A voice processing method, apparatus and electronic device

Abstract

The application provides a speech processing method, device and electronic equipment; the method comprises the following steps: in response to entering a translation mode, obtaining first input audio; the translation mode is used for translating input audio into a target language; the target language is the output language of the translation mode; determining the first input language of the first input audio; if the first input language is the target language, performing speech recognition on the first input audio to obtain a recognition result; outputting the recognition result as the processing result of the translation mode.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and more particularly to a speech processing method, apparatus and electronic device. Background Technology

[0002] In current multi-person conversation scenarios, multiple participants often speak in different languages. Therefore, each participant needs a translation function to quickly and accurately understand the semantics expressed by other participants and translate their statements into the corresponding text in the preset language for output. However, in some scenarios, the results are poor. Summary of the Invention

[0003] This application provides a voice processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a voice processing method, the method comprising: In response to entering translation mode, the first input audio is acquired; translation mode is used to translate the input audio into the target language; the target language is the output language of translation mode. Determine the first input language of the first input audio; If the first input language is the target language, perform speech recognition on the first input audio to obtain the recognition result; The recognition results are output as the processing result of the translation pattern.

[0005] Secondly, embodiments of this application provide a voice processing apparatus, the apparatus comprising: The acquisition module is used to acquire the first input audio in response to entering translation mode; the translation mode is used to translate the input audio into the target language; the target language is the output language of the translation mode; The determination module is used to determine the first input language of the first input audio. The recognition module is used to perform speech recognition on the first input audio if the first input language is the target language, and obtain the recognition result. The first output module is used to output the recognition result as the processing result of the translation mode.

[0006] Thirdly, embodiments of this application provide an electronic device, which includes a processor; The processor is used to respond to the initiation of a multi-person dialogue, enter the translation mode and acquire the first input audio; the translation mode is used to translate the input audio into the target language; the target language is the output language of the translation mode; the first input language of the first input audio is determined; if the first input language is the target language, speech recognition is performed on the first input audio to obtain the recognition result; the recognition result is output as the processing result of the translation mode.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the voice processing method provided in embodiments of this application when executed by a processor.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the voice processing method provided in embodiments of this application. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a speech processing method provided in an embodiment of this application. Figure 1 ; Figure 2 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 2 ; Figure 3 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 3 ; Figure 4 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 4 ; Figure 5 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 5 ; Figure 6 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 6 ; Figure 7 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 7 ; Figure 8 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 8 ; Figure 9 This is a schematic diagram of a multi-user conversation scenario provided in an embodiment of this application; Figure 10 A flowchart illustrating a specific implementation of a speech processing method provided in this application embodiment; Figure 11 This is a logic block diagram of a voice processing device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In cross-language intelligent conferencing scenarios, to reduce the learning cost for users of multiple languages, speech-to-text translation components installed on the user's smart terminal are typically used to translate the user's speech and output the translated text in the user-defined language. This allows users to read the output text to aid their understanding of the speaker's content. However, because translation components usually focus on the semantics of the speech, when the speaker's language is the same as the user-defined language, there can be a significant discrepancy between the translated text and the speaker's direct speech. This can prevent users from understanding the speech through the translated text, especially when the speaker's pronunciation is unclear or their speaking speed is fast. Users may not be able to understand the speech directly from the received audio, potentially causing them to miss important content in the conversation and causing inconvenience.

[0012] To address the issue of inaccurate translated text caused by the audio language being the same as the preset language in the aforementioned translation mode, which could lead to users missing important conversation content, this application provides a speech processing method applicable to devices capable of receiving speech or audio in multi-person conversation scenarios. Figure 1 A flowchart illustrating the interaction of a voice processing method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, this method can be implemented through steps S1 to S4: Step S1: In response to entering translation mode, the first input audio is acquired; translation mode is used to translate the input audio into the target language; the target language is the output language of translation mode.

[0013] In the embodiments of this application, translation mode refers to a working mode in which a smart device translates received audio into a target language and outputs the translated content in the form of text in the target language. This mode can typically be executed by a speech translation component (such as conventional translation software or a translation application interface of a large language model).

[0014] In the embodiments of this application, the target language can be specified by the user, for example, the target language can be specified as English or Chinese; in addition, the default value of the target language can be used directly without any additional setting operation. The default value of the target language can be set by the voice translation component itself. For example, the target language can be set to be consistent with the system language of the operating system running the voice translation component.

[0015] In the embodiments of this application, when a user operates their smart terminal or translation device to enter translation mode, the smart terminal or translation device can respond to entering translation mode and begin acquiring the first input audio.

[0016] The first input audio may include audio transmitted from other devices obtained by the smart terminal or translation device through the Internet or other communication methods (such as Ethernet or LAN); in some scenarios (such as offline multi-person conversations or translation software testing), it may also include audio directly collected by the smart terminal or translation device through a voice input terminal (such as a microphone).

[0017] The first input audio can be a continuously input speech stream, a speech token sampled at a fixed sampling rate, or an audio segment with speech endpoints.

[0018] Step S2: Determine the first input language of the first input audio.

[0019] In the embodiments of this application, the first input language of the first input audio can be determined using a language recognition model or using a language recognition application interface of a large language model.

[0020] The specific steps for determining a language using a language identification model may include: 1) Audio preprocessing; specifically, this may include converting the input audio to a fixed sampling rate (e.g., 16kHz mono or 32kHz mono), as well as noise reduction and endpoint detection.

[0021] 2) Extract speech features; speech features may include acoustic features such as Mel frequency cepstrum coefficients (MFCC) and filter bank features (FBank).

[0022] 3) Input the speech features into the language recognition model and obtain the language prediction probability output by the language recognition model; wherein, the model structure of the language recognition model can be a combination of convolutional neural network and recurrent neural network, or a Transformer network model.

[0023] 4) Post-processing: The language with the highest predicted language probability is identified as the language to be recognized.

[0024] The specific steps for determining the language using the language identification application interface of a large language model are as follows: 1) Obtain the first input audio through the language recognition application interface; 2) The speech recognition application interface performs autonomous processing and language recognition on the first input audio, and finally outputs the recognized language.

[0025] Step S3: If the first input language is the target language, perform speech recognition on the first input audio to obtain the recognition result.

[0026] In the embodiments of this application, Automatic Speech Recognition (ASR) refers to the process of recognizing language information in speech and outputting the recognized content in the form of text. During this process, the language of the language information in the speech is typically kept consistent with the language of the output text.

[0027] Speech recognition of audio can be achieved using the Whisper database, or real-time speech recognition of audio streams can be achieved by combining the Whisper database with microphone callback and real-time monitoring functions.

[0028] After performing speech recognition, the resulting text can be obtained. Additionally, the recognition result may also include the audio generated from the speech-recognized text.

[0029] In the embodiments of this application, since speech recognition can usually only be performed in one language, when the first input language is the target language, the first input language and the output language are consistent, satisfying the speech recognition conditions. Meanwhile, because translation engines focus on semantic expression, they first analyze the semantics of the speech stream before converting it into target language text for output. This can result in significant discrepancies between the translated text and the actual speech. Therefore, the text output accuracy of speech recognition is usually higher than that of speech translation. Thus, in this case, speech recognition can be used to perform speech recognition on the first input audio to obtain the recognition result.

[0030] Step S4: Output the recognition result as the processing result of the translation mode.

[0031] In the embodiments of this application, within the application execution logic of the smart terminal device, when an application is not used for a period of time, the memory space occupied by that application will be cleared and released. When the user wants to use the application again, they need to click to start it again. To avoid the translation mode stopping working and causing the smart terminal to be unable to immediately collect conversation audio and process the audio signal, the recognition result can be used as the processing result of the translation mode. In this way, even if the translation component in the translation mode is not used during speech recognition, the use of the output interface in the translation mode will not cause the translation mode to stop running, thus improving the user experience of the translation mode.

[0032] In terms of specific output format, the recognition result text can be output directly; for users who require audio output (such as users with reading difficulties), or when the speaker's pronunciation is unclear, preset voices can be used to generate output audio in combination with the recognition result text and play the audio while the text is being output; in addition, audio files of the recognition result can be generated and download or playback interfaces can be provided, so that users can download or play the audio files when needed.

[0033] Based on the above embodiments disclosed in this application, after entering the translation mode, when a first input audio with language information consistent with the target language is obtained, speech recognition is performed on the first input audio instead of speech translation, and the result of speech recognition is output as the processing result of the translation mode. This can avoid the problem of inaccurate translation text output caused by the language of the audio being the same as the preset language, and improve the accuracy of the text output of the translation mode.

[0034] In some embodiments, such as Figure 2 As shown, Figure 2 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 2 Step S4 can be achieved through steps S41 to S43: Step S41: In translation mode, perform speech translation on the first input audio to obtain the first translation result.

[0035] In the embodiments of this application, since switching the speech translation component to the speech recognition component will consume more system resources and will also generate a certain delay, in order to avoid frequently switching between speech translation and speech recognition working modes in translation mode, when the first input language of the first input audio is the same as the target language, the difference between the translation result and the recognition result can be compared, and the decision on whether to switch between speech translation and speech recognition can be made based on these differences.

[0036] In the embodiments of this application, when the first input language of the first input audio is the same as the target language, the first input audio can also be translated by speech to obtain a first translation result. Speech translation can be implemented by a translation program or a speech translation component, as described in the relevant embodiments of step S1, which will not be repeated here.

[0037] Step S42: Determine the difference information between the first translation result and the recognition result.

[0038] In embodiments of this application, a first translation result and a recognition result can be compared to obtain difference information between the first translation result and the recognition result. This difference information may include, but is not limited to, semantic differences, keyword differences, and language structure differences.

[0039] For example, if the first translation result is "I like rock climbing" and the recognition result is "I like mountain climbing", the difference information between the first translation result and the recognition result can include semantic differences and keyword differences, while their language structures are the same; if the first translation result is "His wife is a famous singer" and the recognition result is "His wife is a fairly famous singer", the difference information between the first translation result and the recognition result can include semantic differences, keyword differences, and language structure differences.

[0040] Step S43: If the difference information meets the target conditions, the recognition result is output as the processing result of the translation mode.

[0041] In the embodiments of this application, when the difference between the first translation result and the recognition result is sufficiently large or exceeds the user's expectations, the difference can be considered to meet the target condition. For example, the keyword difference condition can be set to 2; when the keyword difference between the first translation result and the recognition result is greater than 2, the difference can be considered to meet the target condition. The semantic difference condition can be set to 90% (cosine similarity of semantic features); when the semantic difference between the first translation result and the recognition result is greater than 90%, the difference can be considered to meet the target condition. In terms of language structure, if the language structures of the first translation result and the recognition result are different, the difference can be considered to meet the target condition. In addition, the user can also set other target conditions to measure the difference between the first translation result and the recognition result.

[0042] In the embodiments of this application, if the difference information meets the target conditions, the first translation result can be considered inaccurate. At this time, the recognition result can be output as the processing result of the translation mode, and the current working component can be switched from the speech translation component to the speech recognition component.

[0043] In some embodiments, if the difference information does not meet the target conditions, it can be considered that the difference between the first translation result and the recognition result is not large, and the first translation result is close to the recognition result. Therefore, in order to avoid the output stuttering caused by switching between performing speech translation and speech recognition, the first translation result can be output as the processing result of the translation mode.

[0044] Based on the above embodiments disclosed in this application, when the language of the input audio is the same as the target language, the translation result and the recognition result can be compared, and the recognition result can be output when the difference information between the translation result and the recognition result meets the target conditions. This can avoid outputting translation results with low accuracy to a certain extent and fully guarantee the accuracy of the output results in the translation mode.

[0045] In some embodiments, such as Figure 3 As shown, Figure 3 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 3 The step S3, "performing speech recognition on the first input audio and obtaining the recognition result," can be achieved through steps S31 and S32: Step S31: Obtain the second input audio; wherein the second input audio is connected to the first input audio, and the generation timestamp of the second input audio is earlier than the generation timestamp of the first input audio.

[0046] In the embodiments of this application, the input audio can be continuously acquired in translation mode. However, when different users speak in sequence, there may be speaking intervals, meaning that there may be periods of inactivity during which no speaking activity occurs. Therefore, blank audio may be acquired. Here, blank audio refers to an audio file that does not contain speech features.

[0047] To determine the input scenario of the first input audio, a second input audio that is connected to the first input audio and whose generation timestamp is earlier than the generation timestamp of the first input audio can be obtained, and it can be determined whether the second input audio conforms to the blank audio generated during the speaking interval.

[0048] Step S32: If the second input audio contains blank audio that meets the conditions, then perform speech recognition on the first input audio to obtain the recognition result.

[0049] In the embodiments of this application, the blank audio that meets the conditions refers to the blank audio contained in the second input audio that is adjacent to the first input audio, and the playback duration (or recording duration) of the blank audio adjacent to the first input audio is greater than the output delay of the translation mode, so as to ensure that the operating system (or session environment) in the translation mode has stopped executing the translation task before the first input audio is acquired.

[0050] For example, assuming the output delay of the translation component in translation mode is 5 seconds, when speaker 1 starts speaking at time 00:00, the user can receive the translated text at time 00:05. If the speaker stops speaking at time 01:20, and no other speaker speaks after that time, the translation component will stop outputting the translated text at time 01:25. If speaker 2 speaks at time 01:24, since the translation component does not stop working, it will continue processing the audio generated by speaker 2. However, if speaker 2 speaks in the target language, the translation component will still process the audio, resulting in a sudden switch from outputting translated text to outputting recognized text. In this case, the playback duration (or recording duration) of the blank audio is 4 seconds, which clearly does not meet the condition. However, when speaker 2 starts speaking at time 01:30, the translation component has already stopped outputting the translated text, and the playback duration of the blank audio is 10 seconds, meeting the condition. Therefore, it is no longer necessary to use the translation component for speech translation.

[0051] In the embodiments of this application, if the second input audio contains blank audio that meets the conditions, it indicates that the current system or translation mode is not executing a translation task. At this time, if the first input language of the first input audio is the same as the target language, since the translation mode has stopped executing the translation task, the switching process from translation to recognition can be skipped, and the first input audio can be directly subjected to speech recognition to obtain the recognition result.

[0052] Based on the above embodiments disclosed in this application, when there is a blank audio preceding the first input audio and the translation mode has stopped performing the translation task, speech recognition can be performed on the first input audio in the target language. This can eliminate one translation-to-recognition switching process and improve the accuracy of the output results when the input language of the input audio is the same as the target language.

[0053] In some embodiments, such as Figure 4 As shown, Figure 4 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 4 The step S3, "perform speech recognition on the first input audio and obtain the recognition result", can also be achieved through steps S33 and S34: Step S33: Determine the language style of the first input audio.

[0054] In the embodiments of this application, considering that language style may lead to differences between translation and recognition results—for example, a colloquial language style may further reduce translation accuracy, i.e., increase the difference between translation and recognition results, while a formal language style may make the translation and recognition results closer, i.e., reduce the difference between translation and recognition results—then, if the first input language is the target language, the language style of the first input audio can be further determined to decide whether to maintain translation or switch to recognition based on the language style.

[0055] Step S34: If the language style is colloquial, then perform speech recognition on the first input audio to obtain the recognition result.

[0056] In the embodiments of this application, if the language style of the first input audio is colloquial, speech recognition can be performed on the first input audio to obtain the recognition result, so as to avoid the problem of inaccurate translation results caused by the processing method of maintaining translation.

[0057] It should be noted that in steps S31 and S32, since the conditions in these steps already indicate that there is no ongoing translation task, there is no need to determine whether to maintain translation or switch to recognition. However, it may be necessary to switch back from recognition mode to translation mode if the language of subsequent input audio differs from the target language. Therefore, for steps S33 and S34, a precondition can be added: before the first input audio, the audio following the first input audio is not blank audio, or the recording duration (or playback duration) of the following blank audio is less than the output delay of the translation mode. This is done to minimize the problem of frequent switching between translation and recognition modes. However, since there is a possibility of switching from recognition to translation after processing the input audio using speech recognition, the processes of steps S33 and S34 can also serve as a supplement to the processes of steps S31 and S32. That is, after determining that there is blank speech that meets the conditions before the first input audio, the language style of the first input audio is further determined. In this way, when the language style is colloquial, it can be determined that it is necessary to use the speech processing method of speech recognition in the current scenario, without having to save the subsequent switching process from recognition to translation.

[0058] Based on the above embodiments disclosed in this application, speech recognition can be performed on the first input audio when the language style of the first input audio is colloquial, thereby obtaining the recognition result and improving the accuracy of the processing result of the colloquial input audio.

[0059] In some embodiments, such as Figure 5 As shown, Figure 5 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 5 The above-mentioned speech processing method may further include steps S5 and S6: Step S5: If the language style is written, in translation mode, perform speech translation on the first input audio to obtain the first translation result.

[0060] In the embodiments of this application, as described in the above embodiments, when speech recognition is enabled to process input audio, a switching process from speech recognition to speech translation will be added in subsequent processes. To save on this switching process in subsequent audio processing, the first input audio can be translated into speech in translation mode, provided the language style is formal and the translation result is close to the recognition result, thus obtaining a first translation result.

[0061] Step S6: Output the first translation result as the processing result of the translation mode.

[0062] In the embodiments of this application, when the language style is formal, the translation result is close to the recognition result, indicating that the translation result can have high accuracy. After obtaining the first translation result, the first translation result can be output as the processing result of the translation mode.

[0063] Based on the above embodiments disclosed in this application, when the language style is written, the speech translation method can be maintained unchanged, and the output result can be used as the processing result of the translation mode. This can ensure the accuracy of the output result while avoiding the subsequent switching process from recognition mode to translation mode, further increasing the immediacy of the output result and reducing the output latency.

[0064] In some embodiments, the difference information is the error rate, such as... Figure 6 As shown, Figure 6 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 6 The above-mentioned speech processing method may further include: Step S44: Determine the number of characters recognized in the recognition result.

[0065] In the embodiments of this application, the error rate refers to the ratio of the number of distinguishable characters in the first translation result relative to the number of characters in the recognition result to the total number of characters in the recognition result. To determine the number of distinguishable characters, the number of recognized characters in the recognition result can first be determined. For example, if the first input language of the first input audio is Chinese, then one Chinese character represents one character; if the first input language is English, then one letter represents one character.

[0066] Step S45: Determine the number of distinguishing characters in the first translation result relative to the recognition result.

[0067] In the embodiments of this application, the number of distinguishable characters refers to the number of characters in the first translation result that are different from the recognized characters in the recognition result. For example, if the first translation result is "I like to go mountain climbing", and the recognition result is "I like to climb mountains", then the distinguishable character is "go", and the number of distinguishable characters is 1; if the first translation result is "I like to go rock climbing", and the recognition result is "I like to climb mountains", then the distinguishable characters are "go", "climb", and "rock", and the number of distinguishable characters is 3.

[0068] Step S46: Based on the number of distinguished characters and the number of recognized characters, determine the error rate of the first translation result relative to the recognition result.

[0069] In the embodiments of this application, the error rate of the first translation result relative to the recognition result can be calculated based on the number of distinguished characters and the number of recognized characters.

[0070] Using the above example, if the first translation result is "I like to go mountain climbing", and the recognition result is "I like to climb mountains", then the number of recognized characters is 5, and the error rate is 20%; if the first translation result is "I like to go rock climbing", and the recognition result is "I like to climb mountains", then the error rate is 60%.

[0071] Step S47: If the error rate is greater than or equal to the target threshold, determine that the difference information meets the target condition.

[0072] In the embodiments of this application, if the error rate is greater than or equal to the target threshold, the difference between the first translation result and the recognition result can be considered to meet the target condition. At this time, the processing mode of the input audio can be switched to the recognition mode. The target threshold can be set by the user according to their needs, for example, it can be set to 20% or 5%.

[0073] In one possible embodiment, if the word error rate is less than the target threshold, it can be considered that the difference information does not meet the target condition. In this case, the processing method of the input audio can be kept unchanged as speech translation, so as to reduce the number of times the switching process from speech translation to speech recognition is executed.

[0074] Based on the above embodiments disclosed in this application, the error rate can be used as difference information, and the error rate can be used to determine whether the first translation result and the recognition result meet the target conditions, which can improve the accuracy and reliability of the target conditions to a certain extent.

[0075] In some embodiments, the distinguishing characters between the first translation result and the recognition result include redundant characters, missing characters, and replaced characters; such as Figure 7 As shown, Figure 7A flowchart illustrating a speech processing method provided in this application embodiment. Figure 7 Step S46 can be achieved through steps S461 to S463: Step S461: Determine the first character count of the redundant character, the second character count of the missing character, and the third character count of the replacement character; the sum of the first character count, the second character count, and the third character count is equal to the number of distinguishing characters.

[0076] In embodiments of this application, the distinguishing characters may include redundant characters, missing characters, and replacement characters. Replacement characters refer to different characters in the same structural position within the language structure of the first translation result and the recognition result. For example, if the first translation result is "I like climbing mountains" and the recognition result is "I want to climb mountains," then the replacement character is "like." Redundant characters refer to characters present in the first translation result but not in the recognition result. For example, if the first translation result is "I like going to climb mountains" and the recognition result is "I like climbing mountains," then the redundant character is "go." Missing characters refer to characters not present in the first translation result but present in the recognition result. For example, if the first translation result is "I like swimming" and the recognition result is "I prefer swimming," then the missing character is "prefer."

[0077] In the embodiments of this application, based on the above meaning, the replacement character also belongs to the category of redundant characters or missing characters. Therefore, in the actual process, the replacement character between the first translation result and the recognition result can be determined first, and then the redundant characters and missing characters can be determined from the remaining characters. That is to say, once a character is identified as a replacement character, that character will no longer be identified as a redundant character or a missing character.

[0078] Step S462: Determine the first weight value of the first number of characters, the second weight value of the second number of characters, and the third weight value of the third number of characters respectively; the first weight value and the second weight value are both less than the third weight value.

[0079] In embodiments of this application, the user can set a first weight value for the first number of characters, a second weight value for the second number of characters, and a third weight value for the third number of characters. Typically, replacing characters can maximally disrupt the semantics of language. For example, the meanings of "His wife is a fairly famous singer" and "His wife is a famous singer" are completely different. Therefore, the third weight value can usually be greater than the first and second weight values.

[0080] Step S463: Based on the first weight value, the first number of characters, the second weight value, the second number of characters, the third weight value, the third number of characters, and the number of recognized characters, determine the error rate of the first translation result relative to the recognition result.

[0081] In the embodiments of this application, the error rate of the first translation result relative to the recognition result can be calculated based on the first weight value, the first number of characters, the second weight value, the second number of characters, the third weight value, the third number of characters, and the number of recognized characters. The calculation method is shown in Formula 1 below: (Formula 1); in, Indicates the number of characters recognized; Indicates the number of the first character; Indicates the number of the second character; Indicates the number of the third character; Indicates the first weight value; This represents the second weight value; This represents the third weight value. Wherein, we can set... , and The normalized weight values ​​are, i.e. .

[0082] Based on the above embodiments disclosed in this application, weights can be added to the error rate, and the weight value of the replaced character can be higher than the weight value of other distinguishing characters. To a certain extent, the error rate can more accurately indicate the difference between the first translation result and the recognition result, thereby improving the accuracy and reliability of the error rate.

[0083] In some embodiments, such as Figure 8 As shown, Figure 8 A flowchart illustrating a speech processing method provided in this application embodiment. Figure 8 The above-mentioned speech processing method may further include steps S7 to S9: Step S7: Obtain the third input audio, the input time of the third input audio is later than the input time of the first input audio.

[0084] In embodiments of this application, a third input audio can be obtained after the first input audio. The third input audio can be continuous with the first input audio or separated by blank audio.

[0085] Step S8: Determine that the input language of the third input audio is different from the target language, and perform translation processing on the third input audio to obtain the second translation result.

[0086] In the embodiments of this application, after acquiring the third input audio, a language recognition algorithm can be used to identify the input language of the third input audio. If the input language of the third input audio is different from the target language, since the first input audio was processed using speech recognition before the third input audio, there is a switching process to switch the speech processing method from speech recognition to speech translation. After switching the speech processing method from speech recognition to speech translation, the third input audio can be translated to obtain the second translation result.

[0087] Step S9: Output the second translation result as the processing result of the translation mode.

[0088] In the embodiments of this application, the second translation result can be output as the processing result of the translation mode. Similarly, the output method can be text, or a combination of text and speech. The speech is played simultaneously while the text is output. To avoid the speech information received by the user being too cluttered, an output speech file can also be provided for the user to actively click to play or download.

[0089] Based on the above embodiments disclosed in this application, when a third input audio with a different language than the target language is obtained after obtaining the first input audio, the third input audio can be processed using a speech translation method, which can improve the accuracy of the output results in the translation mode to a certain extent.

[0090] This application also provides an electronic device, which includes a processor; The processor is used to respond to the initiation of a multi-person dialogue, enter the translation mode and acquire the first input audio; the translation mode is used to translate the input audio into the target language; the target language is the output language of the translation mode; the first input language of the first input audio is determined; if the first input language is the target language, speech recognition is performed on the first input audio to obtain the recognition result; the recognition result is output as the processing result of the translation mode.

[0091] In some embodiments, the processor can also be used to execute the steps in the above method embodiments to achieve the same technical effect as the method embodiments, which will not be repeated here.

[0092] The following describes the application of the processing method provided in the embodiments of this application in a real-world scenario.

[0093] On smart devices such as smartphones, computers, tablets, and wearable devices, speech-to-text conversion has become an essential function. Especially in smart conferencing scenarios, multilingual speech-to-text functionality is necessary to provide users with a better experience.

[0094] Typical application scenario: User A starts a multi-person remote online meeting on their device, during which conversations will be conducted using multiple languages ​​(e.g., ...). Figure 9 As shown in the diagram, User A enables the Speech-to-Text function on their device, setting the target language to English. Regardless of the languages ​​spoken by other participants, the speech will be transcribed into English. User A's computer runs meeting transcription software, which uses a multilingual speech translation engine to convert different languages' speech into target language text. When the source language (e.g., User C speaking English) and the target language are the same, the text presented to User A should be as accurate as possible, matching the pronunciation. However, because the translation engine prioritizes semantic expression, it first analyzes the semantics of the speech stream before converting it into target language text for output. Therefore, the translation engine's result may only match the meaning, with some discrepancy from the pronunciation. For example, a Chinese speech might read, "His wife is a fairly famous singer," while the translation engine's output text might be, "His wife is a famous singer." Therefore, when the source speech (User C's speech) and the target language (User A's set text language category) are the same, a speech recognition engine is needed to provide text that matches the pronunciation.

[0095] To achieve this functionality, this proposal suggests a multilingual speech-to-text implementation scheme. When the source and target languages ​​are the same, the difference between the translated and recognized text is calculated. If the difference is less than a threshold (which has little impact on user experience), the switch from the translation engine to the recognition engine is not performed. Only when the difference exceeds a threshold sufficient to affect user experience is the switch performed. By reducing the number of switches, a smoother user experience is provided.

[0096] The technical advantages of this solution are as follows: 1) The switching from translation engine to recognition engine is done dynamically, which can take into account the differences in text caused by different factors, such as speaking style, language category, environmental noise, etc.

[0097] 2) Compared with traditional implementation solutions, it reduces the number of switching operations, ensuring a smooth user experience as much as possible.

[0098] 3) Only one engine needs to be running, avoiding the resource consumption or increased costs caused by multiple engines working simultaneously.

[0099] The specific plan is as follows: To achieve the function of transcribing source language speech into target language text in multiple languages, we adopt a cascaded process of language identification, speech recognition, and translation engine. The first step is language identification, determining the language type (LID) of the source speech. The second step involves calling either the translation or recognition engine. If the source language and the target language set by the user are inconsistent, the speech translation engine is invoked; otherwise, the automatic speech recognition engine (ASR) is switched to support the transcription process from multilingual speech to target language text. When the source and target are detected to be different languages ​​again, the recognition engine switches to the translation engine.

[0100] During this process, the constant switching between the recognition and translation engines can cause stuttering and speech loss, impacting the user experience. We observed that different speech-to-text translation models exhibit varying performance due to differences in training data and strategies. In some languages, the difference between speech translation and recognition results is minimal when the source and target languages ​​are identical, while in others, the difference can be significant. This difference is influenced by factors such as the translation engine, language type, and speaking style. For example, colloquial speech shows a larger difference between translation and recognition, while standard written language shows a smaller difference. Therefore, this solution leverages this difference to propose a dynamic measurement of the discrepancy between translation and recognition, reducing the problem of frequent engine switching.

[0101] Specific steps are as follows Figure 10 As shown.

[0102] Step S1001: Obtain the input speech stream.

[0103] Step S1002: Language recognition is performed on the input speech stream.

[0104] That is, determine the language category of the speech stream. Once the language category information is obtained, execute step S1003.

[0105] Step S1003, consistency judgment.

[0106] Specifically, it determines whether the language category information of the input speech stream is consistent with the target language (the output language set by the user). If they are consistent, steps S1004 and S1005 are executed; if they are inconsistent, only step S1005 is executed, and step S1004 is not executed.

[0107] Step S1004: Start the speech recognition engine.

[0108] The speech recognition engine is then activated to perform speech recognition on the input speech stream and obtain the recognized text. After step S1005 is completed, step S1006 is executed.

[0109] Step S1005: Invoke the speech translation engine.

[0110] That is, the speech translation engine is invoked to translate the input speech stream to obtain the translated text, and then step S1006 is executed after step S1004 is completed.

[0111] Step S1006: Measure the difference between translated text and recognized text.

[0112] That is, compare the translated text and the recognized text, measure the difference between the translated text and the recognized text by the difference degree, and then perform step S1007.

[0113] Step S1007: Determine whether the difference is greater than the threshold.

[0114] When the difference is greater than the threshold, proceed to step S1008; when the difference is less than or equal to the threshold, proceed to step S1009.

[0115] Step S1008: Output the recognized text.

[0116] Switch the speech translation engine to the speech recognition engine and output the recognized text.

[0117] Step S1009: Output the translated text.

[0118] Turn off the speech recognition engine and output the translated text.

[0119] The specific execution logic is as follows: 1) Determine the language category of the speech stream.

[0120] 2) If the language category of the speech stream changes and is inconsistent with the target text language set by the user, the speech translation engine will continue to be called (Note: After the multilingual speech translation engine is initialized, it is not necessary to restart the engine when the target language changes).

[0121] 3) If the language category of the audio stream changes and matches the target text set by the user, the background speech recognition engine will start, and the text displayed will still be the translation result.

[0122] 4) Calculate the difference between the recognized and translated texts. This proposal suggests using the weighted word error rate (WER) as a metric for measuring the difference. Specifically: Using the identified text as the reference text and the translated text as the hypothetical result, a weighted error rate is calculated between the reference text and the hypothetical result, including insertion error (I), deletion error (D), and substitution error (S). Among these, substitution error best reflects the text differences. For example, the difference between the reference text "I like climbing mountains" and the hypothetical result "I like rock climbing" has the greatest impact on user experience; the difference between the reference text "I like climbing mountains" and the hypothetical result "I like going to climb mountains" is more acceptable to users.

[0123] The design's difference score, DiffScore, is calculated as w × S + I + D, where w is greater than 1 (essentially setting the weight of replacement errors to be greater than that of insertion and deletion errors). When DiffScore < a preset threshold, the translation result continues to be used, and the recognition engine exits. During this process, the user will not experience any content loss or lag; when DiffScore > the preset threshold, the recognition engine switches back to operation.

[0124] In the above solution, each reduction in the switching from the translation engine to the recognition engine also reduces the process of switching back from recognition to translation, thus reducing the occurrence of two stutters.

[0125] Running both translation and recognition engines simultaneously consumes excessive computing resources for edge-based models and increases costs for cloud-based models, especially for commercial engines. Therefore, this solution aims to provide users with a smooth user experience while minimizing computational resource consumption and costs.

[0126] This application provides a voice processing device, such as... Figure 11 As shown, Figure 11 This application provides a logic block diagram of a voice processing device 1100, which may include: The first acquisition module 1101 is used to acquire the first input audio in response to entering the translation mode; the translation mode is used to translate the input audio into a target language; the target language is the output language of the translation mode. The first determining module 1102 is used to determine the first input language of the first input audio; The recognition module 1103 is used to perform speech recognition on the first input audio if the first input language is the target language, and obtain the recognition result; The first output module 1104 is used to output the recognition result as the processing result of the translation mode.

[0127] In some embodiments, the first output module 1104 includes: The first translation submodule is used to perform speech translation on the first input audio in translation mode to obtain the first translation result; The first determining submodule is used to determine the difference information between the first translation result and the recognition result; The first output submodule is used to output the recognition result as the processing result of the translation mode if the difference information meets the target conditions.

[0128] In some embodiments, the identification module 1103 includes: The acquisition submodule is used to acquire the second input audio; wherein the second input audio is contiguous with the first input audio, and the generation timestamp of the second input audio is earlier than the generation timestamp of the first input audio; The first recognition submodule is used to perform speech recognition on the first input audio if the second input audio contains blank audio that meets the conditions, and obtain the recognition result.

[0129] In some embodiments, the identification module 1103 includes: The second determining submodule is used to determine the language style of the first input audio; The second recognition submodule is used to perform speech recognition on the first input audio if the language style is colloquial, and obtain the recognition result.

[0130] In some embodiments, the voice processing device 1100 further includes: The second translation submodule is used to perform speech translation on the first input audio in translation mode if the language style is written, to obtain the first translation result; The second output submodule is used to output the first translation result as the processing result of the translation mode.

[0131] In some embodiments, the difference information is the word error rate, and the speech processing device 1100 further includes: The second determining module is used to determine the number of characters recognized in the recognition result; The third determining module is used to determine the number of distinguishing characters in the first translation result relative to the recognition result; The fourth determining module is used to determine the error rate of the first translation result relative to the recognition result based on the number of distinguished characters and the number of recognized characters; The fifth determination module is used to determine whether the difference information meets the target condition if the error rate is greater than or equal to the target threshold.

[0132] In some embodiments, the distinguishing characters between the first translation result and the recognition result include redundant characters, missing characters, and replaced characters; the fourth determining module includes: The third determination submodule is used to determine the first character count of redundant characters, the second character count of missing characters, and the third character count of replacement characters; the sum of the first character count, the second character count, and the third character count is equal to the number of distinguishing characters; The fourth determination submodule is used to determine the first weight value of the first number of characters, the second weight value of the second number of characters, and the third weight value of the third number of characters, respectively; the first weight value and the second weight value are both less than the third weight value; The fifth determination submodule is used to determine the error rate of the first translation result relative to the recognition result based on the first weight value, the first number of characters, the second weight value, the second number of characters, the third weight value, the third number of characters, and the number of recognized characters.

[0133] In some embodiments, the voice processing device 1100 further includes: The second acquisition module is used to acquire the third input audio, the input time of the third input audio being later than the input time of the first input audio; The translation module is used to determine that the input language of the third input audio is different from the target language, and to translate the third input audio to obtain a second translation result; The second output module is used to output the second translation result as the processing result of the translation mode.

[0134] It should be noted that, in the embodiments of this application, if the above-described processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0135] This application provides another electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0136] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0137] This application provides a computer program including computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes some or all of the steps in the above-described method.

[0138] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0139] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0140] Figure 12 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 12 As shown, the hardware entity of the electronic device 1200 includes a processor 1201 and a memory 1202, wherein the memory 1202 stores a computer program that can run on the processor 1201, and the processor 1201 executes the program to implement the steps in the method of any of the above embodiments.

[0141] The memory 1202 stores computer programs that can run on the processor. The memory 1202 is configured to store instructions and applications that can be executed by the processor 1201. It can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1201 and various modules in the electronic device 1200. It can be implemented by flash memory or random access memory (RAM).

[0142] When processor 1201 executes a program, it implements the steps of any of the above-mentioned processing methods. Processor 1201 typically controls the overall operation of electronic device 1200.

[0143] This application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the speech processing method as described in any of the above embodiments.

[0144] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0145] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0146] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0147] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0150] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0152] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0153] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A speech processing method, the method comprising: In response to entering translation mode, acquire the first input audio; The translation mode is used to translate the input audio into the target language; The target language is the output language of the translation mode; Determine the first input language of the first input audio; If the first input language is the target language, perform speech recognition on the first input audio to obtain the recognition result; The recognition result is output as the processing result of the translation pattern.

2. The method according to claim 1, wherein outputting the recognition result as the processing result of the translation pattern includes: In the translation mode, the first input audio is translated into speech to obtain a first translation result; Determine the difference information between the first translation result and the recognition result; If the difference information meets the target conditions, the recognition result is output as the processing result of the translation mode.

3. The method according to claim 1, wherein performing speech recognition on the first input audio to obtain a recognition result includes: Obtain a second input audio; wherein the second input audio is contiguous with the first input audio, and the generation timestamp of the second input audio is earlier than the generation timestamp of the first input audio; If the second input audio contains blank audio that meets the conditions, then speech recognition is performed on the first input audio to obtain the recognition result.

4. The method according to claim 1, wherein performing speech recognition on the first input audio to obtain a recognition result includes: Determine the language style of the first input audio; If the language style is colloquial, then speech recognition is performed on the first input audio to obtain the recognition result.

5. The method according to claim 4, further comprising: If the language style is a written style, in the translation mode, the first input audio is translated by speech to obtain a first translation result; The first translation result is output as the processing result of the translation mode.

6. The method according to claim 2, wherein the difference information is the error rate, and the method further comprises: Determine the number of characters recognized in the recognition result; Determine the number of distinguishable characters in the first translation result relative to the recognition result; Based on the number of distinguished characters and the number of recognized characters, the error rate of the first translation result relative to the recognition result is determined; If the error rate is greater than or equal to the target threshold, the difference information is determined to meet the target condition.

7. The method according to claim 6, wherein the distinguishing characters of the first translation result relative to the recognition result include redundant characters, missing characters, and replaced characters; and determining the error rate of the first translation result relative to the recognition result based on the number of distinguishing characters and the number of recognized characters includes: The number of the first character of the redundant character, the number of the second character of the missing character, and the number of the third character of the replacement character are determined respectively; The sum of the first character count, the second character count, and the third character count is equal to the number of distinguishing characters; A first weight value for the first number of characters, a second weight value for the second number of characters, and a third weight value for the third number of characters are determined respectively; the first weight value and the second weight value are both less than the third weight value; Based on the first weight value, the first number of characters, the second weight value, the second number of characters, the third weight value, the third number of characters, and the number of recognized characters, the error rate of the first translation result relative to the recognition result is determined.

8. The method according to any one of claims 1 to 7, wherein the method further comprises: Obtain a third input audio, wherein the input time of the third input audio is later than the input time of the first input audio; If the input language of the third input audio is determined to be different from the target language, the third input audio is translated to obtain a second translation result. The second translation result is output as the processing result of the translation mode.

9. A voice processing apparatus, the apparatus comprising: The first acquisition module is used to acquire the first input audio in response to entering translation mode; The translation mode is used to translate the input audio into the target language; The target language is the output language of the translation mode; The first determining module is used to determine the first input language of the first input audio; The recognition module is configured to perform speech recognition on the first input audio if the first input language is the target language, and obtain a recognition result; The first output module is used to output the recognition result as the processing result of the translation mode.

10. An electronic device, the electronic device comprising a processor; in, The processor is used to respond to the initiation of a multi-person dialogue, enter translation mode, and acquire the first input audio; The translation mode is used to translate the input audio into the target language; The target language is the output language of the translation mode; Determine the first input language of the first input audio; If the first input language is the target language, perform speech recognition on the first input audio to obtain a recognition result; output the recognition result as the processing result of the translation mode.

Images