Cross-language speech recognition method and device, electronic equipment and storage medium
By utilizing the source language acoustic model of a high-resource language to generate phoneme sequences and converting them into cross-language conversion prompts for input into a large language model, the problem of wasted computational resources in cross-language speech recognition is solved, and efficient target language text recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 镁佳(北京)科技有限公司
- Filing Date
- 2026-05-13
- Publication Date
- 2026-06-16
AI Technical Summary
Existing technologies for cross-language speech recognition require target language speech annotation data to update or retrain acoustic models, resulting in wasted computing resources and low deployment efficiency, especially in low-resource language scenarios where effective recognition is difficult.
Phoneme sequences are generated using the source language acoustic model of a high-resource language, and then input into a large language model through cross-language conversion prompts to achieve target language text recognition, avoiding reliance on target language speech annotation data and retraining of the acoustic model.
It enables cross-language speech recognition without target language speech annotation data, reducing computing resource consumption and deployment costs while improving recognition efficiency.
Smart Images
Figure CN122224147A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech recognition technology, specifically to cross-language speech recognition methods, devices, electronic devices, and storage media. Background Technology
[0002] Automatic Speech Recognition (ASR) technology aims to convert human speech signals into corresponding text content. Current mainstream speech recognition systems are typically built independently for a single target language, and the training and adaptation of their acoustic models rely on speech-text pairing data of the target language.
[0003] In scenarios requiring support for new languages or low-resource languages, the relevant technologies have the following drawbacks: Regardless of whether de novo training, multilingual joint training, or transfer fine-tuning strategies are employed, a certain amount of target language speech annotation data is required to participate in the parameter updates or adaptation training of the acoustic model. For languages with scarce or no annotation data, the acoustic model struggles to obtain effective training supervision, leading to model failure to converge or recognition performance falling short of practical requirements. Furthermore, each expansion to a new language necessitates restarting the model training process, preventing the direct reuse of deployed source language acoustic model assets, resulting in wasted computational resources and low engineering deployment efficiency.
[0004] Therefore, how to achieve cross-language speech recognition without requiring target language speech annotation data or retraining acoustic model parameters has become a pressing technical problem in this field. Summary of the Invention
[0005] This application provides a cross-language speech recognition method, apparatus, electronic device, and storage medium to solve the problem of how to achieve cross-language speech recognition without completely eliminating the need for target language speech annotation data and retraining acoustic model parameters.
[0006] Firstly, this application provides a cross-language speech recognition method, which includes: Acquire the speech signal to be recognized in the target language; The speech signal to be recognized is input into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language. Based on the phoneme sequence of the source language, construct cross-language conversion prompt information; The cross-language conversion prompts are input into a large language model to generate target language text recognition results corresponding to the speech signal to be recognized. The generation of target language text recognition results does not require the use of speech-text annotation data of the target language for model training or parameter updates.
[0007] In one optional implementation, the speech signal to be recognized is input into a source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized, characterized in that it includes: The speech signal to be recognized is input into the source language acoustic model. During the process of receiving the speech signal to be recognized and performing forward inference, the model parameters of the source language acoustic model remain frozen. Obtain the phoneme sequence output by the acoustic model of the source language that conforms to the pronunciation rules of the first language.
[0008] In one alternative implementation, the source language acoustic model is an end-to-end acoustic model based on a Transformer encoder architecture or a Conformer encoder architecture.
[0009] In one alternative implementation, cross-language translation prompts are constructed based on the source language phoneme sequence, including: The source language phoneme sequence is concatenated with a preset natural language instruction template to form a prompt text that instructs the large language model to perform a cross-language transcription task from the phonemes of the first language to the target language text.
[0010] In one alternative implementation, the cross-language conversion prompt information further includes at least one set of example data pairs, which contain example source language phoneme sequences and their corresponding correct target language text, to guide the large language model to generate target language text recognition results based on context learning.
[0011] In an alternative implementation, the method further includes, before inputting cross-language translation prompts into the large language model: The training process of the large language model includes pre-training of the autoregressive language model based on multilingual text corpus. The multilingual text corpus includes text data corresponding to the source language and text data corresponding to the target language. However, the training process of the large language model does not use speech data or speech-text pairing data of the target language.
[0012] Secondly, this application provides a cross-language speech recognition device, characterized in that the device comprises: The speech acquisition module is used to acquire the speech signal to be recognized in the target language; The phoneme generation module is used to input the speech signal to be recognized into the source language acoustic model and generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language. The prompt building module is used to construct cross-language translation prompts based on the source language phoneme sequence; The text generation module is used to input cross-language conversion prompts into a large language model to generate target language text recognition results corresponding to the speech signal to be recognized. The generation process of target language text recognition results does not require the use of speech-text annotation data of the target language for model training or parameter updates.
[0013] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the cross-language speech recognition method of the first aspect or any corresponding embodiment described above.
[0014] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the cross-language speech recognition method of the first aspect or any corresponding embodiment described above.
[0015] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the cross-language speech recognition method of the first aspect or any corresponding embodiment described above.
[0016] According to the cross-language speech recognition method provided in this application, compared with the prior art, the following beneficial technical effects can be achieved: It acquires the speech signal to be recognized in the target language, so that the cross-language speech recognition processing flow can be started without relying on the target language annotation data; it inputs the speech signal to be recognized into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language. The amount of speech-text annotation data resources of the first language is greater than that of the target language. Therefore, the acoustic model assets of the existing high-resource language can be directly reused as a fixed acoustic feature extractor, avoiding the need to re-collect annotation data or retrain the acoustic model for the target language, thus reducing acoustic complexity. This reduces computational resource consumption and deployment costs at the source language level. Based on the source language phoneme sequence, cross-language conversion prompts are constructed, transforming the acoustic recognition problem into a text-to-text cross-language transcription task. This allows the subsequent large language model to utilize its pre-trained cross-language semantic alignment capabilities to generate target language text. The cross-language conversion prompts are input into the large language model to generate target language text recognition results corresponding to the speech signal to be recognized. The generation of target language text recognition results does not require the use of target language speech-text annotation data for model training or parameter updates. Thus, cross-language speech recognition is achieved without the need for target language speech annotation data or retraining of acoustic model parameters, reducing deployment costs and computational resource consumption for low-resource languages. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a flowchart illustrating a cross-language speech recognition method according to an embodiment of this application; Figure 3 This is a flowchart illustrating another cross-language speech recognition method according to an embodiment of this application; Figure 4 This is a structural block diagram of a cross-language speech recognition device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0021] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0022] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, application 101 is installed in terminal device 110, and user 130 can interact with application 101 through terminal device 110 and / or access device of terminal device 110.
[0023] For example, application 101 can be any application that provides question-and-answer related services. For instance, application 101 could be a question-and-answer interactive application, such as a text-to-text application, an image-to-text application, etc. Figure 1 In the application scenario shown, if application 101 is active, the terminal device 110 can display the interface 102 of application 101. The interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.
[0024] In some embodiments, terminal device 110 is communicatively connected to server 120 to provide services to application 101. Terminal device 110 may be a mobile terminal, fixed terminal, or portable terminal, etc., including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.
[0025] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this application.
[0026] The embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present; no limitations are imposed on the embodiments of this application. Furthermore, the embodiments are primarily described below with reference to terminal device 110. It should be understood that the actions described relative to terminal device 110 can be performed by application 101 on terminal device 110, or can be performed by application 101 in conjunction with its server (e.g., server 120).
[0027] In related technologies, cross-language speech recognition attempts to leverage high-resource languages to assist low-resource language recognition tasks. A typical approach is to construct a multilingual shared acoustic encoder, mixing data from multiple languages during the training phase to learn a universal acoustic representation. However, when supporting new languages, this approach still requires incorporating the target language's speech data into the acoustic model's training or fine-tuning process. For languages with extremely scarce or even nonexistent speech annotation data, the acoustic model lacks effective training supervision signals, making it difficult to learn the acoustic feature distribution of that language, resulting in poor performance in downstream recognition tasks. Furthermore, each expansion to a new language requires restarting the acoustic model's training process; the deployed source language acoustic model cannot be directly reused as a fixed feature extractor, leading to a waste of computational resources.
[0028] Another related technique achieves cross-linguistic acoustic adaptation by constructing a phoneme mapping table between the source and target languages. Specifically, after obtaining the output of the source language acoustic model, it is converted to the phoneme space of the target language using manually defined mapping rules. However, this approach heavily relies on linguistic expertise, and the mapping rules are usually designed for specific language pairs. When the two languages differ significantly in phonological systems, syllable structures, or pronunciation methods, forced mapping can introduce substantial acoustic representation biases, making it impossible to guarantee recognition accuracy.
[0029] Furthermore, the participation of large language models in cross-lingual speech recognition tasks is limited in related technologies. They are typically used only as independent text post-processing modules to perform linguistic error correction or refinement on the generated initial recognition text. The cross-lingual semantic alignment knowledge learned by large language models during the pre-training phase from massive multilingual text corpora is not fully utilized in the acoustic decoding stage, resulting in a mismatch between the model's capabilities and the core task of cross-lingual speech recognition.
[0030] According to an embodiment of this application, a cross-language speech recognition method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0031] This embodiment provides a cross-language speech recognition method that can be used on the aforementioned mobile terminals, such as mobile phones and tablet computers. Figure 2 This is a flowchart of a cross-language speech recognition method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the speech signal to be recognized in the target language.
[0032] Specifically, the target language refers to the language type of the text to be output in this speech recognition task, such as Swahili, Burmese, German, and other low-resource languages. The speech signal to be recognized refers to the raw audio data containing the speech content of the target language, which can be obtained through methods such as real-time acquisition via microphone, reading audio files from storage media, or receiving network audio streams. For example, a user can record a segment of English speech using the recording function of their terminal device as the speech signal to be recognized.
[0033] In this step, by acquiring the speech signal to be recognized in the target language, we provide an input basis for subsequent cross-language speech recognition by reusing the source language acoustic model and the large language model without relying on the target language speech-text annotation data.
[0034] In one possible implementation, obtaining the speech signal to be recognized in the target language includes: receiving real-time speech data entered by the user through the voice input interface of the terminal device, and performing preprocessing operations such as noise reduction and format conversion on the speech data to obtain the speech signal to be recognized that meets the input requirements of the subsequent acoustic model.
[0035] Step S202: Input the speech signal to be recognized into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language.
[0036] Specifically, the source language acoustic model refers to a neural network model that has been trained using speech-text annotation data in the first language and can map speech signals to a sequence of pronunciation symbols in the first language. For example, an end-to-end acoustic model based on the Transformer or Conformer architecture. The source language phoneme sequence refers to a discrete symbol sequence output by the source language acoustic model that conforms to the pronunciation rules of the first language. For example, the Chinese pinyin sequence "ha lou wo de". The first language refers to a resource-rich language with a large-scale open-source speech dataset, such as high-resource languages like Chinese and English. Speech-text annotation data refers to paired training samples that contain speech signals and their corresponding text transcriptions. For example, a Chinese speech audio and its corresponding text annotation "你好世界". The resource volume of speech-text annotation data refers to the total scale of annotated speech data available for model training, usually measured in hours of duration or the number of samples.
[0037] In this step, by inputting the speech signal to be recognized in the target language into the source language acoustic model trained using the high-resource first language, a phoneme sequence that conforms to the pronunciation rules of the first language is output as the cross-language intermediate representation. This allows for the direct reuse of existing acoustic model assets as fixed acoustic feature extractors, avoiding the need to re-collect annotation data or train an acoustic model for the target language, and reducing the computational resource consumption and deployment costs at the acoustic level.
[0038] In a possible implementation, the source language acoustic model uses an end-to-end model based on the Conformer encoder architecture. After the model is trained on a large-scale Chinese Mandarin speech dataset, the model parameters are kept frozen. When receiving an English speech signal, the model forcibly maps it to a Chinese pinyin sequence that is approximately similar to the English pronunciation. For example, it maps "hello" to "ha lou", and this pinyin sequence serves as the input basis for subsequent cross-language conversion.
[0039] Step S203, construct cross-language conversion prompt information based on the source language phoneme sequence.
[0040] Specifically, the cross-language conversion prompt information refers to the input text used to guide the large language model to transcribe the source language phoneme sequence into the target language text, usually composed of the source language phoneme sequence and a natural language instruction. For example, for the source language phoneme sequence "ha lou wo de", the prompt information "Please transcribe the following Chinese pinyin into the corresponding English sentence, requiring smooth semantics: ha lou wo de" can be constructed.
[0041] In this step, the traditional acoustic recognition problem is transformed into a text-to-text cross-language transcription task by concatenating the source language phoneme sequence with the natural language instruction template to form cross-language translation prompts. This allows the subsequent large language model to utilize its cross-language semantic alignment capabilities learned during the pre-training phase to generate target language text. The acoustic model and the large language model achieve decoupled interaction through text-based prompts, eliminating the need for joint training or parameter adaptation.
[0042] In one possible implementation, the cross-language translation prompts also include at least one set of example data pairs, each containing an example source language phoneme sequence and its corresponding correct target language text. For example, example pairs such as "'ai la wu you'→'I love you'" and "'san ke you'→'thank you'" could be pre-added to the prompts to guide the large language model to more accurately perform the transcription task from source language phonemes to target language text based on a context learning paradigm.
[0043] Step S204: Input the cross-language conversion prompt information into the large language model to generate the target language text recognition result corresponding to the speech signal to be recognized. The generation process of the target language text recognition result does not require the use of speech-text annotation data of the target language for model training or parameter update.
[0044] Specifically, a large language model refers to a large neural network model that is pre-trained autoregressively based on massive multilingual text corpora and possesses cross-lingual semantic understanding and text generation capabilities, such as ChatGLM, Qwen, and GPT. Target language text refers to natural language text content represented using a target language scripting system, such as the English sentence "Hello world" or the German sentence "Hallo Welt". The target language text recognition result refers to the final output of the large language model, which is the target language script sequence corresponding to the semantics of the speech signal to be recognized.
[0045] In this step, cross-language conversion prompts are input into a large language model. The large language model then utilizes its pre-training stage-learned cross-language semantic alignment knowledge between source and target language texts to transcribe the source language phoneme sequence into semantically fluent target language text. On one hand, the entire generation process does not require the use of target language speech-text annotation data for training or parameter updates, completely eliminating dependence on target language speech annotation resources. On the other hand, the large language model, with its powerful semantic understanding capabilities, can effectively correct pronunciation deviations generated by the acoustic model during forced cross-language interpretation, transforming acoustic "misidentification" into correct semantic expression, achieving zero-shot cross-language speech recognition.
[0046] In one possible implementation, the large language model is an autoregressive language model pre-trained on multilingual text corpora. Its pre-training corpus includes source language text data and target language text data, but its training process does not use target language speech data or speech-text pairing data. For example, given the input cross-language conversion prompt "Please transcribe the following Chinese pinyin into the corresponding English sentence: ha lou wo de", the large language model outputs the target language text recognition result "Hello world".
[0047] The cross-language speech recognition method provided in this embodiment acquires the speech signal to be recognized in the target language, so that the cross-language speech recognition processing can be initiated without relying on the target language annotation data. The speech signal to be recognized is input into the source language acoustic model to generate the source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language. The amount of speech-text annotation data resources of the first language is greater than that of the target language. Therefore, the existing acoustic model assets of the high-resource language can be directly reused as a fixed acoustic feature extractor, avoiding the need to collect annotation data or train the acoustic model again for the target language, thus reducing the computational resource consumption at the acoustic level. This reduces deployment costs; based on the source language phoneme sequence, cross-language conversion prompts are constructed, transforming the acoustic recognition problem into a text-to-text cross-language transcription task. This allows the subsequent large language model to utilize its pre-trained cross-language semantic alignment capabilities to generate target language text. The cross-language conversion prompts are input into the large language model to generate target language text recognition results corresponding to the speech signal to be recognized. The generation of target language text recognition results does not require the use of target language speech-text annotation data for model training or parameter updates. Thus, cross-language speech recognition is achieved without the need for target language speech annotation data or retraining of acoustic model parameters, reducing deployment costs and computational resource consumption for low-resource languages.
[0048] This embodiment provides a cross-language speech recognition method, which can be used in the aforementioned mobile terminals, such as mobile phones and tablets. Figure 3 This is a flowchart of another cross-language speech recognition method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the speech signal to be recognized in the target language.
[0049] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.
[0050] Step S302: Input the speech signal to be recognized into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language.
[0051] Specifically, step S302 above includes the following steps: Step S3021: The speech signal to be recognized is input into the source language acoustic model. During the process of receiving the speech signal to be recognized and performing forward inference, the model parameters of the source language acoustic model remain frozen.
[0052] Specifically, keeping the model parameters frozen means that after receiving the speech signal to be recognized, during the forward propagation calculation to generate the output sequence, all weight and bias parameters of the source language acoustic model are not updated or adjusted. By keeping the model parameters frozen, the deployed source language acoustic model does not need any form of fine-tuning or adaptation training for the target language and can be directly reused as a fixed acoustic feature extractor, avoiding the computational resource consumption caused by restarting the model training process every time a new language is added.
[0053] In one possible implementation, the source language acoustic model is a pre-trained and deployed Mandarin Chinese acoustic model based on a Conformer encoder architecture. When an English speech signal is input to the model, the model encodes and decodes the input speech based on its existing knowledge of Chinese pronunciation during forward inference, outputting an approximate Chinese pinyin sequence. The entire process does not involve any parameter update operations.
[0054] Step S3022: Obtain the phoneme sequence output by the source language acoustic model that conforms to the pronunciation rules of the first language.
[0055] Specifically, first language pronunciation rules refer to the norms of a first language in terms of phoneme system, syllable structure, and tone patterns. They stipulate the types, combinations, and pronunciation characteristics of legal phonemes in the language, such as the initials, finals, and tone system of Mandarin Chinese.
[0056] In this step, by obtaining the phoneme sequence that conforms to the pronunciation rules of the first language output by the acoustic model of the source language, the speech signal of any target language is uniformly mapped to the pronunciation symbol space of the first language, forming a cross-language intermediate representation that can be understood by the subsequent large language model, thus realizing the conversion from acoustic modality to text symbol modality.
[0057] In one possible implementation, taking Mandarin Chinese as the first language as an example, the source language acoustic model outputs a phoneme sequence that conforms to the rules of Chinese Pinyin for a Swahili speech input. For example, the Swahili "habari" is mapped to the Pinyin sequence "ha ba li". This Pinyin sequence will serve as the basis for the next step of building cross-language conversion prompts.
[0058] Step S303: Construct cross-language conversion prompt information based on the source language phoneme sequence.
[0059] Specifically, step S303 above includes the following steps: Step S3031: The source language phoneme sequence is concatenated with a preset natural language instruction template to form a prompt text that instructs the large language model to perform a cross-language transcription task from the phonemes of the first language to the target language text.
[0060] Specifically, the preset natural language instruction template refers to a pre-designed text template that includes a task description and a target language type identifier, such as "Please transcribe the following [source language name] pinyin into the corresponding [target language name] sentence." The phonemes of the first language refer to the phoneme sequence output by the source language acoustic model in step S302 that conforms to the pronunciation rules of the first language. The prompt text refers to the complete input text formed after embedding the source language phoneme sequence into the natural language instruction template.
[0061] In this step, the acoustic recognition problem is reconstructed into a text-to-text cross-lingual transcription task by concatenating phoneme sequences with natural language instruction templates. The large language model does not need to understand speech modalities; it only needs to rely on the semantic alignment knowledge between the source and target languages learned during the text pre-training stage to complete the mapping from the first language phoneme sequence to the target language text, thus achieving complete decoupling between the acoustic module and the language module.
[0062] In one possible implementation, the natural language instruction template is "Please transcribe the following Chinese pinyin into the corresponding English sentence, requiring semantic fluency:", and the source language phoneme sequence is concatenated to the end of the template to form a prompt text such as "Please transcribe the following Chinese pinyin into the corresponding English sentence, requiring semantic fluency: ha lou wo de", which is then input into the large language model.
[0063] Step S304: Input the cross-language conversion prompt information into the large language model to generate the target language text recognition result corresponding to the speech signal to be recognized. The generation process of the target language text recognition result does not require the use of speech-text annotation data of the target language for model training or parameter update.
[0064] Please see details Figure 2 Step S204 of the illustrated embodiment will not be described again here.
[0065] This implementation first acquires the target language speech signal to be recognized; then, it inputs the speech signal into the source language acoustic model with its parameters frozen. Utilizing high-resource first language acoustic knowledge, the target language speech is forcibly mapped into a phoneme sequence conforming to the pronunciation rules of the first language, achieving zero-cost reuse of the acoustic model assets. Next, the phoneme sequence is concatenated with a natural language instruction template to generate cross-language conversion prompts, reconstructing the speech recognition task into a text transcription task. Finally, the large language model generates the target language text recognition result based on cross-language semantic alignment knowledge. Throughout the entire processing chain, the target language speech-text annotation data is not needed, the source language acoustic model parameters remain unchanged, and the acoustic module and the large language model are flexibly decoupled through text-based prompts. Zero-sample cross-language speech recognition is achieved without requiring target language speech annotation data or retraining the acoustic model parameters.
[0066] In one alternative implementation, the source language acoustic model is an end-to-end acoustic model based on a Transformer encoder architecture or a Conformer encoder architecture.
[0067] Specifically, the Transformer encoder architecture refers to a network structure based on a multi-head self-attention mechanism and a stacked feedforward neural network. Its core is to calculate the correlation strength between positions in the input sequence through self-attention layers, thereby achieving global context modeling. The Conformer encoder architecture refers to a network structure that introduces convolutional modules on top of the Transformer encoder. By inserting depthwise separable convolutional layers between the self-attention layers, it can capture both global dependencies and local detailed features of the sequence.
[0068] In this embodiment, by adopting a Transformer or Conformer encoder architecture as the core structure of the source language acoustic model, and using a self-attention mechanism to perform global acoustic modeling of the speech spectrum, the general acoustic features of any language speech signal can be effectively extracted and mapped to the phonetic symbol space of the first language, providing a high-quality intermediate representation for subsequent cross-language prompting construction.
[0069] In one possible implementation, the source language acoustic model employs an end-to-end model based on a Conformer encoder architecture. This model consists of multiple stacked Conformer coding blocks, each containing a self-attention module, a convolutional module, and a feedforward module. The model is trained on large-scale Mandarin Chinese speech data using a CTC / Attention joint loss and then deployed. It outputs a Chinese pinyin sequence for any input language speech, and the model parameters remain fixed during the inference phase.
[0070] In some optional implementations, the cross-language conversion prompts also include at least one set of example data pairs, which contain example source language phoneme sequences and their corresponding correct target language texts, to guide the large language model to generate target language text recognition results based on context learning.
[0071] Specifically, example data pairs refer to pre-constructed example input-output pairs consisting of source language phoneme sequences and their exact corresponding target language characters, such as "'ai la wu you'→'I love you'" and "'san ke you'→'thank you'". Contextual learning refers to the method by which a large language model infers a task pattern from a small number of examples provided in the prompts without updating its parameters, and then performs similar tasks on new inputs accordingly.
[0072] In this embodiment, by embedding at least one set of example data pairs in the cross-language conversion prompt information, a task demonstration is provided for the large language model, enabling it to quickly understand the mapping rules from source language phonemes to target language text without any fine-tuning for the speech recognition task, thereby improving the accuracy of zero-sample cross-language transcription.
[0073] In one possible implementation, for English speech recognition scenarios, the following cross-language conversion prompt is constructed: "Please transcribe the following Chinese pinyin into the corresponding English sentences. Examples: 'ai la wu you' → 'I love you'; 'san ke you' → 'thank you'; 'ha lou' → 'hello'. Now please transcribe: 'wo yao yi bei kafei'". After inputting this prompt into a large language model, the model infers and outputs "I want a cup of coffee" based on the example.
[0074] In an optional implementation, before inputting the cross-language conversion prompt information into the large language model in step S204, the method further includes: Autoregressive language models are pre-trained based on multilingual text corpora, which include text data corresponding to the source language and text data corresponding to the target language. However, the training process of the large language model does not use speech data or speech-text pairing data of the target language.
[0075] Specifically, multilingual text corpora refer to large-scale plain text datasets containing text content in two or more natural languages, such as Chinese Wikipedia text, English Wikipedia text, and German news text. Autoregressive language models are generative language models trained on the Transformer decoder architecture by predicting the next word in a sequence, such as the Kimi, Qwen, and ChatGPT series models. Their training objective is to predict subsequent words based on the preceding words.
[0076] In this embodiment, the large language model has learned the semantic alignment relationship between the source and target languages through multilingual text corpora during the pre-training stage, enabling it to generate cross-language text by transcribing or translating source language text into target language text. This pre-training process uses only plain text data, without introducing target language speech data or speech-text pairing data, ensuring that the large language model's cross-language semantic knowledge originates entirely from the text modality. Based on this, when receiving cross-language conversion prompts composed of source language phoneme sequences, the large language model can directly invoke the cross-language semantic alignment knowledge learned during the pre-training stage to complete the text-to-text transcription task, without requiring any further adaptation training for the speech modality. This maintains zero dependence on target language speech annotation data even during the inference stage.
[0077] In one possible implementation, open-source large language models such as ChatGLM or Qwen, which have been pre-trained on multilingual text corpora including Chinese and English, are selected as the text generation module. This model has never encountered English or German speech data during the pre-training phase, but it has learned the semantic correspondences of a large number of parallel Chinese-English and Chinese-German texts. When the input is "Please transcribe the following Chinese pinyin into the corresponding English sentence: ha lou wo de", the model directly outputs "Hello world" based on its cross-language knowledge at the text level.
[0078] This embodiment also provides a cross-language speech recognition device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0079] This embodiment provides a cross-language speech recognition device, such as... Figure 4 As shown, it includes: The speech acquisition module 401 is used to acquire the speech signal to be recognized in the target language; The phoneme generation module 402 is used to input the speech signal to be recognized into the source language acoustic model and generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of the first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language. The prompt construction module 403 is used to construct cross-language conversion prompt information based on the source language phoneme sequence; The text generation module 404 is used to input cross-language conversion prompts into the large language model to generate target language text recognition results corresponding to the speech signal to be recognized. The generation process of target language text recognition results does not require model training or parameter updates using speech-text annotation data of the target language.
[0080] In some alternative implementations, the phoneme generation module 402 includes: The input unit is used to input the speech signal to be recognized into the source language acoustic model. During the process of receiving the speech signal to be recognized and performing forward inference, the model parameters of the source language acoustic model remain frozen. The phoneme sequence acquisition unit is used to acquire the phoneme sequence output by the source language acoustic model that conforms to the pronunciation rules of the first language.
[0081] In some alternative implementations, the source language acoustic model is an end-to-end acoustic model based on a Transformer encoder architecture or a Conformer encoder architecture.
[0082] In some alternative implementations, the prompt building module 403 includes: The splicing unit is used to splice the source language phoneme sequence with a preset natural language instruction template to form a prompt text that instructs the large language model to perform a cross-language transcription task from the phonemes of the first language to the target language text.
[0083] In some optional implementations, the cross-language conversion prompts also include at least one set of example data pairs, which contain example source language phoneme sequences and their corresponding correct target language texts, to guide the large language model to generate target language text recognition results based on context learning.
[0084] In some alternative implementations, the text generation module 404 further includes: The training unit is used for pre-training an autoregressive language model based on a multilingual text corpus. The multilingual text corpus includes text data corresponding to the source language and text data corresponding to the target language. However, the training process of the large language model does not use speech data or speech-text pairing data of the target language.
[0085] The cross-language speech recognition device provided in this application can execute the cross-language speech recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.
[0086] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0087] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0088] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0089] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the cross-language speech recognition method of embodiments of this application.
[0090] Figure 5The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0091] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the cross-language speech recognition method shown in the above embodiments is implemented.
[0092] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0093] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.
Claims
1. A cross-language speech recognition method, characterized in that, The method includes: Acquire the speech signal to be recognized in the target language; The speech signal to be identified is input into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be identified. The source language acoustic model is trained using speech-text annotation data of a first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language. Based on the source language phoneme sequence, construct cross-language conversion prompt information; The cross-language conversion prompt information is input into a large language model to generate a target language text recognition result corresponding to the speech signal to be recognized. The generation process of the target language text recognition result does not require model training or parameter updates using speech-text annotation data of the target language.
2. The method according to claim 1, characterized in that, The step of inputting the speech signal to be recognized into the source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized includes: The speech signal to be recognized is input into the source language acoustic model. During the process of receiving the speech signal to be recognized and performing forward inference, the model parameters of the source language acoustic model remain frozen. Obtain the phoneme sequence output by the source language acoustic model that conforms to the pronunciation rules of the first language.
3. The method according to claim 1 or 2, characterized in that, The source language acoustic model is an end-to-end acoustic model based on the Transformer encoder architecture or the Conformer encoder architecture.
4. The method according to claim 1, characterized in that, The construction of cross-language conversion prompts based on the source language phoneme sequence includes: The source language phoneme sequence is concatenated with a preset natural language instruction template to form a prompt text that instructs the large language model to perform a cross-language transcription task from the phonemes of the first language to the target language text.
5. The method according to claim 1, characterized in that, The cross-language conversion prompt information also includes at least one set of example data pairs, which contain example source language phoneme sequences and their corresponding correct target language texts, to guide the large language model to generate the target language text recognition results based on context learning.
6. The method according to claim 1, characterized in that, Before inputting the cross-language conversion prompts into the large language model, the method further includes: Autoregressive language model pre-training is performed based on multilingual text corpora, wherein the multilingual text corpora include text data corresponding to the source language and text data corresponding to the target language, and the training process of the large language model does not use speech data or speech-text pairing data of the target language.
7. A cross-language speech recognition device, characterized in that, The device includes: The speech acquisition module is used to acquire the speech signal to be recognized in the target language; A phoneme generation module is used to input the speech signal to be recognized into a source language acoustic model to generate a source language phoneme sequence corresponding to the speech signal to be recognized. The source language acoustic model is trained using speech-text annotation data of a first language, and the amount of speech-text annotation data resources of the first language is greater than that of the target language. The prompt construction module is used to construct cross-language conversion prompt information based on the source language phoneme sequence; The text generation module is used to input the cross-language conversion prompt information into the large language model to generate the target language text recognition result corresponding to the speech signal to be recognized. The generation process of the target language text recognition result does not require model training or parameter updates using the speech-text annotation data of the target language.
8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the cross-language speech recognition method according to any one of claims 1 to 7 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the cross-language speech recognition method according to any one of claims 1 to 7.
10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the cross-language speech recognition method according to any one of claims 1 to 7.