Enhancing speech-to-text performance of hybrid languages

Abstract

The invention relates to enhancing voice-to-text performance of mixed languages. According to the embodiment of the invention, the mixed language speech recognition model is trained, so that speech recognition executed by the model can be enhanced. The model manager may control training and / or operation of the model. A hybrid language data generation manager may generate an enhanced hybrid language dataset, which may enhance training of the model. The trimmer can facilitate configuration of hyper-parameters of the model. Audio-based information and transcriptions representative of the dataset may be applied to the model to facilitate model training. The FAL evaluator may determine the fidelity, accuracy, and latency of the model to perform speech recognition on audio-based information and transcriptions. Based on this determination, the hybrid language data generation process and / or hyper-parameters may be updated to enhance further training of the model, thereby enhancing fidelity, accuracy, and / or latency with respect to speech recognition performance of the model. The model manager may control one or more iterations of the model training.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 592,912, filed on October 24, 2023, entitled “Speech-to-Text Technology with Better Performance for Mixed Languages,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to technical fields such as data processing, artificial intelligence (AI), and machine learning, and in particular to a speech-to-text service applied to artificial intelligence, and more specifically, to a system for enhancing speech-to-text performance of mixed languages. Background Art

[0004] A speech-to-text application may be used to recognize words in a person's speech audio and convert the speech audio into text words corresponding to the words in the audio, and may present the text words as output results. For example, a speech-to-text application may analyze speech audio of a person speaking a particular language, and based on the analysis of the audio, perform speech recognition to recognize the words in the audio. The application may convert the recognized words into individual text words, which may represent the individual spoken words.

[0005] The above description is intended only to provide a background overview about speech-to-text applications and is not intended to be exhaustive. Summary of the invention

[0006] A simplified summary is presented below in order to provide a basic understanding of some aspects described herein. This summary is not an extensive overview of the disclosed subject matter. It is neither intended to identify the key or essential elements of the disclosure nor to delimit its scope. Its sole purpose is to present certain concepts in a simplified form as a prelude to a more detailed description that will be presented later.

[0007] In some embodiments, the disclosed subject matter may include a method that may include: generating, by a trained model of a system including at least one processor, mixed language data based on instruction data related to generation of the mixed language data and seed topic data related to a set of seed topics, wherein the mixed language data may include respective first text words in a first language and respective second text words in a second language, the respective first text words and the respective second text words being determined based on corresponding simulated conversations between respective simulated speakers, wherein the respective second text words may be interspersed with the respective first text words. The method may also include: determining, by the trained model of the system, as part of training of the speech recognition model, a quality metric indicating speech recognition performance of the speech recognition model in identifying information related to spoken mixed language words representing the mixed language data and generating textual transcription mixed language words representing the spoken mixed language words, wherein the quality metric may represent fidelity, accuracy, and latency related to speech recognition performance.

[0008] In certain embodiments, the disclosed subject matter may include a system that may include: at least one memory storing computer executable components, and at least one processor that may execute the computer executable components stored in the at least one memory. The computer executable components may include a mixed language data generator that may determine and generate mixed language data based on instruction information related to mixed language data generation and seed topic data related to a set of seed topics, wherein the mixed language data may include individual first text words in a first language and individual second text words in a second language, wherein the individual first text words and the individual second text words may be determined based on corresponding simulated conversations between individual simulated speakers, wherein the individual second text words may be interspersed with the individual first text words. The computer executable components may also include an evaluator that may determine, in relation to the training of a speech recognition model, a quality metric that may represent the speech recognition performance of the speech recognition model in recognizing information related to spoken mixed language words representing the mixed language data and generating textual transcription mixed language words characterizing the spoken mixed language words, wherein the quality metric may be related to fidelity, accuracy, and latency associated with the speech recognition performance of the speech recognition model.

[0009] In other embodiments, the disclosed subject matter may include a non-transitory machine-readable medium including executable instructions that, when executed by at least one processor, may implement the execution of operations. The operations may include: determining mixed language data based on instruction data related to generation of mixed language data and seed topic data related to a set of seed topics, wherein the mixed language data may include individual first text words in a first language and individual second text words in a second language, wherein the individual first text words and the individual second text words may be determined based on corresponding simulated conversations between individual simulated speakers, wherein the individual second text words may be mixed with the individual first text words in the same sentence. The operations may also include: with respect to training of a speech recognition model, determining a quality metric rating value that may represent the speech recognition performance of the speech recognition model in identifying information related to a spoken mixed language word that may characterize the mixed language data and generating a textual transcription of the spoken mixed language word that characterizes the mixed language data, wherein the quality metric rating value may represent the fidelity, accuracy, and latency associated with the speech recognition performance of the speech recognition model.

[0010] The following description and accompanying drawings set forth certain illustrative aspects of the present disclosure in detail. However, these aspects represent only a few of the various methods in which the principles of the disclosed various aspects may be employed, and the present disclosure is intended to include all of these aspects and their equivalents. Other advantages and features will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A block diagram of a non-limiting example system is shown that can be ideal for training a speech recognition model that can be ideal for recognizing mixed-language words in speech and converting those words into text data in accordance with various aspects and implementations of the disclosed subject matter.

[0012] Figure 2 A block diagram is depicted of a non-limiting example model manager component 102, which can include a mixed language data generation manager (MLDGM) component that can ideally determine and generate a mixed language dataset that can be used to train a speech recognition model, and a fidelity-accuracy-latency (FAL) evaluator component that can perform FAL evaluations related to the performance of the speech recognition model as part of model training, in accordance with various aspects and embodiments of the disclosed subject matter.

[0013] Figure 3 A block diagram of a speech recognition component including a speech recognition model is shown in accordance with various aspects and implementations of the disclosed subject matter.

[0014] Figure 4A graph is presented of non-limiting example speech recognition results of a non-limiting example trained speech recognition model (e.g., the Whisper-Mixed Cantonese and English (MCE) model) trained using the enhanced mixed-language dataset and enhanced FAL evaluation techniques described herein, and certain other trained speech recognition models trained using existing language datasets and existing training techniques, performing speech recognition on an audio recording containing spoken Cantonese words.

[0015] Figure 5 A graph is presented of non-limiting example speech recognition results relating to speech recognition performed on an audio recording comprising spoken mixed Cantonese and English words by a non-limiting example trained speech recognition model (e.g., a Whisper-MCE model) and certain other trained speech recognition models, in accordance with various aspects and embodiments of the disclosed subject matter.

[0016] Figure 6 A flow chart of an example method is shown that can be ideal for training a speech recognition model that can be ideal for recognizing mixed-language words in speech and converting those words into text data, in accordance with various aspects and implementations of the disclosed subject matter.

[0017] Figure 7 and Figure 8 A flowchart depicts an example method that can ideally determine and generate an enhanced mixed-language dataset that can be used to train a speech recognition model to enable the speech recognition model to ideally recognize mixed-language words in speech and convert those words into text data, in accordance with various aspects and embodiments of the disclosed subject matter.

[0018] Fig. 9 and Fig.10 A flowchart of an example method is shown that can ideally train a speech recognition model based at least in part on information related to an enhanced mixed-language dataset and / or perform updates based at least in part on FAL evaluations to facilitate training of the speech recognition model in accordance with various aspects and embodiments of the disclosed subject matter.

[0019] Fig.11 A flow chart is depicted of an example method that can desirably perform speech recognition and speech-to-text for mixed-language speech in accordance with various aspects and implementations of the disclosed subject matter.

[0020] Fig.12 An example block diagram of an example computing environment is shown in which various embodiments of the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0021] Various aspects of the disclosed subject matter are now described with reference to the accompanying drawings, wherein like reference numerals are used to refer to like elements. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it is apparent that these aspects can be implemented without these specific details. In other examples, well-known structures and devices are shown in block diagram form for ease of description of one or more aspects.

[0022] The present disclosure generally relates to systems, mechanisms, methods, and techniques that can ideally (e.g., appropriately, accurately, quickly, efficiently, reliably, enhanced, or optimally) perform speech recognition (e.g., automatic speech recognition (ASR)) on mixed-language speech audio of a person, the systems, mechanisms, methods, and techniques that convert individual recognized spoken words (e.g., mixed-language words) in the audio into individual text words or characters that correspond to and are representative of the spoken words, and present the text words or characters as output.

[0023] Some existing speech-to-text systems can perform speech recognition on human speech audio and convert spoken words into text words with varying success and accuracy. Some existing speech-to-text systems can perform speech recognition on mixed-language speech audio. Typically, existing systems can use a speech recognition audio dataset for a speech recognition model to train the speech recognition model to perform speech recognition on audio of a person speaking the words.

[0024] However, existing speech-to-text systems may be defective in many aspects. The defect of some existing speech-to-text systems may be that they cannot perform speech recognition on audio speech involving mixed languages ​​(especially challenging mixed language sets), such as Cantonese and English, with sufficiently high accuracy, or may not be able to perform speech recognition on mixed language audio speech at all. The defect of some existing speech-to-text systems may be that these existing systems may undesirably pay very high costs, including financial costs, computing costs and / or time costs (e.g., undesirably high delays and slow speech recognition speeds) when performing speech recognition on mixed languages, even if they can perform speech recognition on speech audio with a certain accuracy. Another defect of some existing speech-to-text systems may be that existing speech recognition audio data sets for training speech recognition models may be defective with respect to certain languages ​​and mixed languages ​​(including, for example, Cantonese, or mixed Cantonese and English). However, another shortcoming of some existing speech-to-text systems may be that they adopt word error rate (WER) technology or character error rate (CER), and WER and CER may not be ideal (e.g., may be inappropriate, unacceptable, inaccurate or suboptimal) when evaluating the performance of speech recognition systems in certain languages ​​or mixed languages ​​(e.g., Cantonese, or a mixture of Cantonese and English).

[0025] It may be desirable (e.g., appropriate, beneficial, advantageous, useful, improved, or optimal) for speech-to-text systems, methods, and techniques to quickly, efficiently, and accurately perform speech recognition on audio of mixed-language speech (e.g., Cantonese and English, or other languages) and convert spoken words into corresponding text words or characters. It may also be desirable to appropriately or enhance training of speech recognition (e.g., speech-to-text) models so that they can accurately, efficiently, and quickly perform speech recognition on mixed-language speech audio. In addition, it may also be more desirable to accurately and efficiently evaluate the performance of speech recognition systems for various types of languages ​​or mixed languages ​​(e.g., including Cantonese or mixed Cantonese and English) to facilitate fine-tuning and training of speech recognition models. The disclosed subject matter can solve and overcome the above-mentioned deficiencies and other deficiencies of existing systems and technologies, because the systems, methods and technologies disclosed herein can ideally enhance (e.g., improve, enhance or optimize) the speech recognition performance of mixed-language speech audio and convert spoken words into corresponding text words or characters, so that a high level of accuracy in speech-to-text recognition and conversion can be ideally achieved, while also achieving speech-to-text recognition and conversion at an ideally faster speed than existing speech-to-text systems, methods and technologies. In order to facilitate ideal speech recognition, the disclosed subject matter can employ systems, methods and technologies for generating an ideal mixed-language audio dataset (e.g., a mixed Cantonese and English audio dataset), which can be used to train a speech recognition model (e.g., an enhanced speech recognition model) to be able to ideally perform speech recognition on audio of mixed-language spoken words of a person. In addition, with respect to the performance of such speech recognition, the disclosed subject matter may employ systems, methods, and techniques that may ideally employ enhanced performance evaluation techniques that may ideally (e.g., accurately, efficiently, enhanced, or optimally) evaluate the performance of speech recognition systems for various types of languages ​​or mixed languages ​​(e.g., including Cantonese or a mixture of Cantonese and English) to facilitate fine-tuning and training of speech recognition models.

[0026] To this end, a technique is presented that can desirably (e.g., automatically, dynamically, appropriately, reliably, efficiently, enhanced, and / or optimally) generate an enhanced mixed-language dataset and generate an enhanced trained mixed-language speech recognition model based at least in part on the enhanced mixed-language dataset. The system may include a model manager component that can control the generation of the enhanced mixed-language dataset, and control the training and / or operation of the mixed-language speech recognition model, which is based at least in part on the enhanced mixed-language dataset. In some embodiments, the model manager component may include or employ one or more artificial intelligence-based models, such as, for example, a generative pre-trained transformer (GPT) model (e.g., a fourth-generation GPT (GPT-4) model), another type of transformer-based model, another type of multimodal large language model, and / or another type of artificial intelligence-based model. The model manager component may include a mixed-language data generation manager (MLDGM) component, which may be part of a trained model (e.g., a trained artificial intelligence-based model) or may be associated with a trained model. The MLDGM component (e.g., an MLDGM component of a trained model or an MLDGM component employing a trained model) can generate an enhanced mixed-language dataset including individual words in a mixed language (e.g., at least two different languages) based at least in part on instructions and / or seed topics related to mixed-language data generation (e.g., seed topics that can enable conversations related to the seed topics or other topics), wherein the words of the dataset can, for example, include individual first words in a first language (e.g., Cantonese or other desired first language) and individual second words in a second language (e.g., English or other desired second language that may be different from the first language), wherein the individual second words can be interspersed with the individual first words according to defined model management criteria, such as described herein. In some embodiments, the MLDGM component can generate and present as output a transcription (e.g., text transcription data) that can represent the mixed-language dataset. It should be recognized and understood that while the terms "word" and / or "sub-word" may be used frequently herein, depending on the language referenced with respect to a mixed-language dataset or transcription (e.g., a text transcription generated by a speech recognition model), a "word" or "sub-word" may refer to characters of a language (e.g., characters of Chinese or another language), or even a single character of a language (e.g., a single character of English), and / or may refer to alphanumeric characters of such language.

[0027] In some embodiments, the audio recorder component can record a human speaker speaking individual words of a mixed language to generate audio content including individual spoken words of the mixed language (e.g., individual orally spoken words). Optionally, in other embodiments, the MLDGM component can employ a speech generator component that can generate audio content that simulates (e.g., simulates, emulates, or synthesizes) individual words spoken by a speaker (e.g., as synthesized speech) corresponding to and / or representative of mixed language words in a mixed language dataset (e.g., spoken mixed language words), and presents it as an output to generate individual spoken words of the mixed language contained in such audio content.

[0028] In some embodiments, the system may include a fine tuner component that can facilitate configuration (e.g., setting) of hyperparameters and / or parameters (e.g., weights or other parameters) of a mixed language speech recognition model. The system may also include an audio converter component that can generate audio-based information representing audio content (e.g., a visualization, graphical, or spectral representation of the audio content), the audio content including individual spoken words (e.g., individual spoken words of a human speaker or a simulated speaker) of a mixed language representing a mixed language data set. In some embodiments, the audio-based information may be or may include a spectrogram (e.g., a log-mel spectrogram) that can represent individual spoken words spoken by a speaker (e.g., a human speaker or a simulated speaker) in the audio content. The system may also include a tokenizer component that can tokenize individual words or subwords of the text transcription data of the mixed language data set based at least in part on the results of the analysis of the text transcription data to generate a set of tokens, wherein each token of the token group can represent a corresponding word or subword contained in the text transcription data.

[0029] As part of an iteration of training of the speech recognition model, the tuner component can input (e.g., apply) audio-based information (e.g., spectrogram information) into the speech recognition model, and the tokenizer component can input word tuples (e.g., can sequentially input individual word tuples) into the speech recognition model. The speech recognition model can analyze the audio-based information and word tuples (e.g., perform artificial intelligence-based analysis). In some embodiments, the speech recognition model can include a transformer-based encoder-decoder architecture that includes an encoder component (e.g., including a set of encoder blocks that can be composed of multiple neural network layers) and a decoder component associated with the encoder component (e.g., including a set of decoder blocks that can be composed of multiple neural network layers), such as described herein. As part of the analysis, the speech recognition model can perform next word tuple prediction to facilitate predicting, recognizing, and / or determining words contained in the audio-based information. Based at least in part on the results of such analysis by the speech recognition model, the speech recognition model can recognize or determine words (e.g., spoken words) in the audio-based information and convert the recognized words into text words that can represent the spoken words contained in the audio-based information, and the speech recognition model can also be trained. The speech recognition model can generate and present as an output a transcription that includes individual text words that can represent the spoken mixed-language words contained in the audio-based information (e.g., text-transcribed mixed-language words).

[0030] In some embodiments, a model manager component (e.g., a trained artificial intelligence-based model of or associated with the model manager component) may include or employ a fidelity-accuracy-latency (FAL) evaluator component that can determine quality metrics associated with the performance of a speech recognition model (e.g., during each iteration of training of the speech recognition model) in performing speech recognition on audio-based information in accordance with defined model management criteria, including a fidelity metric, an accuracy metric, and a latency metric. The fidelity metric value can represent the fidelity of a text transcribed mixed-language word to audio content including a spoken mixed-language word. The accuracy metric value can represent the accuracy of a text transcribed mixed-language word relative to a spoken mixed-language word. The latency metric value can represent the latency (e.g., amount of time) associated with the speech recognition model recognizing a spoken mixed-language word and generating a text transcribed mixed-language word.

[0031] Based at least in part on the results of the determination of such quality metrics, the FAL evaluator component (or another component of the model manager component) can determine updates to the mixed-language data generation process (e.g., modifications to parameters and / or operations in the mixed-language data generation process, or parameters and / or operations associated with the mixed-language data generation process), and / or updates to hyperparameters or parameters of the speech recognition model (e.g., modifications to hyperparameters or parameters) that can enhance further training and performance of the speech recognition model, including enhancing fidelity, accuracy, and / or latency associated with the speech recognition (e.g., ASR) performance of the speech recognition model. The model manager component can manage (e.g., control) one or more iterations of training of the speech recognition model according to defined model management criteria, including making one or more corresponding updates to the mixed-language data generation process and / or hyperparameters or parameters of the speech recognition model, such as described herein.

[0032] The disclosed subject matter can ideally (e.g., automatically, dynamically, appropriately, reliably, efficiently, enhanced and / or optimally) generate a mixed-language dataset by employing a model manager component, an MLDGM component and a FAL evaluator component associated therewith, and the enhanced techniques described herein (e.g., mixed-language dataset generation and FAL evaluation techniques), and the mixed-language dataset can be enhanced compared to mixed-language datasets generated using prior art techniques; the enhanced mixed-language dataset can be used to train a speech recognition model to create an enhanced trained speech recognition model, and the enhanced trained speech recognition model can provide improved speech recognition compared to the existing trained speech recognition model, especially in mixed-language speech recognition. By adopting the FAL evaluator and enhanced FAL evaluation techniques described herein, the presently disclosed subject matter can also ideally enhance the determination of speech recognition performance and problems (e.g., errors) of a speech recognition model (e.g., a mixed-language speech recognition model being trained) compared to existing techniques for determining performance and errors associated with training speech recognition models; and can also enhance modifications related to the training of speech recognition models compared to existing techniques for training speech recognition models, and the performance of speech recognition models trained using existing techniques, so that the training of the speech recognition model (e.g., iterative training) can be improved and / or perfected, and the resulting trained speech recognition model can be enhanced.

[0033]

[0013] The above and other aspects and embodiments of the disclosed subject matter are described in conjunction with the accompanying drawings.

[0034] Now refer to the accompanying drawings, Figure 1A block diagram of a non-limiting example system 100 is shown that can desirably (e.g., automatically, dynamically, appropriately, efficiently, reliably, enhanced, and / or optimally) train a speech recognition model (e.g., an artificial intelligence-based speech-to-text model) that can desirably (e.g., accurately, efficiently, reliably, enhanced, and / or optimally) recognize mixed-language words in speech and convert those words into text data, in accordance with various aspects and embodiments of the disclosed subject matter. The system 100 can include a model manager component 102 that can manage desirably training a speech recognition model 104 (e.g., an artificial intelligence-based speech-to-text model) of a speech recognition component 106 using desirably (e.g., enhanced, reliable, and / or optimal) mixed-language datasets composed of multiple languages ​​and enhanced FAL rating techniques (and mechanisms), such as described herein.

[0035] According to various embodiments, the speech recognition component 106 including the speech recognition model 104 can be part of or associated with the device 108 (e.g., communicatively connected to the device 108). For example, the device 108 can be a computer, which can include the speech recognition component 106. As another example, a first device (e.g., a computer, a server, or other type of device) can include the speech recognition component 106, and a second device (e.g., a mobile phone, a virtual assistant (VA) device, or other type of device) can access the first device or communicate with the first device (e.g., via a wireless or wired communication network) to access, use, and / or communicate with the speech recognition component 106. The device (e.g., 108) can be, for example, a computer, a laptop, a server, a wireless, mobile or smart phone, an electronic tablet, a VA device, electronic glasses, an electronic watch or other electronic personal device, an electronic gaming device, an Internet of Things (IoT) device (e.g., a health monitoring device, a toaster, a coffee machine, blinds, a music player, a speaker, a telemetry device, a smart meter, a machine-to-machine (M2M) device or other type of IoT device), a networked vehicle device (e.g., a car, an airplane, a train, a rocket and / or other at least partially automated means of transportation (e.g., a drone)), a personal digital assistant (PDA), a dongle (e.g., a universal serial bus (USB) or other type of dongle), a communication device, or other type of device. In some embodiments, the non-limiting term UE can be used to describe the device.

[0036] According to various embodiments, the model manager component 102 may include a mixed language data generation manager (MLDGM) component 110 and a FAL evaluator component 112. The MLDGM component 110 may determine and generate an enhanced mixed language data set, which may include at least two languages, wherein the mixed language data set may be used to ideally train the speech recognition model 104. In some embodiments, the mixed language data set may include each first word in a first language (e.g., Cantonese or other desired first language) and each second word in a second language (e.g., English or other desired second language that may be different from the first language), wherein each second word may be interspersed (e.g., inserted, intermixed, or mixed) with each first word (e.g., each second word is interspersed between each first word). For example, with respect to one or more sentences of the mixed language data set, some of the first words in the first language and some of the second words in the second language may appear in the same sentence. In certain embodiments, a mixed language dataset may include or be related to one or more conversations between entities (e.g., simulated speakers) that are related to one or more topics (e.g., news, sports, music, food, movies, television shows, politics, law, and / or other topics) that may be related to people's daily lives, such as described herein. It should be recognized and understood that while various aspects of the subject matter disclosed herein may be related to Cantonese and English, the techniques, aspects, and embodiments described herein may be applied to, extended to, or used in virtually any desired language (Spanish, French, German, Chinese, Japanese, Vietnamese, Korean, Italian, Russian, Turkish, Portuguese, Arabic, Persian, Hindi, or other desired language), and / or desired language dialects, to be used with, in addition to, or in place of Cantonese and / or English. Furthermore, it should be recognized and understood that while the terms “word” and / or “sub-word” may be used frequently herein, depending on the language referenced with respect to a mixed-language dataset or transcription (e.g., a text transcription generated by speech recognition model 104), a “word” or “sub-word” may refer to a character in one language (e.g., a character or symbol in Chinese or another language that may represent a word or sub-word in another language, or even a single character in one language (e.g., a single character in English)), and / or may refer to an alphanumeric character in such a language.

[0037] According to various embodiments, the model manager component 102 can employ one or more AI-based models (e.g., one or more trained AI-based models) that can perform all or at least a portion of the respective functions of the MLDGM component 110 and the FAL evaluator component 112. In some embodiments, the model manager component 102 can include a trained AI-based model that can include or be associated with (e.g., communicatively connected to) the MLDGM component 110 and the FAL evaluator component 112. In other embodiments, the model manager component 102 can include a first trained AI-based model that can include or be associated with (e.g., communicatively connected to) the MLDGM component 110 and the FAL evaluator component 112, and a second trained AI-based model that can include or be associated with the FAL evaluator component 112. In some embodiments, the one or more AI-based models can be GPT-type models, such as, for example, GPT-4 models. In other embodiments, the one or more AI-based models may be another type of GPT model, another type of transformer-based model, another type of multimodal large language model, and / or another type of AI-based model.

[0038] Further with respect to the speech recognition model 104, according to various embodiments, the speech recognition model 104 can be a desired artificial intelligence-based model (e.g., machine learning (ML), neural network, or other type of artificial intelligence-based model) that can be used or suitable for performing speech-to-text recognition (e.g., automatic speech-to-text recognition). In some embodiments, the speech recognition model 104 can be a weakly supervised deep learning acoustic model that can employ an encoder-decoder converter architecture, such as described herein. For example, the speech recognition model 104 can be a Whisper-type ML model, such as described herein. In other embodiments, the speech recognition model 104 can be another desired type of artificial intelligence-based model that can be used or suitable for performing speech-to-text recognition.

[0039] In some embodiments, the system 100 may include a tuner component 114 (e.g., a configuration component) that can fine-tune or configure the speech recognition model 104, or implement fine-tuning or configuration, based at least in part on a set of hyperparameters, parameters, and / or other information (e.g., parameters or other settings or configurations). For example, the tuner component 114 can pass a desired set of hyperparameters, parameters, and / or other information to the speech recognition model 104, wherein the speech recognition model 104 can be configured based at least in part on the set of hyperparameters and / or other information. Specific hyperparameters in the set of hyperparameters can be based at least in part on certain factors, such as the artificial intelligence-based model type of the speech recognition model 104, adjustments or improvements to one or more hyperparameters (e.g., to implement training or improve training of the speech recognition model 104), and / or other factors. For example, the speech recognition model 104 can be an enhanced (e.g., modified and improved) Whisper-type model, such as an enhanced Whisper-tiny model, an enhanced Whisper-base model, an enhanced Whisper-small model, an enhanced Whisper-medium model, or an enhanced Whisper-large model. The first set of hyperparameters can be used to configure (e.g., initially configure, or improve the configuration in subsequent iterations) an enhanced Whisper-tiny model, the second set of hyperparameters can be used to configure an enhanced Whisper-base model, the third set of hyperparameters can be used to configure an enhanced Whisper-small model, the fourth set of hyperparameters can be used to configure an enhanced Whisper-medium model, and the fifth set of hyperparameters can be used to configure an enhanced Whisper-large model. In some embodiments, the speech recognition model 104 can be an enhanced Whisper-mixed language-small model (e.g., a fine-tuned Whisper-mixed language-small model), but in other embodiments, the speech recognition model 104 can be an enhanced Whisper-mixed language model of different types.

[0040] In some embodiments, the system 100 may include an audio recorder component 116 that can be used to record a speaker (e.g., a person speaking a word) speaking each first word and each second word in a mixed language dataset (e.g., a speaker having a conversation using each first word and each second word in text transcription data of the mixed language dataset), and generate an audio recording (e.g., recorded audio content) including each word (e.g., each first word and each second word) spoken by the speaker, such as described herein. In some embodiments, the system 100 may also include an audio converter component 118 (e.g., a spectrogram generator component) that can generate a spectrogram that can characterize each word spoken by the speaker in the audio recording. In some embodiments, the spectrogram can be a log-mel spectrogram, but in other embodiments, the spectrogram can be a different type of spectrogram (e.g., a mel spectrogram or other type of spectrogram). According to various embodiments, the audio converter component 118 can be part of the speech recognition component 106 (e.g., in Figure 1 ), may be a separate component associated with the speech recognition component 106 (eg, communicatively connected with the speech recognition component 106), or may be part of another component associated with the speech recognition component 106.

[0041] In some embodiments, the system 100 may include a tokenizer component 120 that may tokenize individual words or subwords of individual words (e.g., syllables or other parts of words) of the text transcription data of the mixed language data set based at least in part on the results of the analysis of the text transcription data to generate a set of tokens, wherein each token of the token group may represent a corresponding word or subword contained in the text transcription data. For example, based at least in part on the analysis results, the tokenizer component 120 may determine and generate a first token that represents a first word or subword in a first language; a second token that represents a second word or subword in the first language or a second language; a third token that represents a third word or subword in the first language or a second language; and so on, to generate corresponding tokens for other words in the text transcription data. According to various embodiments, the tokenizer component 120 may be part of the speech recognition component 106 (e.g., in Figure 1 ), may be a separate component associated with the speech recognition component 106 (eg, communicatively connected with the speech recognition component 106), or may be part of another component associated with the speech recognition component 106.

[0042] In some embodiments, the audio converter component 118 can input (e.g., apply) the spectrogram to a speech recognition model 104 (e.g., managed by the model manager component 102), and the tokenizer component 120 can input (e.g., apply) the word tuples to the speech recognition model 104 to facilitate training the speech recognition model 104. The speech recognition model 104 can perform an artificial intelligence-based analysis of the spectrogram information and the word tuples of the spectrogram. Based at least in part on the results of the artificial intelligence-based analysis of the spectrogram information, the speech recognition model 104 can be trained (at least initially or partially trained) and can perform speech recognition on the spectrogram information to generate a transcription including text data that can characterize individual words (e.g., individual mixed-language words) spoken in the audio content (e.g., an audio recording), as well as characterize individual words represented in the spectrogram, such as described herein.

[0043] In some embodiments, the speech recognition component 106 (e.g., the speech recognition model 104 or another component of the speech recognition component 106) can determine a loss function associated with speech recognition performance and the generation of transcriptions based at least in part on comparing output data (e.g., predicted word units representing words, or transcribed text data) output by the speech recognition model 104 with input data (e.g., input word units, or text transcription data of a mixed language dataset), wherein the loss function can represent the degree of error or inaccuracy in the speech recognition performance and the generation of transcriptions, such as described herein. According to various embodiments, the speech recognition component 106 can feed back (e.g., backpropagate) information related to the loss function to the tuner component 114 and / or the model manager component 102 to facilitate determining and / or performing adjustments (e.g., modifications) to hyperparameters of the speech recognition model 104, other parameters of the speech recognition model 104, or other functions associated with training the speech recognition model 104, or the process of generating the mixed language dataset, such as described herein. For example, based at least in part on the determination of the loss function, the speech recognition component 106 (or the model manager component 102 or the tuner component 114) can determine updates (e.g., adjustments) that can be made to one or more hyperparameters of the speech recognition model 104 to fine-tune and enhance (e.g., improve or increase) the performance and accuracy of the speech recognition model 104 when performing speech recognition on audio content of mixed-language speech (or a spectrogram representing mixed-language speech) and converting mixed-language spoken words in the audio content into transcriptions, which include text data and represent or correspond to the mixed-language spoken words. Based at least in part on the updates, the tuner component 114 can update one or more hyperparameters of the speech recognition model 104 to facilitate improving the training of the speech recognition model 104, reducing the loss function, and enhancing the performance and accuracy of the speech recognition model 104.

[0044] According to various embodiments, additionally and optionally, the model manager component 102 may also employ a FAL evaluator component 112 to evaluate the accuracy and efficiency of the speech recognition model 104 in performing speech recognition and converting speech to text. As disclosed, existing WER and CER techniques may be defective, inefficient and / or inaccurate in determining errors in words or characters and / or determining other problems associated with speech recognition performance, and may be defective, inefficient and / or inaccurate in determining training updates for speech recognition models, particularly when performing speech recognition on mixed-language speech. Compared to existing WER or CER techniques, the FAL evaluator component 112 may employ an enhanced FAL rating technique (and corresponding mechanism) that may ideally determine an enhanced FAL rating that may more ideally (e.g., accurately, reliably, efficiently, enhanced or optimally) determine the performance quality and / or errors of the speech recognition model 104 with respect to speech recognition performance, particularly when performing speech recognition on mixed-language speech. The FAL evaluator component 112 can determine and evaluate the fidelity, accuracy, and latency associated with the speech recognition performance of the speech recognition model 104 for mixed-language speech of audio content (e.g., a spectrogram characterizing the audio content including the mixed-language speech) based at least in part on a set of validation data or a set of test data, which validation data or test data can include text transcription data of the mixed-language dataset or a portion thereof or information about or related thereto, or can be determined or derived based on text transcription data of the mixed-language dataset or a portion thereof or information about or related thereto (e.g., by the model manager component 102). The fidelity factor can be or can be related to the fidelity of the transcription generated by the speech recognition model 104 relative to the original audio content. The FAL evaluator component 112 can determine and use a fidelity factor (e.g., a fidelity score, rating, or value (e.g., a fidelity metric) that characterizes fidelity) to evaluate the speech recognition model 104's ability to capture the content and meaning of speech (e.g., mixed-language speech) in the original audio content in a transcription generated by the speech recognition model 104, including transcribed text data, and can involve assessing the accuracy of the transcription (e.g., the accuracy of words and / or characters in the transcription compared to the words spoken in the audio content) and ensuring that the transcribed text data in the transcription retains the intended message, tone, and context of the spoken words in the audio content.

[0045] With respect to the accuracy factor, the FAL evaluator component 112 can determine and / or measure the accuracy or correctness of the transcription (e.g., an accuracy score, rating, or value (e.g., an accuracy metric) that characterizes the accuracy). The accuracy factor can relate to the ability of the FAL evaluator component 112 to determine and evaluate the speech recognition model 104 to correctly recognize the spoken words of the audio content and convert them into text data (e.g., written text), including accurately representing the tone and pronunciation of individual words in the various languages ​​contained in the audio content (e.g., particularly with respect to more challenging languages, such as Chinese).

[0046] With respect to latency factors, the FAL evaluator component 112 can determine, measure, or track the time required for the speech recognition model 104 to process the audio content (e.g., a spectrogram representing the audio content including mixed-language speech) and generate a transcription representing the spoken words contained in the audio content, because it may be desirable (e.g., useful, desired, or beneficial) to evaluate the responsiveness and efficiency of the speech recognition model 104 in generating transcriptions in a timely manner. In some implementations, the FAL evaluator component 112 can determine (e.g., calculate) a latency score, rating, or value (e.g., a latency metric) that can characterize the amount of latency.

[0047] According to various embodiments, the model manager component 102 can use the determination of fidelity, accuracy, and / or latency (e.g., a score or rating of fidelity, accuracy, and / or latency) as feedback that can be used to further train (e.g., refine or update the training) the speech recognition model 104 in a subsequent (e.g., next) iteration of the training of the speech recognition model 104. For example, based at least in part on the determination of fidelity, accuracy, and / or latency, the model manager component 102 employing the update component 122 can determine, perform, and / or implement the execution of updates (e.g., adjustments or modifications) to hyperparameters of the speech recognition model 104, other parameters associated with training the speech recognition model 104, a process for generating a mixed-language dataset (e.g., adjusting a conversation prompt for conducting a conversation in a mixed language, or otherwise adjusting the process), instructions for the process for generating a mixed-language dataset, topics for implementing the generation of a mixed-language dataset, and / or other functions or features associated with the training of the speech recognition model 104, such as described herein. If the update involves modifications related to the generation of a mixed-language dataset for a subsequent (e.g., next) iteration of training of the speech recognition model 104, the model manager component 102 can control the operation of the MLDGM component 110 and / or information (e.g., instructions and / or seed topics) input to the MLDGM component 110 based at least in part on the updated update information, so that the subsequent mixed-language dataset generated by the MLDGM component 110 is enhanced compared to the previous mixed-language dataset to facilitate enhanced training and further training of the speech recognition model 104 during subsequent training iterations.

[0048] In some embodiments, the updating may involve the model manager component 102 or another component determining and selecting, or randomly determining and selecting, one or more topics to be used as seed topics for the next round of generating the mixed language dataset by the MLDGM component 110. In some embodiments, in terms of randomly determining and selecting topics, the model manager component 102 or another component may perform such random determination and selection of topics based at least in part on a random sampling function, a random number, a seed value, and / or another randomization factor. For example, the model manager component 102 may include or may manage a random number generator (e.g., a true or pseudo-random number generator) that may generate one (or more) random numbers based at least in part on a random sampling function, a seed value, and / or another randomization factor, wherein the random number may be used to randomly determine or select a topic from a set of topics (e.g., determine or select a topic corresponding to the random number).

[0049] If the update involves modifying hyperparameters associated with the speech recognition model 104, the model manager component 102 can pass the update information to the tuner component 114. The tuner component 114 can perform or implement the modification of the hyperparameters of the speech recognition model 104 (e.g., for reconfiguring the speech recognition model 104) based at least in part on the update information, so as to enhance the training or further training of the speech recognition model 104 during subsequent training iterations. In some embodiments, if there is a first modification determined at least in part based on a loss function (e.g., a first modification to the hyperparameter) and a second modification determined at least in part based on fidelity, accuracy, and / or latency (e.g., a second modification to the hyperparameter), the model manager component 102 or another component (e.g., the tuner component 114) can reconcile any differences between the first modification and the second modification according to the defined model management criteria. For example, the model manager component 102 or another component (e.g., the tuner component 114) can select the first modification instead of the second modification, or vice versa, to be used as part of the update. Optionally, the model manager component 102 or other components (e.g., the tuner component 114) can determine a third modification (e.g., a third modification to the hyperparameters) based at least in part on the first modification and the second modification, and can use the third modification as part of the update.

[0050] The model manager component 102 can manage determining and executing the required number of iterations for updates related to hyperparameters, parameters, processes for generating a mixed-language dataset, instructions for the processes for generating a mixed-language dataset, topics for facilitating the generation of a mixed-language dataset, and / or other functions or features associated with the training of the speech recognition model 104, and the required number of iterations for the training of the speech recognition model 104 (e.g., based at least in part on the updates), in accordance with defined model management criteria, until the speech recognition model 104 reaches (e.g., achieves) a desired performance level, or reaches defined model training termination (e.g., stops) criteria, as described herein. As needed, the model manager component 102 can continue to refine or update the training of the speech recognition model 104 over time as the model manager component 102 receives additional information (e.g., feedback information or other information).

[0051] According to various embodiments, the model manager component 102 can manage the generation and training of corresponding speech recognition models for each language (e.g., English, Spanish, German, Chinese, or other languages) or each mixed language (e.g., Cantonese and English, German and English, Chinese and Spanish, Cantonese and Italian, or other mixed languages), and / or each language dialect. The technology of the subject matter of the present disclosure (e.g., employed by the model manager component 102, the speech recognition component 106, and / or other components described herein) can ideally, flexibly, and scalably manage the generation and training of corresponding speech recognition models for each language or each mixed language and / or each language dialect to ideally meet (e.g., achieve) different language recognition wishes (e.g., needs) or specifications (e.g., requirements).

[0052] The trained speech recognition model 104 can be used for any of a variety of services, applications, or use cases. As some non-limiting examples, the trained speech recognition model 104 can be used for transcription services, language learning platforms, call center automation, multilingual voice assistants, multilingual speech recognition or translation applications or services for mobile devices or computers, multilingual Internet of Things (IoT) devices or applications, and / or other desired services, applications, or use cases.

[0053] refer to Figure 2 (as well as Figure 1 ), Figure 2A block diagram of a non-limiting example model manager component 102 is depicted. According to various aspects and embodiments of the disclosed subject matter, the model manager component 102 can include an MLDGM component 110 that can desirably (e.g., automatically, dynamically, appropriately, reliably, efficiently, enhanced, and / or optimally) determine and generate a mixed language dataset that can be used to train a speech recognition model 104; and a FAL evaluator component 112 that can perform a FAL evaluation related to the performance of the speech recognition model 104 as part of the model training. The model manager component 102 can include the MLDGM component 110, the FAL evaluator component 112, an update component 122, and / or one or more other components. According to various embodiments, the MLDGM component 110 may include a content collector component 202 (e.g., an engineer agent), an evaluator component 204 (e.g., a critic agent), an extractor component 206 (e.g., a manager agent), a dialogue simulator component 208 (e.g., a speaker agent), an assessment component 210 (e.g., a commentator agent), a topic determination component 212, and a text generator component 214. In certain embodiments, the MLDGM component 110 may include a speech generator component 216 and / or another component. The FAL evaluator component 112 may include a fidelity determination component 218, an accuracy determination component 220, a delay determination component 222, and / or another component. According to various embodiments, the model manager component 102 may include (as depicted) or be associated with (e.g., communicatively connected to) a processor component 224 and a data repository 226.

[0054] In some embodiments, the model manager component 102 can employ one or more AI-based (e.g., trained AI-based) models (e.g., GPT-type models, such as GPT-4 models, another type of transformer-based model, another type of multimodal large language model, and / or another type of AI-based model). For example, the model manager component 102 can employ an AI-based model, which can be the MLDGM component 110 and the FAL evaluator component 112, and / or can be used by the MLDGM component 110 and the FAL evaluator component 112, or the model manager component 102 can employ an AI-based model (which can be the MLDGM component 110 and / or can be used by the MLDGM component 110) and another AI-based model (which can be the FAL evaluator component 112 and / or can be used by the FAL evaluator component 112). In certain implementations, one or more artificial intelligence-based models (eg, models of or associated with the MLDGM component 110 ) may be large language models capable of generating data, such as mixed-language data.

[0055] In some embodiments, the MLDGM component 110 (e.g., an artificial intelligence-based model of or associated with the MLDGM component 110) can receive instructions related to data collection and / or generation of a mixed language dataset, and / or seed topic information related to seed topics that can be used to enable generation of a mixed language dataset (e.g., generation of mixed language data related to seed topics and / or other topics derived from or at least partially based on seed topics). These instructions can be created by one or more users (e.g., an operator, technician, or other user), and / or can be generated or customized by the model manager component 102, and / or can be determined based at least in part on feedback information received from the FAL evaluator component 112, the speech recognition component 106, another component, and / or a user. Prior art techniques that employ simple or fixed prompts may undesirably result in or cause undesirable limitations on the number, diversity, and / or creativity of language datasets. To avoid or mitigate such undesirable limitations, the model manager component 102 and / or a user may determine and / or set seed topics (e.g., a topic seed list) to enrich the creativity of the mixed language data determined and generated by the MLDGM component 110. In some embodiments, with respect to generating a mixed language dataset and using the mixed language dataset to implement an iteration of training the speech recognition model 104, in order to facilitate the ideal determination and generation of the mixed language dataset for that iteration, the topic determination component 212 may determine and generate (e.g., randomly determine and generate, or otherwise determine and generate) one or more seed topics based at least in part on a previous mixed language dataset (e.g., a previous or other previous iteration of model training, a previous or other previous mixed language dataset), which can be used to facilitate the determination and generation of the mixed language dataset so as to avoid language data of the same or similar format.

[0056] The MLDGM component 110 can adopt a multi-agent system and process to generate an ideal high-quality mixed language dataset. The language dataset generation framework can decompose the task into less complex ideal sub-tasks and integrate multiple professional agents (e.g., engineer agents, critic agents, manager agents, speaker agents, commentator agents and / or other agents) that can each make unique contributions to the creation and improvement of the mixed language dataset. The multi-agent system architecture adopted by the MLDGM component 110 can achieve efficient processing of different languages ​​and data types, and can enhance the authenticity and applicability of the generated mixed language dataset in practical applications (e.g., practical applications of speech-to-text conversion or other practical applications).

[0057] In some embodiments, the content collector component 202 (e.g., an engineer agent) may include and / or utilize a web crawler and / or other tool to search and / or collect (e.g., retrieve) corresponding content items from various data sources (e.g., various web or online websites, servers, and / or devices) based at least in part on instructions (e.g., instructions for data collection) and / or seed topics. The web crawler and / or other tool can discover and collect a large amount of content in at least two languages, which will become part of the mixed language data set. The content collector component 202 can perform content collection within a desired time period. The corresponding content items may include various video content items, various audio content items, and / or various text content items (e.g., articles, news items, video content, audio content, multimedia content, and / or other types of content), which may use corresponding languages, including at least two languages ​​(e.g., Cantonese, English, and / or other languages). This role of the content collector component 202 can ensure that the raw data including at least two languages ​​continues to flow in, ideally reflecting or representing the current language usage and trends of at least two languages.

[0058] The content collector component 202 or another component of the MLDGM component 110 or the model manager component 102 can annotate (e.g., identify, tokenize, or otherwise annotate) individual content items based at least in part on the content type of the content item (or portion thereof) (e.g., video, audio, text, multimedia, or other type of content), the language of the content item (or portion thereof) (e.g., Cantonese, English, Italian, Spanish, Hindi, Japanese, German, or other languages), the part of speech of the content item (or portion thereof) (e.g., noun, pronoun, verb, adjective, adverb, preposition, conjunction, interjection, or other part of speech), the dialect of the content item (or portion thereof), and / or other types of annotations.

[0059] The evaluator component 204 (e.g., a critic agent) can evaluate (e.g., perform an artificial intelligence-based evaluation or analysis) the performance and efficacy of the web crawler and / or other tools developed and / or used by the content collector component 202 to collect various content items to ensure ideal data diversity in the collected content items, check and identify errors in the collected content items, check and determine the efficiency of collecting various content items, check and determine whether norms and / or guidelines related to content collection (e.g., ethical, guiding or specified web scraping or data collection guidelines) are followed. The evaluator component 204 can determine and generate feedback information and / or content collection updates based at least in part on the evaluation results of the performance and efficacy of the web crawler and / or other tools of the content collector component 202 or other tools associated with the content collector component 202 to collect various content items. The assessor component 204 can provide (e.g., communicate or transmit) the feedback information and / or content collection updates to the model manager component 102 (e.g., the content collector component 202 or other components of the MLDGM component 110, or the model manager component 102) and / or the user. The feedback information and / or content collection updates can be used to improve the collection of content by the content collector component 202 (e.g., improve the data diversity of the collected content, reduce or mitigate errors in the collected content, improve the efficiency of content collection, and / or mitigate or eliminate deviations from specifications or guidelines related to content collection) and ensure that content collection tools (e.g., web crawlers and / or other tools) can remain robust and effective over time. The evaluator component 204 or another component of the model manager component 102 and / or a user may determine content collection updates based at least in part on the evaluation results of the collected content items and applicable specifications and / or guidelines related to the content collection, wherein the results may represent or be related to the data diversity of the collected content items, any errors identified in the collected content items, the efficiency of collecting the content items, and / or the degree of compliance with the specifications or guidelines related to collecting the content items. The model manager component 102 (e.g., the evaluator component 204 or another component of the model manager component 102) may update (e.g., modify or adjust) the content collector component 202 and / or a web crawler or other data collection tool or a web crawler or data collection tool associated therewith in accordance with defined model management standards, at least in part based on the updated update information.

[0060] In some embodiments, the extractor component 206 (e.g., a manager agent) can analyze each content item and, based at least in part on the analysis results, identify and / or extract desired (e.g., relevant and / or desired) topics, keywords, and / or other topics in and / or from each content item in accordance with defined model management criteria. For example, the extractor component 206 can employ a desired trained natural language processing model that can identify topics, topics, keywords, and / or other desired topics that may be prevalent (e.g., frequently or typically occurring) in each content item based at least in part on the analysis results (e.g., artificial intelligence-based analysis) of each content item. These topics can be related to people's daily lives and can be related to, for example, the environment, social media, history, literature, hobbies, interests, music, art, movies, television shows, technology, news, pets, health and fitness, work, study, shopping, news (e.g., local news, national news, and / or international news), sports, entertainment, travel, food, weather, politics, law, and / or other topics. The extractor component 206 can determine and / or decide ideal (e.g., most ideal (e.g., most suitable, most desired, most enhanced, or optimal)) models and techniques for processing different types of data, ensuring that the mixed-language dataset generation phase of the process is ideally targeted and efficient.

[0061] The dialogue simulator component 208 (e.g., speaker agent) can simulate (e.g., synthesize or emulate) one or more corresponding dialogues between various speakers (e.g., simulated or emulated speakers) speaking in a mixed language (e.g., at least two languages, such as Cantonese and English (and / or another desired language)) about one or more corresponding topics according to defined model management criteria, at least in part based on topics, keywords and / or other topics extracted from various content items. The simulated dialogues between the various speakers can be simulations or emulations of how speakers (e.g., native speakers speaking the mixed language) would discuss (e.g., might or probably would discuss) the identified topics, the simulations or emulations being conducted at least in part based on (e.g., using) desired (e.g., relevant and / or desired) topics, keywords and / or other topics according to different prompts (e.g., dialogue prompts). With respect to each of the one or more corresponding conversations being simulated, the individual speakers of the simulated conversation can chat with each other (e.g., operated or managed by the conversation simulator component 208), and the conversation simulator component 208 can generate text data (e.g., text transcription) based at least in part on (e.g., corresponding to, representing, or transcribing) the simulated conversation.

[0062] In some embodiments, as one or more corresponding conversations are simulated and generated, and as individual text data corresponding to the one or more corresponding conversations are generated, the assessment component 210 (e.g., a reviewer agent) can assess (e.g., evaluate) the grammar, wording, coherence, and / or other qualities of the text data corresponding to the one or more corresponding simulated conversations in mixed languages ​​between the respective speakers (e.g., simulated speakers). The assessment component 210 can assess the individual text data to ensure that the corresponding text data of the one or more simulated conversations meets (e.g., meets or exceeds) the required (e.g., defined and / or high) language standards and can represent the natural language that people can typically use in a mixed language environment, where a person can use multiple languages ​​to speak individual words in a conversation (e.g., in the same sentence). In some embodiments, the assessment component 210 can include, employ, or incorporate a mechanism that can score, classify, and / or sort the text data items of the individual simulated conversations based, at least in part, on quality metrics (e.g., grammar, wording, coherence, and / or other quality metrics). The assessment component 210 can use or select the best performing data set (e.g., the best performing text data item) from among the various text data (e.g., the top k text data or the best performing text data determined in other ways) to update (e.g., modify and / or enhance) the prompts of the speaker (e.g., the simulated speaker of the speaker agent), thereby enabling a continuous improvement cycle (e.g., a feedback loop) that can refine (e.g., enhance) the data output (e.g., the mixed-language text data) in successive cycles (e.g., iterations) of the simulated dialogue that are associated with iterations of generating mixed-language data sets and using these mixed-language data sets to iteratively train the speech recognition model 104. For example, the assessment component 210 can determine and generate an update that includes update information related to the best performing data set. The assessment component 210 or another component of the MLDGM component 110 or the model manager component 102 can update the dialogue simulator component 208 (e.g., the artificial intelligence-based model of the speaker agent or an artificial intelligence-based model associated therewith) based at least in part on the updated information regarding the best performing data set input to the dialogue simulator component 208 to improve or refine the prompts (e.g., dialogue prompts) of the speaker (e.g., the simulated speaker of the speaker agent) and / or other functions of the dialogue simulator component 208 to improve or refine the data output (e.g., mixed-language text data) of the dialogue simulator component 208 in subsequent cycles of dialogue simulation performed by the dialogue simulator component 208 according to defined model management standards.

[0063] In some embodiments, the MLDGM component 110 and / or the trained model can employ a desired algorithm (e.g., a mixed language dataset generation algorithm) that can implement (e.g., for each of one or more iterations) determining and / or generating all or at least a portion of the mixed language dataset. For example, the MLDGM component 110 and / or the trained model can utilize the following non-limiting example algorithm (e.g., Algorithm 1) and associated pseudo code to implement determining and / or generating all or at least a portion of the mixed language dataset, such as as follows:

[0064] Algorithm 1 Generate data

[0065] Input: None

[0066] Output: CSV file

[0067] 1: Initialize csv_file1

[0068] 2: For i=1,2,...,160, do

[0069] 3: df ← pd.read_csv(csv_file i )

[0070] 4: topic_list←df['topic'].unique()

[0071] 5: For the topic in topic_list, do

[0072] 6: instance_list←filter_data(df)

[0073] 7: instance1, instance2

[0074] random.sample(instance_list['topic'],2)

[0075] 8: prompt←instruction+topic+instance1+instance2

[0076] 9: df′←GPT4_Generate(prompt)

[0077] 10: pd.to_csv(csv_file i+1 ,df′)

[0078] 11: End for

[0079] 12: End for

[0080] For example, according to the present example algorithm (e.g., Algorithm 1), the MLDGM component 110 and / or the trained model may start with a desired number of CSV files (e.g., 160 CSV files or another desired number of CSV files) of comma-separated value (CSV) files, and may loop (e.g., iterate) through the CSV files, wherein, for a desired number of iterations (e.g., 160 iterations, or another desired number of iterations), the MLDGM component 110 and / or the trained model may (e.g., for each iteration) perform the following operations: read the CSV files into a data frame (df) (e.g., pd.read_csv); identify unique topics in the data frame and generate a topic list (e.g., topic_list) including the unique topics; for each Topic (e.g., a topic in a topic list): filter data (e.g., filter_data(df)) to obtain instances related to the topic and generate an instance list (e.g., instance_list) including the instances; randomly select or sample two instances (e.g., instance1, instance2) from the filtered data; create a prompt (e.g., a conversation prompt) using the instruction, topic, and / or the two instances; generate new data (e.g., df') based at least in part on the prompt using a trained model (e.g., a GPT-4 model or other trained model) and save the generated data to a new CSV file; and (e.g., during the next iteration) repeat by continuing the process for each CSV file. In some embodiments, the MLDGM component 110 and / or the trained model can determine and / or generate all or a desired portion of a mixed language dataset, such as described herein, based at least in part on the application or execution of the example algorithm by the MLDGM component 110 and / or the trained model.

[0081] The topic list can include an ideal number of different topics (e.g., current and / or related topics, such as described herein) that can ensure ideal data diversity for the mixed-language dataset. With respect to each topic, the MLDGM component 110 and / or the trained model can provide, generate, or utilize an ideal number of instances (e.g., more than 10 instances, or 10 instances or less, if desired) that are related to respective conversations (e.g., mixed-language conversations) that may occur between people regarding the topic. In some embodiments, the MLDGM component 110 and / or the trained model can utilize two instances (or another desired number of instances) and the desired topics of the topic list (e.g., for each topic of the topic list) as prompts, according to the algorithm, to determine and generate additional instances related to respective conversations about the topic. In some embodiments, the MLDGM component 110 and / or the trained model may perform a re-sampling of instances generated in a previous operation (e.g., using two (or more) instances and a desired topic as prompts) to determine and generate (e.g., cause to be generated) a set (e.g., a collection) of updated instances related to the topic, which may be used as prompts for further generating instances about the topic (or another topic). This iterative process employed by the MLDGM component 110 and / or the trained model may allow the MLDGM component 110 and / or the trained model to continuously expand the mixed-language dataset while maintaining the diversity and richness of words (e.g., mixed-language words) and topics in the mixed-language dataset.

[0082] With respect to the random sampling of instances disclosed in the example algorithm 1, the random sampling of instances can be performed based at least in part on a random sampling function, a random number, a seed value, and / or another randomization factor. For example, the model manager component 102 can include or can manage a random number generator (e.g., a real or pseudo-random number generator) that can generate one (or more) random numbers based at least in part on a random sampling function, a seed value, and / or other randomization factors, wherein the random number can be used to randomly sample or select instances from the filtered data (e.g., sample or select instances corresponding to the random number).

[0083] It should be recognized and understood that this non-limiting example algorithm (e.g., Algorithm 1) is only an example algorithm that the MLDGM component 110 and / or trained model can utilize to implement all or at least a portion of determining and / or generating a mixed language dataset, and according to other embodiments, the MLDGM component 110 and / or trained model can utilize another desired algorithm to implement all or at least a portion of determining and / or generating a mixed language dataset, for example, such as described herein. It should also be recognized and understood that although Algorithm 1 references GPT4 as a trained model, in other embodiments, the trained model can be a different type of model than the GPT4 model. In addition, it should be recognized and understood that although Algorithm 1 references the use of a CSV file, in other embodiments, a file of a different type (e.g., structured or unstructured) format (e.g., another structured format or an unstructured natural language format) can be used with the example Algorithm 1 or another desired algorithm that the MLDGM component 110 and / or trained model can utilize to facilitate determining and / or generating all or at least a portion of a mixed language dataset.

[0084] The mixed language dataset generated by the MLDGM component 110 and / or the trained model may include a relatively large number of sentences (e.g., tens of thousands of sentences, or even more than 15,000 sentences as desired), including all or a desired portion of the sentences being mixed language sentences, which may include a combination of words and characters from two (or more) languages, wherein the individual sentences may be related to and cover the desired individual topics. In the mixed language dataset, there may be various numbers of sentences related to various topics, wherein certain topics may be associated with relatively large numbers of sentences, respectively, and certain other topics may be associated with relatively small numbers of sentences, respectively, depending on the individual topics (e.g., individual topic types), the current corresponding relevance or topicality of the individual topics, and / or other factors. In addition, the number or frequency of second language words or characters interspersed with first language words or characters, and the ratio of first language words or characters to second language words or characters, may vary based at least in part on the topic type, current language usage characteristics associated with the first language and the second language, and / or other factors. As a non-limiting example, with respect to a non-limiting mixed-language dataset, a food topic (e.g., a food category) may have a relatively high frequency of second language words or characters (e.g., English words) interspersed with first language words or characters (e.g., Chinese characters), while the ratio of first language words or characters to second language words or characters is relatively low (e.g., 3:1, or other relatively low ratios). In contrast, in a non-limiting mixed-language dataset, a technology news topic may have a relatively low frequency of second language words or characters (e.g., English words) interspersed with first language words or characters (e.g., Chinese characters), while the ratio of first language words or characters to second language words or characters is relatively high (e.g., 8:1, or other relatively high ratios). For each mixed-language dataset, the respective frequencies and ratios associated with each topic may change (e.g., over time, and / or as the speech recognition model 104 is iteratively trained and as updates related to the generation of the mixed-language dataset are performed).

[0085] In some embodiments, the MLDGM component 110 can output (e.g., present or transmit as output) a mixed language dataset, including transcriptions of respective text data corresponding to or representing one or more respective mixed language simulated conversations (e.g., text transcription data), wherein respective second words of the second language can be interspersed (e.g., inserted, intermingled, or mixed) with respective first words of the first language (e.g., between respective first words of the first language). In certain embodiments, the MLDGM component 110 can separately annotate (e.g., identify, mark, or otherwise annotate) respective items of the transcribed text data (e.g., respective words, respective phrases, or other respective text data items) to facilitate the training of the speech recognition model 104.

[0086] In some embodiments, an audio recorder component 116 (e.g., managed by the model manager component 102 or another component of the system 100) can record a speaker (e.g., a person speaking a word) speaking each first word and each second word in a mixed language dataset (e.g., a speaker having a conversation using each first word and each second word in a text transcription data of the mixed language dataset), and generate an audio recording (e.g., recorded audio content) including each word (e.g., each first word and each second word) spoken by the speaker, such as described herein. Such an audio recording of a person speaking a mixed language word in a mixed language dataset may be more desirable than using a simulated speaker to simulate or synthesize the pronunciation of the words in the mixed language dataset, because speech synthesis technology, or at least some speech synthesis technology, may not be ideally (e.g., appropriately or acceptably) advanced enough to perform mixed language speech synthesis. The MLDGM component 110 can ideally organize the mixed language dataset so that the audio content representing the mixed language dataset can ideally (e.g., appropriately, enhanced, or optimally) include a variety of audio samples that represent a variety of mixed languages ​​in the target domain.

[0087] In other embodiments, additionally and optionally, the MLDGM component 110 may employ a speech generator component 216 that may generate audio content that simulates individual words of a mixed language data set spoken by a speaker (e.g., as synthesized speech) corresponding to and / or representative of the mixed language words of the mixed language data set, and present it as output. For example, the speech generator component 216 may desirably (e.g., appropriately, acceptably, and / or enhanced) include the ability to synthesize or simulate the pronunciation of words in a mixed language data set, where the spoken conversation represented or presented in the mixed language data set is interspersed with words of different languages, and accordingly, the speech generator component 216 may be used to generate audio content that simulates individual words of the mixed language data set spoken by a speaker (e.g., as synthesized speech), and present it as output.

[0088] In some embodiments, the audio content of the spoken mixed language words representing the mixed language data set (e.g., a mixed language audio data set) may include various parts of the audio content, and the various parts of the audio content may include various instances of conversations related to various topics (e.g., a list of topics) (e.g., various instance parts of the conversation), wherein each instance may include one or more sentences, and these sentences include mixed language words in two (or more) languages ​​(e.g., second language words and first language words are interspersed in the same sentence). The various parts of the audio content including the various instances may have corresponding time lengths and may include corresponding numbers of two (or more) language words. As a non-limiting example of a mixed language data set, the length of a relatively large part (e.g., most) of the various parts of the audio content may be in the range of 5 seconds to 12 seconds, and the length of a relatively small part may be less than 5 seconds or greater than 12 seconds (e.g., the length may reach or even exceed 25 seconds), although for other mixed language data sets, the corresponding lengths of the various parts of the audio content and the time length distribution of the various parts of the audio content may be different from this non-limiting example.

[0089] In some embodiments, the model manager component 102 (e.g., the MLDGM component 110, the trained model, or other components of the model manager component 102), the audio recorder component 116 (e.g., controlled by the model manager component 102), or the speech generator component 216 (e.g., controlled by the model manager component 102) can sample or distribute (e.g., randomly sample or distribute) audio content (e.g., portions of audio content, such as individual audio files and / or segments of individual conversations or spoken words) to divide the portions of audio content representing the mixed-language data set into a group (e.g., a set) of training audio content portions and a group of validation or test audio content portions in a desired ratio (e.g., a ratio of training audio content portions to validation or test audio content portions of 9:1, or other desired ratios higher or lower than 9:1). The random sampling or distribution of the audio content portions can be performed at least in part based on a random sampling or distribution function, a random number, a seed value, and / or another randomization factor. For example, the model manager component 102 may include or may manage a random number generator (e.g., a true or pseudo-random number generator) that can generate random numbers based at least in part on a random sampling or distribution function, a seed value, and / or other randomization factors, wherein the random numbers can be used to determine, identify, or select an audio content portion that can be assigned to a validation or test audio content portion group (or, alternatively, to a training audio content portion group).

[0090] The audio content representing the spoken mixed-language words of the mixed-language dataset may be preprocessed (e.g., managed by the model manager component 102 or the speech recognition component 106) to generate features representing the audio content that may be ideal (e.g., appropriate, acceptable, or optimal) for training the speech recognition model 104. In some embodiments, the audio converter component 118 may generate a spectrogram that may include visual information (e.g., visual or graphical features) that may characterize individual words spoken by a speaker in the audio content (e.g., an audio recording created using a human speaker speaking mixed-language speech, or audio content generated using a simulated speaker synthesizing mixed-language speech). In some embodiments, the spectrogram may be a log-mel spectrogram that captures the frequency content of the audio content over time and provides an ideally compact representation of individual words spoken by a speaker in the audio content. The log-mel spectrogram may employ a mel scale when representing frequencies in the frequency content. In other embodiments, the spectrogram may be a different type of spectrogram (eg, a mel spectrogram or other type of spectrogram) that captures the frequency content of the audio content over time and provides an ideal representation of each word spoken by a speaker in the audio content.

[0091] The tokenizer component 120 may generate a token group including individual tokens based at least in part on the analysis results of the text transcription data, according to a desired token generation and / or model training format, which tokens may represent individual words or subwords (e.g., syllables, fragments, or other parts of words) in the corresponding words (e.g., mixed language words) in the text transcription data (e.g., annotated text transcription data) of the mixed language dataset, such as described herein. In some embodiments, with respect to multi-task training of the speech recognition model 104, the tokenizer component 120 may generate and format individual tokens representing individual words or subwords in a desired (e.g., specific or particular) multi-task model training format, which may enable the speech recognition model 104 to be trained concurrently, simultaneously, or in parallel on multiple related tasks (such as speech recognition, language modeling, and / or another desired task). In some embodiments, a token may include or be related to SOT (e.g., start of transcription, start of turn, or start of track), language (LANG), task, timestamp (TS), text, or EOT (e.g., end of transcription, end of turn, or end of track). SOT can be a start boundary token related to the start of text, language can be related to the language represented by the token (e.g., Cantonese, English, or other language), task can be related to the type of task (e.g., transcription, speech recognition, or other type of task), timestamp can be related to the time associated with the token (e.g., month, day, year, moment, hour, minute, second, or other time indication) and / or to the temporal position of the token and the word or subword it represents relative to other words or subwords, text can be related to the output type (e.g., text data as output), and EOT can be an end boundary token related to the end of text. In some embodiments, the tokenizer component 120 can generate each token of the token group for each word or subword, and the generation can be performed concurrently, simultaneously, or in parallel with the generation of the spectrogram by the audio converter component 118.

[0092] During each iteration of generating the mixed-language dataset, the model manager component 102 (e.g., the MLDGM component 110, and / or another component of the model manager component 102) can pair (e.g., associate, link, map, or otherwise pair) the audio content representing the mixed-language dataset with the transcription (and / or word tuples) representing the mixed-language dataset to facilitate training of the speech recognition model 104.

[0093] With respect to the training and operation of the speech recognition model 104, during the fine-tuning process, the fine-tuner component 114 can initialize, fine-tune, or configure the speech recognition model 104, or implement fine-tuning or configuration, based at least in part on a set of hyperparameters, parameters (e.g., model weights or biases, or other parameters), and / or other information (e.g., parameters or other settings or configurations). For example, the fine-tuner component 114 can pass desired hyperparameters, parameter groups, and / or other information to the speech recognition model 104, where the speech recognition model 104 can be configured (e.g., hyperparameters or other parameters can be configured or set) based at least in part on the set of hyperparameters, parameters, and / or other information. Specific hyperparameters in the hyperparameter group can be based at least in part on certain factors, such as the artificial intelligence-based model type of the speech recognition model 104, the adjustment or improvement of one or more hyperparameters (e.g., to facilitate the training or improvement of the speech recognition model 104), and / or other factors. For example, the hyperparameters may include or be related to the number of epochs, the batch size, the number of layers (e.g., neural network layers), the number of nodes in each layer, and / or another desired hyperparameter associated with the speech recognition model 104. In some embodiments, during initial iterative training and / or subsequent iterative training of the speech recognition model 104, the hyperparameters and / or parameters, or at least some of the hyperparameters and / or parameters, may be determined or learned through a pre-training task (e.g., a relatively large-scale pre-training task), such as unsupervised training on an ideal (e.g., relatively large or massive) amount of audio data. According to defined model management standards, the desired hyperparameters and / or parameter groups used by the speech recognition model 104, and / or the operations or processes performed by the speech recognition model 104, may be updated (e.g., adjusted, modified, or reconfigured) based at least in part on the results of a FAL evaluation, WER, CER, and / or other evaluation of the quality of speech recognition performed by the speech recognition model 104, such as described herein.

[0094] The audio converter component 118 (or another component) can input or apply the spectrogram information of the spectrogram to the speech recognition model 104, and / or the word segmenter component 120 can input or apply the word tuple associated with the spectrogram to the speech recognition model 104 (e.g., managed by the model manager or the speech recognition component 106). The speech recognition model 104 (e.g., a mixed language speech recognition model) can analyze (e.g., perform artificial intelligence-based analysis) the spectrogram information of the spectrogram and / or the individual word tuples of the word tuples. In some embodiments, the speech recognition model 104 can analyze the spectrogram information and the individual word tuples simultaneously, concurrently, or in parallel.

[0095] Based at least in part on the results of analyzing the spectrogram information and / or individual tokens, the speech recognition model 104 can be trained to perform speech recognition, can learn to perform speech recognition, and can perform speech recognition on information about or representing audio content (e.g., spectrogram information, tokens, and / or other information), the audio content including spoken words of a mixed-language data set. During the process of training the speech recognition model 104 and performing speech recognition by the speech recognition model 104, the speech recognition model 104 can perform next token prediction, can generate predicted tokens as output, and / or can determine and generate transcriptions as output, including transcribed text data (e.g., transcribed text data representing mixed-language words represented in the information input to the speech recognition model 104), such as described herein.

[0096] refer to Figure 3 (as well as Figure 1 and Figure 2 ), Figure 3 A block diagram of a speech recognition component 106 including a speech recognition model 104 is shown, in accordance with various aspects and implementations of the disclosed subject matter. According to various implementations, the speech recognition component 106 can include the speech recognition model 104, a tuner component 114, an audio converter component 118, a tokenizer component 120, a loss determination component 302, an update component 304, a processor component 306, and / or a data repository 308.

[0097] In some embodiments, the speech recognition model 104 can employ a transformer-based encoder-decoder architecture, which can be composed of an encoder component (ENCODERCOMP) 310 and a decoder component (DECODERCOMP) 312. As disclosed, in some embodiments, the data input to the encoder component 310 can be annotated speech-related data (e.g., text transcription data, such as annotated text transcription data, and / or spectrogram information), which can represent a mixed-language dataset for analysis and processing by the speech recognition model 104 to facilitate training the speech recognition model 104 and / or performing speech recognition on the input data to generate a transcription including transcribed text data (e.g., including text words in a mixed language), which can represent and / or correspond to spoken mixed-language words in audio content, which can represent words in the mixed-language dataset (e.g., an enhanced mixed-language dataset). The speech recognition model 104 employing the encoder component 310 and the decoder component 312 can perform next word-gram prediction to predict the next word-gram in a word-gram sequence based at least in part on the results of analyzing and processing (e.g., encoding and decoding) the input data, where the next word-gram can represent the next word or sub-word in a word sequence (e.g., a sentence (or other set of words) of spoken mixed-language words in audio content), which word sequence can be represented by a word-gram sequence, such as described herein.

[0098] The encoder component 310 may include one or more encoder blocks (e.g., transformer encoder subcomponents), such as encoder blocks (ENCBLK) 314, 316, and / or 318, which may be associated (e.g., communicatively connected) with each other (e.g., the output of encoder block 314 may be associated with the input of encoder block 316, and the output of encoder block 316 may be associated with the input of the next encoder block). The encoder blocks (e.g., 314, 316, and / or 318) may process or analyze the audio features represented in the spectrogram information and extract a high-level representation of the audio features based at least in part on the results of such processing or analysis (e.g., a sequence of context-aware representations of the spectrogram information and the associated audio features may be generated). The encoder blocks (e.g., 314, 316, and / or 318) may include multiple layers, each of which may include a self-attention layer and a feed-forward neural network layer, such as described herein.

[0099] The decoder component 312 may include one or more decoder blocks (e.g., converter decoder subcomponents), such as decoder blocks (DECBLK) 320, 322, and / or 324, which may be associated (e.g., communicatively connected) with each other (e.g., the output of decoder block 320 may be associated with the input of decoder block 322, and the output of decoder block 322 may be associated with the input of the next decoder block). The output of the encoder component 310 may be associated with the respective input ends (e.g., respective input ports) of the respective decoder blocks (e.g., 320, 322, and / or 324) of the decoder component 312. The decoder blocks (e.g., 320, 322, and / or 324) may generate output transcriptions (e.g., text transcriptions) of mixed-language words representing the mixed-language dataset based at least in part on the results of processing or analyzing the encoded audio representations of the audio features. The decoder blocks (e.g., 320, 322, and / or 324) may include multiple layers, each of which may include a self-attention layer and a feed-forward neural network layer, such as described herein.

[0100] The speech recognition model 104 may also include an activator component (ACTVTR) 326 that may be associated with the input of the encoder component 310 and may be used to implement activation of the speech recognition model 104. In some embodiments, the activator component 326 may include 2x one-dimensional (1D) convolutional layers (Conv1D), Gaussian error linear unit (GELU) activator functions that may perform convolution operations on one-dimensional sequence data (e.g., spectrogram information, word or text transcription data, or other data) as part of the analysis of such data. The Conv1D layer may create a convolution kernel that may be convolved with the layer input (e.g., sequence data) in a single spatial or temporal dimension to generate an output tensor. The GELU activation of the activator function may include performing a desired GELU activation operation that may weight the data (e.g., sequence data) input to the activator component 326 by its probability under a Gaussian distribution (e.g., individual weight values ​​may be applied to the data). The activator component 326 (e.g., using Conv1D) can generally be used to analyze time signals or text data. In other embodiments, the activator component 326 may include features different from the Conv1D layer and / or GELU, such as, for example, a rectified linear unit (ReLU), an exponential linear unit (ELU), or a parameter ReLU (PReLU) (e.g., instead of GELU). The output data output from the activator component 326 (e.g., generated based at least in part on the analysis of the input data input to the activator component 326) can be input to the encoder component 310 (e.g., input to the encoder block 314). The output data input to the encoder component 310 can implement (e.g., enable) the encoder component 310 to perform sinusoidal position encoding on the data. The temporal information of the audio features of the audio content (e.g., the spectrogram information representing the audio content) can be combined to the speech recognition model 104 using the sinusoidal position encoding of the data, which can assist the speech recognition model 104 in understanding the order of the input data (e.g., the spectrogram information representing the audio content).

[0101] In some embodiments, each of the individual encoder blocks (e.g., 314, 316, and / or 318) of the encoder component 310 may include a respective self-attention component (SELF-ATT) (e.g., 328, 330, and / or 332) and a respective multi-layer perception (MLP) component (e.g., 334, 336, and / or 338), which may be associated with (e.g., in communication with) the respective self-attention component (e.g., 328, 330, and / or 332). The respective self-attention components (e.g., 328, 330, and / or 332) may be or may include a desired number of respective self-attention layers that may enable the self-attention component (and the speech recognition model) to determine (e.g., dynamically or automatically determine) the relative importance of respective data items (e.g., words or subwords) in a sequence of data items, thereby enabling the self-attention component (and the model) to capture long-range dependencies in the data (e.g., words). For example, a self-attention component (e.g., 328, 330, or 332) can analyze words (e.g., mixed-language words) in a sentence to determine and / or obtain a context for the sentence, which can facilitate processing (e.g., natural language processing) of a sequence of words in the sentence (e.g., spectrogram information representing a sentence, or a sequence of lemmas representing a sequence of words), and identify the words. As part of the self-attention process, the self-attention component (e.g., 328, 330, or 332) can, for example, assign a query to each word in the sentence, compare the query associated with the sentence with a key, which can be determined or derived from the words in the sentence, so as to determine or identify the information most relevant to the sentence. The self-attention component can combine the various information items from the self-attention process (and as part of the self-attention process) and weight the various information items based on their respective relevance to determine and generate a contextual representation of each word in the sentence. In some embodiments, position-based information can be associated with (e.g., added to) a representation of a word in a sentence to facilitate (e.g., enable) the self-attention component to understand or determine the order and arrangement of words in the sentence.

[0102] In some embodiments, each self-attention component (e.g., 328, 330 and / or 332) can utilize three matrices, including a query matrix, a key matrix, and a value matrix, which enable each self-attention component to determine, understand, and / or process the relationship between words in a sentence (or other paragraph or word set). The query matrix can focus on the words of interest in the sentence (or other paragraph or word set), the key matrix can determine or measure the correlation between words in the sentence, and the value matrix can provide context, thereby facilitating the determination or generation of the final or overall context representation of the focus word. The query matrix, the key matrix, and the value matrix can work together to enable the self-attention component (e.g., 328, 330, or 332) to ideally determine, identify, or capture the various relationships and dependencies between the various words in the sentence.

[0103] Further with respect to the query matrix, the query matrix can represent a focus word about which the self-attention component (e.g., 328, 330, or 332) is determining context. Based at least in part on the results of the analysis of information about the sentence, the self-attention component can utilize the query matrix of the word to transform the word representation and determine and / or generate a query vector that can be compared to other words in the sentence.

[0104] The self-attention component (e.g., 328, 330, or 332) can use a key matrix to determine and / or generate key vectors for words in a sentence based at least in part on the results of an analysis of information about the sentence (e.g., spectrogram information and / or word units). The self-attention component can use the key vector to determine or measure the correlation or similarity between a focus word (e.g., using an associated query vector) and other words in the sentence. A higher correlation or similarity score between a query vector associated with the focus word and a key vector can indicate a relatively stronger (e.g., relatively more important or greater) relationship between the corresponding (e.g., corresponding) words, whereas a relatively lower correlation or similarity score between a query vector and a key vector can indicate a relatively weaker relationship between the corresponding words.

[0105] The self-attention component (e.g., 328, 330, or 332) can use the value matrix to determine and / or generate the value vector of the word in the sentence, wherein each value vector can contain the corresponding context information of the corresponding word. The self-attention component can determine the weighted sum of the value vector after determining (e.g., calculating) the corresponding relevance or similarity score based at least in part on each query vector and key vector. The self-attention component can determine the weight of each value vector in the value vector at least in part based on the relevance or similarity score, so that (and / or ensure) the final or overall context representation (can) be more affected by the relevant words in the sentence. In some embodiments, as part of the self-attention process, the self-attention component (e.g., 328, 330, or 332) can adopt, determine, adjust, and / or apply each attention weight, such as, for example, the query weight, key weight, and value weight associated with the query component (e.g., query matrix), key component (e.g., key matrix), and value component (e.g., value matrix), respectively, so as to determine, identify, or capture the corresponding relationships and dependencies between the words in the sentence.

[0106] Each MLP component (e.g., 334, 336, and / or 338) may be, or may include, each feed-forward layer (e.g., each feed-forward neural network layer), which may include neurons (e.g., fully connected neurons) that may have an activation function (e.g., a nonlinear activation function or a linear activation function). In some embodiments, each MLP component (e.g., 334, 336, and / or 338) may include three or more layers (e.g., an input layer, an output layer, and one or more layers (e.g., hidden layers) between the input layer and the output layer) of nonlinear activation nodes. For a fully connected MLP component, each node in a layer may be associated with a node in the next layer of the MLP component (e.g., may be connected to a node in the next layer through a corresponding weight value). In certain embodiments, each MLP component (e.g., 334, 336, and / or 338) may be trained and learned using a back-propagation technique. For example, in an MLP component (e.g., 334, 336, or 338), learning in perception (e.g., as part of supervised learning) can be performed or occur by modifying (e.g., adjusting, updating, or changing) connection weights (e.g., connection weights between nodes) after processing each item of data, the modification being based at least in part on the amount of error in the output determined relative to (e.g., compared to) an expected output (e.g., an expected result), wherein error information can be back-propagated to facilitate determining and modifying the connection weights. Respective MLP components (e.g., 334, 336, and / or 338) can receive respective output data from respective self-attention components (e.g., 328, 330, and / or 332), analyze and process such respective output data (such as described herein), and transform the respective output data to generate outputs (e.g., encoded output data) that can be output from encoder component 310 (e.g., via an output layer of the MLP component) and input to decoder component 312.

[0107] In some embodiments, each decoder block (e.g., 320, 322, and / or 324) of the decoder component 312 can include each self-attention component (e.g., 340, 342, and / or 344), each cross-attention component (CRS-ATT) (e.g., 346, 348, and / or 350), and each MLP component (e.g., 352, 354, and / or 356), wherein the corresponding self-attention component can be associated with (e.g., communicatively connected to) the corresponding cross-attention component, and the corresponding cross-attention component can be associated with the corresponding MLP component. Each decoder block (e.g., 320, 322, and / or 324) of the decoder component 312 can receive each encoding information from each MLP component (e.g., 334, 336, and / or 338) of each encoder block (e.g., 314, 316, and / or 318) of the encoder component 310. The decoder blocks (e.g., 320, 322, and / or 324) of the decoder component 312 can analyze the respective encoded information and can use the respective encoded representations contained in the respective encoded information as context to facilitate prediction, determination, and / or generation of an output sequence of words (e.g., text words) that can represent words (e.g., spoken words) contained in the audio content. In some embodiments, the decoder component 312 can predict, determine, and / or generate one word-gram at a time (e.g., using autoregressive decoding or another decoding technique), which word-gram can represent a word or a sub-word. In some embodiments, the decoder component 312 can initiate (e.g., start) decoding of the encoded information (e.g., encoded information representing a sentence or other word set) by using a special word-gram (e.g., SOT word-gram (or beginning of sentence (BOS) word-gram or other desired type of initialization word-gram)) as an initial input for analysis by the decoder block (e.g., 320, 322, or 324). The processing and decoding of the encoded information of a sentence (or other set of words) and the prediction of the next word-gram by the various decoder blocks (e.g., 320, 322, and / or 324) of the decoder component 312 can continue, for example, until the decoder component 312 processes the EOT word-gram (or other required special ending word-gram, such as an end-of-sentence (EOS) word-gram).

[0108] Each self-attention component (e.g., 340, 342, and / or 344) of the decoder component 312 can be, or can include, a desired number of corresponding self-attention layers, and each self-attention component (e.g., 340, 342, and / or 344) of the decoder component 312 can be similar in structure and / or can operate similarly to each self-attention component (e.g., 328, 330, and / or 332) of the encoder component 310, except that, for example, each self-attention component (e.g., 340, 342, and / or 344) of the decoder component 312 can operate on each word of the target sequence, and the self-attention layers of each self-attention component (e.g., 340, 342, and / or 344) of the decoder component 312 can focus on or analyze earlier positions in the output sequence, while future positions can be masked out. Each self-attention component (e.g., 340, 342, and / or 344) can determine (e.g., calculate) the interaction between each target word and other target words in the target sequence.

[0109] Each cross-attention component (e.g., 346, 348, and / or 350) can be, or can include, each cross-attention layer that can be used to facilitate predicting, determining, and / or generating a next word based, at least in part, on a context represented in an encoded representation contained in a portion of the encoded information being processed and / or one or more previously predicted words. As part of the decoding process, each cross-attention component (e.g., 346, 348, and / or 350) can determine (e.g., calculate) each attention weight between the sequence of encoded information from the encoder component 310 and the sequence of decoded information generated by the decoder component 312. In some embodiments, when processing the encoded information, the cross-attention component (e.g., 346, 348, or 350) can utilize a query vector (which can be taken from the decoded information generated by the decoder component 312) and a key vector and a value vector (which can be taken from the encoded information received from the encoder component 310).

[0110] Thus, during various operations of the decoding process, various self-attention components (e.g., 340, 342, and / or 344) and various cross-attention components (e.g., 346, 348, and / or 350) can utilize attention, such as self-attention, cross-attention, and / or multi-head attention, to focus on the encoding representations and previously predicted tokens contained in the respective encoding information (e.g., tokens previously predicted and generated by the decoder component 312 when decoding the encoding information) when analyzing and processing the respective encoding information. This can enable the decoder component 312 to determine which parts of the encoding information (e.g., which encoding representations or other parts of the encoding information) and which parts of the previously predicted tokens are relevant to predicting and generating the next token (e.g., the next token that can represent the next word or subword in the predicted token sequence).

[0111] The individual MLP components (e.g., 352, 354, and / or 356) of the decoder component 312 may be, or may include, individual feed-forward layers and may be similar to the individual MLP components (e.g., 334, 336, and / or 338) of the encoder component 310, except that the individual MLP components (e.g., 352, 354, and / or 356) of the decoder component 312 may be constructed (e.g., configured) to facilitate decoding the encoded information received from the encoder component 310 and generating predictions (e.g., predicting the next token in a token sequence, the next token representing a word or sub-word), determinations, and / or probabilities about individual words contained in the encoded information (e.g., corresponding to audio content and associated spectrogram information), which predictions, determinations, and / or probabilities may be output as output data (e.g., a predicted token sequence, which may be a text transcription of words recognized by the speech recognition model 104 regarding the input information (e.g., spectrogram information and / or tokens representing spoken words) input to the speech recognition model 104).

[0112] As described above, the decoder component 312 can also input the previous predicted word-gram output from each MLP component (e.g., 352, 354 and / or 356) back to the decoder component 312 (e.g., the self-attention component (e.g., 340)) for further analysis and processing in relation to other portions of the encoded information, such as described herein. In some embodiments, the decoder component 312 can employ the learned positional encoding as part of the decoder component 312 input and processing of the word-gram. In some embodiments, the word-gram can be in a multi-task model training format, such as described herein.

[0113] Further with respect to the FAL evaluator component 112, as part of each iteration of training of the speech recognition model 104, the FAL evaluator component 112 (e.g., the fidelity determination component 218, the accuracy determination component 220, the latency determination component 222, and / or other components of the FAL evaluator component 112) can analyze (e.g., evaluate or compare) output data (e.g., predicted tokens or transcriptions including transcribed text data) generated and output by the speech recognition model 104 and a validation dataset or test dataset (e.g., text transcription data of a mixed-language dataset or a portion thereof, or information related to or associated therewith). Based at least in part on the analysis results, the fidelity determination component 218 can determine a fidelity (e.g., a fidelity score, rating, or value representing the fidelity) to evaluate the ability of the speech recognition model 104 to capture the content and meaning of the speech (e.g., mixed-language speech) in the transcription generated thereby. Evaluations regarding fidelity (e.g., evaluations performed by fidelity determination component 218) can involve assessing the accuracy of the transcription (e.g., the accuracy of the words and / or characters in the transcription as compared to the words spoken in the audio content) and ensuring that the transcribed text data in the transcription retains the intended message, tone, and context of the spoken words in the audio content.

[0114] Additionally, as part of each iteration of model training, based at least in part on the analysis results, the accuracy determination component 220 of the FAL evaluator component 112 can determine the accuracy of the transcription generated by the speech recognition model 104 and / or measure the accuracy or correctness (e.g., an accuracy score, rating, or value representing accuracy) of the transcription generated by the speech recognition model 104. The accuracy factor can relate to the accuracy determination component 220 determining and evaluating the ability of the speech recognition model 104 to correctly recognize spoken words (e.g., spoken mixed-language words) of the audio content and convert them into text data (e.g., written text), including accurately representing the tone and pronunciation of individual words in the individual languages ​​contained in the audio content.

[0115] As part of each iteration of model training, the delay determination component 222 can determine, measure, or track the time it takes for the speech recognition model 104 to process the audio content (e.g., a spectrogram representing the audio content including mixed-language speech) and generate a transcription representing the spoken words (e.g., spoken mixed-language words) contained in the audio content. In some embodiments, the delay determination component 222 can determine (e.g., calculate) a delay score, rating, or value (e.g., a delay metric value), which can represent the amount of delay.

[0116] In some implementations, the FAL evaluator component 112 can determine a FAL rating (e.g., a FAL score) (e.g., for iterations) based at least in part on fidelity (e.g., a fidelity rating or score), accuracy (e.g., an accuracy rating or score), and latency (e.g., a latency rating or score) (e.g., as a function) according to defined model management criteria, which rating can represent and relate to fidelity, accuracy, and latency associated with speech recognition performance and transcription generation, the transcription representing spoken words presented in audio content associated with (e.g., corresponding to or representing) a mixed-language dataset. In some implementations, the FAL evaluator component 112 can determine (e.g., calculate) a FAL rating (e.g., a FAL score) based at least in part on (e.g., in accordance with or as a function of) the following non-limiting example equation, as follows:

[0117]

[0118] Wherein F may be or may represent fidelity, L may be or may represent delay, M may be or may represent maximum delay, S may be or may represent the number of replacements of second language (e.g., English or other second language) words (or characters) and first language (e.g., Chinese or other first language) characters (or words), I may be or may represent the number of insertions of second language words (or characters) and first language characters (or words), D may be or may represent the number of deletions of second language words (or characters) and first language characters (or words), N may be or may represent the number of deletions of second language words (or characters) and first language characters (or words) in the reference, and α, β, and γ may be or may represent respective weights (e.g., respective weight values), which may be assigned to each part of the equation based at least in part on respective scenarios (e.g., respective speech recognition or translation scenarios) or applications. Accuracy can be based at least in part on (e.g., can be a function of) the variables S, I, D, and / or N (e.g., accuracy can be equal to (1-(S+I+D) / N) or a function of (1-(S+I+D) / N)). The disclosed techniques and equations for determining, estimating, and / or calculating accuracy regarding speech recognition performance and text transcription (e.g., speech recognition transcription) generation can be more desirable (e.g., more accurate, more appropriate, more enhanced, or optimal) than existing accuracy determination techniques that only consider character errors in the transcription, and, in general, the disclosed FAL evaluation, FAL determination, and FAL equation can be more desirable (e.g., more accurate, more appropriate, more enhanced, or optimal) than existing techniques for determining performance, quality, and accuracy of speech recognition and text transcription generation (e.g., WER, CER, and other existing techniques). Regarding weight values, for example, in real-time speech recognition or translation scenarios, the demand for fast response (e.g., fast speech recognition response) may be relatively high, and the model manager component 102 (or another component or user) may assign a relatively higher weight value to γ. As another example, in certain digital human application scenarios, the overall meaning of the conversation may be more desirable (e.g., more desired or more important) than word-by-word consistency, and the model manager component 102 (or other components or users) may assign a relatively lower weight value to β.

[0119] It should be appreciated and understood that this non-limiting example equation for determining a FAL rating is only one non-limiting example equation for determining a FAL rating generated by the speech recognition component 106 employing the speech recognition model 104 for speech recognition performance and transcriptions associated with a mixed-language dataset. In other embodiments, the FAL evaluator component 112 can employ another desired equation (e.g., a FAL rating determination equation) to determine the FAL rating generated by the speech recognition component 106 employing the speech recognition model 104 for speech recognition performance and transcriptions associated with a mixed-language dataset, where such other desired equation can be based at least in part on fidelity (e.g., a fidelity rating or score), accuracy (e.g., an accuracy rating or score), and latency (e.g., a latency rating or score).

[0120] In some embodiments, additionally or alternatively, as part of each iteration of model training, the FAL evaluator component 112 may also determine (e.g., calculate or measure) a WER, CER, and / or other error associated with the speech recognition performance of the speech recognition model 104 based at least in part on analyzing transcriptions (including transcribed text data) and / or results of predicting word units associated with (e.g., compared with) a validation dataset or a test dataset.

[0121] During each iteration of model training, the update component 122 can determine updates to parameters, processes, operations, functions, training procedures (e.g., model training procedures), topics (e.g., seed topics), content collection, prompts (e.g., dialogue prompts), simulated dialogue generation, mixed language dataset generation, and / or other features of a trained model (e.g., of or associated with the model manager component 102) and / or the MLDGM component 110, and / or updates to parameters, hyperparameters, processes, operations, functions, and / or other features of the speech recognition model 104 and / or the speech recognition component 106, based at least in part on FAL evaluation results (e.g., fidelity, accuracy, and / or latency results, ratings, or scores; or an overall FAL rating or score), WER, CER, and / or other error determinations associated with the speech recognition performance of the speech recognition model 104 during the iteration, according to defined model management criteria. In some implementations, the update component 122 can coordinate with the evaluator component 204 regarding updates to content collected by the content collector component 202 and / or associated web crawlers or tools, and / or coordinate with the assessment component 210 regarding updates to prompts or dialog simulations to facilitate generation of a mixed language dataset. The update component 122 can communicate corresponding update information to each of the MLDGM component 110, the trained model, the model manager component 102, the FAL evaluator component 112, the speech recognition model 104, the speech recognition component 106, the tuner component 114, the audio converter component 118, and / or other components (e.g., other components of the system 100) to update (e.g., modify, adjust, or reconfigure) these corresponding components based at least in part on the corresponding update information according to defined model management criteria.

[0122] In some embodiments, in addition to determining the fidelity, accuracy, latency, WER, CER, and / or other errors associated with the speech recognition performance of the speech recognition model 104 during an iteration, a loss function associated with the speech recognition performance may also be determined. For example, the speech recognition component 106 may include a loss determination component 302 that may determine (e.g., calculate) a loss function associated with word-gram prediction during each iteration of training of the speech recognition model 104. The loss function may be a measurement or determination of the difference (e.g., amount of error) between the prediction of a word-gram by the speech recognition model 104 when performing speech recognition on input data input to the speech recognition model 104 and the word-gram of the input data or the actual word-gram representing the input data. For example, the loss determination component 302 can compare a first predicted word-gram with a first actual word-gram (the first actual word-gram corresponds to the first predicted word-gram in a sequence of predicted word-grams and a sequence of actual word-grams (e.g., for one or more sequences of predicted word-grams and one or more corresponding sequences of actual word-grams)), compare a second predicted word-gram with a second actual word-gram (the second actual word-gram corresponds to the second predicted word-gram in a sequence of predicted word-grams and a sequence of actual word-grams), and / or compare another predicted word-gram with another actual word-gram (the other actual word-gram corresponds to the other predicted word-gram in a sequence of predicted word-grams and a sequence of actual word-grams). Based at least in part on the comparison results, the loss determination component 302 can determine whether the first predicted word-gram satisfies defined matching criteria with respect to the first actual word-gram (e.g., it can be determined whether the first predicted word-gram matches or is the same as the first actual word-gram, or otherwise satisfies defined matching criteria), determine whether the second predicted word-gram satisfies defined matching criteria with respect to the second actual word-gram, and / or determine whether another predicted word-gram satisfies defined matching criteria with respect to another actual word-gram. The loss determination component 302 can determine a loss function based at least in part on (e.g., as a function of) the comparison of each predicted word-gram to each actual word-gram. The higher the loss function, the greater the difference and / or amount of error between the predicted word-gram and the actual word-gram. In some embodiments, the loss determination component 302 can pass loss function information indicating or specifying the loss function associated with the sequence of predicted word-grams to the update component 304.

[0123] In some embodiments, the update component 304 can determine updates (e.g., modifications, adjustments, reconfigurations, or changes) to parameters, hyperparameters, processes, and / or procedures of the speech recognition model 104 or the speech recognition component 106 based at least in part on a loss function associated with a predicted word-gram sequence (e.g., based at least in part on an analysis of loss function information associated with the sequence), the predicted word-gram sequence being associated with the current iteration of training of the speech recognition model 104, according to defined model management criteria. For example, the update component 304 of the speech recognition component 106 can back-propagate or facilitate back-propagation of a loss function (e.g., a loss associated with the loss function) through the speech recognition model 104. In certain embodiments, to facilitate such back-propagation of the loss function, the update component 304 can employ or perform (e.g., execute) one or more gradient-based enhancement (e.g., optimization) algorithms, techniques, or processes, such as stochastic gradient descent or another desired gradient-based enhancement algorithm, technique, or process, to determine updates to parameters, hyperparameters, functions, and / or procedures of the speech recognition model 104. Updating the parameters, hyperparameters, functions and / or processes of the speech recognition model 104 can facilitate enhancing the accuracy and / or efficiency of the speech recognition model 104 in performing speech recognition on input data (e.g., input data representing spoken words (such as mixed-language spoken words) in audio content) to generate transcribed text data, which represents the spoken words contained or represented in the input data.

[0124] In some implementations, the update component 304 can communicate update information to the tuner component 114, which can be related to updates to or about parameters, hyperparameters, functions, and / or processes of the speech recognition model 104. The tuner component 114 can update (e.g., modify, adjust, reconfigure, or alter) the parameters, hyperparameters, functions, and / or processes of the speech recognition model 104 based at least in part on the update information.

[0125] In other embodiments, additionally or alternatively, the update component 304 can communicate update information related to the loss function and / or updates to parameters, hyperparameters, functions, and / or processes of the speech recognition model 104 (e.g., can be a preliminary determination of updates) to the FAL evaluator component 112 or the update component 122 of the model manager component 102. The FAL evaluator component 112 or the update component 122 of the model manager component 102 can determine updates to parameters, hyperparameters, processes, operations, functions, and / or other features of the speech recognition model 104 (and / or updates to parameters, processes, operations, functions, training procedures, topics, content collection, prompts, simulated dialog generation, mixed language dataset generation, and / or other features of the trained model, the MLDGM component 110, and / or the model manager component 102) based at least in part on the FAL evaluation results, loss function, WER, CER, and / or other error results, according to defined model management criteria, such as described herein. For example, the FAL evaluator component 112 or the update component 122 can reconcile the update information related to the loss function with other evaluation results (e.g., FAL evaluation results (e.g., FAL ratings or scores), WERs, and / or CERs) according to defined model management criteria, or can combine the update information related to the loss function with other evaluation results as part of determining updates to various components. In some embodiments, the FAL evaluator component 112 or the update component 122 can also reconcile or combine any updates determined by the MLDGM component 110 with respect to the determination and generation of enhanced mixed-language datasets (e.g., updates related to collecting content items, conversation prompts, simulated conversations between speakers, and / or other functions, operations, or features) with or into updates determined by the FAL evaluator component 112 or the update component 122, such as described herein.

[0126] Updates to parameters, processes, operations, functions, training procedures (e.g., model training procedures), topics (e.g., seed topics), content collection, prompts (e.g., dialogue prompts), simulated dialogue generation, mixed-language dataset generation, and / or other features of a trained model (e.g., of or associated with the model manager component 102) and / or the MLDGM component 110 can enhance the generation of a next (or subsequent) mixed-language dataset by the MLDGM component 110, which can be used for the next (or subsequent) iteration of training (e.g., further training, refinement training, or fine-tuning training) of the speech recognition model 104, which can enhance the training and performance of the speech recognition model 104 and can result in or generate an enhanced (e.g., improved or optimized) trained speech recognition model 104. Updates to parameters, hyperparameters, processes, operations, functions, and / or other features of the speech recognition model 104 and / or the speech recognition component 106 can enhance the training and performance of the speech recognition model 104 and can result in or generate an enhanced trained speech recognition model 104.

[0127] To facilitate improving the performance and training of the speech recognition model 104, the model manager component 102, the speech recognition component 106, and other components of the system 100 may perform their respective functions and operations for a desired number of iterations of speech recognition model 104 training, e.g., until the performance of the (trained) speech recognition model 104 is determined (e.g., determined by the model manager component 102) to meet (e.g., meet or exceed, or be at or greater than) a defined speech recognition model performance criterion (e.g., with respect to fidelity, accuracy, latency, WER, CER, loss function, and / or other desired performance metric). ), or until it is determined (e.g., determined by the model manager component 102) that a defined model training termination (e.g., stopping) criterion (e.g., a defined threshold model training termination criterion) is met (e.g., until the number of training iterations of the speech recognition model 104 meets (e.g., is at or greater than) a defined model training iteration number threshold), or until the training and / or performance improvement of the speech recognition model 104 between each (e.g., continuous) iteration is less than a defined training and / or performance improvement threshold amount), according to the defined model management criteria, at least for the initial training phase of the speech recognition model 104. As needed, after training the trained speech recognition model 104, the model manager component 102, the speech recognition component 106 and other components of the system 100 may also perform updates and / or improvements (e.g., regular or dynamic updates and / or improvements) on the trained speech recognition model 104 to further improve the performance of the trained speech recognition model 104.

[0128]

[0026] Further with respect to the model manager component 102 and the speech recognition component 106, according to various embodiments, the model manager component 102 and / or the speech recognition component 106, respectively, may be, may include, or may employ an artificial intelligence component. The artificial intelligence component can perform artificial intelligence-based analysis on the data, such as information about: generation of a mixed-language dataset, training of the speech recognition model 104, performance of the speech recognition model 104, performing FAL evaluations and / or other evaluations (e.g., WER, CER, or loss functions), updating (e.g., adapting, modifying, or reconfiguring) parameters, processes, operations, functions, training procedures, topics, content collection, prompts, simulated dialogue generation, mixed-language dataset generation, and / or other features of a trained model of the model manager component 102 or associated with the model manager component 102, updating parameters, hyperparameters, processes, operations, functions, and / or other features of the speech recognition model 104 and / or the speech recognition component 106, feedback and / or backpropagation related to the performance and / or training of the speech recognition model 104, and / or other aspects or features associated with the model manager component 102, the speech recognition component 106, and / or the speech recognition model 104. In some embodiments, with respect to a model (e.g., a trained model of the model manager component 102 or the speech recognition model 104), depending on the type of model, the artificial intelligence component can, based at least in part on an analysis of input information, input the information into the (trained) model for analysis by the model to update the model or generate output data or results (e.g., a mixed language dataset, a transcription including transcribed text data recognized by the speech recognition model, predicted word units, model update information that can be used to update another model (e.g., the speech recognition model 104) , artificial intelligence related data, and / or other data or results).

[0129] In connection with or as part of such AI-based analysis, the AI ​​component may employ, construct (e.g., construct or create) and / or import AI-based techniques and algorithms, AI models (e.g., untrained or trained models), neural networks (e.g., untrained or trained neural networks), decision trees, Markov chains (e.g., untrained or trained Markov chains), and / or graph mining to provide and / or generate predictions, inferences, calculations, predictions, estimates, derivations, forecasts, detections, and / or calculations to facilitate determining or learning data patterns in data, determining or learning correlations between one (or more) data items and another (or more) data items (e.g., another (or more) data items or the occurrence of an event related thereto), determining or learning a correlation, relationship, or causal relationship between one event and another event (e.g., the occurrence of another event), determining or learning relationships between components of or associated with the system 100 (e.g., encoder components, decoder components, activator components, encoder blocks, decoder blocks, or other components or functions), determining or learning enhancements to mixed-language datasets and / or mixed-language dataset generation, determining or learning updates to the speech recognition model 104, determining or learning enhancements to fidelity, accuracy, and / or latency associated with the speech recognition model 104 performing speech recognition, performing other desired functions or operations, and / or the subject matter automatically performs one or more functions or features of the disclosed subject matter as described in detail herein.

[0130] The artificial intelligence component may employ various artificial intelligence-based schemes to perform various embodiments / examples disclosed herein. In order to provide or assist in making many determinations described herein related to the disclosed subject matter (e.g., determining, ascertaining, inferring, calculating, predicting, predicting, estimating, deducing, prospecting, detecting, calculating), the artificial intelligence component may examine all or a subset of data to which it is authorized to access (e.g., model training data; information related to mixed language data sets; information related to FAL evaluation and / or other evaluations (e.g., WER, CER or loss function); feedback information; back propagation information; and / or other information, such as described herein), and may provide reasoning or determine the state of the system and / or environment based on a set of observations captured through events and / or data. For example, determination may be used to identify a specific context or action, or a probability distribution of states may be generated. Determination may be probabilistic, that is, a probability distribution of relevant states is calculated based on a consideration of data and events. Determination may also refer to a technique used to combine higher-level events from a set of events and / or data.

[0131] In some embodiments, with respect to probability, the artificial intelligence component, the trained model, or the speech recognition model 104 may employ one or more threshold probabilities (e.g., threshold probability values) to facilitate making determinations. As a non-limiting example, when making a determination related to generating a mixed language data set for training the speech recognition model 104, as part of an artificial intelligence-based information analysis, the artificial intelligence component and / or the trained model may determine a probability (e.g., a performance enhancement probability related to the speech recognition performance of the speech recognition model 104), and may determine whether the probability (e.g., probability value) meets (e.g., reaches or exceeds; or is at or greater than) a defined and applicable threshold probability (e.g., a threshold minimum probability value related to speech recognition performance enhancement). The artificial intelligence component and / or the trained model may make a determination (or prediction or inference) related to generating a mixed language data set based at least in part on the results of analyzing (e.g., comparing) the probability to the defined and applicable threshold probability (e.g., the highest probability value (or a set of higher probability values) that meets the threshold minimum probability value may be determined). In some embodiments, the artificial intelligence component, trained model, or trained speech recognition model 104 can make a determination (or prediction or inference) without utilizing any threshold probability. For example, the artificial intelligence component, trained model, or trained speech recognition model 104 can perform such a determination (or prediction or inference) based at least in part on a probability determined to be the highest probability compared to other probabilities, which probability is associated or related to generating a mixed language data set, a performance evaluation (e.g., a FAL evaluation, a WER, a CER, or a loss function), a parameter, a hyperparameter, a function, an operation, a process, a training procedure, or other aspects or features under consideration, without utilizing or considering any threshold probability.

[0132] These determinations can all result in constructing new events or actions from a set of observed events and / or stored event data, regardless of whether the events are closely related in time and whether the events and data come from one or more event and data sources. In certain embodiments, the components disclosed herein (e.g., an artificial intelligence component, a trained model, a speech recognition model 104, and / or another component) can employ various classification (explicit training (e.g., via training data) and implicit training (e.g., via observed behavior, preferences, historical information, receiving external information, etc.)) schemes and / or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) to perform automatic and / or determined operations related to the claimed subject matter. Therefore, the classification schemes and / or systems can be used to automatically learn and perform a number of functions, actions, and / or determinations.

[0133] In some embodiments, the artificial intelligence component can use a classifier that can perform artificial intelligence-based analysis on the data. The classifier can map the input attribute vector z = (z1, z2, z3, z4, ..., zn) to the confidence that the input belongs to a certain category, such as f (z) = confidence (class). This classification can use probability and / or statistical-based analysis (for example, considering utility and cost factors in the analysis) to determine the actions to be automatically performed. Support vector machine (SVM) can be used as an example of a classifier that can be used. The working principle of SVM is to find a hypersurface in the space of possible inputs, where the hypersurface attempts to separate the triggering criteria from the non-triggering events. Intuitively, this makes the classification correct when the test data is close to the training data but not exactly the same. Other directed and undirected model classification methods include, for example: naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models and / or probabilistic classification models that provide different independence patterns, any of which can be adopted. The classification used in this article also includes statistical regression for developing priority models.

[0134] Further with respect to the processor component 224 and the data repository 226 of or associated with the model manager component 102, the processor component 224 can be associated with (e.g., communicatively connected to) the model manager component 102 and / or other components of the system 100, and can work in conjunction with the model manager component 102 and / or other components of the system 100, including the speech recognition component 106, the MLDGM component 110, the FAL evaluator component 112, the tuner component 114, the audio recorder component 116, the audio converter component 118, the word segmenter component 120, the update component 122, the data repository 226 and / or other components of the model manager component 102 and / or the system 100, to facilitate performing various functions and operations of the model manager component 102 and / or the system 100. The processor component 224 can employ one or more processors (e.g., one or more central processing units (CPUs), accelerators, graphics processing units (GPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), microprocessors, controllers, and / or microcontrollers that can process information related to: data, instructions, files, transcriptions, services, applications, content collections, language datasets (e.g., mixed language datasets), words, subwords, spoken words, word-grams, audio content, multimedia content, spectrograms, AI / ML-based models, AI-related data, topics, loss functions, FAL evaluations, WERs, CERs, training data, feedback information, updates, predictions, inferences, thresholds (e.g., maximum, minimum, or other thresholds), weight values, data processing operations, messages, notifications, alerts, warnings, preferences (e.g., user or client preferences), hash values, metadata, hyperparameters, parameters, tables, mappings, policies, defined model management standards, algorithms (e.g., enhanced mixed-language dataset generation management algorithm, enhanced FAL evaluation algorithm, enhanced model training algorithm, artificial intelligence algorithm, hashing algorithm, data compression algorithm, data decompression algorithm and / or other algorithms), interfaces, protocols, tools and / or other information to facilitate the operation of the model manager component 102 and / or the system 100 and control the flow of data between the model manager component 102 and / or other components associated with the model manager component 102 and / or the system 100 (e.g., speech recognition component 106, tuner component 114, audio recorder component 116, audio converter component 118, word segmenter component 120, network devices or components, communication networks, devices (e.g., 108), nodes, applications, services, users, or other entities).

[0135] The data repository 226 may store data structures (e.g., user data, metadata), code structures (e.g., modules, objects, hash values, classes, programs), or instructions, and information related to: data, instructions, files, transcriptions, services, applications, content collections, language datasets (e.g., mixed language datasets), words, subwords, spoken words, word-grams, audio content, multimedia content, spectrograms, AI / ML based models, AI related data, topics, loss functions, FAL evaluations, WER, CER, training data, feedback information, updates, predictions, inferences, thresholds (e.g., the most recent The data repository 226 may include information such as a maximum value, minimum value, or other threshold value, weight values, data processing operations, messages, notifications, alerts, warnings, preferences (e.g., user or client preferences), hash values, metadata, hyperparameters, parameters, tables, mappings, policies, defined model management standards, algorithms (e.g., enhanced mixed language dataset generation management algorithms, enhanced FAL evaluation algorithms, enhanced model training algorithms, artificial intelligence algorithms, hashing algorithms, data compression algorithms, data decompression algorithms, and / or other algorithms), interfaces, protocols, tools, and / or other information to facilitate control or execution of operations associated with the model manager component 102 and / or the system 100. The data repository 226 may include volatile and / or non-volatile memory, such as described herein. In one aspect, the processor component 224 can be functionally coupled (e.g., via a memory bus) to the data repository 226 to store and retrieve information needed to at least partially operate and / or impart functionality to the model manager component 102, the speech recognition component 106, the MLDGM component 110, the FAL evaluator component 112, the tuner component 114, the audio recorder component 116, the audio converter component 118, the word segmenter component 120, the update component 122, the processor component 224, the data repository 226 and / or other components of the model manager component 102 and / or the system 100, and / or substantially any other operational aspect of the model manager component 102 and / or the system 100.

[0136] The data repository 226 may include volatile memory and / or non-volatile memory. By way of example and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, non-volatile memory express (NVMe), fiber architecture NVMe (NVMe-oF), persistent memory (PMEM), or PMEM-oF. Volatile memory may include random access memory (RAM) that may be used as an external cache memory. By way of example and not limitation, RAM may be in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memories of various aspects of the present disclosure are intended to include, but are not limited to, these and other suitable types of memories.

[0137] Further with respect to, or related to, the processor component 306 and the data repository 308 of the speech recognition component 106, the processor component 306 can be associated with (e.g., communicatively connected to) the speech recognition component 106 and / or other components of the system 100, and can work in conjunction with the speech recognition component 106 and / or other components of the system 100, including the tuner component 114, the audio converter component 118, the word segmenter component 120, the loss determination component 302, the update component 304, the audio recorder component 116, the data repository 308, and / or other components of the speech recognition component 106 and / or the system 100, to facilitate performing various functions and operations of the speech recognition component 106 and / or the system 100. The processor component 224 can employ one or more processors (e.g., one or more CPUs, accelerators, GPUs, ASICs, FPGAs, microprocessors, controllers, and / or microcontrollers) that can process information related to: data, instructions, files, transcriptions, services, applications, language datasets (e.g., mixed language datasets), words, subwords, spoken words, word-grams, audio content, multimedia content, spectrograms, AI / machine learning based models, AI related data, topics, loss functions, training data, feedback and / or back-propagation information, updates, predictions, inferences, thresholds (e.g., maximum, minimum, or other thresholds), weight values, data processing operations, messages, notifications, alarms, alerts, preferences (e.g., , user or client preferences), hash values, metadata, hyperparameters, parameters, tables, mappings, policies, defined model management standards, algorithms (e.g., enhanced speech recognition performance and management algorithms, enhanced model training algorithms, artificial intelligence algorithms, hashing algorithms, data compression algorithms, data decompression algorithms, and / or other algorithms), interfaces, protocols, tools, and / or other information to facilitate the operation of the speech recognition component 106 and / or the system 100 and control the flow of data between the speech recognition component 106 and / or other components associated with the speech recognition component 106 and / or the system 100 (e.g., the model manager component 102, the audio recorder component 116, network devices or components, communication networks, devices, nodes, applications, services, users, or other entities).

[0138] The data repository 308 can store data structures (e.g., user data, metadata), code structures (e.g., modules, objects, hash values, classes, programs), or instructions, and information related to the following: data, instructions, files, transcriptions, services, applications, language datasets (e.g., mixed language datasets), words, sub-words, spoken words, word-grams, audio content, multimedia content, spectrograms, artificial intelligence / machine learning based models, artificial intelligence related data, topics, loss functions, training data, feedback and / or back-propagation information, updates, predictions, inferences, thresholds (e.g., maximum, minimum or other thresholds), weight values, data processing operations, messages, notifications, alarms, alerts, preferences (e.g., user or client preferences), hash values, metadata, hyperparameters, parameters, tables, mappings, policies, defined model management standards, algorithms (e.g., enhanced speech recognition performance and management algorithms, enhanced model training algorithms, artificial intelligence algorithms, hashing algorithms, data compression algorithms, data decompression algorithms and / or other algorithms), interfaces, protocols, tools and / or other information to facilitate control or execution of operations associated with the speech recognition component 106 and / or the system 100. The data repository 308 may include volatile and / or non-volatile memory, such as described herein. In one aspect, the processor component 306 may be functionally coupled (e.g., via a memory bus) to the data repository 308 to store and retrieve information required to at least partially operate and / or impart functionality to the speech recognition component 106, the speech recognition model 104, the fine tuner component 114, the audio converter component 118, the word segmenter component 120, the loss determination component 302, the update component 304, the processor component 306, the data repository 308 and / or the speech recognition component 106 and / or other components of the system 100, and / or substantially any other operational aspects of the speech recognition component 106 and / or the system 100.

[0139] Convert to reference Figure 4 and Figure 5 (as well as Figure 1-Figure 3 ). Figure 4 presents a diagram of non-limiting example speech recognition results 400 relating to speech recognition performed on an audio recording including spoken Cantonese words, performed using a trained speech recognition model (e.g., the Whisper-MCE model) described in the non-limiting examples herein (the recognition model being trained based on an enhanced mixed-language dataset and an enhanced FAL evaluation technique) and certain other trained speech recognition models trained using existing language datasets and existing training techniques, and Figure 5A diagram of a non-limiting example speech recognition result 500 is presented, the recognition result involving speech recognition performed on an audio recording including spoken mixed Cantonese and English words, the speech recognition being performed by a trained speech recognition model (e.g., Whisper-MCE model) described in the non-limiting example and certain other trained speech recognition models in accordance with various aspects and embodiments of the disclosed subject matter. The trained speech recognition model in the non-limiting example may be a Whisper-small model trained and fine-tuned using an enhanced mixed-language dataset and enhanced FAL evaluation techniques described herein, wherein, in this non-limiting example, the enhanced mixed-language dataset may include Cantonese and English language words, and the trained speech recognition model of this example may be referred to as Whisper-MCE. Other trained speech recognition models include a trained Whisper-large speech recognition model (Whisper-large model (e.g., Whisper-large-v2 model)) and a trained Whisper-small speech recognition model (Whisper-small model), which are trained using existing language datasets and training techniques, wherein the existing language datasets include Cantonese and English language words.

[0140] For real-world applications, latency and accuracy may be important factors in determining or dictating the experience (e.g., user experience) of speech recognition performed by a speech recognition application, service, or model. To ideally balance latency and accuracy for speech recognition applications, a Whisper-small model is selected as a starting model for training and fine-tuning (e.g., utilizing an enhanced mixed-language dataset, enhanced FAL evaluation, and updates) to generate a trained Whisper-MCE model. The trained Whisper-MCE model, by being based in part on the Whisper-small model as a starting point, may have an ideally smaller model size and lower computational and resource usage (e.g., lower computational and resource requirements) compared to a Whisper-large or other larger model, while still maintaining and / or achieving an ideal (e.g., appropriate, acceptable, useful, beneficial, or reasonable) accuracy of speech recognition performance.

[0141] The example speech recognition results 400 include an example speech recognition result 402 for performing speech recognition on audio recording related information including 18 seconds of Cantonese spoken words by a trained Whisper-large model; an example speech recognition result 404 for performing speech recognition on audio recording related information including 18 seconds of Cantonese spoken words by a trained Whisper-small model; and an example speech recognition result 406 for performing speech recognition on audio recording related information including 18 seconds of Cantonese spoken words by a trained Whisper-MCE model. Each speech recognition result 402, 404, and 406 can present a respective recognition result 408 and a respective delay 410, associated with a respective performance of performing speech recognition on the audio recording related information including the Cantonese spoken words.

[0142] The example speech recognition results 500 include an example speech recognition result 502 for performing speech recognition on audio recording related information including 15 seconds of Cantonese and English mixed language spoken words by a trained Whisper-large model; an example speech recognition result 504 for performing speech recognition on audio recording related information including 15 seconds of Cantonese and English mixed language spoken words by a trained Whisper-small model; and an example speech recognition result 506 for performing speech recognition on audio recording related information including 15 seconds of Cantonese and English mixed language spoken words by a trained Whisper-MCE model. Each speech recognition result 502, 504, and 506 can present a respective recognition result 508 and a respective delay 510, associated with a respective performance of performing speech recognition on audio recording related information including Cantonese and English mixed language spoken words.

[0143] from Figure 4 The various example speech recognition results 402, 404 and 406 presented in Figure 5It can be observed from the various example speech recognition results 502, 504 and 506 presented in that the trained Whisper-MCE model can provide speech recognition results that are much more accurate than the trained Whisper-small model, and can at least be comparable to the accuracy of the trained Whisper-large model, particularly with respect to Cantonese and English mixed-language spoken words, and the trained Whisper-MCE model can have (e.g., significantly) lower latency (e.g., less speech recognition processing and / or reaction time) than the trained Whisper-large model (e.g., with respect to performing speech recognition on information related to an audio recording including Cantonese and English mixed-language spoken words, the latency of the trained Whisper-MCE model can be ideally as low as 2.6s, while the latency of the trained Whisper-large model can be significantly higher than 6.4s), and can be comparable to the low latency (e.g., 2.5s) of the trained Whisper-small model. The example speech recognition result 400 and the example speech recognition result 500 (particularly the example speech recognition result 500) show that when compared with other types of models (including the trained Whisper-large model), the trained Whisper-MCE model can outperform existing baseline models in various aspects and metrics, such as accurate content capture (e.g., the trained Whisper-MCE model can have more accurate content capture), recognition accuracy (e.g., the trained Whisper-MCE model can have higher recognition accuracy), and speech recognition speed (e.g., the trained Whisper-MCE model can have faster speech recognition speed). That is, the trained Whisper-MCE model, when compared with the trained Whisper-large model (e.g., the baseline Whisper-large-v2 model), has a stronger ability in accurately capturing the original audio content of mixed-language speech, achieving higher recognition accuracy, and exhibiting faster recognition speed, thereby outperforming other existing models in the task of recognizing mixed-language speech.Therefore, it can be observed from the example speech recognition results 400 and the example speech recognition results 500 that the trained Whisper-MCE model trained and fine-tuned using the enhanced mixed-language dataset and enhanced FAL evaluation technique described herein may be a more ideal (e.g., more useful, more acceptable, more enhanced, or optimal) model for speech recognition (e.g., speech recognition applications) than the existing trained Whisper models (such as the existing trained Whisper-large model and the existing trained Whisper-small model) trained using the existing language dataset and the existing training technique, particularly in performing speech recognition on spoken mixed-language words.

[0144] It should be recognized and understood that one or more components (e.g., model manager component 102, speech recognition component 106, device 108, tuner component 114, audio recorder component 116, audio converter component 118, word segmenter component 120, artificial intelligence component, or other component) of the systems (e.g., system 100 or other systems) or methods described herein may include or be associated with various other types of components, such as a display screen (e.g., a touch screen display or a non-touch screen display), audio functionality (e.g., an amplifier, speaker, or audio interface), or other interfaces to facilitate presenting information to a user, entity, or other component (e.g., other device or other server), and / or perform other desired functions or operations.

[0145] The above-described systems and / or devices have been described with respect to the interaction between several components. It should be understood that these systems and components may include a specified component or subcomponent, a portion of a specified component or subcomponent, and / or another component. A subcomponent may also be implemented as a component that is communicatively coupled to other components, rather than being contained in a parent component. In addition, one or more components and / or subcomponents may also be combined into a component to provide comprehensive functionality. For the sake of brevity, a component may also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0146] In view of the example systems and / or devices described herein, reference Figure 6-Figure 11, can further understand the example methods that can be implemented according to the disclosed subject matter. For ease of explanation, the example methods disclosed herein are presented and described in the form of a series of actions; however, it should be understood that the disclosed subject matter is not limited by the order of actions, because some actions can occur in different orders and / or concurrently with other actions shown and described in this article. For example, the methods disclosed herein can also be optionally represented as a series of interrelated states or events, such as in a state diagram. In addition, when different entities perform different parts of the method, an interaction diagram can represent the method according to the disclosed subject matter. In addition, not all illustrated actions need to implement the method according to the subject specification. It should be further understood that the methods disclosed throughout the subject specification can be stored on a manufactured article to facilitate the transportation and transmission of this method to a computer, which is executed by a processor or stored in a memory.

[0147] Figure 6 A flow chart of an example method 600 is shown, which can desirably (e.g., automatically, dynamically, appropriately, efficiently, reliably, enhanced, and / or optimally) train a speech recognition model (e.g., an AI-based speech-to-text model) that can desirably (e.g., accurately, efficiently, reliably, enhanced, and / or optimally) recognize mixed-language words in speech and convert those words into text data, in accordance with various aspects and embodiments of the disclosed subject matter. For example, a system including a model manager component can employ method 600, which can include or be associated with a processor component, a data repository, and / or other components. The model manager component can include a trained model (e.g., a trained GPT-4 model or other type of trained AI-based model), which can include or be associated with an MLDGM component and a FAL evaluator component.

[0148] At 602, mixed language data can be generated by a trained model, the generation being based at least in part on instruction data related to generation of the mixed language data and seed topic data related to a set of seed topics, wherein the mixed language data can include respective first text words in a first language and respective second text words in a second language, the respective first text words and respective second text words can be determined at least in part based on respective simulated conversations between respective simulated speakers, wherein respective second text words can be interspersed with respective first text words. The model manager component employs the trained model and the MLDGM component, and can determine and generate the mixed language data (e.g., an enhanced mixed language data set) based at least in part on the instruction data and the seed topic data. The mixed language data can include respective first text words in a first language and respective second text words in a second language, the words can be determined at least in part based on respective simulated conversations between respective simulated speakers (e.g., determined by the trained model and / or the MLDGM component), wherein respective second text words can be interspersed with respective first text words, such as described herein.

[0149] At 604, as part of the training of the speech recognition model, a quality metric may be determined by the trained model, wherein the quality metric may represent the speech recognition performance of the speech recognition model in recognizing information related to a spoken mixed-language word (the spoken mixed-language word may represent mixed-language data) and generating a textual transcription mixed-language word (the textual transcription mixed-language word may represent the spoken mixed-language word), wherein the quality metric may represent fidelity, accuracy, and latency related to the speech recognition performance. As part of the training of the speech recognition model, a model manager component using the trained model and a FAL evaluator component may determine a quality metric, wherein the quality metric may represent the speech recognition performance of the speech recognition model in recognizing information related to a spoken mixed-language word (the spoken mixed-language word may represent mixed-language data) and generating a textual transcription mixed-language word (the textual transcription mixed-language word may represent the spoken mixed-language word), wherein the quality metric may represent fidelity, accuracy, and latency related to the speech recognition performance, such as described herein.

[0150] Figure 7 and Figure 8A flow chart of an example method 700 is depicted, which can desirably (e.g., automatically, dynamically, appropriately, efficiently, reliably, enhanced, and / or optimally) determine and generate an enhanced mixed-language dataset that can be used to train a speech recognition model (e.g., an artificial intelligence-based speech-to-text model) to enable the speech recognition model to desirably (e.g., accurately, efficiently, reliably, enhanced, and / or optimally) recognize mixed-language words in speech and convert these words into text data, according to various aspects and embodiments of the disclosed subject matter. For example, a system including a model manager component can employ method 700, which can include or be associated with a processor component, a data repository, and / or other components. The model manager component can include a trained model (e.g., a trained GPT-4 model or other type of trained artificial intelligence-based model), which can include or be associated with an MLDGM component. The MLDGM component (e.g., an MLDGM component that employs a trained model) can include, for example, a content collector component, an evaluator component, an update component, an extractor component, a dialog simulator component, an assessment component, and / or other components, such as described herein. The model manager component can manage (eg, control) the performance of the various operations of method 700 .

[0151] At 702, various content items may be collected from various data sources based at least in part on instructions and / or seed topics, wherein the various content items may include corresponding languages. For example, a content collector component (e.g., an engineer agent) may utilize a web crawler and / or other tools to search and / or collect (e.g., retrieve) corresponding content items from various data sources (e.g., various networks or online websites, servers, and / or devices) based at least in part on instructions (e.g., instructions for data collection) and / or seed topics. The content collector component may perform content collection within a desired time period. The various content items may include various video content items, various audio content items, and / or various text content items (e.g., articles, news items, video content, audio content, multimedia content, and / or other types of content), which may use corresponding languages, including at least two languages ​​(e.g., Cantonese, English, and / or another language). According to various embodiments, method 700 may proceed to reference numeral 704 and / or reference numeral 710.

[0152] The performance and efficacy of the collection of the various content items can be evaluated at 704. The evaluator component (e.g., critic agent) can evaluate the performance and efficacy of the web crawler and / or other tools developed and / or used by the content collector component to collect the various content items to ensure ideal data diversity in the collected content items, check and identify errors in the collected content items, check and determine the efficiency of collecting the various content items, check and determine whether the norms and / or guidelines related to content collection (e.g., ethical, guiding or specified web scraping or data collection guidelines) are followed.

[0153] At 706, feedback information and / or content collection updates can be determined based at least in part on the evaluation results.The evaluator component can determine and generate feedback information and / or content collection updates based at least in part on the performance evaluation results and the efficacy of the respective content item collections.

[0154] At 708, the collection of content items (e.g., the next or subsequent collection) can be modified based at least in part on the feedback information and / or the content collection update. The evaluator component or another component (e.g., an update component of the mixed language data generation manager component) can modify (e.g., update) the collection of content items by the web crawler and / or other tools of the content collector component based at least in part on the feedback information and / or the content collection update to enhance (e.g., improve) the collection of content items by the content collector component.

[0155] At 710, desired topics, keywords, and / or other topics may be extracted from each content item based at least in part on the results of the analysis of each content item. An extractor component (e.g., a manager agent) may analyze each content item and, based at least in part on the results of the analysis, may identify and / or extract desired (e.g., relevant and / or desired) topics, keywords, and / or other topics in / from each content item. For example, the extractor component may employ a desired trained natural language processing model that may identify topics, themes, keywords, and / or other desired topics that may be prevalent (e.g., frequently or typically appearing) in each content item based at least in part on the results of the analysis of each content item (e.g., an artificial intelligence-based analysis). At this point, method 700 may proceed to reference point A, where method 700 may begin at reference point A, such as Figure 8 As depicted in and described herein.

[0156] At 712, one or more corresponding mixed language conversations related to one or more respective topics may be simulated based at least in part on the desired topics, keywords, and / or other topics. A conversation simulator component (e.g., a speaker agent) may simulate one or more corresponding conversations between respective speakers (e.g., simulated speakers) speaking in a mixed language (e.g., at least two languages, such as Cantonese and English) based at least in part on the desired topics, keywords, and / or other topics extracted from the respective content items, and the conversations are about one or more respective topics. The simulated conversations between the respective speakers may be simulated how expected speakers (e.g., native speakers speaking the mixed language) discuss (e.g., may or probably discuss) the identified topics, based at least in part on (e.g., using) different prompts (e.g., conversation prompts) for the desired (e.g., relevant and / or desired) topics, keywords, and / or other topics. With respect to each of the one or more corresponding conversations being simulated, the respective speakers of the simulated conversations may chat with each other (e.g., operated or managed by the conversation simulator component), and the conversation simulator component may generate text data based at least in part on (e.g., corresponding to, representing, or transcribing) the simulated conversations. According to various implementations, method 700 may proceed to reference numeral 714 and / or reference numeral 720 .

[0157] At 714, the grammar, wording, coherence and / or other qualities of the text data corresponding to one or more corresponding simulated dialogues can be evaluated. At 716, the best performing items in the text data can be determined at least in part based on the evaluation results. At 718, the prompts of the simulated speaker can be updated (e.g., enhanced or improved) at least in part based on the text data items that perform best. The evaluation component (e.g., commentator agent) can evaluate (e.g., evaluate) the grammar, wording, coherence and / or other qualities (e.g., quality characteristics) of the text data corresponding to the simulated dialogues of one or more corresponding mixed languages ​​between the speakers (e.g., simulated speakers). The evaluation component can evaluate the text data to ensure that the text data of one or more corresponding simulated dialogues meet (e.g., meet or exceed) the required (e.g., defined and / or high) language standards and can represent the natural language that can be generally used in the mixed language background. In some embodiments, the assessment component can employ or incorporate a mechanism that can score (e.g., determine or calculate grammar scores, rhetoric scores, coherence scores, and / or other quality metric scores), classify, and / or sort individual text data items based at least in part on quality metrics (e.g., grammar, wording, coherence, and / or other quality metrics). The assessment component can use or select the best performing data set (e.g., the best performing text data item) from the text data (e.g., top-k data) to update (e.g., modify and / or enhance) the prompts of the speaker (e.g., the simulated speaker of the speaker agent), thereby enabling a continuous improvement cycle (e.g., feedback loop) that can refine (e.g., enhance) the data output (e.g., mixed-language text data) in a continuous cycle (e.g., iteration).

[0158] At 720, text data (e.g., a mixed language data set) associated with or representing one or more corresponding simulated conversations in a mixed language and associated with one or more respective topics can be presented as output. The MLDGM component can present (e.g., communicate, display, or otherwise present) text data associated with or representing one or more corresponding simulated conversations in a mixed language as output (e.g., information associated with an enhanced mixed language data set can be presented as output). In some embodiments, additionally or alternatively, the MLDGM component (e.g., employing a speech generator component) can generate audio content corresponding to and / or representing the text data and present it as output.

[0159] Fig. 9 and Fig.10A flow chart of an example method 900 is shown, which can be based on various aspects and embodiments of the presently disclosed subject matter, at least in part, based on information related to an enhanced mixed language data set to ideally (e.g., automatically, dynamically, appropriately, reliably, efficiently, enhanced and / or optimally) train a speech recognition model (e.g., a mixed language speech recognition model) and / or perform updates based at least in part on FAL evaluations to facilitate the training of the speech recognition model. Method 900 can be employed by a system including, for example, a model manager component, a trained model (e.g., a trained GPT-4 model or other type of trained artificial intelligence-based model), an MLDGM component, a FAL evaluator component, a speech recognition component (e.g., including a mixed language speech recognition model), a fine tuner component, and / or another component, which components can include a processor component, a data repository, and / or other components, or be associated with a processor component, a data repository, and / or other components, respectively. The trained model can include or be associated with an MLDGM component and a FAL evaluator component. In some embodiments, the speech recognition component can include or employ a speech recognition model, such as described herein. The model manager component may manage (eg, control) the execution of the various operations of method 900 .

[0160] At 902, the speech recognition model can be configured (e.g., fine-tuned) based at least in part on a set of hyperparameters and the desired size of the speech recognition model. The fine-tuner component can pass or apply the set of hyperparameters and / or a set of parameters to the speech recognition model (e.g., a mixed language speech recognition model) to facilitate the configuration (e.g., fine-tuning) of the model, wherein the hyperparameter group can be determined and / or selected based at least in part on the desired size of the speech recognition model (e.g., tiny, basic, small, medium or large). The speech recognition model can receive information related to the hyperparameter group and / or the parameter group, and can be configured at least in part based on the hyperparameter group and / or the parameter group. In some embodiments, the speech recognition model can be pre-trained or initially trained, such as described herein.

[0161] At 904, the speech recognition model can receive audio content or a spectrogram representing the audio content (e.g., a log-mel spectrogram), and the audio content can be a representative of a mixed language data set. The mixed language data set can be an enhanced mixed language data set, which can be generated by a mixed language data generation manager component, such as described herein. The mixed language data set can include text data in at least two languages, wherein each second text data item of a second language (e.g., English or another desired language) can be interspersed between each first text item of a first language (e.g., Cantonese or another desired language different from the second language). The text data can be annotated (e.g., recognized, word-unit or annotated) speech data, so as to facilitate the training of the mixed language speech recognition model. In some embodiments, the audio converter component can generate a spectrogram representing the audio content based at least in part on the result of analyzing the audio content. The model manager component can manage the audio content or the spectrogram input or application to the speech recognition model, and the speech recognition model can receive the audio content or the spectrogram. For example, the audio converter component (e.g., managed by the model manager component) can input or apply the spectrogram to the speech recognition model.

[0162] At 906, the speech recognition model may receive textual transcription data representing the mixed-language data set or a word-gram that may characterize the textual transcription data. The textual transcription data or word-gram may represent annotated speech data of the mixed-language data set. The model manager component may input or apply the textual transcription data or word-gram to the speech recognition model and / or may manage the input or application of the textual transcription data or word-gram to the speech recognition model, which may receive the textual transcription data or word-gram.

[0163] At 908, the speech recognition model can analyze the audio content or spectrogram and the text transcription data or word units (e.g., perform artificial intelligence-based analysis). In some implementations, the speech recognition model can analyze the audio content or spectrogram and the text transcription data or word units simultaneously, concurrently, or in parallel.

[0164] At 910, based at least in part on the analysis results, the speech recognition model can predict or determine individual word-grams that represent individual words or sub-words contained in the audio content or in the spectrogram. At 912, based at least in part on the analysis results and one or more previously predicted or determined word-grams, the speech recognition model can predict a next word-gram that can represent a word or sub-word (e.g., the next word or sub-word in a sequence of words or sub-words that appear or are contained in the audio content or in the spectrogram). At 914, based at least in part on the analysis results and one or more predicted or determined word-grams, the speech recognition model can determine a text transcription that includes words and / or characters in a mixed language (e.g., transcribed words and / or characters) that can represent the audio content and / or spectrogram. In some embodiments, as part of the analysis, the speech recognition model can employ an encoder component that processes and encodes the audio content or spectrogram to generate encoded data, and a decoder component that can decode the encoded data to predict or determine a word unit, predict the next word unit, and / or generate a text transcript representing the encoded data and correspondingly representing the audio content and / or spectrogram, such as described herein. At this point, method 900 can proceed to reference point B, where method 900 can start from reference point B, such as Fig.10 As depicted in and described herein.

[0165] At 916, a loss associated with any differences between the predicted word-grams and the actual word-grams representing the text transcription data can be determined. A loss component of or associated with the speech recognition model can determine (e.g., calculate) a loss (e.g., a loss function) associated with any differences between the predicted word-grams (e.g., predicted by the model) and the actual word-grams representing the text transcription data.

[0166] At 918, one or more parameters or hyperparameters of the speech recognition model may be updated based at least in part on the loss to facilitate training (e.g., fine-tuning, further training, or perfecting training) of the speech recognition model. An update component or a loss component (or another component of the model or another component associated with the model) may determine an update based at least in part on the loss, the update including one or more modified parameters or hyperparameters, such as described herein. An update component or a loss component (or another component of the model or another component associated with the model) may update (e.g., modify) or facilitate updating (e.g., reconfiguring) the speech recognition model based at least in part on the update (e.g., one or more modified parameters or hyperparameters) to facilitate training the speech recognition model, such as described herein.

[0167] At 920, the speech recognition model can present the text transcription as output. Based at least in part on the determination of the text transcription, the speech recognition model can present the text transcription including words and / or characters (eg, transcribed words and / or characters) of the mixed language as output.

[0168] At 922, the fidelity of the text transcription to the audio content can be determined based at least in part on the results of analyzing the text transcription and the audio content. For example, the FAL evaluator component can analyze (e.g., evaluate) the text transcription (e.g., at least a portion of the text transcription) and the audio content (e.g., at least a portion of the audio content, the at least a portion corresponding to the above-mentioned portion of the text transcription), and based at least in part on the analysis results, the FAL evaluator component can determine (e.g., calculate) a fidelity (e.g., a fidelity score, rating, or value), which can represent the degree (e.g., accuracy) of the speech recognition model to capture the content (e.g., speech content, such as spoken words) and meaning of the original audio content (e.g., spoken by a speaker corresponding to the enhanced mixed-language dataset) in terms of the intended message, tone, and context of the spoken content (e.g., spoken words) of the audio content, to ensure that the text transcription retains the intended message, tone, and context of the spoken content.

[0169] At 924, the accuracy of the text transcription relative to the audio content can be determined based at least in part on the analysis results of the text transcription and the audio content. For example, based at least in part on the analysis results, the FAL evaluator component can determine (e.g., calculate or measure) the accuracy (e.g., accuracy score, rating, or value) or correctness of the text transcription relative to the spoken words in the audio content (e.g., compared to the spoken words). The determination or measurement of the accuracy of the text transcription can involve the FAL evaluator component evaluating the ability of the speech recognition model to accurately (correctly) recognize the spoken words (or tokens representing the spoken words) and convert them into text data (e.g., written text), including accurately representing the tone and pronunciation associated with each of the multiple mixed languages ​​in the audio content.

[0170] At 926, the amount of delay associated with the speech recognition model processing and generating a text transcription representing the audio content can be determined based at least in part on an analysis of the amount of time spent during processing the audio content or spectrogram and generating the text transcription. The FAL evaluator component can determine (e.g., calculate, track, or otherwise determine) the amount of delay associated with the above-mentioned processing and generation of the speech recognition model and / or a corresponding delay score, rating, or value based at least in part on an analysis of the amount of time spent during processing the audio content or spectrogram and generating the text transcription representing the audio content by the speech recognition model. By determining the delay, the FAL evaluator component can evaluate and determine the speed, responsiveness, and efficiency of the speech recognition model in generating a text transcription representing the audio content (e.g., spoken words).

[0171] At 928, a generation process for a mixed language dataset for subsequent model training and / or updates to hyperparameters for configuring a speech recognition model may be determined based at least in part on the fidelity, accuracy, and / or latency. In some embodiments, a FAL evaluator component (or another component, such as an update component of a MLDGM component, or an update component associated with a MLDGM component) may determine updates (e.g., update information) to a generation process for a mixed language dataset and / or updates to hyperparameters or parameters for configuring a speech recognition model based at least in part on the fidelity, accuracy, and / or latency (e.g., a fidelity score, an accuracy score, and / or a latency score), such as described herein.

[0172] At 930, the process of generating the mixed language data set and / or the hyperparameters for configuring the speech recognition model can be updated based at least in part on the update. The FAL evaluator component (or another component, such as an update component of the MLDGM component, or an update component associated with the MLDGM component) can update the process of generating the mixed language data set and / or the hyperparameters (and / or parameters) for configuring the speech recognition model based at least in part on the update. In some embodiments, if the update involves an update to the hyperparameters and / or speech recognition model parameters, the FAL evaluator component (or other component) can reconcile such updates to the hyperparameters and / or parameters with any other updates to the hyperparameters and / or parameters (if any) determined by the speech recognition component (e.g., the update component or loss component of the FAL) to facilitate determining the required unified updates to the hyperparameters and / or parameters for configuring (e.g., reconfiguring or fine-tuning) the speech recognition model. The MLDGM component employing the update process can determine and generate another mixed-language dataset (e.g., another further enhanced mixed-language dataset) that can be used as input to the speech recognition model, and / or the fine-tuner component can configure or facilitate configuration of hyperparameters and / or other parameters of the speech recognition model (e.g., updated hyperparameters and / or other parameters) as part of another iteration of speech recognition model training. In some embodiments, method 900 can return to reference number 902 (e.g., via reference point C), where one or more iterations of method 900 can be performed (e.g., performed by the MLDGM component, the fine-tuner component, the speech recognition component, the speech recognition model, the FAL evaluator component, and / or other components) to facilitate training of the speech recognition model (e.g., further training or refining the training).

[0173] Fig.11 A flow diagram of an example method 1100 is depicted, which can desirably (e.g., automatically, dynamically, appropriately, reliably, efficiently, enhanced, and / or optimally) perform speech recognition and speech-to-text (e.g., perform ASR on mixed-language speech) on mixed-language speech in accordance with various aspects and embodiments of the disclosed subject matter. For example, a system including a trained speech recognition model (e.g., a trained mixed-language speech recognition model) and / or a model manager component can employ method 1100, which can include or be associated with a processor component, a data repository, and / or other components. In some embodiments, the model manager component can manage (e.g., control) the performance of various operations of method 1100.

[0174] At 1102, a trained speech recognition model may receive audio content or a spectrogram representing the audio content (e.g., a log-mel spectrogram), and the audio content may include spoken words in a mixed language. For example, the audio content may include spoken words, each first word using a first language, each second word using a second language, wherein each second word may be interspersed with each first word (e.g., each second word is inserted, intermixed, or mixed between each first word).

[0175] At 1104, the trained speech recognition model may analyze the audio content or spectrogram. At 1106, based at least in part on the results of the analysis of the audio content or spectrogram (e.g., performing an artificial intelligence-based analysis), the trained speech recognition model may determine (e.g., recognize) the spoken words of the mixed language contained in the audio content. At 1108, the trained speech recognition model may present (e.g., generate and present) a text transcription of the spoken words (and / or characters) of the mixed language as an output, which may represent the spoken words contained in the audio content.

[0176] In order to provide additional context for the various embodiments described herein, Fig.12 The following discussion is intended to provide a concise, general description of a suitable computing environment 1200 in which various embodiments of the embodiments described herein may be implemented. Although the above embodiments have been described in the general context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will recognize that the embodiments may also be combined with other program modules and / or implemented as a combination of hardware and software.

[0177] Generally speaking, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In addition, those skilled in the art will understand that these methods can be implemented by other computer system configurations, including single-processor or multi-processor computer systems, microcomputers, mainframe computers, Internet of Things devices, distributed computing systems, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc., each of which can be operatively connected to one or more associated devices.

[0178] The illustrated embodiments of the embodiments herein may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0179] Computing devices typically include various media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, which are used differently in this article as follows. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by a computer, including volatile media and non-volatile media, removable media and non-removable media. By way of example and not limitation, computer-readable storage media or machine-readable storage media can be implemented in association with any method or technology for storing information (such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data).

[0180] Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disk (BD) or other optical disk storage, magnetic tape, cassette, magnetic disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store the desired information. In this regard, the terms "tangible" or "non-transitory" as applied to storage, memory, or computer-readable media herein should be understood to exclude only the propagation of transitory signals themselves from the modifier, and do not disclaim the right to all standard storage, memory, or computer-readable media that propagate not only transitory signals themselves.

[0181] One or more local or remote computing devices may access the computer-readable storage medium, for example, via access requests, queries, or other data retrieval protocols, and perform various operations on the information stored by the medium.

[0182] Communication media typically embodies computer readable instructions, data structures, program modules, or other structured or unstructured data in a data signal, such as a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term "modulated data signal" or "signal" refers to a signal that has one or more characteristics set or changed to encode information in one or more signals. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media.

[0183] Reference again Fig.12, an example environment 1200 for implementing various embodiments of various aspects described herein includes a computer 1202, which includes a processing unit 1204, a system memory 1206, and a system bus 1208. The system bus 1208 couples system components including, but not limited to, the system memory 1206 to the processing unit 1204. The processing unit 1204 can be any of a variety of commercially available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1204.

[0184] The system bus 1208 may be any of several types of bus structures, and it may further interconnect with a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 1206 includes ROM 1210 and RAM 1212. A basic input / output system (BIOS) may be stored in a nonvolatile memory such as ROM, an erasable programmable read-only memory (EPROM), or an EEPROM, wherein the BIOS contains basic routines that help to transfer information between elements within the computer 1202, such as during startup. The RAM 1212 may also include high-speed RAM, such as static RAM for caching data.

[0185] The computer 1202 also includes an internal hard disk drive (HDD) 1214 (e.g., EIDE, SATA), one or more external storage devices 1216 (e.g., a magnetic floppy disk drive (FDD) 1216, a memory stick or flash drive reader, a memory card reader, etc.), and an optical drive 1220 (e.g., read or write from a CD-ROM, DVD, BD, etc.). Although the internal HDD 1214 shown is located in the computer 1202, the internal HDD 1214 can also be configured in a suitable chassis (not shown) for external use. In addition, although not shown in the environment 1200, a solid-state drive (SSD) can be used as a supplement or alternative to the HDD 1214. The HDD 1214, the external storage device 1216, and the optical drive 1220 can be connected to the system bus 1208 via a HDD interface 1224, an external storage interface 1226, and an optical drive interface 1228, respectively. The interface 1224 for external drive implementations may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are also within the contemplated scope of the embodiments described herein.

[0186] The drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For the computer 1202, the drives and storage media can store data in any suitable digital format. Although the above description of computer-readable storage media refers to various types of storage devices, it should be understood by those skilled in the art that other types of computer-readable storage media can also be used in this example operating environment, whether these storage media are currently existing or developed in the future, and any such storage media can contain computer-executable instructions for performing the methods described herein.

[0187] Many program modules may be stored in the drives and RAM 1212, including an operating system 1230, one or more application programs 1232, other program modules 1234, and program data 1236. All or portions of the operating system, application programs, modules, and / or data may also be cached in RAM 1212. The systems and methods described herein may be implemented using various commercially available operating systems or combinations of operating systems.

[0188] Computer 1202 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment of operating system 1230, and the emulated hardware may optionally be different from the hardware in the operating system. Fig.12 . In this embodiment, operating system 1230 may include one of multiple virtual machines (VM) hosted by computer 1202. In addition, operating system 1230 may provide a runtime environment, such as a Java runtime environment or a .NET framework, for application 1232. A runtime environment is a consistent execution environment that allows application 1232 to run on any operating system that includes a runtime environment. Similarly, operating system 1230 may support containers, and application 1232 may take the form of a container, which is a lightweight, independent, executable software package that includes, for example, code, runtime, system tools, system libraries, and settings for the application.

[0189] In addition, the computer 1202 may also enable a security module, such as a trusted processing module (TPM). For example, through the TPM, the boot component hashes the next boot component of the time and waits for the result to match the security value before loading the next boot component. This process can be performed at any layer of the code execution stack of the computer 1202, for example, applied to the application execution layer or the operating system (OS) kernel layer, thereby achieving security at any layer of code execution.

[0190] A user may enter commands and information into the computer 1202 through one or more wired or wireless input devices (e.g., a keyboard 1238, a touch screen 1240, and a pointing device such as a mouse 1242). Other input devices (not shown) may include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control or other remote control, a joystick, a virtual reality controller and / or a virtual reality headset, a game controller, a stylus, an image input device (e.g., a camera), a gesture sensor input device, a visual movement sensor input device, an emotion or facial detection device, a biometric input device (e.g., a fingerprint or iris scanner), or the like. These input devices and other input devices are typically connected to the processing unit 1204 through an input device interface 1244, which may be coupled to the system bus 1208 or may be connected through other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an infrared port, Interfaces, etc.

[0191] A monitor 1246 or other type of display device may also be connected to the system bus 1208 via an interface, such as a video adapter 1248. In addition to the monitor 1246, computers typically include other peripheral output devices (not shown), such as speakers, printers, and the like.

[0192] The computer 1202 can operate in a network environment using logical connections to communicate with one or more remote computers, such as remote computer 1250, via wired or wireless communications. The remote computer 1250 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other common network node, and typically includes many or all of the described elements associated with the computer 1202, although only a memory / storage device 1252 is shown for simplicity. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1254 and / or a larger network, such as a wide area network (WAN) 1256. Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global communication network, such as the Internet.

[0193] When used in a LAN networking environment, the computer 1202 can be connected to the local network 1254 through a wired and / or wireless communication network interface or adapter 1258. The adapter 1258 can facilitate wired or wireless communication with the LAN 1254, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter 1258 in a wireless mode.

[0194] When used in a wide area network environment, the computer 1202 may include a modem 1260 or may be connected to a communication server on the wide area network 1256 via other means to establish communications on the wide area network 1256, such as through the Internet. The modem 1260 may be an internal or external device, and may be a wired or wireless device, and may be connected to the system bus 1208 via the input device interface 1244. In a network environment, program modules related to the computer 1202 or parts thereof may be stored in the remote memory / storage device 1252. It will be appreciated that the network connections shown are only examples, and other means may be used to establish a communications link between computers.

[0195] When used in a LAN or WAN network environment, the computer 1202 can access a cloud storage system or other network-based storage system to supplement or replace the external storage device 1216 described above. In general, the connection between the computer 1202 and the cloud storage system can be established through the LAN 1254 or WAN 1256 (e.g., through the adapter 1258 or modem 1260, respectively). After the computer 1202 is connected to the associated cloud storage system, the external storage interface 1226 can manage the storage provided by the cloud storage system with the adapter 1258 and / or modem 1260, just as it manages other types of external storage. For example, the external storage interface 1226 can be configured to provide access to cloud storage sources as if these storage sources were physically connected to the computer 1202.

[0196] Computer 1202 may be operable to communicate with any wireless device or entity operatively arranged in wireless communication, such as printers, scanners, desktop and / or portable computers, portable data assistants, communication satellites, any device or location associated with a wirelessly detectable tag (e.g., a newsstand, a newsstand, a store shelf, etc.), and telephones. This may include Wireless Fidelity (Wi-Fi) and Wireless technology. Therefore, the communication can be a predetermined structure of a traditional network or just an ad hoc communication between at least two devices.

[0197] Wi-Fi, or Wireless Fidelity, allows people to connect to the Internet from their couch at home, in their hotel room, or in a conference room at work, without the need for wires. Wi-Fi is a wireless technology, similar to that used in cell phones, that enables such devices (e.g., computers) to send and receive data indoors and outdoors, anywhere within range of a base station. Wi-Fi networks use a radio technology called IEEE802.11 (a, b, g, etc.) to provide secure, reliable, and fast wireless connections. Wi-Fi networks can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE802.3 or Ethernet). Wi-Fi networks operate on the unlicensed 2.4 and 5 GHz radio bands with data transfer rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), for example, or using products that include both bands (dual-band), so the network can provide real-world performance similar to the basic 10BaseT wired Ethernet networks used in many offices.

[0198] The various aspects or features described herein can be implemented as methods, devices, systems or manufactured products using standard programming or engineering techniques. In addition, the various aspects or features disclosed in this disclosure can also be implemented by program modules, which at least implement one or more methods disclosed herein, and the program modules are stored in a memory and executed by at least one processor. Other combinations of hardware and software or hardware and firmware can enable or implement the various aspects described herein, including the disclosed methods. The term "manufactured product" used herein is intended to cover computer programs accessible from any computer-readable device, carrier or storage medium. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip, etc.), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk (BD), etc.), a smart card, and a memory device including a volatile memory and / or a non-volatile memory (e.g., a flash memory device, such as a card, a stick, a key drive, etc.) or a similar device. According to various implementations, a computer-readable storage medium may be a non-temporary computer-readable storage medium and / or a computer-readable storage device may include a computer-readable storage medium.

[0199] As used in this specification, the term "processor" may refer to substantially any computing processing unit or device, including but not limited to: a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. The processor may be or may include multiple processors, for example, a distributed processor or parallel processor on a single machine or multiple machines. In addition, the processor may include or refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable gate array (PGA), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a state machine, discrete gate or transistor logic, discrete hardware components, or any combination thereof, intended to perform the functions described herein. In addition, the processor may utilize nanoscale architectures, such as but not limited to transistors, switches, and gates based on molecules and quantum dots, to optimize space usage or enhance the performance of user devices. The processor may also be implemented as a combination of computing processing units.

[0200] The processor may facilitate various types of operations, such as by executing computer-executable instructions. When the processor executes instructions to perform an operation, this may include the processor performing (e.g., directly performing) the operation and / or the processor performing the operation indirectly, such as by facilitating (e.g., facilitating the operation), directing, controlling, or cooperating with one or more other devices or components to perform the operation. In some implementations, a memory may store computer-executable instructions, and a processor may be communicatively coupled to the memory, wherein the processor may access or retrieve the computer-executable instructions from the memory, and may facilitate execution of the computer-executable instructions to perform the operation.

[0201] In some implementations, the processor may be or may include one or more processors that may be used to support a virtualized computing environment or a virtualized processing environment. A virtualized computing environment may support one or more virtual machines representing a computer, server, or other computing device. In such a virtualized virtual machine, components such as processors and storage devices may be virtualized or logically represented.

[0202] In this specification, terms such as "store", "storage", "data repository", "data storage", "database", and substantially any other information storage component related to the operation and functionality of the component are used to refer to an entity embodied by a "memory component", "memory", or a component including a memory. It is understood that the memory and / or memory component described herein may be a volatile memory or a non-volatile memory, and may also include a volatile memory and a non-volatile memory.

[0203] By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) as external buffer memory. By way of illustration and not limitation, RAM may come in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRS DRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). In addition, the memory components of the systems or methods disclosed herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0204] In the present disclosure, the terms "component", "system", "platform", "framework", "layer", "interface", "agent", etc. may refer to and / or may include entities related to computers or entities related to operating machines having one or more specific functions. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be (but is not limited to) a process running on a processor, a processor, an object, an executable file, an execution thread, a computer-executable instruction, a program, and / or a computer. As an illustration, both an application running on a server and a server may be a component. One or more components may be located in a process and / or execution thread, and a component may be located on one computer and / or distributed between two or more computers.

[0205] In another example, the various components can be executed by various computer-readable media on which various data structures are stored. The components can communicate via local and / or remote processes, such as communicating according to signals with one or more data packets (e.g., data of a component interacting with another component in a local system, a distributed system, and / or data across a network (such as the Internet) with other systems via signals). As another example, a component can be a device that provides a specific function by a mechanical part operated by an electrical or electronic circuit, which is operated by a software or firmware application executed by a processor. In this case, the processor can be inside or outside the device and can execute at least part of the software or firmware application. As another example, a component can be a device that provides a specific function by an electronic component without a mechanical part, wherein the electronic component can include a processor or other means of executing software or firmware, and the software or firmware at least partially gives the function of the electronic component. In one aspect, the component can simulate the electronic component via a virtual machine, for example, in a cloud computing system.

[0206] A communication device, such as described herein, may be or may include, for example, a computer, a laptop, a server, a mobile phone (e.g., a smartphone), an electronic tablet or pad, an electronic gaming device, an electronic head-mounted device or body-mounted device (e.g., electronic glasses, a smart watch, an augmented reality (AR) / virtual reality (VR) headset, or other types of electronic head-mounted devices or body-mounted devices), a set-top box, an Internet Protocol (IP) television (IPTV), an IoT device (e.g., a medical device, an electronic speaker with a voice controller, a camera device, a security device, a tracking device, an appliance, or other IoT device), or other desired communication device type.

[0207] In addition, the term "or" is intended to be an inclusive or rather than an exclusive or. That is, unless stated otherwise or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is true in any of the above cases.

[0208] As used herein, the terms "example," "exemplary," and / or "exemplary" are used to refer to an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to these examples. In addition, any aspect or design described herein as "example," "exemplary," and / or "exemplary" is not necessarily to be construed as superior or more beneficial to other aspects or designs, nor is it meant to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art. In addition, to the extent the terms "including," "having," "comprising," and other similar words are used in the detailed description or claims, these terms are intended to express inclusiveness in a manner similar to the term "comprise" as an open transition word, without excluding any additional or other elements.

[0209] It should be recognized and understood that components (e.g., model manager components, models or trained models, speech recognition components, speech recognition models, devices, MLDGM components, FAL evaluator components, audio recorder components, artificial intelligence components, processor components, data repositories, or other components) as described with respect to particular systems or methods may include the same or similar functionality as various components (e.g., separately named components or similarly named components) described with respect to other systems or methods disclosed herein.

[0210] The foregoing includes examples of systems and methods that provide advantages of the disclosed subject matter. Of course, it is not possible to describe every conceivable combination of components or methods for purposes of describing the disclosed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the disclosed subject matter are possible. In addition, when the terms "including," "having," "having," and the like are used in the detailed description, claims, appendices, and drawings, these terms are intended to be interpreted in a manner similar to the word "comprising" when used as a transitional word in a claim, indicating an inclusive meaning.

Claims

1. A method comprising: generating, by a trained model of a system including at least one processor, mixed language data based on instruction data related to generation of mixed language data and seed topic data related to a set of seed topics, wherein the mixed language data includes respective first text words in a first language and respective second text words in a second language, the respective first text words and the respective second text words being determined based on corresponding simulated conversations between respective simulated speakers, wherein the respective second text words are interspersed with the respective first text words; and As part of training of a speech recognition model, a quality metric is determined by the trained model of the system indicating speech recognition performance of the speech recognition model in identifying information related to spoken mixed-language words representing the mixed-language data and generating text transcribed mixed-language words representing the spoken mixed-language words, wherein the quality metric indicates fidelity, accuracy, and latency related to the speech recognition performance.

2. The method according to claim 1, further comprising: Based on the analysis result of the textual transcription mixed-language word, the transcription of the mixed-language data, and / or the information related to the audio content including the spoken mixed-language word: determining, by the trained model of the system, a fidelity metric indicating the fidelity of the textual transcription mixed-language word to the audio content including the spoken mixed-language word; determining, from the trained model of the system, an accuracy metric indicating the accuracy of the textually transcribed mixed-language words relative to the spoken mixed-language words; as well as A delay metric value is determined from the trained model of the system, the delay metric value being indicative of the delay associated with the speech recognition model recognizing the spoken mixed-language word and generating the text transcribed mixed-language word.

3. The method according to claim 2, further comprising: With respect to an iteration of the training of the speech recognition model, determining, by the trained model of the system, an update related to the training of the speech recognition model based on the fidelity metric, the accuracy metric, and / or the latency metric in accordance with a defined model management standard, the update comprising update information; as well as Regarding subsequent iterations of the training of the speech recognition model, based on the update information, the trained model of the system initiates modifications to the hyperparameters of the speech recognition model, the first parameters of the speech recognition model, the second parameters of the trained model, the third parameters related to the generation of the mixed-language data, and / or the processes related to the generation of the mixed-language data.

4. The method according to any one of claims 1 to 3, wherein: In the same sentence, the second parts of the second text words are interspersed with the first parts of the first text words.

5. The method according to any one of claims 1 to 3, wherein: The respective first text words include respective characters associated with the first language.

6. The method according to claim 1, further comprising: collecting, by the trained model of the system, respective content items from respective data sources based on the instruction data and the seed topic data, wherein the respective content items include respective text words in respective languages; extracting topics and keywords from the respective content items based on the first results of analyzing the respective content items by the trained model of the system; and The trained model of the system simulates the corresponding simulated dialogue between the respective simulated speakers based on the second result of analyzing the topic and the keywords, wherein the simulated dialogue includes respective spoken dialogue words related to the topic and the keywords, wherein the respective spoken dialogue words include a first part of the respective first text words of the first language and a second part of the respective second text words of the second language.

7. The method according to claim 6, further comprising: evaluating, by the trained model of the system, performance and efficacy associated with obtaining the respective content items via one or more web crawler devices with respect to the collection according to a defined performance metric and a defined efficacy metric; determining, by the trained model of the system, content collection update data or feedback data based on the evaluation of the performance and the efficacy; as well as Modifications to subsequent collection of respective subsequent content items from at least some of the respective data sources and / or modifications to the operation of the one or more web crawler devices are initiated by the trained model of the system.

8. The method according to claim 6, further comprising: evaluating, by the trained model of the system, grammar, wording and / or coherence of the corresponding simulated dialogue based on a first result of analyzing the corresponding simulated dialogue according to a grammar metric, wording metric and / or coherence metric, wherein the corresponding simulated dialogue comprises the respective first text words and the respective second text words, wherein the corresponding simulated dialogue is associated with the iteration of the training of the speech recognition model; sorting, by the trained model of the system, the respective text data items of the corresponding simulated conversation based on the second result of the evaluation; determining, by the trained model of the system, from the respective text data items, based on the ranking, a higher performing text data item that is ranked higher than a lower performing text data item in terms of the grammar, the wording, and / or the coherence; Determining, by the trained model of the system, prompt update information based on the higher performance text data item; and One or more conversation prompts associated with simulating the corresponding simulated conversations between the respective simulated speakers are modified by the trained model of the system so as to facilitate utilization of the one or more modified conversation prompts during simulation of corresponding subsequent simulated conversations between the respective simulated speakers to generate subsequent mixed-language data for subsequent iterations of the training of the speech recognition model.

9. The method according to claim 1, further comprising: The system starts setting hyperparameters and parameters of the speech recognition model; generating, by the system, a log-mel spectrogram based on a first result of analyzing the audio content, the log-mel spectrogram comprising spectrogram information representing audio content, the audio content comprising the spoken mixed-language words representing the mixed-language data; Based on the second result of analyzing the mixed language data, the system generates respective word-grams representing respective text words or respective text sub-words in the mixed language data, wherein the respective text words include the respective first text words in the first language and the respective second text words in the second language, wherein the respective text sub-words are respective parts of some text words in the respective text words; and The system starts inputting the log-mel spectrogram and the individual word units into the speech recognition model so as to perform speech recognition on the log-mel spectrogram and perform the training of the speech recognition model.

10. The method according to claim 9, wherein: The speech recognition model encodes the spectrogram information to generate encoded information representing the spectrogram information based on analysis of the spectrogram information, wherein the speech recognition model decodes the encoded information to predict corresponding decoded word elements representing the respective text words, the respective text words including the respective first text words of the first language and the respective second text words of the second language.

11. The method according to claim 10, wherein: The speech recognition model generates a speech recognition transcription based on the decoded tokens, the speech recognition transcription comprising the text transcription mixed-language words, wherein the text transcription mixed-language words represent the spoken mixed-language words and represent the individual text words, the individual text words comprising the individual first text words in the first language and the individual second text words in the second language.

12. The method according to claim 10, wherein: The speech recognition model predicts a next decoded word-gram of the corresponding decoded word-gram based on a previous decoded word-gram of the corresponding decoded word-gram and an analysis of the encoding information, wherein the next decoded word-gram is associated with a next spoken word or a next spoken sub-word of the spoken mixed-language word.

13. A system comprising: at least one processor, at least one memory storing one or more computer executable programs, When the one or more computer executable programs are executed by the at least one processor, the at least one processor is caused to implement the method according to claims 1-12.

14. A non-transitory machine-readable medium having executable instructions stored thereon, wherein the executable instructions, when executed by at least one processor, implement the method according to any one of claims 1 to 12.

Images