Speech recognition model training method and device based on transliteration translation preference alignment

Abstract

The embodiment of the specification provides a speech recognition model training method and device based on phonetic translation and meaning translation preference alignment, wherein the speech recognition model training method based on phonetic translation and meaning translation preference alignment comprises: processing sample audio according to a phonetic translation strategy and a meaning translation strategy to obtain phonetic translation text and meaning translation text; constructing a target sample according to the sample audio, the phonetic translation text, the meaning translation text, and preference annotation information corresponding to the sample audio; processing the target sample by using a speech recognition model to obtain predicted text; performing multi-dimensional evaluation on the predicted text based on a reward model of the speech recognition model to obtain evaluation information; determining a reward loss value and a preference loss value according to the evaluation information; and fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used for phonetic translation recognition and / or meaning translation recognition on audio.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of speech recognition technology, and in particular to a method and apparatus for training a speech recognition model based on the alignment of transliteration and semantic translation preferences. Background Technology

[0002] With the development of computer technology, speech recognition technology is being applied in an increasing number of scenarios. However, existing Automatic Speech Recognition (ASR) models often face a conflict when processing cross-linguistic personal names, place names, brand names, and technical terms: "transliteration results retain pronunciation features but lack semantic clarity" versus "paraphrasing results convey meaning but lose the original pronunciation." Different application scenarios (such as news broadcasting, judicial records, and customer service quality inspection) have significantly different preferences for transliteration versus paraphrasing, and existing fine-tuning methods lack the ability to model and align these preferences, resulting in outputs that fail to meet business needs. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a method for training a speech recognition model based on transliteration-translation preference alignment. One or more embodiments of this specification also relate to a speech recognition method, a speech recognition model training device based on transliteration-translation preference alignment, a speech recognition device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a method for training a speech recognition model based on transliteration-translation preference alignment is provided, comprising:

[0005] The sample audio is processed according to the transliteration strategy and the free translation strategy to obtain the transliterated text and the free translation text. The target sample is constructed based on the sample audio, the transliterated text, the free translation text and the preference annotation information corresponding to the sample audio.

[0006] The target sample is processed using a speech recognition model to obtain a first predicted text and a second predicted text. The first predicted text and the second predicted text are evaluated in multiple dimensions based on the reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text.

[0007] The reward loss value is determined based on the first evaluation information, and the preference loss value is determined based on the first evaluation information and the second evaluation information;

[0008] The speech recognition model is fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used to perform transliteration recognition and / or paraphrase recognition of audio.

[0009] According to a second aspect of the embodiments of this specification, a speech recognition method is provided, comprising:

[0010] Obtain the audio to be processed and determine the preference tag corresponding to the audio to be processed;

[0011] The target speech recognition model is used to process the audio to be processed according to the preference labels to obtain multiple candidate texts, wherein the target speech recognition model is obtained by the above method;

[0012] The target text corresponding to the audio to be processed is determined from the plurality of candidate texts based on the preference tags.

[0013] According to a third aspect of the embodiments of this specification, a speech recognition model training apparatus based on transliteration-translation preference alignment is provided, comprising:

[0014] The processing module is configured to process the sample audio according to the transliteration strategy and the semantic translation strategy to obtain the transliterated text and the semantic translation text, and to construct the target sample based on the sample audio, the transliterated text, the semantic translation text and the preference annotation information corresponding to the sample audio.

[0015] The evaluation module is configured to process the target sample using a speech recognition model to obtain a first predicted text and a second predicted text, and to perform multi-dimensional evaluation of the first predicted text and the second predicted text based on the reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text.

[0016] The determination module is configured to determine a reward loss value based on the first evaluation information, and to determine a preference loss value based on the first evaluation information and the second evaluation information;

[0017] The fine-tuning module is configured to fine-tune the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used to perform transliteration recognition and / or paraphrase recognition of audio.

[0018] According to a fourth aspect of the embodiments of this specification, a voice recognition device is provided, comprising:

[0019] The acquisition module is configured to acquire the audio to be processed and determine the preference tag corresponding to the audio to be processed;

[0020] The processing module is configured to process the audio to be processed according to the preference tags using a target speech recognition model to obtain multiple candidate texts, wherein the target speech recognition model is obtained by the above method;

[0021] The determination module is configured to determine the target text corresponding to the audio to be processed from among the plurality of candidate texts based on the preference tags.

[0022] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0023] Memory and processor;

[0024] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration and translation preference alignment.

[0025] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration-translation preference alignment.

[0026] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration-translation preference alignment.

[0027] The speech recognition model training method based on transliteration and semantic translation preference alignment provided in this embodiment introduces a transliteration and semantic translation alignment mechanism during the fine-tuning of the speech recognition model. This allows the trained speech recognition model to output recognition results that conform to user preferences according to scenario requirements. First, sample audio is processed according to transliteration and semantic translation strategies to obtain transliterated and semantically translated texts. At this point, target samples are constructed based on the sample audio, transliterated text, semantically translated text, and the corresponding preference annotation information of the sample audio. Then, the speech recognition model is used to process the target samples to obtain a first predicted text and a second predicted text. The reward model based on the speech recognition model then... The first and second predicted texts are evaluated from multiple dimensions to obtain the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text. Based on this, the reward loss value can be determined according to the first evaluation information, and the preference loss value can be determined according to the first and second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration recognition and / or paraphrasing recognition of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a speech recognition model training method based on transliteration-translation preference alignment, provided in one embodiment of this specification.

[0029] Figure 2 This is a schematic diagram of model training in a speech recognition model training method based on transliteration-translation preference alignment provided in one embodiment of this specification;

[0030] Figure 3 This is a flowchart illustrating a speech recognition method provided in one embodiment of this specification;

[0031] Figure 4 This is a schematic diagram of the structure of a speech recognition model training device based on transliteration and translation preference alignment provided in one embodiment of this specification;

[0032] Figure 5 This is a schematic diagram of the structure of a speech recognition device provided in one embodiment of this specification;

[0033] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0034] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0035] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0037] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0038] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0039] Automatic speech recognition (ASR) is a computer technology that automatically converts human speech signals into corresponding text.

[0040] This specification provides a method for training a speech recognition model based on transliteration-translation preference alignment. One or more embodiments of this specification relate to a speech recognition method, a speech recognition model training device based on transliteration-translation preference alignment, a speech recognition device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0041] Automatic speech recognition (ASR) is a core technology for achieving natural speech interaction, playing a crucial role in scenarios such as training, customer service, and script quality control. Enterprises in these applications require not only highly accurate recognition results but also systems capable of outputting expected text content tailored to specific business scenarios to ensure effective communication and service quality. However, speech recognition tasks are inherently challenging due to limitations such as telephone channels, background noise, and regional accents. Traditional language understanding systems often lack sufficient support for spoken expressions, and the significant differences between spoken and written input further complicate the recognition process.

[0042] Currently, most mainstream ASR models are trained on large-scale public datasets and possess strong general recognition capabilities, but they perform poorly in recognizing cross-language personal names, place names, brand names, and industry terms. In these scenarios, ASR often faces a conflict between transliteration and semantic translation: transliteration can preserve pronunciation features but has limited semantic expression; semantic translation can convey meaning but may lose the original phonological form. Different business scenarios have different preferences for results. For example, training scenarios tend to favor transliteration to preserve pronunciation features, customer service prefers semantic translation to ensure semantic clarity, and script quality inspection requires flexible switching between transliteration and semantic translation.

[0043] However, most existing fine-tuning methods focus on optimizing overall recognition performance without explicitly introducing transliteration / translation preference modeling and alignment mechanisms. This makes it difficult for the model to output results that meet specific business needs when recognizing cross-linguistic mixed readings or cross-domain terms. Therefore, how to effectively introduce transliteration and translation preference constraints during the fine-tuning process of speech recognition pre-trained models, so that the model can flexibly adjust its output according to different scenario requirements, has become a technical problem that urgently needs to be solved in this embodiment.

[0044] The speech recognition model training method based on transliteration and semantic translation preference alignment provided in this embodiment introduces a transliteration and semantic translation alignment mechanism during the fine-tuning of the speech recognition model. This allows the trained speech recognition model to output recognition results that conform to user preferences according to scenario requirements. First, sample audio is processed according to transliteration and semantic translation strategies to obtain transliterated and semantically translated texts. At this point, target samples are constructed based on the sample audio, transliterated text, semantically translated text, and the corresponding preference annotation information of the sample audio. Then, the speech recognition model is used to process the target samples to obtain a first predicted text and a second predicted text. The reward model based on the speech recognition model then... The first and second predicted texts are evaluated from multiple dimensions to obtain the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text. Based on this, the reward loss value can be determined according to the first evaluation information, and the preference loss value can be determined according to the first and second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration recognition and / or paraphrasing recognition of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0045] See Figure 1 , Figure 1 A flowchart is shown of a speech recognition model training method based on transliteration-translation preference alignment according to an embodiment of this specification, which specifically includes the following steps.

[0046] Step S102: Process the sample audio according to the transliteration strategy and the semantic translation strategy to obtain the transliterated text and the semantic translation text. Construct the target sample based on the sample audio, the transliterated text, the semantic translation text and the preference annotation information corresponding to the sample audio.

[0047] The speech recognition model training method based on transliteration and semantic translation preference alignment provided in this embodiment can be applied to speech recognition model training in any scenario. For example, the trained speech recognition model can be used in training scenarios, customer service scenarios, or terminal device command interaction scenarios, so that the trained speech recognition model can automatically recognize the audio input by the user and generate text that meets the requirements of the current scenario for subsequent business use.

[0048] Specifically, the transliteration strategy refers to the strategy of transliterating sample audio to preserve the pronunciation features of the sample audio and obtain the corresponding transliterated text. The transliterated text is the text directly generated according to the audio content of the sample audio. Conversely, the semantic translation strategy refers to the strategy of semantic translation of sample audio to generate semantically translated text containing semantic meaning. The semantically translated text is the text generated after recognizing the semantic meaning of the sample audio. For example, if the sample audio content is "abcd", where "abcd" is the name of a mountain in the place name domain and the name of an app in the internet domain, when performing speech recognition on the sample audio, in a training scenario, transliteration results are preferred to preserve pronunciation features, while in a customer service scenario, semantic translation results are preferred to ensure semantic clarity. Therefore, in the sample construction stage, by constructing the transliterated and semantically translated texts corresponding to the sample audio, it is convenient for the subsequent training model to make transliteration and semantic translation decisions according to the needs of the scenario, thereby ensuring that the model outputs speech recognition results that conform to the scenario.

[0049] Correspondingly, preference labeling information specifically refers to the labeling information of the corresponding transliteration preference or semantic translation preference when performing speech recognition on sample audio. This information is used to construct target samples and facilitate the subsequent training of the model by clearly defining the label corresponding to each sample audio, so that the model can learn the ability of transliteration recognition and semantic translation recognition.

[0050] Based on this, in order to introduce a transliteration and semantic translation alignment mechanism during the fine-tuning of the speech recognition model, so that the trained speech recognition model can output recognition results that meet user preferences according to scenario requirements, the sample audio can be processed first according to the transliteration and semantic translation strategies to obtain transliterated text and semantic text. At this time, the target sample can be constructed based on the sample audio, transliterated text, semantic text, and the preference annotation information corresponding to the sample audio. Then, the speech recognition model can be used to process the target sample to obtain the first predicted text and the second predicted text. The reward model of the speech recognition model is used to evaluate the first predicted text and the second predicted text in multiple dimensions to obtain the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text. On this basis, the reward loss value can be determined based on the first evaluation information, and the preference loss value can be determined based on the first evaluation information and the second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration recognition and / or semantic recognition of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0051] In one or more embodiments provided in this example, the step of processing the sample audio according to the transliteration strategy and the semantic translation strategy to obtain the transliterated text and the semantic translation text includes:

[0052] Acquire sample audio; input the sample audio into the speech recognition model for processing according to the transliteration strategy, or process the sample audio using a preset dictionary and pronunciation rules, and determine the transliterated text based on the processing result; process the sample audio using parallel corpus, translation model or semantic rewriting algorithm according to the semantic translation strategy, and determine the semantic text based on the processing result.

[0053] Specifically, a speech recognition model refers to a model that has not yet undergone fine-tuning for speech recognition, such as Whisper; correspondingly, a pre-defined vocabulary refers to a dictionary that records professional terms, and pronunciation rules refer to the rules governing the pronunciation standards of different character units and word units; parallel corpora refer to standard corpora that are associated with different semantic meanings, and their corresponding paraphrased texts can be determined by matching them with sample audio; correspondingly, a translation model refers to a model that translates the content corresponding to sample audio, and a semantic rewriting algorithm refers to an algorithm that rewrites semantic meanings.

[0054] Based on this, before constructing target samples, for any sample audio, the sample audio can be acquired first. At this time, the sample audio can be input into the speech recognition model for processing according to the transliteration strategy, or the sample audio can be processed using a preset dictionary and pronunciation rules to determine the transliterated text containing pronunciation features based on the processing results. At the same time, the sample audio can also be processed using parallel corpora, translation models, or semantic rewriting algorithms according to the semantic translation strategy, and the semantic text containing semantic features can be determined based on the processing results, so that the transliterated text and the semantic text can be combined to construct sample pairs for training the speech recognition model.

[0055] In summary, the collaborative mechanism of speech feature analysis and deep semantic processing significantly improves the accuracy and naturalness of speech content conversion. Speech feature analysis ensures high-fidelity reproduction of the input audio, while deep semantic processing optimizes the logical coherence of the text expression. The combination of the two effectively overcomes the error problems of traditional speech recognition in dialects, accents, and complex contexts, providing more reliable technical support for multilingual intelligent interactive systems and greatly enhancing user experience and system usability.

[0056] In one or more embodiments provided in this example, constructing a target sample based on the sample audio, the transliterated text, the translated text, and the preference annotation information corresponding to the sample audio includes:

[0057] Preference data pairs are constructed based on the transliterated text and the translated text, and the preference data pairs are labeled according to the set labeling rules to obtain preference labeling information; triple samples are constructed based on the preference data pairs, the sample audio, and the preference labeling information, and the triple samples are used as target samples.

[0058] Specifically, the preference data pair refers to the data pair after combining the transliterated text and the translated text, and the annotation rule refers to the rule for adding preference annotation information to the sample audio. It is set according to whether the sample audio is suitable for transliteration or translation recognition operation. The annotation rule can be implemented by setting judgment criteria or by using a discriminant model. This embodiment does not make any limitations here.

[0059] Based on this, after obtaining the transliterated and translated texts, preference data pairs can be constructed based on the transliterated and translated texts. At this time, the preference data pairs can be labeled according to the set labeling rules to obtain preference labeling information. The preference labeling information can reflect the speech recognition preference corresponding to the sample audio. Therefore, triple samples can be constructed based on the preference data pairs, sample audios and preference labeling information. The triple samples can be used as target samples for subsequent model training.

[0060] In practical applications, to address the "transliteration / translation conflict" that exists during the recognition of cross-linguistic personal names, place names, brand names, and industry terms, a dataset with preference annotations can be constructed to ensure the trained speech recognition model achieves higher accuracy. This includes the following:

[0061] (1) Transliteration processing: Based on the original output of the speech recognition model, or combined with a proper noun dictionary and pronunciation rules, transliterated text that retains pronunciation features is automatically generated.

[0062] (2) Paraphrasing: Using parallel corpora, machine translation or semantic rewriting tools, generate paraphrased texts that convey semantic meaning.

[0063] (3) Preference data pair construction: for a given audio input Forming preference data pairs ,in This represents the transliteration results generated by the model. This represents the paraphrased results generated manually or according to rules. It is then combined with manual or semi-automatic discrimination mechanisms to perform preference labeling on both. .

[0064] (4) Data structuring: Finally, training triplet samples are formed. ,in It is audio input. is a candidate preference data pair, where p is the scene preference label.

[0065] This allows us to establish a preference dataset that reflects real business needs. It not only covers speech features and semantic information, but also explicitly introduces the "user's or scenario's preference for transliteration / translation," providing high-quality supervision signals for subsequent preference alignment training.

[0066] In summary, by constructing preference data pairs and generating standardized annotation information to form structured triplet samples, the problems of uneven data quality and redundant annotations in traditional preference learning are effectively solved. This mechanism significantly improves the model's accuracy in recognizing user intent, making the speech translation output more consistent with actual interaction scenarios, greatly enhancing the system's adaptability and service matching degree in multilingual environments, and providing a more efficient and reliable training foundation for intelligent interactive applications.

[0067] Step S104: Process the target sample using a speech recognition model to obtain a first predicted text and a second predicted text. Based on the reward model of the speech recognition model, evaluate the first predicted text and the second predicted text in multiple dimensions to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text.

[0068] Specifically, after constructing the target samples as described above, the speech recognition model can be trained based on these samples. During training, to enable the speech recognition model to learn the ability to switch between transliteration and semantic translation in different scenarios, a reward model can be used for multi-dimensional evaluation after the model processes the samples. The model can then be fine-tuned based on the evaluation results. In other words, the speech recognition model can be used to process the target samples, resulting in a first predicted text and a second predicted text. Then, the reward model of the speech recognition model can be used to perform multi-dimensional evaluations on the first and second predicted texts respectively, obtaining first evaluation information for the first predicted text and second evaluation information for the second predicted text. Subsequently, the loss value is calculated by combining the evaluation information to complete the fine-tuning of the speech recognition model.

[0069] Specifically, the first predicted text and the second predicted text refer to two texts obtained by randomly sampling after the speech recognition model performs speech recognition on the sample audio in the target sample. The reward model refers to the model that evaluates the predictive ability of the speech recognition model, which can perform multi-dimensional evaluation processing, such as preference dimension and / or semantic dimension. Correspondingly, the first evaluation information is the descriptive information that integrates the multi-dimensional evaluation results for the first predicted text, and the second evaluation information is the descriptive information that integrates the multi-dimensional evaluation results for the second predicted text.

[0070] In one or more embodiments provided in this example, the first predicted text is evaluated in multiple dimensions based on the reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text, including:

[0071] A preferred text is determined from the target sample, and the preferred text and the first predicted text are input into the reward model of the speech recognition model for processing. Based on the processing result, preference matching evaluation information corresponding to the preference matching dimension, semantic similarity evaluation information corresponding to the semantic similarity dimension, and keyword matching evaluation information corresponding to the keyword matching dimension are determined. The preference matching evaluation information, the semantic similarity evaluation information, and the keyword matching evaluation information are fused to obtain the first evaluation information corresponding to the first predicted text.

[0072] Specifically, the preference text refers to the transliterated or paraphrased text in the target sample that corresponds to the preference annotation information. The preference matching dimension refers to the dimension that evaluates the output text against the specified preference text in terms of overall structure and key segments. The preference matching evaluation information is the information obtained after evaluating the preference matching dimension, which can be in numerical or textual form. Correspondingly, the semantic similarity dimension refers to the dimension that measures whether the output text is semantically similar to the specified preference text. The semantic similarity evaluation information is the information obtained after evaluating the semantic similarity dimension. The keyword matching dimension refers to the dimension that evaluates the fidelity or correctness of the interpretation of names, place names, brand names, or terms. The keyword matching evaluation information is the information obtained after evaluating the keyword matching dimension.

[0073] Based on this, when using the reward model to evaluate the first predicted text, the preferred text can be determined in the target sample. At this time, the preferred text and the first predicted text can be input into the reward model of the speech recognition model for processing. The reward model can output the preference matching evaluation information corresponding to the preference matching dimension, the semantic similarity evaluation information corresponding to the semantic similarity dimension, and the keyword matching evaluation information corresponding to the keyword matching dimension. Then, the preference matching evaluation information, semantic similarity evaluation information, and keyword matching evaluation information can be fused to obtain the first evaluation information corresponding to the first predicted text.

[0074] In summary, a dynamic quality quantification mechanism is constructed by innovatively integrating preference matching, semantic similarity, and keyword matching as three evaluation dimensions. This mechanism effectively avoids the one-sidedness of a single evaluation indicator, significantly improves the objectivity and accuracy of the evaluation results, provides a high-precision optimization basis for speech recognition models, greatly enhances the output stability and interaction consistency of the system in dynamic interactive scenarios, and promotes the development of intelligent voice services towards a more accurate and fluent direction.

[0075] In one or more embodiments provided in this example, the determination of preference matching evaluation information corresponding to the preference matching dimension includes:

[0076] The reward model is used to perform character unit matching and text segment matching on the preferred text and the first predicted text. Based on the character unit matching results and text segment matching results, the preference matching evaluation information for the corresponding preference matching dimension is determined.

[0077] The determination of semantic similarity evaluation information corresponding to the semantic similarity dimension includes:

[0078] The reward model is used to calculate the similarity between the preferred text and the first predicted text to obtain semantic similarity evaluation information for the corresponding semantic similarity dimension;

[0079] The determination of keyword matching evaluation information for the corresponding keyword matching dimension includes:

[0080] Keywords are extracted from the first predicted text using the reward model, and the keywords are matched with benchmark keywords. Based on the matching results, keyword matching evaluation information for the corresponding keyword matching dimension is determined.

[0081] Specifically, character unit matching refers to the matching operation performed at the character unit level, while text segment matching refers to the matching operation performed at the text segment level. Correspondingly, similarity calculation refers to the semantic similarity calculation operation. Also, keywords refer to the keywords extracted from the predicted text, and benchmark keywords refer to standard professional terms.

[0082] Based on this, when determining the preference matching evaluation information for the preference matching dimension, a reward model can be used to perform character unit matching and text segment matching on the preference text and the first predicted text, thereby determining the preference matching evaluation information for the corresponding preference matching dimension based on the character unit matching results and the text segment matching results; when determining the semantic similarity evaluation information for the corresponding semantic similarity dimension, a reward model can be used to calculate the similarity between the preference text and the first predicted text, thereby obtaining the semantic similarity evaluation information for the corresponding semantic similarity dimension based on the similarity calculation results; when determining the keyword matching evaluation information for the corresponding keyword matching dimension, a reward model can be used to extract keywords from the first predicted text, at which point the keywords can be matched with benchmark keywords, thereby determining the keyword matching evaluation information for the corresponding keyword matching dimension based on the matching results.

[0083] In practical applications, after constructing the transliteration-translation preference dataset, fine-tuning with preference constraints can be performed on a pre-trained speech recognition model (such as Whisper) to enable the model to learn the ability to switch between transliteration and translation in different scenarios. Based on this, training samples with preference labels can be obtained. Then, reinforcement learning can be used to fine-tune the pre-trained speech recognition model so that it can stably align with transliteration / translation preferences in different scenarios.

[0084] In the policy modeling phase, the speech recognition model can be defined as a conditional policy. Where x is the audio input, The label is controlled by preference (transliteration / translation / adaptation), and the output y is the recognition sequence (predicted text).

[0085] In the reward model construction stage, the reward model R(x,y,p) comprehensively considers semantic consistency, proper name fidelity, and preference matching degree. The output of the reward model can be expressed by the following formula (1):

[0086] (1)

[0087] in, And it is adjustable; = (When p is "paraphrasing first") or = (When p is "transliteration priority"). See "Adaptive Preferences" at the end of the article for details on adaptive scenarios. Items representing the degree of matching of corresponding preferences. This indicates items with corresponding semantic consistency. Items indicating consistency of proper names.

[0088] Among them, preference matching degree [0,1] can be used to measure the degree of matching between the output and the target candidate specified by the preference (the speech recognition result corresponding to the preference annotation information) in terms of the overall and key segments. It is composed of two weighted parts: character-level similarity and key segment consistency, and can be expressed by the following formula (2):

[0089] (2)

[0090] Character / sub-word consistency CharF: uses normalized edit distance; Key fragment consistency KeyF1: calculates fragment-level F1 for the set of key fragments K(y⋆,p) in the target candidate. When p = "transliteration priority", K focuses more on proper name fragments with consistent syllables / phonemes; when p = "interpretation priority", K focuses more on semantic phrase / term interpretation fragments.

[0091] Semantic consistency This can measure the sentence-level semantic similarity between the output and the target candidate. Using vector cosine similarity and linear normalization, it can be expressed by the following formula (3):

[0092] (3)

[0093] Here, e() is a statement vector encoder (such as bert-embedding).

[0094] proper name consistency The accuracy of the translation (in the case of transliteration) or the correctness of the interpretation (in the case of free translation) of personal names, place names, brand names and terms can be expressed by the following formula (4):

[0095] (4)

[0096] in For target candidate This is an indicator function that takes the value 1 when the candidate input y contains the target proper name e, and 0 otherwise.

[0097] Based on the above processing, the first predicted text can be evaluated in multiple dimensions, thereby obtaining its corresponding first evaluation information, which can then be used to fine-tune the model.

[0098] It should be noted that the processing procedure for the second evaluation information corresponding to the second predicted text can be referred to the processing procedure for the first evaluation information corresponding to the first predicted text in the above embodiments, and will not be elaborated on in this embodiment.

[0099] In summary, through a multi-level matching mechanism at the character unit, segment, and keyword levels, the grammatical structure, semantic coherence, and core elements of speech recognition output are accurately analyzed. This mechanism achieves in-depth refinement of evaluation dimensions, effectively avoiding the coarse-grained coverage of text features by traditional methods, significantly improving the detail and reliability of evaluation results, providing high-value feedback for model optimization, and greatly enhancing the output accuracy and user-perceived fluency of the voice interaction system.

[0100] Step S106: Determine the reward loss value based on the first evaluation information, and determine the preference loss value based on the first evaluation information and the second evaluation information.

[0101] Step S108: Fine-tune the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used to perform transliteration recognition and / or semantic translation recognition of audio.

[0102] Specifically, after obtaining the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text, in order to improve the prediction ability of the speech recognition model, a reward loss value can be determined based on the first evaluation information, and a preference loss value can be determined based on the first evaluation information and the second evaluation information. Subsequently, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model for audio transliteration and / or paraphrasing recognition. This can effectively improve the model's proper noun fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0103] Specifically, the reward loss value refers to the loss value used to optimize the model's single-sequence policy based on the output of the reward model, while the preference loss value refers to the loss value introduced to address the high variance and convergence instability during reinforcement learning training, ensuring model convergence stability. Correspondingly, the target speech recognition model refers to the speech recognition model obtained after fine-tuning the existing speech recognition model. It can determine whether to perform transliteration or semantic translation based on the input audio samples, thereby obtaining recognition results that conform to the speech recognition scenario and meet the speech recognition needs of different scenarios.

[0104] In one or more embodiments provided in this example, the step of fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain the target speech recognition model includes:

[0105] The speech recognition loss value corresponding to the speech recognition model is calculated based on the first predicted text and the target sample; the speech recognition loss value, the reward loss value and the preference loss value are fused to obtain the target loss value; the speech recognition model is fine-tuned based on the target loss value until a target speech recognition model that meets the training stopping condition is obtained, wherein the speech recognition model is a pre-trained speech recognition model.

[0106] Specifically, the speech recognition loss value refers to the loss value calculated using the cross-entropy loss function for speech recognition. Correspondingly, the target loss value refers to the loss value obtained by fusing the speech recognition loss value, reward loss value, and preference loss value. It can be determined by weighted average calculation or other statistical methods; this embodiment does not impose any limitations. The training stopping condition refers to the condition for stopping the training of the speech recognition model. It can be the number of iterations, a loss value comparison condition, or a validation set verification condition; this embodiment does not impose any limitations.

[0107] Based on this, when fine-tuning the model, in order to improve the prediction accuracy of the model, the speech recognition loss value corresponding to the speech recognition model can be calculated based on the first predicted text and the target sample. At this time, the speech recognition loss value, reward loss value and preference loss value can be fused to obtain the target loss value. Then, the speech recognition model can be fine-tuned based on the target loss value until the target speech recognition model that meets the training stopping condition is obtained. Here, the speech recognition model is the pre-trained speech recognition model.

[0108] In practical applications, the optimization objective of a single-sequence strategy can be to maximize the expected reward (reward loss value), which can be achieved by the following formula (5):

[0109] (5)

[0110] The corresponding policy gradient is shown in formula (6):

[0111] (6)

[0112] Here, b(x) is the baseline function used to reduce variance.

[0113] The calculation of the preference loss value can be understood as the pairwise preference robustness loss, which can solve the problems of high variance and unstable convergence in reinforcement learning training. By introducing pairwise preference optimization: under the same audio x, sample two outputs y+ (first predicted text) and y− (second predicted text), and use the reward difference for ranking learning, which can be expressed by the following formula (7):

[0114] (7)

[0115] Where σ is the sigmoid function. This mechanism ensures that sequences with higher rewards receive larger update weights during training, thereby stabilizing the model's preference learning process.

[0116] Based on this, the fusion of loss values ​​can be expressed by the following formula (8):

[0117] (8)

[0118] in It uses conventional speech recognition loss (such as cross-entropy) to ensure the model's basic recognition capability.

[0119] The speech recognition model trained in this way can select either transliteration or semantic translation when processing any audio, based on the corresponding scene information, thereby improving the accuracy of the recognition results for subsequent business services.

[0120] In summary, an adaptive optimization mechanism is constructed by dynamically fusing speech recognition loss, reward loss, and preference loss. This mechanism effectively solves the performance imbalance problem caused by fixed loss weights in traditional fine-tuning, significantly improves model training efficiency and convergence speed, enables pre-trained speech recognition models to quickly achieve accurate optimization in complex speech scenarios, and greatly enhances the system's response speed and output reliability in real-time interaction, providing a more efficient and robust model deployment foundation for intelligent voice services.

[0121] In one or more embodiments provided in this example, after the step of fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain the target speech recognition model is executed, the method further includes:

[0122] Obtain task feedback information of the target speech recognition model performing the speech recognition task; determine optimized sample audio based on the task feedback information, and construct optimized transliterated text and optimized semantic text corresponding to the optimized sample audio; optimize the target speech recognition model using the optimized sample audio, the optimized transliterated text and the optimized semantic text, and perform the speech recognition task based on the optimized target speech recognition model.

[0123] Specifically, task feedback information refers to the feedback information collected when the target speech recognition model performs a speech recognition task. The speech recognition task specifically refers to the task performed by the target speech recognition model after deployment in a specific business scenario. Correspondingly, optimized sample audio refers to sample audio used for model optimization. Optimized transliterated text and optimized semantic text are the semantic and transliterated texts corresponding to the optimized sample audio. Their related descriptions can be found in the descriptions of transliterated and semantic texts in the above embodiments, and will not be elaborated upon further in this embodiment.

[0124] Therefore, in order to optimize the model after deployment by incorporating user feedback, after obtaining the task feedback information of the target speech recognition model performing the speech recognition task, the optimized sample audio can be determined based on the task feedback information. At this time, the optimized transliterated text and optimized semantic text corresponding to the optimized sample audio can be constructed. Then, the optimized sample audio, optimized transliterated text, and optimized semantic text can be used to optimize the target speech recognition model, and the speech recognition task can be performed based on the optimized target speech recognition model.

[0125] In summary, an innovative task-feedback-driven closed-loop optimization mechanism is constructed. By dynamically generating highly relevant optimization samples through real-time analysis of actual interaction feedback, this mechanism effectively solves the performance drift problem of the model in real-world scenarios. This mechanism significantly improves the adaptability and task completion accuracy of speech recognition in complex environments, reduces reliance on manual tuning, and enables continuous evolution of model capabilities. The optimized system exhibits higher stability and user satisfaction in practical applications, providing an iterative and highly reliable optimization path for intelligent voice services.

[0126] See Figure 2 The schematic diagram illustrates the speech recognition model training method based on transliteration and semantic translation preference alignment provided in this embodiment. To introduce a transliteration and semantic translation alignment mechanism during the fine-tuning of the speech recognition model, enabling the trained model to output recognition results that conform to user preferences based on scenario requirements, the sample audio is first processed according to transliteration and semantic translation strategies to obtain transliterated and semantically translated texts. At this point, a target sample is constructed based on the sample audio, transliterated text, semantically translated text, and the corresponding preference annotation information. Then, the speech recognition model is used to process the target sample to obtain a first predicted text and a second predicted text. The reward from the speech recognition model is then calculated. The model performs multi-dimensional evaluations on the first and second predicted texts to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text. Based on this, a reward loss value can be determined according to the first evaluation information, and a preference loss value can be determined according to the first and second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration and / or paraphrasing of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0127] See Figure 3 , Figure 3 A flowchart of a speech recognition method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0128] Step S302: Obtain the audio to be processed and determine the preference tag corresponding to the audio to be processed.

[0129] Step S304: The target speech recognition model is used to process the audio to be processed according to the preference tags to obtain multiple candidate texts, wherein the target speech recognition model is obtained by the above method.

[0130] Step S306: Determine the target text corresponding to the audio to be processed from the plurality of candidate texts according to the preference tag.

[0131] Specifically, the audio to be processed refers to the audio processed in the business scenario where the target speech recognition model is deployed, such as audio from training scenarios, audio recognized in customer service dialogue scenarios, and audio of voice commands submitted through interactive instructions. Preference labels refer to the labels determined by transliteration or semantic translation during the recognition processing of the audio to be processed. Multiple candidate texts refer to the multiple texts obtained after the target speech recognition model recognizes the text to be processed according to the preference labels; the target text is the text extracted from these multiple preference texts for subsequent use.

[0132] Based on this, after obtaining the audio to be processed, the preference label corresponding to the audio to be processed can be determined first. Then, the target speech recognition model can be used to process the audio to be processed according to the preference label, thereby obtaining multiple candidate texts. After that, the target text corresponding to the audio to be processed can be determined from the multiple candidate texts according to the preference label, so that it can be used later.

[0133] In practical applications, users or the system specify transliteration, paraphrasing, or adaptive mode using preference control labels at the start of inference. For adaptive mode, the system automatically infers whether transliteration or paraphrasing should be preferred based on scene information or a lightweight discriminator. Furthermore, the model can generate multiple candidate results during decoding, which may differ in semantic expression and pronunciation fidelity. The system can then reorder or filter these candidate results according to the specified preference, prioritizing the output of results consistent with the preference, thus ensuring that the recognized content meets business requirements. This ensures that after preference control processing, the system outputs recognition results that match the scene requirements, improving practicality and reliability across languages ​​and scenarios. For example, when a user interacts with a smart device, submitting control commands via voice, a target speech recognition model can be used to recognize the user's voice commands. Since the control of the smart device needs to be completed according to the user's intention (e.g., a robot vacuum cleaner sweeping), paraphrasing recognition can be used to recognize the voice control commands. The resulting recognition result is then used to control the smart device to perform the corresponding task.

[0134] In other words, the model training and application processing provided in this embodiment mainly consist of the following three steps:

[0135] (1) Constructing a transliteration-translation preference dataset: By collecting voice input and its corresponding transliteration and translation candidates, and combining manual or semi-automatic annotation of preference labels, training samples that can reflect actual business needs are formed, providing high-quality supervision signals for model learning.

[0136] (2) Preference alignment fine-tuning based on reinforcement learning: On the basis of the pre-trained speech recognition model, a preference conditionalization mechanism is introduced, and a reward function and a pairwise preference optimization method are designed. Through reinforcement learning training, the model can not only maintain the basic accuracy of speech recognition, but also explicitly learn the preference selection logic between transliteration and translation.

[0137] (3) Controllable generation of preferences during the inference stage: In practical applications, users or the system can specify preference patterns (transliteration, translation, or adaptive). After generating candidate results, the model filters and sorts them according to preference information and outputs the final recognition results that meet the needs of the scenario.

[0138] Through the above steps, the speech recognition model achieves controllable output in cross-language and industry terminology scenarios, significantly improving the fidelity of proper nouns and semantic consistency, and meeting the needs of various business scenarios such as training, customer service, and script quality inspection.

[0139] In summary, to introduce a transliteration and semantic alignment mechanism during the fine-tuning of the speech recognition model, enabling the trained speech recognition model to output recognition results that conform to user preferences based on scenario requirements, the sample audio can be processed first according to transliteration and semantic translation strategies to obtain transliterated and semantically translated texts. At this point, target samples can be constructed based on the sample audio, transliterated text, semantically translated text, and the corresponding preference annotation information of the sample audio. Then, the speech recognition model can be used to process the target samples to obtain the first and second predicted texts. The reward model of the speech recognition model is used to evaluate the first and second predicted texts in multiple dimensions to obtain the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text. Based on this, the reward loss value can be determined based on the first evaluation information, and the preference loss value can be determined based on the first and second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration and / or semantic translation recognition of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0140] Corresponding to the above method embodiments, this specification also provides an embodiment of a speech recognition model training device based on transliteration-translation preference alignment. Figure 4 This diagram illustrates a structural schematic of a speech recognition model training device based on transliteration-translation preference alignment, according to one embodiment of this specification. Figure 4 As shown, the device includes:

[0141] Processing module 402 is configured to process sample audio according to transliteration strategy and semantic translation strategy to obtain transliterated text and semantic translation text, and construct target sample according to the sample audio, the transliterated text, the semantic translation text and the preference annotation information corresponding to the sample audio;

[0142] Evaluation module 404 is configured to process the target sample using a speech recognition model to obtain a first predicted text and a second predicted text, and to perform multi-dimensional evaluation of the first predicted text and the second predicted text based on a reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text.

[0143] The determination module 406 is configured to determine a reward loss value based on the first evaluation information, and to determine a preference loss value based on the first evaluation information and the second evaluation information;

[0144] The fine-tuning module 408 is configured to fine-tune the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used to perform transliteration recognition and / or paraphrase recognition of audio.

[0145] In an optional embodiment, processing the sample audio according to a transliteration strategy and a semantic translation strategy to obtain transliterated text and semantic translation text includes:

[0146] Acquire sample audio; input the sample audio into the speech recognition model for processing according to the transliteration strategy, or process the sample audio using a preset dictionary and pronunciation rules, and determine the transliterated text based on the processing result; process the sample audio using parallel corpus, translation model or semantic rewriting algorithm according to the semantic translation strategy, and determine the semantic text based on the processing result.

[0147] In an optional embodiment, constructing the target sample based on the sample audio, the transliterated text, the translated text, and the preference annotation information corresponding to the sample audio includes:

[0148] Preference data pairs are constructed based on the transliterated text and the translated text, and the preference data pairs are labeled according to the set labeling rules to obtain preference labeling information; triple samples are constructed based on the preference data pairs, the sample audio, and the preference labeling information, and the triple samples are used as target samples.

[0149] In an optional embodiment, the first predicted text is evaluated in multiple dimensions based on the reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text, including:

[0150] A preferred text is determined from the target sample, and the preferred text and the first predicted text are input into the reward model of the speech recognition model for processing. Based on the processing result, preference matching evaluation information corresponding to the preference matching dimension, semantic similarity evaluation information corresponding to the semantic similarity dimension, and keyword matching evaluation information corresponding to the keyword matching dimension are determined. The preference matching evaluation information, the semantic similarity evaluation information, and the keyword matching evaluation information are fused to obtain the first evaluation information corresponding to the first predicted text.

[0151] In an optional embodiment, determining the preference matching evaluation information corresponding to the preference matching dimension includes:

[0152] The reward model is used to perform character unit matching and text segment matching on the preferred text and the first predicted text. Based on the character unit matching results and text segment matching results, preference matching evaluation information for the corresponding preference matching dimension is determined. Specifically, determining the semantic similarity evaluation information for the corresponding semantic similarity dimension includes: using the reward model to calculate the similarity between the preferred text and the first predicted text to obtain semantic similarity evaluation information for the corresponding semantic similarity dimension. Furthermore, determining the keyword matching evaluation information for the corresponding keyword matching dimension includes: using the reward model to extract keywords from the first predicted text, matching the keywords with benchmark keywords, and determining keyword matching evaluation information for the corresponding keyword matching dimension based on the matching results.

[0153] In an optional embodiment, fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain the target speech recognition model includes:

[0154] The speech recognition loss value corresponding to the speech recognition model is calculated based on the first predicted text and the target sample; the speech recognition loss value, the reward loss value and the preference loss value are fused to obtain the target loss value; the speech recognition model is fine-tuned based on the target loss value until a target speech recognition model that meets the training stopping condition is obtained, wherein the speech recognition model is a pre-trained speech recognition model.

[0155] In an optional embodiment, after the step of fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain the target speech recognition model is performed, the method further includes:

[0156] Obtain task feedback information of the target speech recognition model performing the speech recognition task; determine optimized sample audio based on the task feedback information, and construct optimized transliterated text and optimized semantic text corresponding to the optimized sample audio; optimize the target speech recognition model using the optimized sample audio, the optimized transliterated text and the optimized semantic text, and perform the speech recognition task based on the optimized target speech recognition model.

[0157] The speech recognition model training device based on transliteration and semantic translation preference alignment provided in this embodiment introduces a transliteration and semantic translation alignment mechanism during the fine-tuning of the speech recognition model. This allows the trained speech recognition model to output recognition results that conform to user preferences according to scenario requirements. First, sample audio is processed according to transliteration and semantic translation strategies to obtain transliterated and semantically translated texts. At this point, target samples are constructed based on the sample audio, transliterated text, semantically translated text, and the corresponding preference annotation information of the sample audio. Then, the speech recognition model is used to process the target samples to obtain a first predicted text and a second predicted text. The reward model based on the speech recognition model then... The first and second predicted texts are evaluated from multiple dimensions to obtain the first evaluation information corresponding to the first predicted text and the second evaluation information corresponding to the second predicted text. Based on this, the reward loss value can be determined according to the first evaluation information, and the preference loss value can be determined according to the first and second evaluation information. Based on this, the speech recognition model can be fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model that can be used for transliteration recognition and / or paraphrasing recognition of audio. This can effectively improve the model's proper name fidelity and semantic consistency in cross-language and cross-domain scenarios, thereby meeting the speech recognition needs of different application scenarios.

[0158] The above is an illustrative scheme of a speech recognition model training device based on transliteration-translation preference alignment according to this embodiment. It should be noted that the technical solution of this speech recognition model training device based on transliteration-translation preference alignment belongs to the same concept as the technical solution of the speech recognition model training method based on transliteration-translation preference alignment described above. Details not described in detail in the technical solution of the speech recognition model training device based on transliteration-translation preference alignment can be found in the description of the technical solution of the speech recognition model training method based on transliteration-translation preference alignment described above.

[0159] Corresponding to the above method embodiments, this specification also provides embodiments of a speech recognition device. Figure 5 A schematic diagram of the structure of a speech recognition device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0160] The acquisition module 502 is configured to acquire the audio to be processed and determine the preference tag corresponding to the audio to be processed;

[0161] The processing module 504 is configured to process the audio to be processed according to the preference tags using a target speech recognition model to obtain multiple candidate texts, wherein the target speech recognition model is obtained by the above method;

[0162] The determination module 506 is configured to determine the target text corresponding to the audio to be processed from among the plurality of candidate texts based on the preference label.

[0163] The above is an illustrative scheme of a speech recognition device according to this embodiment. It should be noted that the technical solution of this speech recognition device and the technical solution of the speech recognition method described above belong to the same concept. For details not described in detail in the technical solution of the speech recognition device, please refer to the description of the technical solution of the speech recognition method described above.

[0164] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0165] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0166] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0167] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0168] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration-translation preference alignment.

[0169] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the above-described technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences. For details not described in detail in the technical solution of the computing device, please refer to the description of the above-described technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences.

[0170] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration-translation preference alignment.

[0171] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the above-described technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences. For details not described in detail in the technical solution of the storage medium, please refer to the description of the above-described technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences.

[0172] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described speech recognition model training method or speech recognition method based on transliteration-translation preference alignment.

[0173] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the above-mentioned technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences. For details not described in detail in the technical solution of the computer program product, please refer to the description of the above-mentioned technical solution of the speech recognition model training method or speech recognition method based on the alignment of transliteration and translation preferences.

[0174] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0175] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0176] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0178] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A method for training a speech recognition model based on transliteration-interpretation preference alignment, characterized in that, The method comprises the following steps: According to the transliteration strategy and the interpretation strategy, the sample audio is processed to obtain transliterated text and interpreted text, wherein the transliteration strategy is used to retain the pronunciation features in the sample audio, and the interpretation strategy is used to obtain the interpreted text containing semantic meaning. According to the sample audio, the transliterated text, the interpreted text and the preference annotation information corresponding to the sample audio, a target sample is constructed; The target sample is processed by using a speech recognition model to obtain a first predicted text and a second predicted text. The first predicted text and the second predicted text are evaluated in multiple dimensions based on a reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text. The first evaluation information corresponding to the first predicted text is obtained by evaluating the first predicted text in multiple dimensions based on the reward model of the speech recognition model, which includes: determining a preferred text in the target sample, and inputting the preferred text and the first predicted text into the reward model of the speech recognition model for processing; determining preference matching evaluation information corresponding to the preference matching dimension, semantic similarity evaluation information corresponding to the semantic similarity dimension, and keyword matching evaluation information corresponding to the keyword matching dimension according to the processing result; and fusing the preference matching evaluation information, the semantic similarity evaluation information and the keyword matching evaluation information to obtain the first evaluation information corresponding to the first predicted text; A reward loss value is determined according to the first evaluation information, and a preference loss value is determined according to the first evaluation information and the second evaluation information; The speech recognition model is fine-tuned based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used for transliteration recognition and / or interpretation recognition of audio.

2. The method of claim 1, wherein, The method comprises the following steps: Obtaining a sample audio; According to the transliteration strategy, the sample audio is input into the speech recognition model for processing, or the sample audio is processed by using a preset dictionary and pronunciation rules, and transliterated text is determined according to the processing result; According to the interpretation strategy, the sample audio is processed by using parallel corpus, translation model or semantic rewriting algorithm, and interpreted text is determined according to the processing result.

3. The method of claim 1, wherein, The method comprises the following steps: According to the transliterated text and the interpreted text, preference data pairs are constructed, and the preference data pairs are annotated according to a set annotation rule to obtain preference annotation information; Based on the preference data pairs, the sample audio and the preference annotation information, a triple sample is constructed, and the triple sample is used as a target sample.

4. The method of claim 1, wherein, The determination of the preference matching evaluation information corresponding to the preference matching dimension comprises: The reward model is used for word unit matching and text segment matching of the preference text and the first predicted text, and preference matching evaluation information corresponding to a preference matching dimension is determined according to a word unit matching result and a text segment matching result; The determination of the semantic similarity evaluation information corresponding to the semantic similarity dimension comprises: The reward model is used for similarity calculation of the preference text and the first predicted text, and the semantic similarity evaluation information corresponding to the semantic similarity dimension is obtained; The determination of the keyword matching evaluation information corresponding to the keyword matching dimension comprises: The reward model is used for extracting keywords in the first predicted text, matching the keywords with reference keywords, and determining the keyword matching evaluation information corresponding to the keyword matching dimension according to a matching result.

5. The method of claim 1, wherein, The fine-tuning of the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model comprises: calculating a speech recognition loss value corresponding to the speech recognition model according to the first predicted text and the target sample; fusing the speech recognition loss value, the reward loss value and the preference loss value to obtain a target loss value; fine-tuning the speech recognition model based on the target loss value until a target speech recognition model that meets a training stop condition is obtained, wherein the speech recognition model is a pre-trained speech recognition model.

6. The method according to any one of claims 1 to 5, characterized in that, After the step of fine-tuning the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, the method further comprises: obtaining task feedback information of the target speech recognition model performing a speech recognition task; determining an optimized sample audio according to the task feedback information, and constructing an optimized transliterated text and an optimized interpreted text corresponding to the optimized sample audio; optimizing the target speech recognition model using the optimized sample audio, the optimized transliterated text and the optimized interpreted text, and performing the speech recognition task based on the optimized target speech recognition model.

7. A voice recognition method characterized by, comprises: obtaining a to-be-processed audio and determining a preference label corresponding to the to-be-processed audio; processing the to-be-processed audio according to the preference label using a target speech recognition model to obtain a plurality of candidate texts, wherein the target speech recognition model is obtained by the method of any one of claims 1 to 6; determining a target text corresponding to the to-be-processed audio from the plurality of candidate texts according to the preference label.

8. A speech recognition model training device based on transliteration-translation preference alignment, characterized in that, comprises: a processing module configured to process a sample audio according to a transliteration strategy and an interpretation strategy to obtain a transliterated text and an interpreted text, wherein the transliteration strategy is used to retain pronunciation features in the sample audio, and the interpretation strategy is used to generate an interpreted text containing semantic meaning, and a target sample is constructed according to the sample audio, the transliterated text, the interpreted text and preference annotation information corresponding to the sample audio; The evaluation module is configured to process the target sample by using a speech recognition model to obtain a first predicted text and a second predicted text, perform multi-dimensional evaluation on the first predicted text and the second predicted text based on a reward model of the speech recognition model to obtain first evaluation information corresponding to the first predicted text and second evaluation information corresponding to the second predicted text; wherein performing multi-dimensional evaluation on the first predicted text based on the reward model of the speech recognition model to obtain the first evaluation information corresponding to the first predicted text comprises: determining a preferred text in the target sample, and inputting the preferred text and the first predicted text into the reward model of the speech recognition model for processing; determining, according to a processing result, preferred matching evaluation information corresponding to a preferred matching dimension, semantic similarity evaluation information corresponding to a semantic similarity dimension, and keyword matching evaluation information corresponding to a keyword matching dimension; and fusing the preferred matching evaluation information, the semantic similarity evaluation information, and the keyword matching evaluation information to obtain the first evaluation information corresponding to the first predicted text; The determination module is configured to determine a reward loss value according to the first evaluation information, and determine a preference loss value according to the first evaluation information and the second evaluation information; The fine-tuning module is configured to fine-tune the speech recognition model based on the reward loss value and the preference loss value to obtain a target speech recognition model, wherein the target speech recognition model is used for phonetic translation recognition and / or semantic translation recognition on audio.

9. A speech recognition apparatus characterized by comprising: The method comprises: The acquisition module is configured to acquire audio to be processed, and determine a preferred label corresponding to the audio to be processed; The processing module is configured to process the audio to be processed according to the preferred label by using a target speech recognition model to obtain a plurality of candidate texts, wherein the target speech recognition model is obtained by the method of any one of claims 1 to 6; The determination module is configured to determine a target text corresponding to the audio to be processed from the plurality of candidate texts according to the preferred label.

10. A computing device, comprising: The method comprises: A memory and a processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, realize the steps of the method of any one of claims 1 to 7.

11. A computer readable storage medium, characterized in that, The memory has stored computer executable instructions, and the computer executable instructions, when executed by the processor, realize the steps of the method of any one of claims 1 to 7.

12. A computer program product, characterised in that, The computer program or instructions comprise steps of the method of any one of claims 1 to 7.

Images