Training method and device of heterogeneous language model, equipment and storage medium

Abstract

Embodiments of the present application provide a method, device and storage medium for training a heterogeneous language model. The method comprises: obtaining a voice training sample set; training a first initial network model and a second initial network model using the voice training sample set to obtain at least two first network models and at least two second network models; the first network model and the second network model have different structures, the first network model is used to process an input pinyin sequence to obtain at least one character sequence corresponding to the pinyin sequence, and the second network model is used to determine a target character sequence corresponding to the pinyin sequence from the at least one character sequence; and determining the heterogeneous language model according to the at least two first network models and the at least two second network models. The method, device and storage medium for training the heterogeneous language model provided by the embodiments of the present application are used to improve the accuracy of the language model.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a training method, apparatus, device, and storage medium for a heterogeneous language model. Background Technology

[0002] Automatic Speech Recognition (ASR) systems are systems that recognize speech to obtain corresponding text. An ASR system includes an acoustic model (AM) and a language model (LM). AM is used to obtain the corresponding pinyin based on the speech. LM is an isomorphic language model, which is used to obtain the text based on the pinyin.

[0003] In related technologies, a speech dataset is typically used to train an initial language model (LM) to obtain the LM in the aforementioned ASR system. However, due to information privacy restrictions, the number of speech samples in the speech dataset is relatively small, and the isomorphic language model is either an n-gram model or a deep neural network (DNN), which results in a low accuracy of the trained LM (i.e., the accuracy of the text obtained through the LM is low). Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for training heterogeneous language models to solve the problem of low accuracy in language models.

[0005] In a first aspect, embodiments of this application provide a method for training a heterogeneous language model, comprising:

[0006] Obtain the speech training sample set;

[0007] Using a speech training sample set, a first initial network model and a second initial network model are trained to obtain at least two first network models and at least two second network models. The first network models and the second network models have different structures. The first network model is used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network model is used to determine the target text sequence corresponding to the pinyin sequence from at least one text sequence.

[0008] Determine the heterogeneous language model based on at least two first network models and at least two second network models.

[0009] Optionally, a heterogeneous language model is determined based on at least two first network models and at least two second network models, including:

[0010] Based on at least two first network models and at least two second network models, determine at least four first language models;

[0011] Obtain the voice verification sample set;

[0012] Based on the speech verification sample set, determine the error rate of each first language model;

[0013] Determine heterogeneous language models based on at least four first language models and error rates.

[0014] Optionally, at least four first language models are determined based on at least two first network models and at least two second network models, including:

[0015] At least two first network models and at least two second network models are randomly combined to obtain at least four first language models.

[0016] Optionally, at least four first language models are determined based on at least two first network models and at least two second network models, including:

[0017] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0018] For each second network model, the model parameters of the second network model are converted into binary to obtain a second initial parameter sequence; the second initial parameter sequence is cross-processed and mutated to obtain at least two second intermediate parameter sequences; the model parameters of the second network model are replaced with the model parameters corresponding to at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model.

[0019] Randomly combine at least two third network models corresponding to each of the at least two first network models and at least two fifth network models corresponding to each of the at least two second network models to obtain at least four first language models.

[0020] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples; for each of the at least four first language models, the first language model includes a first network model and a second network model;

[0021] Based on the speech verification sample set, determine the error rate of the first language model, including:

[0022] Multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each pinyin verification sample.

[0023] The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0024] Optionally, a heterogeneous language model is determined based on at least four first language models and error rates, including:

[0025] Determine if at least one of the four first language models has an error rate lower than a preset value;

[0026] If so, the first language model with an error rate lower than the preset value is identified as a heterogeneous language model;

[0027] If not, obtain the first model parameter sequence corresponding to the model parameters of a preset number of first language models from at least four first language models, so as to obtain at least one second language model corresponding to the preset number of first language models, and determine the heterogeneous language model based on the multiple second language models and the error rate of each second language model.

[0028] Optionally, a heterogeneous language model is determined based on at least two first network models and at least two second network models, including:

[0029] Based on at least two first network models, determine a first target network model, and based on at least two second network models, determine a second target network model;

[0030] The first target network model and the second target network model are identified as heterogeneous language models.

[0031] Optionally, a first target network model is determined based on at least two first network models, including:

[0032] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0033] Based on multiple third network models, the first target network model is determined.

[0034] Optionally, a first target network model is determined based on multiple third network models, including:

[0035] Obtain the voice verification sample set;

[0036] Based on the voice verification sample set, determine the error rate of each third network model;

[0037] The first target network model is determined based on multiple third network models and error rates.

[0038] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples;

[0039] Based on the speech verification sample set, determine the error rate of the third network model, including:

[0040] Multiple pinyin verification samples are processed sequentially through the third network model and any second network model to obtain the text output results corresponding to each pinyin verification sample.

[0041] The error rate of the third network model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0042] Optionally, a first target network model is determined based on multiple third network models and error rates, including:

[0043] Determine if there is a third network model among multiple third network models with an error rate lower than a preset value;

[0044] If so, the third network model with an error rate less than the preset value will be identified as the first target network model;

[0045] If not, obtain the first initial parameter sequence corresponding to the model parameters of a preset number of third network models among the multiple third network models, so as to obtain at least one fourth network model corresponding to each of the preset number of third network models, and determine the first target network model based on the multiple fourth network models and the error rate of each fourth network model.

[0046] Optionally, a first target network model is determined based on at least two first network models, and a second target network model is determined based on at least two second network models, including:

[0047] The first weights of each first network model and the second weights of each second network model are generated using a weight generation model.

[0048] Based on the first weight, the model parameters of each first network model are fused to obtain the first target model parameters; based on the second weight, the model parameters of each second network model are fused to obtain the second target model parameters.

[0049] The model parameters of the first network model are replaced with the parameters of the first target model to obtain the first target network model; the model parameters of the second network model are replaced with the parameters of the second target model to obtain the second target network model.

[0050] Optionally, the method further includes:

[0051] Obtain the voice verification sample set;

[0052] Based on the speech verification sample set, determine the error rate of the heterogeneous language model, and determine the reward value based on the error rate;

[0053] Based on the reward value, update the model parameters of the weight generation model, regenerate the first weights of each first network model and the second weights of each second network model to obtain a new heterogeneous language model.

[0054] Optionally, the speech training sample set includes at least two sample subsets; using the speech training sample set, the first initial network model and the second initial network model are trained to obtain at least two first network models and at least two second network models, including:

[0055] The first initial network model is trained using at least two sample subsets respectively, to obtain the first network model corresponding to each of the at least two sample subsets;

[0056] The second initial network model is trained using at least two sample subsets respectively, to obtain the second network model corresponding to each of the at least two sample subsets;

[0057] The first network model corresponding to each of at least two sample subsets is determined as at least two first network models; the second network model corresponding to each of at least two sample subsets is determined as at least two second network models.

[0058] Secondly, embodiments of this application provide a training apparatus for a heterogeneous language model, comprising: a processing module; the processing module is used for:

[0059] Obtain the speech training sample set;

[0060] Using a speech training sample set, a first initial network model and a second initial network model are trained to obtain at least two first network models and at least two second network models. The first network models and the second network models have different structures. The first network model is used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network model is used to determine the target text sequence corresponding to the pinyin sequence from at least one text sequence.

[0061] Determine the heterogeneous language model based on at least two first network models and at least two second network models.

[0062] Optionally, the processing module is specifically used for:

[0063] Based on at least two first network models and at least two second network models, determine at least four first language models;

[0064] Obtain the voice verification sample set;

[0065] Based on the speech verification sample set, determine the error rate of each first language model;

[0066] Determine heterogeneous language models based on at least four first language models and error rates.

[0067] Optionally, the processing module is specifically used for:

[0068] At least two first network models and at least two second network models are randomly combined to obtain at least four first language models.

[0069] Optionally, the processing module is specifically used for:

[0070] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0071] For each second network model, the model parameters of the second network model are converted into binary to obtain a second initial parameter sequence; the second initial parameter sequence is cross-processed and mutated to obtain at least two second intermediate parameter sequences; the model parameters of the second network model are replaced with the model parameters corresponding to at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model.

[0072] Randomly combine at least two third network models corresponding to each of the at least two first network models and at least two fifth network models corresponding to each of the at least two second network models to obtain at least four first language models.

[0073] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples; for each of the at least four first language models, the first language model includes a first network model and a second network model;

[0074] The processing module is specifically used for:

[0075] Multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each pinyin verification sample.

[0076] The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0077] Optionally, the processing module is specifically used for:

[0078] Determine if at least one of the four first language models has an error rate lower than a preset value;

[0079] If so, the first language model with an error rate lower than the preset value is identified as a heterogeneous language model;

[0080] If not, obtain the first model parameter sequence corresponding to the model parameters of a preset number of first language models from at least four first language models, so as to obtain at least one second language model corresponding to the preset number of first language models, and determine the heterogeneous language model based on the multiple second language models and the error rate of each second language model.

[0081] Optionally, the processing module is specifically used for:

[0082] Based on at least two first network models, determine a first target network model, and based on at least two second network models, determine a second target network model;

[0083] The first target network model and the second target network model are identified as heterogeneous language models.

[0084] Optionally, the processing module is specifically used for:

[0085] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0086] Based on multiple third network models, the first target network model is determined.

[0087] Optionally, the processing module is specifically used for:

[0088] Obtain the voice verification sample set;

[0089] Based on the voice verification sample set, determine the error rate of each third network model;

[0090] The first target network model is determined based on multiple third network models and error rates.

[0091] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples;

[0092] The processing module is specifically used for:

[0093] Multiple pinyin verification samples are processed sequentially through the third network model and any second network model to obtain the text output results corresponding to each pinyin verification sample.

[0094] The error rate of the third network model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0095] Optionally, the processing module is specifically used for:

[0096] Determine if there is a third network model among multiple third network models with an error rate lower than a preset value;

[0097] If so, the third network model with an error rate less than the preset value will be identified as the first target network model;

[0098] If not, obtain the first initial parameter sequence corresponding to the model parameters of a preset number of third network models among the multiple third network models, so as to obtain at least one fourth network model corresponding to each of the preset number of third network models, and determine the first target network model based on the multiple fourth network models and the error rate of each fourth network model.

[0099] Optionally, the processing module is specifically used for:

[0100] The first weights of each first network model and the second weights of each second network model are generated using a weight generation model.

[0101] Based on the first weight, the model parameters of each first network model are fused to obtain the first target model parameters; based on the second weight, the model parameters of each second network model are fused to obtain the second target model parameters.

[0102] The model parameters of the first network model are replaced with the parameters of the first target model to obtain the first target network model; the model parameters of the second network model are replaced with the parameters of the second target model to obtain the second target network model.

[0103] Optionally, the processing module is also used for:

[0104] Obtain the voice verification sample set;

[0105] Based on the speech verification sample set, determine the error rate of the heterogeneous language model, and determine the reward value based on the error rate;

[0106] Based on the reward value, update the model parameters of the weight generation model, regenerate the first weights of each first network model and the second weights of each second network model to obtain a new heterogeneous language model.

[0107] Optionally, the speech training sample set includes at least two sample subsets; the processing module is specifically used for:

[0108] The first initial network model is trained using at least two sample subsets respectively, to obtain the first network model corresponding to each of the at least two sample subsets;

[0109] The second initial network model is trained using at least two sample subsets respectively, to obtain the second network model corresponding to each of the at least two sample subsets;

[0110] The first network model corresponding to each of at least two sample subsets is determined as at least two first network models; the second network model corresponding to each of at least two sample subsets is determined as at least two second network models.

[0111] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0112] The memory stores instructions that the computer executes;

[0113] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0114] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0115] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0116] This application provides a method, apparatus, device, and storage medium for training a heterogeneous language model. The method includes: acquiring a speech training sample set; using the speech training sample set to train a first initial network model and a second initial network model to obtain at least two first network models and at least two second network models; the first network models and the second network models have different structures, the first network models are used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network models are used to determine the target text sequence corresponding to the pinyin sequence from the at least one text sequence; and determining a heterogeneous language model based on the at least two first network models and the at least two second network models. Attached Figure Description

[0117] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0118] Figure 1 A schematic diagram illustrating an application scenario of the training method for heterogeneous language models provided in this application embodiment;

[0119] Figure 2 A flowchart illustrating a training method for a heterogeneous language model provided in an embodiment of this application;

[0120] Figure 3 A flowchart illustrating a method for determining a heterogeneous language model, as provided in this application embodiment;

[0121] Figure 4 A flowchart illustrating another method for determining heterogeneous language models provided in this application embodiment;

[0122] Figure 5 A flowchart illustrating another method for determining a heterogeneous language model provided in this application embodiment;

[0123] Figure 6 A flowchart illustrating a method for updating heterogeneous language models provided in this application embodiment;

[0124] Figure 7 An architecture diagram of a heterogeneous language model based on GMMA provided for embodiments of this application;

[0125] Figure 8 An architecture diagram of obtaining a heterogeneous language model based on RLGMA provided for embodiments of this application;

[0126] Figure 9 A schematic diagram of the structure of a training device for a heterogeneous language model provided in an embodiment of this application;

[0127] Figure 10 A hardware schematic diagram of the electronic device provided in the embodiments of this application.

[0128] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0129] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0130] In related technologies, due to information privacy restrictions, the number of speech samples in speech datasets is relatively small, and the isomorphic language model is an n-gram model or a deep neural network (DNN). Therefore, the accuracy of the language model obtained after training the initial LM with a speech dataset with a small number of speech samples is usually low.

[0131] In this application, in order to train a language model with a relatively small number of speech samples using a speech dataset, the inventors conceived of first obtaining multiple first network models and multiple second network models by first training the speech sample set. The first network models and the second network models have different structures and functions. Then, based on the multiple first network models and the multiple second network models, a heterogeneous language model is determined, which enables the training of a heterogeneous language model with a relatively small number of speech samples using a speech dataset.

[0132] The following is combined with Figure 1 The application scenarios of the training method for heterogeneous language models provided in the embodiments of this application are described.

[0133] Figure 1 This diagram illustrates the application scenarios to which the training method for the heterogeneous language model provided in this embodiment of the application is applicable. For example... Figure 1 As shown, the application scenario includes: a first initial network model, a second initial network model, multiple first network models, multiple second network models, and a heterogeneous language model.

[0134] Multiple first network models are obtained by training the first initial network model using a speech training sample set.

[0135] Multiple second network models are obtained by training the second initial network model using a speech training sample set.

[0136] The first and second network models have different structures and functions.

[0137] The structure of the first network model is the same as that of the n-gram model. The structure of the second network model is the same as that of Deep Neural Networks (DNNs).

[0138] The function of the first network model is to process the input pinyin sequence to obtain at least one corresponding text sequence. The function of the second network model is to determine the target text sequence corresponding to the pinyin sequence from at least one text sequence.

[0139] The heterogeneous language model is determined based on multiple first network models and multiple second network models.

[0140] exist Figure 1 In the application scenario shown, a heterogeneous language model is determined based on multiple first network models and multiple second network models. This allows for the training of a heterogeneous language model with high accuracy using a speech dataset with a small number of speech samples, thereby improving the accuracy of the language model.

[0141] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0142] Figure 2 This is a flowchart illustrating a training method for a heterogeneous language model provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0143] S201, Obtain the speech training sample set.

[0144] Optionally, the execution subject of the heterogeneous language model training method provided in this application embodiment is an electronic device, or it can be a heterogeneous language model training device set in an electronic device. The heterogeneous language model training device can be implemented by a combination of software and / or hardware.

[0145] Optionally, the speech training sample set includes at least one speech dataset obtained from a pre-defined open website. The pre-defined open website could be, for example, OpenSLR 1. The at least one speech dataset could include, for example, at least one dataset selected from SLR18, SLR33, SLR38, SLR47, SLR62, SLR68, SLR93, etc.

[0146] S202, using the speech training sample set, train the first initial network model and the second initial network model to obtain at least two first network models and at least two second network models.

[0147] The initial network model can be, for example, an initial n-gram model.

[0148] The second initial network model can be, for example, an initial deep neural network (DNN).

[0149] The first network model processes the input pinyin sequence to obtain at least one corresponding text sequence. The pinyin sequence is a pinyin sequence corresponding to a speech training sample in a speech training sample set. The second network model determines the target text sequence corresponding to the pinyin sequence from the at least one text sequence.

[0150] It should be noted that the first network model and the first initial network model have the same structure, and the second network model and the second initial network model have the same structure.

[0151] Alternatively, the first network model and the second network model can be obtained through the following methods 11 and 12.

[0152] Method 11: In the process of training the first initial network model using the speech training sample set, at least two first network models are obtained; in the process of training the second initial network model using the speech training sample set, at least two second network models are obtained.

[0153] Optionally, during the training of the first initial network model using the speech training sample set, the model parameters of the first initial network model are updated according to the speech training sample set, and the network models after the X1, X2, X3... updates are determined as at least two first network models.

[0154] Optionally, X1, X2, X3... can be sequentially increasing positive integers. For example, X1 is 50, X2 is 100, X3 is 300, etc. In this application, the specific values ​​of X1, X2, X3... are not limited.

[0155] In method 11, the method for obtaining at least two second network models is similar to the method for obtaining at least two first network models, and will not be described in detail here.

[0156] Method 12: The speech training sample set includes at least two sample subsets; the first initial network model is trained using at least two sample subsets respectively to obtain the first network model corresponding to each sample subset; the second initial network model is trained using at least two sample subsets respectively to obtain the second network model corresponding to each sample subset; the first network models corresponding to each of the at least two sample subsets are determined as at least two first network models; the second network models corresponding to each of the at least two sample subsets are determined as at least two second network models.

[0157] For example, when at least two sample subsets include SLR18 and SLR33, SLR18 can be used to train the first initial network model to obtain a first network model, and SLR33 can be used to train the first initial network model to obtain another first network model.

[0158] In Method 12, the method for obtaining at least two second network models is similar to the method for obtaining at least two first network models, and will not be described in detail here.

[0159] S203, Determine the heterogeneous language model based on at least two first network models and at least two second network models.

[0160] Alternatively, the heterogeneous language model can be determined using methods 21 and 22 as follows.

[0161] Method 21: Determine at least four first language models based on at least two first network models and at least two second network models; obtain a speech verification sample set; determine the error rate of each first language model based on the speech verification sample set; and determine a heterogeneous language model based on the at least four first language models and the error rates.

[0162] In some embodiments, at least two first network models and at least two second network models are randomly combined to obtain at least four first language models.

[0163] It should be noted that for a detailed explanation of method 21, please refer to the following sections. Figure 3 The specific details of the illustrated embodiment are as follows.

[0164] Method 22: Determine a first target network model based on at least two first network models, and determine a second target network model based on at least two second network models; define the first target network model and the second target network model as a heterogeneous language model. The first target network model and the first network model have the same structure, and the second target network model has the same structure as the second network model.

[0165] For a detailed explanation of method 22, please refer to the following sections. Figure 4 The specific details of the illustrated embodiment are as follows.

[0166] exist Figure 2 In the training method for the heterogeneous language model provided in the embodiment, a speech training sample set is used to train the first initial network model and the second initial network model to obtain at least two first network models and at least two second network models. Based on the at least two first network models and at least two second network models, the heterogeneous language model is determined. A speech training sample set with fewer speech samples can be used to obtain a heterogeneous language model with higher accuracy, thereby improving the accuracy of the heterogeneous language model.

[0167] In this application, the first network model and the second network model have different structures and functions. Therefore, the heterogeneous language model determined based on at least two first network models and at least two second network models also includes two network models with different structures and functions. In the heterogeneous language model provided in the embodiments of this application, the two network models with different structures and functions are used in combination. This allows for the initial determination of the approximate range of the result (i.e., processing the input pinyin sequence to obtain at least one character sequence corresponding to the pinyin sequence), followed by a more refined judgment of this range (e.g., determining the target character sequence corresponding to the pinyin sequence from at least one character sequence), thereby improving the accuracy of the heterogeneous language model.

[0168] In related technologies, isomorphic language models include a network model (such as an n-gram model or DNN) that directly obtains the target text sequence corresponding to the input pinyin sequence. However, it lacks the process of determining the target text sequence from at least one obtained text sequence. Therefore, the accuracy of isomorphic language models is usually poor.

[0169] Based on the above embodiments, the following is combined with Figure 3 The specific execution process of method 21 above will be explained.

[0170] Figure 3 This is a flowchart illustrating a method for determining heterogeneous language models, provided as an embodiment of this application. Figure 3 As shown, the method includes:

[0171] S301, for each of the at least two first network models, perform binary conversion on the model parameters of the first network model to obtain a first initial parameter sequence; perform crossover and mutation on the first initial parameter sequence to obtain at least two first intermediate parameter sequences; replace the model parameters of the first network model with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0172] Optionally, at least two first intermediate parameter column orders can be obtained using the following method 30.

[0173] Method 30: Perform a crossover and mutation process on the first initial parameter sequence to obtain a first intermediate parameter sequence; perform another crossover and mutation process on the obtained first intermediate parameter sequence to obtain another first intermediate parameter sequence; determine the above two first intermediate parameter sequences as at least two first intermediate parameter sequences.

[0174] S302, for each of the at least two second network models, perform binary conversion on the model parameters of the second network model to obtain a second initial parameter sequence; perform crossover and mutation on the second initial parameter sequence to obtain at least two second intermediate parameter sequences; replace the model parameters of the second network model with the model parameters corresponding to the at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model.

[0175] Optionally, a method similar to method 30 described above can be used to perform crossover and mutation processing on the second initial parameter sequence to obtain at least two second intermediate parameter sequences.

[0176] S303, randomly combine at least two third network models corresponding to each of the at least two first network models and at least two fifth network models corresponding to each of the at least two second network models to obtain at least four first language models.

[0177] It should be noted that S301 to S303 above is an explanation of determining at least four first language models based on at least two first network models and at least two second network models.

[0178] S304, Obtain the voice verification sample set.

[0179] The voice verification sample set differs from the voice training sample set. The voice verification sample set may include, for example, at least one dataset from SLR18 and SLR68 in a pre-defined open website.

[0180] The voice verification sample set includes multiple voice verification samples, a corresponding pinyin verification sample for each voice verification sample, and the corresponding text verification result for each pinyin verification sample.

[0181] S305, determine the error rate of each first language model based on the speech verification sample set.

[0182] Optionally, for each of at least four first language models, each first language model includes a first network model and a second network model; the error rate of the first language model is determined based on the speech verification sample set, including:

[0183] Multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each pinyin verification sample.

[0184] The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0185] In this application, "passing sequentially" means that after one network model processes a sample, it provides intermediate results to another network model, so that the other network model processes the intermediate results to obtain the output results.

[0186] S306, Determine heterogeneous language models based on at least four first language models and error rates.

[0187] Alternatively, the heterogeneous language model can be determined using methods 31 and 32 as follows.

[0188] Method 31: The first language model with the lowest error rate among at least four first language models is identified as the heterogeneous language model.

[0189] Method 32: Determine whether there is a first language model with an error rate lower than a preset value among at least four first language models;

[0190] If so, the first language model with an error rate lower than the preset value is identified as a heterogeneous language model;

[0191] If not, obtain the first model parameter sequence corresponding to the model parameters of a preset number of first language models from at least four first language models, so as to obtain at least one second language model corresponding to the preset number of first language models, and determine the heterogeneous language model based on the multiple second language models and the error rate of each second language model.

[0192] Optionally, the preset number of first language models can be a preset number of first language models with the lowest error rates among at least four first language models, or it can be any preset number of first language models among at least four first language models.

[0193] Optionally, when the first language model includes a first network model and a second network model, the first model parameter sequence includes a first initial parameter sequence corresponding to the model parameters of the first network model and a second initial parameter sequence corresponding to the model parameters of the second network model in the first language model.

[0194] Optionally, when the first language model includes a third network model and a fifth network model, the first model parameter sequence includes the third initial parameter sequence corresponding to the model parameters of the third network model in the first language model and the fifth initial parameter sequence corresponding to the model parameters of the fifth network model.

[0195] Optionally, a heterogeneous language model is determined based on multiple second language models and the error rate of each second language model. This includes: repeatedly executing a method similar to method 32 for multiple second language models until an Nth language model with an error rate less than a preset value is obtained, and identifying the Nth language model with an error rate less than the preset value as a heterogeneous language model. Here, N can be a positive integer greater than or equal to 2. It should be noted that in the first language model, second language model, ..., Nth language model, each subsequent language model is obtained based on the previous language model. For example, the second language model is obtained based on the first language model.

[0196] The following section uses the example of a first language model including a third network model and a fifth network model to illustrate the method for obtaining at least one second language model corresponding to the first language model:

[0197] The model parameters of the third network model are converted into binary form to obtain the third initial parameter sequence;

[0198] Cross-processing and mutation processing are performed on the third initial parameter sequence to obtain at least one third intermediate parameter sequence. The model parameters of the third network model are replaced with the model parameters corresponding to at least one third intermediate parameter sequence to obtain at least one sixth network model corresponding to the third network model.

[0199] The model parameters of the fifth network model are converted into binary form to obtain the fifth initial parameter sequence;

[0200] Crossover and mutation processing are performed on the fifth initial parameter sequence to obtain at least one fifth intermediate parameter sequence. The model parameters of the fifth network model are replaced with the model parameters corresponding to at least one fifth intermediate parameter sequence to obtain at least one seventh network model corresponding to the fifth network model.

[0201] By randomly combining multiple sixth network models and multiple seventh network models, at least one second language model corresponding to the first language model can be obtained.

[0202] Specifically, S304 to S306 are explanations of how to determine a heterogeneous language model based on at least four first language models.

[0203] exist Figure 3In the method for determining heterogeneous language models provided in the embodiments, at least one third network model corresponding to each first network model and at least one fifth network model corresponding to each second network model are randomly combined to obtain at least four first language models. This method can obtain multiple first language models even when there are few speech samples in the speech training sample set. Furthermore, the error rate of each first language model is determined based on the speech verification sample set. The heterogeneous language model is determined based on at least four first language models and the error rates, which can improve the accuracy of the heterogeneous language model.

[0204] Based on the above embodiments, the following is combined with Figure 4 The specific execution process of method 22 above will be explained.

[0205] Figure 4 A flowchart illustrating another method for determining heterogeneous language models provided in this application embodiment. Figure 4 As shown, the method includes:

[0206] S401, for each of the at least two first network models, perform binary conversion on the model parameters of the first network model to obtain the first initial parameter sequence corresponding to the model parameters of the first network model; perform crossover and mutation processing on the first initial parameter sequence to obtain at least two first intermediate parameter sequence orders; replace the model parameters of the first network model with the model parameters corresponding to the at least two first intermediate parameter sequence orders to obtain at least two third network models corresponding to the first network model.

[0207] Optionally, a method similar to the above method 30 can be used to perform crossover and mutation processing on the first initial parameter sequence to obtain at least two first intermediate parameter sequence sequences.

[0208] Specifically, a binary conversion of a first intermediate parameter column is performed to obtain the model parameters corresponding to the first intermediate parameter column order. Then, the model parameters of the first network model are replaced with the model parameters corresponding to the first intermediate parameter column order to obtain a third network model corresponding to the first network model.

[0209] S402. Based on multiple third network models, determine the first target network model.

[0210] In S402, the multiple third network models include at least two third network models corresponding to each of the at least two first network models in S401.

[0211] In some embodiments, S402 specifically includes: acquiring a voice verification sample set; determining the error rate of each third network model based on the voice verification sample set; and determining a first target network model based on the plurality of third network models and the error rate of each third network model.

[0212] The voice verification sample set here can be the same as the voice verification sample set in S304.

[0213] In some embodiments, for each of the plurality of third network models, the error rate of the third network model is determined based on the speech verification sample set, including: processing the plurality of pinyin verification samples sequentially through the third network model and any second network model to obtain the text output results corresponding to each of the plurality of pinyin verification samples; and determining the error rate of the third network model as the ratio of the number of pinyin verification samples whose text output results and text verification results are different to the total number of the plurality of pinyin verification samples.

[0214] Optionally, any second network model can be any one of at least two second network models, or it can be the second network model after the Y-th update of the model parameters of the second initial network model. Optionally, Y can be the maximum value among Y1, Y2, Y3..., or it can be a preset fixed value. Wherein, Y1, Y2, Y3... represent the number of times the model parameters of the second initial network model are updated.

[0215] It should be noted that, for a third network model, in the process of determining the error rate of the third network model, any of the aforementioned second network models are selected and fixed.

[0216] Alternatively, a method similar to S306 can be used to determine the first target network model based on multiple third network models and the error rate of each third network model, which will not be elaborated here.

[0217] S403. For each of the at least two second network models, perform binary conversion on the model parameters of the second network model to obtain the second initial parameter sequence corresponding to the model parameters of the second network model; perform crossover and mutation processing on the second initial parameter sequence to obtain at least two second intermediate parameter sequences; replace the model parameters of the second network model with the model parameters corresponding to the at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model.

[0218] Specifically, the method of S403 is similar to that of S401, and will not be described in detail here.

[0219] S404. Based on multiple fifth network models, determine the second target network model.

[0220] In S404, the multiple fifth network models include at least two fifth network models corresponding to each of the at least two second network models in S403.

[0221] In some embodiments, S404 specifically includes: acquiring a voice verification sample set; determining the error rate of each fifth network model based on the voice verification sample set; and determining a second target network model based on the plurality of fifth network models and the error rate of each fifth network model.

[0222] In some embodiments, for each of the plurality of fifth network models, determining the error rate of the fifth network model based on the speech verification sample set includes: processing the plurality of speech verification samples sequentially through the first target network model and the fifth network model to obtain the text output results corresponding to each of the plurality of speech verification samples;

[0223] The error rate of the fifth network model is determined by the ratio of the number of voice verification samples whose text output and text verification results are different to the total number of multiple voice verification samples.

[0224] Alternatively, a method similar to S306 can be used to determine the second target network model based on multiple fifth network models and the error rate of each fifth network model, which will not be elaborated here.

[0225] S405. The first target network model and the second target network model are determined to be heterogeneous language models.

[0226] exist Figure 4 In the method provided in the embodiment, the first initial parameter sequence is subjected to cross-processing and mutation processing to obtain at least one third network model corresponding to the first network model, which enables the acquisition of multiple third network models even when the number of speech samples in the speech training sample set is small. The second initial parameter sequence is subjected to cross-processing and mutation processing to obtain at least one fifth network model corresponding to the second network model, which also enables the acquisition of multiple fifth network models even when the number of speech samples in the speech training sample set is small. Further, the error rate of each third network model and the error rate of each fifth network model are determined based on the speech verification sample set. A first target network model is determined based on the multiple third network models and their error rates, and a second target network model is determined based on the multiple fifth network models and their error rates. This improves the accuracy of the obtained first and second target network models, thereby improving the accuracy of the obtained language model.

[0227] It should be noted that the above Figure 3 and Figure 4 The example illustrates a specific method for obtaining heterogeneous language models based on the Genetic Matching Merge Algorithm (GMMA).

[0228] Based on the above embodiments, this application also provides a method for determining heterogeneous language models, which is described below in conjunction with... Figure 5The examples illustrate the method.

[0229] Figure 5 A flowchart illustrating another method for determining heterogeneous language models provided in this application embodiment. Figure 5 As shown, the method includes:

[0230] S501. Generate the first weights of each first network model and the second weights of each second network model through the weight generation model.

[0231] S502. Based on the first weight, the model parameters of each first network model are fused to obtain the first target model parameters.

[0232] Optionally, the model parameters of each first network model are weighted and summed according to the first weight to achieve the fusion of the model parameters of each first network model.

[0233] For example, if there are 2 first network models, and the weights of the two first network models are A1 and A2 respectively, and the model parameters of one first network model include X11, X12, and X13, and the model parameters of the other first network model include X21, X22, and X23, then the target model parameters include A1*X11+A2*X21, A1*X12+A2*X22, and A1*X13+A2*X23.

[0234] S503. Based on the second weight, the model parameters of each second network model are fused to obtain the second target model parameters.

[0235] Specifically, the method of S502 is similar to that of S503, and will not be described in detail here.

[0236] S504. Replace the model parameters of the first network model with the parameters of the first target model to obtain the first target network model.

[0237] Here, the first network model is any one of the first network models in S501 above.

[0238] S505. Replace the model parameters of the second network model with the parameters of the second target model to obtain the second target network model.

[0239] Here, the second network model is any one of the second network models in S502 above.

[0240] S501 to S505 are detailed descriptions of determining a first target network model based on at least two first network models and determining a second target network model based on at least two second network models.

[0241] S506. The first target network model and the second target network model are determined to be heterogeneous language models.

[0242] In this application, a weight generation model is used to generate first weights for each first network model and second weights for each second network model to obtain first target model parameters and second target network models. The first network model with the first target model parameters is determined as the first target network model, and the second network model with the second target model parameters is determined as the second target network model. A speech training sample set with fewer speech samples is used to obtain a heterogeneous language model with higher accuracy, thereby improving the efficiency of obtaining the heterogeneous language model.

[0243] In some embodiments, the method provided in this application further includes: obtaining a speech verification sample set; determining the error rate of the heterogeneous language model based on the speech verification sample set; determining the reward value based on the error rate; updating the model parameters of the weight generation model based on the reward value, and regenerating the first weights of each first network model and the second weights of each second network model to update the heterogeneous language model.

[0244] Optionally, the reward value is equal to the product of 1 minus the error rate difference and 100%.

[0245] In the above Figure 5 Based on this, the following will combine Figure 6 The method for updating heterogeneous language models is explained.

[0246] Figure 6 A flowchart illustrating a method for updating heterogeneous language models provided in an embodiment of this application. Figure 6 As shown, the method includes:

[0247] S601. Using a weight generation model with the i-th set of model parameters, generate the first weights corresponding to each first network model and the second weights corresponding to each second network model.

[0248] S602. According to the first weight, the model parameters of each first network model are fused to obtain the first target model parameters; according to the second weight, the model parameters of each second network model are fused to obtain the second target model parameters.

[0249] S603. Replace the model parameters of the first network model with the parameters of the first target model to obtain the first target network model; replace the model parameters of the second network model with the parameters of the second target model to obtain the second target network model; determine the first target network model and the second target network model as the i-th heterogeneous language model.

[0250] S604. Obtain the voice verification sample set.

[0251] S605. Based on the speech verification sample set, determine the error rate of the i-th heterogeneous language model.

[0252] S606. Determine the i-th reward value based on the error rate of the i-th heterogeneous language model.

[0253] S607. Determine whether the i-th reward value is greater than or equal to the preset reward value.

[0254] If yes, then execute S608; otherwise, execute S609.

[0255] S608, the i-th heterogeneous language model is determined as the final heterogeneous language model.

[0256] S609. Based on the reward value, determine the (i+1)th set of model parameters for the weight generation model, and repeat S601 to S609 based on the obtained (i+1)th set of model parameters. Initially, i equals 1.

[0257] Optionally, the reward value can be substituted into a preset formula, and the preset formula after substituting the reward value can be processed to obtain the (i+1)th set of model parameters.

[0258] For example, a preset formula can have the form of Formula 1 as follows:

[0259]

[0260] Where J is the objective function, ω a Here, T represents the model parameters for the weight generation model, where T is the total weight after dividing the series of weights that need to be merged across all models (including the first network models and the second network models in S601), and a is the model parameters for the weight generation model. t Let a be the weight of the t-th part of the model in T. (t-1) Let R be the weight of the (t-1)th part of the model in T, R be the reward value, and E be ω. a The mathematical expectation, P is ω a The probability function, For ω a The gradient of , where log represents the logarithm operation.

[0261] It should be noted that Equation 1 can be processed using the gradient descent convergence method to obtain ω. a and the obtained ω a , which is determined to be the model parameters of the (i+1)th group.

[0262] It should be noted that, based on the obtained i+1 group of model parameters, the process of repeatedly executing S601 to S610 is to update the model parameters of the model generated by updating the weights according to the reward value, and to regenerate the first weights of each first network model and the second weights of each second network model in order to obtain the explanation of the new heterogeneous language model.

[0263] exist Figure 6 In the method provided in the embodiment, during each execution of S601 to S609, a weight generation model generates first weights corresponding to each first network model and second weights corresponding to each second network model. Based on the first and second weights, a heterogeneous language model is obtained, which can improve the efficiency of obtaining the heterogeneous language model, and thus improve the efficiency of obtaining the final heterogeneous language model. Further, based on the error rate of the i-th heterogeneous language model, the i-th reward value is determined, and based on the i-th reward value, the i+1-th set of model parameters of the weight generation model is determined. S601 to S609 are executed repeatedly, and the heterogeneous language model whose reward value is greater than or equal to the preset reward value is determined as the final heterogeneous language model.

[0264] Unlike existing technologies, where LM (Literature Model) is a homogeneous language model, which includes a single network model that directly derives the target text sequence from the input Pinyin sequence, lacking the process of determining the target text sequence from at least one obtained text sequence, resulting in poor accuracy, this application employs a heterogeneous language model. This model comprises two network models with distinct structures and functions. The combined use of these two models allows for the initial determination of the approximate range of the result (i.e., processing the input Pinyin sequence to obtain at least one corresponding text sequence), followed by a more refined judgment of this approximate range (e.g., determining the target text sequence from at least one text sequence), thereby improving the accuracy of the heterogeneous language model.

[0265] Specifically, Figure 5 and Figure 6 The method provided in the embodiment is a specific method for obtaining heterogeneous language models based on the Reinforcement Learning Guided Merge Algorithm (RLGMA).

[0266] Figure 7 This is an architecture diagram of a heterogeneous language model based on GMMA, provided for embodiments of this application. Figure 7 As shown, it includes: at least two first network models and at least two second network models.

[0267] Optionally, at least two first network models include first network models 11 to 1M, and at least two second network models include second network models 21 to 2N. M is the total number of first network models, and N is the total number of second network models. M and N are both integers greater than or equal to 2.

[0268] Specifically, in Figure 3Based on this, the first fusion process includes, for example, the relevant processing involved in S301 in obtaining at least two third network models corresponding to each first network model for any one of the at least two first network models;

[0269] The second fusion process includes, for example, the relevant processing involved in S302 in obtaining at least two fifth network models corresponding to each second network model for any one of the at least two second network models;

[0270] The third process includes, for example, the following: obtaining at least four first language models in S303 to S306, acquiring a speech verification sample set, determining the error rate of each first language model based on the speech verification sample set, and related processing involved in determining the heterogeneous language model based on at least four first language models and the error rate.

[0271] Specifically, in Figure 4 Based on this, the first fusion process includes, for example, S401 obtaining at least two third network models corresponding to each first network model, and S402 related processing involved in determining the first target network model based on multiple third network models.

[0272] The second fusion process includes, for example, the process of obtaining at least two fifth network models corresponding to each second network model in S403, and the related processing involved in determining the second target network model based on multiple fifth network models in S404.

[0273] The third processing includes, for example, the processing in S405 that identifies the first target network model and the second target network model as heterogeneous language models.

[0274] Figure 8 The following is an architecture diagram of a heterogeneous language model based on RLGMA provided in an embodiment of this application. The heterogeneous language model includes: a weight generation model, first weights corresponding to each first network model, and second weights corresponding to each second network model.

[0275] In each first network model, the first weight corresponding to the i-th first network model is A. 1i , i = 1…M.

[0276] In each of the second network models, the second weight corresponding to the j-th second network model is B. 2j , j = 1…N.

[0277] exist Figure 8The process can be repeated as follows: The weight generation model generates the first weights for each first network model and the second weights for each second network model based on the reward value. The first and second network models obtain the language model based on the first and second weights. The first and second weights are continuously updated until the reward value is greater than or equal to the preset reward value, at which point the final heterogeneous language model is obtained.

[0278] Figure 9 This is a schematic diagram of the structure of a training device for a heterogeneous language model provided in an embodiment of this application. Figure 9 As shown, the training device 10 for the heterogeneous language model includes: a processing module; the processing module 101 is used for:

[0279] Obtain the speech training sample set;

[0280] Using a speech training sample set, a first initial network model and a second initial network model are trained to obtain at least two first network models and at least two second network models. The first network models and the second network models have different structures. The first network model is used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network model is used to determine the target text sequence corresponding to the pinyin sequence from at least one text sequence.

[0281] Determine the heterogeneous language model based on at least two first network models and at least two second network models.

[0282] The heterogeneous language model training device provided in this application embodiment can execute the above-described heterogeneous language model training method. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0283] Optionally, the processing module 101 is specifically used for:

[0284] Based on at least two first network models and at least two second network models, determine at least four first language models;

[0285] Obtain the voice verification sample set;

[0286] Based on the speech verification sample set, determine the error rate of each first language model;

[0287] Determine heterogeneous language models based on at least four first language models and error rates.

[0288] Optionally, the processing module 101 is specifically used for:

[0289] At least two first network models and at least two second network models are randomly combined to obtain at least four first language models.

[0290] Optionally, the processing module 101 is specifically used for:

[0291] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0292] For each second network model, the model parameters of the second network model are converted into binary to obtain a second initial parameter sequence; the second initial parameter sequence is cross-processed and mutated to obtain at least two second intermediate parameter sequences; the model parameters of the second network model are replaced with the model parameters corresponding to at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model.

[0293] Randomly combine at least two third network models corresponding to each of the at least two first network models and at least two fifth network models corresponding to each of the at least two second network models to obtain at least four first language models.

[0294] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples; for each of the at least four first language models, the first language model includes a first network model and a second network model;

[0295] Processing module 101 is specifically used for:

[0296] Multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each pinyin verification sample.

[0297] The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0298] Optionally, the processing module is specifically used for:

[0299] Determine if at least one of the four first language models has an error rate lower than a preset value;

[0300] If so, the first language model with an error rate lower than the preset value is identified as a heterogeneous language model;

[0301] If not, obtain the first model parameter sequence corresponding to the model parameters of a preset number of first language models from at least four first language models, so as to obtain at least one second language model corresponding to the preset number of first language models, and determine the heterogeneous language model based on the multiple second language models and the error rate of each second language model.

[0302] Optionally, the processing module 101 is specifically used for:

[0303] Based on at least two first network models, determine a first target network model, and based on at least two second network models, determine a second target network model;

[0304] The first target network model and the second target network model are identified as heterogeneous language models.

[0305] Optionally, the processing module 101 is specifically used for:

[0306] For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model.

[0307] Based on multiple third network models, the first target network model is determined.

[0308] Optionally, the processing module 101 is specifically used for:

[0309] Obtain the voice verification sample set;

[0310] Based on the voice verification sample set, determine the error rate of each third network model;

[0311] The first target network model is determined based on multiple third network models and error rates.

[0312] Optionally, the voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each of the multiple pinyin verification samples;

[0313] Processing module 101 is specifically used for:

[0314] Multiple pinyin verification samples are processed sequentially through the third network model and any second network model to obtain the text output results corresponding to each pinyin verification sample.

[0315] The error rate of the third network model is determined by the ratio of the number of pinyin verification samples whose text output and text verification results are different to the total number of multiple pinyin verification samples.

[0316] Optionally, the processing module 101 is specifically used for:

[0317] Determine if there is a third network model among multiple third network models with an error rate lower than a preset value;

[0318] If so, the third network model with an error rate less than the preset value will be identified as the first target network model;

[0319] If not, obtain the first initial parameter sequence corresponding to the model parameters of a preset number of third network models among the multiple third network models, so as to obtain at least one fourth network model corresponding to each of the preset number of third network models, and determine the first target network model based on the multiple fourth network models and the error rate of each fourth network model.

[0320] Optionally, the processing module 101 is specifically used for:

[0321] The first weights of each first network model and the second weights of each second network model are generated using a weight generation model.

[0322] Based on the first weight, the model parameters of each first network model are fused to obtain the first target model parameters; based on the second weight, the model parameters of each second network model are fused to obtain the second target model parameters.

[0323] The model parameters of the first network model are replaced with the parameters of the first target model to obtain the first target network model; the model parameters of the second network model are replaced with the parameters of the second target model to obtain the second target network model.

[0324] Optionally, the processing module 101 is also used for:

[0325] Obtain the voice verification sample set;

[0326] Based on the speech verification sample set, determine the error rate of the heterogeneous language model, and determine the reward value based on the error rate;

[0327] Based on the reward value, update the model parameters of the weight generation model, regenerate the first weights of each first network model and the second weights of each second network model to obtain a new heterogeneous language model.

[0328] Optionally, the speech training sample set includes at least two sample subsets; the processing module 101 is specifically used for:

[0329] The first initial network model is trained using at least two sample subsets respectively, to obtain the first network model corresponding to each of the at least two sample subsets;

[0330] The second initial network model is trained using at least two sample subsets respectively, to obtain the second network model corresponding to each of the at least two sample subsets;

[0331] The first network model corresponding to each of at least two sample subsets is determined as at least two first network models; the second network model corresponding to each of at least two sample subsets is determined as at least two second network models.

[0332] The heterogeneous language model training device 10 provided in this application embodiment can execute the above-described heterogeneous language model training method. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0333] Figure 10 A hardware schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 10 As shown, the electronic device 20 may include a transceiver 201, a memory 202, and a processor 203. The transceiver 201 may include a transmitter and / or a receiver. The transmitter may also be referred to as a transmitter, transmitter, transmitting port, or transmitting interface, etc., and the receiver may also be referred to as a receiver, receiver, receiving port, or receiving interface, etc. Exemplarily, the transceiver 201, memory 202, and processor 203 are interconnected via a bus 204.

[0334] Memory 202 is used to store instructions executed by the computer;

[0335] The processor 203 is used to execute computer execution instructions stored in the memory 202, so that the processor 203 executes the training method of the heterogeneous language model.

[0336] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement a method for training a heterogeneous language model.

[0337] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement a method for training heterogeneous language models.

[0338] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0339] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One process or at least one process and / or block Figure 1 A device that provides the function specified in one or more boxes.

[0340] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One process or at least one process and / or block Figure 1 The function specified in one or more boxes.

[0341] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One process or at least one process and / or block Figure 1 The steps of the function specified in one or more boxes.

[0342] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0343] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". In this application, the terms "first", "second", etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "at least one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0344] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0345] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a heterogeneous language model, characterized in that, include: Obtain the speech training sample set; Using the aforementioned speech training sample set, a first initial network model and a second initial network model are trained to obtain at least two first network models and at least two second network models. The first network models and the second network models have different structures. The first network model is used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network model is used to determine the target text sequence corresponding to the pinyin sequence from the at least one text sequence. Based on the at least two first network models and the at least two second network models, at least four first language models are determined; Obtain the voice verification sample set; The voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each pinyin verification sample; for each of the at least four first language models, the first language model includes the first network model and the second network model; The multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each of the multiple pinyin verification samples; The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output results and text verification results are different to the total number of the multiple pinyin verification samples. Based on the at least four first language models and the error rate, a heterogeneous language model is determined; or, For each first network model, the model parameters of the first network model are converted into binary form to obtain a first initial parameter sequence; the first initial parameter sequence is then subjected to crossover and mutation processing to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model; a first target network model is determined based on the multiple third network models; or, a first weight is generated for each first network model using a weight generation model; the model parameters of each first network model are fused based on the first weight to obtain first target model parameters; the model parameters of the first network models are then replaced with the first target model parameters to obtain the first target network model. Based on the at least two second network models, determine the second target network model; The first target network model and the second target network model are identified as heterogeneous language models.

2. The training method for heterogeneous language models according to claim 1, characterized in that, The determination of at least four first language models based on the at least two first network models and the at least two second network models includes: The at least two first network models and the at least two second network models are randomly combined to obtain at least four first language models.

3. The training method for heterogeneous language models according to claim 1, characterized in that, The determination of at least four first language models based on the at least two first network models and the at least two second network models includes: For each first network model, the model parameters of the first network model are converted into binary to obtain a first initial parameter sequence; the first initial parameter sequence is cross-processed and mutated to obtain at least two first intermediate parameter sequence orders; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequence orders to obtain at least two third network models corresponding to the first network model. For each second network model, the model parameters of the second network model are converted into binary to obtain a second initial parameter sequence; the second initial parameter sequence is cross-processed and mutated to obtain at least two second intermediate parameter sequences; the model parameters of the second network model are replaced with the model parameters corresponding to the at least two second intermediate parameter sequences to obtain at least two fifth network models corresponding to the second network model. At least two third network models corresponding to each of the at least two first network models and at least two fifth network models corresponding to each of the at least two second network models are randomly combined to obtain at least four first language models.

4. The training method for heterogeneous language models according to claim 1, characterized in that, The step of determining the heterogeneous language model based on the at least four first language models and the error rate includes: Determine that among the at least four first language models, there is a first language model with an error rate less than a preset value; If so, the first language model with an error rate less than a preset value is identified as the heterogeneous language model; If not, then obtain the first model parameter sequence corresponding to the model parameters of a preset number of first language models among the at least four first language models, so as to obtain at least one second language model corresponding to the preset number of first language models, and determine the heterogeneous language model based on the multiple second language models and the error rate of each second language model.

5. The training method for heterogeneous language models according to claim 1, characterized in that, The step of determining the first target network model based on multiple third network models includes: Obtain the voice verification sample set; Based on the aforementioned voice verification sample set, determine the error rate of each third network model; Based on the plurality of third network models and the error rate, a first target network model is determined.

6. The training method for heterogeneous language models according to claim 5, characterized in that, The voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each pinyin verification sample; Based on the aforementioned speech verification sample set, the error rate of the third network model is determined, including: The multiple pinyin verification samples are processed sequentially through the third network model and any of the second network models to obtain the text output results corresponding to each of the multiple pinyin verification samples; The error rate of the third network model is determined by the ratio of the number of pinyin verification samples whose text output results and text verification results are different to the total number of the multiple pinyin verification samples.

7. The training method for heterogeneous language models according to claim 5, characterized in that, The step of determining the first target network model based on the plurality of third network models and the error rate includes: Determine whether there is a third network model among the plurality of third network models with an error rate less than a preset value; If so, the third network model with an error rate less than a preset value is determined as the first target network model; If not, then obtain the first initial parameter sequence corresponding to the model parameters of a preset number of third network models among the multiple third network models, so as to obtain at least one fourth network model corresponding to each of the preset number of third network models, and determine the first target network model based on the multiple fourth network models and the error rate of each fourth network model.

8. The training method for heterogeneous language models according to claim 1, characterized in that, The step of determining the second target network model based on the at least two second network models includes: The second weights of each second network model are generated using a weight generation model. Based on the second weight, the model parameters of each second network model are fused to obtain the second target model parameters; The second target network model is obtained by replacing the model parameters of the second network model with the parameters of the second target model.

9. The training method for heterogeneous language models according to claim 8, characterized in that, The method further includes: Obtain the voice verification sample set; Based on the speech verification sample set, the error rate of the heterogeneous language model is determined, and the reward value is determined based on the error rate. Based on the reward value, the model parameters of the weight generation model are updated, and the first weights of each first network model and the second weights of each second network model are regenerated to obtain a new heterogeneous language model.

10. The training method for a heterogeneous language model according to any one of claims 1 to 9, characterized in that, The speech training sample set includes at least two sample subsets; the step of using the speech training sample set to train a first initial network model and a second initial network model to obtain at least two first network models and at least two second network models includes: The first initial network model is trained using the at least two sample subsets respectively, to obtain the first network model corresponding to each of the at least two sample subsets; The second initial network model is trained using the at least two sample subsets respectively, to obtain the second network model corresponding to each of the at least two sample subsets; The first network model corresponding to each of the at least two sample subsets is determined as the at least two first network models; the second network model corresponding to each of the at least two sample subsets is determined as the at least two second network models.

11. A training device for a heterogeneous language model, characterized in that, include: Processing module; The processing module is used for: Obtain the speech training sample set; Using the aforementioned speech training sample set, a first initial network model and a second initial network model are trained to obtain at least two first network models and at least two second network models. The first network models and the second network models have different structures. The first network model is used to process the input pinyin sequence to obtain at least one text sequence corresponding to the pinyin sequence, and the second network model is used to determine the target text sequence corresponding to the pinyin sequence from the at least one text sequence. Based on the at least two first network models and the at least two second network models, at least four first language models are determined; Obtain the voice verification sample set; The voice verification sample set includes multiple pinyin verification samples and the text verification results corresponding to each pinyin verification sample; for each of the at least four first language models, the first language model includes the first network model and the second network model; The multiple pinyin verification samples are processed sequentially through the first network model and the second network model to obtain the text output results corresponding to each of the multiple pinyin verification samples; The error rate of the first language model is determined by the ratio of the number of pinyin verification samples whose text output results and text verification results are different to the total number of the multiple pinyin verification samples. Based on the at least four first language models and the error rate, a heterogeneous language model is determined; or, For each first network model, the model parameters of the first network model are converted into binary form to obtain a first initial parameter sequence; the first initial parameter sequence is then subjected to crossover and mutation processing to obtain at least two first intermediate parameter sequences; the model parameters of the first network model are replaced with the model parameters corresponding to the at least two first intermediate parameter sequences to obtain at least two third network models corresponding to the first network model; a first target network model is determined based on the multiple third network models; or, a first weight is generated for each first network model using a weight generation model; the model parameters of each first network model are fused based on the first weight to obtain first target model parameters; the model parameters of the first network models are then replaced with the first target model parameters to obtain the first target network model. Based on the at least two second network models, determine the second target network model; The first target network model and the second target network model are identified as heterogeneous language models.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 10.

Images