Processing method, training method, interaction method, server, vehicle and medium

The pre-trained natural language processing model annotates the voice request data, which solves the problem of high difficulty in labeling training data of the vehicle natural language processing model, improves the labeling efficiency and accuracy, and ensures the performance and voice interaction quality of the vehicle acoustic model.

CN119091869APending Publication Date: 2024-12-06GUANGZHOU XIAOPENG MOTORS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411218050.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The training data labeling of the on-board natural language processing model is difficult and inefficient, resulting in poor model performance and voice interaction quality.

Method used

By using pre-trained multiple natural language processing models to predict speech request data, determine semantic units and phoneme labels, reduce manual annotation dependence, improve labeling efficiency and accuracy, and generate sufficient sample labels through model training methods to improve the model performance.

Benefits of technology

It improves the labeling accuracy and reliability of voice request data, reduces manual labeling dependence, and ensures the performance and voice interaction quality of the vehicle-mounted acoustic model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091869A_ABST
    Figure CN119091869A_ABST
Patent Text Reader

Abstract

The invention discloses a voice request processing method, a model training method, a voice interaction method, a server, a vehicle and a computer readable storage medium, and the processing method comprises the steps: obtaining to-be-processed voice request data, and carrying out the processing of the to-be-processed voice request data according to a plurality of pre-trained natural language processing models, and determining a prediction result of each natural language processing model for the voice request data, and determining a semantic unit tag of the voice request data and a phoneme tag corresponding to the semantic unit tag according to each prediction result, thereby completing processing of the voice request data. Thus, both semantic unit labeling and phoneme labeling of the voice request data are realized, and the degree of dependence of the sample labels required for training the vehicle-mounted acoustic model on manpower is reduced, so that the vehicle-mounted acoustic model can be trained through a sufficient number of voice request samples and the corresponding sample labels, and the training efficiency is improved. And the performance of the vehicle-mounted acoustic model and the quality of voice interaction can be guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data annotation technology, and in particular to a method for processing voice requests, a model training method, a voice interaction method, a server, a vehicle, and a computer-readable storage medium. Background Art

[0002] With the introduction and application of in-vehicle voice interaction, vehicles in related technologies can understand the sounds in the cabin through pre-deployed in-vehicle natural language processing models to perform corresponding operations, such as starting the air conditioner based on the voice recognition result after the driver says "turn on the air conditioner". However, since the performance of the in-vehicle natural language processing model depends on the training data, and the annotation of training data is usually done manually, the annotation is difficult and inefficient, resulting in insufficient amount of training data, which to some extent affects the performance of the in-vehicle natural language processing model and the quality of the in-vehicle voice interaction function. Summary of the invention

[0003] The present application provides a method for processing a voice request, a model training method, a voice interaction method, a server, a vehicle, and a computer-readable storage medium.

[0004] A method for processing a voice request provided in an embodiment of the present application includes:

[0005] Get the voice request data to be processed;

[0006] Determine, according to a plurality of pre-trained natural language processing models, a prediction result of each of the natural language processing models for the voice request data, wherein the prediction result includes phoneme prediction results of a plurality of semantic units;

[0007] According to the prediction results of each of the natural language processing models, the semantic unit tags of the voice request data and the phoneme tags corresponding to the semantic unit tags are determined to complete the processing of the voice request data.

[0008] In the voice request processing method provided in the embodiment of the present application, the server can obtain the voice request data to be processed, and determine the prediction results of each natural language processing model for the voice request data based on multiple pre-trained natural language processing models, and determine the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label based on the prediction results of each natural language processing model to complete the processing of the voice request data.

[0009] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0010] In certain embodiments of the present application, the phoneme prediction result includes prediction probabilities of multiple phoneme predictions, and determining the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label according to the prediction result of each of the natural language processing models, and completing the processing of the voice request data, including:

[0011] According to the same phoneme prediction among the plurality of phoneme predictions, the plurality of phoneme predictions and the prediction probabilities of the plurality of phoneme predictions are combined to obtain a plurality of first phonemes and a first probability of each of the first phonemes;

[0012] The phoneme label is determined according to a second phoneme among the plurality of first phonemes and the first probability of the second phoneme, wherein the second phoneme is a preset number of first phonemes among the plurality of first phonemes whose first probability is the highest.

[0013] In this way, in an embodiment of the present application, the predictions of the same phoneme and the prediction probabilities of the same phoneme predictions in the prediction results output by the natural language processing model can be merged to obtain multiple first phonemes and the first probability of each first phoneme, and then the phoneme labels of the voice request data are determined based on a preset number of first phonemes with the highest first probability among the multiple first phonemes, thereby reducing the computing resources required to determine the phoneme labels of the voice request data to a certain extent.

[0014] In some embodiments of the present application, the phoneme label includes a target phoneme and a target prediction probability of the target phoneme, the prediction result includes a prediction vector, each vector element of the prediction vector includes the phoneme prediction results of the multiple semantic units, and the determining the phoneme label according to the second phoneme among the multiple first phonemes and the first probability of the second phoneme includes:

[0015] The target phoneme and the target prediction probability of the target phoneme are determined according to the second phoneme and the first probability of the second phoneme in the vector element at the same position in each of the prediction vectors to obtain the phoneme label.

[0016] Thus, in the implementation mode of the present application, the phoneme label of the semantic unit corresponding to the voice request data can be determined based on the second phoneme and the first probability of the second phoneme in each vector element at the same position in each prediction vector output by each natural language processing model.

[0017] In certain embodiments of the present application, the method further comprises:

[0018] A length alignment process is performed on each of the prediction vectors.

[0019] Thus, in the implementation manner of the present application, each prediction vector may be length aligned so that each prediction vector includes the same number of vector elements, thereby ensuring the robust determination of the phoneme label to a certain extent.

[0020] In certain embodiments of the present application, determining the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label according to the prediction result of each of the natural language processing models, and completing the processing of the voice request data includes:

[0021] According to the pre-trained generative model and the second phoneme with the highest first probability in each vector element, the semantic unit corresponding to the voice request data is predicted to obtain the semantic unit label.

[0022] Thus, in an embodiment of the present application, the semantic unit corresponding to the voice request data can be predicted based on the pre-trained generative model and the second phoneme with the highest first probability in each vector element, thereby obtaining the semantic unit label of the voice request data.

[0023] The present application embodiment provides a model training method, including:

[0024] Acquire a speech request sample and a sample label of the speech request sample, wherein the sample label includes a semantic unit label and a phoneme label corresponding to the semantic unit label, and the sample label is obtained by processing the speech request sample using the speech request processing method.

[0025] An on-board acoustic model is trained based on the voice request sample and the sample label, wherein the on-board acoustic model is used to recognize phonemes corresponding to the voice request.

[0026] In the model training method provided in the embodiment of the present application, the server can obtain voice request samples and sample labels of the voice request samples, and train an on-board acoustic model based on the voice request samples and the sample labels, wherein the on-board acoustic model is used to identify the phonemes corresponding to the voice requests.

[0027] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0028] In certain embodiments of the present application, the phoneme label includes a target phoneme and a target prediction probability of the target phoneme, and the vehicle-mounted acoustic model is trained according to the voice request sample and the sample label, including:

[0029] Determining, according to the vehicle-mounted acoustic model, a first semantic unit prediction result corresponding to the voice request sample;

[0030] Determining a first loss function value according to the first semantic unit prediction result and the semantic unit label of the voice request sample;

[0031] The vehicle-mounted acoustic model is trained according to the first loss function value.

[0032] Thus, in an embodiment of the present application, the first loss function value can be calculated by the on-board acoustic model through the first semantic unit prediction result output by the voice request sample and the semantic unit label in the voice request sample to train the on-board acoustic model.

[0033] In certain embodiments of the present application, the phoneme label includes a target phoneme and a target prediction probability of the target phoneme, and the vehicle-mounted acoustic model is trained according to the voice request sample and the sample label, including:

[0034] Determine, according to the vehicle-mounted acoustic model and the pre-trained teacher model, a second prediction result of the vehicle-mounted acoustic model and a third prediction result of the teacher model, wherein the second prediction result includes a second semantic unit prediction result and a first prediction probability for the target phoneme, and the third prediction result includes a third semantic unit prediction result and a second prediction probability for the target phoneme;

[0035] Determine a second loss function value according to the second semantic unit prediction result, the third phonetic unit prediction result, the first prediction probability and the second prediction probability;

[0036] The vehicle-mounted acoustic model is trained according to the second loss function value.

[0037] Thus, in an embodiment of the present application, the second loss function value can be calculated to train the on-board acoustic model by using the second semantic unit prediction result output by the teacher model through the voice request sample and the second prediction probability for the target phoneme, and the third semantic unit prediction result output by the on-board acoustic model through the voice request sample and the third prediction probability for the target phoneme.

[0038] The present application provides a voice interaction method, including:

[0039] Get voice request;

[0040] Predicting the phonemes corresponding to the voice request using a pre-trained vehicle-mounted acoustic model, wherein the vehicle-mounted acoustic model is trained using the above-mentioned model training method;

[0041] Determining, according to the phonemes, a natural language text corresponding to the voice request;

[0042] The voice interaction is completed according to the natural language text.

[0043] In the voice interaction method of the implementation mode of the present application, the vehicle can obtain a voice request and use a pre-trained on-board acoustic model to predict the phonemes corresponding to the voice request, and determine the natural language text corresponding to the voice request based on the phonemes, and complete the voice interaction based on the natural language text.

[0044] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0045] An embodiment of the present application provides a server, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned method for processing voice requests, or the above-mentioned model training method, or the above-mentioned voice interaction method is implemented.

[0046] An embodiment of the present application provides a vehicle, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned voice interaction method is implemented.

[0047] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, it implements the above-mentioned method for processing voice requests, or implements the above-mentioned model training method, or implements the above-mentioned voice interaction method.

[0048] The server, vehicle and computer-readable storage medium provided in the embodiments of the present application can determine the semantic unit labels and phoneme labels of the voice request data through the prediction results of multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0049] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 This is a schematic diagram of the in-vehicle voice interaction process;

[0052] Figure 2 is a schematic diagram of the acoustic model;

[0053] Figure 3 A flowchart of a method for processing a voice request in certain implementation modes of the present application;

[0054] Figure 4 A flowchart of a method for processing a voice request in certain implementation modes of the present application;

[0055] Figure 5 A schematic diagram of application scenarios in certain embodiments of the present application;

[0056] Figure 6 A schematic diagram of application scenarios in certain embodiments of the present application;

[0057] Figure 7 A schematic diagram of application scenarios in certain embodiments of the present application;

[0058] Figure 8 A schematic diagram of application scenarios in certain embodiments of the present application;

[0059] Fig. 9 A schematic diagram of application scenarios in certain embodiments of the present application;

[0060] Fig.10 A flowchart of a model training method in certain implementation modes of the present application;

[0061] Fig.11 A flowchart of a model training method in certain implementation modes of the present application;

[0062] Fig.12 A flowchart of a model training method in certain implementation modes of the present application;

[0063] Fig.13 A schematic diagram of application scenarios in certain embodiments of the present application;

[0064] Fig.14 A schematic diagram of application scenarios in certain embodiments of the present application;

[0065] Fig.15 This is a flowchart of the voice interaction method in certain implementation modes of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0067] With the introduction and widespread application of in-vehicle voice interaction, vehicles in related technologies are usually equipped with a pre-trained natural language processing model to understand and process the voice of the user in the car and output the corresponding natural language processing results. The vehicle can perform corresponding operations based on the natural language processing results output by the model, thereby completing the voice interaction with the user. For example, when the user says "turn on the air conditioner", the vehicle can use the pre-trained natural language processing model to recognize the captured user voice and control the vehicle air conditioner to turn on according to the recognition result.

[0068] For a clearer explanation of the process of in-vehicle voice interaction, please refer to Figure 1 , Figure 1 This is a schematic diagram of the in-vehicle voice interaction process. Figure 2 is a schematic diagram of the acoustic model. That is, Figure 1 and Figure 2 As shown, after the user speaks a voice in the cockpit, the vehicle can capture the voice to form audio data, and use the pre-trained acoustic model to encode, decode, and softmax classify the audio data to obtain the phoneme (or pinyin) prediction corresponding to the audio data. Then, the phoneme prediction is converted into text using the decoding graph converted from the pre-trained language model, thereby completing the conversion from speech to text. Finally, the display content of the central control screen is refreshed according to the converted text, and / or the text is used to control the operating status of one or more vehicle components.

[0069] However, unlike smart home devices such as smart speakers, the sound environment of a vehicle changes with factors such as the location of the vehicle, the degree of window raising, the wind speed outside the cabin, and the conversations inside the cabin, presenting a complex and changeable situation that is difficult to list. Figure 1 and Figure 2 As for the acoustic model shown, the acoustic model may be used in a complex, changeable, or even harsh sound environment, which poses great challenges to the acoustic model.

[0070] Furthermore, in order to enable the acoustic model to be used in complex, changeable, and even harsh sound environments, a traditional solution is to collect massive amounts of acoustic data and train the acoustic model with this massive amount of acoustic data so that the performance of the acoustic model is strong enough to cover all types of sound environments.

[0071] It is understandable that in order to ensure the training effect of the acoustic model, the massive amount of acoustic data collected needs to be labeled. It is also understandable that when there is a lot of acoustic data, data labeling is more difficult, and the time and human resources required for data labeling are both high.

[0072] Based on the above problems that may be encountered, please refer to Figure 3 , the embodiment of the present application provides a method for processing a voice request, including:

[0073] 01: Get the voice request data to be processed;

[0074] 02: Determine, based on multiple pre-trained natural language processing models, a prediction result of each natural language processing model for the voice request data, wherein the prediction result includes phoneme prediction results of multiple semantic units;

[0075] 03: According to the prediction results of each natural language processing model, the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label are determined to complete the processing of the voice request data.

[0076] The embodiment of the present application provides a device for processing voice requests. The method for processing voice requests of the embodiment of the present application can be implemented by the device for processing voice requests of the embodiment of the present application. Specifically, the processing device includes an acquisition module, a determination module and a processing module. Among them, the acquisition module is used to acquire the voice request data to be processed. The determination module is used to determine the prediction results of each natural language processing model for the voice request data according to multiple natural language processing models that have been trained in advance, and the prediction results include phoneme prediction results of multiple semantic units. The labeling module is used to determine the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label according to the prediction results of each natural language processing model, so as to complete the processing of the voice request data.

[0077] The embodiment of the present application also provides a server, which includes a memory and a processor. The method for processing voice requests of the embodiment of the present application can be implemented by the server of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain voice request data to be processed, and to determine the prediction results of each natural language processing model for the voice request data based on multiple natural language processing models that have been trained in advance, and to determine the semantic unit labels of the voice request data and the phoneme labels corresponding to the semantic unit labels based on the prediction results of each natural language processing model, and complete the processing of the voice request data, and the prediction results include the phoneme prediction results of multiple semantic units.

[0078] Specifically, in an embodiment of the present application, when the server obtains the voice request data, it may call multiple pre-trained natural language processing models to perform multiple predictions on the voice request data to obtain the prediction results of each natural language processing model for the voice request data, and then determine the semantic unit label corresponding to the voice request data and the phoneme label corresponding to the semantic unit label based on the prediction results of each natural language processing model, thereby completing the processing of the voice request data.

[0079] It is understood that, in the implementation of the present application, the voice request data can be set according to the actual situation. For example, in one example, the voice request data is the voice request collected online, such as the sound emitted when the user says the text "turn on the air conditioner". In another example, the voice request data is the voice request collected online and the natural language text corresponding to the voice request, such as the text "turn on the air conditioner" and the sound emitted when the user says the text "turn on the air conditioner".

[0080] It can also be understood that in the implementation of the present application, the multiple pre-trained natural language processing models can also be set according to actual conditions. For example, in one example, the multiple pre-trained natural language processing models are pre-trained large language models (LLM).

[0081] And, it can be understood that, in the implementation mode of the present application, the prediction result of the natural language processing model for the voice request data includes the phoneme prediction results of multiple semantic units. Among them, the semantic unit can be understood as a token, such as "character" in the Chinese context and "word" in the English context. Correspondingly, the phoneme of the semantic unit can be understood as pronunciation, such as "the pinyin of 'character'" in the Chinese context and "the phonetic symbol of 'word'" in the English context.

[0082] For example, in an example where the voice request data is a voice request collected online and the natural language text corresponding to the voice request, a natural language processing model predicts the natural language text "navigation to the department store", and the prediction result obtained is {"guidance":"dǎo","navigation":"háng","go":"qù","hundred":"bǎi","goods":"huò","business":"shāng","field":"cháng"}. In another example where the voice request data is a voice request collected online, a natural language processing model predicts a segment of voice request data, and the prediction result obtained is {"guidance":"dǎo","navigation":"háng","go":"qù","hundred":"bǎi","goods":"huò","business":"shāng","field":"cháng"}.

[0083] It can also be understood that in the implementation manner of the present application, the server may integrate the prediction results of multiple natural language processing models for voice request data to obtain semantic unit labels of the voice request data and phoneme labels corresponding to the semantic unit labels, and ensure that the semantic unit labels and phoneme labels of the voice request data can rely on the prediction results support of multiple natural language processing models, thereby ensuring the accuracy and reliability of the semantic unit labels and phoneme labels.

[0084] Furthermore, when different natural language processing models make predictions for the same voice request data, the prediction results of different natural language processing models may be different or similar. Therefore, in one example, the server can determine the semantic unit labels and phoneme labels based on the results with the largest number of identical results among the prediction results of each natural language processing model.

[0085] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0086] See also Figure 4 In some embodiments of the present application, the phoneme prediction result includes prediction probabilities of multiple phoneme predictions, and step 03 includes:

[0087] 030: According to the same phoneme prediction in the multiple phoneme predictions, merging the multiple phoneme predictions and the prediction probabilities of the multiple phoneme predictions to obtain multiple first phonemes and a first probability of each first phoneme;

[0088] 031: Determine a phoneme label according to a second phoneme among a plurality of first phonemes and a first probability of the second phoneme, wherein the second phoneme is a preset number of first phonemes with the highest first probability among the plurality of first phonemes.

[0089] The determination module of the implementation mode of the present application is also used to merge multiple phoneme predictions and the prediction probabilities of multiple phoneme predictions based on the same phoneme prediction in multiple phoneme predictions to obtain multiple first phonemes and the first probability of each first phoneme, and to determine the phoneme label based on the second phoneme in the multiple first phonemes and the first probability of the second phoneme, the second phoneme being a preset number of first phonemes with the highest first probability among the multiple first phonemes.

[0090] The processor of the embodiment of the present application is also used to merge multiple phoneme predictions and the prediction probabilities of multiple phoneme predictions based on the same phoneme prediction in multiple phoneme predictions to obtain multiple first phonemes and the first probability of each first phoneme, and to determine the phoneme label based on the second phoneme in the multiple first phonemes and the first probability of the second phoneme, the second phoneme being a preset number of first phonemes with the highest first probability among the multiple first phonemes.

[0091] Specifically, in order to reduce the amount of computational effort in voice request data annotation, in an embodiment of the present application, the prediction results of each natural language processing model can be combined in terms of phoneme predictions and prediction probabilities, and a preset number of phoneme predictions with the highest prediction probabilities can be selected from the combined phoneme predictions for subsequent phoneme label determination. In other words, the TopN phoneme predictions from the combined phoneme predictions can be used for subsequent phoneme label determination.

[0092] The merging process of “prediction probability of phoneme prediction” may include: assuming that the modeling unit of a model is 4 Chinese characters, which are divided into W1, W2, W3 and W4, the pinyin of W1 is PY1, the pinyin of W2 is PY2, the pinyin of W3 is PY3, and the pinyin of W4 is PY1, the model predicts a voice request data, and predicts that the voice request data is a Chinese character W T , and predict the Chinese character W T The probabilities of falling on W1, W2, W3 and W4 are Prep1, Prep2, Prep3 and Prep4 respectively. T The probability that the pinyin of falls on the pinyin of the four Chinese characters W1, W2, W3 and W4 (that is, the predicted probability of the phoneme prediction corresponding to the semantic unit) is also Prep1, Prep2, Prep3 and Prep4.

[0093] Furthermore, since the pinyin of W1 and W4 are both PY1, the probability of falling on W1 Prep1 and the probability of falling on W4 Prep4 can be combined, and then the Chinese character W T The probability that the pinyin falls on PY1, PY2 and PY3 (ie, the first probability of the first phoneme corresponding to the semantic unit) is Prep1+Prep4, Prep2 and Prep3 respectively.

[0094] It can be understood that the modeling unit can be understood as the number of semantic units used in natural language processing modeling, and can also be understood as the number of categories that can be divided into when natural language processing modeling classifies a semantic unit.

[0095] In addition, the selection process of the second phoneme may include: again taking the above-mentioned probabilities of W1, W2, W3 and W4 as Prep1, Prep2, Prep3 and Prep4 respectively, and setting Prep1+Prep4>Prep2>Prep3, the preset number is 2, then due to the Chinese character W predicted by the model T The probability that the pinyin falls on PY1, PY2 and PY3 are Prep1+Prep4, Prep2 and Prep3 respectively. Therefore, in the three phoneme prediction-prediction probability groups corresponding to Prep1+Prep4 of PY1, Prep2 of PY2 and Prep3 of PY3, the second phoneme is "PY1 corresponding to the highest value (Prep1+Prep4)" and "PY2 corresponding to Prep2 with the second highest value".

[0096] To clearly explain the implementation of this application, please refer to Figure 5 , Figure 5 This is a schematic diagram of an application scenario in some embodiments of the present application. That is, in one example, the open source big model 1 is implemented based on 5000 modeling units, and the open source big model 2 is implemented based on 6500 modeling units.

[0097] Further, open source big model 1 and open source big model 2 respectively predict the voice request data of "navigation to department store". Open source big model 1 predicts that the voice request data corresponds to 7 tokens (such as "navigation to department store"), and outputs the prediction probabilities of a total of 7*5000 pinyins corresponding to these 7 tokens, and one of the 7*1 pinyin sequences is "dao3 hang2 qu4 bai3 huo4 shang1 chang2". Among them, Figure 5 The "numbers that appear after letters" represent pinyin tones, such as "dao3" means "the third tone of 'dao'", which is "dǎo".

[0098] Correspondingly, the open source large model 2 predicts that the voice request data corresponds to 6 tokens (such as "navigation department store"), and outputs the prediction probabilities of a total of 6*5000 pinyins corresponding to these 6 tokens, one of which is a 7*1 pinyin sequence "dao3 hang2 bai3 huo4 shang1 chang2".

[0099] Further, for the prediction probability of 7*5000 pinyin output by the open source large model 1, the prediction probability of 1*5000 pinyin corresponding to each token is merged to merge 1*5000 pinyin into 1*1500 pinyin, so that the prediction probability of 1*5000 pinyin corresponding to each token is converted to the prediction probability of 1*1500 pinyin. Correspondingly, for the prediction probability of 6*6500 pinyin output by the open source large model 2, the prediction probability of 1*6500 pinyin corresponding to each token is merged to merge 1*5000 pinyin into 1*1500 pinyin, so that the prediction probability of 1*6500 pinyin corresponding to each token is converted to the prediction probability of 1*1500 pinyin.

[0100] Furthermore, for the prediction probability of 7*1500 pinyins of the open source model 1, the N with the highest probability values ​​are selected from the prediction probability of 1*1500 pinyins corresponding to each token, so that the prediction probability of 1*1500 pinyins corresponding to each token is converted to the prediction probability of 1*N pinyins, and then, the prediction probability of 7*1500 pinyins of the open source model 1 is converted to the prediction probability of 7*N pinyins. Correspondingly, for the prediction probability of 6*1500 pinyins of the open source model 2, the N with the highest probability values ​​are selected from the prediction probability of 1*1500 pinyins corresponding to each token, so that the prediction probability of 1*1500 pinyins corresponding to each token is converted to the prediction probability of 1*N pinyins, and then, the prediction probability of 6*1500 pinyins of the open source model 2 is converted to the prediction probability of 6*N pinyins.

[0101] Therefore, the server can determine the pinyin corresponding to the voice request data based on the prediction probabilities of the 7*N pinyins of the open source big model 1 and the prediction probabilities of the 7*N pinyins of the open source big model 2, thereby completing the pinyin labeling of the voice request data.

[0102] In this way, in an embodiment of the present application, the predictions of the same phoneme and the prediction probabilities of the same phoneme predictions in the prediction results output by the natural language processing model can be merged to obtain multiple first phonemes and the first probability of each first phoneme, and then the phoneme labels of the voice request data are determined based on a preset number of first phonemes with the highest first probability among the multiple first phonemes, thereby reducing the computing resources required to determine the phoneme labels of the voice request data to a certain extent.

[0103] In addition, it should be noted that the modeling unit can be a character or a pinyin. And, in the case where the modeling unit is a character, the phoneme merging, probability merging and second phoneme determination can be performed through the above steps 030-031, and in the case where the modeling unit is a pinyin, since the modeling unit is a pinyin, the situation where repeated phoneme predictions exist in the prediction results can be ignored, and then the above step 030 can be skipped, and step 031 can be directly executed to determine the phoneme label through the second phoneme.

[0104] It should also be noted that, in the implementation manner of the present application, the specific value of the preset number can be set according to the actual situation. For example, in one example, the preset number is a positive integer less than or equal to 10. In another example, the preset number is 10.

[0105] In some embodiments of the present application, the phoneme tag includes a target phoneme and a target prediction probability of the target phoneme, and the prediction result includes a prediction vector, each vector element of the prediction vector includes the phoneme prediction results of multiple semantic units, and then, step 031 includes:

[0106] According to the second phoneme and the first probability of the second phoneme in the vector element at the same position in each prediction vector, the target phoneme and the target prediction probability of the target phoneme are determined to obtain the phoneme label.

[0107] The processing module of the embodiment of the present application is also used to determine the target phoneme and the target prediction probability of the target phoneme according to the second phoneme and the first probability of the second phoneme in the vector element with the same position in each prediction vector to obtain the phoneme label.

[0108] The processor of the embodiment of the present application is also used to determine the target phoneme and the target prediction probability of the target phoneme according to the second phoneme and the first probability of the second phoneme in the vector element with the same position in each prediction vector to obtain the phoneme label.

[0109] That is, in an embodiment of the present application, for each prediction vector output by the natural language processing model, the server can determine the target phoneme and target prediction probability corresponding to the semantic unit of the voice request data at that position based on the second phoneme and the first probability in the vector element at the same position in each prediction vector.

[0110] To more clearly illustrate the implementation method of this application, please refer to Figure 5 , and see Figure 6 , Figure 6 Schematic diagram of application scenarios in certain embodiments of the present application. Figure 5As shown, the open source large language model 1 predicts the voice request data. The predicted voice request data may include 7 tokens, each token may correspond to the prediction probability of 1*5000 pinyins. After merging the pinyins and the prediction probabilities, each token may correspond to the prediction probability of 1*1500 pinyins. The 4 with the highest probability values ​​among the 1*1500 pinyin prediction probabilities corresponding to each token are retained, so that each token may correspond to the prediction probability of 1*4 pinyins.

[0111] Similar, such as Figure 5 As shown, the open source large language model 2 predicts the voice request data, and the predicted voice request data may include 6 tokens, each token may correspond to the prediction probability of 1*6500 pinyins, and after the pinyins and the prediction probabilities are merged, each token may correspond to the prediction probability of 1*1500 pinyins, and the 4 with the highest probability values ​​in the prediction probabilities of 1*1500 pinyins corresponding to each token are retained, so that each token may correspond to the prediction probability of 1*4 pinyins, that is, Figure 6 As shown in the upper right part of .

[0112] Furthermore, according to the prediction of the first token of the voice request data by the open source large language model 1, the probability (i.e., the first probability) that the pinyin of the first token falls on "dao3", "dao2", "dao4", and "dang3" (i.e., the second phoneme) is 0.6, 0.05, 0.05, and 0.04, respectively. Figure 6 In contrast, according to the prediction of the first token of the voice request data by the open source large language model 2, the probability that the pinyin of the first token falls on "dao3", "dang4", "dao1" and "tao1" are 0.7, 0.1, 0.1 and 0.05 respectively, that is, Figure 6 The upper right part shows the content.

[0113] Furthermore, for the pinyin of the first token, both the open source large language model 1 and the open source large language model 2 output the probability that the pinyin of the first token falls on "dao3", so the "probability that the pinyin of the first token output by the open source large language model 1 falls on 'dao3'" and the "probability that the pinyin of the first token output by the open source large language model 2 falls on 'dao3'" are merged to obtain the final probability that the pinyin of the first token falls on "dao3" (that is, the target prediction probability).

[0114] For example, Figure 6As shown in the lower part, we can perform a weighted sum based on the "probability that the pinyin of the first token output by the open source large language model 1 falls on 'dao3'" and the "probability that the pinyin of the first token output by the open source large language model 2 falls on 'dao3'" to make the probability that the first token falls on "dao3" 0.6*0.5+0.7*0.5, which is 0.65.

[0115] Also, except for the predictions of open source big language model 1 and open source big language 2 that the first token of the voice request data falls on "dao3", the predictions of open source big language model 1 and open source big language 2 that the first token of the voice request data falls on other pinyins are not repeated, so the predictions of open source big language model 1 and open source big language 2 that the first token of the voice request data falls on other pinyins can be directly used as the final target phoneme and target prediction probability. For example, since open source big language model 2 does not output the probability that the first token of the voice request data falls on "dao2", the prediction probability of the target phoneme "dao2" corresponding to the first token of the voice request data is "the product of '0.05 output by open source big language model 1' and 'the weight 0.5 corresponding to open source big language model 1'", that is, 0.025.

[0116] In addition, it is understandable that the "weight corresponding to the open source large language model 1" can be set according to the actual situation, and the above 0.5 is only one of the feasible examples. In another example, the server determines the label of the voice request data through the prediction results of X open source large language models (including the open source large language model 1), and the weight of the open source large language model 1 is 1 / X.

[0117] Thus, in the implementation mode of the present application, the phoneme label of the semantic unit corresponding to the voice request data can be determined based on the second phoneme and the first probability of the second phoneme in each vector element at the same position in each prediction vector output by each natural language processing model.

[0118] In some embodiments of the present application, the voice interaction method further includes:

[0119] Perform length alignment on each prediction vector.

[0120] The speech interaction device of the embodiment of the present application further includes an alignment module. The alignment module is used to perform length alignment processing on each prediction vector.

[0121] The processor of the embodiment of the present application is also used to perform length alignment processing on each prediction vector.

[0122] Specifically, in order to avoid the situation where the prediction vectors output by different natural language processing models have different lengths, when determining the phoneme label of the semantic unit corresponding to the voice request data based on the vector elements at the same position of each prediction vector, the prediction vectors output by some natural language processing models include the vector elements at a certain position, but the prediction vectors output by other natural language processing models do not include the vector elements at that position, resulting in errors in the determination of the phoneme labels. Therefore, in an embodiment of the present application, each prediction vector can be length aligned to make the length of each prediction vector the same, so that each prediction vector includes the same number of vector elements.

[0123] To more clearly illustrate the implementation of this application, please refer to Figure 7 , Figure 7 Schematic diagram of application scenarios in certain embodiments of the present application. Figure 7 As shown, in one example, the server calls 5 open source large language models to predict the voice request data of "navigation to the department store", and after merging the prediction results of the decoder output of each open source large language model with the same phoneme probability, the 10 first phonemes with the highest first probability are selected from the multiple first phonemes of each semantic unit of the voice request data of each open source large language model, thereby obtaining the prediction probability of 7*10 pinyins corresponding to 7 tokens output by the open source large language model 1 (i.e., decoder 1), the prediction probability of 6*10 pinyins corresponding to 6 tokens output by the open source large language model 2 (i.e., decoder 2), the prediction probability of 8*10 pinyins corresponding to 8 tokens output by the open source large language model 3 (i.e., decoder 3), the prediction probability of 7*10 pinyins corresponding to 7 tokens output by the open source large language model 4 (i.e., decoder 4), and the prediction probability of 7*10 pinyins corresponding to 7 tokens output by the open source large language model 5 (i.e., decoder 5).

[0124] Next, the prediction vector output by each open source large language model is aligned in length using the placeholder “*”. For example, for the prediction vector output by the open source large language model 1 containing 7 semantic units, in order to match the prediction vector of the open source large language model 1 with the prediction vector of the open source large language model 3 containing 8 semantic units, a placeholder “*” is added to the end of the prediction vector output by the open source large language model 1. For example, a placeholder “*” is added to the end of “dao3 hang2qu4 bai3 huo4 shang1 chang2” to obtain “dao3 hang2 qu4 bai3huo4 shang1 chang2*”.

[0125] Next, the length alignment of each open source large language model is completed through the placeholder "*", so that the output of each open source large language model is converted to "the prediction probability of 8*10 pinyins corresponding to 8 tokens". Since the 5 open source large language models output a prediction probability of 1*10 pinyins for any semantic unit of the voice request data, and thus any semantic unit of the voice request data corresponds to a prediction probability of 1*50 pinyins, the prediction probabilities of 1*50 pinyins corresponding to each semantic unit are merged and weighted, so that the prediction probability of 1*50 pinyins corresponding to each semantic unit is converted to the prediction probability of 1*M pinyins, thereby obtaining the prediction probability of 8*M pinyins corresponding to the voice request data. Among them, M is a positive integer less than or equal to 50, such as Figure 7 16, 15, 14, etc.

[0126] Furthermore, since the semantic unit label corresponding to the voice request data is "a semantic unit combination excluding the 8th semantic unit", the prediction probability of 1*M pinyins corresponding to the last semantic unit in the prediction probability of 8*M pinyins is removed, that is, Figure 7 Shown in the lower right corner.

[0127] Thus, in the implementation manner of the present application, each prediction vector may be length aligned so that each prediction vector includes the same number of vector elements, thereby ensuring the robust determination of the phoneme label to a certain extent.

[0128] In certain embodiments of the present application, step 03 includes:

[0129] According to the pre-trained generative model and the second phoneme with the highest first probability in each vector element, the semantic unit corresponding to the voice request data is predicted to obtain the semantic unit label.

[0130] The processing module of the implementation mode of the present application is also used to predict the semantic unit corresponding to the voice request data based on the pre-trained generative model and the second phoneme with the highest first probability in each vector element to obtain the semantic unit label.

[0131] The processor of the embodiment of the present application is also used to predict the semantic unit corresponding to the voice request data based on the pre-trained generative model and the second phoneme with the highest first probability in each vector element to obtain the semantic unit label.

[0132] Specifically, in the implementation of the present application, a pre-trained generative model can be used to predict the second phoneme with the highest first probability in each vector element output by each natural language processing model, thereby obtaining the semantic unit label of the voice request data. Figure 8 , Figure 8 A schematic diagram of an application scenario in certain embodiments of the present application.

[0133] For example, in the prediction vector output by a certain open source large language model, the pinyin with the highest probability value among the 1*10 pinyin prediction probabilities corresponding to the first token of the voice request data is "dao3", the pinyin with the highest probability value among the 1*10 pinyin prediction probabilities corresponding to the second token is "hang2", ..., and the pinyin with the highest probability value among the 1*10 pinyin prediction probabilities corresponding to the seventh token is "chang2", so "dao3 hang2 qu4 bai3 huo4shang1chang2" can be used as the prediction vector with the highest score of the open source large language model, and the vector is input into the generative model. Similarly, the prediction vectors with the highest scores of other open source large language models are input into the generative model.

[0134] In one example, the generative model is a BERT (Bidirectional Encoder Representations from Transformers) model.

[0135] Thus, in the implementation mode of the present application, the semantic unit corresponding to the voice request data can be predicted based on the pre-trained generative model and the second phoneme with the highest first probability in each vector element, thereby obtaining the semantic unit label of the voice request data.

[0136] See also Fig. 9 , Fig. 9 Schematic diagram of application scenarios in certain embodiments of the present application. That is, in one example, for each of the massive voice request data collected online, the server can use multiple open source large language models to predict the voice request data, predict each semantic unit (i.e. text) corresponding to the voice request data, and after converting the semantic unit into pinyin, predict the predicted probability of each semantic unit (i.e. Fig. 9 The softmax in ) determines the predicted probability of the phoneme prediction corresponding to each semantic unit.

[0137] Next, multiple semantic units output by each large language model are aligned.

[0138] Then, for the prediction probabilities of the phoneme predictions corresponding to each aligned semantic unit, the TopN phoneme predictions corresponding to each semantic unit with the highest prediction probabilities are selected.

[0139] Then, a weighted sum is performed on the TopN phoneme predictions corresponding to each semantic unit and the prediction probabilities of these TopN phoneme predictions to obtain multiple phoneme predictions corresponding to each semantic unit and these multiple weighted probabilities, thereby obtaining the phoneme label corresponding to each semantic unit.

[0140] Furthermore, the semantic unit with the highest prediction probability among all the phoneme predictions corresponding to each semantic unit output by each large language model is input into the generative model. In other words, the semantic unit combination with the highest score of each large language model is input into the generative model to obtain the semantic unit label corresponding to the voice request data.

[0141] Therefore, the word-probability pair consisting of the semantic unit label and the phoneme label corresponding to each semantic unit is used as the label of the semantic request data to complete the labeling of the semantic request data.

[0142] See also Fig.10 Corresponding to the above-mentioned method for processing voice requests, the embodiment of the present application further provides a model training method, including:

[0143] 11: Obtain a voice request sample and a sample label of the voice request sample, wherein the sample label includes a semantic unit label and a phoneme label corresponding to the semantic unit label, and the sample label is obtained by processing the voice request sample using the voice request processing method described above;

[0144] 12: Based on the voice request samples and sample labels, an on-board acoustic model is trained, where the on-board acoustic model is used to identify the phonemes corresponding to the voice request.

[0145] The embodiment of the present application provides a model training device. The model training method of the embodiment of the present application can be implemented by the model training device of the embodiment of the present application. Specifically, the training device includes a sample acquisition module and a model training module. Among them, the sample acquisition module is used to obtain voice request samples and sample labels of voice request samples, and the sample labels include semantic unit labels and phoneme labels corresponding to the semantic unit labels. The sample labels are obtained by processing the voice request samples using the above-mentioned voice request processing method. The model training module is used to train an on-board acoustic model based on the voice request samples and sample labels, and the on-board acoustic model is used to identify the phonemes corresponding to the voice request.

[0146] The embodiment of the present application also provides a server, which includes a memory and a processor. The model training method of the embodiment of the present application can be implemented by the server of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain voice request samples and sample labels of voice request samples, and is used to train an on-board acoustic model based on the voice request samples and sample labels, wherein the sample labels include semantic unit labels and phoneme labels corresponding to the semantic unit labels, and the sample labels are obtained by processing the voice request samples using the above-mentioned voice request processing method, and the on-board acoustic model is used to identify the phonemes corresponding to the voice request.

[0147] Specifically, in the implementation manner of the present application, the server may use the voice request data collected online as the voice request samples to be labeled, and use the prediction results of multiple natural language understanding models for the voice request samples to determine the phoneme labels and semantic unit labels of the voice request samples. Then, based on the voice request samples, the phoneme labels of the voice request samples, and the semantic unit labels of the voice request samples, the model training is performed to obtain an on-board acoustic model that can be deployed in the vehicle and can recognize the phonemes corresponding to the voice requests.

[0148] It can be understood that, in the implementation manner of the present application, the specific process of obtaining the phoneme label and semantic unit label of the voice request sample can refer to the aforementioned content, and will not be repeated here to avoid repetition.

[0149] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0150] See also Fig.11 In some embodiments of the present application, the phoneme tag includes a target phoneme and a target predicted probability of the target phoneme, and step 12 includes:

[0151] 120: Determine, according to the vehicle-mounted acoustic model, a first semantic unit prediction result corresponding to the voice request sample;

[0152] 121: determining a first loss function value according to the first semantic unit prediction result and the semantic unit label of the voice request sample;

[0153] 122: According to the first loss function value, a vehicle-mounted acoustic model is trained.

[0154] The model training module of the implementation mode of the present application is also used to determine the first semantic unit prediction result corresponding to the voice request sample based on the vehicle-mounted acoustic model, and to determine the first loss function value based on the first semantic unit prediction result and the semantic unit label of the voice request sample, and to train the vehicle-mounted acoustic model based on the first loss function value.

[0155] The processor of the embodiment of the present application is also used to determine the first semantic unit prediction result corresponding to the voice request sample based on the vehicle-mounted acoustic model, and to determine the first loss function value based on the first semantic unit prediction result and the semantic unit label of the voice request sample, and to train the vehicle-mounted acoustic model based on the first loss function value.

[0156] Specifically, in order to improve the training efficiency of the on-board acoustic model, in an embodiment of the present application, the on-board acoustic model can be used to predict the prediction result of the semantic unit corresponding to the voice request sample, calculate the corresponding first loss function value, and train the on-board acoustic model through the first loss function value.

[0157] Therefore, in an embodiment of the present application, the server can calculate the first loss function value based on the token_label (semantic unit label) corresponding to the sample label of a voice request sample and the token_model (first semantic unit prediction result) obtained by predicting the voice request sample by the on-board acoustic model.

[0158] In one example, the first loss function value is a cross entropy loss function value.

[0159] Thus, in an embodiment of the present application, the first loss function value can be calculated by the on-board acoustic model through the first semantic unit prediction result output by the voice request sample and the semantic unit label in the voice request sample to train the on-board acoustic model.

[0160] See also Fig.12 In some embodiments of the present application, the phoneme tag includes a target phoneme and a target predicted probability of the target phoneme, and step 12 includes:

[0161] 123: Determine, according to the vehicle-mounted acoustic model and the pre-trained teacher model, a second prediction result of the vehicle-mounted acoustic model and a third prediction result of the teacher model, wherein the second prediction result includes a second semantic unit prediction result and a first prediction probability for a target phoneme, and the third prediction result includes a third semantic unit prediction result and a second prediction probability for the target phoneme;

[0162] 124: Determine a second loss function value according to the second semantic unit prediction result, the third phonetic unit prediction result, the first prediction probability, and the second prediction probability;

[0163] 125: According to the second loss function value, the vehicle acoustic model is trained.

[0164] The model training module of the implementation mode of the present application is also used to determine the second prediction result of the vehicle-mounted acoustic model and the third prediction result of the teacher model based on the vehicle-mounted acoustic model and the pre-trained teacher model, and to determine the second loss function value based on the second semantic unit prediction result, the third speech unit prediction result, the first prediction probability and the second prediction probability, and to train the vehicle-mounted acoustic model based on the second loss function value, the second prediction result includes the second semantic unit prediction result and the first prediction probability for the target phoneme, and the third prediction result includes the third semantic unit prediction result and the second prediction probability for the target phoneme.

[0165] The processor of the embodiment of the present application is also used to determine the second prediction result of the on-board acoustic model and the third prediction result of the teacher model based on the on-board acoustic model and the pre-trained teacher model, and to determine the second loss function value based on the second semantic unit prediction result, the third speech unit prediction result, the first prediction probability and the second prediction probability, and to train the on-board acoustic model based on the second loss function value, the second prediction result includes the second semantic unit prediction result and the first prediction probability for the target phoneme, and the third prediction result includes the third semantic unit prediction result and the second prediction probability for the target phoneme.

[0166] Specifically, in the implementation of the present application, in order to ensure that the vehicle can robustly call the vehicle acoustic model, the server can complete the training of the vehicle acoustic model by distillation learning. That is, the second loss function value is calculated by the first prediction result of the pre-trained teacher model for the voice request sample and the second prediction result of the untrained vehicle acoustic model (or student model) for the voice request sample, and the vehicle acoustic model is trained by the second loss function value.

[0167] In one example, the second loss function value is a KL divergence (Kullback-Leibler divergence) loss function value.

[0168] It can be understood that in the implementation mode of the present application, the phoneme label includes the target phoneme and the target prediction probability of the target phoneme, such as the aforementioned 1*16 represents the target prediction probability of 16 target phonemes corresponding to a token, such as the aforementioned 1*16 represents the target prediction probability of 16 target phonemes corresponding to a token, and the aforementioned 1*15 represents the target prediction probability of 15 target phonemes corresponding to a token.

[0169] Therefore, the server can determine the token_model_teacher (second semantic unit prediction result) obtained by the teacher model to predict the voice request sample, the prediction probability prep_model_teacher (first prediction probability) of token_model_teacher in the target phoneme phoneme_lable according to the "target prediction probability prep_lable (target prediction probability) of multiple target phonemes phoneme_lable (target phoneme) corresponding to each token_label (semantic unit label)" in the sample label of a voice request sample, and determine the token_model_student (third semantic unit prediction result) obtained by the on-board acoustic model to predict the voice request sample, the prediction probability prep_model_student (second prediction probability) of token_model_student in the target phoneme phoneme_lable, and calculate the second loss function value through token_model_teacher, token_model_student, prep_model_teacher and prep_model_student.

[0170] It can be understood that compared with the method of calculating the loss through all the results output by the teacher model and all the results output by the vehicle-mounted acoustic model, the implementation method of the present application requires fewer computing resources when calculating the second loss function value through the above-mentioned ttoken_model_teacher, token_model_student, prep_model_teacher and prep_model_student, thereby improving the model training efficiency to a certain extent.

[0171] Thus, in an embodiment of the present application, the second loss function value can be calculated to train the on-board acoustic model by using the second semantic unit prediction result output by the teacher model through the voice request sample and the second prediction probability for the target phoneme, and the third semantic unit prediction result output by the on-board acoustic model through the voice request sample and the third prediction probability for the target phoneme.

[0172] In certain embodiments of this application, please refer to Fig.13 and Fig.14 , Fig.13 and Fig.14 All of them are schematic diagrams of application scenarios in certain embodiments of the present application. That is, in one example, since the server can obtain sample labels of voice request samples collected online through the prediction results of multiple natural language processing models, the server can complete the training of the vehicle-mounted acoustic model through a small number of sample labels of voice request samples manually annotated and a large number of sample labels of voice request samples annotated through the prediction results of multiple natural language processing models.

[0173] Furthermore, during the training process, the target prediction probability of the target phoneme (i.e. Fig.13 The softmax label in ) calculates the corresponding loss function value to train the model.

[0174] Optionally, compared with the sample labels of voice request samples annotated by the prediction results of multiple natural language processing models, the sample labels of voice request samples manually annotated have higher credibility. Furthermore, in one example, when the sample labels of voice request samples are manually annotated, the weight of the first loss function value can be set to 1, and the weight of the second loss function value can be set to 0. After the first loss function value, the weight of the first loss function value, the second loss function value, and the weight of the second loss function value are weightedly summed to obtain the final loss function value, the final loss function value is used to train the on-board acoustic model.

[0175] Also, when the sample labels of voice request samples are annotated by the prediction results of multiple natural language processing models, the weight of the first loss function value can be set to 0.8, and the weight of the second loss function value can be set to 0.2. The first loss function value, the weight of the first loss function value, the second loss function value, and the weight of the second loss function value are weighted and summed to obtain the final loss function value, and the on-board acoustic model is trained using the final loss function value.

[0176] See also Fig.15 Corresponding to the above-mentioned model training method, the embodiment of the present application also provides a voice interaction method, including:

[0177] 21: Get voice request;

[0178] 22: Using a pre-trained vehicle acoustic model, predict the phonemes corresponding to the voice request, wherein the vehicle acoustic model is trained using the above model training method;

[0179] 23: Determine the natural language text corresponding to the voice request based on the phonemes;

[0180] 24: Complete voice interaction based on natural language text.

[0181] The embodiment of the present application provides a voice interaction device. The voice interaction method of the embodiment of the present application can be implemented by the voice interaction device of the embodiment of the present application. Specifically, the interaction device includes a request acquisition module, a phoneme prediction module, a text determination module and an interaction module. Among them, the request acquisition module is used to obtain a voice request. The phoneme prediction module is used to predict the phonemes corresponding to the voice request using a pre-trained on-board acoustic model, and the on-board acoustic model is trained by the above-mentioned model training method. The text determination module is used to determine the natural language text corresponding to the voice request based on the phonemes. The interaction module is used to complete the voice interaction based on the natural language text.

[0182] The embodiment of the present application also provides a vehicle, which includes a memory and a processor. The voice interaction method of the embodiment of the present application can be implemented by the vehicle of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain a voice request, and to predict the phoneme corresponding to the voice request using a pre-trained on-board acoustic model, and to determine the natural language text corresponding to the voice request based on the phoneme, and to complete the voice interaction based on the natural language text, and the on-board acoustic model is trained by the above-mentioned model training method.

[0183] Specifically, in the implementation of the present application, after the vehicle-mounted acoustic model is deployed, the vehicle can capture the voice request issued by the user to obtain the voice request, and call the vehicle-mounted acoustic model to process the voice request to obtain the phoneme corresponding to the voice request. Then, the phoneme is converted into the corresponding text to complete the voice recognition. Finally, voice interaction is performed through the converted text, such as generating a control instruction for turning on the air conditioner through text, and refreshing the display content of the vehicle's central control screen through text to let the user know the execution status of the vehicle's voice interaction function.

[0184] Thus, in the implementation mode of the present application, the semantic unit labels and phoneme labels of the voice request data can be determined by the prediction results of the multiple natural language processing models that have been pre-trained for the voice request data, so that the semantic unit labeling and phoneme labeling of the voice request data can be achieved, and the accuracy and reliability of the semantic unit labels and phoneme labels can be guaranteed to a certain extent. Therefore, the degree of reliance on manual labor for the sample labels required for the training of the on-board acoustic model can be reduced, so that the on-board acoustic model can be trained with a sufficient number of voice request samples and corresponding sample labels, and the performance of the on-board acoustic model can be guaranteed, and then when the voice interaction is performed using the phonemes predicted by the on-board acoustic model, the quality of the voice interaction can be guaranteed.

[0185] In addition, it can be understood that the training method of the vehicle-mounted acoustic model can refer to the aforementioned model training method, and to avoid repetition, it will not be repeated here.

[0186] It is also understood that the specific method of converting phonemes to text can be set according to actual conditions. For example, in one example, the vehicle can complete the conversion of phonemes to text through a pre-trained on-board language model. For example, in another example, the vehicle can convert phonemes to text through a "decoding graph converted from a pre-trained language model".

[0187] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, it implements the above-mentioned method for processing voice requests, or implements the above-mentioned model training method, or implements the above-mentioned voice interaction method.

[0188] In the description of this specification, the descriptions with reference to the terms "specifically", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0189] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0190] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for processing a voice request, It is characterized in that include: Get the voice request data to be processed; Determine, according to a plurality of pre-trained natural language processing models, a prediction result of each of the natural language processing models for the voice request data, wherein the prediction result includes phoneme prediction results of a plurality of semantic units; According to the prediction results of each of the natural language processing models, the semantic unit tags of the voice request data and the phoneme tags corresponding to the semantic unit tags are determined to complete the processing of the voice request data.

2. The method according to claim 1, It is characterized in that The phoneme prediction result includes prediction probabilities of a plurality of phoneme predictions, and determining, according to the prediction results of each of the natural language processing models, a semantic unit label of the voice request data and a phoneme label corresponding to the semantic unit label, and completing the processing of the voice request data, including: According to the same phoneme prediction among the plurality of phoneme predictions, the plurality of phoneme predictions and the prediction probabilities of the plurality of phoneme predictions are combined to obtain a plurality of first phonemes and a first probability of each of the first phonemes; The phoneme label is determined according to a second phoneme among the plurality of first phonemes and the first probability of the second phoneme, wherein the second phoneme is a preset number of first phonemes among the plurality of first phonemes whose first probability is the highest.

3. The method according to claim 2, It is characterized in that The phoneme label includes a target phoneme and a target prediction probability of the target phoneme, the prediction result includes a prediction vector, each vector element of the prediction vector includes the phoneme prediction results of the multiple semantic units, and the determining the phoneme label according to a second phoneme among the multiple first phonemes and the first probability of the second phoneme includes: The target phoneme and the target prediction probability of the target phoneme are determined according to the second phoneme and the first probability of the second phoneme in the vector element at the same position in each of the prediction vectors to obtain the phoneme label.

4. The method according to claim 3, It is characterized in that The method further comprises: A length alignment process is performed on each of the prediction vectors.

5. The method according to claim 3, It is characterized in that The step of determining the semantic unit label of the voice request data and the phoneme label corresponding to the semantic unit label according to the prediction result of each of the natural language processing models, and completing the processing of the voice request data, includes: According to the pre-trained generative model and the second phoneme with the highest first probability in each vector element, the semantic unit corresponding to the voice request data is predicted to obtain the semantic unit label.

6. A model training method, It is characterized in that include: Obtaining a speech request sample and a sample label of the speech request sample, wherein the sample label includes a semantic unit label and a phoneme label corresponding to the semantic unit label, and the sample label is obtained by processing the speech request sample using the method described in any one of claims 1 to 5, An on-board acoustic model is trained based on the voice request sample and the sample label, wherein the on-board acoustic model is used to recognize phonemes corresponding to the voice request.

7. The method according to claim 6, It is characterized in that The phoneme label includes a target phoneme and a target prediction probability of the target phoneme, and the vehicle-mounted acoustic model is trained according to the voice request sample and the sample label, including: Determining, according to the vehicle-mounted acoustic model, a first semantic unit prediction result corresponding to the voice request sample; Determining a first loss function value according to the first semantic unit prediction result and the semantic unit label of the voice request sample; The vehicle-mounted acoustic model is trained according to the first loss function value.

8. The method according to claim 6, It is characterized in that The phoneme label includes a target phoneme and a target prediction probability of the target phoneme, and the vehicle-mounted acoustic model is trained according to the voice request sample and the sample label, including: Determine, according to the vehicle-mounted acoustic model and the pre-trained teacher model, a second prediction result of the vehicle-mounted acoustic model and a third prediction result of the teacher model, wherein the second prediction result includes a second semantic unit prediction result and a first prediction probability for the target phoneme, and the third prediction result includes a third semantic unit prediction result and a second prediction probability for the target phoneme; Determine a second loss function value according to the second semantic unit prediction result, the third phonetic unit prediction result, the first prediction probability and the second prediction probability; The vehicle-mounted acoustic model is trained according to the second loss function value.

9. A voice interaction method, It is characterized in that include: Get voice request; Predicting the phonemes corresponding to the voice request using a pre-trained vehicle-mounted acoustic model, wherein the vehicle-mounted acoustic model is trained by the method according to any one of claims 6 to 8; Determining, according to the phonemes, a natural language text corresponding to the voice request; The voice interaction is completed according to the natural language text.

10. A server, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A vehicle, It is characterized in that The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to claim 9 is implemented.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 9 is implemented.