Voice interaction method, vehicle and computer readable storage medium

Abstract

The invention discloses a voice interaction method, a vehicle and a computer readable storage medium, and the method comprises the steps: obtaining voice request data and gender information, carrying out the down-sampling and natural language processing of the voice request data according to the gender information, and determining a voice request data processing result; and completing voice interaction according to a voice request data processing result. Thus, down-sampling can be carried out based on the gender information in the processing process of the voice request data, the processing difficulty of the voice request data is reduced, the processing efficiency of the voice request data is improved, the accuracy of voice request data processing results of users with different genders is guaranteed, and the user experience is improved. The accuracy of the voice request data processing result is guaranteed, the voice interaction function based on the voice request data processing result is stably executed, and the user experience of the voice interaction function and the vehicle driving experience are guaranteed.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a voice interaction method, a vehicle and a computer-readable storage medium. Background Art

[0002] With the emergence of user needs and the upgrading of natural language processing technology, voice interaction functions are gradually applied in various vehicle models, allowing users to control vehicles through voice. However, due to the differences in the way different users speak, it is difficult for vehicles to accurately recognize the voices of different users, which in turn affects the execution effect of the in-vehicle voice interaction function, and users have a poor experience in using the in-vehicle voice interaction function. Summary of the invention

[0003] The present application provides a voice interaction method, a vehicle and a computer-readable storage medium.

[0004] The present application provides a voice interaction method, including:

[0005] Acquire voice request data and gender information, wherein the gender information is used to indicate the gender of a triggering object of the voice request data;

[0006] Downsampling and natural language processing are performed on the voice request data according to the gender information to determine a voice request data processing result;

[0007] The voice interaction is completed according to the voice request data processing result.

[0008] Thus, in the implementation manner of the present application, the voice request data can be downsampled and naturally language processed according to the gender information to obtain the voice request data processing result, so that the voice request data processing process can be downsampled based on the gender information, so that the difficulty of processing the voice request data can be reduced, the processing efficiency of the voice request data can be improved, and the voice request data can be processed in a targeted manner based on the gender information, so that the accuracy of the voice request data processing results of users of different genders can be guaranteed, the voice interaction function based on the voice request data processing result can be robustly executed, and the user experience of the voice interaction function and the vehicle driving experience can be guaranteed.

[0009] In certain embodiments of the present application, downsampling and natural language processing are performed on the voice request data according to the gender information to determine the voice request data processing result, including:

[0010] Downsampling the voice request data according to the gender information;

[0011] According to the down-sampling result, the natural language processing is performed on the voice request data to determine the voice request data processing result.

[0012] In this way, in the implementation mode of the present application, natural language processing of voice request data can be performed based on the downsampling results of the voice request data, which reduces the difficulty of natural language processing of voice request data to a certain extent.

[0013] In some implementations of the present application, the voice request data includes multiple consecutive frames of sub-data, and downsampling the voice request data according to the gender information includes:

[0014] In a case where the gender information is the first information, determining the downsampling result according to the even-numbered frame sub-data in the voice request data; and / or

[0015] In the case where the gender information is the second information, the downsampling result is determined according to odd-numbered frame sub-data in the voice request data.

[0016] Thus, in an embodiment of the present application, when the gender information is the first information, the even-numbered frame sub-data within the voice request data can be sampled to determine the result of downsampling, and / or when the gender information is the second information, the odd-numbered frame sub-data within the voice request data can be sampled to determine the result of downsampling, and then the natural language processing of the voice request data can be completed based on the even-numbered frame sub-data or the odd-numbered frame sub-data, thereby reducing the difficulty of natural language processing of the voice request data to a certain extent.

[0017] In certain embodiments of the present application, performing the natural language processing on the voice request data according to the downsampling result to determine the voice request data processing result includes:

[0018] Determining a query mapping vector according to the downsampling result;

[0019] Determining a key mapping vector and a value mapping vector according to the voice request data;

[0020] According to the query mapping vector, the key mapping vector and the value mapping vector, natural language processing is performed on the voice request data to determine a processing result of the voice request data.

[0021] Thus, in an embodiment of the present application, a query mapping vector can be determined according to the result of the downsampling, and a key mapping vector and a value mapping vector can be determined according to the voice request data, that is, natural language processing is performed on the voice request data according to the query mapping vector, the key mapping vector and the value mapping vector, thereby realizing natural language processing based on self-attention, and ensuring the accuracy and reliability of the voice request data processing results to a certain extent.

[0022] In certain embodiments of the present application, downsampling and natural language processing are performed on the voice request data according to the gender information to determine the voice request data processing result, including:

[0023] The downsampling and natural language processing are performed on the voice request data according to the gender information and a pre-trained natural language processing model to determine a processing result of the voice request data.

[0024] Thus, in the implementation manner of the present application, the downsampling and natural language processing of the voice request data can be completed through a pre-trained natural language processing model, so that the processing of the voice request data can be completed based on the neural network model, thereby ensuring the accuracy and reliability of the voice request data processing results to a certain extent.

[0025] In certain embodiments of the present application, the natural language processing model includes a plurality of model layers connected in sequence, each of the model layers includes a plurality of model sub-blocks connected in sequence, and the downsampling and natural language processing are performed on the voice request data according to the gender information and the pre-trained natural language processing model to determine the processing result of the voice request data, including:

[0026] According to the first model sub-block in each of the model layers and the gender information, the first input data is downsampled and natural language processed to obtain the second input data of the next model sub-block, wherein the first input data of the first model sub-block in the first model layer is the voice request data;

[0027] According to each model sub-block after the first model sub-block in each model layer, the natural language processing is performed on the second input data to determine the voice request data processing result.

[0028] Thus, in the implementation manner of the present application, the input data can be downsampled by the first model sub-block in each model layer to reduce the difficulty of processing the input data, and the input data can be processed for natural speech by each model sub-block after the first model sub-block in each model layer to ensure the processing effect of the input data, thereby reducing the difficulty of processing the voice request data to a certain extent, and ensuring the accuracy and reliability of the voice request processing data to a certain extent.

[0029] In certain embodiments of the present application, the number of hidden layer neural network units in the later model layer is higher than the number of hidden layer neural network units in the previous model layer.

[0030] Thus, in the implementation mode of the present application, high-dimensional features can be obtained based on more hidden layer neural network units, which ensures the processing effect of natural language processing to a certain extent.

[0031] In certain embodiments of the present application, the training step of the natural language processing model includes:

[0032] Obtaining voice request sample data and a gender information label corresponding to the voice request sample data;

[0033] Performing model training according to the voice request sample data to determine a reference model;

[0034] The reference model is trained according to the voice request sample data and the gender information label to determine the natural language processing model.

[0035] In this way, in the implementation mode of the present application, voice request sample data and the gender information label corresponding to the voice request sample data can be obtained, and model training can be performed based on the voice request sample data to determine the reference model, and the reference model can be trained based on the voice request sample data and the gender information label to determine the natural language processing model, so that the training effect and model performance of the natural language processing model can be guaranteed.

[0036] An embodiment of the present application provides a vehicle, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned voice interaction method is implemented.

[0037] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the above-mentioned voice interaction method is implemented.

[0038] The vehicle and computer-readable storage medium provided in the embodiments of the present application can downsample and perform natural language processing on voice request data according to gender information to obtain voice request data processing results, so that downsampling can be performed based on gender information during the processing of voice request data, thereby reducing the difficulty of processing voice request data, improving the processing efficiency of voice request data, and also allowing voice request data to be processed in a targeted manner based on gender information, thereby ensuring the accuracy of voice request data processing results for users of different genders, and stably executing voice interaction functions based on voice request data processing results, thereby ensuring the user experience of voice interaction functions and vehicle driving experience.

[0039] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 A schematic diagram of the speech recognition process;

[0042] Figure 2 is a schematic diagram of a speech recognition model;

[0043] Figure 3 A flowchart of a voice interaction method in certain implementation modes of the present application;

[0044] Figure 4 A flowchart of a voice interaction method in certain implementation modes of the present application;

[0045] Figure 5 A flowchart of a voice interaction method in certain implementation modes of the present application;

[0046] Figure 6 A schematic diagram of application scenarios in certain embodiments of the present application;

[0047] Figure 7 A flowchart of a voice interaction method in certain implementation modes of the present application;

[0048] Figure 8 A flowchart of a voice interaction method in certain implementation modes of the present application;

[0049] Fig. 9 A schematic diagram of application scenarios in certain embodiments of the present application;

[0050] Fig.10A flowchart of a voice interaction method in certain implementation modes of the present application;

[0051] Fig.11 A schematic diagram of application scenarios in certain embodiments of the present application;

[0052] Fig.12 A schematic diagram of an application scenario in certain embodiments of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0054] In order to improve the performance of the in-vehicle voice interaction function, the solutions in the relevant technologies can be simply divided into two types: one is the voice interaction solution based on the cloud model, and the other is the voice interaction solution based on the vehicle side. In the voice interaction solution based on the cloud model, a natural language processing model with a large number of parameters is deployed in the cloud for the vehicle to call and complete the voice interaction with the user. In the voice interaction solution based on the vehicle side, the vehicle can complete the voice interaction with the user through a natural language processing model with a small number of parameters deployed locally. It is understandable that these two solutions have their own advantages and disadvantages in terms of robustness and accuracy.

[0055] It is also understandable that the computing resources on both the cloud and vehicle sides are limited. Therefore, when the number of parameters and computing power are limited by computing resources, optimizing the model structure to improve model performance is a more important algorithmic problem.

[0056] In addition, since the voice data required for voice recognition model training is difficult to collect and relatively scarce, the difficulty and scarcity of voice data suitable for in-vehicle voice interaction scenarios are even more serious. In addition, when the data required for training is relatively scarce, there is less voice data for users with higher vocal cord frequencies, such as women and children, while there is little voice data for mid-bass users, such as adult male users, resulting in the voice recognition accuracy for users with higher vocal cord frequencies being generally lower than that for users with mid-bass.

[0057] In addition, in the related art, the speech recognition process and model structure of the speech recognition model can be found in Figure 1 and Figure 2 , Figure 1 A schematic diagram of the speech recognition process. Figure 2 is a schematic diagram of the speech recognition model. That is, Figure 1 and Figure 2As shown in FIG. 1 , after acquiring the user's voice, feature extraction is performed on the user's voice to obtain T*80 voice features, where T is the number of frames or the time dimension, and 80 is the dimension of the filterbank feature.

[0058] Furthermore, in order to reduce computer power, the speech features are compressed in the time dimension after passing through the downsampling module. The downsampling module is completed through the conv2d operator, and the kernel-stride of the conv2d operator is usually set to 4, that is, 1 frame is convolved every 4 frames, and the time dimension of the speech features is converted to T / 4. Subsequently, the speech features with a time dimension of T / 4 are input into N conformer-block modules. Among them, each conformer-block module has the same size and is simply stacked, the hidden layer size is H (including the size of each layer h1, h2, h3...), and the time dimension is T / 4. Finally, the output results of the N conformer-block modules are predicted by softmax to obtain the acoustic probability matrix, and then the acoustic probability matrix is ​​decoded to obtain the result, and the output frame number is T / 4.

[0059] It is understandable that before the downsampling module, the T*80 speech features contain more original information, and downsampling the T*80 speech features will cause greater information loss and performance loss. Also, after deep abstraction, the time dimension of the deep features is higher and there is structural redundancy.

[0060] Based on the above problems that may be encountered, please refer to Figure 3 , the present application embodiment provides a voice interaction method, including:

[0061] 01: Acquire voice request data and gender information, where the gender information is used to indicate the gender of the triggering object of the voice request data;

[0062] 02: Downsample and perform natural language processing on the voice request data based on the gender information to determine the processing result of the voice request data;

[0063] 03: Complete voice interaction based on the voice request data processing results.

[0064] The embodiment of the present application provides a voice interaction device. The voice interaction method of the embodiment of the present application can be implemented by the voice interaction device of the embodiment of the present application. Specifically, the position determination device includes an acquisition module, a processing module and an interaction module. Among them, voice request data and gender information are acquired, and the gender information is used to indicate the gender of the triggering object of the voice request data. The processing module is used to downsample and perform natural language processing on the voice request data according to the gender information, and determine the processing result of the voice request data. The interaction module is used to complete the voice interaction according to the processing result of the voice request data.

[0065] The embodiment of the present application also provides a vehicle, which includes a memory and a processor. The voice interaction method of the embodiment of the present application can be implemented by the vehicle of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain voice request data and gender information, downsample and perform natural language processing on the voice request data according to the gender information, determine the processing result of the voice request data, and complete the voice interaction according to the processing result of the voice request data, and the gender information is used to indicate the gender of the triggering object of the voice request data.

[0066] Specifically, in an implementation manner of the present application, the vehicle can obtain voice request data of a user in the cabin, as well as gender information that can characterize the gender of the user, and then downsample and perform natural language processing on the voice request data based on the gender information to obtain a voice request data processing result, and finally complete voice interaction with the user based on the voice request data processing result.

[0067] For example, the vehicle can capture the sound in the cabin to obtain the user voice request data when the user in the cabin wants to control the start of the vehicle air conditioning and thus says the sentence "turn on the air conditioning". Also, when it is determined that the gender information of the user in the cabin is "female", the previously obtained voice request data can be downsampled and natural language processing can be performed based on the gender information of "female" to process part of the voice request data, reduce the difficulty of voice request data processing, and achieve targeted processing of voice request data for female users. Finally, the control instruction for controlling the start of the air conditioning can be output, that is, the result of the voice request data processing, and the action of executing the air conditioning start-up according to the control instruction can be combined with the voice interaction of the user in the cabin.

[0068] Thus, in the implementation manner of the present application, the voice request data can be downsampled and naturally language processed according to the gender information to obtain the voice request data processing result, so that the voice request data processing process can be downsampled based on the gender information, so that the difficulty of processing the voice request data can be reduced, the processing efficiency of the voice request data can be improved, and the voice request data can be processed in a targeted manner based on the gender information, so that the accuracy of the voice request data processing results of users of different genders can be guaranteed, the voice interaction function based on the voice request data processing result can be robustly executed, and the user experience of the voice interaction function and the vehicle driving experience can be guaranteed.

[0069] It is understood that, in the embodiment of the present application, the gender information may indicate the gender of the speaker in the vehicle cabin space, or may indicate the gender of the triggering object of the voice request data. In one example, the gender information may represent three categories: female, male, and child.

[0070] It can also be understood that in the embodiments of the present application, the voice request data can represent the sound emitted by the user in the cabin. In one example, the voice request data is a continuous multi-frame filterbank feature, that is, after feature extraction of the collected voice of the user in the cabin, a multi-frame filterbank feature is obtained as the voice request data.

[0071] See also Figure 4 In certain embodiments of the present application, step 02 includes:

[0072] 020: Downsample the voice request data based on gender information;

[0073] 021: According to the downsampling result, natural language processing is performed on the voice request data to determine the processing result of the voice request data.

[0074] The processing module of the implementation mode of the present application is also used to downsample the voice request data according to the gender information, and to perform natural language processing on the voice request data according to the downsampling result to determine the voice request data processing result.

[0075] The processor of the embodiment of the present application is also used to downsample the voice request data according to the gender information, and to perform natural language processing on the voice request data according to the downsampling result to determine the voice request data processing result.

[0076] Specifically, in the embodiment of the present application, the vehicle can downsample the voice request data according to the gender information. For example, when the speaker is male and the voice request data is composed of multiple frames of data (such as multiple frames of filterbank feature data), the even-numbered frames of the voice request data are sampled as the result of downsampling. When the speaker is female and the voice request data is composed of multiple frames of data, the odd-numbered frames of the voice request data are sampled as the result of downsampling.

[0077] After completing the downsampling of the voice request data, the vehicle can perform natural language processing on the voice request data based on the downsampling results, such as calculating the attention score based on the downsampling results, and then processing the voice request data based on the calculated attention score, thereby realizing natural language processing of the voice request data based on the attention mechanism.

[0078] In this way, in the implementation mode of the present application, natural language processing of voice request data can be performed based on the downsampling results of the voice request data, which reduces the difficulty of natural language processing of voice request data to a certain extent.

[0079] In some embodiments of the present application, the voice request data includes continuous multi-frame sub-data, and step 02 includes:

[0080] In the case where the gender information is the first information, determining a downsampling result according to even-numbered frame sub-data in the voice request data; and / or

[0081] When the gender information is the second information, the downsampling result is determined according to the odd-numbered frame sub-data in the voice request data.

[0082] The processing module of the implementation mode of the present application is also used to determine the downsampling result based on the even-numbered frame sub-data in the voice request data when the gender information is the first information; and / or to determine the downsampling result based on the odd-numbered frame sub-data in the voice request data when the gender information is the second information.

[0083] The processor of the embodiment of the present application is also used to determine the downsampling result based on the even-numbered frame sub-data in the voice request data when the gender information is the first information; and / or to determine the downsampling result based on the odd-numbered frame sub-data in the voice request data when the gender information is the second information.

[0084] Specifically, in the implementation manner of the present application, the vehicle may sample even-numbered frame sub-data or odd-numbered frame sub-data within the voice request data according to the specific gender of the speaker.

[0085] In one example, when the speaker is a user with a high vocal frequency such as a female or a child, all odd-numbered frame sub-data in the voice request data are sampled as the result of downsampling. In another example, when the speaker is a user with a low vocal frequency such as a male, all even-numbered frame sub-data in the voice request data are sampled as the result of downsampling.

[0086] Thus, in an embodiment of the present application, when the gender information is the first information, the even-numbered frame sub-data within the voice request data can be sampled to determine the result of downsampling, and / or when the gender information is the second information, the odd-numbered frame sub-data within the voice request data can be sampled to determine the result of downsampling, and then the natural language processing of the voice request data can be completed based on the even-numbered frame sub-data or the odd-numbered frame sub-data, thereby reducing the difficulty of natural language processing of the voice request data to a certain extent.

[0087] See also Figure 5 In some implementations of the present application, step 021 includes:

[0088] 0210: Determine the query mapping vector according to the downsampling result;

[0089] 0211: Determine a key mapping vector and a value mapping vector according to the voice request data;

[0090] 0212: Perform natural language processing on the voice request data according to the query mapping vector, the key mapping vector, and the value mapping vector to determine a processing result of the voice request data.

[0091] The processing module of the implementation mode of the present application is also used to determine the query mapping vector based on the downsampling result, and to determine the key mapping vector and the value mapping vector based on the voice request data, and to perform natural language processing on the voice request data based on the query mapping vector, the key mapping vector and the value mapping vector to determine the voice request data processing result.

[0092] The processor of the embodiment of the present application is also used to determine a query mapping vector based on the downsampling result, and to determine a key mapping vector and a value mapping vector based on the voice request data, and to perform natural language processing on the voice request data based on the query mapping vector, the key mapping vector and the value mapping vector to determine the voice request data processing result.

[0093] To more clearly illustrate the implementation of this application, please refer to Figure 6 , Figure 6 Schematic diagram of application scenarios in certain embodiments of the present application. Figure 6 As shown, for the voice request data X containing t frame sub-data, the voice request data X is downsampled. For example, when the speaker is a user with a higher vocal frequency such as a female or a child, all odd-numbered frame sub-data in the t frame sub-data are sampled as the downsampling result Y, and the time dimension is converted from t to t / 2.

[0094] Next, the downsampled result Y is used to calculate the query mapping vector W Q At the same time, the key vector W is calculated using the voice request data X K and value vector W V . Among them, the query mapping vector W Q The corresponding time dimension is t / 2, and the key vector W K The corresponding time dimension and value vector W V The corresponding time dimension is t.

[0095] Next, according to the query mapping vector W Q , key vector W K and the value vector W V , calculate the attention score Attention.

[0096] Among them, the calculation formula of the attention score Attention can be shown as follows, namely:

[0097]

[0098] Therefore, after obtaining the attention score Attention, corresponding natural language processing such as convolution processing can be performed based on the attention score Attention.

[0099] In this way, in the implementation mode of the present application, the query mapping vector can be determined according to the downsampling result, and the key mapping vector and the value mapping vector can be determined according to the voice request data, that is, the voice request data is subjected to natural language processing according to the query mapping vector, the key mapping vector and the value mapping vector, thereby realizing natural language processing based on self-attention, and ensuring the accuracy and reliability of the voice request data processing results to a certain extent.

[0100] See also Figure 7 In certain embodiments of the present application, step 02 includes:

[0101] 022: Based on the gender information and the pre-trained natural language processing model, downsample and perform natural language processing on the voice request data to determine the processing result of the voice request data.

[0102] The processing module of the implementation mode of the present application is also used to downsample and perform natural language processing on the voice request data according to the gender information and the pre-trained natural language processing model to determine the processing result of the voice request data.

[0103] The processor of the implementation mode of the present application is also used to downsample and perform natural language processing on the voice request data according to the gender information and the pre-trained natural language processing model, and determine the processing result of the voice request data.

[0104] Specifically, in the implementation manner of the present application, downsampling and natural language processing of voice request data can be completed according to a pre-trained natural language processing model, thereby ensuring the accuracy of the voice request data processing results.

[0105] In one example, the natural language processing model is a comformer model.

[0106] Thus, in the implementation manner of the present application, the downsampling and natural language processing of the voice request data can be completed through a pre-trained natural language processing model, so that the processing of the voice request data can be completed based on the neural network model, thereby ensuring the accuracy and reliability of the voice request data processing results to a certain extent.

[0107] See also Figure 8 In certain embodiments of the present application, the natural language processing model includes a plurality of model layers connected in sequence, each model layer includes a plurality of model sub-blocks connected in sequence, and the above step 022 includes:

[0108] 0220: down-sampling and natural language processing the first input data according to the first model sub-block and the gender information in each model layer to obtain the second input data of the next model sub-block, wherein the first input data of the first model sub-block in the first model layer is the voice request data;

[0109] 0221: According to each model sub-block after the first model sub-block in each model layer, natural language processing is performed on the second input data to determine the voice request data processing result.

[0110] The processing module of the implementation mode of the present application is also used to downsample and perform natural language processing on the first input data according to the first model sub-block and gender information in each model layer to obtain the second input data of the subsequent model sub-block, and to perform natural language processing on the second input data according to each model sub-block after the first model sub-block in each model layer to determine the voice request data processing result, and the first input data of the first model sub-block in the first model layer is the voice request data.

[0111] The processor of the embodiment of the present application is also used to downsample and perform natural language processing on the first input data according to the first model sub-block and gender information in each model layer to obtain the second input data of the subsequent model sub-block, and to perform natural language processing on the input data according to each model sub-block after the first model sub-block in each model layer to determine the voice request data processing result, and the first input data of the first model sub-block in the first model layer is the voice request data.

[0112] Specifically, in the implementation manner of the present application, the natural language processing model can be divided into a plurality of model layers connected in sequence, and each layer includes a plurality of model sub-blocks connected in sequence.

[0113] Furthermore, the first model sub-block in each model layer can downsample the input data according to the gender information, and perform natural language processing on the input data according to the downsampling result. Wherein, for the first model layer, the input data of the first model sub-block in the model layer is the above-mentioned voice request data, and for any model layer from the second to the last model layer, the input data of the first model sub-block in the model layer can be the output result of the last model sub-block in the previous model layer. And, for each module sub-block located after the first model sub-block in each model layer, the module sub-block can perform natural language processing such as convolution on the output result of the previous module sub-block.

[0114] Thus, in the implementation manner of the present application, the input data can be downsampled by the first model sub-block in each model layer to reduce the difficulty of processing the input data, and the input data can be processed for natural speech by each model sub-block after the first model sub-block in each model layer to ensure the processing effect of the input data, thereby reducing the difficulty of processing the voice request data to a certain extent, and ensuring the accuracy and reliability of the voice request processing data to a certain extent.

[0115] To more clearly illustrate the implementation of this application, please refer to Fig. 9 , Fig. 9 Schematic diagram of application scenarios in certain embodiments of the present application. Fig. 9 For example, Fig. 9 A natural language processing model including three model layers and each model layer including three model sub-blocks comformer-block is shown. The first, second and third model sub-blocks in the first model layer are comformer-blocks respectively. 1-1 、comformer-block 1-2 、comformer-block 1-3 , let the first, second and third model sub-blocks in the second model layer be comformer-block 2-1 、comformer-block 2-2 、comformer-block 2-3 , taking the first model layer and the second model layer as an example, we have:

[0116] After extracting the features of the user's voice in the cockpit and obtaining T*80 (T is the number of frames and also the time dimension, "80" refers to the dimension of the filterbank feature) frames of voice feature data, the comformer-block 1-1 According to the gender information, the even-numbered frame sub-data or the odd-numbered frame sub-data in the speech feature data can be sampled as the down-sampling result, and then the query vector W can be calculated according to the down-sampling result. q , and calculate the key vector W based on the speech feature data of T frames K and value vector W V , and then, according to the query vector W q , key vector W K and value vector W V Attention scores are calculated to perform natural language processing on the speech feature data of T frames.

[0117] Next, the compressor-block 1-2 Can be directly connected to the comformer-block1-1 The output of the natural language processing, such as according to the comformer-block 1-1 The output of the calculation results is used to calculate the attention score, and the attention score is used to calculate the comformer-block 1-1 The output of the rectifier block is processed by convolution. 1-3 Can be directly connected to the comformer-block 1-2 The output results are processed by natural language processing.

[0118] Then, the comformer-block 2-1 According to gender information, sample comformer-block 1-3 The even-numbered frame sub-data or odd-numbered frame sub-data in the output result of is used as the down-sampling result, and then the query vector W is calculated according to the down-sampling result. q , and calculate the key vector W based on the speech feature data of T frames K Sum value vector W V , and, according to the query vector W q , key vector W K and value vector W V Calculate attention scores to composer-block 1-3 The output results are processed by natural language processing.

[0119] Then, the compressor-block 2-2 Can be directly connected to the comformer-block 2-1 The output of the natural language processing, such as according to the comformer-block 2-1 The output of the calculation results is used to calculate the attention score, and the attention score is used to calculate the comformer-block 2-1 The output of the rectifier block is processed by convolution. 2-3 Can be directly connected to the comformer-block 2-2 The output results are processed by natural language processing.

[0120] It can be understood that the functions of each model sub-block in the third model layer can be found in comformer-block 2-1 、comformer-block 2-2 and comformer-block 2-3, to avoid repetition, it will not be described in detail here. It can also be understood that the softmax probability can be calculated according to the output result of the last model sub-block in the third model layer to obtain the acoustic probability matrix, and the speech recognition result, that is, the text recognition result of the speech request data, can be obtained according to the acoustic probability matrix.

[0121] In addition, it can be understood that for the speech feature data with a dimension of T*80, down-sampled according to the gender information is performed when it is sent to the first conformer-block of the first module layer, while the subsequent conformer-blocks do not perform sampling. The time dimension of the final output speech feature data is T / 2. In other words, the speech feature data is processed by the first module layer, and the time dimension changes from T to T / 2.

[0122] It can also be understood that for the speech feature data output by the first module layer, the speech feature data is downsampled according to the gender information when entering the first conformer-block of the second model layer, while the subsequent conformer-blocks do not perform sampling. The time dimension of the final output speech feature data is T / 4. In other words, the speech feature data is processed by the second module layer, and the time dimension changes from T / 2 to T / 4.

[0123] Also, it can be understood that, for the speech feature data output by the second module layer, the speech feature data is downsampled according to the gender information when entering the first conformer-block of the third model layer, while the subsequent conformer-blocks do not perform sampling, and the time dimension of the final output speech feature data is T / 8. In other words, after the speech feature data is processed by the third module layer, the time dimension changes from T / 4 to T / 8.

[0124] Furthermore, since the time dimension decreases with the increase of model depth, in order to ensure reliable processing of high-dimensional features, in one example, the number of hidden neural network units in the later model layer is higher than the number of hidden neural network units in the previous model layer. Fig. 9 , the hidden layer neural network units in the first model layer are H / 2, the hidden layer neural network units in the second model layer are H, and the hidden layer neural network units in the third model layer are H*2.

[0125] Thus, in the implementation mode of the present application, high-dimensional features can be obtained based on more hidden layer neural network units, which ensures the processing effect of natural language processing to a certain extent.

[0126] It is understandable that Fig. 9For the top of the pyramid-shaped composer model (i.e., the first model layer), the speech feature data is not extracted through the hidden layer and is scattered in the features of different T dimensions. Therefore, in order to avoid information loss, large-scale downsampling is abandoned.

[0127] Furthermore, for Fig. 9 For the middle tower (i.e. the second model layer) of the inverted pyramid-shaped conformer model, the speech feature data is initially extracted, the information in different T dimensions is concentrated, and the sensitivity to the T dimension is reduced. At this time, downsampling is increased and the model size is expanded to ensure that enough information is retained without increasing the amount of calculation.

[0128] Furthermore, for Fig. 9 When the bottom of the pyramid-shaped composer model (i.e., the third model layer) is reached, after two consecutive downsamplings in the T dimension, the temporal connection of the speech feature data is concentrated in the features at a certain time T. At this time, the downsampling intensity can be further increased, and the model size can be further expanded to increase the model modeling capability. It can be understood that compared with the first and second model layers, the model size of the third model layer is larger, but because the T dimension is lower, the total amount of calculation has not increased.

[0129] Therefore, based on Fig. 9 The inverted pyramid-shaped compformer model can retain the shallow time dimension information as much as possible and compress the deep time dimension information, so that the model redundancy can be reduced and the modeling ability can be used in important places. At the same time, by compressing the time dimension and enlarging the size, the modeling ability is increased, and the same amount of calculation has better results. In addition, because it is downsampling the time dimension instead of reducing the dimension through convolution, it can save the model parameters and efficiently discard the T / 2 features.

[0130] See also Fig.10 In certain embodiments of the present application, the training step of the natural language processing model includes:

[0131] 11: Obtain voice request sample data and gender information labels corresponding to the voice request sample data;

[0132] 12: Perform model training based on voice request sample data and determine the reference model;

[0133] 13: Train the reference model based on the voice request sample data and gender information labels to determine the natural language processing model.

[0134] The voice interaction device of the embodiment of the present application further includes a sample acquisition module, a first training module and a second training module. The sample acquisition module is used to acquire voice request sample data and gender information labels corresponding to the voice request sample data. The first training module is used to perform model training based on the voice request sample data and determine a reference model. The second training module is used to train the reference model based on the voice request sample data and the gender information label and determine a natural language processing model.

[0135] The processor of the embodiment of the present application is also used to obtain voice request sample data and gender information labels corresponding to the voice request sample data, and to perform model training based on the voice request sample data to determine a reference model, and to train the reference model based on the voice request sample data and the gender information labels to determine a natural language processing model.

[0136] Specifically, in the implementation manner of the present application, the training process of the natural language processing model may include two parts, one is pre-training (Pre-Train), and the other is fine-tuning (FuneTuning, FT) training.

[0137] Specifically, in one example, based on a pre-trained gender discriminator, the gender of the voice request sample data is determined to mark the gender information label of the voice request sample data as one of the three labels: female, child and male. Then, in the pre-training stage, the voice request sample data and random gender information are input into the natural language processing model to be trained, so that the natural language processing model to be trained randomly samples the odd-numbered frame sub-data or even-numbered frame sub-data of the voice request sample data according to the random gender information to calculate the query vector Q, and performs natural language processing according to the query vector Q to output the result, so that in the pre-training stage, the natural language processing model to be trained can improve the situation of small number of high-pitched training data (i.e., voice request sample data with gender information labels marked as female or child) by sharing parameters.

[0138] Next, in the fine-tuning stage, the voice request sample data and the real gender information labels of the voice request sample data are annotated and input into the natural language processing model completed in the pre-training stage, that is, the reference model mentioned above, so that the model can learn "to sample the even-numbered frame sub-data in the voice request data when the gender information is female or child, and to sample the odd-numbered frame sub-data in the voice request data when the gender information is male", thereby completing the fine-tuning training of the natural language processing model.

[0139] Further, in one example, see Fig.12 , Fig.12This is a schematic diagram of an application scenario in some embodiments of the present application. That is, after the fine-tuned and trained natural language processing model is deployed to the vehicle, the gender information of the vehicle owner can be obtained during the actual prediction process of the model for downsampling of the voice request data.

[0140] In one example, the vehicle may periodically capture images inside the cabin and / or capture voice information of the user inside the cabin, and send the images inside the cabin and / or the voice information of the user inside the cabin to a server. The server then identifies the gender information of the received images inside the cabin and / or the voice information of the user inside the cabin to output the gender information of the user inside the cabin, and sends the gender information to the vehicle, so that the vehicle can downsample the acquired voice request data according to the gender information periodically sent down by the server.

[0141] In one example, the vehicle may identify the acquired voice request data through a locally deployed gender recognition model to identify the gender information of the speaker corresponding to the voice request data.

[0142] In this way, in the implementation mode of the present application, voice request sample data and the gender information label corresponding to the voice request sample data can be obtained, and model training can be performed based on the voice request sample data to determine the reference model, and the reference model can be trained based on the voice request sample data and the gender information label to determine the natural language processing model, so that the training effect and model performance of the natural language processing model can be guaranteed.

[0143] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which implements the above-mentioned voice interaction method when the computer program is executed by one or more processors.

[0144] In the description of this specification, the descriptions with reference to the terms "specifically", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0145] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0146] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A voice interaction method, characterized in that: include: Acquire voice request data and gender information, wherein the gender information is used to indicate the gender of a triggering object of the voice request data; Downsampling and natural language processing are performed on the voice request data according to the gender information to determine a voice request data processing result; The voice interaction is completed according to the voice request data processing result.

2. The method according to claim 1, characterized in that The downsampling and natural language processing of the voice request data according to the gender information to determine the voice request data processing result includes: Downsampling the voice request data according to the gender information; According to the down-sampling result, the natural language processing is performed on the voice request data to determine the voice request data processing result.

3. The method according to claim 2, characterized in that The voice request data includes a plurality of continuous frames of sub-data, and down-sampling the voice request data according to the gender information includes: In a case where the gender information is the first information, determining the downsampling result according to the even-numbered frame sub-data in the voice request data; and / or In the case where the gender information is the second information, the downsampling result is determined according to the odd-numbered frame sub-data in the voice request data.

4. The method according to claim 2, characterized in that: The performing the natural language processing on the voice request data according to the down-sampling result to determine the voice request data processing result includes: Determining a query mapping vector according to the downsampling result; Determining a key mapping vector and a value mapping vector according to the voice request data; According to the query mapping vector, the key mapping vector and the value mapping vector, natural language processing is performed on the voice request data to determine a processing result of the voice request data.

5. The method according to claim 1, characterized in that The downsampling and natural language processing of the voice request data according to the gender information to determine the voice request data processing result includes: The downsampling and natural language processing are performed on the voice request data according to the gender information and a pre-trained natural language processing model to determine a processing result of the voice request data.

6. The method according to claim 5, characterized in that The natural language processing model includes a plurality of model layers connected in sequence, each of the model layers includes a plurality of model sub-blocks connected in sequence, and performing the downsampling and natural language processing on the voice request data according to the gender information and the pre-trained natural language processing model to determine the processing result of the voice request data, including: According to the first model sub-block in each of the model layers and the gender information, the first input data is downsampled and natural language processed to obtain the second input data of the next model sub-block, wherein the first input data of the first model sub-block in the first model layer is the voice request data; According to each model sub-block after the first model sub-block in each model layer, the natural language processing is performed on the second input data to determine the voice request data processing result.

7. The method according to claim 5, characterized in that The number of hidden layer neural network units in the later model layer is higher than the number of hidden layer neural network units in the earlier model layer.

8. The method according to claim 5, characterized in that The training steps of the natural language processing model include: Obtaining voice request sample data and a gender information label corresponding to the voice request sample data; Performing model training according to the voice request sample data to determine a reference model; The reference model is trained according to the voice request sample data and the gender information label to determine the natural language processing model.

9. A vehicle, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 8 is implemented.

Images