Voice interaction method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide a voice interaction method and device, electronic equipment and readable storage medium, and relate to the technical field of computer. In the embodiments of the present application, keyword detection can be performed on the target voice by a keyword detection model subjected to supplementary training. Since the keyword detection model is subjected to supplementary training based on at least the audio frames in a predetermined range in the first sample, and the predetermined range takes the end frame corresponding to the keyword in the first sample as the starting point and takes a predetermined number of frames as the length of the predetermined range, the trigger time point of the keyword will be concentrated in each audio frame in the predetermined range when the keyword detection model subjected to supplementary training performs voice interaction, thereby avoiding the problems of early triggering and delayed triggering, and improving the accuracy of the trigger time point of the keyword.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a voice interaction method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the development of computer technology, electronic devices are becoming increasingly intelligent. Currently, most electronic devices support voice-based human-computer interaction. When users interact with electronic devices via voice, the devices need to accurately and promptly recognize the user's voice.

[0003] In this system, after receiving a user's voice, the electronic device performs speech recognition. When the device detects specific keywords in the user's voice, it executes corresponding actions based on those keywords. However, in related technologies, the electronic device still suffers from problems such as premature or delayed keyword triggering when recognizing user voice. Summary of the Invention

[0004] In view of this, embodiments of this application provide a voice interaction method, device, electronic device, and readable storage medium to avoid the problems of premature or delayed keyword triggering, thereby improving the accuracy of keyword triggering timing.

[0005] Firstly, a voice interaction method is provided, the method comprising:

[0006] Identify the target speech.

[0007] Based on the keyword detection model, keyword detection is performed on the target speech to determine the keyword detection result. The keyword detection model is supplemented by training on at least a predetermined range of audio frames in the first sample. The predetermined range takes the end frame corresponding to the keyword in the first sample as the starting point and a predetermined number of frames as the length of the predetermined range.

[0008] In response to detecting that the target speech includes the keyword, the pre-stored instruction corresponding to the keyword is executed.

[0009] In some embodiments, the keyword detection model is supplemented with training based on the following steps:

[0010] Obtain a pre-trained keyword detection model and a first training set, wherein the first training set includes at least the first sample, the first sample label corresponding to the first sample, and the range label of the predetermined range corresponding to the first sample.

[0011] Based on the range marker, audio frames within a predetermined range are determined in the first sample.

[0012] The model parameters of the keyword detection model are adjusted based on the audio frames within the predetermined range and the first sample label.

[0013] In some embodiments, adjusting the model parameters of the keyword detection model based on the audio frames within the predetermined range and the first sample label includes:

[0014] The first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model.

[0015] Determine the target probabilities corresponding to the audio frames within the predetermined range in the first probability vector.

[0016] Max pooling is performed on each of the target probabilities to determine the second probability vector.

[0017] The model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label.

[0018] In some embodiments, the keyword detection model is pre-trained based on the following steps:

[0019] Obtain a second training set, which includes at least a second sample and the second sample label corresponding to the second sample.

[0020] The second sample is input into the initial keyword detection model to determine the third probability vector output by the initial keyword detection model.

[0021] The model parameters of the initial keyword detection model are adjusted based on the third probability vector and the second sample label.

[0022] In some embodiments, the method further includes:

[0023] Obtain the sample to be processed and the corresponding speech text.

[0024] Force alignment is performed on the sample to be processed and the speech text to determine the first sample and the predetermined range corresponding to the first sample.

[0025] In some embodiments, the forced alignment of the sample to be processed and the speech text includes:

[0026] The sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed.

[0027] Based on the first text probability distribution and the speech text, determine the second text probability distribution of the speech text in the sample to be processed.

[0028] Based on the second text probability distribution, the sample to be processed and the speech text are forcibly aligned.

[0029] In some embodiments, determining the second text probability distribution of the speech text in the sample to be processed, based on the first text probability distribution and the speech text, includes:

[0030] Determine the maximum probability of each character in the spoken text in the first text probability distribution.

[0031] Based on the maximum probability, determine the target position of each character in the speech text in the first text probability distribution.

[0032] Based on the target location, determine the matching path of the speech text in the first text probability distribution, and then determine the second text probability distribution.

[0033] Secondly, a voice interaction device is provided, the device comprising:

[0034] The target speech determination module is configured to determine the target speech.

[0035] The keyword detection module is configured to execute a pre-trained keyword detection model to detect keywords in the target speech and determine the keyword detection result. The keyword detection model is supplemented by training on at least a predetermined range of audio frames in the first sample. The predetermined range starts from the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length.

[0036] The keyword execution module is configured to execute a pre-stored instruction corresponding to the keyword in response to detecting that the target speech includes the keyword.

[0037] In some embodiments, the keyword detection model is supplemented by training based on the following modules:

[0038] The first acquisition module is configured to acquire a pre-trained keyword detection model and a first training set, wherein the first training set includes at least the first sample, the first sample label corresponding to the first sample, and the range marker of the predetermined range corresponding to the first sample.

[0039] The first determining module is configured to perform the task of determining audio frames within a predetermined range in the first sample based on the range marker.

[0040] The supplementary training module is configured to adjust the model parameters of the keyword detection model based on the audio frames within the predetermined range and the first sample label.

[0041] In some embodiments, the supplementary training module is specifically configured to perform:

[0042] The first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model.

[0043] Determine the target probabilities corresponding to the audio frames within the predetermined range in the first probability vector.

[0044] Max pooling is performed on each of the target probabilities to determine the second probability vector.

[0045] The model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label.

[0046] In some embodiments, the keyword detection model is pre-trained based on the following modules:

[0047] The second acquisition module is configured to acquire a second training set, which includes at least a second sample and a second sample label corresponding to the second sample.

[0048] The second determining module is configured to input the second sample into the initial keyword detection model and determine the third probability vector output by the initial keyword detection model.

[0049] The pre-training module is configured to adjust the model parameters of the initial keyword detection model based on the third probability vector and the second sample label.

[0050] In some embodiments, the apparatus further includes:

[0051] The third acquisition module is configured to acquire the sample to be processed and the corresponding speech text of the sample to be processed.

[0052] The forced alignment module is configured to perform forced alignment of the sample to be processed and the speech text to determine the first sample and the predetermined range corresponding to the first sample.

[0053] In some embodiments, the forced alignment module is specifically configured to perform:

[0054] The sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed.

[0055] Based on the first text probability distribution and the speech text, determine the second text probability distribution of the speech text in the sample to be processed.

[0056] Based on the second text probability distribution, the sample to be processed and the speech text are forcibly aligned.

[0057] In some embodiments, the forced alignment module is specifically configured to perform:

[0058] Determine the maximum probability of each character in the spoken text in the first text probability distribution.

[0059] Based on the maximum probability, determine the target position of each character in the speech text in the first text probability distribution.

[0060] Based on the target location, determine the matching path of the speech text in the first text probability distribution, and then determine the second text probability distribution.

[0061] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0063] In this embodiment, keyword detection of target speech can be performed using a supplementally trained keyword detection model. Since the keyword detection model is supplementally trained based on at least a predetermined range of audio frames in the first sample, and the predetermined range starts from the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length, the supplementally trained keyword detection model ensures that the keyword triggering time points are concentrated within the aforementioned predetermined range during voice interaction. This avoids premature and delayed triggering issues and improves the accuracy of keyword triggering timing. Attached Figure Description

[0064] The above and other objects, features and advantages of the present application will become clearer from the following description of embodiments of the present application with reference to the accompanying drawings, in which:

[0065] Figure 1 This is a flowchart of the voice interaction method in the embodiments of this application;

[0066] Figure 2 This is a schematic diagram of the first sample according to an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the keyword detection results in an embodiment of this application;

[0068] Figure 4 This is a flowchart of another voice interaction method in the embodiments of this application;

[0069] Figure 5 This is a schematic diagram of the first probability vector in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of another first probability vector in an embodiment of this application;

[0071] Figure 7 This is a flowchart of another voice interaction method in the embodiments of this application;

[0072] Figure 8 This is a flowchart of another voice interaction method in the embodiments of this application;

[0073] Figure 9 This is a schematic diagram of the probability distribution of the first text in an embodiment of this application;

[0074] Figure 10 This is a schematic diagram illustrating the maximum probability corresponding to the voice text in the embodiments of this application;

[0075] Figure 11 This is a schematic diagram of the second text probability distribution in an embodiment of this application;

[0076] Figure 12 This is a schematic diagram illustrating the forced alignment result of an embodiment of this application;

[0077] Figure 13 This is a schematic diagram of the structure of the voice interaction device according to an embodiment of this application;

[0078] Figure 14 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0079] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0080] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0081] Unless the context explicitly requires it, words such as "including" or "contains" in the instruction manual should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0082] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0083] Currently, voice interaction is one of the main methods of human-computer interaction, and most electronic devices support voice-based human-computer interaction. When users interact with electronic devices via voice, the electronic devices need to accurately and promptly recognize the user's voice.

[0084] When an electronic device receives a user's voice, it performs speech recognition. When the electronic device detects specific keywords in the user's voice, it performs corresponding operations based on those keywords.

[0085] In related technologies, keyword detection is achieved by training a model. For example, the technology uses training samples as input and obtains the model's recognition results for the training samples. Then, based on the model's output results and the corresponding sample labels of the training samples, the model is trained so that it can recognize keywords in the user's speech.

[0086] However, in practical applications, since users need to speak words containing keywords in sequence, it is difficult to determine the exact time when the model in the relevant technology will trigger the detection of keywords.

[0087] In one scenario, if the keyword is "next song," the model in the relevant technology might trigger the detection of that keyword upon detecting the word "next." This causes the model to trigger earlier than intended, meaning the model might still trigger detection even when the user hasn't actually said "next song" (e.g., the user says "next time" or "next one"), resulting in lower keyword detection accuracy.

[0088] In another scenario, the model in the relevant technology may trigger detection some time after the user has spoken all the keywords. This would result in a longer time from when the user speaks the words until the electronic device responds, leading to a poor human-computer interaction experience.

[0089] To address the aforementioned issues, related technologies incorporate a large number of negative samples related to keywords during model training, thereby making the timing of keyword triggering more accurate. For example, if the keyword is "next song," the technology adds negative samples related to the keyword such as "next," "next," and "song" during model training, enabling the trained model to trigger detection only after detecting all audio corresponding to "next song."

[0090] However, if a large number of negative samples related to keywords are added during the model training process, the model training process will consume more time and computing power, and the problem of sample imbalance will also occur.

[0091] Therefore, how to effectively solve the problem of keywords being triggered too early or too late during voice interaction is an urgent issue that needs to be addressed.

[0092] To address the aforementioned issues, this application provides a voice interaction method that can be applied to electronic devices. The electronic device can be a terminal or a server. The terminal can be a smartphone, tablet computer, or personal computer (PC), etc. The server can be a single server, a server cluster configured in a distributed manner, or a cloud server.

[0093] In this embodiment, the electronic device can determine the target speech. After determining the target speech, the electronic device can execute the aforementioned voice interaction method to trigger the detection of keywords in the target speech at an accurate time point. Specifically, as shown... Figure 1 As shown, the voice interaction method in this application embodiment may include the following steps:

[0094] In step S110, the target speech is determined.

[0095] In one scenario, the electronic device in this embodiment can capture user speech using its own built-in audio capture device (such as a microphone in a smartphone, tablet, or smart speaker) and use that speech as the target speech. In another scenario, the electronic device can also capture user speech using an external audio capture device (such as a microphone connected via wired or wireless connection) and use that speech as the target speech.

[0096] In one optional implementation, after determining the target speech, the present application embodiment can perform feature extraction processing on the target speech to determine the audio features corresponding to the target speech, so as to facilitate subsequent model processing.

[0097] The audio features corresponding to the target speech can be spectral features (Fbank). In this embodiment, information in the target speech can be extracted through processes such as pre-emphasis, framing, windowing, Fourier transform, and mel filter bank filtering to determine the corresponding Fbank features. Alternatively, the audio features corresponding to the target speech can also be mel-scale frequency cepstral coefficients (MFCC) features, etc.

[0098] This application embodiment can effectively extract information from the target speech by performing feature extraction processing, making subsequent model processing more efficient.

[0099] In step S120, based on the keyword detection model, keyword detection is performed on the target speech to determine the keyword detection result.

[0100] The keyword detection model is trained on at least a predetermined range of audio frames from the first sample. The predetermined range begins with the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length. This length can be any number of audio frames, which can be set appropriately based on the actual situation; for example, the length could be 20, 30, or 40 frames. Furthermore, the keyword detection model can be a neural network model built on Temporal Convolutional Networks (TCN), or other applicable models.

[0101] like Figure 2 As shown, Figure 2 This is a schematic diagram of the first sample 21 in an embodiment of this application. The first sample 21 is speech consisting of 13 audio frames, namely, the first sample 21 includes audio frames A to M. The numerical numbers below each audio frame are used to represent the order of each audio frame in the first sample 21.

[0102] Combination Figure 2 The first sample 21 shown in this application embodiment can be used to supplement the training of the keyword detection model based on a predetermined range of audio frames in the first sample 21. The audio composed of audio frames D, E and F is the keyword audio 211, and the audio frames G and H are the audio 212 within the predetermined range.

[0103] In this embodiment, the keyword detection model can determine whether the audio contains corresponding keywords based on the features in the audio. Furthermore, this embodiment can supplement the keyword detection model based on audio 212, that is, calculate the loss using the audio features corresponding to audio 212, and adjust the model parameters of the keyword detection model based on the loss corresponding to audio 212, so that the loss is transmitted to each audio frame in audio 212. This ensures that when the supplementally trained keyword detection model performs voice interaction, the triggering time of keywords will be concentrated in each audio frame of audio 212, thus controlling the keyword triggering time to the end of the keyword and avoiding the problems of premature or delayed triggering.

[0104] In step S130, in response to detecting that the target speech contains keywords, the pre-stored instruction corresponding to the keywords is executed.

[0105] Pre-stored instructions can be semantically corresponding to keywords. When the keyword detection model detects that the target speech contains keywords, the electronic device can execute the pre-stored instructions corresponding to the keywords, thereby enabling the user to control the electronic device to perform specific actions through voice, thus completing human-computer voice interaction. The electronic device can pre-store one or more pre-stored instructions and establish a correspondence between the pre-stored instructions and keywords.

[0106] For example, in one scenario, there can be a one-to-one correspondence between stored instructions and keywords, meaning one stored instruction corresponds to one keyword. In another scenario, there can be a one-to-many or many-to-many correspondence between stored instructions and keywords, meaning one stored instruction corresponds to multiple keywords, one keyword corresponds to multiple stored instructions, or multiple keywords correspond to multiple stored instructions.

[0107] In practical applications, if a pre-stored instruction corresponds to multiple keywords, the user can trigger the pre-stored instruction by saying one of those keywords. If one keyword corresponds to one pre-stored instruction, the user can trigger a series of pre-stored instructions in the electronic device by saying that keyword. If multiple keywords correspond to multiple pre-stored instructions, the user can trigger a series of pre-stored instructions in the electronic device by saying one of those keywords. Therefore, by setting different correspondences between pre-stored instructions and keywords, the triggering methods of human-computer voice interaction can be effectively expanded, and the convenience of human-computer voice interaction can be improved.

[0108] In this embodiment, keyword detection of target speech can be performed using a supplementally trained keyword detection model. Since the keyword detection model is supplementally trained based on at least a predetermined range of audio frames in the first sample, and the predetermined range starts from the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length, the supplementally trained keyword detection model ensures that the keyword triggering time points are concentrated within the aforementioned predetermined range during voice interaction. This avoids premature and delayed triggering issues and improves the accuracy of keyword triggering timing.

[0109] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the keyword detection results in an embodiment of this application.

[0110] in, Figure 3 The left vertical axis represents the numerical value of the target speech audio sampling point, the right vertical axis represents the category index of the pre-stored instruction, and the horizontal axis represents time (in seconds). Figure 3 The broken line in the diagram is used to represent the pre-stored instruction that has been triggered.

[0111] Depend on Figure 3 As can be seen, since the keyword detection model is supplemented with training based on at least a predetermined range of audio frames in the first sample, and the predetermined range takes the end frame corresponding to the keyword in the first sample as the starting point and the predetermined number of frames as the length of the predetermined range, the keyword triggering time point in this embodiment will be concentrated within the time period after the keyword ends during voice interaction, avoiding the problems of premature triggering and delayed triggering, and improving the accuracy of the keyword triggering time point.

[0112] In one alternative implementation, such as Figure 4 As shown, the keyword detection model can be further trained based on the following steps:

[0113] In step S410, the pre-trained keyword detection model and the first training set are obtained.

[0114] The first training set includes at least a first sample, a first sample label corresponding to the first sample, and a range marker for a predetermined range corresponding to the first sample.

[0115] by Figure 2 Taking the first sample 21 as an example, the first training set may include the first sample 21, the first sample label corresponding to the first sample 21, and the range marker corresponding to the predetermined range of the first sample 21. The first sample label is the text corresponding to the first sample 21, and the range marker is used to mark each audio frame (i.e., audio frames G and H) in the audio 212.

[0116] In step S420, audio frames within a predetermined range are determined in the first sample based on the range marker.

[0117] by Figure 2 Taking the first sample 21 shown as an example, in this embodiment of the application, audio frame G and audio frame H in audio 212 can be determined according to the range marking.

[0118] In step S430, the model parameters of the keyword detection model are adjusted based on the audio frames within a predetermined range and the first sample label.

[0119] Through the embodiments of this application, the model parameters of the keyword detection model can be adjusted according to the audio frames and the first sample labels within a predetermined range to achieve supplementary training of the keyword detection model. As a result, when the keyword detection model is used for voice interaction, the triggering time of the keywords will be concentrated in each audio frame within the predetermined range, avoiding the problems of premature triggering and delayed triggering, and improving the accuracy of the timing of keyword triggering.

[0120] In an optional implementation, step S430 may include the following steps:

[0121] In step S431, the first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model.

[0122] The keyword detection model can output a first probability vector corresponding to the first sample based on the first sample or the Fbank features corresponding to the first sample. This first probability vector can be used to characterize the probability that each audio frame in the first sample belongs to a certain word.

[0123] Specifically, in this embodiment of the application, the first sample, the Fbank feature corresponding to the first sample, or other audio features corresponding to the first sample can be input into the pre-trained keyword detection model so that the pre-trained keyword detection model outputs a first probability vector.

[0124] For example, the first sample is a 3-second audio clip, and the keyword detection model is a neural network model built based on the TCN model, with 23 Mel filter banks in the TCN model. During supplementary training, this embodiment can first perform feature extraction on the first sample with a frame shift of 10 milliseconds to determine the 300×23-dimensional Fbank features corresponding to the first sample. Here, 300 represents that the Fbank features include 300 frames, and 23 is the dimension determined by the number of Mel filter banks.

[0125] Then, in this embodiment, the 300×23-dimensional Fbank features corresponding to the first sample can be input into the pre-trained keyword detection model to determine a 300×5-dimensional first probability vector. Here, 5 can represent the number of pre-set probability categories, which can be set according to actual conditions. This embodiment uses 5 probability categories as an example, which may include 4 pre-set keywords (e.g., answering the phone, hanging up the phone, previous song, and next song) and no detected keywords.

[0126] For example, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the first probability vector in an embodiment of this application. Wherein, Figure 5 It includes 5 probability categories (i.e. Figure 5 The data is categorized into categories 0, 1, 2, 3, and 4. Category 0 indicates that no keywords were detected, while categories 1, 2, 3, and 4 correspond to different keywords. Additionally, Figure 5 It also includes n audio frames (n is a natural number greater than or equal to 1), if Figure 5 This represents the first probability vector of the above 300×5 dimensions, so n equals 300.

[0127] Combination Figure 5 The first probability vector shown in this embodiment can be represented by a matrix, which represents the probability that each frame of the first sample belongs to each probability category.

[0128] In step S432, the probability of each target corresponding to the audio frame within the predetermined range in the first probability vector is determined.

[0129] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the first probability vector in an embodiment of this application. Wherein, Figure 6 The dashed box in the figure is used to represent the predetermined range 61. That is, the audio frames within the predetermined range 61 are audio frame 3 and audio frame 4. Furthermore, the 10 probabilities shown in the predetermined range 61 are the target probabilities determined in the embodiments of this application.

[0130] It should be noted that, in order to clearly illustrate the predetermined scope, the embodiments of this application... Figure 6 Only audio frames 3 and 4 are used as audio frames within the predetermined range 61. In practical applications, the length of the predetermined range can be set according to the actual situation. For example, if the first probability vector is the aforementioned 300×5-dimensional vector, the predetermined range in this embodiment can be 20 frames, 30 frames, or 40 frames, etc.

[0131] In step S433, max pooling is performed on each target probability to determine the second probability vector.

[0132] Max pooling reduces the dimensionality of the matrix composed of the probabilities of each target. Specifically, if the matrix composed of the probabilities of each target is a 30×5 matrix, then when max pooling is performed on this matrix in this embodiment, the dimensionality can be reduced for probability categories where no keywords were detected (such as those mentioned above). Figure 5 and Figure 6 The row containing category 0 is given the minimum value, and the other rows are given the maximum value, thus obtaining a 1×5 second probability vector.

[0133] In step S434, the model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label.

[0134] In this embodiment, loss calculation can be performed based on the second probability vector and the first sample label to determine the loss function, thereby adjusting the model parameters of the pre-trained keyword detection model according to the loss function. For example, this embodiment can perform binary cross-entropy loss on each probability category for the second probability vector, and then take the average of the binary cross-entropy losses as the loss function, thereby adjusting the model parameters of the pre-trained keyword detection model according to the loss function.

[0135] Through the embodiments of this application, the model parameters of the keyword detection model can be adjusted according to the audio frames and the first sample labels within a predetermined range to achieve supplementary training of the keyword detection model. As a result, when the keyword detection model is used for voice interaction, the triggering time of the keywords will be concentrated in each audio frame within the predetermined range, avoiding the problems of premature triggering and delayed triggering, and improving the accuracy of the timing of keyword triggering.

[0136] In one alternative implementation, such as Figure 7 As shown, the keyword detection model can be pre-trained based on the following steps:

[0137] In step S710, the second training set is obtained.

[0138] The second training set includes at least the second sample and the corresponding second sample label. Furthermore, the second sample can be the same audio sample as the first sample, or it can be a different audio sample.

[0139] In step S720, the second sample is input into the initial keyword detection model to determine the third probability vector output by the initial keyword detection model.

[0140] In step S730, the model parameters of the initial keyword detection model are adjusted based on the third probability vector and the second sample label.

[0141] Through the embodiments of this application, an initial keyword detection model can be pre-trained to obtain a pre-trained keyword detection model. In the subsequent supplementary training process, the embodiments of this application can directly obtain the pre-trained keyword detection model, thereby saving the time required for supplementary training and improving the training efficiency of supplementary training.

[0142] During the supplementary training process in this embodiment, it is necessary to determine a predetermined range of audio frames in the first sample, with the predetermined range starting from the end frame corresponding to the keyword in the first sample. Therefore, an accurate correspondence needs to be established between the keyword and each audio frame of the first sample.

[0143] In one alternative implementation, such as Figure 8 As shown, embodiments of this application may further include the following steps:

[0144] In step S810, the sample to be processed and the corresponding speech text are obtained.

[0145] The sample to be processed can be the original speech sample, which includes the audio corresponding to the speech text, and the speech text can be the semantics corresponding to the sample to be processed.

[0146] In step S820, the sample to be processed and the speech text are forcibly aligned to determine the first sample and the predetermined range corresponding to the first sample.

[0147] By forcibly aligning the sample to be processed and the speech text, the embodiments of this application can make the semantics corresponding to each audio frame in the sample to be processed correspond to the semantics shown in the speech text, thereby enabling the embodiments of this application to accurately determine the end frame corresponding to the keyword during supplementary training, and thus accurately determine the predetermined range corresponding to the first sample.

[0148] Specifically, in one optional implementation, step S820 may include the following steps:

[0149] In step S821, the sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed.

[0150] The pre-trained speech recognition model can be an Automatic Speech Recognition (ASR) model. After receiving the sample to be processed, the ASR model can perform speech recognition on the sample to be processed, thereby determining the probability distribution vector (i.e., the first text probability distribution) corresponding to the sample to be processed.

[0151] Specifically, in the first text probability distribution output by the ASR model, the probability categories can be a dictionary composed of individual characters, with each probability category representing a single character. Taking English as an example, the dictionary corresponding to the above probability categories can include the 26 English letters, spaces, single quotes, and null values ​​(i.e., no content), totaling 29 probability categories.

[0152] For example, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the probability distribution of the first text in an embodiment of this application.

[0153] exist Figure 9 In the diagram, the vertical axis represents the probability categories in the first text probability distribution, and the horizontal axis represents the time axis (in frames) corresponding to the first text probability distribution. Figure 9 The colors in the image can be used to represent the probability of each frame under each probability category; darker colors represent lower probabilities, and lighter colors represent higher probabilities. Figure 9 In the diagram, each white area represents the probability category with the highest probability for each frame.

[0154] In step S822, based on the first text probability distribution and the speech text, the second text probability distribution of the speech text in the sample to be processed is determined.

[0155] Since the probability categories in the first text probability distribution include both characters in the spoken text and characters not contained in the spoken text, this embodiment of the application can filter the probabilities in the first text probability distribution using the spoken text to select the probability distribution corresponding to each character in the spoken text (i.e., the second text probability distribution).

[0156] In an optional implementation, step S822 may include the following steps:

[0157] In step S8221, the maximum probability of each character in the speech text in the first text probability distribution is determined.

[0158] For example, such as Figure 10 As shown, Figure 10 for Figure 9 The probability distribution obtained after step S8221 above is, i.e. Figure 10 This is a schematic diagram illustrating the maximum probability corresponding to the voice text in the embodiments of this application.

[0159] exist Figure 10 In the diagram, the vertical axis represents the probability categories in the first text probability distribution, and the horizontal axis represents the time axis (in frames) corresponding to the first text probability distribution. Figure 10The colors in the text can be used to represent the probability of each word in the first text probability distribution; darker colors represent lower probabilities, and lighter colors represent higher probabilities. Figure 10 In the text, the lightest white area represents the highest probability of each word in the first text probability distribution.

[0160] In step S8222, the target position of each character in the speech text in the first text probability distribution is determined according to the maximum probability.

[0161] In other words, in this embodiment of the application, the position of the highest probability corresponding to each character in the speech text in the first text probability distribution can be determined, and this position can be used as the target position.

[0162] In step S8223, based on the target location, the matching path of the speech text in the first text probability distribution is determined to determine the second text probability distribution.

[0163] For example, if the voice text is “I HAD THAT CURIOSITY BESIDE ME AT THIS MOMENT”, then the second text probability distribution of this application embodiment is the maximum probability corresponding to each letter in the above voice text.

[0164] like Figure 11 As shown, Figure 11 This is a schematic diagram of the second text probability distribution in an embodiment of this application.

[0165] exist Figure 11 In this embodiment of the application, the target positions can be marked and matching paths determined based on the target positions corresponding to each letter in the above-mentioned speech text, thereby determining the second text probability distribution. Specifically, as follows... Figure 11 As shown, Figure 11 Each white rectangular region in the image represents the target position corresponding to each letter, and its length represents the duration (in frames) corresponding to each letter.

[0166] Therefore, through the embodiments of this application, forced alignment can be performed between the sample to be processed and the speech text to determine the probability distribution of the second text, which means that the semantics corresponding to each audio frame in the sample to be processed corresponds to the semantics shown in the speech text. This allows the embodiments of this application to accurately determine the end frame corresponding to the keyword during supplementary training, and thus accurately determine the predetermined range corresponding to the first sample.

[0167] In step S823, the sample to be processed and the speech text are forcibly aligned according to the second text probability distribution.

[0168] like Figure 12 As shown, Figure 12This is a schematic diagram of the forced alignment result in an embodiment of this application.

[0169] Depend on Figure 12 As can be seen, through the forced alignment in the embodiments of this application, the human voice part in the sample to be processed can be forcibly aligned with the text in the speech text, so that the embodiments of this application can accurately determine the end frame corresponding to the keyword during supplementary training, and thus accurately determine the predetermined range corresponding to the first sample.

[0170] In this embodiment of the application, after the sample to be processed and the speech text are forcibly aligned, the aligned sample to be processed can be used as the first sample, and the first sample can be used for supplementary training of the keyword detection model.

[0171] Based on the same technical concept, embodiments of this application also provide a voice interaction device, such as... Figure 13 As shown, the device includes:

[0172] The target speech determination module 131 is configured to determine the target speech.

[0173] The keyword detection module 132 is configured to execute a pre-trained keyword detection model to detect keywords in the target speech and determine the keyword detection result. The keyword detection model is supplemented by training on at least a predetermined range of audio frames in the first sample. The predetermined range takes the end frame corresponding to the keyword in the first sample as the starting point and a predetermined number of frames as the length of the predetermined range.

[0174] The keyword execution module 133 is configured to execute a pre-stored instruction corresponding to the keyword in response to detecting that the target speech includes the keyword.

[0175] In some embodiments, the keyword detection model is supplemented by training based on the following modules:

[0176] The first acquisition module is configured to acquire a pre-trained keyword detection model and a first training set, wherein the first training set includes at least the first sample, the first sample label corresponding to the first sample, and the range marker of the predetermined range corresponding to the first sample.

[0177] The first determining module is configured to perform the task of determining audio frames within a predetermined range in the first sample based on the range marker.

[0178] The supplementary training module is configured to adjust the model parameters of the keyword detection model based on the audio frames within the predetermined range and the first sample label.

[0179] In some embodiments, the supplementary training module is specifically configured to perform:

[0180] The first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model.

[0181] Determine the target probabilities corresponding to the audio frames within the predetermined range in the first probability vector.

[0182] Max pooling is performed on each of the target probabilities to determine the second probability vector.

[0183] The model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label.

[0184] In some embodiments, the keyword detection model is pre-trained based on the following modules:

[0185] The second acquisition module is configured to acquire a second training set, which includes at least a second sample and a second sample label corresponding to the second sample.

[0186] The second determining module is configured to input the second sample into the initial keyword detection model and determine the third probability vector output by the initial keyword detection model.

[0187] The pre-training module is configured to adjust the model parameters of the initial keyword detection model based on the third probability vector and the second sample label.

[0188] In some embodiments, the apparatus further includes:

[0189] The third acquisition module is configured to acquire the sample to be processed and the corresponding speech text of the sample to be processed.

[0190] The forced alignment module is configured to perform forced alignment of the sample to be processed and the speech text to determine the first sample and the predetermined range corresponding to the first sample.

[0191] In some embodiments, the forced alignment module is specifically configured to perform:

[0192] The sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed.

[0193] Based on the first text probability distribution and the speech text, determine the second text probability distribution of the speech text in the sample to be processed.

[0194] Based on the second text probability distribution, the sample to be processed and the speech text are forcibly aligned.

[0195] In some embodiments, the forced alignment module is specifically configured to perform:

[0196] Determine the maximum probability of each character in the spoken text in the first text probability distribution.

[0197] Based on the maximum probability, determine the target position of each character in the speech text in the first text probability distribution.

[0198] Based on the target location, determine the matching path of the speech text in the first text probability distribution, and then determine the second text probability distribution.

[0199] In this embodiment, keyword detection of target speech can be performed using a supplementally trained keyword detection model. Since the keyword detection model is supplementally trained based on at least a predetermined range of audio frames in the first sample, and the predetermined range starts from the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length, the supplementally trained keyword detection model ensures that the keyword triggering time points are concentrated within the aforementioned predetermined range during voice interaction. This avoids premature and delayed triggering issues and improves the accuracy of keyword triggering timing.

[0200] Figure 14 This is a schematic diagram of an electronic device according to an embodiment of this application. For example... Figure 14 As shown, Figure 14 The illustrated electronic device is a general address lookup device, comprising a general computer hardware architecture, including at least a processor 141 and a memory 142. The processor 141 and memory 142 are connected via a bus 143. The memory 142 is adapted to store instructions or programs executable by the processor 141. The processor 141 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 141 executes the instructions stored in the memory 142, thereby performing the method flow described in the embodiments of this application to process data and control other devices. The bus 143 connects the aforementioned components together, and also connects these components to a display controller 144, a display device, and an input / output (I / O) device 145. The input / output (I / O) device 145 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 145 is connected to the system via an input / output (I / O) controller 146.

[0201] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0203] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0204] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0205] Another embodiment of this application relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0206] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0207] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A voice interaction method, characterized in that, The method includes: Identify the target speech; Based on a keyword detection model, keyword detection is performed on the target speech to determine the keyword detection result. The keyword detection model is supplemented with training based on at least a predetermined range of audio frames in the first sample. The predetermined range starts at the end frame corresponding to the keyword in the first sample and has a predetermined number of frames as its length. In response to detecting that the target speech includes the keyword, the pre-stored instruction corresponding to the keyword is executed; The keyword detection model is further trained based on the following steps: Obtain a pre-trained keyword detection model and a first training set, wherein the first training set includes at least the first sample, the first sample label corresponding to the first sample, and the range label of the predetermined range corresponding to the first sample; Based on the range marker, audio frames within a predetermined range are determined in the first sample; and The model parameters of the keyword detection model are adjusted based on the audio frames within the predetermined range and the first sample label; the adjustment of the model parameters of the keyword detection model based on the audio frames within the predetermined range and the first sample label includes: The first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model. Determine the target probabilities corresponding to the audio frames within the predetermined range in the first probability vector; Max pooling is performed on each of the stated target probabilities to determine the second probability vector; and The model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label. The adjustment of the model parameters of the pre-trained keyword detection model includes: For the second probability vector, perform binary cross-entropy loss on each probability category, then take the average of the binary cross-entropy losses as the loss function, and adjust the model parameters of the pre-trained keyword detection model according to the loss function. Determining the second probability vector includes: When performing max pooling on the matrix composed of the probabilities of each target, the minimum value is taken in the row where the probability category of no keyword is detected, and the maximum value is taken in the other rows to obtain the second probability vector. Prior to the supplementary training, the method further includes: Obtain the sample to be processed and the corresponding speech text of the sample to be processed; Force alignment is performed on the sample to be processed and the speech text to determine the first sample and the predetermined range corresponding to the first sample; The forced alignment of the sample to be processed and the speech text includes: The sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed. The probability categories of the first text probability distribution are a dictionary composed of each character, and each probability category represents a character. Based on the first text probability distribution and the speech text, determine the second text probability distribution of the speech text in the sample to be processed; Based on the second text probability distribution, the sample to be processed and the speech text are forcibly aligned.

2. The method according to claim 1, characterized in that, The keyword detection model is pre-trained based on the following steps: Obtain a second training set, which includes at least a second sample and the second sample label corresponding to the second sample; The second sample is input into the initial keyword detection model to determine the third probability vector output by the initial keyword detection model; and The model parameters of the initial keyword detection model are adjusted based on the third probability vector and the second sample label.

3. The method according to claim 1, characterized in that, The step of determining the second text probability distribution of the speech text in the sample to be processed based on the first text probability distribution and the speech text includes: Determine the maximum probability of each character in the spoken text in the first text probability distribution; Based on the maximum probability, determine the target position of each character in the speech text within the first text probability distribution; and Based on the target location, determine the matching path of the speech text in the first text probability distribution, and then determine the second text probability distribution.

4. A voice interaction device, characterized in that, The device includes: The target speech determination module is configured to determine the target speech. A keyword detection module is configured to execute a pre-trained keyword detection model to detect keywords in the target speech and determine the keyword detection result. The keyword detection model is supplemented with training based on at least a predetermined range of audio frames in a first sample. The predetermined range starts at the end frame corresponding to a keyword in the first sample and has a predetermined number of frames as its length. The keyword execution module is configured to execute a pre-stored instruction corresponding to the keyword in response to detecting that the target speech includes the keyword. The keyword detection model is further trained based on the following steps: Obtain a pre-trained keyword detection model and a first training set, wherein the first training set includes at least the first sample, the first sample label corresponding to the first sample, and the range label of the predetermined range corresponding to the first sample; Based on the range marker, audio frames within a predetermined range are determined in the first sample; and The model parameters of the keyword detection model are adjusted based on the audio frames within the predetermined range and the first sample label; the adjustment of the model parameters of the keyword detection model based on the audio frames within the predetermined range and the first sample label includes: The first sample is input into the pre-trained keyword detection model to determine the first probability vector output by the pre-trained keyword detection model. Determine the target probabilities corresponding to the audio frames within the predetermined range in the first probability vector; Max pooling is performed on each of the stated target probabilities to determine the second probability vector; and The model parameters of the pre-trained keyword detection model are adjusted based on the second probability vector and the first sample label. The adjustment of the model parameters of the pre-trained keyword detection model includes: For the second probability vector, perform binary cross-entropy loss on each probability category, then take the average of the binary cross-entropy losses as the loss function, and adjust the model parameters of the pre-trained keyword detection model according to the loss function. Determining the second probability vector includes: When performing max pooling on the matrix composed of the probabilities of each target, the minimum value is taken in the row where the probability category of no keyword is detected, and the maximum value is taken in the other rows to obtain the second probability vector. The device further includes: The third acquisition module is configured to acquire the sample to be processed and the corresponding speech text of the sample to be processed. The forced alignment module is configured to force alignment of the sample to be processed and the speech text to determine the first sample and the predetermined range corresponding to the first sample. The forced alignment module is specifically configured to execute: The sample to be processed is input into a pre-trained speech recognition model to determine the first text probability distribution corresponding to the sample to be processed. The probability categories of the first text probability distribution are a dictionary composed of each character, and each probability category represents a character. Based on the first text probability distribution and the speech text, determine the second text probability distribution of the speech text in the sample to be processed; Based on the second text probability distribution, the sample to be processed and the speech text are forcibly aligned.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-3.

Images