Multi-speaker speech keyword recognition method, device and apparatus

Abstract

Embodiments of the present application provide a multi-speaker speech keyword recognition method, device and equipment, relating to the technical field of speech processing, wherein the method comprises: obtaining multi-speaker audio to be recognized; based on a keyword information extractor, extracting a keyword embedding condition matrix corresponding to a target keyword from the multi-speaker audio; inputting the keyword embedding condition matrix and the multi-speaker audio to a text-aware speech separation module to separate a potential keyword audio sequence from the multi-speaker audio by the text-aware speech separation module; based on a keyword search module, determining a recognition probability of the potential keyword audio sequence for the target keyword. Thus, the keyword context information is used to guide the extraction of keyword audio in multi-speaker mixed audio, effectively making up for the defect that the current related technology cannot handle human voice interference, and improving the accuracy of the keyword recognition result of the multi-speaker audio in the speech keyword detection technology.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a method, apparatus and device for multi-speaker speech keyword recognition. Background Technology

[0002] With the continuous promotion of intelligent voice technology, smart devices with voice interaction capabilities have been widely adopted. Currently, the intelligent voice assistants of various smart electronic devices are equipped with corresponding wake-up word detection technology, which uses specific wake-up keywords to activate the intelligent voice assistant and initiate the human-computer interaction process. Examples include Xiaomi's "Xiao Ai" and Huawei's "Xiao Yi Xiao Yi," among others.

[0003] Currently, wake-up word detection technology mainly involves running a wake-up word detection model continuously on the device 24 / 7 to detect the user's input speech stream and determine whether the user has used a wake-up word to wake up the device. However, wake-up technology has certain requirements for the signal-to-noise ratio (SNR) of the input signal; performance is poor when the SNR is low, especially in multi-speaker scenarios. Existing noise reduction technologies are poor at handling human voice interference and cannot effectively remove human voice interference, resulting in poor keyword wake-up performance of the device.

[0004] Currently, the industry has not proposed a better solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for multi-speaker speech keyword recognition, which at least solves the problem in the current related technologies that the keyword detection performance of multi-speaker speech is poor due to the inability to effectively eliminate human voice interference.

[0006] In a first aspect, embodiments of this application provide a method for multi-speaker speech keyword recognition, comprising: acquiring multi-speaker audio to be recognized; extracting a keyword embedding condition matrix corresponding to a target keyword from the multi-speaker audio based on a keyword information extractor; inputting the keyword embedding condition matrix and the multi-speaker audio to a text-aware speech separation module, so that the text-aware speech separation module separates a potential keyword audio sequence from the multi-speaker audio; the potential keyword audio sequence contains at least one potential keyword audio with a fixed order, each potential keyword audio being a predicted audio segment containing the target keyword; and determining the recognition probability of the potential keyword audio sequence for the presence of the target keyword based on a keyword search module.

[0007] Secondly, embodiments of this application provide a multi-speaker speech keyword recognition device, comprising: an acquisition unit configured to acquire multi-speaker audio to be recognized; an extraction unit configured to extract a keyword embedding condition matrix corresponding to a target keyword from the multi-speaker audio based on a keyword information extractor; a separation unit configured to input the keyword embedding condition matrix and the multi-speaker audio to a text-aware speech separation module, so that the text-aware speech separation module separates a potential keyword audio sequence from the multi-speaker audio; the potential keyword audio sequence includes at least one potential keyword audio with a fixed order, each potential keyword audio being a predicted audio segment containing the target keyword; and a recognition unit configured to determine the recognition probability of the potential keyword audio sequence for the presence of the target keyword based on a keyword search module.

[0008] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-speaker speech keyword recognition method of any embodiment of this application.

[0009] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the multi-speaker speech keyword recognition method of any embodiment of this application.

[0010] The beneficial effects of the embodiments of this application are as follows:

[0011] When processing multi-speaker audio, a keyword information extractor is used to extract the keyword embedding condition matrix of the corresponding target keywords. The keyword embedding condition matrix and the multi-speaker audio are then input into the text-aware speech separation module. Text-guided extraction of keyword audio is used to enable the model to have text-aware capabilities by utilizing keyword context information. This results in the output of a sequence of potential keyword audio with a fixed order, rather than outputting audio in a random order. This approach can solve the problem of human voice interference without increasing keyword detection overhead, effectively improving the separation quality of keyword-related audio. It effectively compensates for the shortcomings of current related technologies in handling human voice interference and improves the accuracy of keyword recognition results for multi-speaker audio in speech keyword detection technology. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating an example of a multi-speaker speech keyword recognition method according to an embodiment of this application is shown;

[0014] Figure 2 A schematic diagram of the architecture of an example of a multi-speaker speech keyword recognition system according to an embodiment of this application is shown;

[0015] Figure 3 A schematic diagram of the architecture of an example keyword search module according to an embodiment of this application is shown;

[0016] Figure 4 This diagram illustrates a performance comparison of TSKim chn1 with other models in terms of KWS performance according to an embodiment of this application.

[0017] Figure 5 A structural block diagram of an example of a multi-speaker speech keyword recognition device provided according to an embodiment of this application is shown;

[0018] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that keyword detection (or keyword detection system, KWS), also known as wake word detection (WWD), has been widely used in intelligent interaction interfaces on numerous edge devices. Despite the excellent performance of existing systems, many challenges remain in complex acoustic environments, such as ambient noise or overlapping interference from multiple speakers. The low signal-to-noise ratio (SNR) of the wake word audio severely hinders the effectiveness of wake-up systems.

[0021] For robust KWS, a common solution is to integrate a Speech Enhancement (SE) front-end to handle noisy speech, thereby improving its resilience to background noise. Current related techniques involve jointly training a fully convolutional denoising UNet architecture and a speech classifier, using multi-scale L1 loss for the SE task and binary cross-entropy loss for speech classification. Furthermore, some experts and scholars have proposed applying a curriculum training strategy to enhance noise robustness. Additionally, some related techniques explore combining a self-supervised pre-trained speech representation model, Wav2Vec, with an SE module for speech classification under noisy conditions. Another significant approach in current related techniques proposes a front-end called SE-SPP Net, which simultaneously performs SE and estimates the Speech Presence Probability (SPP) for robust KWS in noisy environments.

[0022] However, simply handling ambient noise is insufficient for KWS systems, as they may also operate in environments with significant interference from other speakers. This interference, known as overlapping speech, has a far greater impact on KWS system performance than ambient noise. Overlapping speech contains segments resembling wake words, which are difficult to eliminate without specific design, increasing the risk of false alarms. Therefore, this application focuses on robust KWS against multi-speaker interference to specifically address the overlapping speech problem in KWS tasks. In some related technologies, experts and scholars have proposed Text Dependent Speech Enhancement (TDSE) to handle ambient noise and human speech signals. This involves training a front-end SE model to directly predict keyword audio, regardless of the noise type. However, this work raises some concerns. Each mixed audio sample in the TDSE training corpus contains a keyword speech, which could potentially cause the model to overfit to the keyword. During inference, the model may still tend to predict keyword speech even when it is absent, increasing the risk of false alarms.

[0023] Robustness is crucial for Keyword Search (KWS) systems operating in noisy real-world environments. While environmental noise has been extensively studied, research on multi-person mixed speech scenarios for KWS is limited. Although speech separation methods have been successfully applied to multi-person mixed speech recognition, research on designing specific speech separation techniques for KWS is scarce.

[0024] In response, the inventors of this application propose a multi-task framework for SE and KWS, introducing an audio-based context bias module to distinguish keywords in low SNR scenarios.

[0025] Figure 1 A flowchart illustrating an example of a multi-speaker speech keyword recognition method according to an embodiment of this application is shown.

[0026] Regarding the execution subject of the method in the embodiments of this application, it can be any controller or processor with computing or processing capabilities to automatically identify whether target keywords exist in the audio of multiple speakers, thereby improving the reliability of keyword detection results in multi-speaker scenarios. In some examples, it can be integrated into an electronic device through software, hardware, or a combination of both, and the type of electronic device can be diverse, such as a mobile phone, tablet computer, or desktop computer, etc.

[0027] like Figure 1 As shown, in step S110, the audio of the multiple speakers to be identified is obtained.

[0028] Here, multi-speaker audio can refer to audio data collected from a multi-speaker environment by a device audio acquisition device, such as capturing speaker speech in a noisy multi-speaker environment.

[0029] In step S120, based on the keyword information extractor, a keyword embedding condition matrix corresponding to the target keywords is extracted from the audio of multiple speakers.

[0030] Here, the keyword information extractor can employ various text feature extractors. In one example of an embodiment of this application, the keyword information extractor can use plain text as a condition, or it can use a certain number of keyword audio clips as a condition to increase the amount of implicit information; more details will be elaborated below.

[0031] In step S130, the keyword embedding condition matrix and multi-speaker audio are input to the text-aware speech separation module so that the text-aware speech separation module can separate the potential keyword audio sequence from the multi-speaker audio.

[0032] Here, the latent keyword audio sequence contains at least one latent keyword audio segment with a fixed order, and each latent keyword audio segment is a predicted audio segment containing the target keyword. Therefore, using keyword features instead of a speech enhancement front-end to guide audio separation, while leveraging keyword contextual information to give the model text-awareness capabilities, prioritizes outputting keyword-related audio rather than outputting audio in random order, effectively solving the problem of human voice interference without increasing detection overhead.

[0033] In step S140, based on the keyword search module, the probability of identifying the potential keyword audio sequence for the existence of the target keyword is determined.

[0034] Here, the keyword search module can employ various audio recognition networks, such as deep learning models, to identify and determine whether the target keyword exists in the potential keyword audio sequence. In a further implementation example, the recognition probability is compared with a preset probability threshold, and the result of the comparison determines whether the mixed audio signal (or multi-speaker audio) contains the target keyword.

[0035] This application addresses the keyword detection problem in long-tail, but under-researched, noisy multi-speaker scenarios. By utilizing a text-aware separator, keyword-related audio can be effectively extracted from aliased speech, while simultaneously improving the separation quality of keyword-related audio. This effectively compensates for the shortcomings of previous systems in handling voice interference, thereby improving the accuracy of speech keyword detection results.

[0036] Figure 2 A schematic diagram of the architecture of an example of a multi-speaker speech keyword recognition system according to an embodiment of this application is shown.

[0037] like Figure 2 As shown, to extract potential keyword speech from overlapping speech, a multi-speaker speech keyword recognition system includes a keyword information extractor, a text-aware separator (i.e., a text-aware speech separation module), and a keyword search (KWS) module, forming a text-aware separation system for multi-person conversation KWS. During the inference process, the text-aware separator receives mixed audio and outputs keyword-related audio in a fixed order. The KWS module then calculates the probability that the mixed signal contains the keyword. In this embodiment, the focus is on KWS for a fixed single keyword, making both the text-aware separator and the KWS module customized for a single preset keyword.

[0038] Reference Figure 2 In the example, the keyword information extractor is used to extract a keyword embedding condition matrix corresponding to the target keyword from multi-speaker audio based on a list of keyword audios.

[0039] In some implementations, the keyword information extractor includes an audio encoder. Regarding the implementation details of step S120 above, based on the audio encoder, each labeled keyword audio from a preset keyword audio list can be matched with multi-speaker audio to extract at least one keyword embedding corresponding to the multi-speaker audio. Furthermore, based on each keyword embedding, a keyword embedding condition matrix is ​​constructed. Here, each labeled keyword audio is a labeled audio segment containing a keyword, and each labeled audio segment has a corresponding speaker's speech background.

[0040] More specifically, the keyword information extractor processes text-aware keyword fragments to generate conditional information. In this embodiment, the focus is primarily on the scenario of preset keywords, where using text-aware audio fragments from different speakers with different speech backgrounds as conditions significantly helps the text-aware separator separate audio related to keywords. Conversely, in the scenario of preset keywords, using plain text as conditions would result in the conditions in this embodiment being equivalent to a learnable vector, greatly reducing the amount of implicit information. Therefore, the condition list in this embodiment includes a certain number of keyword audio fragments, specifically employing the widely used ECAPA-TDNN network as the audio encoder to extract keyword embeddings. Since the focus of this embodiment is not on speaker information, the keyword information extractor is jointly trained with the entire network to ensure that the generated embeddings are directly optimized for the text-aware separation task. For a condition list of size N: A, |A|=N. The condition list will be encoded as a matrix C∈R. N×D , where D represents the feature dimension. This process can be represented as:

[0041]

[0042] Where F is the audio encoder (ECAPA-TDNN), a i It is the i-th audio segment in the condition list, c i It is its corresponding embedding.

[0043] Furthermore, the text-aware speech separation module includes an encoder, a decoder, and a separator. The encoder contains 1-D convolutional layers, the decoder contains 1-D transposed convolutional layers, and the separator contains multiple sequentially cascaded text-aware skip memory blocks. Specifically, the last text-aware skip memory block in the separator contains a Mem-LSTM (Memory-long-short-term memory) network, and the other text-aware skip memory blocks each contain cascaded Seg-LSTM (Segment-long-short-term memory) and Mem-LSTM networks.

[0044] Specifically, the text-aware speech separation module follows the TasNet backbone architecture, comprising an encoder, a separator, and a decoder. The encoder contains a 1-D convolutional layer, and the decoder contains a 1-D transposed convolutional layer. The encoder is responsible for encoding the signal into features, while the decoder reconstructs the signal back from the processed features. The separator consists of multiple text-aware skip memory (TSkim) blocks, enabling excellent performance in low-latency and real-time continuous speech separation.

[0045] As a further optimization of the embodiments of this application, the text-aware skip memory block includes a cross-Conformer block, the output of which is used as the input of the Seg-LSTM. Here, the cross-Conformer block includes multiple feedforward neural network layers, a multi-head cross-attention layer, convolutional layers, and a layer normalization module. Specifically, the query of the multi-head cross-attention layer is defined by the mixed signal features corresponding to the multi-speaker audio, and the keys and values ​​of the multi-head cross-attention layer are defined by the keyword embedding condition matrix.

[0046] Specifically, each text-aware skip memory block includes a basic SKim block, a Seg-LSTM, and a Mem-LSTM (except for the last block). A cross-Conformer block is introduced at the beginning of each SKim block to facilitate information integration between the mixed features and the conditional matrix. The cross-Conformer block contains three feedforward neural network (FFN) modules, a multi-head cross-attention layer (MHCA), a convolutional layer (Conv), and a post-normalization (LN) module. The FFN and Conv modules follow the same architecture as the Conformer. MHCA utilizes the mixed input as the query and the conditional matrix as the key and value to achieve effective information fusion. The input features of the i-th SKim block are represented as W. i ∈R T×D Where T and D are the time steps and feature dimensions, respectively, the mapping function of the i-th cross-Conformer module can be expressed as:

[0047]

[0048] Among them, W i This is the output of the cross-Conformer block, which is then processed by Seg-LSTM and Mem-LSTM. (The last part, "c", appears to be a typo and can be left as is.) i and h i This represents the globally synchronized memory state. The output of the i-th TSKim block can be represented as:

[0049]

[0050] Among them, c i+1 and h i+1 W is used as the initial state for the Seg-LSTM in the next TSKim block. i+1 The output is sent to the next TSKIM block. The output of the last TSKIM block is merged and then fed into the transposed convolutional decoder to estimate the separated signals.

[0051] In some examples of embodiments of this application, the text-aware speech separation module employs the temporal SI-SNR loss function. For the proposed text-aware separation network, embodiments of this application employ the temporal SI-SNR loss function and design two training modes, the main difference being whether or not audio unrelated to keywords is involved, and whether or not PIT is used to enhance separation performance. In embodiments of this application, the keyword-related reference signal and the mixed signal are respectively represented as S... k and X k The reference signal unrelated to the keyword is represented as S. u In the first training mode, similar to target speaker extraction, the model always outputs keyword-related audio in the speaker's phonology channel, effectively eliminating the permutation problem. This text-guided permutation determinism loss L... k It can be represented as:

[0052]

[0053] in, and These represent the separate speech sounds of speaker one and speaker two, respectively.

[0054] Furthermore, regarding the second training mode, the text-aware speech separation module can be trained using a permutation-invariant training method. This allows for the inclusion of additional audio samples unrelated to keywords during training, and the PIT (Permutation Invariant Training) criterion is applied to these samples. The final mixture loss L can be expressed as:

[0055]

[0056] Among them, S u1 and S u2 This represents two reference signals unrelated to the keywords. In this case, due to the involvement of PIT, it naturally encounters potential permutation problems. These two training modes can be seen as a trade-off between separation performance and permutation issues; more specific details and advantages will be discussed below in conjunction with other examples.

[0057] It should be noted that in multi-person conversation environments, handling overlapping audio is usually achieved by applying a Speech Separation (SS) model. Permutation Independence Training (PIT) is a very effective training technique for SS systems. It calculates the loss of all possible permutations of the separated signals and selects the permutation with the minimum loss to optimize the model. PIT was initially used for signal-level multi-speaker separation and subsequently applied to multi-speaker automatic speech recognition (ASR). Then, to further improve performance, multi-person conversation speech frameworks based on fully neural end-to-end PIT were widely proposed. Although PIT performs well in both signal and SS performance, it is not directly applicable to multi-person conversation KWS for two key reasons: (1) PIT does not focus on keywords. In multi-person conversation KWS, the focus is entirely on recognizing keywords, rather than the speech content. Since PIT ignores keyword information, it may lead to suboptimal performance. (2) Increased computational overhead and risk of false alarms. When using PIT for a front-end SS with multiple output channels, the KWS module must process all separated speech channels simultaneously. This doubles the processing cost of the KWS module and increases the risk of false alarms.

[0058] In this application embodiment, a text-aware speech separation method for multi-person conversation KWS is proposed, combining the SS module and the KWS module. Its contributions are summarized as follows:

[0059] This paper addresses the important but poorly studied problem of multi-person conversation KWS. In this embodiment, a novel text-aware SS module is proposed to eliminate the keyword speech permutation problem introduced by PIT. This method not only significantly improves the performance of subsequent multi-person conversation KWS, but also reduces the computational overhead of the backend KWS model even when the front-end model outputs multiple separate speech samples.

[0060] Through the embodiments of this application, in multi-person conversation KWS, the SS module using a hybrid standard on mixed speech, which includes both text-aware standards containing keywords and PIT-based non-keyword negative speech, is crucial to prevent the front-end module from overfitting keyword predictions.

[0061] This application proposes a novel text-aware speech separation method that can effectively select speech channels with specific keywords from mixed speech. Here, PIT is used to process mixed non-keyword speech, and text-guided permutation determination training is introduced for speech containing keywords. Experimental results show that the text-aware speech separation method provided in this application can effectively alleviate the permutation problem of audio containing keywords and achieve better separation performance. Therefore, compared with KWS systems using traditional speech separation methods, the multi-speaker KWS performance of the system proposed in this application is significantly better.

[0062] In some examples of embodiments of this application, the keyword search module includes a multi-scale deep temporal convolution (MDTC) model. By employing MDTC to construct the KWS module, multi-scale features with different receptive fields are extracted based on multi-scale deep temporal convolution (MDTC), achieving good performance while retaining a small number of parameters.

[0063] Figure 3 A schematic diagram of the architecture of an example keyword search module according to an embodiment of this application is shown.

[0064] like Figure 3 As shown, the keyword search module adopts an MDTC architecture with m DTC stacks, and each DTC stack contains n DTC blocks.

[0065] In this embodiment of the application, for training the KWS module (MDTC), binary classification is used for preset fixed keywords, and the following enhanced maximum pooling loss function is adopted:

[0066]

[0067] Where, p i Let represent the predicted posterior probability of the keyword at time step i, y∈{0,1} be the actual label of the utterance, m be the minimum number of frames for the keyword, and M be the total number of time steps for each speech segment.

[0068] Furthermore, embodiments of this application experimentally validate the provided model to compare the impact of different models and training modes. For all training modes employed, embodiments of this application carefully limit the amount of data and the consistency of training steps to ensure fair comparison.

[0069] Specifically, the experimental configuration mainly involves configuring the dataset, model configuration, and the selection method for multi-person conversation KWS.

[0070] For dataset configuration, the Libri2Mix and Snips2mix datasets were selected. Specifically, the Libri2Mix dataset is a noisy speech separation dataset with an average SI-SNR of -2dB. In this embodiment, train-100 was used as the training dataset containing keyword-independent audio, and test was used for testing. In the Snips2mix dataset, Hey-snips is a crowdsourced dataset where the preset "hey snips" is used as the wake word. In this embodiment, overlapping keyword-related audio was simulated according to a Libri2mix script. Each heysnips audio was mixed with another non-keyword audio randomly selected from the Librispeech train-100 dataset to construct the Snips2mix dataset.

[0071] It should be noted that WHAM! noise was also added to the mixed signal, resulting in an average SI-SNR of -2.5dB. The original splits of all datasets were maintained during simulation. For testing, an additional 10,000 keyword-independent audio samples were simulated using negative samples from the hey-snips test set and audio from the librispeech test-clean set. The libri2mix test set was also included. Overall, the number of positive and negative samples during testing was 3,000 and 13,000, respectively.

[0072] In terms of model configuration, all splitter models contain 6 blocks. Within each block, the LSTM has a hidden dimension of 256. The audio segment size, i.e., the feature length processed by the Seg-LSTM, is set to 48. The encoder and decoder both have a stride of 16, while the convolutional kernel size is 32. The hidden dimension of the cross-conformer is set to 128. In this embodiment, the condition list contains audio segments randomly selected from the Hey-snips training set, which remains constant during training and testing. For the KWS module, the number of stacks in the MDTC is 3, and the size of each stack is 4. The hidden dimension of the MDTC is set to 32. Training and inference are both performed using the ESPnet toolkit.

[0073] Regarding the selection of the Keyword Sounding Module (KWS) for multi-person conversations, although the KWS module is typically trained on non-overlapping audio, this embodiment employs three selection methods to merge multiple outputs for better comparison with the KWS performance of the PIT model. In the Oracle selection method, this embodiment selects the output channel that maximizes the SI-SNR. Oracle selection reveals the upper limit of system performance because it depends on the reference of the keyword audio. In the maximum selection method, this embodiment applies the keyword recognition module to two output channels and selects the highest score as the final output. In the no-selection method, since this embodiment imposes permutation constraints on keyword-related audio, channel selection is unnecessary. Therefore, this embodiment applies the KWS module only to the output of the primary channel (preset as the keyword channel).

[0074]

[0075] Table 1 shows the performance of the SS model trained solely on Snips2mix.

[0076] This indicates that only L is used during training. k This is unstable. Training focuses solely on keyword-related audio, while the test scenarios contain various negative samples. This mismatch can cause the model to either overfit to predict keyword-related audio or produce outputs with excessive noise. To address this issue, this application embodiment further introduces LPIT and includes non-keyword speech in the training data. Table 2 compares two training modes between Skim and TSKim trained on the joint datasets of Snips2mix and Libri2mix.

[0077]

[0078] Table 2 shows the performance of the speech separation models trained on Snips2mix and Libri2mix.

[0079] As shown in Table 2, including non-keyword speech in the PIT training is both effective and necessary for achieving text-aware speech separation. The hybrid training standard proposed in this application helps achieve better separation performance when keyword audio is included, while there is no significant degradation in non-keyword audio.

[0080] It should be noted that KWS performance is greatly influenced by separation performance, but is not solely determined by it. This application uses the same KWS model pre-trained on non-overlapping Hey-snips data to evaluate the impact of different SS models. The baseline SS model is Skim trained using LPIT with the maximum selection method. Its performance is compared to the TSKim SS model trained using Lk+LPIT. All SS models are trained on the merged datasets of Snips2mix and Libri2mix. Furthermore, this application's embodiments demonstrate oracle selection results to illustrate the gap between this application's embodiments and the upper limit.

[0081] Figure 4 A schematic diagram illustrating a performance comparison of TSKim chn1 with other models in KWS performance according to an embodiment of this application is shown. Figure 4 As shown, the method TSKimchn1 according to the embodiments of this application has a lower false alarm rejection rate, indicating superior performance under the same false alarm conditions. Therefore, the method TSKimchn1 of the embodiments of this application significantly outperforms the benchmark in KWS performance, highlighting the effectiveness of text-aware separation in multi-speaker KWS of the embodiments of this application. Meanwhile, the method of the embodiments of this application eliminates the selection method among multiple outputs, reducing computational overhead. Furthermore, by comparing the performance gap between the method of the embodiments of this application and its oracle, it can be concluded that the model of the embodiments of this application effectively alleviates the permutation problem in the output of keyword-related audio. It is worth noting that many related works train the KWS module together with the front-end module to obtain further performance improvements. However, in the work of the embodiments of this application, the training of the KWS module must be performed after the training process of text-aware separation to mitigate the impact of insufficient audio separation and potential permutation problems. Considering that the performance gap of the separation model may lead to further differences and make fair comparison challenging, the KWS module was not tuned using separated audio.

[0082] Regarding the impact of text-aware conditions, the condition list is crucial for assisting the text-aware separation system. This application's embodiments construct different condition lists for comparative experiments to study the effects of different conditions. Here, the plain text condition is labeled "zero audio".

[0083]

[0084] Table 3 shows the false alarm rejection rate (FRR) and memory usage relative to Skim at a false alarm rate of 0.5.

[0085] As shown in Table 3, it is evident that using audio as a text-aware condition, which contains richer information than plain text, leads to superior performance. Furthermore, since the keyword information extraction module in this embodiment is trained from scratch, the system performance may degrade to the level of plain text conditions when the number of audio samples is insufficient. While a larger condition list generally achieves better performance, it also introduces additional computational overhead. Meanwhile, the relative performance improvement gradually diminishes, making a condition size of 50 a seemingly well-balanced choice.

[0086] This application provides a novel text-aware speech separation method that addresses the long-tailed but poorly studied problem of multi-speaker keyword detection. The method proposes a text-aware separator with a deterministic training criterion. Experiments demonstrate the effectiveness and necessity of the introduced training pattern. Although the output order is significantly limited without significantly reducing speech separation performance, the method of this application achieves superior keyword detection performance and reduces computational overhead compared to PIT separation. In the future, the technical solution provided in this application will be extended to the field of robust open-ended vocabulary keyword detection.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] like Figure 5 The diagram shown is a schematic diagram of a multi-speaker voice keyword recognition device provided in an embodiment of this application. This multi-speaker voice keyword recognition device can execute the multi-speaker voice keyword recognition method described in any of the above embodiments and is configured in a terminal.

[0089] The multi-speaker speech keyword recognition device 500 provided in this embodiment includes an acquisition unit 510, an extraction unit 520, a separation unit 530, and a recognition unit 540.

[0090] The acquisition unit 510 is configured to acquire multi-speaker audio to be identified.

[0091] The extraction unit 520 is configured to extract a keyword embedding condition matrix corresponding to the target keyword from the multi-speaker audio based on the keyword information extractor.

[0092] The separation unit 530 is configured to input the keyword embedding condition matrix and the multi-speaker audio into the text-aware speech separation module, so that the text-aware speech separation module can separate the latent keyword audio sequence from the multi-speaker audio; the latent keyword audio sequence contains at least one latent keyword audio with a fixed order, and each latent keyword audio is a predicted audio segment containing the target keyword.

[0093] The recognition unit 540 is configured to determine the recognition probability of the potential keyword audio sequence for the existence of the target keyword based on the keyword search module.

[0094] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the multi-speaker speech keyword recognition methods described above.

[0095] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described multi-speaker speech keyword recognition methods.

[0096] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a multi-speaker speech keyword recognition method.

[0097] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a multi-speaker speech keyword recognition method according to another embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0098] One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0099] The device for performing the multi-speaker speech keyword recognition method may further include an input device 630 and an output device 640.

[0100] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0101] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-speaker speech keyword recognition method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the multi-speaker speech keyword recognition method in the above-described method embodiments.

[0102] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] Input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. Output device 640 may include display devices such as a display screen.

[0104] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they perform the multi-speaker speech keyword recognition method in any of the above method embodiments.

[0105] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0106] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0107] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0108] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0109] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0110] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for recognizing keywords in multi-speaker speech, comprising: Acquire the audio of the multiple speakers to be identified; Based on the keyword information extractor, a keyword embedding condition matrix corresponding to the target keywords is extracted from the multi-speaker audio. The keyword information extractor includes an audio encoder; The keyword embedding condition matrix and the multi-speaker audio are input into the text-aware speech separation module, so that the text-aware speech separation module can separate the latent keyword audio sequence from the multi-speaker audio; the latent keyword audio sequence contains at least one latent keyword audio with a fixed order, and each latent keyword audio is a predicted audio segment containing the target keyword; Based on the keyword search module, the probability of recognizing the target keyword in the potential keyword audio sequence is determined; The step of extracting the keyword embedding condition matrix corresponding to the target keywords from the multi-speaker audio based on the keyword information extractor includes: Based on the audio encoder, each labeled keyword audio in the preset keyword audio list is matched with the multi-speaker audio to extract at least one keyword embedding corresponding to the multi-speaker audio; each labeled keyword audio is a labeled audio segment with the keyword, and each labeled audio segment has a corresponding speaker voice background. Based on each of the aforementioned keyword embeddings, a keyword embedding condition matrix is ​​constructed.

2. The method according to claim 1, wherein, The text-aware speech separation module includes an encoder, a decoder, and a separator. The encoder includes a 1-D convolutional layer, the decoder includes a 1-D transposed convolutional layer, and the separator includes multiple sequentially cascaded text-aware skip memory blocks. The last text-aware skip memory block in the separator contains a Mem-LSTM, and the other text-aware skip memory blocks contain cascaded Seg-LSTM and Mem-LSTM.

3. The method according to claim 2, wherein, The text-aware skip memory block includes a cross-Conformer block, the output of which is used as the input of the Seg-LSTM; the cross-Conformer block includes multiple feedforward neural network layers, a multi-head cross-attention layer, a convolutional layer, and a layer normalization module; wherein, the query of the multi-head cross-attention layer is defined by the mixed signal features corresponding to the multi-speaker audio, and the key and value of the multi-head cross-attention layer are defined by the keyword embedding condition matrix.

4. The method according to claim 3, wherein, The text-aware speech separation module uses the temporal SI-SNR loss function.

5. The method according to claim 4, wherein, The text-aware speech separation module is trained using a permutation-independent training method.

6. The method according to claim 1, wherein, The keyword search module includes a multi-scale deep temporal convolution model.

7. The method according to claim 6, wherein, The loss function of the multi-scale deep temporal convolution model is: in, p i Indicates at time step i The predicted posterior probability of the keyword. y ∈ {0, 1} are the true reality labels of the discourse. m It is the minimum frame count for the keyword, and M It represents the total number of time steps for each audio segment.

8. A multi-speaker speech keyword recognition device, comprising: The acquisition unit is configured to acquire multi-speaker audio to be identified; The extraction unit is configured to extract a keyword embedding condition matrix corresponding to the target keyword from the multi-speaker audio based on a keyword information extractor. The keyword information extractor includes an audio encoder; The separation unit is configured to input the keyword embedding condition matrix and the multi-speaker audio into the text-aware speech separation module, so that the text-aware speech separation module can separate the latent keyword audio sequence from the multi-speaker audio; the latent keyword audio sequence contains at least one latent keyword audio with a fixed order, and each latent keyword audio is a predicted audio segment containing the target keyword; The recognition unit is configured to determine the recognition probability of the potential keyword audio sequence for the existence of the target keyword based on the keyword search module; The step of extracting the keyword embedding condition matrix corresponding to the target keywords from the multi-speaker audio based on the keyword information extractor includes: Based on the audio encoder, each labeled keyword audio in the preset keyword audio list is matched with the multi-speaker audio to extract at least one keyword embedding corresponding to the multi-speaker audio; each labeled keyword audio is a labeled audio segment with the keyword, and each labeled audio segment has a corresponding speaker voice background. Based on each of the aforementioned keyword embeddings, a keyword embedding condition matrix is ​​constructed.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

Images