Keyword identification method and apparatus, electronic device, and storage medium

Abstract

The application provides a keyword recognition method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: determining a set of available modal pairs based on at least one registration modal feature and at least one query modal feature, wherein the modal pairs in the set of available modal pairs are combinations of any query modal feature and any registration modal feature; obtaining a hidden state vector of each modal pair, and aggregating all the hidden state vectors to generate an aggregated feature vector for determining a recognition result. The application constructs a recognition framework that can dynamically determine modal pairs and aggregate corresponding features according to the actual available registration and query modal combinations, solves the problem of sharp performance degradation or failure in the modal missing scene, realizes flexible compatibility and adaptive fusion of different modal inputs, and significantly enhances the generalization ability, robustness and practicality of the keyword recognition system in complex and variable real application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a keyword recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the popularity of Internet of Things and smart home, smart vehicle and other applications, the keyword recognition system (KWS) as the key entry of human-computer interaction is increasingly adaptable to different scenarios and user personalized experience requirements, and higher requirements are put forward for recognition technology that can flexibly handle multiple input forms and maintain high accuracy.

[0003] To meet the above requirements, the self-defined keyword recognition system in the prior art can be mainly divided into audio registration type and text registration type according to different registration methods. Among them, some advanced schemes start to introduce multi-modal information to improve robustness in complex environments, such as combining video signals synchronized with query audio for joint decision. Such multi-modal systems usually construct a fixed processing pipeline, in which the query audio features, query video features and features of a preset registration modality (such as text) are fused and processed, and finally input to a classifier to obtain a recognition result.

[0004] The above-mentioned prior art scheme usually presets in the design that all modal information used for joint decision is complete and available. Such a processing architecture constructed based on a fixed combination of modalities has inherent limitations when facing the application scenarios of missing or incomplete modalities in the real world. For example, when the user selects a modality that is not preset by the fixed architecture for keyword registration, or the video modality cannot be effectively collected due to camera obstruction, the user being out of the field of view, etc. during query, the processing flow of the entire recognition system is interrupted or invalid due to mismatched input information, thereby severely limiting its generalization ability and actual availability under diversified and uncertain interaction conditions. SUMMARY

[0005] The present application provides a keyword recognition method, device, electronic device and storage medium to solve the problem of performance degradation or failure of multi-modal systems in the prior art due to the preset of all complete and available modalities, and to realize flexible compatibility and adaptive fusion of different modal inputs.

[0006] The present application provides a keyword recognition method, comprising the following steps:

[0007] determine a set of available modal pairs based on the registration modal features obtained in the at least one registration modal and the query modal features obtained in the at least one query modal; the modal pairs in the set of available modal pairs are combinations of any of the query modal features and any of the registration modal features;

[0008] obtain a hidden state vector corresponding to each of the modal pairs in the set of available modal pairs;

[0009] aggregate all the hidden state vectors to generate an aggregated feature vector;

[0010] determine a recognition result based on the aggregated feature vector; the recognition result is used to represent whether a preset keyword is contained in a query input corresponding to any query modal feature.

[0011] According to the keyword recognition method provided by the application, the registration modal at least includes one of text registration, audio registration and video registration, and the query modal at least includes one of voice query and video query.

[0012] When the query modal includes both the voice query and the video query, the voice query and the video query are synchronous.

[0013] According to the keyword recognition method provided by the application, when it is determined that the query modal includes voice query, before determining the set of available modal pairs, the following steps are iteratively performed until a first preset stop condition is met:

[0014] frame-level segmentation is performed on the query voice signal in the query input to obtain a query audio frame-level representation sequence;

[0015] determine a noise reduction mask, the noise reduction mask is determined based on the correlation calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and at least one of the query video signal of the video query and the registration modal signal in the registration modal;

[0016] de-noise the query voice signal using the noise reduction mask.

[0017] According to the keyword recognition method provided by the application, the determination of the noise reduction mask comprises:

[0018] respectively calculate the cross-attention calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal to create a video-associated attention output sequence;

[0019] respectively calculate cross attention calculation results between each of the query audio frame level representations in the query audio frame level representation sequence and each of the enrollment modal signals to create an enrollment associated attention output sequence corresponding to each of the enrollment modal signals;

[0020] respectively determine a corresponding modal mask according to the video associated attention output sequence and / or each of the enrollment associated attention output sequences;

[0021] perform weighted fusion on all modal masks to generate the noise reduction mask.

[0022] According to the keyword recognition method provided by the application, the determination of the corresponding modal mask according to the video associated attention output sequence and each of the enrollment associated attention output sequences comprises:

[0023] input the video associated attention output sequence and the enrollment associated attention output sequence into a mask generation network respectively, and obtain the modal mask output by the mask generation network;

[0024] The mask generation network comprises a convolution layer and a nonlinear activation function layer. The convolution layer is used for feature extraction on the video associated attention output sequence or the enrollment associated attention output sequence to generate a feature matrix. The nonlinear activation function layer is used for element-wise probability operation on the feature matrix to generate the modal mask.

[0025] According to the keyword recognition method provided by the application, the obtaining of the hidden state vector corresponding to each of the available modal pairs comprises:

[0026] determine a similarity matrix between the query modal feature and the enrollment modal feature in each of the modal pairs, wherein the similarity matrix represents the matching degree of the query modal feature and the enrollment modal feature at different time points;

[0027] input the similarity matrix into a recurrent neural network to obtain the hidden state vector corresponding to each of the modal pairs output by the recurrent neural network.

[0028] According to the keyword recognition method provided by the application, the aggregation of all the hidden state vectors to generate an aggregated feature vector comprises:

[0029] splicing the hidden state vectors corresponding to all the modal pairs in the available modal pair set to generate the aggregated feature vector.

[0030] According to the keyword recognition method provided by the application, the determination of the recognition result based on the aggregated feature vector comprises:

[0031] inputting the polyphonic feature vector into a classifier network, and obtaining the recognition result output by the classifier network.

[0032] According to the keyword recognition method provided in the application, the method further comprises:

[0033] A training sample set is collected, and the training sample set comprises multiple groups of training samples, each group of training samples comprising at least one registration input sample in a registration mode, at least one query input sample in a query mode, and a pre-labeled recognition result label and a clean audio signal label;

[0034] The training process of a to-be-trained keyword recognition model is iteratively performed using the training sample set until a second preset stop condition is met.

[0035] In each training process, a batch of training samples is randomly selected from the training sample set, and the following training steps are performed until all training samples in the batch are traversed:

[0036] A group of training samples selected from the batch is taken as input of the to-be-trained keyword recognition model, and a predicted recognition result output by the to-be-trained keyword recognition model and a noise reduction speech training signal and a frame-level audio-video fusion vector sequence generated in the process are obtained;

[0037] A decision loss between the predicted recognition result and the recognition result label of the training sample is calculated.

[0038] A noise reduction loss between the noise reduction speech training signal and the clean audio signal label of the training sample is calculated.

[0039] After the frame-level audio-video fusion vector sequence is mapped to a feature space where text phoneme features are located, weighted processing based on a time window center Gaussian distribution is performed to generate phoneme-level fusion feature representation, so as to calculate a spatial alignment loss between the fusion feature representation and the text phoneme feature.

[0040] Based on the decision loss, the noise reduction loss and the spatial alignment loss, parameters of the to-be-trained keyword recognition model are updated.

[0041] The to-be-trained keyword recognition model at least comprises a feature encoder, the mask generation network, the recurrent neural network and the classifier network.

[0042] According to the keyword recognition method provided in the application, after a batch of training samples is randomly selected from the training sample set in each training process, the method further comprises:

[0043] Randomly remove the training samples in the batch related to at least one registration modality and / or at least one query modality.

[0044] According to the keyword recognition method provided by the application, the frame-level audio-video fusion vector sequence is determined based on the following manner:

[0045] The audio feature sequence corresponding to the audio signal sample and the video feature sequence corresponding to the video signal sample in the training sample are exchanged in features on at least one time frame with a preset probability;

[0046] The frame-level audio-video fusion vector sequence is obtained by fusing the audio signal sample and the video signal sample after the feature exchange.

[0047] According to the keyword recognition method provided by the application, the parameters of the to-be-trained keyword recognition model are updated based on the decision loss, the noise reduction loss and the spatial alignment loss, including:

[0048] The parameters of the mask generation network are updated by integrating the decision loss, the spatial alignment loss and the noise reduction loss;

[0049] The parameters of the feature encoder are updated by integrating the decision loss and the spatial alignment loss;

[0050] The parameters of the classifier network and the recurrent neural network are updated based on the decision loss.

[0051] According to the keyword recognition method provided by the application, the parameters of the mask generation network are updated by integrating the decision loss, the spatial alignment loss and the noise reduction loss, including:

[0052] The combined loss is obtained by weighted sum of the decision loss, the noise reduction loss and the spatial alignment loss;

[0053] The parameters of the mask generation network are updated based on the combined loss;

[0054] The weight coefficient of the weighted sum is dynamically determined based on the change gradient of the decision loss, the noise reduction loss and the spatial alignment loss in the historical training process.

[0055] According to the keyword recognition method provided by the application, the parameters of the mask generation network are updated based on the combined loss, including:

[0056] According to the keyword recognition method provided by the application, the decision loss is binary cross entropy loss, the noise reduction loss is scale invariant signal-to-noise ratio improvement loss, and the spatial alignment loss is mean square error loss.

[0057] The application further provides a keyword recognition device, comprising the following structure:

[0058] a feature extraction unit configured to determine a set of available modal pairs based on preset registration modal features obtained in at least one registration modal and query modal features obtained in at least one query modal, wherein the modal pairs in the set of available modal pairs are combinations of any of the query modal features and any of the registration modal features;

[0059] a feature analysis unit configured to obtain a hidden state vector corresponding to each of the modal pairs in the set of available modal pairs;

[0060] a feature aggregation unit configured to aggregate all the hidden state vectors to generate an aggregated feature vector;

[0061] a result recognition unit configured to determine a recognition result based on the aggregated feature vector, wherein the recognition result is used to represent whether a preset keyword is contained in a query input corresponding to any of the query modal features.

[0062] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the keyword recognition method according to any of the above when executing the program.

[0063] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the keyword recognition method according to any of the above.

[0064] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the keyword recognition method according to any of the above.

[0065] The keyword recognition method, device, electronic device, and storage medium provided by the application can dynamically determine modal pairs and aggregate corresponding features of the modal pairs by constructing an identification framework capable of combining actual available registration and query modalities, thereby solving the problem of sharp performance degradation or failure in a modality missing scenario, achieving flexible compatibility and adaptive fusion of different modal inputs, and significantly enhancing the generalization ability, robustness, and practicality of a keyword recognition system in a complex and variable real application scenario. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort based on these drawings.

[0067] Figure 1 is a flowchart of the keyword recognition method provided by the present application.

[0068] Figure 2 is a flowchart of the noise reduction of the query speech signal provided by the present application.

[0069] Figure 3 is a flowchart of the generation of the noise reduction mask provided by the present application.

[0070] Figure 4 is an architecture diagram of the keyword recognition system provided by the present application.

[0071] Figure 5 is a training flowchart of the keyword recognition model provided by the present application.

[0072] Figure 6 is a structure diagram of the keyword recognition device provided by the present application.

[0073] Figure 7 is a structure diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will combine the drawings in the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of protection of the present application.

[0075] It should be noted that in the description of the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices comprising a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element. The above terms can be understood according to their specific meanings in the present application by those skilled in the art.

[0076] The terms "first", "second", and the like in the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0077] With the gradual evolution of human-computer interaction towards universalization, personalization and diversification, as a key entry in the human-computer interaction process, the keyword recognition system plays an important role in the wake-up device and task start in the conversational artificial intelligence system, and its role is particularly crucial in the application scenarios of voice assistants, intelligent vehicle systems, smart homes, etc.

[0078] Traditional keyword recognition systems usually rely on fixed and predefined keyword configurations, although they have certain practical value in terms of universality, but there are obvious limitations in flexibility and user personalization needs. In order to overcome this problem, custom keyword recognition technology has emerged, which enables users to register custom keywords according to personal preferences, thereby significantly improving the freedom and adaptability of the system.

[0079] Existing custom keyword recognition systems are mainly divided into two categories: audio registration and text registration. The audio registration method allows users to input keyword templates through recording, which has the advantages of recording pronunciation characteristics and adapting to individual differences, but at the same time is easily affected by environmental noise, resulting in a decline in recognition performance. The text registration method matches the user input keyword text with the speech to be recognized, avoiding the problem of audio collection, and has good structural stability, but requires high accuracy of user pronunciation, especially in dialect, accent and other diversified language environments.

[0080] In addition, the existing custom keyword recognition system is mostly limited to single modal registration (i.e. only supports audio registration or text registration), and there is still a deficiency in the joint use of multi-modal information. For example, it cannot combine the lip movement information in the video to assist in determining the pronunciation person, or automatically make robust judgments in the case of missing modalities, thereby limiting the recognition ability and practicality of the custom keyword recognition system in complex scenarios.

[0081] In order to solve the technical bottlenecks of the existing self-defined keyword recognition method in aspects of single modal registration, poor noise robustness, rough spatial alignment and the like, the present application provides a self-defined keyword recognition method based on modal flexible fusion, which can support keyword registration and recognition of three modalities of text registration, audio registration and video registration in a unified model architecture, and significantly improve the adaptability and recognition accuracy under multi-scene and multi-modal conditions.

[0082] As a core innovation point, in view of the fact that part of the registration modal information may be missing in the actual scene (such as no effective video registration or no text registration provided), the present application provides a modal adaptable fusion method, which supports degradation to a non-full modal mode in the case of missing any registration modal, optimizes the effects of single modal, double modal and triple modal at the same time, so that the system can still maintain high-precision recognition ability when the registration modal information is incomplete, and enhance the generalization and practicality of keyword recognition.

[0083] The following will be described in combination with Figures 1-7 The keyword recognition method, device, electronic equipment and storage medium provided by the present application are described.

[0084] Figure 1 The flowchart of the keyword recognition method provided by the present application is shown in FIG. 1, and the present application embodiment provides a keyword recognition method which can be applied to an electronic equipment. Figure 1 The electronic equipment can be any device with computing processing capability, such as a smart phone, a tablet computer, a personal computer, a server, a smart speaker, a smart television, a vehicle information entertainment system or any other form of smart terminal. For the convenience of description, the system is uniformly referred to as a system hereinafter.

[0085] In the present embodiment, the keyword recognition method provided by the present application mainly includes but is not limited to the following steps:

[0086] Step 101, based on the registration modal features obtained under the preset at least one registration modal and the query modal features obtained under the at least one query modal, a set of available modal pairs is determined.

[0087] The registration modal can refer to the information form used by the user to pre-enter the self-defined keyword information into the system. The registration modal can be one or a combination of multiple. In an optional embodiment, the registration modal can include but is not limited to: text registration, i.e. the user inputs the text string of the keyword through the keyboard or handwriting; audio registration, i.e. the user records a segment of speech containing the keyword through the microphone; and video registration, i.e. the user records a segment of facial video speaking the keyword through the camera, especially the lip movement video.

[0088] When a user enters personal custom keyword information, a unique ID can be assigned to each user, and the registered text, registered audio and / or registered video entered by each user are bound to the ID assigned thereto and uploaded to a background storage, which can be stored uniformly in a cloud server.

[0089] As an optional embodiment, for any registration modality, each user can record multiple pieces of information. For example, user A can record two different registered texts (different registered texts correspond to different registered keywords) in the same time period or different time periods. In this way, any registered keyword can be identified in the subsequent keyword identification process.

[0090] The registration modality feature is a mathematical representation, such as a high-dimensional vector or a vector sequence, obtained by feature encoding of the registration input of the above registration modality.

[0091] The query modality refers to the form of real-time information provided by the user when triggering keyword identification in actual use. The query modality can be one or multiple. In an optional embodiment, the query modality can include but is not limited to: audio query, i.e., real-time acquisition of user sound signals through a microphone; video query, i.e., real-time acquisition of user video pictures through a camera. The query modality feature is a mathematical representation obtained by feature encoding of the query input of the above query modality.

[0092] The available modality pair set refers to the set of all modality pairs that can be actually used for comparison and analysis in the current specific identification task. All modality pairs in the available modality pair set are formed by traversing any combination of the current available query modality feature and any preset registration modality feature. For example, if the user presets two registration modalities of text and audio, and provides two query modalities of audio and video at the time of query, the available modality pair set at this time includes four modality pairs: (text registration, audio query), (text registration, video query), (audio registration, audio query), and (audio registration, video query). By determining the available modality pair set, the system can dynamically identify all currently available information processing paths.

[0093] Step 102, obtaining a hidden state vector corresponding to each modality pair in the available modality pair set.

[0094] The hidden state vector is a high-dimensional feature vector that can contain sequence context dependency relationships obtained by processing through a time series modeling network (such as a recurrent neural network).

[0095] In this step 102, for each modality pair in the set of determined available modality pairs, the system independently executes a processing flow to generate its corresponding hidden state vector. The processing flow can include calculating the similarity between the enrollment modality feature and the query modality feature in each modality pair, and modeling the temporal change of the similarity in the time dimension, thereby obtaining a hidden state vector that can represent the matching degree of this modality pair and its temporal dynamics.

[0096] Step 103, aggregate all the hidden state vectors to generate an aggregated feature vector. In this embodiment, the system combines all the available hidden state vectors (the number of which depends on the number of available modality pairs) to generate a single aggregated feature vector.

[0097] The aggregation method used can include but is not limited to concatenation, element-wise addition, element-wise multiplication, or weighted summation through an attention network, etc. For example, if two hidden state vectors are obtained in the previous step, they can be concatenated by connecting the two vectors head to tail to form an aggregated feature vector with a dimension of the sum of the two. The aggregated feature vector collects analysis information from all available modality pairs, forming a panoramic description of the current query event.

[0098] Step 104, determine the recognition result based on the aggregated feature vector; the recognition result is used to represent whether the query input corresponding to any query modality feature contains the preset keyword.

[0099] The system uses the aggregated feature vector generated in the previous step as the final basis for decision-making, and determines the final recognition result through a preset decision mechanism. The recognition result is used to represent whether the query input contains the user's custom keyword.

[0100] For example, the aggregated feature vector can be input into a preset classifier, and a confidence score can be output by the classifier. Then the confidence score is compared with a preset decision threshold. If the score is greater than or equal to the decision threshold, the recognition result is "hit" or "yes", otherwise it is "miss" or "no".

[0101] The keyword recognition method provided by the present application solves the problem of rapid performance degradation or failure in the absence of modalities by constructing an identification framework that can dynamically determine modality pairs and aggregate their corresponding features according to the actual available enrollment and query modalities, and realizes flexible compatibility and adaptive fusion of different modal inputs, significantly enhancing the generalization ability, robustness and practicality of the keyword recognition system in complex and variable real application scenarios.

[0102] As an optional embodiment, the registration modalities include at least one of text registration, audio registration and video registration, and the query modalities include at least one of voice query and video query.

[0103] In the present embodiment, the specific types of the registration modalities and the query modalities are exemplarily described. The registration modalities, as the forms for the user to input the custom keyword information, in an optional embodiment, can include:

[0104] Text registration, i.e., the user inputs one or more custom text strings as keywords through the interactive interface of the system, such as "turn on the big light" or "navigate back home".

[0105] Audio registration, i.e., the user records a voice clip of pronouncing one or more custom keywords through the microphone of the system.

[0106] Video registration, i.e., the user records a facial video of pronouncing one or more custom keywords through the camera of the system, which is mainly used to capture the lip movement information related to pronunciation.

[0107] The user can select any one or a combination of the above registration modalities to register the keywords. As an optional implementation, the system can assign a unique user identification (ID) to each user. When the user inputs the personal custom keyword information, the system can bind the registered text, the registered audio and / or the registered video input by the user with the user identification assigned to the user, and upload the bound information to the background server for storage, for example, can be uniformly stored in the data space dedicated to the user in the cloud server.

[0108] As another optional implementation, for any registration modality, each user can record multiple pieces of information. For example, user A can record two different registration texts (different registration texts correspond to different registration keywords, such as "keyword 1: play music" and "keyword 2: increase volume") in the same time period or different time periods. In this way, in the subsequent keyword recognition process, the system can recognize any keyword registered by the user. For example, for the same keyword "hello", the user can record multiple registration audios, so that the system can learn the pronunciation features of the user in different states (such as speed, emotion change), thereby improving the robustness of recognition.

[0109] Query modalities, as the information form when the user actually triggers recognition, in an optional embodiment, can include:

[0110] Audio query, i.e., the system collects the environmental sound signals (which may contain the user query voice) in real time through the microphone, which is the most basic and common query modality.

[0111] The video query is a video picture collected by the camera in real time, and in particular, a video information containing the face of the user.

[0112] In actual application, the query mode can be an audio query only (for example, the user is driving and is not facing the vehicle-mounted camera), can be an audio query and a video query simultaneously, and in some special cases, can be a video query only, that is, only the query video input by the user is used to realize the recognition of the keyword by capturing the lip movement information related to pronunciation.

[0113] It should be noted that when the query mode simultaneously includes the voice query and the video query, the voice query and the video query occur synchronously.

[0114] Synchronous occurrence means that the query audio signal collected by the audio query and the query video signal collected by the video query are generated by the same event (that is, the sound production behavior of the user) in the same time period, and the two are strictly aligned or can be aligned through technical means in the time stamp. Such natural and physical synchronicity provides a basis for subsequent joint analysis and processing through multi-modal information, for example, the lip movement video synchronized with the audio signal can be analyzed to assist in judging the pronunciation content or suppressing environmental noise, thereby improving the accuracy of recognition.

[0115] The keyword recognition method provided by the application limits the specific types of the registration mode and the query mode, so that it can cover the current mainstream human-computer interaction scene, and the sources and characteristics of different modal information are clear, which provides a clear input definition for subsequent multi-modal feature extraction and fusion processing, and enhances the implementability and scene adaptability of the scheme.

[0116] Based on the content of the above embodiment, as an optional embodiment, a multi-modal noise reduction technical scheme is provided to improve the quality of the query audio signal.

[0117] When the system detects that the current query input contains an audio query, an optional audio preprocessing process (that is, multi-modal noise reduction) can be started before the subsequent recognition process such as determining the available modal pair set is performed, which aims to use other available multi-modal information as an aid to purify the input query audio signal to suppress environmental noise, thereby improving the accuracy and robustness of subsequent keyword recognition.

[0118] Figure 2 is a flowchart of the noise reduction process for the query voice signal provided by the application, as shown in Figure 2 mainly includes but is not limited to the following steps:

[0119] After starting to perform the noise reduction on the query speech signal, first, frame-level segmentation is performed on the query audio signal in the query input to obtain a sequence of query audio frame-level representations.

[0120] Specifically, by segmenting the continuous time-domain query audio signal into a series of short-time audio frames that are consecutive in time, and extracting acoustic features (referred to as query audio frame-level representations) for each short-time audio frame to form a time-sequential feature sequence, i.e., a sequence of query audio frame-level representations. Each query audio frame-level representation can be, but is not limited to, Mel-Frequency Cepstral Coefficients (MFCC), Log-Mel Spectrogram (LMS), and other audio features.

[0121] Further, a noise reduction mask is determined. The noise reduction mask is a matrix or tensor with the same or compatible dimensions as the time-frequency representation of the audio signal to be processed, and the values inside the noise reduction mask are usually used to represent the degree to which the energy of the corresponding time-frequency unit in the original audio signal should be preserved.

[0122] In this embodiment, the generation of the noise reduction mask can be based on the results of a correlation calculation. Specifically, the core idea of the correlation calculation is to measure the correlation strength between each frame of the query audio signal and all currently available auxiliary information sources. The auxiliary information sources can include features of the query video signal synchronized with the query audio signal (when the video query modality also exists) and / or features of at least one registered modality signal (such as features of registered text information or features of registered audio signals or features of registered video signals) pre-recorded by the user. For example, the auxiliary information sources include only one or more of any combination of the query video signal, the registered text information, the registered audio signal, and the registered video signal.

[0123] By calculating the correlation between each query audio frame-level representation in the sequence of query audio frame-level representations and the one or more auxiliary information sources described above, the system can comprehensively determine which parts of the query audio are more likely to belong to the target content spoken by the target speaker and which parts are more likely to belong to noise. Based on the results of these comprehensive determinations, a noise reduction mask for guiding noise reduction is finally determined. This design embodies the flexibility of the keyword recognition method provided by the present application, i.e., the system will utilize all available auxiliary information at the moment to perform noise reduction, and will not forcibly require all auxiliary information to exist at the same time.

[0124] Finally, the query audio signal can be denoised using the noise reduction mask. The specific implementation of the denoising can be: first, the original query audio signal is converted into a time-frequency domain by a short-time Fourier transform (STFT) or the like to obtain a spectrum graph; then, the noise reduction mask determined in the previous step is applied to the spectrum graph by element multiplication or the like to attenuate or suppress the energy components corresponding to the noise in the spectrum graph; finally, the spectrum graph processed by the mask is reconstructed into a time-domain waveform by an inverse short-time Fourier transform (iSTFT) or the like, and a denoised query audio signal is obtained. The denoised query audio signal can be used for subsequent feature extraction and keyword recognition processes.

[0125] It should be noted that the above denoising process can be iteratively performed, that is, the output of one denoising can be used as the input of the next denoising, and the steps of determining the noise reduction mask and using the noise reduction mask for denoising can be repeatedly performed. The termination condition of the iteration is to determine whether the first preset termination condition is met, which can include but is not limited to: reaching a preset number of iterations, the improvement of the denoising effect being lower than a preset threshold, or the processing delay exceeding a preset upper limit. By iterative execution, better denoising effect can be obtained in a scene with low real-time requirement, and feature purity can be further improved. The number of iterations can be determined according to actual use requirements. When the real-time requirement is not high, the denoising can be iterated multiple times to obtain better effect; if the real-time requirement is high, the denoising can be performed once to improve the effect without introducing too much time delay.

[0126] The multi-modal denoising process introduced by the keyword recognition method provided by the application can flexibly use the currently available multi-modal information to perform directional noise suppression on the input query audio signal before executing the core recognition logic, thereby significantly improving the recognition performance and stability of the system in various complex noise environments.

[0127] Figure 3 is a flowchart of the process of generating a noise reduction mask provided by the application, as Figure 3 The implementation of determining a noise reduction mask provided by the embodiment can be divided into a structured processing flow including parallel computing and subsequent fusion.

[0128] First, the system determines whether there is a query video signal synchronized in time with the query audio signal to be denoised. At the same time, the user's pre-registered keyword information, i.e., the registered text information, the registered video signal and / or the registered audio signal (if any) is retrieved from the storage, and the corresponding feature representation is extracted.

[0129] Then, the system initiates multiple independent computing streams in parallel to obtain relevance assessments of the query audio signal with respect to different auxiliary information sources (query video signal, registered text information, registered video signal, and / or registered audio signal), such as computing, for each query audio frame-level representation in the query audio frame-level representation sequence, a relevance mapping between the query audio frame-level representation and the feature representation of the other auxiliary information source via a Cross-Attention mechanism. The computing streams can include:

[0130] First, the system computes a Cross-Attention result between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal to create a video-related attention output sequence.

[0131] Cross-Attention is a technique that can model the dependency between two different sequences. In this embodiment, each query audio frame-level representation in the query audio frame-level representation sequence is taken as a Query to perform attention computation with the feature representation of the entire query video signal as Key and Value. After performing this operation for all frames in the query audio frame-level representation sequence, a video-related attention output sequence is obtained. Each element in the video-related attention output sequence integrates the information in the entire query video signal that is most relevant to the corresponding query audio frame, thereby quantifying the physical synchronization and relevance between audio and video.

[0132] Second, the system computes a Cross-Attention result between each query audio frame-level representation in the query audio frame-level representation sequence and each registered modality signal to create a registered-related attention output sequence for each registered modality signal.

[0133] This step is performed in parallel for all available registered modality signals. For example, if there are both registered text information and registered audio signal, the system performs the following operations:

[0134] (1) Perform a Cross-Attention computation with each query audio frame-level representation in the query audio signal as Query and the feature representation of the entire registered text information as Key and Value to create a text-related attention output sequence.

[0135] (2) Perform another Cross-Attention computation with each query audio frame-level representation in the query audio signal as Query and the feature representation of the entire registered audio signal as Key and Value to create an audio-related attention output sequence. Through this step, the consistency of the query audio signal with the standard pronunciation of the registered keywords (from the registered text information) and the similarity of the query audio signal with the user's voiceprint features (from the registered audio signal) can be quantified.

[0136] Then, a corresponding modal mask is determined for each of the video-related attention output sequence and / or the text registration-related attention output sequence.

[0137] To convert the obtained abstract attention output sequence into a specific control signal, the system determines a corresponding modal mask for each attention output sequence (e.g., a video-related attention output sequence, a text registration-related attention output sequence, etc.) independently. The modal mask is a matrix compatible with the time-frequency representation dimension of the query audio, and each element value of the matrix represents the degree of retention of the corresponding component of the audio signal based on the judgment of the corresponding auxiliary information source.

[0138] In an optional embodiment, the process of determining the modal mask can include inputting the attention output sequence into a network comprising a convolutional layer and a nonlinear activation function for processing. The output value range of the activation function can be set to [0, 1], and its physical meaning can be regarded as the retention probability of the corresponding query audio frame.

[0139] Finally, all modal masks are weighted and fused to generate the denoising mask, including combining all available modal masks (the number of which depends on the number of currently available auxiliary information sources) through a fusion strategy to generate a final, single denoising mask.

[0140] In an optional embodiment, the fusion strategy is weighted fusion, that is, each modal mask is multiplied by a corresponding weight coefficient and then summed. The weight coefficient can be a preset fixed value, or a learnable parameter learned during model training, to represent the importance of different auxiliary information sources in different noise scenarios, or to understand the contribution of each auxiliary information source to the denoising task under different conditions. For example, in a dim light and poor video quality scenario, the model training may automatically reduce the weight of the modal mask corresponding to the video modality, and in a noisy background voice, the weight of the modal mask corresponding to the video modality that is strongly related to the lip movement information may be increased.

[0141] The denoising method provided in the embodiment can effectively improve the feature purity and discriminability in the subsequent recognition stage, and the technical principle analysis is as follows:

[0142] (1) Utilize the strong physical correlation of audio and video to separate noise sources: The production of speech is physically synchronized and strongly correlated with the movement of the vocal organs, especially the lips. This invention uses this principle to effectively distinguish between query audio frames consistent with the current lip movement pattern (judged as valid speech) and inconsistent query audio frames (judged as noise, especially background voices or non-speech noise) by calculating the correlation between the query audio signal and the query video signal (containing lip movement features), providing a strong physical constraint for separating valid query speech features in complex acoustic environments.

[0143] (2) Utilize the semantic prior of registered text information for content focus: Registered text information provides the standard phoneme composition of keywords, which constitutes a high-level semantic constraint on audio content. During noise reduction, the system will tend to retain audio segments whose acoustic feature evolution path matches the phoneme sequence. This is equivalent to introducing a content filter, making the noise reduction process not only energy-based denoising, but also content-based focusing, thereby enhancing audio features related to keywords.

[0144] (3) Utilize the voiceprint characteristics of registered audio signals for identity locking: Registered audio signals contain unique voiceprint features of the user (such as fundamental frequency, formant, etc.). By comparing the query audio signal with the registered audio signal, the system can identify and enhance the audio parts that match the user's voiceprint characteristics, providing another basis for distinguishing target speakers from interfering speakers, achieving identity-based directional speech enhancement.

[0145] (4) Adopt adaptive fusion strategy to improve system robustness: By setting learnable weight coefficients to fuse each modal mask, an adaptive information fusion framework is constructed, allowing the system to learn the reliability of different modal masks in specific scenarios during training and dynamically adjust their contribution during inference, ensuring that the system can still combine available information in the best way in any case of missing or degraded modal information, thereby ensuring the stability of the noise reduction effect and the generalization ability to variable environments.

[0146] In summary, the keyword recognition method provided by the invention decomposes the determination process of the noise reduction mask into parallel cross-attention calculations, independent modal mask generation, and final weighted fusion, constructing a clear and scalable multi-modal information fusion framework that can effectively utilize cross-attention mechanisms to capture deep cross-modal associations and provide a reliable technical path for generating high-quality noise reduction masks.

[0147] As an optional embodiment, the conversion process from the abstract attention output sequence to the specific modality mask is implemented through a specially designed neural network module. The system configures an independent, parameter-unshared mask generation network for each attention output sequence (e.g., a video-related attention output sequence, a registration-related attention output sequence, etc.).

[0148] Specifically, the step of determining a corresponding modality mask can include:

[0149] First, the video-related attention output sequence and the registration-related attention output sequence are respectively input into a mask generation network to obtain the modality mask output by the mask generation network.

[0150] For example, for the video-related attention output sequence, the system inputs it as a whole into its exclusive mask generation network, and after a series of operations inside the network, it obtains the output audio mask for video guidance. Similarly, each registration-related attention output sequence is also sent into its exclusive mask generation network to generate its corresponding modality mask.

[0151] Secondly, the internal structure of the mask generation network is further preferred in this embodiment, which can include convolutional layers and nonlinear activation function layers, etc.

[0152] Among them, the convolutional layer is used to extract features from the video-related attention output sequence or the registration-related attention output sequence to generate a feature matrix.

[0153] In a preferred embodiment, the convolutional layer can adopt a one-dimensional convolutional layer. The convolutional layer slides in the time dimension of the attention output sequence through its internal convolution kernel to effectively integrate the context information of adjacent time frames and further abstract and transform the input attention features, thereby extracting more robust high-level features for distinguishing speech and noise to generate a feature matrix. In addition, by configuring the channel number and other parameters of the convolutional layer, the dimension of the feature can also be adjusted to match the requirements of subsequent processing.

[0154] The nonlinear activation function layer is used to perform element-wise probability operation on the feature matrix to generate the modality mask.

[0155] In a preferred embodiment, the nonlinear activation function can be a sigmoid activation function (such as Sigmoid function). The nonlinear activation function layer will perform a nonlinear transformation on each element of the feature matrix output by the previous convolutional layer independently. The Sigmoid function can smoothly map its input in the range of arbitrary real numbers to the open interval (0, 1). In the application scenario of the present application, the physical meaning of the output value between (0, 1) is the reservation probability of the corresponding time-frequency unit of the audio signal. A value close to 1 indicates that the component should be reserved, and a value close to 0 indicates that the component should be suppressed. Therefore, by performing such element-by-element probability operations on the feature matrix, the system can finally generate a numerical modal mask that can be directly used for subsequent masking operations.

[0156] The keyword recognition method provided by the present application explicitly implements the generation process from attention output to modal mask through a network comprising a convolutional layer and a nonlinear activation function. The use of the convolutional layer enables the system to effectively refer to the context association information of the previous and subsequent time frames when determining whether the current frame is noise, avoiding misjudgment based on instantaneous features and improving the smoothness and temporal rationality of the decision. The introduction of the nonlinear activation function layer successfully converts the abstract feature vector into a reservation probability with clear physical meaning, providing a quantitative and refined control basis for subsequent noise reduction processing, so that the target speech and noise can be more accurately separated, and the quality of the final noise reduction mask is improved.

[0157] As an optional embodiment, the present application provides an implementation process for generating a corresponding hidden state vector for each available modal pair, which can be specifically divided into a processing process comprising similarity calculation and temporal modeling, and can specifically include:

[0158] First, determine the similarity matrix between the query modal feature and the registration modal feature in each of the modal pairs, aiming to quantify the matching degree between the query modal feature and the registration modal feature in a modal pair.

[0159] In this embodiment, the similarity matrix is a two-dimensional matrix, whose rows and columns correspond to the constituent units of the registration modal feature (for example, the phoneme sequence of the registration text information, or the frame sequence of the registration audio signal) and the constituent units of the query modal feature (for example, the query audio frame level representation sequence of the query audio signal). Each element in the similarity matrix represents the similarity or matching score of the two modal features at a specific constituent unit.

[0160] In an optional embodiment, the similarity or matching score can be obtained by calculating the cosine similarity (Cosine Similarity) between the two feature vectors.

[0161] By determining the similarity matrix of each modal pair, the system can obtain a quantitative representation representing the matching degree of the query modal feature and the registration modal feature at different time points.

[0162] Considering that the obtained similarity matrix only reflects the isolated and instantaneous matching at each time point, lacking the modeling ability of context information, the application continues to input the similarity matrix into a recurrent neural network to obtain the hidden state vector corresponding to each modal pair output by the recurrent neural network.

[0163] To solve the problem of lack of temporal context of the similarity matrix, the embodiment introduces a recurrent neural network for temporal modeling. The recurrent neural network is a network structure good at processing sequence data, which can capture and remember long-range dependencies in sequence data through its internal recurrent connection.

[0164] In a preferred embodiment, the recurrent neural network can be a Gated Recurrent Unit (GRU) network or a Long Short-Term Memory (LSTM) network, etc.

[0165] The system will input the similarity matrix generated in the previous step into the recurrent neural network according to the time dimension of the query modal signal (i.e. the column of the matrix) as a sequence. The recurrent neural network will process the similarity information at each time step in turn, and combine the hidden state stored at the previous time to update the hidden state at the current time. When the entire sequence is processed, the final hidden state output by the recurrent neural network is taken as the hidden state vector corresponding to the modal pair. The hidden state vector is no longer a collection of instantaneous matching scores, but a deep feature representation containing the entire sequence matching pattern, sequential dependency and context information.

[0166] The keyword recognition method provided by the application explicitly defines the process of obtaining the hidden state vector as first calculating the instantaneous similarity matrix, and then performing temporal modeling through the recurrent neural network, effectively overcoming the limitation of lacking context information when only based on similarity scores. Especially the introduction of the recurrent neural network can learn and understand the evolution rule and dependency of the similarity score in the time dimension, so that the finally generated hidden state vector can more accurately represent the matching degree of a keyword as a complete temporal pattern, rather than a coincidental similarity of scattered fragments, significantly improving the reliability and accuracy of the subsequent aggregation and decision steps.

[0167] As an optional embodiment, the application provides a specific and efficient implementation of weighting all the obtained hidden state vectors to generate an aggregated feature vector.

[0168] Specifically, after completing the independent analysis of all modal pairs in the available modal pair set and obtaining the hidden state vector corresponding to each modal pair, the system will perform a concatenation operation, that is, by combining multiple independent hidden state vectors in a predefined order, connecting them end to end into a longer, higher-dimensional vector.

[0169] For example, suppose in a keyword recognition task, the system determines that there are three available modality pairs in the set of available modality pairs and generates three hidden state vectors accordingly, namely vector A, vector B, and vector C. If each vector has a dimension of 128, the system will concatenate these three vectors to generate a final aggregated feature vector. The dimension of this aggregated feature vector will be 128 + 128 + 128 = 384, and its internal structure can be represented as [the content of vector A, the content of vector B, and the content of vector C].

[0170] It should be noted that since the number of mode pairs in the available mode pair set changes dynamically based on the actual input, the dimension of the aggregated feature vector generated by concatenation is also variable. For example, when there is only one mode pair in the available mode pair set, the aggregated feature vector is the hidden state vector of that mode pair itself; when there are six mode pairs, the dimension of the aggregated feature vector will be six times the dimension of a single hidden state vector. This dynamically changing structure is one of the core mechanisms for achieving flexible mode fusion in this invention, and the subsequent decision module (such as the classifier network) will be designed to handle this variable-dimensional input, or to unify it to a fixed dimension through methods such as zero padding.

[0171] The keyword recognition method provided by this invention explicitly defines the aggregation operation as concatenation that completely preserves all the information contained in the analysis results of each independent modality pair (i.e., each hidden state vector), and aggregates this information from different analysis dimensions into a unified feature representation. This approach provides the most comprehensive and original decision-making basis for the subsequent classifier network, enabling the classifier to comprehensively learn and discriminate complex association patterns from all available modality pairs in a higher-dimensional feature space, thereby improving the accuracy of the final recognition result.

[0172] As an optional embodiment, in the keyword recognition method provided by the present invention, the final keyword recognition decision process is completed by a pre-trained classifier network.

[0173] The classifier network is a neural network model capable of performing a classification task. In an optional embodiment, the classifier network can be a Multi-Layer Perceptron (MLP) composed of one or more fully connected layers.

[0174] Specifically, after obtaining the aggregated feature vector, the system performs the following steps:

[0175] First, the aggregated feature vector is input into a classifier network to obtain the recognition result output by the classifier network.

[0176] The aggregated feature vector, as a high-dimensional feature representation containing analysis information of all modal pairs, is sent to the classifier network. The classifier network makes a final decision on the high-dimensional feature through a series of nonlinear transformations inside the classifier network.

[0177] In a preferred embodiment, the output layer of the classifier network usually contains an activation function (such as a Sigmoid function), so that it finally outputs a value between 0 and 1. This value is taken as a specific form of the recognition result, for example, it can be a confidence score representing the probability or confidence level of the query input containing the preset keyword.

[0178] After obtaining the confidence score, the system can further obtain a binary final recognition result (yes / no) by comparing the score with a preset decision threshold. For example, if the confidence score is greater than or equal to the decision threshold, it is determined as a "hit", i.e., the keyword is recognized; otherwise, it is determined as a "miss".

[0179] The classifier network is trained based on a pre-constructed full modal pair set, which includes full modal pair samples obtained by combining all registered modal features under registered modalities and all query modal features under query modalities two by two, and each full modal pair sample is labeled with a recognition result label.

[0180] The embodiment further illustrates the training manner of the classifier network. In order to enable the classifier network to correctly process the aggregated feature vectors of variable dimensions generated by different modal pair combinations, the training process is based on a specially constructed full modal pair set which aims to cover all theoretically possible modal pair combinations. For example, if the system supports three registration modalities of text, audio and video and two query modalities of audio and video, the full modal pair set will contain all training samples under 6 combinations. In the training process, the system will provide the classifier network with the aggregated feature vectors generated by different numbers and types of modal pairs (from 1 to 6), and use the corresponding recognition result labels (for example, labeled as 1 for "key word" and 0 for "non-key word") as a supervision signal to optimize the classifier network.

[0181] The keyword recognition method provided by the application explicitly realizes the final recognition decision through a specially trained classifier network, effectively utilizes the powerful nonlinear classification ability of the deep neural network, and can learn a complex decision boundary from a high-dimensional aggregated feature vector.

[0182] In subsequent embodiments, it will be introduced that the application will also ensure that the classifier network can stably process the dimension change of the aggregated feature vector caused by the actual input modal missing by training on a set covering all modal combinations, so that the entire keyword recognition method has the ability to make reliable decisions under any modal combination, greatly enhancing the generalization and practicality of the system.

[0183] To solve the problem that there is still a lack of a keyword recognition system that can adaptively cope with complex acoustic environments (such as strong noise interference) and has multi-modal compatibility in the current technology, the embodiment of the application provides a new self-defined keyword recognition system, which aims to realize a stable and accurate keyword recognition system that can flexibly switch or fuse available modal data under the condition of complete or missing different modal information. The keyword recognition system not only has strong robustness to noise environment, but also can dynamically integrate audio, text and video information according to the actual input condition, so as to ensure user experience while significantly improving adaptability and recognition performance in diversified practical application scenarios.

[0184] Figure 4 is the architecture schematic diagram of the keyword recognition system provided by the application, and the specific process of a keyword recognition method provided by the embodiment of the application will be described in detail below with reference to the accompanying drawings. Figure 4 The embodiment ingeniously fuses keyword registration modules of different modalities in one framework and can flexibly cope with the case of modal missing.

[0185] Specifically, the keyword recognition method of the embodiment can be applied to a keyword recognition system including a text encoder, an audio encoder, a video encoder, a multi-modal noise reduction module, a multi-modal alignment module, and a decision maker, and can support one or more of text, audio, and video as registered modalities of keyword registration information, and also support one or more of audio and video as query modalities for keyword information recognition.

[0186] First, data preparation and feature extraction are performed.

[0187] The system can receive registered text information, registered audio signals, or registered video signals as registered inputs, and perform feature extraction on these registered inputs through corresponding text encoders, audio encoders, or video encoders to obtain corresponding registered modality features.

[0188] For registered text information, a word segmentation operation can be performed using natural language processing techniques to convert the text into phoneme representation, and the classification layer weights of a pre-trained model are used as high-dimensional vector representation of the corresponding phonemes to provide text features that are distinguishable at the phoneme level.

[0189] For registered audio signals, they can be zero-padded to a uniform length and input into a pre-trained speech representation model (such as the Whisper-tiny model) to obtain high-dimensional semantic embedding vectors that can effectively capture speech content, pronunciation patterns, and speaker characteristics from the output layer of the speech representation model.

[0190] For registered video signals, a pre-trained video feature extractor containing a residual connection network and a three-dimensional convolutional neural network (such as ResNet+3D CNN) structure can be selected to extract features from the lip and surrounding facial regions in the video frames, focusing on capturing the user's lip movement information.

[0191] At the same time, the system receives real-time query audio signals and / or query video signals as query inputs, and obtains query modality features through corresponding audio encoders and video encoders.

[0192] Further, the system also performs an optional multi-modal noise reduction process aimed at noise reduction and other auxiliary operations on the query audio and registered video signals to achieve effective differentiation and noise suppression of the target speaker and the interfering speaker in the speech signal.

[0193] Specifically, the system performs frame-level segmentation on the query audio signal to obtain a sequence of query audio frame-level representations. Subsequently, a noise reduction mask is determined. The determination of the noise reduction mask is based on the correlation calculation between each query audio frame-level representation in the sequence of query audio frame-level representations and the query video signal (which contains lip movement information that is strictly aligned in time with the query audio) and all available enrollment modalities (enrollment text information, enrollment audio signal and enrollment video signal). The correlation calculation is achieved through a cross-attention mechanism to create multiple associated attention output sequences, and the corresponding modality masks are determined according to these sequences, respectively.

[0194] Among them, compared with the enrollment text information, the enrollment audio signal and the enrollment video signal, the query video signal performs particularly well in assisting in distinguishing the target speaker from the interfering speaker.

[0195] Finally, all modality masks are weighted and fused to generate a final noise reduction mask, and the query audio signal is denoised using the noise reduction mask.

[0196] The denoised query audio signal will be re-encoded to obtain cleaner audio features. The number of iterations of this denoising process can be determined according to actual use requirements. When the real-time requirement is not high, multiple iterations of denoising can be performed to obtain better results. If the real-time requirement is high, one denoising can achieve effect improvement while not introducing too much time delay.

[0197] Finally, the core keyword recognition process is performed.

[0198] (1) The system determines a set of available modality pairs based on all currently available query modality features, which may have been denoised, and all preset enrollment modality features.

[0199] (2) For each modality pair in the set of available modality pairs, the system independently calculates a similarity matrix between its enrollment modality features and query modality features, and inputs the similarity matrix into a recurrent neural network to obtain a hidden state vector that can represent the timing matching pattern of the modality pair.

[0200] (3) All available hidden state vectors obtained are spliced to generate an aggregated feature vector. The composition of the aggregated feature vector is dynamic, and its composition varies from one to six, ensuring the generalization ability to any modality missing scenario.

[0201] (4) The obtained polymeric feature vector is input into a classifier network. The classifier network outputs a confidence score between 0 and 1. The system compares the score with a preset decision threshold. If the score is greater than or equal to the decision threshold, the recognition result is determined as a "hit" (a self-defined keyword); otherwise, a "miss" result is returned.

[0202] The embodiment of the present application realizes a multi-modal fusion, self-adaptive robust, and spatial alignment fine self-defined keyword recognition method, so that the system can still maintain high-precision recognition ability when the modal information is incomplete, and the generalization and practicability of the system are enhanced, which has important significance for promoting the practicability and popularity of the intelligent voice interaction system.

[0203] Figure 5 is a training process schematic diagram of the keyword recognition model provided by the present application, as shown in Figure 5 The embodiment provides a method for designing training of a to-be-trained keyword recognition model. The to-be-trained keyword recognition model at least includes the feature encoder, the mask generation network, the recurrent neural network, and the classifier network mentioned in the above embodiment. Through the joint optimization training process, the keyword recognition model after training is simultaneously improved in the three key abilities of noise reduction, alignment, and decision. The specific implementation of the training method can include:

[0204] First, a training sample set is collected.

[0205] In order to ensure that the to-be-trained keyword recognition model can learn rich multi-modal correlation information, the embodiment will pre-construct a high-quality training sample set. Specifically, a recording and video recording device facing the speaker can be used to record strictly aligned audio signal samples and video signal samples in various home scenes (such as living rooms and bedrooms), and record the corresponding speech content as text information samples. From these text information samples, select keywords suitable for human-computer interaction (for example, nouns), and cut out the corresponding audio and video clips from the audio signal samples and video signal samples.

[0206] In order to be close to the actual use scene, the audio signal samples can be simulated at different signal-to-noise ratio levels (such as 0 decibels, 3 decibels, 6 decibels, and 9 decibels). The noise data can use various home noises (such as television background sound, electric fan steady-state noise, and music sound). Each group of training samples includes these registered input samples (text information samples, audio signal samples, and video signal samples) and query input samples (noisy audio and video), and pre-annotates the "true value" label for supervised learning, that is, the recognition result label (for example, 1 represents "keyword", and 0 represents "no") and the clean audio signal label (that is, the original audio without noise).

[0207] Secondly, the training sample set is used to iteratively train a keyword recognition model to be trained. The entire training process is an iterative optimization process until a second preset cutoff condition is met. This second preset cutoff condition may be reaching a preset number of training epochs, or the performance indicators (such as accuracy) of the trained model on the validation set no longer improving.

[0208] In each training iteration, the system randomly selects a batch of training samples from the training sample set, and performs the following core training steps for each group of training samples within that batch:

[0209] First, a set of training samples is used as input to the keyword recognition model to be trained, in order to obtain the various outputs of the model. After receiving registration input and query input, the keyword recognition model performs forward computation through its internal modules, generating intermediate or final results for calculating different loss functions, specifically including:

[0210] The final predicted recognition result (a confidence score) is output by the decision module (which includes a recurrent neural network and a classifier network).

[0211] The noise-reduced speech training signal obtained after processing by the multimodal noise reduction module.

[0212] The frame-level audio and video fusion vector sequence obtained after processing by the feature encoder and multimodal alignment module.

[0213] Second, calculate three different losses, including:

[0214] (1) Compare the predicted recognition result with the recognition result label pre-annotated in the training sample to calculate the decision loss, which is used to evaluate the accuracy of the model's final decision.

[0215] (2) The noise reduction training signal is compared with the clean audio signal label pre-labeled in the training sample to calculate the noise reduction loss, which is used to evaluate the ability of the noise reduction module to recover clean speech.

[0216] (3) The frame-level audio-video fusion vector sequence is mapped to the feature space of the text phoneme features through a learnable spatial mapping network. Then, a Gaussian-like distribution weighting process based on the center of the time window is performed on the mapped sequence to aggregate it into a phoneme-level fusion feature representation. Finally, this phoneme-level fusion feature representation is compared with the text phoneme features extracted from the text information of the training samples to calculate the spatial alignment loss. This loss is used to optimize the semantic consistency of features of different modalities.

[0217] Third, update the model parameters based on the calculated three types of losses.

[0218] The system combines the computed decision loss, noise reduction loss and spatial alignment loss (e.g., by weighted summation) to form a total joint loss. Then, the gradient of all learnable parameters in the model is computed according to the joint loss by a backpropagation algorithm, and the parameters are adjusted and updated slightly using the gradient. The updated parameters are at least from the model components of the feature encoder, the mask generation network, the recurrent neural network and the classifier network.

[0219] The keyword recognition method provided by the embodiment can simultaneously optimize the noise reduction, alignment and decision of the system in an unified framework through a combined loss function, and the collaborative optimization ensures that the modules in the keyword recognition model can cooperate with each other to serve the final keyword recognition goal, thereby constructing a powerful model with high performance under various complex conditions.

[0220] To improve the fusion effect of multi-modal information in the unified semantic space, the application introduces a multi-modal spatial alignment mechanism. The core of the multi-modal spatial alignment mechanism is to effectively convert the continuous and frame-level audio-video fusion features into discrete phoneme-level features corresponding to the text phonemes. The specific implementation manner can include:

[0221] First, the frame-level audio-video fusion vector sequence is mapped to the feature space where the text phoneme features are located in the training sample through a learnable spatial mapping network.

[0222] The frame-level audio-video fusion vector sequence is a time-sequenced representation obtained by fusing audio and video features. To solve the problem that the original spaces of different modal features are inconsistent, the embodiment inputs the frame-level audio-video fusion vector sequence into a spatial mapping network. The spatial mapping network can be composed of one or more fully connected layers, and the parameters of the spatial mapping network are learnable. The audio-video fusion features inputted through the spatial mapping network are linearly or nonlinearly transformed from the original feature space to a new latent space. The latent space is designed to have the same dimension and similar distribution characteristics as the feature space where the text phoneme features are located, thereby laying a foundation for subsequent direct comparison and alignment.

[0223] Second, the mapped sequence is weighted processed based on the time window center Gaussian distribution to generate phoneme-level fusion feature representation.

[0224] To realize the conversion from the frame level to the phoneme level and avoid the error propagation problem that may be introduced by traditional frame alignment, the processing process of the embodiment can be further decomposed as follows:

[0225] For each phoneme contained in the keyword text in the training sample, a frame sequence interval corresponding to the phoneme on the time axis, i.e., a time window, is determined in advance by means of a technical means such as forced alignment, and a center frame position of the time window is found.

[0226] A weight of a Gaussian distribution type is constructed with the center frame of the time window as a reference point, that is, the center frame is given the highest weight, and the weight of a frame farther away from the center frame is reduced according to the decay law of a Gaussian function.

[0227] Each spatially mapped frame-level vector in the time window is multiplied by its corresponding weight, and then all the weighted vectors are summed or averaged. Through this weighted aggregation process, multiple frame-level vectors in a time window are collapsed or aggregated into a single fusion feature representation that can represent the overall features of the phoneme.

[0228] Finally, after obtaining the phoneme-level fusion feature representation corresponding to each phoneme in the keyword, the spatial alignment loss between the fusion feature representation and the text phoneme feature can be calculated. For example, the spatial alignment loss can be obtained by calculating the mean squared error (MSE) between the two sets of phoneme-level vector sequences.

[0229] The method for calculating the spatial alignment loss provided in the present application explicitly combines spatial mapping and Gaussian-like weighted aggregation in the conversion process from frame-level features to phoneme-level features. By raising the granularity of alignment from unstable frame level to phoneme level with more semantic stability, and using a weighting strategy to highlight the core features of each phoneme, the present application realizes more fine and robust cross-modal modeling of the pronunciation units of keywords, effectively improving the consistency and discriminability of different modalities in a unified space.

[0230] To ensure that the keyword recognition method provided in the present application can still maintain stable and accurate performance in the real world under the scene of missing or incomplete modalities, a training strategy simulating the missing of modalities is introduced in the training phase of the model.

[0231] Specifically, after a batch of training samples is randomly selected from a complete training sample set, but before the batch of samples is sent to the model to be trained for calculation, the system performs a random removal operation.

[0232] The random removal operation refers to temporarily removing data related to one or more modalities from the batch of training samples with a certain preset probability. This operation does not physically delete the data, but simulates the situation where the modality information is not available in the current training iteration.

[0233] For example, for a training sample originally containing text, audio, and video three registration modal input, the system can randomly ignore the text registration input with a certain probability, and only use the audio registration input and the video registration input for subsequent model training.

[0234] For example, for a training sample originally containing audio and video two query modal input, the system can randomly ignore the video query input with a certain probability, and only use the audio query input for subsequent model training.

[0235] The system can also randomly eliminate registration modalities and query modalities at the same time to simulate more complex modal missing scenarios. For example, only provide text registration input and audio query input, and ignore all other modalities.

[0236] By randomly and dynamically eliminating modalities in each batch of training, the keyword recognition model to be trained will be forced to learn how to make optimal decisions under various incomplete information conditions. For example, when the video modality is eliminated, the model must learn to rely more on audio and text information; when the text registration modality is eliminated, the model learns to extract key features only from audio or video registration information.

[0237] The training strategy introduced in this embodiment to simulate modal missing forces the model to learn the complementarity and redundancy between different modal information, which can greatly enrich the diversity of training data, so that the keyword recognition model to be trained will not rely too much on any single modality or fixed modality combination during optimization. The generalization ability and robustness of the final keyword recognition method in real applications are significantly enhanced, so it can calmly cope with various modal missing situations caused by user usage habits, device status or environmental restrictions, etc.

[0238] To enhance the interaction and synergy between the audio modality and the video modality, when generating the frame-level audio-video fusion vector sequence used for subsequent spatial alignment, the present application further introduces a random replacement mechanism. Instead of directly fusing the original audio and video features, the random replacement mechanism first constructs a more robust intermediate representation through a feature exchange operation. Its implementation can include the following steps:

[0239] First, the audio feature sequence corresponding to the audio signal sample and the video feature sequence corresponding to the video signal sample in the training sample are exchanged at least one time frame with a preset probability.

[0240] After obtaining the audio feature sequence (e.g., composed of a series of audio feature frame vectors) time-aligned with the audio signal sample and the video feature sequence (e.g., composed of a series of video feature frame vectors) time-aligned with the video signal sample, the system traverses the time frames of the two sequences. At each time frame, the system randomly decides whether to perform a feature exchange operation according to a preset probability (e.g., 15%).

[0241] Feature exchange refers to replacing the audio feature vector of a time frame with the video feature vector time-aligned therewith and / or replacing the video feature vector of a time frame with the audio feature vector time-aligned therewith at the time frame. The feature exchange operation is independently performed at different time frames.

[0242] Secondly, the audio signal sample and the video signal sample after the feature exchange are fused to obtain the frame-level audio-video fusion vector sequence.

[0243] After the feature exchange operation described above is completed, the system obtains two new feature sequences in which some features are exchanged. Then, the system fuses the two new feature sequences.

[0244] In an optional embodiment, the fusion operation is concatenation, i.e., the audio feature vector and the video feature vector (which have or have not been exchanged) at each time frame are concatenated to form a new vector with a higher dimension. The new vectors generated at all time frames are arranged in sequence to form the final frame-level audio-video fusion vector sequence.

[0245] The present application introduces a random feature exchange mechanism before audio-video feature fusion, introduces a structured noise and uncertainty to the to-be-trained keyword recognition model, and the effect is similar to an effective regularization method, which forces the audio and video feature encoders to learn to generate more shared and complementary feature representations containing the information of the other modalities. At the same time, it also forces the downstream modules of the to-be-trained keyword recognition model to learn to cooperatively process information from different channels instead of relying on fixed input structures, greatly enhancing the robustness of the final learned audio-video joint representation and improving the generalization ability of the model when the quality of a certain modality information is poor or partially missing, thereby laying a solid foundation for subsequent implementation of higher-precision spatial alignment.

[0246] As an optional embodiment, the updating of the parameters of the to-be-trained keyword recognition model based on the decision loss, the noise reduction loss, and the spatial alignment loss specifically includes but is not limited to the following implementation steps:

[0247] update parameters of the mask generation network based on the decision loss, the spatial alignment loss and the noise reduction loss;

[0248] update parameters of the feature encoder based on the decision loss and the spatial alignment loss;

[0249] update parameters of the classifier network and the recurrent neural network based on the decision loss.

[0250] In the training process of the keyword recognition model to be trained, the updating process of the model parameters is not simply applying the gradients of all losses to all model component modules (mainly including the feature encoder, the mask generation network, the recurrent neural network and the classifier network, etc.) without distinction, but a more refined asymmetric updating mechanism that can reflect the internal logical relationship between different loss functions and different modules is adopted. The principle of the asymmetric updating mechanism is derived from the chain rule of loss backpropagation in neural networks, that is, the updating of the parameters of a module will be affected by the gradients flowing through it. The steps of the parameter updating can be:

[0251] First, update parameters of the classifier network and the recurrent neural network based on the decision loss loss3.

[0252] The decision loss loss3 is calculated directly based on the final output of the classifier network and the recurrent neural network as the core of the decision maker. Therefore, in the backpropagation process, the gradients generated by loss3 will be directly used to update the parameters of these two modules to optimize the final decision-making ability of the model.

[0253] In a preferred embodiment, the decision loss loss3 can be calculated using the binary cross-entropy loss function. Based on the recognition result label y (where y 1 represents a keyword and 0 represents a non-keyword) and the confidence score of the predicted recognition result output by the model (a value between 0 and 1), the decision loss loss3 can be calculated by the following formula:

[0254] ].

[0255] Second, update parameters of the feature encoder based on the decision loss loss3 and the spatial alignment loss loss2.

[0256] The spatial alignment loss loss2 is calculated based on the comparison between the fused feature representation output by the feature encoder (after processing by the alignment module) and the text phoneme features.

[0257] In a preferred embodiment, the spatial alignment loss loss2 can be calculated using a mean squared error loss function (MSE Loss) to supervise the alignment between the fused audio-visual representation and the phoneme-level text phoneme features, which can be obtained by the following formula:

[0258] ;

[0259] wherein, is the t-th frame of the fused audio-visual representation, is the text phoneme feature of the n-th phoneme, M represents the spatial mapping network, w represents the weight of the Gaussian-like weighting strategy, and N represents the total number of phonemes.

[0260] The gradient of loss2 will directly act on the feature encoder, but since the output of the feature encoder will continue to propagate forward and ultimately affect the calculation of the decision loss loss3, the gradient of the decision loss loss3 will also be backpropagated to the feature encoder. Therefore, the parameter update of the feature encoder is affected by the gradients of both the spatial alignment loss loss2 and the decision loss loss3.

[0261] Again, the parameters of the mask generation network are updated by combining the decision loss loss3, the spatial alignment loss loss2 and the noise reduction loss loss1.

[0262] The noise reduction loss loss1 is calculated directly according to the output of the mask generation network, which is the core of the noise reduction module.

[0263] In a preferred embodiment, the noise reduction loss can be calculated using a scale-invariant signal-to-noise ratio improvement loss (SISNR). In the time domain, the noise reduction speech training signal is compared with the clean audio signal label of the training sample, and the noise reduction loss loss1 can be obtained by the following formula:

[0264] ;

[0265] wherein s is the projection component of the noise reduction speech training signal in the direction of the clean audio signal label, and e is the noise component.

[0266] The gradient of the noise reduction loss loss1 directly acts on the mask generation network, and the output of the noise reduction module is used as the input of all subsequent modules, so the gradients of the spatial alignment loss loss2 and the decision loss loss3 are also ultimately back propagated to the mask generation network. Therefore, the parameter update of the mask generation network is affected by the gradients of all three losses, namely the noise reduction loss loss1, the spatial alignment loss loss2 and the decision loss loss3.

[0267] The application decomposes the parameter update process into asymmetric updates of different modules in the keyword recognition model, accurately reflects the differentiated optimization of different loss functions on different modules in the joint training framework, ensures that each module can be effectively optimized according to the loss directly related to it, and also enables the upstream module to simultaneously perceive and respond to the performance of the downstream task through the cascade propagation of the gradient, thereby realizing the end-to-end and truly collaborative optimization of the entire model, and finally constructing a keyword recognition model with highly coordinated functions and strong overall performance.

[0268] Figure 6 is a structural schematic diagram of the keyword recognition device provided by the application, as shown in Figure 6 The keyword recognition device provided by the application mainly includes but is not limited to the following components:

[0269] The feature extraction unit 11 is configured to determine a set of available modal pairs based on the registration modal features obtained under at least one registration modality and the query modal features obtained under at least one query modality; the modal pair in the set of available modal pairs is a combination of any query modal feature and any registration modal feature;

[0270] The feature analysis unit 12 is configured to obtain a hidden state vector corresponding to each modal pair in the set of available modal pairs;

[0271] The feature aggregation unit 13 is configured to aggregate all the hidden state vectors to generate an aggregated feature vector;

[0272] The result recognition unit 14 is configured to determine a recognition result based on the aggregated feature vector; the recognition result is used to represent whether the pre-set keyword is contained in the query input corresponding to any query modal feature.

[0273] It should be noted that the keyword recognition device provided by the application can execute the keyword recognition method described in any of the above embodiments when it is actually operated, and this embodiment will not be described here.

[0274] The keyword recognition device provided by the application solves the problem of sharp performance degradation or failure in the modal missing scene, realizes flexible compatibility and adaptive fusion of different modal inputs, and significantly enhances the generalization ability, robustness and practicality of the keyword recognition system in complex and variable real application scenes.

[0275] Figure 7 is a structural schematic diagram of an electronic device provided by the application, as Figure 7 shown, the electronic device can include a processor (processor) 710, a communication interface (communications interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, the memory 730 complete mutual communication through the communication bus 740. The processor 710 can call the logic instruction in the memory 730 to execute the keyword recognition method, which includes: determining a set of available modal pairs based on the registration modal features obtained under at least one preset registration modal and the query modal features obtained under at least one query modal; the modal pair in the set of available modal pairs is a combination of any query modal feature and any registration modal feature; obtaining the hidden state vector corresponding to each modal pair in the set of available modal pairs; aggregating all the hidden state vectors to generate an aggregated feature vector; determining the recognition result based on the aggregated feature vector; the recognition result is used to represent whether the preset keyword is contained in the recognition result of the query input corresponding to any query modal feature.

[0276] In addition, the logic instruction in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product stored in a storage medium includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and various program code storage media.

[0277] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the keyword recognition method provided by any of the above embodiments, the method comprising: determining a set of available modal pairs based on preset registration modal features obtained in at least one registration modality and query modal features obtained in at least one query modality; the modal pair in the set of available modal pairs is a combination of any of the query modal features and any of the registration modal features; obtaining a hidden state vector corresponding to each modal pair in the set of available modal pairs; aggregating all the hidden state vectors to generate an aggregated feature vector; determining a recognition result based on the aggregated feature vector; and the recognition result is used to represent whether a preset keyword is contained in a query input corresponding to any query modal feature.

[0278] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a keyword recognition method provided by any of the above embodiments, the method comprising: determining a set of available modal pairs based on preset registration modal features obtained in at least one registration modality and query modal features obtained in at least one query modality; the modal pair in the set of available modal pairs is a combination of any of the query modal features and any of the registration modal features; obtaining a hidden state vector corresponding to each modal pair in the set of available modal pairs; aggregating all the hidden state vectors to generate an aggregated feature vector; determining a recognition result based on the aggregated feature vector; and the recognition result is used to represent whether a preset keyword is contained in a query input corresponding to any query modal feature.

[0279] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0280] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0281] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A keyword spotting method, characterized by, The method comprises the following steps: Based on the preset at least one registration modal feature obtained under the registration modal, and at least one query modal feature obtained under the query modal, a set of available modal pairs is determined; the modal pair in the set of available modal pairs is a combination of any query modal feature and any registration modal feature; Obtain the hidden state vector corresponding to each modal pair in the set of available modal pairs; Aggregate all the hidden state vectors to generate an aggregated feature vector; Based on the aggregated feature vector, determine the recognition result; the recognition result is used to represent whether the query input corresponding to any query modal feature contains a preset keyword; The keyword recognition method further comprises: Collect a training sample set, which comprises multiple groups of training samples, each group of training samples comprising at least one registration input sample under a registration modal, at least one query input sample under a query modal, and a pre-labeled recognition result label and a clean audio signal label; Iteratively perform the training process of a to-be-trained keyword recognition model using the training sample set until a second preset stopping condition is met; In each training process, a batch of training samples is randomly selected from the training sample set, and the following training steps are performed until all training samples in the batch are traversed: A group of training samples selected from the batch is used as the input of the to-be-trained keyword recognition model, and a predicted recognition result output by the to-be-trained keyword recognition model and a noise reduction speech training signal and a frame-level audio-video fusion vector sequence generated during the process are obtained; Calculate the decision loss between the predicted recognition result and the recognition result label of the training sample; Calculate the noise reduction loss between the noise reduction speech training signal and the clean audio signal label of the training sample; After mapping the frame-level audio-video fusion vector sequence to the feature space where the text phoneme feature in the training sample is located, perform weighted processing based on the time window center of the Gaussian distribution to generate a phoneme-level fusion feature representation, and calculate the spatial alignment loss between the fusion feature representation and the text phoneme feature; Based on the decision loss, the noise reduction loss and the spatial alignment loss, update the parameters of the to-be-trained keyword recognition model; The to-be-trained keyword recognition model comprises at least a feature encoder, a mask generation network, a recurrent neural network and a classifier network.

2. The keyword spotting method of claim 1, wherein, The registration modal comprises at least one of text registration, audio registration and video registration, and the query modal comprises at least one of voice query and video query; When the query modal comprises both voice query and video query, the voice query and the video query occur synchronously.

3. The keyword spotting method of claim 2, wherein, In the case where it is determined that the query modal comprises voice query, before determining the set of available modal pairs, the following steps are iteratively performed until a first preset stopping condition is met: Frame-level segmentation is performed on the query voice signal in the query input to obtain a query audio frame-level representation sequence; determining a noise reduction mask, the noise reduction mask being determined based on a result of a correlation calculation between each query audio frame-level representation in the sequence of query audio frame-level representations and the query video signal of the video query and / or at least one registered modality signal in the registered modalities; de-noising the query speech signal using the noise reduction mask.

4. The keyword spotting method of claim 3, wherein, The determining a noise reduction mask comprises: calculating a cross-attention calculation result between each query audio frame-level representation in the sequence of query audio frame-level representations and the query video signal to create a video-associated attention output sequence; calculating a cross-attention calculation result between each query audio frame-level representation in the sequence of query audio frame-level representations and each registered modality signal to create a registered-associated attention output sequence corresponding to each registered modality signal; determining a corresponding modality mask according to the video-associated attention output sequence and / or each registered-associated attention output sequence respectively; weighting and fusing all modality masks to generate the noise reduction mask.

5. The keyword spotting method of claim 4, wherein, The determining a corresponding modality mask according to the video-associated attention output sequence and / or each registered-associated attention output sequence respectively comprises: inputting the video-associated attention output sequence and / or the registered-associated attention output sequence into a mask generation network to obtain the modality mask output by the mask generation network; The mask generation network comprises a convolutional layer and a nonlinear activation function layer, the convolutional layer is used for feature extraction of the video-associated attention output sequence or the registered-associated attention output sequence to generate a feature matrix, and the nonlinear activation function layer is used for element-wise probability operation of the feature matrix to generate the modality mask.

6. The keyword spotting method of claim 1, wherein, The obtaining the hidden state vector corresponding to each modality pair in the set of available modality pairs comprises: determining a similarity matrix between the query modality feature and the registered modality feature in each modality pair, the similarity matrix representing a matching degree of the query modality feature and the registered modality feature at different time points; inputting the similarity matrix into a recurrent neural network to obtain the hidden state vector corresponding to each modality pair output by the recurrent neural network.

7. The keyword spotting method of claim 1, wherein, The aggregating all hidden state vectors to generate an aggregated feature vector comprises: splicing the hidden state vectors corresponding to all modality pairs in the set of available modality pairs to generate the aggregated feature vector.

8. The keyword spotting method according to any one of claims 1 to 7, characterized in that, The determining a recognition result based on the aggregated feature vector comprises: inputting the aggregated feature vector into a classifier network to obtain the recognition result output by the classifier network.

9. The keyword spotting method of claim 8, wherein, After randomly selecting a batch of training samples from the training sample set in each training process, further comprising: randomly removing training samples related to at least one registered modality and / or at least one query modality in the batch.

10. The keyword spotting method of claim 8, wherein, The frame-level audio-video fusion vector sequence is determined based on the following manner: interchange the audio feature sequence corresponding to the audio signal sample and the video feature sequence corresponding to the video signal sample in the training sample with each other at at least one time frame with a preset probability; fuse the audio signal sample and the video signal sample after the feature interchange to obtain the frame-level audio-video fusion vector sequence.

11. The keyword spotting method of claim 8, wherein, The updating of the parameters of the to-be-trained keyword recognition model based on the decision loss, the noise reduction loss, and the spatial alignment loss comprises: The updating of the parameters of the mask generation network based on the decision loss, the noise reduction loss, and the spatial alignment loss comprises: The updating of the parameters of the feature encoder based on the decision loss and the spatial alignment loss comprises: The updating of the parameters of the classifier network and the recurrent neural network based on the decision loss comprises:

12. The keyword spotting method of claim 8, wherein, The updating of the parameters of the to-be-trained keyword recognition model based on the decision loss, the noise reduction loss, and the spatial alignment loss comprises: The combination loss is obtained by weighted summation of the decision loss, the noise reduction loss, and the spatial alignment loss; The parameters of the mask generation network of the to-be-trained keyword recognition model are updated based on the combination loss. The weight coefficients of the weighted summation are dynamically determined based on the change gradients of the decision loss, the noise reduction loss, and the spatial alignment loss in the historical training process.

13. The keyword spotting method of claim 12, wherein, The combination loss is obtained by weighted summation of the decision loss, the noise reduction loss, and the spatial alignment loss. The decision loss is a binary cross-entropy loss, the noise reduction loss is a scale-invariant signal-to-noise ratio improvement loss, and the spatial alignment loss is a mean square error loss.

14. The keyword spotting method of claim 8, wherein, The combination loss is obtained by weighted summation of the decision loss, the noise reduction loss, and the spatial alignment loss.

15. A keyword spotting apparatus characterized by comprising: The feature extraction unit is configured to determine a set of available modality pairs based on the preset registration modality features obtained under at least one registration modality and the query modality features obtained under at least one query modality, wherein the modality pairs in the set of available modality pairs are combinations of any of the query modality features and any of the registration modality features; The feature analysis unit is configured to obtain a hidden state vector corresponding to each modality pair in the set of available modality pairs; The feature aggregation unit is configured to aggregate all the hidden state vectors to generate an aggregated feature vector; The result recognition unit is configured to determine a recognition result based on the aggregated feature vector, wherein the recognition result is used to represent whether a preset keyword is contained in a query input corresponding to any query modality feature. The keyword recognition device further comprises a model training unit configured to perform the following operations: A training sample set is collected, wherein the training sample set comprises a plurality of groups of training samples, each group of training samples comprising at least one registration input sample under a registration modality, at least one query input sample under a query modality, and a pre-labeled recognition result label and a clean audio signal label; The training process of a to-be-trained keyword recognition model is iteratively performed using the training sample set until a second preset stop condition is met. In each training process, a batch of training samples is randomly selected from the training sample set, and the following training steps are performed until all training samples in the batch are traversed: ​ taking a set of training samples selected from the batch as input of the to-be-trained keyword recognition model, obtaining a predicted recognition result output by the to-be-trained keyword recognition model and a denoised speech training signal and a frame-level audio-video fusion vector sequence generated in the process; calculating a decision loss between the predicted recognition result and a recognition result label of the training sample; calculating a denoising loss between the denoised speech training signal and a clean audio signal label of the training sample; after mapping the frame-level audio-video fusion vector sequence to a feature space where text phoneme features in the training sample are located, performing weighted processing based on a time window center Gaussian distribution to generate phoneme-level fusion feature representation, so as to calculate a spatial alignment loss between the fusion feature representation and the text phoneme features; updating parameters of the to-be-trained keyword recognition model based on the decision loss, the denoising loss and the spatial alignment loss; the to-be-trained keyword recognition model at least includes a feature encoder, a mask generation network, a recurrent neural network and a classifier network.

16. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, the processor executes the computer program to implement the keyword recognition method in any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, the computer program is executed by the processor to implement the keyword recognition method in any one of claims 1 to 14.

18. A computer program product comprising a computer program, characterized in that, the computer program is executed by the processor to implement the keyword recognition method in any one of claims 1 to 14. the computer program is executed by the processor to implement the keyword recognition method in any one of claims 1 to 14.

Images