Keyword identification method and device, electronic equipment and storage medium

Abstract

The invention provides a keyword recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: determining an available modal pair set based on at least one registration modal feature and at least one query modal feature, modal pairs in the available modal pair set are combinations traversing any query modal feature and any registration modal feature; and acquiring a hidden state vector of each modal pair, and aggregating all the hidden state vectors to generate an aggregated feature vector for determining an identification result. According to the method, an identification framework capable of dynamically determining the modal pairs and aggregating the corresponding characteristics of the modal pairs according to the actually available registration and query modal combination is constructed, so that the problem that the performance is sharply degraded or invalid in a modal missing scene is solved, flexible compatibility and self-adaptive fusion of different modal inputs are realized, and the method and the system have good application prospects. And the generalization ability, robustness and practicability of the keyword recognition system in a complex and changeable real application scene are remarkably enhanced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a keyword recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the popularization of IoT and smart home, smart vehicle and other applications, Keyword Spotting (KWS), as a key entry point for human-computer interaction, is increasingly demanding adaptability in different scenarios and personalized user experience. This places higher demands on recognition technology that can flexibly handle multiple input formats and maintain high accuracy.

[0003] To meet the above requirements, existing custom keyword recognition systems can be mainly divided into audio registration and text registration systems, depending on the registration method. Some advanced solutions, to improve robustness in complex environments, have begun to introduce multimodal information, such as combining video signals synchronized with the query audio for joint decision-making. These multimodal systems typically construct a fixed processing pipeline in which query audio features, query video features, and features of a preset registration modality (such as text) are fused and processed, and finally input into a classifier to obtain the recognition result.

[0004] The aforementioned existing technical solutions typically assume that all modal information used for joint decision-making is complete and readily available. This processing architecture, built upon a fixed modality combination, has inherent limitations when facing common real-world application scenarios with missing or incomplete modalities. For example, when a user selects a modality not pre-defined by the fixed architecture for keyword registration, or when video modalities cannot be effectively acquired during a query due to camera obstruction or the user being out of sight, the entire recognition system's processing flow is interrupted or fails due to input information mismatch, severely limiting its generalization ability and practical usability under diverse and uncertain interaction conditions. Summary of the Invention

[0005] This invention provides a keyword recognition method, device, electronic device, and storage medium to solve the problem that in existing multimodal systems, the performance of the system degrades sharply or fails in scenarios where modalities are missing because all modalities are pre-set to be fully available. This invention achieves flexible compatibility and adaptive fusion of different modal inputs.

[0006] This invention provides a keyword recognition method, comprising the following steps: Based on the registration modality features obtained under at least one pre-defined registration modality and the query modality features obtained under at least one query modality, a set of available modality pairs is determined; the modality pairs in the set of available modality pairs are combinations of any of the query modality features and any of the registration modality features. Obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs; Aggregate all the hidden state vectors to generate an aggregated feature vector; Based on the aggregated feature vector, the recognition result is determined; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of preset keywords.

[0007] According to a keyword recognition method provided by the present invention, the registration modality includes at least one of text registration, audio registration, and video registration, and the query modality includes at least one of voice query and video query; When the query modality includes both voice query and video query, the voice query and the video query occur simultaneously.

[0008] According to a keyword recognition method provided by the present invention, when it is determined that the query modality includes a voice query, before determining the set of available modality pairs, the method further includes iteratively executing the following steps until a first preset cutoff condition is met: The query voice signal in the query input is segmented at the frame level to obtain the query audio frame-level representation sequence; A noise reduction mask is determined based on the correlation calculation between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal of the video query and / or at least one registered modality signal in the registered modalities; The noise reduction mask is used to reduce the noise in the query speech signal.

[0009] According to a keyword recognition method provided by the present invention, determining a noise reduction mask includes: The cross-attention calculation results between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal are calculated respectively to create a video association attention output sequence; The cross-attention calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and each registered modality signal is calculated respectively to create a registration-related attention output sequence corresponding to each registered modality signal; A corresponding modality mask is determined based on the video-associated attention output sequence and / or each of the registered-associated attention output sequences; The noise reduction mask is generated by weighted fusion of all modal masks.

[0010] According to a keyword recognition method provided by the present invention, the step of determining a corresponding modality mask based on the video-related attention output sequence and each of the registered related attention output sequences includes: The video-associated attention output sequence and the registered-associated attention output sequence are respectively input into a mask generation network to obtain the modal mask output by the mask generation network; The mask generation network includes convolutional layers and nonlinear activation function layers. The convolutional layers are used to extract features from the video-associated attention output sequence or the registered-associated attention output sequence to generate a feature matrix. The nonlinear activation function layers are used to perform element-wise probability operations on the feature matrix to generate the modality mask.

[0011] According to a keyword recognition method provided by the present invention, obtaining the hidden state vector corresponding to each mode pair in the available mode pair set includes: Determine a similarity matrix between the query modality feature and the registered modality feature in each modality pair, wherein the similarity matrix characterizes the degree of matching between the query modality feature and the registered modality feature at different time points; The similarity matrix is ​​input into a recurrent neural network to obtain the hidden state vector corresponding to each modality pair output by the recurrent neural network.

[0012] According to a keyword recognition method provided by the present invention, the step of aggregating all the hidden state vectors to generate an aggregated feature vector includes: The hidden state vectors corresponding to all the modality pairs in the available modality pair set are concatenated to generate the aggregated feature vector.

[0013] According to a keyword recognition method provided by the present invention, determining the recognition result based on the aggregated feature vector includes: The aggregated feature vector is input into a classifier network to obtain the recognition result output by the classifier network.

[0014] The keyword recognition method provided by the present invention further includes: Collect a training sample set, which includes multiple sets of training samples. Each set of training samples includes at least one registration input sample under a registration modality, at least one query input sample under a query modality, and pre-labeled recognition result labels and clean audio signal labels. The training process of a keyword recognition model to be trained is iteratively executed using the training sample set until the second preset cutoff condition is met. In each training process, a batch of training samples is randomly selected from the training sample set, and the following training steps are performed until all training samples in the batch are traversed: A set of training samples selected from the batch is used as the input of the keyword recognition model to be trained, and the predicted recognition result output by the keyword recognition model to be trained, as well as the noise-reduced speech training signal and frame-level audio-video fusion vector sequence generated in the process are obtained. Calculate the decision loss between the predicted recognition result and the recognition result label of the training sample; Calculate the noise reduction loss between the denoised speech training signal and the clean audio signal labels of the training samples; After mapping the frame-level audio-video fusion vector sequence to the feature space where the text phoneme features in the training samples are located, a weighted processing based on a Gaussian distribution centered on the time window is performed to generate a phoneme-level fusion feature representation, and to calculate the spatial alignment loss between the fusion feature representation and the text phoneme features. The parameters of the keyword recognition model to be trained are updated based on the decision loss, the noise reduction loss, and the spatial alignment loss. The keyword recognition model to be trained includes at least a feature encoder, a mask generation network, a recurrent neural network, and a classifier network.

[0015] According to a keyword recognition method provided by the present invention, after randomly selecting a batch of training samples from the training sample set in each training process, the method further includes: Randomly remove training samples from the batch that are associated with at least one registered modality and / or at least one query modality.

[0016] According to a keyword recognition method provided by the present invention, the frame-level audio-video fusion vector sequence is determined based on the following method: After swapping the audio feature sequences corresponding to the audio signal samples and the video feature sequences corresponding to the video signal samples in the training samples with a preset probability at at least one time frame; The frame-level audio-video fusion vector sequence is obtained by fusing the audio signal sample and the video signal sample after feature swapping.

[0017] According to a keyword recognition method provided by the present invention, updating the parameters of the keyword recognition model to be trained based on the decision loss, the noise reduction loss, and the spatial alignment loss includes: The parameters of the mask generation network are updated by combining the decision loss, the spatial alignment loss, and the noise reduction loss. The parameters of the feature encoder are updated by combining the decision loss and the spatial alignment loss. The parameters of the classifier network and the recurrent neural network are updated based on the decision loss.

[0018] According to a keyword recognition method provided by the present invention, the parameters of the mask generation network are updated by combining the decision loss, the spatial alignment loss, and the noise reduction loss, including: The decision loss, the noise reduction loss, and the spatial alignment loss are weighted and summed to obtain the combined loss; The parameters of the mask generation network are updated based on the combined loss; The weight coefficients of the weighted summation are dynamically determined based on the gradient changes of the decision loss, the noise reduction loss, and the spatial alignment loss during the historical training process.

[0019] According to a keyword recognition method provided by the present invention, the method further includes: updating the weight coefficients of the weighted fusion of all modality masks to generate the noise reduction mask based on the combined loss.

[0020] According to the keyword recognition method provided by the present invention, the decision loss is a binary cross-entropy loss, the noise reduction loss is a scale-invariant signal-to-noise ratio improvement loss, and the spatial alignment loss is a mean square error loss.

[0021] The present invention also provides a keyword recognition device, comprising the following structure: The feature extraction unit is used to determine a set of available modality pairs based on at least one registration modality feature obtained under a preset registration modality and at least one query modality feature obtained under a query modality; the modality pairs in the set of available modality pairs are combinations of any of the query modality features and any of the registration modality features. The feature analysis unit is used to obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs; The feature aggregation unit is used to aggregate all the hidden state vectors to generate an aggregated feature vector. The result recognition unit is used to determine the recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of preset keywords.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the keyword recognition method as described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the keyword recognition method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the keyword recognition method as described above.

[0025] The keyword recognition method, device, electronic device, and storage medium provided by this invention solve the problem of drastic performance degradation or failure in modality missing scenarios by constructing a recognition framework that can dynamically determine modality pairs and aggregate their corresponding features based on the actual available combination of registration and query modalities. It achieves flexible compatibility and adaptive fusion of different modal inputs, and significantly enhances the generalization ability, robustness, and practicality of the keyword recognition system in complex and ever-changing real-world application scenarios. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the keyword recognition method provided by the present invention.

[0028] Figure 2 This is a schematic diagram of the process for denoising query voice signals provided by the present invention.

[0029] Figure 3 This is a schematic diagram of the process for generating a noise reduction mask provided by the present invention.

[0030] Figure 4 This is a schematic diagram of the keyword recognition system provided by the present invention.

[0031] Figure 5 This is a schematic diagram of the training process of the keyword recognition model provided by the present invention.

[0032] Figure 6 This is a schematic diagram of the keyword recognition device provided by the present invention.

[0033] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0036] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] As human-computer interaction gradually evolves towards universality, personalization, and diversification, keyword recognition systems, as key entry points in the human-computer interaction process, play an important role in conversational artificial intelligence systems in waking up devices and starting tasks. Their role is particularly crucial in application scenarios such as voice assistants, intelligent vehicle systems, and smart homes.

[0038] Traditional keyword recognition systems typically rely on fixed, predefined keyword configurations. While these systems offer some practical value in terms of general applicability, they suffer from significant limitations in terms of flexibility and user personalization. To overcome this issue, custom keyword recognition technology has emerged, enabling users to register custom keywords based on their personal preferences, thereby significantly improving the system's flexibility and adaptability.

[0039] Existing custom keyword recognition systems are mainly divided into two categories: audio registration and text registration. Audio registration allows users to input keyword templates through recording, which has the advantages of recording pronunciation features and adapting to individual differences, but it is also highly susceptible to environmental noise, leading to a decrease in recognition performance. Text registration, on the other hand, matches the keyword text entered by the user with the speech to be recognized, avoiding the problem of recording and acquisition, and has better structural stability, but it requires a high degree of accuracy in the user's pronunciation, and its effectiveness is limited, especially in diverse language environments such as dialects and accents.

[0040] Furthermore, existing custom keyword recognition systems are mostly limited to single-modal registration (i.e., only audio or text registration is supported), and there are still shortcomings in the joint utilization of multimodal information. For example, they cannot combine lip movement information in video to help determine the speaker, or perform robust discrimination automatically in the case of modality absence, thus limiting the recognition capability and practicality of custom keyword recognition systems in complex scenarios.

[0041] To address the technical bottlenecks of existing custom keyword recognition methods, such as single modality registration, poor noise robustness, and coarse spatial alignment, this invention provides a custom keyword recognition method based on flexible modality fusion. This method supports keyword registration and recognition of three modalities—text, audio, and video—within a unified model architecture, significantly improving adaptability and recognition accuracy under multi-scenario and multi-modal conditions.

[0042] As a core innovation, given that some registration modal information may be missing in real-world scenarios (such as no valid video registration or no text registration), this invention provides a modality adaptive fusion method. This method supports degrading to a non-full-modality mode when any registration modality is missing. By simultaneously optimizing the effects of single-modality, dual-modality, and tri-modality, the system can still maintain high-precision recognition capabilities when registration modal information is incomplete, thereby enhancing the generalization and practicality of keyword recognition.

[0043] The following is combined with Figures 1-7 This invention describes the keyword recognition method, apparatus, electronic device, and storage medium provided by the present invention.

[0044] Figure 1 This is a flowchart illustrating the keyword recognition method provided by the present invention, as shown below. Figure 1 As shown, this embodiment of the invention provides a keyword recognition method that can be applied to an electronic device. The electronic device can be any device with computing processing capabilities, such as a smartphone, tablet computer, personal computer, server, smart speaker, smart TV, in-vehicle infotainment system, or any other form of smart terminal. For ease of description, it will be referred to as the system below.

[0045] In this embodiment, the keyword recognition method provided by the present invention mainly includes, but is not limited to, the following steps: Step 101: Based on the registration modality features obtained under at least one preset registration modality and the query modality features obtained under at least one query modality, determine the set of available modality pairs.

[0046] A registration modality refers to the information format used by a user when pre-entering custom keyword information into the system. A registration modality can be single or a combination of multiple modalities. In one optional implementation, the registration modality may include, but is not limited to: text registration, where the user inputs a text string of keywords via keyboard or handwriting; audio registration, where the user records a voice recording containing the keywords using a microphone; and video registration, where the user records a facial video, especially lip movement video, of the user speaking the keywords using a camera.

[0047] When users enter their own custom keyword information, each user can be assigned a unique ID. The registration text, registration audio and / or registration video entered by each user can be bound to the ID and then uploaded to the backend storage, which can be uniformly stored on the cloud server.

[0048] As an optional implementation, each user can record multiple pieces of information for any registration modality. For example, user A can record two different registration texts (each with different registration keywords) at the same time or at different time periods. This allows for the identification of any one of the registration keywords during subsequent keyword recognition.

[0049] Registration modality features are mathematical representations that can be processed by computer models, obtained by encoding the registration inputs of the above-mentioned registration modalities, such as high-dimensional vectors or vector sequences.

[0050] A query modality refers to the real-time information provided by a user when keyword recognition is triggered during actual use. A query modality can be one or more. In one optional implementation, the query modality may include, but is not limited to: audio queries, i.e., user voice signals collected in real-time via a microphone; and video queries, i.e., user video footage collected in real-time via a camera. The query modality feature is the mathematical representation obtained by feature encoding the query input of the above-mentioned query modality.

[0051] The available modality pair set refers to the set of all modality pairs that can actually be used for comparative analysis in the current specific recognition task. All modality pairs in the available modality pair set are formed by combining any currently available query modality feature with any preset registration modality feature. For example, if a user presets two registration modalities (text and audio) and simultaneously provides both audio and video query modalities during the query, then the available modality pair set includes four modality pairs: (text registration, audio query), (text registration, video query), (audio registration, audio query), and (audio registration, video query). By determining the available modality pair set, this invention enables the system to dynamically identify all currently available information processing paths.

[0052] Step 102: Obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs.

[0053] Among them, the hidden state vector is a high-dimensional feature vector that can contain sequence context dependencies after being processed by a temporal modeling network (such as a recurrent neural network).

[0054] In step 102, for each modality pair in the determined set of available modality pairs, the system independently executes a processing flow to generate its corresponding hidden state vector. This processing flow may include calculating the similarity between the registered modality features and the query modality features in each modality pair, and performing temporal modeling on the changes in this similarity over time, thereby obtaining a hidden state vector that can characterize the matching degree of this modality pair and its temporal dynamics.

[0055] Step 103: Aggregate all the hidden state vectors to generate an aggregated feature vector. In this embodiment, the system combines all available hidden state vectors (the number of which depends on the number of available mode pairs) to generate a single aggregated feature vector.

[0056] The aggregation methods employed can include, but are not limited to, concatenation, element-wise addition, element-wise multiplication, or weighted summation using an attention network. For example, if two hidden state vectors were obtained in the previous step, they can be concatenated end-to-end to form an aggregated feature vector with a dimension equal to the sum of the two vectors. This aggregated feature vector concentrates analytical information from all available modality pairs, forming a panoramic description of the current query event.

[0057] Step 104: Based on the aggregated feature vector, determine the recognition result; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of the preset keyword.

[0058] The system uses the aggregated feature vector generated in the previous step as the final decision criterion, and determines the final recognition result through a preset judgment mechanism. The recognition result is used to characterize whether the query input contains user-preset custom keywords.

[0059] For example, the obtained aggregated feature vector can be input into a preset classifier, which outputs a confidence score. The confidence score is then compared with a preset decision threshold. If the score is greater than or equal to the decision threshold, the recognition result is "hit" or "yes", otherwise it is "miss" or "no".

[0060] The keyword recognition method provided by this invention solves the problem of drastic performance degradation or failure in modality missing scenarios by constructing a recognition framework that can dynamically determine modality pairs and aggregate their corresponding features based on the actual available registration and query modality combinations. It achieves flexible compatibility and adaptive fusion of different modality inputs, and significantly enhances the generalization ability, robustness and practicality of the keyword recognition system in complex and ever-changing real-world application scenarios.

[0061] As an optional embodiment, the registration modality includes at least one of text registration, audio registration, and video registration, and the query modality includes at least one of voice query and video query.

[0062] In this embodiment, the specific types of registration and query modalities are illustrated by example. The registration modal, as a form of user input of custom keyword information, may, in an optional embodiment, include: Text registration means that users enter one or more custom text strings as keywords through the system's interactive interface, such as "turn on the headlights" or "navigate home".

[0063] Audio registration refers to users recording audio snippets of themselves saying one or more custom keywords using the system's microphone.

[0064] Video registration involves users recording a video of themselves speaking one or more custom keywords using the system's camera. This video is primarily used to capture lip movement information related to pronunciation.

[0065] Users can choose any one or more combinations of the above registration modalities to register keywords. As an optional implementation, the system can assign a unique user identifier (ID) to each user. When a user enters their custom keyword information, the system can bind the user's entered registration text, registration audio, and / or registration video to the assigned user identifier, and upload the bound information to the backend server for storage, for example, in the user's dedicated data space on a cloud server.

[0066] As another optional implementation, for any registration modality, each user can record multiple messages. For example, user A can record two different registration texts at the same time or at different time periods (different registration texts correspond to different registration keywords, such as "Keyword 1: Play music" and "Keyword 2: Turn up the volume"). In this way, during the subsequent keyword recognition process, the system can recognize any keyword registered by the user. For another example, for the same keyword "hello," a user can record multiple registration audio messages so that the system can learn the user's pronunciation characteristics in different states (such as speech speed and emotional changes), thereby improving the robustness of recognition.

[0067] The query modality, as the form of information when the user actually triggers recognition, may, in one optional embodiment, include: Audio queries, which are ambient sound signals (possibly including the user's voice) collected in real time by the system through a microphone, are the most basic and common query modality.

[0068] Video query refers to video footage captured in real time by the system through a camera, especially video information containing the user's face.

[0069] In practical applications, the query modality can be audio-only (e.g., when the user is driving and not facing the in-vehicle camera), or it can include both audio and video queries. In some special cases, it can also include only video queries, that is, the keywords are identified by capturing lip movement information related to pronunciation based solely on the query video entered by the user.

[0070] It should be noted that when the query modality includes both voice query and video query, the voice query and the video query occur simultaneously.

[0071] Synchronization refers to the fact that the query audio signal collected by an audio query and the query video signal collected by a video query are generated by the same event (i.e., the user's vocalization) within the same time period, and their timestamps are strictly aligned or can be aligned through technical means. This natural, physical synchronicity provides a foundation for subsequent joint analysis and processing using multimodal information. For example, by analyzing lip movement video synchronized with the audio signal, we can help determine the pronunciation content or suppress environmental noise, thereby improving the accuracy of recognition.

[0072] The keyword recognition method provided by this invention limits the specific types of registration modality and query modality, enabling it to cover current mainstream human-computer interaction scenarios. It clarifies the source and characteristics of information from different modalities, providing a clear input definition for subsequent multimodal feature extraction and fusion processing, and enhancing the feasibility and scenario adaptability of the solution.

[0073] Based on the above embodiments, as an optional embodiment, a multimodal noise reduction technical solution is provided to improve the quality of query audio signals.

[0074] When the system detects that the current query input contains an audio query, before executing subsequent recognition processes such as determining the set of available modal pairs, an optional audio preprocessing process (i.e., multimodal noise reduction) can be started. This process aims to use other available multimodal information as an aid to purify the input query audio signal, thereby suppressing environmental noise and improving the accuracy and robustness of subsequent keyword recognition.

[0075] Figure 2 This is a flowchart illustrating the noise reduction process for query voice signals provided by the present invention, as shown below. Figure 2 As shown, the main steps include, but are not limited to, the following: After starting the noise reduction of the query speech signal, the query audio signal used in the query input is first segmented into frames to obtain the query audio frame-level representation sequence.

[0076] Specifically, by segmenting the continuous time-domain query audio signal into a series of short audio frames that are sequential in time, and extracting the acoustic features of each short audio frame (called the query audio frame-level representation), a time-seriesd feature sequence, namely the query audio frame-level representation sequence, is formed. Each query audio frame-level representation can be, but is not limited to, audio features such as Mel-Frequency Cepstral Coefficients (MFCC) and Log-Mel Spectrogram (LMS).

[0077] Further, a noise reduction mask is determined. A noise reduction mask is a matrix or tensor with the same or compatible dimensions as the time-frequency representation of the audio signal to be processed, and its internal values ​​are typically used to characterize the degree to which the energy of the corresponding time-frequency unit in the original audio signal should be preserved.

[0078] In this embodiment, the noise reduction mask can be generated based on the result of a correlation calculation. Specifically, the core idea of ​​this correlation calculation is to measure the correlation strength between each frame of the query audio signal and all currently available auxiliary information sources, using each frame as a reference. The auxiliary information sources may include: features of a query video signal synchronized with the query audio signal (when a video query modality also exists) and / or features of at least one registration modality signal pre-recorded by the user (e.g., features of registration text information, registration audio signal, or registration video signal). For example, the auxiliary information sources may include only one or more of the query video signal, registration text information, registration audio signal, and registration video signal, or any combination thereof.

[0079] By calculating the correlation between each query audio frame-level representation in the query audio frame-level representation sequence and one or more auxiliary information sources, the system can comprehensively determine which parts of the query audio are more likely to belong to the target content spoken by the target speaker and which parts are more likely to be noise. Based on these comprehensive judgments, a noise reduction mask is finally determined to guide the noise reduction process. This design reflects the flexibility of the keyword recognition method provided by this invention, that is, the system utilizes all available auxiliary information for noise reduction without requiring all auxiliary information to be present simultaneously.

[0080] Finally, the denoising mask can be used to denoise the query audio signal. Specifically, the denoising implementation can be as follows: First, the original query audio signal is converted to the time-frequency domain using a method such as Short Time Fourier Transform (STFT) to obtain its spectrum. Then, the denoising mask determined in the previous step is applied to this spectrum using element-wise multiplication to attenuate or suppress the energy components corresponding to noise in the spectrum. Finally, the spectrum after masking is reconstructed into a time-domain waveform using a method such as Inverse Short Time Fourier Transform (iSTFT), thus obtaining the denoised query audio signal. The denoised query audio signal can then be used for subsequent feature extraction and keyword recognition processes.

[0081] It should be noted that the above noise reduction process can be executed iteratively. That is, the output of one noise reduction iteration can be used as the input for the next, repeatedly executing the steps of determining the noise reduction mask and using the mask for noise reduction. The termination condition for the iteration is whether a first preset cutoff condition is met, which may include, but is not limited to: reaching a preset number of iterations, the improvement in noise reduction effect being lower than a preset threshold, or the processing latency exceeding a preset upper limit. Through iterative execution, better noise reduction effects can be obtained in scenarios where real-time requirements are not high, further improving feature purity. The specific number of iterations can be determined according to actual usage needs. When real-time requirements are not high, multiple iterations of noise reduction can be performed to obtain better results; if real-time requirements are high, a single noise reduction iteration can improve the effect without introducing excessive latency.

[0082] The keyword recognition method provided by this invention introduces a multimodal noise reduction process, which can flexibly utilize the currently available multimodal information to perform targeted noise suppression on the input query audio signal before executing the core recognition logic, thereby significantly improving the recognition performance and stability of the system in various complex noise environments.

[0083] Figure 3 This is a schematic diagram of the process for generating a noise reduction mask provided by the present invention, as shown below. Figure 3 As shown, the implementation method for determining a noise reduction mask provided in this embodiment can be decomposed into a structured processing flow that includes parallel computing and subsequent fusion.

[0084] First, the system determines whether there is a query video signal that is time-synchronized with the query audio signal to be denoised. Simultaneously, it retrieves the user's pre-registered keyword information from storage, i.e., registration text information, registration video signal, and / or registration audio signal (if present), and extracts the corresponding feature representations.

[0085] Then, the system will launch multiple independent computation streams in parallel to obtain the correlation evaluation of different auxiliary information sources (query video signal, registered text information, registered video signal and / or registered audio signal) with the query audio signal. For example, for each query audio frame-level representation in the query audio signal, the correlation mapping between its feature representation and other auxiliary information sources is calculated through a cross-attention mechanism. These computation streams may specifically include: First, the cross-attention calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal is calculated to create a video-related attention output sequence.

[0086] Cross-attention computation, performed when a query video signal is determined to be available, is a technique for modeling dependencies between two different sequences. In this embodiment, each query audio frame-level representation in the query audio frame-level representation sequence is used as a query vector to perform attention computation with the feature representations of the entire query video signal, which serve as keys and values. After performing this operation on all frames in the query audio frame-level representation sequence, a video-associative attention output sequence is obtained. Each element of this video-associative attention output sequence incorporates the most relevant information from the entire query video signal to the corresponding query audio frame, thereby quantifying the physical synchronicity and correlation between the audio and video.

[0087] Secondly, the cross-attention calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and each registered modality signal is calculated to create a registration-related attention output sequence corresponding to each registered modality signal.

[0088] This step is performed in parallel for all available registration modal signals. For example, if there are two registration modalities, namely registration text information and registration audio signal, the system will perform the following operations: (1) Perform a cross-attention computation, where each query audio frame level representation of the query audio signal is used as the query vector, and the feature representation of the entire registered text information is used as the key and value, to create a text registration association attention output sequence.

[0089] (2) Perform another cross-attention calculation, where each query audio frame-level representation of the query audio signal is used as the query vector, and the feature representation of the entire registered audio signal is used as the key and value, to create an audio registration association attention output sequence. Through this step, the consistency between the query audio signal and the standard pronunciation of the registered keywords (from the text registration information), as well as the similarity with the user's voiceprint features (from the registered audio signal), can be quantified.

[0090] Then, a corresponding modality mask is determined based on the video-associated attention output sequence and / or each of the registered-associated attention output sequences.

[0091] To transform the obtained abstract attention output sequence into specific control signals, the system independently determines a corresponding modality mask for each attention output sequence (e.g., video association attention output sequence, text registration association attention output sequence, etc.). The modality mask is a matrix compatible with the time-frequency representation dimension of the query audio, where each element represents the degree of preservation of the corresponding component of the audio signal based on the corresponding auxiliary information source.

[0092] In one alternative implementation, the process of determining the modality mask may include feeding the attention output sequence into a network containing convolutional layers and nonlinear activation functions for processing. The output range of the activation function may be set to [0, 1], and its physical meaning can be considered as the probability of preserving the corresponding query audio frame.

[0093] Finally, all modal masks are weighted and fused to generate the denoising mask. This involves combining all available modal masks (the number of which depends on the number of available auxiliary information sources) through a fusion strategy to generate the final, single denoising mask.

[0094] In one optional implementation, the fusion strategy is weighted fusion, which involves multiplying each modal mask by a corresponding weight coefficient and then summing the results. This weight coefficient can be a preset fixed value or a learnable parameter acquired during model training. It characterizes the importance of different auxiliary information sources in different noise scenarios, or can be understood as the contribution of each auxiliary information source to the noise reduction task under different conditions. For example, in dimly lit scenarios with poor video quality, the model may automatically reduce the weight of the modal mask corresponding to the video modality during training, while in noisy background environments, it may increase the weight of the modal mask corresponding to video modalities strongly correlated with lip movement information.

[0095] The reason why the noise reduction method provided in this embodiment can effectively improve the feature purity and discriminative power in the subsequent recognition stage is analyzed by the following technical principle: (1) Noise source separation using the strong physical correlation between audio and video: The generation of speech is physically synchronous and strongly correlated with the movement of the vocal organs (especially the lips). This invention utilizes this principle to effectively distinguish between query audio frames that are consistent with the current lip movement pattern (determined as valid speech) and query audio frames that are inconsistent (determined as noise, especially background human voice or non-speech noise) by calculating the correlation between the query audio signal and the query video signal (including lip movement features). This provides a strong physical constraint for separating valid query speech features in complex acoustic environments.

[0096] (2) Utilizing semantic priors from registered text information for content focusing: Registered text information provides the standard phoneme composition of keywords, which constitutes a high-level semantic constraint on audio content. During the noise reduction process, the system tends to retain audio segments whose acoustic feature evolution paths match the phoneme sequence. This is equivalent to introducing a content filter, making the noise reduction process not only energy-level noise reduction but also content-level focusing, thereby enhancing audio features related to keywords.

[0097] (3) Identification using the voiceprint characteristics of the registered audio signal: The registered audio signal contains the user's unique voiceprint features (such as fundamental frequency, formants, etc.). By comparing the query audio signal with the registered audio signal, the system can identify and enhance the audio parts that match the user's voiceprint characteristics. This provides another basis for distinguishing the target speaker from the interfering speaker, and realizes identity-based targeted speech enhancement.

[0098] (4) Adaptive fusion strategy to improve system robustness: By setting learnable weight coefficients to fuse each modal mask, an adaptive information fusion framework is constructed, which allows the system to learn to evaluate the reliability of different modal masks in specific scenarios during training and dynamically adjust their contribution during inference. This ensures that the system can still combine available information in the best way when any modal information is missing or degraded, thereby ensuring the stability of the noise reduction effect and the ability to generalize to changing environments.

[0099] In summary, the keyword recognition method provided by this invention decomposes the process of determining the denoising mask into parallel cross-attention calculation, independent modal mask generation, and final weighted fusion, thus constructing a well-structured and scalable multimodal information fusion framework. This method can effectively utilize the cross-attention mechanism to capture deep cross-modal correlations, providing a reliable technical path for generating high-quality denoising masks.

[0100] As an optional implementation, the transformation from the abstract attention output sequence to the specific modality mask is achieved through a specially designed neural network module. The system configures an independent mask generation network with non-shared parameters for each attention output sequence (e.g., video association attention output sequence, text registration association attention output sequence, etc.).

[0101] Specifically, the step of determining a corresponding modality mask may include: First, the video-related attention output sequence and the registered related attention output sequence are respectively input into a mask generation network to obtain the modal mask output by the mask generation network.

[0102] For example, for a video-related attention output sequence, the system inputs it as a whole into its dedicated mask generation network. After a series of operations within the network, it outputs an audio mask for video guidance. Similarly, each registered related attention output sequence is also fed into its dedicated mask generation network to generate its corresponding modal mask.

[0103] Secondly, this embodiment further optimizes the internal structure of the mask generation network, which may include convolutional layers and nonlinear activation function layers, etc.

[0104] The convolutional layer is used to extract features from the video-associated attention output sequence or the registered-associated attention output sequence to generate a feature matrix.

[0105] In a preferred embodiment, the convolutional layer can be a one-dimensional convolutional layer. This layer, through its internal convolutional kernel, performs sliding computation along the temporal dimension of the attention output sequence to effectively integrate contextual information from neighboring time frames and further abstract and transform the input attention features. This extracts higher-level features that are more robust to distinguishing between speech and noise, generating a feature matrix. Furthermore, by configuring parameters such as the number of channels in the convolutional layer, the dimensionality of the features can be adjusted to match the requirements of subsequent processing.

[0106] The nonlinear activation function layer is used to perform element-wise probability operations on the feature matrix to generate the modality mask.

[0107] In a preferred embodiment, the nonlinear activation function can be a sigmoid activation function (such as the sigmoid function). The nonlinear activation function layer independently performs a nonlinear transformation on each element of the feature matrix output from the previous convolutional layer. The sigmoid function can smoothly map its input, ranging from any real number, to the open interval (0, 1). In the application scenario of this invention, the physical meaning of assigning this output value between (0, 1) is the probability of preserving the corresponding time-frequency unit of the audio signal. A value close to 1 indicates that the component should be preserved, and a value close to 0 indicates that the component should be suppressed. Therefore, by performing this element-wise probability operation on the feature matrix, the system can ultimately generate a numerical modal mask that can be directly used for subsequent masking operations.

[0108] The keyword recognition method provided by this invention explicitly implements the process from attention output to modality mask generation through a network containing convolutional layers and nonlinear activation functions. The use of convolutional layers allows the system to effectively reference the contextual information of preceding and following frames when determining whether the current frame is noise, avoiding misjudgments based solely on instantaneous features and improving the smoothness and temporal rationality of the decision-making process. The introduction of nonlinear activation function layers successfully transforms abstract feature vectors into retention probabilities with clear physical meaning, providing a quantitative and refined control basis for subsequent noise reduction processing. This enables more accurate separation of target speech from noise, improving the quality of the final noise reduction mask.

[0109] As an optional embodiment, the present invention provides an implementation process for generating the corresponding hidden state vector for each available modality pair. This implementation process can be specifically decomposed into a processing flow that includes similarity calculation and temporal modeling, and may specifically include: First, a similarity matrix is ​​determined between the query modal features and the registered modal features in each modal pair, with the aim of quantifying the degree of matching between the query modal features and the registered modal features in a modal pair.

[0110] In this embodiment, the similarity matrix is ​​a two-dimensional matrix, whose rows and columns correspond to the constituent units of the registered modal features (e.g., phoneme sequences of registered text information, or frame sequences of registered audio signals) and the constituent units of the query modal features (e.g., query audio frame-level representation sequences of query audio signals), respectively. Each element in the similarity matrix represents the similarity or matching score of two modal features on a specific constituent unit.

[0111] In an alternative implementation, the similarity or matching score can be obtained by calculating the cosine similarity between the two feature vectors.

[0112] By determining the similarity matrix for each modality pair, the system can obtain a quantitative representation of the degree of matching between the query modality features and the registered modality features at different time points.

[0113] Considering that the obtained similarity matrix only reflects isolated, instantaneous matching at each time point and lacks the ability to model contextual information, the present invention further inputs the similarity matrix into a recurrent neural network to obtain the hidden state vector corresponding to each modality pair output by the recurrent neural network.

[0114] To address the lack of temporal context in similarity matrices, this embodiment introduces recurrent neural networks (RNNs) for temporal modeling. RNNs are a network structure adept at processing sequential data; they can capture and remember long-range dependencies in sequential data through their internal recurrent connections.

[0115] In a preferred embodiment, the recurrent neural network may be a gated recurrent unit (GRU) network or a long short-term memory (LSTM) network, etc.

[0116] The system inputs the similarity matrix generated in the previous step into the recurrent neural network (RNN) as a sequence, categorized by the time dimension (columns of the matrix) of the query modality signal. The RNN processes the similarity information at each time step sequentially and updates the current hidden state by combining it with the previously stored hidden state. After processing the entire sequence, the final hidden state output by the RNN is used as the hidden state vector corresponding to that modality pair. This hidden state vector is no longer merely a set of instantaneous matching scores, but a deep feature representation containing the entire sequence matching pattern, order dependencies, and contextual information.

[0117] The keyword recognition method provided by this invention explicitly defines the acquisition process of the hidden state vector as first calculating the instantaneous similarity matrix, and then performing temporal modeling through a recurrent neural network. This effectively overcomes the limitation of lacking contextual information when judging solely based on similarity scores. In particular, the introduction of the recurrent neural network enables it to learn and understand the evolution and dependencies of similarity scores over time. This allows the final generated hidden state vector to more accurately represent the degree of matching of a keyword as a complete temporal pattern, rather than the coincidental similarity of scattered fragments, significantly improving the reliability and accuracy of subsequent aggregation and judgment steps.

[0118] As an optional embodiment, the present invention provides a specific and efficient implementation method for weighting all acquired hidden state vectors to generate aggregate feature vectors.

[0119] Specifically, after completing the independent analysis of all modal pairs in the available modal pair set and obtaining the hidden state vector corresponding to each modal pair, the system will perform a concatenation operation, that is, by combining multiple independent hidden state vectors in a predefined order, connecting them end to end into a longer, higher-dimensional vector.

[0120] For example, suppose in a keyword recognition task, the system determines that there are three available modality pairs in the set of available modality pairs and generates three hidden state vectors accordingly, namely vector A, vector B, and vector C. If each vector has a dimension of 128, the system will concatenate these three vectors to generate a final aggregated feature vector. The dimension of this aggregated feature vector will be 128 + 128 + 128 = 384, and its internal structure can be represented as [the content of vector A, the content of vector B, and the content of vector C].

[0121] It should be noted that since the number of mode pairs in the available mode pair set changes dynamically based on the actual input, the dimension of the aggregated feature vector generated by concatenation is also variable. For example, when there is only one mode pair in the available mode pair set, the aggregated feature vector is the hidden state vector of that mode pair itself; when there are six mode pairs, the dimension of the aggregated feature vector will be six times the dimension of a single hidden state vector. This dynamically changing structure is one of the core mechanisms for achieving flexible mode fusion in this invention, and the subsequent decision module (such as the classifier network) will be designed to handle this variable-dimensional input, or to unify it to a fixed dimension through methods such as zero padding.

[0122] The keyword recognition method provided by this invention explicitly defines the aggregation operation as concatenation that completely preserves all the information contained in the analysis results of each independent modality pair (i.e., each hidden state vector), and aggregates this information from different analysis dimensions into a unified feature representation. This approach provides the most comprehensive and original decision-making basis for the subsequent classifier network, enabling the classifier to comprehensively learn and discriminate complex association patterns from all available modality pairs in a higher-dimensional feature space, thereby improving the accuracy of the final recognition result.

[0123] As an optional embodiment, in the keyword recognition method provided by the present invention, the final keyword recognition decision process is completed by a pre-trained classifier network.

[0124] A classifier network is a neural network model capable of performing classification tasks. In an optional implementation, the classifier network can be a multi-layer perceptron (MLP) consisting of one or more fully connected layers.

[0125] Specifically, after obtaining the aggregated feature vector, the system of this invention performs the following steps: First, the aggregated feature vector is input into a classifier network to obtain the recognition result output by the classifier network.

[0126] The aggregated feature vector, a high-dimensional feature representation containing analytical information of all modality pairs, is fed into the classifier network. The classifier network then makes a final decision on this high-dimensional feature through a series of internal nonlinear transformations.

[0127] In a preferred embodiment, the output layer of the classifier network typically includes an activation function (e.g., the sigmoid function) that outputs a value between 0 and 1. This value serves as a specific representation of the recognition result; for example, it could be a confidence score, used to characterize the probability or credibility of the query input containing preset keywords.

[0128] After obtaining the confidence score, the system can further compare the score with a preset decision threshold to obtain a binary final recognition result (yes / no). For example, if the confidence score is greater than or equal to the decision threshold, it is judged as "hit", that is, the keyword is recognized; otherwise, it is judged as "miss".

[0129] The classifier network is trained based on a pre-constructed set of full modality pairs. The set of full modality pairs includes full modality pairs formed by pairwise combinations of registered modality feature samples under all registered modalities and query modality feature samples under all query modalities. Each full modality pair is labeled with a recognition result label.

[0130] This embodiment further illustrates the training method of the classifier network. To enable the classifier network to correctly process aggregated feature vectors of variable dimensions generated from combinations of different modality pairs, its training process is based on a specially constructed set of full modality pairs. This set of full modality pairs aims to cover all theoretically possible combinations of modality pairs. For example, if the system supports three registration modalities (text, audio, and video) and two query modalities (audio and video), the full modality pair set will contain all training samples under all six combinations. During training, the system provides the classifier network with aggregated feature vectors generated from different numbers and types of modality pairs (ranging from one to six), and uses the corresponding recognition result labels (e.g., 1 indicating "keyword" and 0 indicating "non-keyword") as supervision signals to optimize the classifier network.

[0131] The keyword recognition method provided by this invention explicitly achieves the final recognition decision through a specially trained classifier network, effectively utilizing the powerful nonlinear classification capability of deep neural networks, and is able to learn complex decision boundaries from high-dimensional aggregated feature vectors.

[0132] As will be described in subsequent embodiments, the present invention also ensures that the classifier network can robustly handle the situation where the aggregate feature vector dimension changes due to the absence of actual input modalities by training on a set that covers all modality combinations. This enables the entire keyword recognition method to make reliable decisions under any modality combination, greatly enhancing the generalization and practicality of the system.

[0133] To address the current lack of a keyword recognition system capable of adaptively handling complex acoustic environments (such as strong noise interference) and possessing multimodal compatibility, this invention provides a novel custom keyword recognition system. The system aims to achieve stable and accurate keyword recognition by flexibly switching or fusing available modal data regardless of the completeness or absence of modal information. This system not only exhibits strong robustness to noisy environments but also dynamically integrates audio, text, and video information based on actual input conditions. This significantly improves adaptability and recognition performance in diverse practical application scenarios while ensuring a positive user experience.

[0134] Figure 4 This is a schematic diagram of the keyword recognition system provided by the present invention. The following will be discussed in conjunction with the attached diagram. Figure 4 As shown, the specific process of a keyword recognition method provided by an embodiment of the present invention will be described in detail. This embodiment cleverly integrates keyword registration modules of different modalities into one framework and can flexibly deal with the situation of modality missing.

[0135] Specifically, the keyword recognition method of this embodiment can be applied to a keyword recognition system including a text encoder, an audio encoder, a video encoder, a multimodal noise reduction module, a multimodal alignment module, and a decision maker. It can support one or more of the three modalities of text, audio, and video as the registration modal for keyword registration information, and also support the recognition of keyword information under the query modal of one or both of the two modalities of audio and video.

[0136] First, data preparation and feature extraction are performed.

[0137] The system can receive registration text information, registration audio signals, or registration video signals as registration inputs, and extract features from these registration inputs through the corresponding text encoder, audio encoder, or video encoder to obtain the corresponding registration modality features.

[0138] For registered text information, natural language processing techniques can be used for word segmentation to convert the text into phoneme representations. The classification layer weights of the pre-trained model can then be used as high-dimensional vector representations of the corresponding phonemes to provide text features that are discriminative at the phoneme level.

[0139] For the registered audio signal, it can be padded with zeros to a uniform duration and input into a pre-trained speech representation model (such as the Whisper-tiny model) to obtain a high-dimensional semantic embedding vector output by the output layer of the speech representation model, which can effectively capture speech content, pronunciation patterns and speaker characteristics.

[0140] For registered video signals, a pre-trained video feature extractor containing residual connection networks and three-dimensional convolutional neural network (such as ResNet+3D CNN) structures can be used to extract features from the lips and surrounding facial regions in the video frames, in order to focus on capturing the user's lip shape changes.

[0141] Meanwhile, the system receives real-time query audio signals and / or query video signals as query inputs, and obtains query modal features through the corresponding audio encoder and video encoder.

[0142] Furthermore, the system also performs optional multimodal noise reduction processes, which are designed to perform auxiliary operations such as noise reduction on the queried audio and registered video signals, so as to effectively distinguish between the target speaker and the interfering speaker in the speech signal and suppress noise.

[0143] Specifically, the system performs frame-level segmentation on the query audio signal to obtain a query audio frame-level representation sequence. Subsequently, a noise reduction mask is determined. This mask is determined based on the correlation calculation between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal (which contains lip-sync information that is strictly aligned with the query audio in time) and all available registration modal signals (registration text information, registration audio signal, and registration video signal). This correlation calculation is implemented through a cross-attention mechanism to create multiple correlated attention output sequences, and the corresponding modal masks are determined based on these sequences respectively.

[0144] Among them, compared with registered text information, registered audio signals and registered video signals, querying video signals is particularly effective in helping to distinguish between the target speaker and the interfering speaker.

[0145] Finally, all modal masks are weighted and fused to generate the final denoised mask, which is then used to denoise the query audio signal.

[0146] The denoised query audio signal will be re-encoded to obtain cleaner audio features. The number of iterations in this denoising process can be determined according to actual usage requirements. When real-time requirements are not high, multiple iterations of denoising can achieve better results; if real-time requirements are high, a single denoising iteration can improve the effect without introducing too much latency.

[0147] Finally, the core keyword recognition process is executed.

[0148] (1) The system determines a set of available modality pairs based on all currently available query modality features that may have been denoised, and all preset registered modality features.

[0149] (2) For each modality pair in the available modality pair set, the system independently calculates the similarity matrix between its registered modality features and the query modality features, and inputs the similarity matrix into a recurrent neural network to obtain the hidden state vector that can characterize the temporal matching pattern of the modality pair.

[0150] (3) All available hidden state vectors are concatenated to generate an aggregated feature vector. The composition of the aggregated feature vector is dynamic, and its composition varies from one to six, which ensures the generalization ability for any modality missing scenario.

[0151] (4) Input the obtained aggregated feature vector into a classifier network. The classifier network outputs a confidence score between 0 and 1. The system compares the score with a preset decision threshold. If the score is greater than or equal to the decision threshold, the recognition result is determined as "hit" (which is a user-defined keyword); otherwise, the system returns a "miss" result.

[0152] This invention implements a custom keyword recognition method with multi-modal fusion, adaptive robustness, and fine spatial alignment, enabling the system to maintain high-precision recognition capability even when modal information is incomplete. This enhances the system's generalization and practicality, and is of great significance for promoting the practical application and popularization of intelligent voice interaction systems.

[0153] Figure 5 This is a schematic diagram of the training process of the keyword recognition model provided by the present invention, as follows: Figure 5 As shown, this embodiment provides a method for design-training a keyword recognition model to be trained. The keyword recognition model to be trained includes at least the feature encoder, mask generation network, recurrent neural network, and classifier network mentioned in the above embodiment. Through such a joint optimization training process, the trained keyword recognition model achieves simultaneous improvement in the three key capabilities of noise reduction, alignment, and decision-making. Specific implementation methods of this training method may include: First, a training sample set is collected.

[0154] To ensure that the keyword recognition model under training can learn rich multimodal association information, this embodiment will pre-construct a high-quality training sample set. Specifically, a recording device facing the speaker can be used to record strictly aligned audio and video signal samples in various home scenarios (such as the living room and bedroom), while simultaneously recording the corresponding spoken content as text information samples. Suitable keywords (e.g., nouns) for human-computer interaction are selected from these text information samples, and corresponding audio and video clips are extracted from the audio and video signal samples.

[0155] To closely resemble real-world usage scenarios, audio signal samples can be simulated at different signal-to-noise ratio levels (e.g., 0 dB, 3 dB, 6 dB, 9 dB). Various household noises (e.g., TV background noise, steady-state fan noise, music) can be used for noise data. Each training sample set includes registered input samples (text information samples, audio signal samples, and video signal samples) and query input samples (including noise frequencies and videos), pre-labeled with "truth values" for supervised learning: a recognition result label (e.g., 1 for "is a keyword," 0 for "no") and a clean audio signal label (i.e., the original audio without noise).

[0156] Secondly, the training sample set is used to iteratively train a keyword recognition model to be trained. The entire training process is an iterative optimization process until a second preset cutoff condition is met. This second preset cutoff condition may be reaching a preset number of training epochs, or the performance indicators (such as accuracy) of the trained model on the validation set no longer improving.

[0157] In each training iteration, the system randomly selects a batch of training samples from the training sample set, and performs the following core training steps for each group of training samples within that batch: First, a set of training samples is used as input to the keyword recognition model to be trained, in order to obtain the various outputs of the model. After receiving registration input and query input, the keyword recognition model performs forward computation through its internal modules, generating intermediate or final results for calculating different loss functions, specifically including: The final predicted recognition result (a confidence score) is output by the decision module (which includes a recurrent neural network and a classifier network).

[0158] The noise-reduced speech training signal obtained after processing by the multimodal noise reduction module.

[0159] The frame-level audio and video fusion vector sequence obtained after processing by the feature encoder and multimodal alignment module.

[0160] Second, calculate three different losses, including: (1) Compare the predicted recognition result with the recognition result label pre-annotated in the training sample to calculate the decision loss, which is used to evaluate the accuracy of the model's final decision.

[0161] (2) The noise reduction training signal is compared with the clean audio signal label pre-labeled in the training sample to calculate the noise reduction loss, which is used to evaluate the ability of the noise reduction module to recover clean speech.

[0162] (3) The frame-level audio-video fusion vector sequence is mapped to the feature space of the text phoneme features through a learnable spatial mapping network. Then, a Gaussian-like distribution weighting process based on the center of the time window is performed on the mapped sequence to aggregate it into a phoneme-level fusion feature representation. Finally, this phoneme-level fusion feature representation is compared with the text phoneme features extracted from the text information of the training samples to calculate the spatial alignment loss. This loss is used to optimize the semantic consistency of features of different modalities.

[0163] Third, update the model parameters based on the calculated three types of losses.

[0164] The system combines the calculated decision loss, denoising loss, and spatial alignment loss (e.g., through weighted summation) to form a total joint loss. Then, using backpropagation, it calculates the gradient of this joint loss with respect to all learnable parameters in the model and uses this gradient to make minor adjustments and updates to these parameters. The updated parameters originate from at least the model components, including the feature encoder, mask generation network, recurrent neural network, and classifier network.

[0165] The keyword recognition method provided in this embodiment can optimize the three key capabilities of the system—noise reduction, alignment, and decision-making—in an end-to-end manner within a unified framework using a combined loss function. This collaborative optimization approach ensures that the various modules within the keyword recognition model can cooperate with each other to jointly serve the final keyword recognition goal, thereby constructing a powerful model with high performance under various complex conditions.

[0166] To improve the fusion effect of multimodal information in a unified semantic space, this invention introduces a multimodal spatial alignment mechanism. The core of this mechanism lies in effectively converting continuous, frame-level audio-video fusion features into discrete, phoneme-level features corresponding to text phonemes. Specific implementation methods may include: First, the frame-level audio-video fusion vector sequence is mapped to the feature space of the text phoneme features in the training samples through a learnable spatial mapping network.

[0167] The frame-level audio-video fusion vector sequence is a temporal representation obtained after the system fuses audio and video features. To address the issue of inconsistencies in the original feature spaces of different modal features, this embodiment inputs the frame-level audio-video fusion vector sequence into a spatial mapping network. This spatial mapping network can consist of one or more fully connected layers, and its parameters are learnable. The spatial mapping network linearly or non-linearly transforms the input audio-video fusion features from their original feature space to a new latent space. This latent space is designed to have the same dimension and similar distribution characteristics as the feature space containing the text phoneme features, thus laying the foundation for subsequent direct comparison and alignment.

[0168] Secondly, the mapped sequence is subjected to a Gaussian-like distribution weighting based on the center of the time window to generate phoneme-level fusion feature representations.

[0169] To achieve frame-level to phoneme-level conversion and avoid the error propagation problem that may be introduced by traditional frame alignment, the processing procedure in this embodiment can be further broken down as follows: For each phoneme contained in the keyword text in the training samples, the frame sequence interval corresponding to the phoneme on the time axis, i.e. the time window, is determined in advance through techniques such as forced alignment, and the center frame position of the time window is found.

[0170] Using the center frame of the time window as the reference point, a Gaussian-like weight distribution is constructed, where the center frame is given the highest weight, and the weight of frames farther away from the center frame decreases according to a decay law similar to that of a Gaussian function.

[0171] Each spatially mapped frame-level vector within the time window is multiplied by its corresponding weight, and then all weighted vectors are summed or averaged. Through this weighted aggregation process, multiple frame-level vectors within a time window are collapsed or aggregated into a single fused feature representation that can represent the overall characteristics of the phoneme.

[0172] Finally, after obtaining the phoneme-level fused feature representations that correspond one-to-one with each phoneme in the keyword, the spatial alignment loss between the fused feature representations and the text phoneme features can be calculated. For example, this spatial alignment loss can be obtained by calculating the mean squared error (MSE) between these two sets of phoneme-level vector sequences.

[0173] The spatial alignment loss calculation method provided by this invention defines the conversion process from frame-level features to phoneme-level features as a combination of spatial mapping and Gaussian-like weighted aggregation. By improving the granularity of alignment from the unstable frame level to the more semantically stable phoneme level, and using a weighted strategy to highlight the core features of each phoneme, it achieves more refined and robust cross-modal modeling of keyword pronunciation units, effectively improving the consistency and discriminability of representations of different modalities in a unified space.

[0174] To ensure that the keyword recognition method provided by this invention can maintain stable and accurate performance in real-world scenarios where modalities are missing or incomplete, a training strategy simulating modal missingness is introduced during the model training phase.

[0175] Specifically, after randomly selecting a batch of training samples from the complete training sample set, the system performs a random removal operation before sending the batch of samples into the model to be trained for computation.

[0176] The random removal operation refers to temporarily removing data related to one or more modalities from the batch of training samples with a certain preset probability. This operation is not a physical deletion of data, but rather a simulation of the situation where the modality information is unavailable in the current training iteration.

[0177] For example, for a training sample that originally contains three registration modalities: text, audio, and video, the system can randomly ignore the text registration input with a certain probability and use only the audio and video registration inputs for subsequent model training.

[0178] For example, for a training sample that originally contained both audio and video query modal inputs, the system can randomly ignore the video query input with a certain probability and use only the audio query input for subsequent model training.

[0179] The system can also randomly remove both registration and query modalities to simulate more complex modality-deficient scenarios. For example, it can provide only text registration input and audio query input, while ignoring all other modalities.

[0180] By performing this random, dynamic modality removal in each batch of training, the keyword recognition model is forced to learn how to make optimal decisions under various conditions of incomplete information. For example, when the video modality is removed, the model must learn to rely more on audio and text information; when the text registration modality is removed, the model learns to extract key features only from audio or video registration information.

[0181] The simulated modality loss training strategy introduced in this embodiment forces the model to learn the complementarity and redundancy between different modal information, which can greatly enrich the diversity of training data. This ensures that the keyword recognition model under training will not overly depend on any single modality or fixed modality combination during the optimization process, significantly enhancing the generalization ability and robustness of the final keyword recognition method in real applications. This allows it to easily cope with various modality loss situations caused by user habits, device status, or environmental limitations.

[0182] To enhance the interactivity and synergy between audio and video modalities, this invention further introduces a random replacement mechanism when generating the frame-level audio-video fusion vector sequence for subsequent spatial alignment. This random replacement mechanism does not directly fuse the original audio and video features, but instead first constructs a more robust intermediate representation through a feature exchange operation. Its implementation may include the following steps: First, the audio feature sequences corresponding to the audio signal samples and the video feature sequences corresponding to the video signal samples in the training samples are swapped at least once in time frames with a preset probability.

[0183] After acquiring the audio feature sequence (e.g., composed of a series of audio feature frame vectors) aligned with the audio signal sample time, and the video feature sequence (e.g., composed of a series of video feature frame vectors) aligned with the video signal sample time, the system iterates through the time frames of both sequences. At each time frame, the system randomly decides whether to perform a feature swap operation based on a preset probability (e.g., 15%).

[0184] Feature swapping refers to replacing the audio feature vector of a time frame with a video feature vector that is time-aligned, and / or replacing the video feature vector of a time frame with an audio feature vector that is time-aligned. This feature swapping operation is performed independently in different time frames.

[0185] Secondly, the audio signal sample and the video signal sample after feature exchange are fused to obtain the frame-level audio-video fusion vector sequence.

[0186] After completing the feature swapping operation described above, the system will obtain two new feature sequences with some features swapped. The system will then merge these two new feature sequences.

[0187] In one optional implementation, the fusion operation is concatenation, which involves concatenating the audio feature vector (which may or may not have been swapped) with the video feature vector at each time frame to form a new vector with a higher dimension. Arranging the new vectors generated in all time frames in sequence constitutes the final frame-level audio-video fused vector sequence.

[0188] This invention introduces a random feature interchange mechanism before audio-video feature fusion, thereby introducing structured noise and uncertainty into the keyword recognition model to be trained. This acts as an effective regularization method, forcing the audio and video feature encoders to learn to generate more shared and complementary feature representations that include information from each other's modalities, rather than operating independently. Simultaneously, it forces other downstream modules of the keyword recognition model to learn to collaboratively process information from different channels, rather than relying on a fixed input structure. This significantly enhances the robustness of the final learned joint audio-video representation and improves the model's generalization ability when the quality of information in a particular modality is poor or partially missing, thus laying a solid foundation for achieving higher-precision spatial alignment in the future.

[0189] As an optional embodiment, updating the parameters of the keyword recognition model to be trained based on the decision loss, the noise reduction loss, and the spatial alignment loss includes, but is not limited to, the following implementation steps: The parameters of the mask generation network are updated by combining the decision loss, the spatial alignment loss, and the noise reduction loss. The parameters of the feature encoder are updated by combining the decision loss and the spatial alignment loss. The parameters of the classifier network and the recurrent neural network are updated based on the decision loss.

[0190] In this embodiment, during the training process of the keyword recognition model, the update process of the model parameters is not simply an indiscriminate application of all loss gradients to all model components (mainly including the feature encoder, the mask generation network, the recurrent neural network, and the classifier network, etc.). Instead, a more refined asymmetric update mechanism is adopted, which reflects the inherent logical relationship between different loss functions and different modules. The principle of this asymmetric update mechanism originates from the chain rule of backpropagation of loss in a neural network, that is, the parameter update of a module will be affected by all gradients flowing through it. The parameter update steps can be: First, the parameters of the classifier network and the recurrent neural network are updated based on the decision loss loss3.

[0191] The decision loss (loss3) is calculated directly from the final outputs of the classifier network and the recurrent neural network, which form the core of the decision-maker. Therefore, during backpropagation, the gradient generated by loss3 will be directly used to update the parameters of these two modules to optimize the model's final decision-making ability.

[0192] In a preferred embodiment, the decision loss loss3 can be calculated using a binary cross-entropy loss function. Based on the recognition result label y (where y... (1 represents a keyword, 0 represents a non-keyword) and the confidence score of the predicted recognition result output by the model. The decision loss (loss3) is calculated using the following formula: (a value between 0 and 1) ].

[0193] Secondly, the parameters of the feature encoder are updated by combining the decision loss (loss3) and the spatial alignment loss (loss2).

[0194] The spatial alignment loss2 is calculated by comparing the fused feature representation output by the feature encoder (after processing by the alignment module) with the text phoneme features.

[0195] In a preferred embodiment, the spatial alignment loss2 can be calculated using the mean squared error loss function (MSELoss) to supervise the alignment of the fused audio-visual representation with the text phoneme features at the phoneme level, and can be obtained by the following formula: ; in, Let t be the t-th frame of the merged audio and video representation. Let M represent the text phoneme features of the nth phoneme, where M represents the spatial mapping network, w represents the weights of the Gaussian-weighted strategy, and N is the total number of phonemes.

[0196] The gradient of loss2 directly affects the feature encoder, but since the output of the feature encoder also continues to propagate forward, ultimately affecting the calculation of the decision loss (loss3), the gradient of the decision loss (loss3) also propagates back to the feature encoder. Therefore, the parameter update of the feature encoder is simultaneously affected by the gradients of both the spatial alignment loss (loss2) and the decision loss (loss3).

[0197] Next, the parameters of the mask generation network are updated by combining the decision loss (loss3), spatial alignment loss (loss2), and noise reduction loss (loss1).

[0198] The noise reduction loss1 is calculated directly based on the output of the mask generation network, which is the core of the noise reduction module.

[0199] In a preferred embodiment, the noise reduction loss can be calculated using the Scale-Invariant Signal-to-Noise Ratio Improvement Loss (SISNR). In the time domain, the noise-reduced speech training signal is compared with the clean audio signal labels of the training samples, and the noise reduction loss1 can be obtained by the following formula: ; Where s is the projection component of the denoised speech training signal onto the clean audio signal label direction, and e is the noise component.

[0200] The gradient of the denoising loss (loss1) directly affects the mask generation network. Simultaneously, the output of the denoising module serves as the input to all subsequent modules. Therefore, the gradients of the spatial alignment loss (loss2) and the decision loss (loss3) will also eventually propagate back to the mask generation network. Thus, the parameter updates of the mask generation network are simultaneously influenced by the gradients of all three losses: denoising loss (loss1), spatial alignment loss (loss2), and decision loss (loss3).

[0201] This invention decomposes the parameter update process into asymmetric updates of different modules in the keyword recognition model. This accurately reflects the differentiated optimization effects of different loss functions on different modules in the joint training framework, ensuring that each module can be effectively optimized based on its directly related loss. Furthermore, through the cascading propagation of gradients, upstream modules can simultaneously perceive and respond to the performance of downstream tasks, thereby achieving end-to-end, truly collaborative optimization of the entire model. Ultimately, this results in a keyword recognition model with highly coordinated functions and powerful overall performance.

[0202] Figure 6 This is a schematic diagram of the keyword recognition device provided by the present invention, as shown below. Figure 6 As shown, it mainly includes, but is not limited to, the following components: The feature extraction unit 11 is used to determine a set of available modality pairs based on at least one registration modality feature obtained under a preset registration modality and at least one query modality feature obtained under a query modality; the modality pairs in the set of available modality pairs are combinations of any query modality feature and any registration modality feature. Feature analysis unit 12 is used to obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs; Feature aggregation unit 13 is used to aggregate all the hidden state vectors to generate an aggregated feature vector; The result recognition unit 14 is used to determine the recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of preset keywords.

[0203] It should be noted that the keyword recognition device provided by the present invention can execute the keyword recognition method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0204] The keyword recognition device provided by this invention solves the problem of drastic performance degradation or failure in modality missing scenarios by constructing a recognition framework that can dynamically determine modality pairs and aggregate their corresponding features based on the actual available registration and query modality combinations. It achieves flexible compatibility and adaptive fusion of different modality inputs, and significantly enhances the generalization ability, robustness and practicality of the keyword recognition system in complex and ever-changing real-world application scenarios.

[0205] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a keyword recognition method. This method includes: determining a set of available modality pairs based on at least one preset registration modality feature and at least one query modality feature; the modality pairs in the set of available modality pairs are combinations of any query modality feature and any registered modality feature; obtaining the hidden state vector corresponding to each modality pair in the set of available modality pairs; aggregating all the hidden state vectors to generate an aggregated feature vector; and determining a recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains a preset keyword.

[0206] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the keyword recognition method provided in the above embodiments, the method including: determining an available modality pair set based on at least one preset registration modality feature and at least one query modality feature; the modality pair in the available modality pair set is a combination of any query modality feature and any registered modality feature; obtaining the hidden state vector corresponding to each modality pair in the available modality pair set; aggregating all the hidden state vectors to generate an aggregated feature vector; determining the recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains a preset keyword recognition result.

[0208] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the keyword recognition method provided in the above embodiments. The method includes: determining an available modality pair set based on at least one preset registration modality feature and at least one query modality feature; the modality pair in the available modality pair set is a combination of any query modality feature and any registered modality feature; obtaining a hidden state vector corresponding to each modality pair in the available modality pair set; aggregating all the hidden state vectors to generate an aggregated feature vector; and determining a recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains a preset keyword.

[0209] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A keyword recognition method, characterized in that, include: Based on the registration modality features obtained under at least one pre-defined registration modality and the query modality features obtained under at least one query modality, a set of available modality pairs is determined; the modality pairs in the set of available modality pairs are combinations of any of the query modality features and any of the registration modality features. Obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs; Aggregate all the hidden state vectors to generate an aggregated feature vector; Based on the aggregated feature vector, the recognition result is determined; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of preset keywords.

2. The keyword recognition method according to claim 1, characterized in that, The registration modality includes at least one of text registration, audio registration, and video registration; the query modality includes at least one of voice query and video query. When the query modality includes both voice query and video query, the voice query and the video query occur simultaneously.

3. The keyword recognition method according to claim 2, characterized in that, If the query modality is determined to include a voice query, before determining the set of available modality pairs, the following steps are further performed iteratively until a first preset cutoff condition is met: The query voice signal in the query input is segmented at the frame level to obtain the query audio frame-level representation sequence; A noise reduction mask is determined based on the correlation calculation between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal of the video query and / or at least one registered modality signal in the registered modalities; The noise reduction mask is used to reduce the noise in the query speech signal.

4. The keyword recognition method according to claim 3, characterized in that, Determining a noise reduction mask includes: The cross-attention calculation results between each query audio frame-level representation in the query audio frame-level representation sequence and the query video signal are calculated respectively to create a video association attention output sequence; The cross-attention calculation result between each query audio frame-level representation in the query audio frame-level representation sequence and each registered modality signal is calculated respectively to create a registration-related attention output sequence corresponding to each registered modality signal; A corresponding modality mask is determined based on the video-associated attention output sequence and / or each of the registered-associated attention output sequences; The noise reduction mask is generated by weighted fusion of all modal masks.

5. The keyword recognition method according to claim 4, characterized in that, The step of determining a corresponding modality mask based on the video-associated attention output sequence and / or each of the registered associated attention output sequences includes: The video-associated attention output sequence and / or the registered-associated attention output sequence are respectively input into a mask generation network to obtain the modal mask output by the mask generation network; The mask generation network includes convolutional layers and nonlinear activation function layers. The convolutional layers are used to extract features from the video-associated attention output sequence or the registered-associated attention output sequence to generate a feature matrix. The nonlinear activation function layers are used to perform element-wise probability operations on the feature matrix to generate the modality mask.

6. The keyword recognition method according to claim 1, characterized in that, The step of obtaining the hidden state vector corresponding to each mode pair in the set of available mode pairs includes: Determine a similarity matrix between the query modality feature and the registered modality feature in each modality pair, wherein the similarity matrix characterizes the degree of matching between the query modality feature and the registered modality feature at different time points; The similarity matrix is ​​input into a recurrent neural network to obtain the hidden state vector corresponding to each modality pair output by the recurrent neural network.

7. The keyword recognition method according to claim 1, characterized in that, The aggregation of all the hidden state vectors to generate an aggregated feature vector includes: The hidden state vectors corresponding to all the modality pairs in the available modality pair set are concatenated to generate the aggregated feature vector.

8. The keyword recognition method according to any one of claims 1-7, characterized in that, The process of determining the recognition result based on the aggregated feature vector includes: The aggregated feature vector is input into a classifier network to obtain the recognition result output by the classifier network.

9. The keyword recognition method according to claim 8, characterized in that, Also includes: Collect a training sample set, which includes multiple sets of training samples. Each set of training samples includes at least one registration input sample under a registration modality, at least one query input sample under a query modality, and pre-labeled recognition result labels and clean audio signal labels. The training process of a keyword recognition model to be trained is iteratively executed using the training sample set until the second preset cutoff condition is met. In each training process, a batch of training samples is randomly selected from the training sample set, and the following training steps are performed until all training samples in the batch are traversed: A set of training samples selected from the batch is used as the input of the keyword recognition model to be trained, and the predicted recognition result output by the keyword recognition model to be trained, as well as the noise-reduced speech training signal and frame-level audio-video fusion vector sequence generated in the process are obtained. Calculate the decision loss between the predicted recognition result and the recognition result label of the training sample; Calculate the noise reduction loss between the denoised speech training signal and the clean audio signal labels of the training samples; After mapping the frame-level audio-video fusion vector sequence to the feature space where the text phoneme features in the training samples are located, a weighted processing based on a Gaussian distribution centered on the time window is performed to generate a phoneme-level fusion feature representation, and to calculate the spatial alignment loss between the fusion feature representation and the text phoneme features. The parameters of the keyword recognition model to be trained are updated based on the decision loss, the noise reduction loss, and the spatial alignment loss. The keyword recognition model to be trained includes at least a feature encoder, a mask generation network, a recurrent neural network, and a classifier network.

10. The keyword recognition method according to claim 9, characterized in that, After randomly selecting a batch of training samples from the training sample set during each training process, the method further includes: Randomly remove training samples from the batch that are associated with at least one registered modality and / or at least one query modality.

11. The keyword recognition method according to claim 9, characterized in that, The frame-level audio-video fusion vector sequence is determined based on the following method: After swapping the audio feature sequences corresponding to the audio signal samples and the video feature sequences corresponding to the video signal samples in the training samples with a preset probability at at least one time frame; The frame-level audio-video fusion vector sequence is obtained by fusing the audio signal sample and the video signal sample after feature swapping.

12. The keyword recognition method according to claim 9, characterized in that, The step of updating the parameters of the keyword recognition model to be trained based on the decision loss, the noise reduction loss, and the spatial alignment loss includes: The parameters of the mask generation network are updated by combining the decision loss, the spatial alignment loss, and the noise reduction loss. The parameters of the feature encoder are updated by combining the decision loss and the spatial alignment loss. The parameters of the classifier network and the recurrent neural network are updated based on the decision loss.

13. The keyword recognition method according to claim 9, characterized in that, Based on the decision loss, the noise reduction loss, and the spatial alignment loss, the parameters of the keyword recognition model to be trained are updated, including: The decision loss, the noise reduction loss, and the spatial alignment loss are weighted and summed to obtain the combined loss; The parameters of the mask generation network of the keyword recognition model to be trained are updated based on the combined loss. The weight coefficients of the weighted summation are dynamically determined based on the gradient changes of the decision loss, the noise reduction loss, and the spatial alignment loss during the historical training process.

14. The keyword recognition method according to claim 13, characterized in that, Also includes: Based on the combined loss, update the weight coefficients of the weighted fusion of all modal masks to generate the denoising mask.

15. The keyword recognition method according to claim 9, characterized in that, The decision loss is the binary cross-entropy loss, the noise reduction loss is the scale-invariant signal-to-noise ratio improvement loss, and the spatial alignment loss is the mean square error loss.

16. A keyword recognition device, characterized in that, include: The feature extraction unit is used to determine a set of available modality pairs based on at least one registration modality feature obtained under a preset registration modality and at least one query modality feature obtained under a query modality; the modality pairs in the set of available modality pairs are combinations of any of the query modality features and any of the registration modality features. The feature analysis unit is used to obtain the hidden state vector corresponding to each mode pair in the set of available mode pairs; The feature aggregation unit is used to aggregate all the hidden state vectors to generate an aggregated feature vector. The result recognition unit is used to determine the recognition result based on the aggregated feature vector; the recognition result is used to characterize whether the query input corresponding to any query modality feature contains the recognition result of preset keywords.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the keyword recognition method as described in any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the keyword recognition method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the keyword recognition method as described in any one of claims 1 to 15.

Images