Intelligent sound pickup and speech recognition system based on multimodal fusion

Abstract

This invention discloses an intelligent sound pickup and speech recognition system based on multimodal fusion, belonging to the interdisciplinary field of artificial intelligence and speech recognition. The system includes a main control module, multiple sound pickup nodes, and a multimodal fusion engine. The multimodal fusion engine comprises four core components: sound source localization and separation, environmental adaptive noise reduction, cross-modal feature fusion, and dynamic contextual understanding. By acquiring multimodal data through an array of microphones and auxiliary sensor groups, the system achieves sound source localization and separation, dynamic environmental noise suppression, deep fusion of multimodal features, and contextual semantic correction. This invention effectively improves the robustness, accuracy, and intelligent interaction capabilities of speech recognition, enhances the voice interaction experience in complex scenarios such as noisy environments and accent variations, and provides a more reliable voice processing solution for intelligent voice interaction devices.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and speech recognition, specifically involving an intelligent sound pickup and speech recognition system based on multimodal fusion. Background Technology

[0002] With the development of artificial intelligence, voice interaction has become the core interaction method for smart devices, and its demand continues to grow in consumer electronics, industrial control, public services, and other fields due to its natural and convenient advantages. However, in real-world scenarios, problems such as complex environmental interference, overlapping multiple sound sources, and differences in user characteristics lead to a significant decline in the performance of traditional voice recognition systems, making it difficult to meet the requirements of high robustness and high accuracy. This has become a key bottleneck restricting the widespread adoption of intelligent voice technology. The specific shortcomings of existing technologies are as follows:

[0003] Limitations of speech signal acquisition and sound source processing technologies: The design of pickup nodes is simple, with most using a single microphone or a simple linear array, lacking distributed deployment capabilities and unable to cover complex spaces. This leads to speech signal attenuation and increased interference in scenarios where the target speaker is far from the device or in multi-sound-source scenarios, making effective speech extraction difficult. Traditional sound source localization and separation technologies are prone to localization errors and poor separation effects in non-stationary noise and multi-sound-source overlapping scenarios, resulting in excessively low input signal-to-noise ratios for subsequent recognition models.

[0004] Noise suppression technology lacks dynamic adaptability: Traditional noise reduction relies on the assumption of "noise stationarity" and fixed noise spectrum estimation. When faced with non-stationary noise, noise residue or loss and distortion of speech details are likely to occur. Noise reduction parameters are mostly preset manually or only support limited scene switching. They cannot be dynamically adjusted according to real-time noise type and intensity, resulting in extremely poor versatility and adaptability.

[0005] The robustness defects of single-modal recognition and the inadequacy of cross-modal fusion: relying solely on the speech modality, the recognition rate drops sharply when affected by noise, accent, and transmission loss; a few multimodal systems simply use feature splicing or fixed weighting, which cannot explore the intrinsic correlation between modalities, make it difficult to utilize the complementarity of multimodalities, and even cause the fusion effect to be worse than that of single-modal recognition due to interference from redundant information.

[0006] Lack of contextual understanding and intelligent interaction capabilities: Most systems only utilize short-term context, ignoring mid-term conversation topics and long-term user preferences, making it difficult to resolve identification ambiguities; intent recognition relies on keyword matching, without combining domain knowledge graphs and historical interaction habits, making it difficult to identify complex intents; responses are mostly based on fixed templates, unable to be dynamically generated according to real-time context and user intent, resulting in low interaction efficiency and poor experience.

[0007] Therefore, this invention proposes an intelligent sound pickup and speech recognition system based on multimodal fusion. Summary of the Invention

[0008] To overcome the shortcomings and deficiencies of the existing technology, the present invention adopts the following technical solution:

[0009] The intelligent sound pickup and speech recognition system based on multimodal fusion includes a main control module, multiple distributed sound pickup nodes, a communication module, and a multimodal fusion engine.

[0010] The main control module is deployed on a device with strong computing power. It contains a multimodal data coordinator, a recognition result fusion unit, and a system parameter optimizer, which are used for collaborative processing of multimodal data, comprehensive decision-making of recognition results, and global optimization of system parameters.

[0011] Each pickup node includes an array microphone, an auxiliary sensor group, and a local preprocessing unit. The array microphone is used to collect speech signals from different locations in space to support sound source localization and beamforming. The auxiliary sensor group includes an infrared sensor, a vibration sensor, and an environmental noise sensor, which are used to detect the speaker's position and activity status, collect solid-conducted sound, and analyze noise characteristics, respectively. The local preprocessing unit is used to amplify, filter, perform A / D conversion, and compress the collected raw signal.

[0012] The communication module uses wired or wireless communication technology to transmit multimodal data and control commands between the pickup node and the main control module;

[0013] The multimodal fusion engine includes a sound source localization and separation engine, an environment-adaptive noise reduction engine, a cross-modal feature fusion engine, and a dynamic context understanding engine, which are used to deeply fuse multimodal information to improve the robustness and accuracy of speech recognition.

[0014] Preferably, the sound source localization and separation engine is configured as follows:

[0015] The system receives multi-channel speech signals from an array of microphones. First, it preprocesses the signals by unifying the sampling rate, removing DC offset, and normalizing the energy. Then, it uses a generalized cross-correlation algorithm to calculate the signal delay between different microphones. Based on the geometry of the microphone array and the estimated delay, it calculates the spatial coordinates of the sound sources using maximum likelihood estimation. Based on the localization results, it uses an adaptive beamforming algorithm to construct a spatial filter, enhancing the speech signal in the target direction and suppressing interference from other directions. For multiple overlapping sound sources, it further separates them using an independent component analysis method based on a deep neural network, obtaining the separated signals for each sound source.

[0016] Preferably, the environment-adaptive noise reduction engine is configured as follows:

[0017] The statistical characteristics of environmental noise are analyzed in real time using an auxiliary sensor array and voice activity detection technology. When a pure noise segment is detected, the noise power spectrum estimate is updated.

[0018] An improved spectral subtraction method is used for initial denoising of noisy speech. The denoising effect is further optimized by an attention-based encoder-decoder network. The encoder of the encoder-decoder network extracts the time-frequency features of noisy speech through convolutional layers and LSTM layers. The attention layer automatically learns the attention at different frequencies and time points to retain speech features. The decoder maps the encoded features back to the speech spectrum to generate denoised speech. The network training is aimed at the weighted sum of logarithmic spectral distance loss and structural similarity index loss.

[0019] The noise reduction parameters are dynamically adjusted based on the type and intensity of environmental noise.

[0020] Spectral smoothing and phase reconstruction techniques are used for speech enhancement post-processing.

[0021] Preferably, the cross-modal feature fusion engine is configured as follows:

[0022] Extracting multimodal features includes extracting speech features such as Mel frequency cepstral coefficients, perceptual linear prediction, and Mel spectrograms from preprocessed speech signals; extracting visual features of speaker location and activity status from infrared sensor data; extracting environmental features such as noise type, intensity, and temperature from environmental sensors; and extracting contextual features of user habits and dialogue topics from historical interactions.

[0023] Features from different modalities are temporally aligned and scaled; features from each modality are enhanced using a self-attention mechanism.

[0024] A cross-attention mechanism is used to learn the associations between features of different modalities;

[0025] Each modal feature is treated as a graph node to construct an intermodal relationship graph, which is then deeply fused using a graph neural network.

[0026] Finally, the contribution of each modality feature is dynamically adjusted through a gating mechanism to obtain the final fused feature.

[0027] Preferably, the dynamic context understanding engine is configured as follows:

[0028] Construct a multi-level context model that includes short-term context, medium-term context, and long-term context. The short-term context is the content of the most recent 3-5 rounds of dialogue, the medium-term context is the topic and goal of the current session, and the long-term context is the user's historical preferences, habits, and background knowledge.

[0029] A pre-trained language model is used to convert speech recognition results and contextual information into vector representations;

[0030] By calculating the semantic similarity between candidate recognition results and the context, recognition ambiguities are detected and resolved;

[0031] Identify the user's true intent based on semantic representations that incorporate context;

[0032] The recognition results are semantically expanded and corrected using a domain knowledge graph.

[0033] Generate appropriate system responses based on recognition results, user intent, and contextual information; integrate current interaction information into the context model and manage context length through a sliding window mechanism.

[0034] Preferably, the multimodal data coordinator in the main control module is responsible for receiving and integrating multimodal data from different pickup nodes, and managing the data transmission timing to ensure the consistency of each modal data in the time dimension; the recognition result fusion unit performs weighted fusion of results from different recognition engines, selects the optimal recognition result based on confidence evaluation, and handles recognition ambiguity and conflict; the system parameter optimizer dynamically adjusts the pickup sensitivity, noise reduction intensity, and recognition model parameters according to environmental characteristics, user characteristics, and recognition performance indicators to achieve the best recognition effect.

[0035] Preferably, the communication module is adapted to different application scenarios and can adopt wired Ethernet, Wi-Fi, Bluetooth or low power wide area network technology. The transmitted content includes multimodal acquisition data, recognition results, control commands and parameter configuration information.

[0036] Preferably, in the speech recognition stage, the main control module inputs cross-modal fusion features into the CTC or attention mechanism-based recognition model to obtain preliminary recognition results; after correction by the dynamic context understanding engine, the recognition result fusion unit outputs the optimal recognition result; the system parameter optimizer updates the parameters of the noise reduction model and the recognition model by analyzing the recognition accuracy and confidence index, and saves the user interaction history and preferences to optimize subsequent interactions.

[0037] Preferably, the device with strong computing power is a smart speaker, an edge computing device, or a cloud server.

[0038] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0039] 1. This invention achieves dynamic noise suppression and multi-source separation through a sound source localization and separation engine and an environment-adaptive noise reduction engine. Even in non-stationary noise and overlapping speech scenarios, the system can accurately locate sound sources and effectively separate target speech from interfering sound sources. This solves the dual dilemma of "difficult acquisition - noise interference" faced by traditional single-modal systems in complex environments, improving the purity and recognizability of speech signals.

[0040] 2. This invention employs a three-layer fusion architecture of self-attention, cross-attention, and graph neural networks through a cross-modal feature fusion engine to achieve deep fusion of speech, vision, environmental, and contextual features. Through intra-modal enhancement and cross-modal association modeling, the system improves recognition accuracy in complex scenarios such as accent variations, dialect mixing, and noise interference, effectively overcoming the limitations of single-modal information and enhancing the generalization ability and robustness of the technical solution.

[0041] 3. This invention constructs a three-level context model (short-term, medium-term, and long-term) through a dynamic context understanding engine. Combined with a pre-trained language model and domain knowledge graph, it achieves ambiguity resolution, accurate intent recognition, and dynamic response generation. Through semantic similarity calculation and sliding window context management, the system can dynamically correct recognition errors in multi-turn dialogues, understand the user's true intent, and generate system responses that meet the needs of the scenario. This improves the fluency, accuracy, and user experience of intelligent interaction, satisfying the needs of all scenarios from consumer-grade smart speakers to industrial-grade voice control. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 The block diagram of the intelligent sound pickup and speech recognition system based on multimodal fusion of the present invention is shown.

[0044] Figure 2 The flowchart of the sound source localization and separation engine of the present invention is shown;

[0045] Figure 3 A flowchart of the operation of the system of the present invention is shown. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0048] Example 1:

[0049] See Figure 1 As shown, the intelligent voice pickup and speech recognition system based on multimodal fusion in this embodiment aims to solve problems such as difficulty in acquiring voice signals in complex environments, severe noise interference, and low accuracy of single-modal recognition. The system includes:

[0050] Main control module: As the core processing unit of the entire system, it is responsible for the collaborative processing of multimodal data, comprehensive decision-making of recognition results, and global optimization of system parameters. The main control module is deployed on devices with strong computing capabilities, such as smart speakers, edge computing devices, or cloud servers. The main control module includes a multimodal data coordinator, a recognition result fusion unit, and a system parameter optimizer.

[0051] Multimodal data coordinator: Responsible for receiving and integrating multimodal data from different pickup nodes, managing data transmission timing, and ensuring consistency of each modal data in the time dimension.

[0052] Recognition Result Fuder: It performs weighted fusion of results from different recognition engines, selects the optimal recognition result based on confidence evaluation, and handles recognition ambiguity and conflict.

[0053] System parameter optimizer: Based on environmental characteristics, user characteristics, and recognition performance indicators, it dynamically adjusts various system parameters, such as sound pickup sensitivity, noise reduction intensity, and recognition model parameters, to achieve the best recognition effect.

[0054] Multiple microphone nodes: Distributed throughout the voice acquisition area, responsible for the acquisition and preliminary processing of multimodal signals. Each microphone node includes an array microphone and an auxiliary sensor group.

[0055] Array microphone: An array of multiple microphones used to collect voice signals from different locations in space, supporting sound source localization and beamforming.

[0056] Auxiliary sensor group: including infrared sensor (for detecting speaker position and activity status), vibration sensor (for collecting solid-conducted sound), environmental noise sensor (for noise feature analysis), etc., to provide multimodal environmental information.

[0057] Local preprocessing unit: used to perform preliminary processing on the acquired raw signals, including signal amplification, filtering, A / D conversion and compression, to reduce transmission bandwidth requirements.

[0058] Communication module: Used to transmit multimodal data and control commands between the pickup node and the main control module. Depending on the application scenario, wired Ethernet, Wi-Fi, Bluetooth, or low-power wide area network (LPWAN) communication technologies can be used.

[0059] Multimodal fusion engine: This is one of the core innovations of this invention, responsible for deeply fusing information from different modalities to improve the robustness and accuracy of speech recognition. The multimodal fusion engine includes the following key components:

[0060] Sound source localization and separation engine: Configured with time delay estimation (TDE) and independent component analysis (ICA) technologies, this engine enables the localization and separation of multiple sound sources in complex environments. Based on the signal differences collected by the microphone array, the engine calculates the azimuth and distance of the sound sources and separates the target speech from interfering sound sources, providing a clean speech signal for subsequent recognition.

[0061] The environmental adaptive noise reduction engine is configured to combine spectral subtraction, wavelet thresholding, and deep neural network noise reduction techniques to achieve noise suppression in dynamic environments. This engine can analyze environmental noise characteristics in real time and adaptively adjust noise reduction parameters, effectively removing noise while preserving speech details, thus solving the problem of poor performance of traditional noise reduction methods in non-stationary noise environments.

[0062] Cross-modal feature fusion engine: Configured to achieve deep fusion of speech features with other modal features using attention mechanisms and graph neural networks (GNNs). This engine can automatically learn the correlation weights between features of different modalities, focusing on features relevant to speech recognition, suppressing irrelevant interference information, and improving recognition robustness in complex environments.

[0063] Dynamic Context Understanding Engine: Configured based on pre-trained language models and knowledge graphs, this engine enables dynamic modeling and semantic understanding of dialogue context. It combines user history, domain knowledge, and current scene information to perform semantic correction and intent prediction on the recognition results, resolving ambiguity issues in speech recognition and enhancing the system's intelligent interaction capabilities.

[0064] The beneficial effects of this embodiment are as follows: by multimodal data collaborative processing and fusion, the problems of difficult speech acquisition, severe noise interference and low accuracy of single modality recognition in complex environments are solved, the robustness, accuracy and intelligent interaction capabilities of speech recognition are improved, and adaptive noise reduction, sound source localization and separation, deep fusion of cross-modal features and contextual semantic understanding are realized in dynamic environments.

[0065] Example 2:

[0066] Reference Figure 2 The specific configuration and working principle of the sound source localization and separation engine in this invention are as follows:

[0067] The sound source localization and separation engine first receives multi-channel speech signals from the array microphones, and then performs sound source localization and separation through the following steps:

[0068] S11. Signal preprocessing: Perform sampling rate unification, DC offset removal, and energy normalization on the signals of each channel to ensure consistency in subsequent processing.

[0069] S12. Delay Estimation: The generalized cross-correlation (GCC) algorithm is used to calculate the signal delay between different microphones. Wherein, GCC(t) is the generalized cross-correlation function, used to characterize the correlation between two microphone signals x(t) and y(t) over time t, and its peak position corresponds to the signal delay; Let f be the cross-power spectral density of the two microphone signals x(t) and y(t), describing the cross-correlation characteristics of the signals in the frequency domain. f is the frequency variable; t is the time variable.

[0070] S13. Sound source localization: Based on the geometry of the microphone array and the estimated time delay, the spatial coordinates of the sound source are calculated using the maximum likelihood estimation method. ;in, The estimated spatial coordinates of the sound source (including azimuth and elevation angles); The azimuth and elevation angles (spatial parameters) of the sound source to be estimated. , ,..., M is the time delay of each microphone relative to the reference microphone (M is the total number of microphones). It is the likelihood function, which represents the location of a given sound source. At that time, various time delays were observed. , ,..., The probability of.

[0071] S14. Beamforming: Based on the positioning results, an adaptive beamforming algorithm (such as an MVDR beamformer) is used to construct a spatial filter to enhance the speech signal in the target direction and suppress interference from other directions. Where w is the weight vector of the beamformer, used for weighted summation of multi-microphone signals; It is the noise covariance matrix, which describes the statistical properties of noise across multiple microphone channels; For the target direction The steering vector represents the phase / amplitude response of the signal as it arrives at each microphone from the target direction; H is the conjugate transpose operator; This indicates finding the minimum value of the weight vector w; The objective function is denoted as .

[0072] S15. Sound Source Separation: For multiple overlapping sound sources, the Independent Component Analysis (ICA) method based on deep neural networks is used for further separation: s=Wx; where x is the mixed signal collected by the microphone array; W is the separation matrix, which is obtained through an optimization algorithm and used to decompose the mixed signal; s is the signal of each sound source after separation, and the separation matrix W is optimized by minimizing the mutual information of the signals or maximizing the non-Gaussianity.

[0073] Preferably, the specific configuration and working principle of the environment-adaptive noise reduction engine in this invention are as follows: The core of the environment-adaptive noise reduction engine is to achieve noise suppression in dynamic environments while preserving the integrity of the speech signal to the greatest extent. Its workflow includes:

[0074] S21. Noise Characterization: Statistical characteristics of ambient noise are analyzed in real time using an auxiliary sensor array and Voice Activity Detection (VAD) technology. When a pure noise segment is detected (VAD=0), update the noise power spectrum estimate. Wherein, VAD(k) is the speech activity detection result of the k-th frame; "1" indicates speech is present, "0" indicates pure noise; and k is the frame number.

[0075] S22. Initial noise reduction processing: An improved spectral subtraction method is used to perform initial noise reduction on the noisy speech. in, The pure speech power spectrum estimated at frequency f in the k-th frame; Let f be the power spectrum of the noisy speech at frequency f in the k-th frame. α is the noise power spectrum estimated at frequency f in the k-th frame; α is the over-subtraction factor, which controls the noise suppression strength; β is the lower limit factor, which prevents excessive speech distortion and ensures that the estimated clean speech power spectrum is not less than β times the noisy speech power spectrum.

[0076] S23. Deep Neural Network Denoising: This method further optimizes noise reduction by utilizing an attention-based encoder-decoder network. The network structure includes: Encoder: Extracts time-frequency features of noisy speech through convolutional and LSTM layers; Attention Layer: Automatically learns the attention levels at different frequencies and time points, focusing on preserving speech features; Decoder: Maps the encoded features back to the speech spectrum to generate the denoised speech. The network training objective is to minimize the loss function. The first term is the log-spectral distance loss; the second term is the structural similarity index (SSIM) loss. The power spectrum of the real, clean speech at frequency f in the k-th frame; Power spectrum of clean speech estimated by a deep neural network; It is a structural similarity index that measures the structural similarity between the power spectra of real and estimated clean speech. is the weighting coefficient, balancing the proportion of logarithmic spectral distance loss to SSIM loss.

[0077] S24. Adaptive adjustment of noise reduction parameters: Dynamically adjusts noise reduction parameters according to the type and intensity of ambient noise. ;in, It is the basic over-subtraction factor, and γ is the adjustment coefficient. and These are the noise and speech power estimates for the current frame, respectively. The stronger the noise, the larger the over-subtraction factor, and the stronger the noise reduction effect.

[0078] This is the over-subtraction factor for the k-th frame, which is dynamically adjusted according to the noise intensity. γ is the base over-attenuation factor (the default value when the noise intensity is zero); γ is an adjustment coefficient that controls the magnitude of the over-attenuation factor as a function of the noise-to-speech power ratio. This is the noise power estimate for the k-th frame; This is the estimated speech power value for the k-th frame.

[0079] S25. Post-processing of speech enhancement: Spectral smoothing and phase reconstruction techniques are used to reduce music noise introduced during the noise reduction process and improve the naturalness and intelligibility of the speech.

[0080] Preferably, the specific configuration and working principle of the cross-modal feature fusion engine in this invention are as follows: The cross-modal feature fusion engine is responsible for deeply fusing speech features with other modal features (such as vision, environment, location, etc.) to improve the robustness of speech recognition. Its workflow includes:

[0081] S31. Multimodal Feature Extraction: Speech Features: Extract Mel frequency cepstral coefficients (MFCC), perceptual linear prediction (PLP), and Mel spectrogram features from the preprocessed speech signal. Visual Features: Extract speaker location and activity status features from infrared sensor data. Environmental Features: Extract environmental features such as noise type, intensity, and temperature from environmental sensors. Context Features: Extract context features such as user habits and dialogue topics from historical interactions.

[0082] S32. Feature Alignment and Standardization: Perform time alignment and scale standardization on features from different modalities. ;in, It is the standardized value of the i-th feature in the m-th mode, and μm and σm are the mean and standard deviation of all features in the m-th mode, respectively; It is the i-th original feature value in the m-th mode.

[0083] S33. Intramodal Feature Enhancement: Features of each modality are enhanced using a self-attention mechanism. Where Q, K, and V are the query, key, and value matrices, respectively, obtained from the original features through linear transformation, and dk is the feature dimension.

[0084] Q is the query matrix, obtained by linear transformation of the original features, and is used to initiate the "query"; K is the key matrix, obtained by linear transformation of the original features, and is used to match the query matrix; V is the value matrix, obtained by linear transformation of the original features, and is the basis for the calculation of attention output; dk is the feature dimension of the key matrix K, used to scale the dot product result to avoid gradient problems.

[0085] S34. Cross-modal attention fusion: Employing a cross-attention mechanism to learn the associations between features from different modalities. ;in, Features of mode m; Features of mode n; , , It is a learnable weight matrix used to transform modal features into a query, key, and value matrix.

[0086] S35. Graph Neural Network Fusion: Treating each modal feature as a graph node, constructing an inter-modal relationship graph, and performing deep fusion through a graph neural network (GNN): ;in, For the feature σ of node i in the (l+1)th layer, an activation function (such as ReLU or Sigmoid) is used to introduce nonlinearity; Let be the weight matrix of the l-th layer, used for the linear transformation of node features; Let N(i) be the feature of node i in the l-th layer; N(i) is the set of neighbors of node i. is an element of the adjacency matrix, representing the association strength between nodes i and j.

[0087] S36. Feature Fusion Optimization: Dynamically adjust the contribution of each modality feature through a gating mechanism. ; ;in, The gating coefficient for mode m controls the contribution of this mode feature to the fused features; This is the weight matrix for the gating mechanism; The original features of mode m Compared with enhanced features splicing; This is the bias term for the gating mechanism; Features of mode m after attention enhancement; This is the final fusion feature.

[0088] Through the aforementioned multi-layered fusion mechanism, the cross-modal feature fusion engine can fully utilize the complementarity between different modal information to effectively improve the recognition accuracy of speech recognition systems in complex scenarios such as noisy environments, accent variations, and dialect mixing.

[0089] Preferably, the specific configuration and working principle of the dynamic context understanding engine in this invention are as follows: The core function of the dynamic context understanding engine is to combine contextual information to perform semantic correction and intent understanding on the speech recognition results, thereby resolving ambiguity issues in speech recognition. Its workflow includes:

[0090] S41. Context Modeling: Construct a multi-level context model, including: Short-term context: content of the most recent 3-5 rounds of dialogue; Mid-term context: topic and goal of the current session; Long-term context: user's historical preferences, habits and background knowledge.

[0091] S42. Semantic Representation: Pre-trained language models (such as BERT, GPT, etc.) are used to convert speech recognition results and contextual information into vector representations. ;in, The output vector at the [CLS] position in the BERT model serves as a semantic representation. [CLS] is a pre-trained language model (such as a bidirectional Transformer encoder) used for text semantic vectorization; [CLS] is the classification label of BERT, whose output is used for overall semantic representation. This is the current speech recognition result; [SEP] is the BERT delimiter used to separate different text segments. This is contextual information.

[0092] S43. Ambiguity Detection and Resolution: By calculating the semantic similarity between candidate recognition results and the context, ambiguities in the recognition are detected and resolved. ;in, It is a semantic vector of contextual information, and the candidate result with the highest similarity is selected as the optimal recognition result.

[0093] in, For candidate identification results Semantic similarity with context c; For candidate identification results semantic vector; The semantic vector of context information c; The magnitude of the vector is used to normalize the dot product result.

[0094] S44. Intent Recognition: Based on semantic representations that incorporate context, identify the user's true intent. ;in, y represents the predicted user intent category; y represents the intent category. For a given semantic representation When, the conditional probability of intention being y.

[0095] S45. Knowledge Graph Enhancement: Utilizing domain knowledge graphs to semantically expand and correct the recognition results. ;in, s is the recognition result enhanced by the knowledge graph; s is the original speech recognition result; G is the domain knowledge graph, which contains knowledge such as entities and relationships.

[0096] S46. Dynamic Response Generation: Based on the recognition results, user intent, and contextual information, generate an appropriate system response. ;in, The generated system response; ResponseGenerator is the response generation module (which can use template matching, search enhancement, or generative methods). The recognition result is enhanced by the knowledge graph; c represents the identified user intent; c represents contextual information.

[0097] S47. Context Update: Integrates the current interaction information into the context model to support the next round of interaction. The system manages context length through a sliding window mechanism to ensure efficient system operation. This is the updated context information (for use in the next round of interaction); The context update function integrates the current interaction information; This refers to the context information before the update. This is the current speech recognition result; This is the current system response.

[0098] Through the aforementioned dynamic contextual understanding mechanism, the system can better understand the user's real needs. Even if there are errors in the speech recognition results, it can be corrected through contextual information, significantly improving the fluency and accuracy of intelligent interaction.

[0099] The beneficial effects of this embodiment are as follows: by sound source localization and separation, environmental adaptive noise reduction, cross-modal feature fusion and dynamic context understanding engine, the robustness, accuracy and intelligent interaction capabilities of speech recognition in complex environments are improved. Dynamic noise suppression, deep fusion of multimodal information, contextual semantic correction and accurate intent recognition are achieved, which enhances the practicality of the system in complex scenarios and the user experience.

[0100] Example 3:

[0101] Reference Figure 3 As shown, the workflow of an intelligent voice pickup and speech recognition system based on multimodal fusion includes the following steps:

[0102] Step S101: Multimodal signal acquisition and preliminary processing. The pickup node acquires speech signals through an array microphone, and simultaneously acquires multimodal data such as environmental information and speaker status through an auxiliary sensor group. The local preprocessing unit amplifies, filters, and performs A / D conversion on the acquired signals, and transmits the processed data to the main control module through the communication module.

[0103] Step S102: Sound Source Localization and Separation. The sound source localization and separation engine processes the received multi-channel speech signals: it uses a generalized cross-correlation algorithm to calculate the signal delay between microphones; based on the delay information and the microphone array geometry, it estimates the spatial location of the sound sources; it applies adaptive beamforming technology to enhance the speech signal in the target direction; and it uses independent component analysis to separate multiple overlapping sound sources.

[0104] Step S103: Environmental Adaptive Noise Reduction. The environmental adaptive noise reduction engine performs noise reduction processing on the separated target speech: pure noise segments are identified through speech activity detection, noise feature estimation is updated, preliminary noise reduction is performed using an improved spectral subtraction method, and noise is further suppressed using a deep neural network based on an attention mechanism. The noise reduction parameters are dynamically adjusted according to changes in environmental noise to balance the noise reduction effect and speech distortion.

[0105] Step S104: Multimodal Feature Extraction and Fusion. The cross-modal feature fusion engine extracts and fuses multimodal features: it extracts acoustic features such as Mel spectrograms and MFCCs from the denoised speech signal, extracts environmental features and speaker state features from auxiliary sensor data, performs time alignment and standardization processing on different modal features, uses a self-attention mechanism to enhance key features within each modality, and achieves deep fusion of cross-modal features through cross-attention and graph neural networks.

[0106] Step S105: Speech Recognition and Context Understanding. The main control module uses the fused features for speech recognition: the fused features are input into a speech recognition model (such as CTC, Attention-based model) to obtain preliminary recognition results. The dynamic context understanding engine combines contextual information and domain knowledge to correct the recognition results. By calculating semantic similarity, recognition ambiguity is resolved, the optimal recognition result is determined, user intent is identified, and an appropriate system response is generated.

[0107] Step S106: System Optimization and Adaptive Adjustment. The system parameter optimizer adaptively adjusts based on recognition performance and environmental changes: it calculates performance indicators such as recognition accuracy and confidence, analyzes error patterns, determines the direction of system optimization, dynamically adjusts parameters such as the sensitivity and beam direction of the pickup nodes, updates the parameters of the noise reduction model and recognition model, adapts to environmental changes and user characteristics, saves user interaction history and preferences, and optimizes future recognition and response.

[0108] The beneficial effects of this embodiment are as follows: Through the above steps, the present invention can achieve high-quality voice acquisition and high-accuracy voice recognition in complex environments, effectively solve the problem of performance degradation of single-modal voice recognition systems in noisy environments, accent changes and other scenarios, and significantly improve the user experience of intelligent voice interaction.

[0109] All formulas in this invention are dimensionless and calculated numerically. The preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0110] The weighting coefficients of this invention are used to measure the degree of influence of different factors or variables on a certain outcome or decision. The weighting coefficient is defined as the numerical value assigned to each factor when comparing and evaluating multiple factors, reflecting their importance or priority. These weighting coefficients can be determined according to specific circumstances and needs, and are usually jointly formulated and confirmed by professionals or relevant stakeholders. By reasonably setting the weighting coefficients, programs or systems can be helped to make decisions or predictions more accurately.

[0111] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0112] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal fusion-based intelligent sound pickup and speech recognition system, characterized in that, It includes a main control module, multiple distributed pickup nodes, a communication module, and a multimodal fusion engine; The main control module contains a multimodal data coordinator, a recognition result fusion unit, and a system parameter optimizer, which are used for collaborative processing of multimodal data, comprehensive decision-making of recognition results, and global optimization of system parameters. Each pickup node includes an array microphone, an auxiliary sensor group, and a local preprocessing unit. The array microphone is used to collect speech signals from different locations in space to support sound source localization and beamforming. The auxiliary sensor group includes an infrared sensor, a vibration sensor, and an environmental noise sensor, which are used to detect the speaker's position and activity state, collect solid-conducted sound, and analyze noise characteristics, respectively. The local preprocessing unit is used to amplify, filter, perform A / D conversion, and compress the collected raw signal. The communication module uses wired or wireless communication technology to transmit multimodal data and control commands between the pickup node and the main control module; The multimodal fusion engine includes a sound source localization and separation engine, an environment-adaptive noise reduction engine, a cross-modal feature fusion engine, and a dynamic context understanding engine, which are used to deeply fuse multimodal information to improve the robustness and accuracy of speech recognition.

2. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The sound source localization and separation engine is configured as follows: The system receives multi-channel voice signals from array microphones. First, it performs preprocessing on the signals of each channel, including sampling rate unification, DC offset removal, and energy normalization. Then, it uses a generalized cross-correlation algorithm to calculate the signal delay between different microphones. Based on the geometry of the microphone array and the estimated time delay, the spatial coordinates of the sound source are calculated using the maximum likelihood estimation method. Based on the localization results, an adaptive beamforming algorithm is used to construct a spatial filter to enhance the speech signal in the target direction and suppress interference from other directions. For multiple overlapping sound sources, an independent component analysis method based on a deep neural network is used to further separate them, obtaining the separated sound source signals.

3. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The environment-adaptive noise reduction engine is configured as follows: The statistical characteristics of environmental noise are analyzed in real time using an auxiliary sensor array and voice activity detection technology. When a pure noise segment is detected, the noise power spectrum estimate is updated. An improved spectral subtraction method is used to perform preliminary denoising on noisy speech. The denoising effect is further optimized by using an attention-based encoder-decoder network. The encoder of the encoder-decoder network extracts the time-frequency features of noisy speech through convolutional layers and LSTM layers. The attention layer automatically learns the attention at different frequencies and time points to retain key speech features. The decoder maps the encoded features back to the speech spectrum to generate denoised speech. The network training is aimed at a weighted sum of logarithmic spectral distance loss and structural similarity index loss. The noise reduction parameters are dynamically adjusted based on the type and intensity of environmental noise. Spectral smoothing and phase reconstruction techniques are used for speech enhancement post-processing.

4. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The cross-modal feature fusion engine is configured as follows: Extracting multimodal features includes extracting speech features such as Mel frequency cepstral coefficients, perceptual linear prediction, and Mel spectrograms from preprocessed speech signals; extracting visual features of speaker location and activity status from infrared sensor data; extracting environmental features such as noise type, intensity, and temperature from environmental sensors; and extracting contextual features of user habits and dialogue topics from historical interactions. Features from different modalities are temporally aligned and scaled; features from each modality are enhanced using a self-attention mechanism. A cross-attention mechanism is used to learn the associations between features of different modalities; Each modal feature is treated as a graph node to construct an intermodal relationship graph, which is then deeply fused using a graph neural network. Finally, the contribution of each modality feature is dynamically adjusted through a gating mechanism to obtain the final fused feature.

5. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The dynamic context understanding engine is configured as follows: Construct a multi-level context model that includes short-term context, medium-term context, and long-term context. The short-term context is the content of the most recent 3-5 rounds of dialogue, the medium-term context is the topic and goal of the current session, and the long-term context is the user's historical preferences, habits, and background knowledge. A pre-trained language model is used to convert speech recognition results and contextual information into vector representations; By calculating the semantic similarity between candidate recognition results and the context, recognition ambiguities are detected and resolved; Identify the user's true intent based on semantic representations that incorporate context; The recognition results are semantically expanded and corrected using a domain knowledge graph. Generate appropriate system responses based on recognition results, user intent, and contextual information; integrate current interaction information into the context model and manage context length through a sliding window mechanism.

6. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The multimodal data coordinator in the main control module is responsible for receiving and integrating multimodal data from different pickup nodes, and managing the data transmission timing to ensure the consistency of each modal data in the time dimension; the recognition result fusion unit performs weighted fusion of results from different recognition engines, selects the optimal recognition result based on confidence evaluation, and handles recognition ambiguity and conflict. The system parameter optimizer dynamically adjusts the pickup sensitivity, noise reduction intensity, and recognition model parameters based on environmental characteristics, user characteristics, and recognition performance indicators to achieve the best recognition effect.

7. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, The communication module is adaptable to different application scenarios and can use wired Ethernet, Wi-Fi, Bluetooth or low-power wide area network technologies. The transmitted content includes multimodal acquisition data, recognition results, control commands and parameter configuration information.

8. The intelligent sound pickup and speech recognition system based on multimodal fusion according to claim 1, characterized in that, During the speech recognition stage, the main control module inputs cross-modal fusion features into the CTC or attention mechanism-based recognition model to obtain preliminary recognition results; after correction by the dynamic context understanding engine, the recognition result fusion unit outputs the optimal recognition result. The system parameter optimizer updates the parameters of the noise reduction model and the recognition model by analyzing the recognition accuracy and confidence index, and saves user interaction history and preferences to optimize subsequent interactions.

9. The intelligent sound pickup and speech recognition system based on multimodal fusion according to any one of claims 1-8, characterized in that, The main control module is deployed on a device with strong computing capabilities, such as a smart speaker, edge computing device, or cloud server.

Images