Intelligent loudspeaker voice interaction method

Abstract

The invention relates to the technical field of intelligent voice interaction, in particular to an intelligent loudspeaker voice interaction method, which comprises the steps of multi-mode audio signal acquisition, dynamic scene modeling, hierarchical feature extraction, noise suppression, low-rank tensor decomposition, adaptive weight distribution of an attention mechanism model and the like. A voice interaction response heat map is generated through multi-source heterogeneous feature fusion, and core indexes such as the interaction success rate and the response speed are quantitatively evaluated. According to the invention, user voice spectrum features, environmental noise distribution and space sound field information can be comprehensively obtained, the voice recognition precision and robustness in a complex scene are significantly improved, and the operation capability and the intelligent level of an intelligent loudspeaker in a dynamic environment are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent voice technology, specifically a voice interaction method for intelligent speakers. Background Technology

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, smart speakers, as an important voice interaction device, have been widely used in homes, offices, and public places. Through voice recognition and natural language processing technologies, they provide users with convenient functions such as information retrieval, device control, and entertainment playback, greatly improving the efficiency and experience of human-computer interaction.

[0003] However, existing smart speaker voice interaction methods still have many shortcomings in practical applications. For example, in complex acoustic environments, the accuracy of speech recognition is often significantly affected by background noise, reverberation, and other factors, leading to interaction failures or misoperations. Furthermore, existing technologies typically lack a deep understanding of user intent and struggle to perform accurate semantic parsing based on context, thus limiting the level of intelligence in the interaction. Simultaneously, current voice interaction systems have weak personalized service capabilities in multi-user scenarios, failing to effectively differentiate the needs of different users and provide targeted services. These problems not only degrade the user experience but also restrict the widespread application of smart speakers in more scenarios. Therefore, there is an urgent need for a smart speaker voice interaction method that can improve speech recognition accuracy, enhance semantic understanding capabilities, and support multi-user personalized services to address the shortcomings of existing technologies and meet users' growing demand for intelligent interaction. Summary of the Invention

[0004] To address the aforementioned technical challenges, a voice interaction method for smart speakers is proposed. This method first acquires multimodal audio signals to comprehensively obtain the spectral features of the user's speech, environmental noise distribution, and spatial sound field information. Combined with dynamic scene modeling, it achieves high-precision speech recognition and localization in complex backgrounds. Furthermore, by preprocessing the raw audio data through hierarchical feature extraction, noise suppression, and low-rank tensor decomposition, the representational capability of the speech signal is significantly improved. Based on this, an improved attention mechanism model is used to adaptively weight multidimensional features, enhancing the semantic expression of key speech segments. Finally, a low-rank tensor fusion strategy efficiently integrates multi-source heterogeneous features to generate an intuitive voice interaction response heatmap. Based on task requirements, core indicators such as interaction success rate and response speed are quantitatively evaluated. This achieves high-efficiency and high-precision prediction of smart speaker voice interaction behavior, significantly improving operational robustness and intelligence in complex scenarios.

[0005] According to one aspect of this application, a smart speaker voice interaction method is provided, comprising: performing multimodal microphone array calibration to obtain a unified transformation relationship between different microphone coordinate systems. Specifically, a calibration sound source with clearly defined acoustic markings is placed at the center of the workspace, and the microphone array is controlled to synchronously acquire the calibration sound source from at least three different angles to obtain multiple sets of local audio feature data. Subsequently, beamforming or sound source localization calculations are performed on each set of data to extract the key point coordinates of the calibration sound source at each acquisition position. Based on known microphone installation parameters and acquisition angles, the rotation and translation transformation matrices between each microphone coordinate system are accurately solved using maximum likelihood estimation or minimum acoustic error methods, thereby establishing a unified perception coordinate system.

[0006] Collect multimodal audio feature data of the target user. Place the user under test within a fixed area of ​​the speaker's operating range, ensuring their voice is within a detectable range. Set the microphone array parameters and activate the audio sensor and environmental noise monitoring module. While the speaker rotates at a constant speed, the sensor array continuously captures the spectral structure, sound field details, and background noise distribution information of the user's speech. Through a multi-channel signal synchronous acquisition system, record and integrate raw data from different sensors in real time to form a high-dimensional feature set covering the entire voice interaction process.

[0007] Optionally, the audio sensor may be a device based on a MEMS microphone array, or in another preferred embodiment, an active noise-canceling audio sensor may be used to cope with noisy environments, as its anti-interference characteristics are more adapted to high noise conditions and it can obtain higher resolution voice information.

[0008] The acquired raw multimodal audio data undergoes preprocessing. This step aims to eliminate redundant information and enhance data quality, laying the foundation for subsequent feature fusion and analysis. Preprocessing may include: noise suppression using adaptive filtering algorithms; hierarchical feature extraction, optionally applying multi-scale time-frequency pyramid structures from deep learning; and data dimensionality reduction using low-rank tensor decomposition methods to retain key feature components while reducing computational complexity.

[0009] The preprocessed multidimensional features are efficiently fused with task-driven prior knowledge. This step is the core of voice interaction, aiming to achieve adaptive integration of multi-source heterogeneous features through the synergy of low-rank tensor decomposition and attention mechanisms. First, a task-related prior knowledge base is loaded, including historical interaction data and statistical patterns of successful cases. Second, coarse-grained feature alignment is performed, either by manually annotating key speech segments or using an initial alignment algorithm based on time-frequency matching to determine the preliminary correspondences of each modality feature. Then, fine-grained feature optimization is performed. Building upon the coarse alignment, an attention mechanism model is introduced to dynamically adjust the weights of different feature channels, highlighting speech segments that play a crucial role in interaction decisions.

[0010] Based on the fusion results, a voice interaction response heatmap is generated and evaluated. The optimized multidimensional features are mapped to a unified time domain, and a confidence score is calculated for each potential interaction point. This score reflects the reliability of the point as an interaction candidate location. Positive values ​​indicate a high interaction success rate, while negative values ​​indicate a low interaction feasibility. The set of confidence scores for all points constitutes a high-density global interaction scoring field.

[0011] Based on this scoring field, a visual heatmap of voice interaction response can be generated. The calculated confidence scores for each point are displayed intuitively on a timeline using color gradient rendering, forming a heat map. Operators can clearly observe the optimal interaction periods and their priority ranking, enabling them to quickly formulate interaction strategies. It can also quantitatively evaluate interaction performance indicators. Based on industry standards or task requirements, key interaction evaluation parameters, such as response stability index, anti-interference ability, and task completion rate, are extracted and calculated from the scoring field data. Finally, a prediction report is output, automatically generating an analysis report that includes the heatmap, quantitative values ​​of key performance indicators, and recommended interaction schemes based on preset thresholds.

[0012] Compared with existing technologies, this application provides a smart speaker voice interaction method. First, it comprehensively acquires the spectral features of user speech, environmental noise distribution, and spatial sound field information through multimodal audio signal acquisition. Combined with dynamic scene modeling, it achieves high-precision speech recognition and localization in complex backgrounds. Furthermore, by preprocessing the raw data through hierarchical feature extraction, noise suppression, and low-rank tensor decomposition, the representational ability of the speech signal is significantly improved. Based on this, an improved attention mechanism model is used to adaptively weight multidimensional features, strengthening the semantic expression of key speech segments. Finally, through a low-rank tensor fusion strategy, multi-source heterogeneous features are efficiently integrated to generate an intuitive voice interaction response heatmap. Based on task requirements, core indicators such as interaction success rate and response speed are quantitatively evaluated. This not only overcomes the dependence of traditional methods on single-modal data but also, through efficient feature fusion and adaptive weight allocation mechanisms, greatly improves the accuracy and robustness of voice interaction in complex scenarios, significantly enhancing the intelligent operation capability of smart speakers in dynamic environments. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a smart speaker voice interaction method provided in an embodiment of the present invention. Detailed Implementation

[0014] This invention provides a smart speaker voice interaction method, the core of which lies in achieving high-precision voice recognition and localization in complex scenarios through steps such as multimodal audio signal acquisition, preprocessing, feature fusion, and response heatmap generation. The following is in conjunction with the appendix... Figure 1 The specific embodiments of the present invention will be described in detail below.

[0015] First, a multimodal microphone array calibration is performed to obtain a unified transformation relationship between different microphone coordinate systems. In practice, a calibration sound source with clear acoustic markings is placed at the center of the workspace. This sound source can be a standard signal generator with stable frequency and controllable sound pressure level. The microphone array is controlled to synchronously acquire data from this calibration sound source from at least three different angles, ensuring that the data acquired from each angle covers the key areas of the entire workspace. During the acquisition process, each sensor unit of the microphone array records local audio characteristic data of the calibration sound source, including spectral distribution, phase information, and sound field intensity. Subsequently, beamforming or sound source localization calculations are performed on each set of data to extract the key point coordinates of the calibration sound source at each acquisition position. Based on the known microphone installation parameters and acquisition angles, the rotation and translation transformation matrices between each microphone coordinate system are accurately solved using maximum likelihood estimation or minimum acoustic error methods.

[0016] Next, multimodal audio feature data of the target user is collected. The user is placed within a fixed area of ​​the speaker's operating range to ensure that their voice is within the detectable range. Microphone array parameters are set, and the audio sensor and environmental noise monitoring module are activated. While the speaker rotates at a constant speed, the sensor array continuously captures the spectral structure, sound field details, and background noise distribution information of the user's speech. During this process, the audio sensor can be a MEMS microphone array-based device with a sensitivity typically between -38dB and -26dB and a sampling rate of no less than 48kHz to ensure the integrity of high-frequency signals. To cope with noisy environments, in another preferred embodiment, an active noise-canceling audio sensor can be used because its anti-interference characteristics are better suited to high-noise conditions, enabling the acquisition of higher resolution speech information. Through a multi-channel signal synchronous acquisition system, raw data from different sensors are recorded and integrated in real time to form a high-dimensional feature set covering the entire voice interaction process. For example, assuming the microphone array contains M sensors, and each sensor collects data of dimension D, the final high-dimensional feature set can be represented as an M×D matrix F, where each row corresponds to the data collected by one sensor.

[0017] Preprocessing of the acquired raw multimodal audio data aims to eliminate redundant information and enhance data quality, laying the foundation for subsequent feature fusion and analysis. The preprocessing process mainly includes three steps: noise suppression, hierarchical feature extraction, and data dimensionality reduction. In the noise suppression stage, an adaptive filtering algorithm is used to process the raw audio signal. Its core idea is to dynamically adjust the filter coefficients based on the statistical characteristics of the input signal. Assuming the input signal is x(n), the output signal is y(n), and the filter coefficients are w(n), the update formula of the adaptive filtering algorithm is w(n+1) = w(n) + μ·e(n)·x(n), where μ is the step size factor and e(n) is the error signal. This algorithm can effectively remove background noise while retaining the main components of the speech signal. In the feature hierarchical extraction stage, a multi-scale time-frequency pyramid structure from deep learning can be optionally applied to decompose the audio signal into time-frequency feature maps of multiple scales. For example, assuming the time-frequency plot of the input signal is S(ft), a series of feature maps {S1(ft) S2(ft) ... Sk(ft)} can be obtained through multi-scale decomposition, where k is the decomposition level. These feature maps correspond to different time and frequency resolutions, thus comprehensively capturing the multi-level features of the speech signal. In the data dimensionality reduction stage, a low-rank tensor decomposition method is used to retain the main feature components while reducing computational complexity. Assuming the original feature matrix is ​​X with a size of I×J×K, it can be approximately represented by low-rank tensor decomposition as X ≈ G ×1 U1 ×2 U2 ×3 U3, where G is the core tensor, and U1, U2, and U3 are the factor matrices for each dimension. This method can significantly reduce the data dimensionality while maintaining feature expressiveness.

[0018] The core step in voice interaction is to efficiently fuse preprocessed multidimensional features with task-driven prior knowledge. The aim is to achieve adaptive integration of multi-source heterogeneous features through the synergy of low-rank tensor decomposition and attention mechanisms. First, a task-related prior knowledge base is loaded, including historical interaction data and statistical patterns of successful cases. For example, assuming the prior knowledge base contains N historical interaction samples, each with a feature vector fi and a corresponding interaction result yi, the mapping relationship between features and results can be obtained through statistical analysis. Second, coarse-grained feature alignment is performed. This can be done by manually annotating key speech segments or using an initial alignment algorithm based on time-frequency matching to determine the preliminary correspondence between features of each modality. For example, assuming the features of two modalities are A and B, the optimal alignment path can be found by calculating their similarity matrix S(ij) = A(i)·B(j)T. Then, fine-grained feature optimization is performed. Based on the coarse alignment, an attention mechanism model is introduced to dynamically adjust the weights of different feature channels, highlighting speech segments that play a crucial role in interaction decisions. Assuming the input feature is X and the attention weight is α, the improved attention mechanism model can be represented as Y = softmax(W·X + b)·X, where W is the weight matrix and b is the bias term. This model can automatically learn the importance of different features and perform weighted processing on them.

[0019] A voice interaction response heatmap is generated and evaluated based on the fusion results. Optimized multidimensional features are mapped to a unified time domain, and a confidence score is calculated for each potential interaction point. This score reflects the reliability of the point as a candidate interaction location. Positive values ​​indicate a higher interaction success rate, while negative values ​​indicate lower interaction feasibility. The set of confidence scores for all points constitutes a high-density global interaction scoring field. For example, assuming the time axis is divided into T discrete points, and each point has a confidence score of ci, the global interaction scoring field can be represented as C = {c1 c2 ... cT}. A voice interaction response visualization heatmap is generated based on this scoring field. The calculated confidence scores for each point are visually displayed on the time axis using color gradient rendering, forming a heat map. Operators can clearly observe the optimal interaction time periods and their priority ranking, enabling them to quickly formulate interaction strategies. Furthermore, interaction performance indicators can be quantitatively evaluated. Based on industry standards or task requirements, key interaction evaluation parameters, such as response stability index, anti-interference ability, and task completion rate, are extracted and calculated from the scoring field data. Finally, a prediction report is output, automatically generating an analysis report that includes heatmaps, quantitative values ​​of key performance indicators, and recommended interaction schemes based on preset thresholds.

[0020] In practical applications, the method of this invention can significantly improve the voice interaction capabilities of smart speakers in complex environments. For example, in a home environment, when a user issues a voice command, the smart speaker can accurately recognize the command content and respond quickly using the method described above, maintaining high accuracy even with background music playing or multiple people conversing. In an office environment, this method can help the smart speaker distinguish the voices of different speakers, thereby achieving precise task allocation and execution. Furthermore, through the generated voice interaction response heatmap, users can intuitively understand the optimal interaction time, thereby optimizing the timing of issuing voice commands and further improving interaction efficiency.

[0021] In summary, this invention achieves high-efficiency and high-precision prediction of voice interaction behavior of smart speakers through multimodal audio signal acquisition, preprocessing, feature fusion, and response heatmap generation, significantly improving the operational robustness and intelligence level in complex scenarios.

Claims

1. A voice interaction method for an intelligent speaker, characterized in that, include: Perform multimodal microphone array calibration to obtain a unified transformation relationship between different microphone coordinate systems; Collect multimodal audio feature data of the target user, including spectral features, environmental noise distribution, and spatial sound field information; The acquired raw multimodal audio data is preprocessed, including noise suppression, feature layer extraction, and data dimensionality reduction. The preprocessed multidimensional features are efficiently fused with task-driven prior knowledge, and the adaptive integration of multi-source heterogeneous features is achieved through the synergistic effect of low-rank tensor decomposition and attention mechanism. A voice interaction response heatmap is generated based on the fusion results, and a confidence score for each potential interaction point is calculated to form a global interaction scoring field. The interaction performance metrics are evaluated based on the global interaction scoring field, and a prediction report is output.

2. The method according to claim 1, characterized in that, The multimodal microphone array calibration includes: Place a calibration sound source with clear acoustic markings in the center of the workspace, and control the microphone array to synchronously acquire multiple sets of local audio feature data from at least three different angles. Beamforming or sound source localization calculations are performed on each set of data to extract the key point coordinates of the calibration sound source at each acquisition location; Based on the known microphone installation parameters and acquisition angle, the rotation matrix and translation vector between each microphone coordinate system are solved by the maximum likelihood estimation or the method of minimizing acoustic error.

3. The method according to claim 1, characterized in that, The multimodal audio feature data of the target user collected includes: Place the user under test within a fixed area of ​​the speaker's operating range to ensure that their sound emission is within the detectable range; Set the microphone array parameters, activate the audio sensor and environmental noise monitoring module, and enable the speaker to capture the spectral structure, sound field details and background noise distribution information of the user's voice while rotating at a constant speed. The system records and integrates raw data from different sensors in real time through a multi-channel signal synchronous acquisition system, forming a high-dimensional feature set covering the entire voice interaction process.

4. The method according to claim 1, characterized in that, The preprocessing of the acquired raw multimodal audio data includes: An adaptive filtering algorithm is used to suppress noise in the audio data. Feature hierarchical extraction is performed using a multi-scale time-frequency pyramid structure from deep learning. We employ a low-rank tensor decomposition method to reduce the dimensionality of high-dimensional data, preserving key features while reducing computational complexity.

5. The method according to claim 1, characterized in that, The efficient fusion of preprocessed multidimensional features with task-driven prior knowledge includes: Load the prior knowledge base related to the task, which includes statistical patterns of historical interaction data and success cases; Perform coarse-grained feature alignment by manually annotating key speech segments or using an initial alignment algorithm based on time-frequency matching to determine the preliminary correspondence of each modality feature; Fine-grained feature optimization is performed, and an attention mechanism model is introduced on the basis of coarse alignment to dynamically adjust the weights of different feature channels.

6. The method according to claim 5, characterized in that, The dynamic weight adjustment of the attention mechanism model is achieved through the following formula: αi is equal to exp(ei) divided by ∑j=1Nexp(ej), where ei is the importance score of the i-th feature channel and αi is its corresponding attention weight.

7. The method according to claim 1, characterized in that, The generation of the voice interaction response heatmap based on the fusion result includes: The optimized multidimensional features are mapped to a unified time domain, and the confidence score for each potential interaction point is calculated. The confidence scores of each point are displayed on the time axis using color gradient rendering to form a heat map.

8. The method according to claim 1, characterized in that, The interaction performance evaluation metrics based on the global interaction scoring field include: The response stability index is calculated and defined as the average confidence score within the neighborhood of the interaction point. The anti-interference capability is calculated as the probability of a successful interaction point remaining stable when subjected to external noise interference. The task completion rate is calculated as the proportion of voice interaction tasks that are successfully completed within a certain period of time.

9. The method according to claim 1, characterized in that, The output prediction report includes: Generate an analysis report that includes a heatmap of voice interaction response, quantitative values ​​of key performance indicators, and recommended interaction schemes; The recommended interaction scheme is based on a comprehensive evaluation of various performance indicators, and the interaction point with the highest confidence score and that meets the stability requirements is selected as the final recommended position.

10. The method according to claim 1, characterized in that, The audio sensor is a MEMS microphone array-based device or an active noise-canceling audio sensor.

Images