A speech signal recognition and separation method based on robot voiceprint space-time clustering

Abstract

This invention discloses a speech signal recognition and separation method based on spatiotemporal clustering of robot voiceprints. It relates to the fields of robot voice interaction and digital signal processing technology. The method includes: a robot voice pickup module acquiring mixed speech signals in complex scenarios and performing preprocessing; extracting voiceprint feature vectors from the preprocessed mixed speech signals and marking human voice feature anchor points; constructing a feature matrix based on the voiceprint feature vectors and human voice feature anchor points, and selecting target human voice signal clusters through weight optimization and automatic clustering; using LSTM adaptive spectrum compensation to perform spectrum correction on the target human voice signal clusters to obtain a clean speech signal; inputting the clean speech signal into a robot speech recognition model and outputting the final recognition result. This invention can solve the problems of sound source number dependence, poor robustness to non-stationary noise, speech feature distortion, and long processing delay in existing technologies, improving the accuracy of robot speech recognition and real-time interaction capabilities in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of robot voice interaction and digital signal processing technology, and more specifically to a method for voice signal recognition and separation based on robot voiceprint spatiotemporal clustering. Background Technology

[0002] With the popularization of intelligent robot applications, voice interaction has become the core method of human-machine collaboration. However, in actual application scenarios (shopping malls, factories, ruins, etc.), there are common problems such as multiple sound sources superimposed, non-stationary background noise interference, and low signal-to-noise ratio (-5dB~10dB), which restrict the accuracy and real-time performance of robot voice recognition.

[0003] Existing blind source signal separation techniques have the following key drawbacks: 1. Dependence on the number of sound sources: Traditional algorithms (such as Independent Component Analysis (ICA) and Non-negative Matrix Factorization (NMF) require pre-setting the number and location of sound sources, which cannot adapt to scenarios where sound sources change dynamically during robot movement.

[0004] 2. Poor robustness to non-stationary noise: It has poor separation effect on non-stationary noise such as mechanical impact noise and sudden background human voices, and the separated speech signal is severely distorted.

[0005] 3. Voice feature loss: The separation process easily loses detailed voiceprint features, resulting in a significant drop in subsequent speech recognition rate, with recognition accuracy usually below 65%.

[0006] 4. Insufficient real-time performance: The separation scheme based on deep learning has high model complexity, and the signal processing time for a single frame exceeds 20ms, which cannot meet the requirements of real-time voice interaction for robots (end-to-end latency ≤100ms).

[0007] 5. High hardware costs: Some solutions improve separation by increasing the number of microphone arrays, but this is limited by the robot's installation space and increases hardware modification costs by more than 80%.

[0008] In summary, existing technologies cannot meet the actual needs of voice interaction in complex robot scenarios, and there is an urgent need for a blind source signal separation algorithm that does not require prior sound source information, has high robustness, high fidelity, and low latency. Summary of the Invention

[0009] In view of this, the present invention provides a speech signal recognition and separation method based on spatiotemporal clustering of robot voiceprints, which solves the problems of sound source number dependence, poor robustness of non-stationary noise, speech feature distortion and long processing delay in the prior art, and improves the speech recognition accuracy and real-time interaction capability of robots in complex scenarios.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering includes: Step 1: The robot's voice pickup module collects mixed speech signals from complex scenarios and performs preprocessing; Step 2: Extract the voiceprint feature vector from the preprocessed mixed speech signal and mark the human voice feature anchor points; Step 3: Construct a feature matrix based on voiceprint feature vectors and human voice feature anchor points, and select the target human voice signal clusters through weight optimization and automatic clustering; Step 4: Use LSTM adaptive spectrum compensation to perform spectrum correction on the target human voice signal cluster to obtain a clean speech signal; Step 5: Input the clean speech signal into the robot speech recognition model and output the final recognition result.

[0011] Preferably, it also includes: parallel computing acceleration and feedback optimization; Parallel computing acceleration includes: optimizing the clustering in step 3 for GPU parallel computing; Feedback optimization includes feeding back the human voice feature anchor point parameters from step 2 to the next round of signal processing and dynamically adjusting the anchor point screening threshold.

[0012] Preferably, the preprocessing process in step 1 includes: performing frame-by-frame windowing on the mixed speech signal, using a Hamming window, with a frame length of 20ms~30ms and a frame shift of 10ms~15ms, to ensure a balance between temporal continuity and frequency domain resolution.

[0013] Preferably, step 2 includes voiceprint feature extraction and spatiotemporal feature anchor point marking; Voiceprint feature extraction: Extract 12-dimensional MFCC features and 12-dimensional LPCC features from each frame of mixed speech signal, and fuse them to generate a 24-dimensional voiceprint feature vector; Spatiotemporal feature anchor point marking involves performing STFT transformation on the mixed speech signal to obtain the time-frequency domain distribution characteristics; continuous frame regions with spectral peaks concentrated in the 300Hz~3400Hz human voice main frequency band and frame energy greater than the average energy are selected and marked as human voice feature anchor points.

[0014] Preferably, step 3 specifically includes: The voiceprint feature vector is fused with the proportion of human voice main frequency band energy in each frame of signal to construct a 25-dimensional feature matrix; Set the weight coefficients for anchor frames and non-anchor frames for human voice features, and calculate the local density and relative distance of sample points through a weighted distance matrix; Based on the density peak clustering algorithm, the two sample points with the largest product of local density and relative distance are selected as cluster centers to achieve automatic separation of human voice signal clusters and environmental noise clusters. The proportion of anchor frames in each cluster is counted, and the cluster with the highest proportion is selected as the target human voice signal cluster. The corresponding speech frames are then extracted to obtain the separated speech.

[0015] Preferably, step 4 specifically includes: Spectral feature extraction: The separated speech is subjected to STFT transformation to obtain amplitude spectrum and phase spectrum features; Pre-trained LSTM model loading: Load an LSTM spectrum compensation model trained based on log-spectral distance minimization. The model input is the spectral features of the separated speech, and the output is the spectrum compensation coefficient matrix. Spectrum correction: Adaptive correction of the amplitude and phase spectra of the separated speech using a compensation coefficient matrix; Time-domain signal reconstruction: The corrected spectrum is subjected to inverse STFT transformation, and then reconstructed using the overlap-addition method to obtain a high-fidelity pure speech signal.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for speech signal recognition and separation based on robot voiceprint spatiotemporal clustering, with the following beneficial effects: This invention achieves four core beneficial effects through the collaborative design of a three-order closed-loop architecture, all of which have been verified through simulated scenario testing: 1. Strong adaptability to sound sources, requiring no prior information. It can process mixed signals from 1 to 5 sound sources without presetting the number and location of sound sources; in complex scenarios with a signal-to-noise ratio of -5dB to 10dB, the accuracy of human voice signal separation reaches 96.8%, which is 32.5% higher than the traditional ICA algorithm.

[0017] 2. High voice fidelity and significantly improved recognition accuracy.

[0018] The spectrum compensation model controls the log-spectral distance (LSD) of the separated speech to below 0.15, achieving a speech recognition accuracy of 92.3%, which is 28.7% higher than the uncompensated separation algorithm, thus solving the pain point of speech feature distortion after separation.

[0019] 3. Excellent real-time performance, meeting the needs of robot interaction.

[0020] The single-frame signal processing time is ≤5ms, and the end-to-end algorithm latency is ≤100ms, which fully meets the requirements of real-time voice interaction for robots and improves the processing efficiency by more than 4 times compared with traditional deep learning solutions.

[0021] 4. Low deployment cost and strong hardware compatibility

[0022] No additional microphone array is required; it can be directly deployed in the robot's existing single / dual microphone pickup system, reducing hardware modification costs by more than 80% and demonstrating promising industrialization prospects. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 The diagram shows the third-order closed-loop processing architecture of the algorithm provided in this invention.

[0025] Figure 2 The flowchart for extracting voiceprint spatiotemporal feature anchor points provided by the present invention.

[0026] Figure 3 A comparison chart of the clustering effects of the improved density peak clustering provided by this invention.

[0027] Figure 4 The flowchart shows the training and inference process of the LSTM spectrum compensation model provided by this invention.

[0028] Figure 5 The method flowchart provided by the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 5 and Figure 1 As shown, this invention discloses a method for speech signal recognition and separation based on robot voiceprint spatiotemporal clustering, including: Step 1: The robot's voice pickup module collects mixed speech signals from complex scenarios and performs preprocessing; Step 2: Extract the voiceprint feature vector from the preprocessed mixed speech signal and mark the human voice feature anchor points; Step 3: Construct a feature matrix based on voiceprint feature vectors and human voice feature anchor points, and select the target human voice signal clusters through weight optimization and automatic clustering; Step 4: Use LSTM adaptive spectrum compensation to perform spectrum correction on the target human voice signal cluster to obtain a clean speech signal; Step 5: Input the clean speech signal into the robot speech recognition model and output the final recognition result.

[0031] In a specific embodiment of the present invention, the robot's sound pickup module (single / dual microphones) collects mixed speech signals in complex scenarios, including 1 to 5 sound sources such as target human voice, mechanical noise, and crowd noise.

[0032] Furthermore, during the automatic clustering and filtering process, different sound sources are distinguished based on voiceprints to differentiate between human voices and ambient sounds. Then, different voices are distinguished based on different voiceprints. Next, different content is identified based on speech-to-text conversion. Then, the speaker's intention is determined based on the semantic understanding of the large model. Finally, all the speech of the speaker whose intention is most important is found, and the speech information of other non-primary speakers is excluded.

[0033] Specifically, this also includes: parallel computing acceleration and feedback optimization; Parallel computing acceleration includes: optimizing the clustering in step 3 for GPU parallel computing; Feedback optimization includes feeding back the human voice feature anchor point parameters from step 2 to the next round of signal processing and dynamically adjusting the anchor point screening threshold.

[0034] In a specific embodiment of the present invention, real-time optimization and feedback iteration include: 1. Parallel computing acceleration: The clustering process is optimized for GPU parallel computing, keeping the single-frame signal processing time within 5ms and the end-to-end algorithm latency ≤100ms.

[0035] 2. Feedback Optimization: The parameters of the separated voiceprint feature anchor points are fed back to the next round of signal processing to dynamically adjust the anchor point selection threshold and improve the algorithm's adaptability.

[0036] Furthermore, robot speech recognition models are trained using existing open-source large language models or obtained through collaboration with companies that provide mature speech models.

[0037] Specifically, the preprocessing process in step 1 includes: performing frame-by-frame windowing on the mixed speech signal, using a Hamming window, with a frame length of 20ms~30ms and a frame shift of 10ms~15ms, to ensure a balance between temporal continuity and frequency domain resolution.

[0038] Specifically, such as Figure 2 As shown, step 2 includes voiceprint feature extraction and spatiotemporal feature anchor point marking; Voiceprint feature extraction: Extract 12-dimensional MFCC features and 12-dimensional LPCC features from each frame of mixed speech signal, and fuse them to generate a 24-dimensional voiceprint feature vector; Spatiotemporal feature anchor point marking involves performing STFT transformation on the mixed speech signal to obtain the time-frequency domain distribution characteristics; continuous frame regions with spectral peaks concentrated in the 300Hz~3400Hz human voice main frequency band and frame energy greater than the average energy are selected and marked as human voice feature anchor points.

[0039] Specifically, step 3 includes: The voiceprint feature vector is fused with the proportion of human voice main frequency band energy in each frame of signal to construct a 25-dimensional feature matrix; Set the weight coefficients for anchor frames and non-anchor frames for human voice features, and calculate the local density and relative distance of sample points through a weighted distance matrix; Based on the density peak clustering algorithm, the two sample points with the largest product of local density and relative distance are selected as cluster centers to achieve automatic separation of human voice signal clusters and environmental noise clusters. The proportion of anchor frames in each cluster is counted, and the cluster with the highest proportion is selected as the target human voice signal cluster. The corresponding speech frames are then extracted to obtain the separated speech.

[0040] In a specific embodiment of the present invention, such as Figure 3 As shown, the anchor frame ratio in the left image is 92.3% for the improved algorithm, while the ratio in the right image is 65.7% for the traditional algorithm. The improved algorithm fuses the voiceprint feature vector with the proportion of human voice main frequency band energy in each frame to construct a 25-dimensional feature matrix. The weighting coefficient for anchor frames is set to 1.5~2.0 times, and the weighting coefficient for non-anchor frames is set to 0.8~1.0 times. The local density and relative distance of sample points are calculated using a weighted distance matrix. Based on the density peak clustering algorithm, the two sample points with the largest product of local density and relative distance (γ value) are selected as cluster centers, achieving automatic separation of human voice signal clusters from environmental noise clusters without the need to preset the number of sound sources.

[0041] Specifically, such as Figure 4 As shown, step 4 specifically includes: Spectral feature extraction: The separated speech is subjected to STFT transformation to obtain amplitude spectrum and phase spectrum features; Pre-trained LSTM model loading: Load an LSTM spectrum compensation model trained based on log-spectral distance minimization. The model input is the spectral features of the separated speech, and the output is the spectrum compensation coefficient matrix. Spectrum correction: Adaptive correction of the amplitude and phase spectra of the separated speech using a compensation coefficient matrix; Time-domain signal reconstruction: The corrected spectrum is subjected to inverse STFT transformation, and then reconstructed using the overlap-addition method to obtain a high-fidelity pure speech signal.

[0042] The specific embodiments provided by this invention are as follows: Example 1: Service Robot Testing in a Shopping Mall Scenario 1. Test scenario: Shopping mall environment (background noise: crowd noise, broadcast noise, air conditioner fan noise, signal-to-noise ratio 5dB) 2. Test equipment: The service robot is equipped with a dual-microphone pickup system and the algorithm of this invention is deployed.

[0043] 3. Test results: The target human voice separation accuracy was 97.2%, the speech recognition accuracy was 93.1%, and the end-to-end latency was 85ms; the traditional ICA algorithm had a separation accuracy of 64.7%, a recognition accuracy of 64.4%, and a latency of 210ms.

[0044] Example 2: Rescue robot test in a ruins scenario

[0045] 1. Test scenario: Simulated earthquake ruins environment (background noise: mechanical digging sound, wall collapse sound, signal-to-noise ratio -3dB)

[0046] 2. Testing Equipment: The rescue robot is equipped with a single-microphone pickup system, and the algorithm of this invention is deployed.

[0047] 3. Test results: The survivor speech separation accuracy was 95.1%, and key commands such as "help" and "I'm here" were successfully recognized; the traditional NMF algorithm failed in both separation and recognition due to severe noise interference.

[0048] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0049] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for speech signal recognition and separation based on spatiotemporal clustering of robot voiceprints, characterized in that, include: Step 1: The robot's voice pickup module collects mixed speech signals from complex scenarios and performs preprocessing; Step 2: Extract the voiceprint feature vector from the preprocessed mixed speech signal and mark the human voice feature anchor points; Step 3: Construct a feature matrix based on voiceprint feature vectors and human voice feature anchor points, and select the target human voice signal clusters through weight optimization and automatic clustering; Step 4: Use LSTM adaptive spectrum compensation to perform spectrum correction on the target human voice signal cluster to obtain a clean speech signal; Step 5: Input the clean speech signal into the robot speech recognition model and output the final recognition result.

2. The speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering according to claim 1, characterized in that, Also includes: Parallel computing acceleration and feedback optimization; Parallel computing acceleration includes: optimizing the clustering in step 3 for GPU parallel computing; Feedback optimization includes feeding back the human voice feature anchor point parameters from step 2 to the next round of signal processing and dynamically adjusting the anchor point screening threshold.

3. The speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering according to claim 1, characterized in that, The preprocessing process in step 1 includes: performing frame-by-frame windowing on the mixed speech signal, using a Hamming window, with a frame length of 20ms~30ms and a frame shift of 10ms~15ms, to ensure a balance between temporal continuity and frequency domain resolution.

4. The speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering according to claim 3, characterized in that, Step 2 includes voiceprint feature extraction and spatiotemporal feature anchor point marking; Voiceprint feature extraction: Extract 12-dimensional MFCC features and 12-dimensional LPCC features from each frame of mixed speech signal, and fuse them to generate a 24-dimensional voiceprint feature vector. Spatiotemporal feature anchor point marking involves performing STFT transformation on the mixed speech signal to obtain the time-frequency domain distribution characteristics; continuous frame regions with spectral peaks concentrated in the 300Hz~3400Hz human voice main frequency band and frame energy greater than the average energy are selected and marked as human voice feature anchor points.

5. The speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering according to claim 4, characterized in that, Step 3 specifically includes: The voiceprint feature vector is fused with the proportion of human voice main frequency band energy in each frame of signal to construct a 25-dimensional feature matrix; Set the weight coefficients for anchor frames and non-anchor frames for human voice features, and calculate the local density and relative distance of sample points through a weighted distance matrix; Based on the density peak clustering algorithm, the two sample points with the largest product of local density and relative distance are selected as cluster centers to achieve automatic separation of human voice signal clusters and environmental noise clusters. The proportion of anchor frames in each cluster is counted, and the cluster with the highest proportion is selected as the target human voice signal cluster. The corresponding speech frames are then extracted to obtain the separated speech.

6. The speech signal recognition and separation method based on robot voiceprint spatiotemporal clustering according to claim 5, characterized in that, Step 4 specifically includes: Spectral feature extraction: The separated speech is subjected to STFT transformation to obtain amplitude spectrum and phase spectrum features; Pre-trained LSTM model loading: Load an LSTM spectrum compensation model trained based on log-spectral distance minimization. The model input is the spectral features of the separated speech, and the output is the spectrum compensation coefficient matrix. Spectrum correction: Adaptive correction of the amplitude and phase spectra of the separated speech using a compensation coefficient matrix; Time-domain signal reconstruction: The corrected spectrum is subjected to inverse STFT transformation, and then reconstructed using the overlap-addition method to obtain a high-fidelity pure speech signal.

Images