A privacy voice interaction method and device based on air sound to normal voice mapping

Abstract

The application discloses a privacy voice interaction method and device, and relates to the technical field of voice signal processing.The method comprises the following steps: collecting the user's breath sound signal, the breath sound signal is a voice signal generated by non-vocal cord vibration, the sound pressure level is usually lower than 40 dBA, and is 20-30 dB lower than normal voice; extracting acoustic characteristics; converting the acoustic characteristics into the normal voice acoustic characteristics of the user himself through a mapping model, the process is not aimed at extracting semantic information, and the user's voiceprint characteristics are retained; and reconstructing the voice waveform and outputting.The application solves the problem of privacy leakage of public voice interaction.Different from the ASR technology, the application directly converts the signal in the acoustic domain, so that the remote receiver hears the voice of the user at normal volume, the surrounding people cannot identify the breath sound content under the normal social distance, and the continuity of voiceprint authentication and voice communication is not affected.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, specifically to a method and apparatus for converting aerosound signals into a user's normal speech signal in real time in a privacy scenario. Background Technology

[0002] With the increasing popularity of AI voice assistants, more and more users are getting used to interacting with smart devices via voice. However, in many scenarios where speaking is inconvenient, users find it difficult to use voice assistants naturally. For example, in public places (such as offices, subways, and cafes), normal-volume voice input is easily heard by those around, potentially leading to information leaks. Similarly, late at night in quiet environments (such as bedrooms or hotel rooms) when needing to talk to others or operate devices, speaking can disturb their rest. Furthermore, when attending remote meetings, online classes, or in noisy environments, suddenly needing to handle calls or messages presents a challenge: interrupting the current situation to answer loudly, or completing the voice interaction discreetly. These scenarios share a core contradiction—users have a need for voice interaction, but the act of speaking itself can expose privacy, disturb others, or disrupt the social etiquette of the current situation.

[0003] Existing solutions mainly include the following: First, using bone conduction microphones or in-ear microphones to collect speech signals, but these devices are expensive and have a poor user experience; second, directly performing speech recognition on breath sounds (whisper), although modern ASR engines have a certain breath sound recognition capability, the breath sound input itself is still difficult to eliminate the social pressure and privacy leakage risks brought about by the user's lip movements and airflow when speaking; third, using lip reading technology to assist in recognition, but this technology is still in the research stage and has not yet achieved consumer-level popularization.

[0004] More importantly, none of the above solutions have solved a fundamental problem: they either rely on the user's actual voice or only extract semantic text, and cannot fully preserve the user's voiceprint features and speech rhythm without the user speaking. Therefore, they cannot be applied to call scenarios that require voiceprint authentication.

[0005] Furthermore, existing Automatic Speech Recognition (ASR) technology first converts speech into text, and then uses Text-to-Speech (TTS) technology to synthesize the text into speech output. Even when using the speaker's own voiceprint for speech fitting, the synthesized speech often lacks tone, intonation, and emotional expression that match the current dialogue context, resulting in an unnatural call experience and making it unsuitable for call scenarios requiring voiceprint authentication. Simultaneously, this multi-stage processing chain involves multiple computationally intensive stages such as speech recognition, natural language understanding, and speech synthesis, resulting in high computational overhead and processing latency, making it difficult to meet the low-latency requirements of real-time calls.

[0006] Therefore, there is an urgent need for a solution that can protect users' voice privacy while preserving their voice characteristics to achieve seamless communication in various scenarios where it is inconvenient to speak aloud (including but not limited to public places, quiet environments, ongoing meetings or calls). Summary of the Invention

[0007] The present invention aims to solve the above problems and provide a privacy-preserving voice interaction method and device.

[0008] Specifically, the sound pressure level of normal speech is typically in the range of 60-70 dBA, while the sound pressure level of Whisper is usually below 40 dBA, a reduction of 20-30 dB compared to normal speech. At a normal social distance (≥0.5m), this sound pressure level is already lower than the ambient background noise, making it difficult for people in the surrounding area to discern the speech content, but still retaining complete speech information for acoustic feature extraction and mapping processing.

[0009] The core difference of this invention is that it does not aim to extract semantic information, but directly maps the aerophone signal into the speaker's normal speech signal within the acoustic feature domain, preserving all acoustic properties of speech (including fundamental frequency, harmonic structure and voiceprint features). Therefore, it can be directly used in voice calls or voiceprint authentication scenarios.

[0010] The technical solution of the present invention includes: acquiring the user's aerosound signal; extracting the acoustic features of the aerosound signal; converting the acoustic features into the user's normal speech acoustic features through a mapping model; reconstructing the speech waveform based on the normal speech acoustic features and outputting it. Attached Figure Description

[0011] Figure 1 This is the overall system architecture diagram of the present invention.

[0012] Figure 2 This is a schematic diagram of the mapping model structure.

[0013] Figure 3 This is a flowchart of the training method.

[0014] Figure 4 This is a schematic diagram of an application scenario.

[0015] Figure 5 This is a block diagram of the device structure.

[0016] Figure 6 This is a schematic diagram comparing the spectra of aerophones and normal speech. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. Example 1: System Overall Architecture

[0018] like Figure 1 As shown, the overall architecture of the system of the present invention includes five modules: an audio acquisition module 101 acquires the user's aerosound signal; a feature extraction module 102 extracts the Log-Mel spectrum features of the aerosound; a mapping module 103 maps the aerosound features to normal speech features through a trained shallow neural network model (fθ(X,C)); a speech reconstruction module 104 reconstructs the speech waveform through the Griffin-Lim algorithm; and an output module 105 outputs the reconstructed normal speech to the ASR engine or communication link. Example 2: Mapping Model Structure

[0019] like Figure 2 As shown, the mapping model employs a shallow neural network structure. The input layer 201 receives the Mel-spectral features of aerosound. The hidden layer 202 consists of multiple nonlinear activation units; during training, an F0 conditional input can be introduced to recover the fundamental frequency and harmonic structure of normal speech. The output layer 203 outputs the Mel-spectral features of normal speech. The total number of parameters in the model is controlled to within 10k, making it suitable for real-time operation on a mobile terminal NPU. Example 3: Training Method

[0020] like Figure 3 As shown, the model training process includes the following steps: Step 301, collect paired air and normal sound samples; Step 302, extract Mel spectrum features; Step 303, construct the training dataset; Step 304, train the mapping model to minimize the reconstruction loss (L1 / L2 loss function); Step 305, quantize and compress the trained model to adapt it for mobile deployment. Example 4: Application Scenarios

[0021] like Figure 4 As shown, user 601 inputs speech in a public office setting (with other people 604 / 605 around). The sound pressure level of the breathy signal is typically below 40 dBA, 20-30 dB lower than normal speech (60-70 dBA), making it difficult for people in the surrounding area to discern the breathy content at normal social distances. The breathy signal is transmitted to mobile terminal 602 via a receiver. The mobile terminal performs mapping and reconstruction, converting the breathy sound into the user's normal speech. The processed speech is then sent to a remote location via the network, where person 603 hears the user's normal volume speech.

[0022] It should be noted that although existing ASR technologies can recognize aerophone content, the processing object of this invention is not semantic text, but rather the acoustic features themselves. This invention directly maps aerophones to normal sounds within the acoustic domain, preserving voiceprint features, so that the output speech can be directly used in call scenarios. Example 5: Device Structure

[0023] like Figure 5 As shown, the device of the present invention is integrated inside a mobile terminal (phone / wearable device) and includes: an audio acquisition module 501 (microphone); a feature extraction module 502 (Log-Mel / DSP); a mapping module 503 (running on NPU); a reconstruction module 504 (running on DSP); and an output module 505 (speaker / communication port). The modules are arranged in layers inside the casing according to the signal flow direction. Example 6: Comparison of Spectral Features

[0024] like Figure 6 As shown, the left side represents the aeroacoustic spectrum 601, which exhibits a wide-band continuous energy distribution, lacks periodic harmonic structure, and contains a turbulence noise band in the 2-5kHz frequency range. The middle section represents the mapping model fθ(X,C), which converts aeroacoustic features into normal acoustic features through nonlinear mapping. The right side represents the normal speech spectrum 603, containing the fundamental frequency F0 and its integer multiples of harmonics (2F0, 3F0, 4F0, 5F0), exhibiting a comb-like line spectrum structure.

[0025] This invention achieves a complete conversion from "silent" input to normal speech output by mapping the broadband continuous spectrum of aerophones to the normal speech spectrum containing fundamental harmonics, while preserving the speaker's voiceprint characteristics.

Claims

1. A privacy-preserving voice interaction method and device based on mapping aerophones to one's own normal speech, characterized in that, Includes the following steps: Collect the user's aerosound signal, which is a speech signal not generated by vocal cord vibration; extract the acoustic features of the aerosound signal; The acoustic features are input into a mapping model, and the output is the user's normal speech acoustic features; wherein, the mapping process is not for the purpose of extracting semantic information, and the normal speech acoustic features retain the user's voiceprint features; the speech waveform is reconstructed based on the normal speech acoustic features, and the speech waveform is sent to a remote end through a communication link.

2. The method according to claim 1, characterized in that, The aerosonic signal is Whisper speech, which lacks a fundamental frequency component and integer harmonic structure in the spectral domain.

3. The method according to claim 1, characterized in that, The mapping model is a shallow neural network with no more than 10k parameters.

4. The method according to claim 1, characterized in that, The method is executed locally on the mobile terminal, and the reconstruction of the speech waveform is completed locally.

5. The method according to claim 1, characterized in that, The sound pressure level of the aerosonic signal is typically below 40 dBA, which is 20-30 dB lower than normal speech, making it difficult for people in the surrounding area to recognize the content of the speech at normal social distances.

6. A privacy-preserving voice interaction device, characterized in that, include: The audio acquisition module is used to acquire the user's breath sound signals; The feature extraction module is used to extract the acoustic features of the aeroacoustic signal; The mapping module is used to convert the acoustic features into the user's normal speech acoustic features through a mapping model, and the mapping is not for the purpose of extracting semantic information; the reconstruction module is used to reconstruct the speech waveform based on the normal speech acoustic features; the output module is used to send the speech waveform to a remote end through a communication link.

7. The apparatus according to claim 6, characterized in that, The device is integrated into a mobile terminal or wearable device.

8. The apparatus according to claim 6, characterized in that, The aerosonic signal is a speech signal not generated by vocal cord vibration, with a sound pressure level below 40 dBA.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 5.

Images