A voice privacy processing method, a near-ear wearable device and a storage medium

Abstract

The present disclosure relates to the technical field of audio signal processing, and discloses a speech privacy processing method, an ear-wearable device and a storage medium. The method comprises: obtaining an original downlink speech signal; analyzing the original downlink speech signal to obtain fundamental frequency information, filter coefficients and an energy envelope; replacing a periodic excitation signal in the original downlink speech signal with a noise excitation signal having an energy distribution matched with the energy envelope according to the fundamental frequency information to obtain a replaced speech signal; performing spectral shaping processing on the replaced speech signal based on the filter coefficients to obtain a whisper audio signal having whisper speech characteristics; and playing the whisper audio signal. The method can greatly reduce the probability that a third party picks up and identifies the speech content when the earphone is played outside, realize speech privacy protection, ensure that the speech timbre heard by the wearer is natural and complete in layers, and balance the privacy protection effect and the user's listening clarity and naturalness.

Description

Technical Field

[0001] This disclosure relates to the field of audio signal processing technology, and more specifically, to a voice privacy processing method for a near-ear wearable device, a near-ear wearable device, and a computer-readable storage medium. Background Technology

[0002] In recent years, open-back headphones (OWS) have been widely used in the consumer electronics market due to their comfortable fit and good environmental awareness. These devices employ an open acoustic structure, where there is no physical sound barrier between the speaker unit and the external environment. Downlink voice signals radiate into the surrounding space during playback, allowing nearby listeners to hear the conversation, thus posing a risk of voice privacy breaches.

[0003] The aforementioned audio leakage problem is particularly prominent when using near-ear wearable devices to answer phone calls or listen to voice messages in environments such as public places and office areas. Current solutions for this scenario typically struggle to simultaneously preserve speech intelligibility and adaptively perform real-time acoustic feature conversion of downlink speech based on ambient noise levels to eliminate speech content that can be recognized by bystanders. Summary of the Invention

[0004] One objective of this disclosure is to provide a technical solution for real-time voice privacy processing in near-ear wearable devices, in order to solve the technical problem that when near-ear wearable devices play downlink voice messages, the content of the call can be obtained by an eavesdropper due to audio leakage.

[0005] According to a first aspect of this disclosure, a method for voice privacy processing in a near-ear wearable device is provided, the method comprising: Acquire the raw downlink speech signal; The original downlink speech signal is analyzed to obtain the fundamental frequency information, filter coefficients, and energy envelope; Based on the fundamental frequency information, the periodic excitation signal in the original downlink speech signal is replaced with a noise excitation signal whose energy distribution matches the energy envelope, to obtain the replaced speech signal; The replaced speech signal is subjected to spectral shaping based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics. Play the whispered audio signal.

[0006] Optionally, the step of replacing the periodic excitation signal in the original downlink speech signal with a noise excitation signal whose energy distribution matches the energy envelope, based on the fundamental frequency information, to obtain the replaced speech signal, includes: generating a broadband noise signal; modulating the energy of the broadband noise signal according to the energy envelope to obtain a noise excitation signal whose energy distribution matches the energy envelope; and replacing the periodic excitation signal in the original downlink speech signal with the noise excitation signal based on the fundamental frequency information, to obtain the replaced speech signal.

[0007] Optionally, the step of performing spectral shaping processing on the replaced speech signal based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics includes: constructing a synthesis filter based on the filter coefficients; and filtering the replaced speech signal through the synthesis filter to obtain the whispered audio signal.

[0008] Optionally, before playing the whispered audio signal, the method further includes: performing perceptual quality optimization processing on the whispered audio signal.

[0009] Optionally, the perceptual quality optimization processing of the whispered audio signal includes: performing spectrum restoration processing on the whispered audio signal; and / or performing dynamic range compression processing on the whispered audio signal; and / or performing frequency response compensation processing on the whispered audio signal.

[0010] Optionally, before analyzing the original downlink speech signal to obtain the fundamental frequency information, filter coefficients, and energy envelope, the method further includes: performing noise reduction processing on the original downlink speech signal.

[0011] Optionally, the method further includes: acquiring an environmental noise signal; detecting the current environmental noise level based on the environmental noise signal; and, if the current environmental noise level is lower than a preset noise level threshold, performing privacy processing on the original downlink speech signal.

[0012] Optionally, detecting the current environmental noise level based on the environmental noise signal includes: analyzing the sound pressure level or spectral characteristics of the environmental noise signal to obtain the current environmental noise level.

[0013] According to a second aspect of the present disclosure, a near-ear wearable device is provided, including a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to the first aspect of the present disclosure.

[0014] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, the computer program implementing the method according to the first aspect of the present disclosure when executed by a processor.

[0015] This embodiment extracts three core speech features from the original downlink speech data source: fundamental frequency information, filter coefficients, and energy envelope. Based on the fundamental frequency information, it accurately locates and replaces periodic excitation signals in the original speech that are prone to long-distance propagation and cause speech leakage. Then, a noise excitation signal matching the energy envelope is used as a substitute sound source without the risk of periodic leakage, ensuring that the speech loudness dynamically matches the original sound without sounding harsh. Subsequently, filter coefficients representing the speaker's original timbre are reused to complete spectrum shaping, creating a whisper spectrum with prominent low-frequency attenuation and mid-to-high-frequency frictional characteristics. Finally, the modified whisper audio signal is played. The overall solution eliminates periodic sound waves with long-distance propagation capabilities at the source, while completely preserving the formants and loudness variation details of the human voice. This significantly reduces the probability of others picking up and recognizing the speech content when the headphones are played aloud, achieving speech privacy protection, while also ensuring that the wearer hears a natural and complete timbre of the speech, balancing privacy protection with the user's listening clarity and naturalness.

[0016] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0018] Figure 1 This is a schematic diagram of a voice privacy processing system for a near-ear wearable device that can apply the methods of the embodiments of this disclosure.

[0019] Figure 2 This is a flowchart illustrating a voice privacy processing method for near-ear wearable devices according to some embodiments.

[0020] Figure 3 This is a block diagram of a near-ear wearable device according to one embodiment. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0026] Figure 1 This is a schematic diagram of a voice privacy processing system for a near-ear wearable device that can apply the methods of the embodiments of this disclosure. Figure 1 As shown, the voice privacy processing system 1000 includes a downlink signal input module 1100, an environment recognition module 1200, a signal processing module 1300, and an output module 1400. The modules are connected to each other through digital audio signals to form a complete real-time processing link.

[0027] The downlink signal input module 1100 is used to receive downlink audio data streams sent by external terminals such as mobile phones, players, and call base stations connected to the near-ear wearable device, complete audio sampling, quantization, analog-to-digital conversion, and generate raw downlink voice signals in a standardized digital format; and synchronously transmit the raw downlink voice signals to the signal processing module 1300.

[0028] The environmental recognition module 1200 has a built-in microphone for continuously collecting simulated environmental noise signals from the surrounding space. It performs frame segmentation and energy calculation on the environmental noise signals to obtain the real-time environmental noise level. The noise level is compared with a preset noise threshold to identify and distinguish between three usage scenarios: quiet, normal, and noisy. Scene determination results and control commands are generated and sent to the signal processing module 1300. Only when a quiet scenario is identified is a privacy processing command output; in noisy scenarios, a bypass pass-through command is output, allowing the system to skip the voice conversion process and reduce device power consumption.

[0029] The signal processing module 1300 is communicatively connected to the downlink signal input module 1100, the environment recognition module 1200, and the output module 1400, respectively. It is used to receive the original downlink voice signal and the environmental scene control command, and perform privacy-preserving voice processing or direct output to the output module 1400 according to the command.

[0030] The output module 1400 is electrically connected to the open-back headphone speaker hardware and is used to receive the whisper audio signal output by the signal processing module 1300; drive the open-back headphone speaker to play the whisper audio signal and radiate outwards the acoustic speech with whisper acoustic characteristics and extremely low recognizability at a distance.

[0031] The hardware components of a near-ear wearable device typically include: one or more processors (e.g., a digital signal processor (DSP) or a microcontroller (MCU)) for executing real-time voice privacy processing algorithms; memory for storing computer programs and intermediate processing data; a microphone for acquiring ambient sound; and a speaker for playing the processed whispered audio signal. These hardware components collectively constitute the physical basis for the near-ear wearable device to perform the methods of the embodiments of this disclosure.

[0032] This disclosure relates to a real-time voice privacy processing scheme for near-ear wearable devices. Figure 1 In the system architecture shown, near-ear wearable devices (such as open-back headphones, Open Wearable Stereo, OWS) radiate downlink voice signals into the surrounding space during playback due to their open acoustic structure, potentially allowing eavesdroppers to access the conversation content. Related technologies typically lack a processing mechanism that can adaptively trigger based on ambient noise levels and convert downlink voice signals into audio signals that are difficult for eavesdroppers to recognize in real time, resulting in a technical problem of voice privacy leakage. To address this, this disclosure embodiment constructs a real-time voice privacy processing mechanism that combines environmentally aware adaptive triggering with acoustic feature decoupling conversion. When a low ambient noise level is detected, the privacy processing flow is automatically triggered, removing the fundamental frequency component of the downlink voice and replacing it with noise excitation. Then, it is reconstructed based on the spectral envelope, ensuring that the output audio retains speech intelligibility while eliminating individual pitch characteristics, thereby preventing eavesdroppers from recognizing the conversation content.

[0033] The following combination Figure 1 The system / application scenarios described herein illustrate various embodiments of this disclosure.

[0034] <First Embodiment> Figure 2 This is a flowchart illustrating a voice privacy processing method for a near-ear wearable device according to some embodiments. This voice privacy processing method for the near-ear wearable device can be implemented by a voice privacy processing system of the near-ear wearable device; specifically, it can be implemented by, for example... Figure 1 The illustrated voice privacy processing system 1000 for a near-ear wearable device is implemented. For example... Figure 2 As shown, the voice privacy processing method of the near-ear wearable device in this embodiment may include the following steps S210 to S250.

[0035] Step S210: Obtain the raw downlink voice signal.

[0036] In this embodiment, the original downlink voice signal is a digital audio signal transmitted to the near-ear wearable device by external devices such as mobile terminals, audio-visual playback devices, and call base stations connected to the near-ear wearable device. Its content includes voice content such as callers' voices and media voices.

[0037] Specifically, near-ear wearable devices can receive external audio data streams through communication channels such as Bluetooth, wired audio interfaces, and wireless audio links, and obtain the original downlink voice signal after analog-to-digital conversion and sampling quantization.

[0038] The near-ear wearable device of this embodiment is a portable wearable audio device that can fit snugly to the ear and has a built-in audio playback unit and microphone. It includes in-ear Bluetooth headphones, semi-open headphones, and over-ear near-ear headphones, and integrates an audio signal processing chip, a speaker, and a storage module. It can independently execute all the voice privacy algorithms of this application.

[0039] In some examples, the raw downlink speech signal is denoised before step S220 is performed to analyze it.

[0040] Denoising the original downlink speech signal includes removing background noise and / or echo residue from the original downlink speech signal.

[0041] Specifically, adaptive Wiener filtering / spectral subtraction can be used to suppress transmission clutter and circuit noise in the original downlink speech signal, and remove interference noise components that do not carry speech semantics.

[0042] In these examples, filtering out irrelevant clutter interference through a pre-denoising step can improve the accuracy of extracting the three types of acoustic features—fundamental frequency, filter coefficients, and energy envelope—and avoid misjudgment of periodic excitation and modulation distortion caused by clutter, thereby improving the speech clarity and naturalness of the final whispered audio.

[0043] In some embodiments, before performing step S220, the method further includes: acquiring an ambient noise signal; detecting the current ambient noise level based on the ambient noise signal; and performing privacy processing on the original downlink speech signal if the current ambient noise level is lower than a preset noise level threshold.

[0044] This step is the pre-processing environment judgment logic for privacy processing. Before performing privacy processing on the original downlink voice signal through steps S220 to S240, the background noise of the environment around the wearable device is picked up and the noise intensity is quantified. Based on the quietness of the environment, an adaptive decision is made on whether to start the entire voice privacy calculation. The subsequent signal conversion process is only executed in quiet scenarios where voice leakage is likely to occur. In noisy scenarios, the original voice is played directly, thus achieving a balance between computing power and privacy protection.

[0045] The ambient noise signal is the acoustic signal of the surrounding space picked up in real time by the microphone on the near-ear wearable device. It does not include downlink playback voice and only records ambient background sounds, including human voice noise, road noise, indoor background noise and other types of ambient sound waves.

[0046] The current environmental noise level is a numerical indicator obtained by quantifying the environmental noise signal, and is used to objectively characterize the strength level of the current environmental background noise.

[0047] The noise level threshold is a fixed numerical threshold pre-programmed into the device's memory, used as a critical standard to distinguish between quiet environments with high privacy risks and noisy environments with low privacy risks.

[0048] In some examples, the current ambient noise level is detected based on the ambient noise signal, including: performing frame-by-frame statistics and mean smoothing calculation on the sound pressure level of the ambient noise signal within the sliding window length to obtain a stable current ambient noise level.

[0049] This example details the complete sub-steps for noise level detection: the microphone continuously collects ambient noise to generate a continuous digital audio stream, the ambient noise is divided into frames according to the standard frame length of the speech, the sound pressure level of each frame is calculated, and the sliding average of the sound pressure levels of multiple consecutive frames is taken as the final current ambient noise level.

[0050] In these examples, by averaging and smoothing the sound pressure levels of multiple frames to eliminate numerical jumps caused by instantaneous sudden noises, it is possible to avoid frequent start-stop of privacy functions and repeated signal switching caused by a single abnormal noise, thereby improving the continuity of audio playback and the user's listening comfort.

[0051] In other examples, when calculating the average ambient noise sound pressure level, the sliding window length can be dynamically adjusted based on the device's remaining battery power; when the device is low on battery power, the sliding window length is shortened to reduce the amount of computation required for multi-frame buffering.

[0052] In these examples, by dynamically adapting the sliding window length to balance noise detection accuracy and computational overhead, the power consumption caused by continuous environmental detection can be reduced in low-battery scenarios of headphones, thereby extending the single-use time of the device.

[0053] In other examples, the current ambient noise level is detected based on the ambient noise signal, including: extracting the spectral characteristics of the ambient noise signal for analysis to obtain the current ambient noise level.

[0054] Specifically, one could extract the full-band spectral energy features of the environmental noise signal to construct a noise feature vector, and then use a pre-trained classification model to determine the corresponding environmental noise level, which would then serve as the current environmental noise level.

[0055] If the ambient noise level is lower than the preset noise threshold, the privacy processing procedure is triggered, that is, steps S220 to S250 of this embodiment are executed; if the ambient noise level is higher than or equal to the preset noise threshold, the privacy processing procedure is closed and the original downlink voice signal is played directly.

[0056] This triggering mechanism allows privacy processing to be disabled in high-noise environments (where the risk of eavesdropping is low), allowing the original downlink voice signal to be played directly. This avoids unnecessary power consumption from signal processing and improves the naturalness of the call.

[0057] Step S220: Analyze the original downlink speech signal to obtain fundamental frequency information, filter coefficients, and energy envelope.

[0058] In this embodiment, three independent acoustic features are simultaneously separated using a short-time speech analysis algorithm: fundamental frequency information, filter coefficients, and energy envelope. This provides all the underlying feature data support for subsequent replacement of periodic excitation, matching of noise excitation energy, and reconstruction of the whisper spectrum.

[0059] Fundamental frequency information (also known as fundamental tone frequency, F0) is a key acoustic parameter in speech signals that reflects the speaker's pitch characteristics and carries individual speaker identification information. The energy envelope reflects the amplitude variation of the speech signal over time. Filter coefficients describe the speaker's vocal tract resonance characteristics; these coefficients encode the formant structure of the speech signal (i.e., the key carrier of speech intelligibility) but do not contain fundamental frequency (pitch) information.

[0060] In some embodiments, before analyzing the raw downlink audio signal, the raw downlink audio signal can be subjected to frame segmentation and windowing processing. The frame length for frame segmentation can be preset according to the application scenario or specific requirements. For example, the frame length can be 20ms to 30ms, and Hamming windows can be used for windowing.

[0061] In some examples, the fundamental frequency information can be extracted using the autocorrelation function method, which calculates the autocorrelation function for each frame of speech signal, finds the delay corresponding to the first significant peak, and thus estimates the fundamental frequency value of that frame. After processing frame by frame, the fundamental frequency profile sequence is obtained.

[0062] In other examples, the cepstral method can also be used to extract the fundamental frequency. This involves performing cepstral analysis on the speech frame and finding the peak value of the corresponding fundamental frequency in the cepstral domain. Both the autocorrelation function method and the cepstral method are mature algorithms with low computational complexity, suitable for real-time processing scenarios in near-ear wearable devices.

[0063] The energy envelope can be calculated frame by frame using the short-time root mean square energy.

[0064] The filter coefficients can be extracted using linear predictive coding (LPC) analysis, and these coefficients describe the vocal tract resonance characteristics of the speech signal.

[0065] In other examples, during frame-by-frame windowing, the frame length is dynamically adjusted based on the device's remaining battery power. Specifically, the frame length in low-battery scenarios can be longer than that in normal-battery scenarios. For instance, a frame length of 30ms can be used in low-battery scenarios to reduce the number of frames processed per unit time; while a frame length of 20ms can be used in normal-battery scenarios to ensure the precision of feature extraction.

[0066] In these examples, by dynamically and adaptively adjusting the speech analysis frame length, the frequency of feature extraction operations can be reduced when the device is low on power, thus balancing feature accuracy with device battery life.

[0067] In some other examples, the total signal energy of the original speech frame can be calculated frame by frame to obtain the single-frame energy gain. The energy gains of multiple frames with continuous temporal sequence can be smoothly interpolated to fit and generate a continuously changing energy envelope.

[0068] In some other examples, the full-band energy time-series curve can be obtained directly by integrating the original speech spectrum, eliminating the linear prediction operation step and simplifying the energy envelope generation logic.

[0069] Step S230: Based on the fundamental frequency information, the periodic excitation signal in the original downlink speech signal is replaced with a noise excitation signal whose energy distribution matches the energy envelope, thus obtaining the replaced speech signal.

[0070] This embodiment relies on the pre-extracted fundamental frequency information to locate the periodic excitation signal in the original downlink speech signal that is prone to leakage. All such long-distance propagating periodic sound sources are removed. Then, a noise excitation signal with dynamic loudness changes that are completely synchronized with the original human voice is generated as a replacement sound source. The noise excitation signal is then filled back into the time sequence position of the original periodic excitation signal to obtain a replaced speech signal that removes the speech leakage source and retains the original vocal strength and rhythm. This cuts off the basic conditions for long-distance speech leakage at the sound source level.

[0071] A periodic excitation signal is a signal generated by the vibration of the vocal cords during normal speech production. Its period corresponds to the fundamental frequency and is the root cause of the speaker's pitch characteristics in the speech signal. A periodic excitation signal includes at least the fundamental frequency component generated by the periodic vibration of the vocal cords in the original downlink speech signal, and may also include at least one harmonic component of the fundamental frequency. The periodic excitation signal possesses stable, periodic, low-frequency vibration characteristics and is the core sound source component that is easily picked up by people at a distance when the audio is played through headphones, causing leakage of the call content.

[0072] The noise excitation signal is a sound source replacement component constructed from a non-periodic noise signal. It has no fixed vibration period and does not have a low-frequency harmonic structure that can propagate over long distances. It is used to replace the original periodic excitation signal as a new speech source.

[0073] In some embodiments, the periodic excitation signal in the original downlink speech signal is replaced with a noise excitation signal whose energy distribution matches the energy envelope, based on the baseband information, to obtain a replaced speech signal, including: generating a noise signal; modulating the energy of the noise signal according to the energy envelope to obtain a noise excitation signal whose energy distribution matches the energy envelope; and replacing the periodic excitation signal in the original downlink speech signal with a noise excitation signal based on the baseband information to obtain a replaced speech signal.

[0074] In this embodiment, the noise signal is a broadband random noise waveform with no fixed vibration period and a uniform spectral distribution. This serves as the original base material for constructing the noise excitation signal and does not contain low-frequency harmonic components capable of long-distance propagation. For example, the noise signal can be a broadband noise signal (such as white noise) or a colored noise signal. Specifically, it can be a noise sequence with a flat global spectrum output by a built-in pseudo-random noise generator, with the length of the noise sequence consistent with the total duration of the original downlink speech signal.

[0075] Specifically, white noise has a uniform power spectral density across the entire frequency band, is simple to generate, and is suitable for low-end near-ear wearable devices with limited computing power; colored noise (such as pink noise) has a power spectral density that decreases as the frequency increases, and its spectral characteristics are closer to the excitation characteristics of natural ear speech, resulting in a more natural output sound quality, making it suitable for mid-to-high-end devices with higher sound quality requirements.

[0076] Furthermore, the noise signal can be modulated based on the energy envelope to obtain a noise excitation signal whose energy distribution is completely synchronized with the energy envelope.

[0077] In some examples, the specific implementation of energy modulation can be as follows: for each frame of noise signal, calculate the root mean square energy of that frame, and then divide it by the target energy envelope value (i.e., the root mean square energy of the original speech of the corresponding frame) to obtain the frame-by-frame gain coefficient; multiply the noise signal of that frame by the corresponding gain coefficient so that the root mean square energy of the noise excitation of each frame after modulation is consistent with the original speech energy envelope value.

[0078] In other examples, the specific implementation of energy modulation can also adopt the energy modulation method in the Short-Time Fourier Transform (STFT) domain, which adjusts the energy of each frequency band of the noise signal in the frequency domain to achieve more precise energy matching.

[0079] Energy modulation can preserve the energy fluctuations of speech while eliminating periodic fundamental frequency components, thus helping to maintain speech intelligibility.

[0080] In some examples, based on the fundamental frequency information, the periodic excitation signal in the original downlink speech signal is replaced with a noise excitation signal to obtain the replaced speech signal. This can be done by dividing the speech frame into voiced and unvoiced intervals based on the fundamental frequency information, removing all periodic excitation signals in the voiced frame and filling it with the modulated noise excitation signal; the unvoiced frame is not subjected to excitation replacement processing and the original signal remains unchanged; the voiced frame after splicing is spliced ​​with the unmodified unvoiced frame to obtain the replaced speech signal.

[0081] In these examples, by first generating a uniform noise signal and then performing global amplitude modulation based on the energy envelope, while distinguishing between voiced and unvoiced sound frames for differentiated processing, the original speech loudness dynamics can be accurately matched. Only voiced sound frames that pose a risk of leakage are replaced with signals. This ensures privacy protection while preserving the natural human voice details of unvoiced segments to the greatest extent, thus improving the overall audio listening experience.

[0082] In other examples, the removal of periodic excitation signals can be achieved within a linear predictive coding (LPC) analysis-synthesis framework: first, LPC analysis is performed on each frame of the original downlink speech signal to obtain prediction coefficients; then, the speech frame is filtered using a prediction error filter (i.e., an inverse filter whose coefficients are directly determined by the prediction coefficients) to obtain a residual signal, which is an approximate vocal cord excitation signal; periodicity detection is performed on the residual original downlink speech signal to identify and remove periodic pulse components, thereby achieving the removal of periodic excitation signals from the original downlink speech signal.

[0083] In some other examples, the original downlink speech signal can also be directly comb-filtered in the time domain to remove the fundamental frequency and its harmonic components, thus achieving fundamental frequency removal. This method requires less computation and is suitable for embedded devices with limited computing power.

[0084] By replacing periodic vocal cord excitation with non-periodic noise excitation, the pitch perception feature in speech signals can be fundamentally eliminated, making it impossible for listeners to identify the speaker's identity or speech content through pitch.

[0085] Step S240: Perform spectral shaping on the replaced speech signal based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics.

[0086] This step is the whisper spectral feature shaping stage. The replaced speech signal is used as input, and a filtering model is constructed based on the filter coefficients. The overall spectral distribution of the replaced speech signal is reshaped by filtering, which suppresses the low-frequency components that are easy to propagate over long distances and raises the mid-to-high-frequency components corresponding to airflow friction. The intermediate signal that has only completed the sound source replacement is transformed into an audio waveform with spectral characteristics that fit the real human whisper, resulting in a whisper audio signal that can be played directly.

[0087] Spectrum shaping is a spectrum reshaping operation based on linear prediction all-pole filtering. It replicates the original vocal tract formant distribution of the speaker by relying on the filter coefficients, and synchronously adjusts the energy ratio of each frequency band of the replaced speech signal, weakening low-frequency periodic sound waves and strengthening mid-to-high frequency friction noise to create a spectrum shape unique to whispers.

[0088] The whisper audio signal generated after dual processing of excitation replacement and spectrum shaping has a sound source with non-periodic noise excitation, a spectrum with high-frequency friction characteristics of whispers, and a significantly reduced low-frequency long-distance radiation capability, making it difficult for others to hear the speech content clearly from a distance, while preserving the complete semantics of human voice.

[0089] In some examples, the replaced speech signal is spectrally shaped based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics, including: constructing a synthesis filter based on the filter coefficients; and filtering the replaced speech signal through the synthesis filter to obtain the whispered audio signal.

[0090] Synthetic filters are full-pole digital filters built on the principle of linear prediction. The filter poles and weighting parameters completely reuse the filter coefficients extracted from the original speech, which can replicate the original vocal tract formant distribution of the speaker and be used to perform frequency domain shaping on the replaced speech signal.

[0091] Specifically, the filter coefficients can be assigned to the standard all-pole filtering algorithm framework to complete the configuration of the filter transfer function and generate a synthesized filter that can process audio streams in real time.

[0092] Furthermore, the replaced speech signal can be completely input into the synthesis filter to complete the convolution filtering operation, thereby achieving full-band spectrum shaping. After filtering, the output signal has a spectrum that matches the characteristics of real whispers.

[0093] In these examples, by reusing the filter coefficients extracted from the original speech to construct a synthetic filter, the unique formants and timbre features of the speaker are fully preserved. This can ensure that the speech semantics are clear and indistinguishable while reconstructing the whisper spectrum, without the problems of voice distortion and significant decrease in recognizability.

[0094] In other examples, when constructing the synthetic filter, the low-frequency gain of the filter is attenuated and corrected according to the device's preset whisper intensity parameter. The higher the whisper intensity, the stronger the low-frequency suppression of the filter.

[0095] In these examples, the privacy protection level can be flexibly adjusted by modifying the response curve of the adjustable low-frequency gain filter. In quiet scenes, the low-frequency attenuation is increased to enhance the privacy effect, while in noisy scenes, the attenuation is reduced to improve the listening comfort of human voices, making it suitable for various usage environments.

[0096] In some other examples, the spectral shaping process is not achieved by relying on filter coefficients, but by using a pre-stored whisper spectral mask to perform frequency domain amplitude correction on the replaced speech signal.

[0097] This method eliminates the need to solve and store filter coefficients, and directly performs segmented gain adjustment in the frequency domain. The algorithm is simple to implement and is suitable for miniature single-ear headphones with extremely low storage and computing resources.

[0098] In some other examples, the spectrum shaping process can also be superimposed with a fixed high-frequency boost gain to specifically enhance the energy of the 2kHz~5kHz human voice friction frequency band, further conforming to the characteristics of real whispered airflow, without relying on the frequency response adjustment of the filter itself, and the operation logic is simpler.

[0099] In some embodiments, after performing step S240 and before performing step S250, the method further includes: performing perceptual quality optimization processing on the whispered audio signal.

[0100] This step is a post-audio quality restoration process before audio playback, performing multi-dimensional auditory optimization on the whisper audio signal generated after excitation signal replacement and spectrum reshaping. The whisper audio signal after sound source replacement and spectrum reshaping will have auditory defects such as loss of spectral details, excessive volume dynamic range, and frequency response distortion of headphone hardware; this step corrects the above distortion problems through multiple audio optimization algorithms, smooths out speech imperfections, adapts to the acoustic characteristics of headphone hardware, and finally outputs optimized whisper audio with a natural sound, comfortable volume, and no frequency band distortion, which is then sent to the speaker for playback.

[0101] Perceptual quality optimization processing is an audio correction operation oriented towards human auditory perception. It is used to repair sound quality defects such as spectral distortion, volume imbalance, and hardware frequency response deviation caused by whisper conversion algorithms. It includes at least one of the following optional correction methods: spectrum restoration, dynamic range compression, and frequency response compensation.

[0102] Specifically, the whisper audio signal output by spectrum shaping can be used as input, and at least one of the following processing steps can be selectively performed: spectrum restoration, dynamic range compression, and frequency response compensation. After all corrections are completed, the signal is output to a speaker for playback.

[0103] In some examples, perceptual quality optimization processing is performed on the whispered audio signal, including spectral restoration processing of the whispered audio signal.

[0104] In this embodiment, to address the issues of missing spectral details, blurred formants, and frequency band discontinuity distortion in the whispered audio signal obtained after spectral shaping, a neural network trained offline in advance can be used to complete the details and repair distortion in the whispered spectrum. By learning the complete spectral mapping rules between normal speech and standard whispered speech through the network, the lost subtle spectral features of human voice can be restored, eliminating the dryness and distortion problems caused by conversion, and improving the naturalness of human voice in the whispered audio.

[0105] This neural network model is a lightweight neural network specifically designed for whisper spectrum distortion repair, trained offline under supervised conditions. It uses distorted whisper spectra as input and real standard whisper spectra as supervisory labels to achieve parameter convergence, and is deployed locally on a near-ear wearable device for real-time inference. Specifically, it can collect a massive amount of paired standard whisper samples and filtered distorted whisper samples to form a training set, construct a lightweight convolutional network / small Seq2Seq network, use the MSE loss function for supervised training, and after convergence, solidify the model weights and store them in the near-ear wearable device's memory.

[0106] In these examples, spectrum restoration can compensate for the loss of speech details caused by spectrum shaping filtering. Whispers generated by linear filtering alone lose a lot of subtle human voice friction features, resulting in a stiff and mechanical sound. Spectrum restoration can make up for the details and improve the realism of human voices without compromising privacy noise reduction.

[0107] In other examples, the whispered audio signal is subjected to perceptual quality optimization processing, including dynamic range compression processing.

[0108] In this embodiment, to address the issue of excessive volume dynamic difference in the whispered audio signal obtained after spectrum shaping, the difference between the maximum and minimum audio volume is compressed, the harsh peak volume is reduced, and the volume of the weak vocal low section is increased, so that the volume of the entire whispered audio segment is maintained within the comfortable range for the human ear. This avoids problems such as fluctuating volume, harsh popping sounds, and unclear soft content during playback, resulting in optimized whispered audio with a stable and uniform output volume.

[0109] Dynamic range compression is an audio amplitude adaptive adjustment algorithm. By using four types of compression configuration parameters—compression threshold, compression ratio, attack time, and release time—it attenuates the amplitude of high-volume sampling points that exceed the threshold and slightly increases the gain of weak speech that is below the threshold, thereby reducing the overall audio volume fluctuation range.

[0110] Specifically, the whisper audio signal output by spectrum shaping can be used as input, and the average amplitude of the audio in each frame can be calculated frame by frame. The amplitude is compared with a preset compression threshold, and the peak value exceeding the threshold is attenuated according to a fixed compression ratio. At the same time, the low amplitude speech is slightly amplified, and the amplitude-equalized audio waveform is output frame by frame.

[0111] In these examples, dynamic range compression adapts to the hardware playback limitations of near-ear devices and the characteristics of human hearing. After whispering is converted, the difference in the intensity of human voices is amplified, peak volume can easily trigger speaker distortion, and faint whispers can easily become indistinct. Dynamic range compression unifies the volume range, taking into account both the recognizability of soft human voices and the lack of distortion when playing at high volumes, thus improving the overall listening comfort.

[0112] In other examples, the whispered audio signal is subjected to perceptual quality optimization processing, including frequency response compensation processing.

[0113] The speaker and cavity structure of near-ear wearable devices inherently have uneven frequency response. In the whisper signal, the key human voice frequency band will have gain dips or peaks, resulting in thin, muffled, and high-frequency loss of human voice during playback. This embodiment corrects the gain of each frequency band of the whisper audio by using a preset compensation curve to offset the inherent acoustic distortion of the device, so that the whisper spectrum played out conforms to the standard listening curve and restores a balanced and full human voice listening experience.

[0114] Frequency response compensation is a full-band amplitude correction operation for acoustic hardware defects in near-ear wearable devices. It involves pre-measuring the frequency response defect curve of the entire headphone and generating a corresponding reverse compensation filter curve. Through digital filtering, the gain of the input audio is increased or decreased in different frequency bands to offset the frequency band attenuation or over-amplification caused by the hardware.

[0115] Specifically, the device can retrieve the frequency response compensation filter parameters stored locally, input the whisper audio signal into the compensation filter, adjust the energy of the corresponding frequency components according to the compensation coefficients of each frequency band, and output the hardware-adapted equalized audio waveform.

[0116] In these examples, frequency response compensation processing of whispered audio signals can eliminate the distortion of whispered voice timbre caused by near-ear wearable device hardware. After spectrum shaping and spectrum repair, the ideal whispered spectrum will have frequency band imbalance after being played due to the limitation of the speaker cavity. Frequency response compensation corrects the energy of the entire frequency band, ensuring that the whispered friction details in the mid and high frequencies are clear and the low frequencies are not muffled when the user listens, and unifying the playback sound quality performance of different hardware individuals.

[0117] In some further examples, the whispered audio signal is subjected to perceptual quality optimization processing, including at least two of the following: spectral restoration processing of the whispered audio signal; dynamic range compression processing of the whispered audio signal; and frequency response compensation processing of the whispered audio signal.

[0118] In this embodiment, the execution order of spectrum restoration processing, dynamic range compression processing, and frequency response compensation processing can be preset, and the corresponding sensing quality optimization processing steps can be executed sequentially according to the execution order.

[0119] Step S250: Play the whispered audio signal.

[0120] In this embodiment, the whispered audio signal can be transmitted to the sound-generating hardware unit of the wearable device near the ear for acoustic playback; the user wearing the device can hear the clear human voice normally, while the low-frequency long-distance propagation components of the audio have been greatly suppressed, making it difficult for people nearby to identify the content of the voice, thus realizing the final effect of privacy listening.

[0121] As can be seen from steps S210 to S250 above, this embodiment extracts three key acoustic features from the original downlink speech signal: fundamental frequency information representing the periodic characteristics of the speech, filter coefficients representing the timbre of the human voice, and energy envelope representing the loudness change. Based on the fundamental frequency information, it accurately locates and replaces the periodic excitation signal in the original speech that is easy to propagate over long distances and cause speech leakage. It uses noise excitation that matches the energy envelope to eliminate strong periodic harmonic components, and then reuses the filter coefficients extracted from the same speech to complete spectrum shaping. After shaping the whisper spectrum with low-frequency attenuation and prominent mid-to-high frequency friction characteristics, it is played back. On the one hand, it weakens the long-distance propagation ability of speech from the source of sound excitation, greatly reducing the probability of people around recognizing the content of the speech and achieving efficient speech privacy protection. On the other hand, it reuses the feature parameters extracted from the same source speech to complete the signal transformation without the need for independent parameter calibration of multiple modules. This ensures that the loudness change of the whisper after conversion matches the original sound and the timbre is complete and natural, taking into account both privacy protection effect and human hearing clarity and real hearing. The overall process is lightweight and adaptable to the real-time audio processing needs of near-ear wearable devices.

[0122] <Second Embodiment> This embodiment provides a near-ear wearable device. Figure 3 A schematic diagram of the hardware structure of the near-ear wearable device is shown.

[0123] like Figure 3 As shown, the near-ear wearable device 300 includes a processor 310 and a memory 320. The memory can be used to store computer programs, and the processor can be used to retrieve the computer programs from the memory to execute any method embodiment of this disclosure. The processor can be one or more, and these processors can execute instructions individually or jointly. Similarly, the memory can be one or more, and these memories can store the aforementioned computer programs individually or jointly.

[0124] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods in any embodiment of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto; it may also be a temporary storage medium.

[0125] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0126] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having a computer-readable program loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0127] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0128] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0129] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0130] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0132] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0134] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for voice privacy processing in a near-ear wearable device, characterized in that, The method includes: Acquire the raw downlink speech signal; The original downlink speech signal is analyzed to obtain the fundamental frequency information, filter coefficients, and energy envelope; Based on the fundamental frequency information, the periodic excitation signal in the original downlink speech signal is replaced with a noise excitation signal whose energy distribution matches the energy envelope, to obtain the replaced speech signal; The replaced speech signal is subjected to spectral shaping based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics. Play the whispered audio signal.

2. The method according to claim 1, characterized in that, The step of replacing the periodic excitation signal in the original downlink speech signal with a noise excitation signal whose energy distribution matches the energy envelope, based on the fundamental frequency information, to obtain the replaced speech signal, includes: Generate broadband noise signals; The energy of the broadband noise signal is modulated according to the energy envelope to obtain a noise excitation signal whose energy distribution matches the energy envelope; Based on the fundamental frequency information, the periodic excitation signal in the original downlink speech signal is replaced with the noise excitation signal to obtain the replaced speech signal.

3. The method according to claim 1, characterized in that, The step of performing spectral shaping on the replaced speech signal based on the filter coefficients to obtain a whispered audio signal with ear speech characteristics includes: Construct a synthetic filter based on the filter coefficients; The replaced speech signal is filtered by the synthesis filter to obtain the whisper audio signal.

4. The method according to claim 1, characterized in that, Before playing the whispered audio signal, the method further includes: The whispered audio signal is subjected to perception quality optimization processing.

5. The method according to claim 4, characterized in that, The perceptual quality optimization processing of the whispered audio signal includes: Perform spectral restoration processing on the whispered audio signal; and / or, The whispered audio signal is subjected to dynamic range compression processing; and / or, Frequency response compensation processing is performed on the whispered audio signal.

6. The method according to claim 1, characterized in that, Before analyzing the original downlink speech signal to obtain the fundamental frequency information, filter coefficients, and energy envelope, the method further includes: The original downlink speech signal is then denoised.

7. The method according to claim 1, characterized in that, The method further includes: Collect environmental noise signals; The current ambient noise level is detected based on the ambient noise signal. If the current ambient noise level is lower than a preset noise level threshold, the original downlink voice signal is subjected to privacy processing.

8. The method according to claim 7, characterized in that, The step of detecting the current ambient noise level based on the ambient noise signal includes: The sound pressure level or spectral characteristics of the environmental noise signal are analyzed to obtain the current environmental noise level.

9. A near-ear wearable device, characterized in that, It includes a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1-8.

Images