An adaptive speech extraction and tts adaptation system based on audiogram

Abstract

An adaptive voice extraction and TTS adaptation system based on audiogram includes a sound collection and content extraction module, an audiogram adaptive TTS synthesis module, a wireless transmission and secondary gain module, and an audiogram test tool; the sound collection and content extraction module is used for collecting sound signals in different scenes, and identifying the corresponding speaker identity for each piece of text information; the audiogram test tool tests the hearing of the listener, outputs the audiogram parameters of the listener, the audiogram adaptive TTS synthesis module receives the text information, the corresponding speaker identity and the audiogram parameters, and generates a personalized TTS sound file; the wireless transmission and secondary gain module is in communication connection with the audiogram adaptive TTS synthesis module, and after receiving the TTS sound file, the secondary spectral gain adjustment is carried out according to the audiogram parameters. The present application can effectively shield environmental noise and realize individualized adaptation of audiogram.

Description

Technical Field

[0001] This invention belongs to the field of hearing aid technology, specifically relating to a speech extraction and TTS adaptation system based on audiogram adaptive speech extraction. Background Technology

[0002] In daily life, people with hearing impairments mainly rely on hearing aids to obtain sound information from their surroundings. However, traditional hearing aids have many shortcomings: First, they cannot effectively distinguish between target speech and environmental noise. In complex scenarios (such as crowded outdoor areas or large conferences), they collect various types of noise from the environment simultaneously, making it difficult for people with hearing impairments to distinguish useful dialogue content and seriously affecting their communication experience. Second, individual voices differ, with variations in accents, dialects, and personal habits. People with hearing impairments find it difficult to adapt to these differences, resulting in poor speech recognition for some individuals. Third, existing personalized hearing aids are complex in structure and expensive, making them difficult to widely apply.

[0003] Currently, some research and applications have been conducted on speech-to-text assistance technologies for people with hearing impairments. For example, a single-engine multi-microphone speech-to-text system and role separation method (publication number CN116863966A) can achieve multi-person speech-to-text transcription and speaker differentiation in meeting scenarios through multi-microphone arrays and voiceprint clustering technology. However, this technology has significant limitations: it only completes the unidirectional conversion from speech to text and does not have the ability to convert text back into speech adapted to hearing-impaired individuals. Hearing-impaired individuals still need to rely on visual reading of text and cannot use it in non-visual attention scenarios such as driving and sports. At the same time, the general TTS speech synthesis module it uses does not combine individual audiograms for frequency response and loudness compensation. For people with high-frequency or low-frequency hearing loss, the synthesized speech lacks clarity and recognition, and cannot meet personalized hearing aid needs.

[0004] In addition, existing assistive devices generally suffer from two major common defects: First, their functionality is one-way; most devices only support speech-to-text conversion, lacking closed-loop feedback for text-to-speech conversion, preventing hearing-impaired individuals from receiving information through hearing and severely limiting their usage scenarios. Second, synthesized speech is not personalized; mainstream TTS technology uses a universal standard tone and does not perform frequency band enhancement and dynamic compression tailored to the hearing curves of different hearing-impaired individuals (such as steeply sloping or flat hearing loss), resulting in poor listening experience and low comprehension efficiency for users with different hearing impairments. Furthermore, existing hearing testing tools are mostly professional medical equipment, which are expensive and complex to operate, making it difficult to provide convenient daily hearing parameter testing for hearing-impaired individuals, thus hindering personalized adaptation.

[0005] Therefore, there is an urgent need for a hearing assistance system that can effectively shield environmental noise, achieve personalized adaptation based on audiograms, has low cost, and is widely applicable, in order to solve the aforementioned problems of existing technologies. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention aims to provide an audiogram-based adaptive speech extraction and TTS matching system, which solves the technical problems of traditional hearing aids being prone to collecting ambient noise and being unable to perform personalized matching for different hearing-impaired individuals, as well as the high cost, poor applicability, and inability to achieve multi-speaker differentiation and personalized TTS synthesis of existing assistive devices; the system can effectively shield ambient noise and achieve personalized matching of audiograms.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An audiogram-based adaptive speech extraction and TTS adaptation system includes a sound acquisition and content extraction module, an audiogram-based adaptive TTS synthesis module, a wireless transmission and secondary gain module, and an audiogram testing tool. The sound acquisition and content extraction module is used to acquire sound signals in different scenarios, preprocess the sound signals and separate multiple people speaking, convert the separated single speaker's sound signal into text information, and identify the corresponding speaker's identity for each piece of text information. The audiogram testing tool is used to call open-source hearing analysis software to conduct hearing tests on the listener and output the listener's audiogram parameters. The audiogram parameters include a variety of personalized information, including hearing spectral attenuation data at different frequencies. The audiogram adaptive TTS synthesis module receives the text information, the speaker identity corresponding to the text information identifier, and audiogram parameters. Based on the audiogram parameters, it calculates the spectral distribution and gain of each spectrum of the TTS synthesis file adapted to the listener. Combining the text information and the speaker identity corresponding to the identifier, it generates a personalized TTS sound file. The wireless transmission and secondary gain module receives the TTS audio file and then performs secondary spectral gain adjustment based on the audiogram parameters.

[0008] The scenarios include simple scenarios and complex scenarios. Simple scenarios are partial indoor scenarios with a limited number of people, while complex scenarios are large indoor scenarios or outdoor scenarios with many people. The TTS audio files distinguish the voice styles of men and women, as well as people of different ages (including the elderly, middle-aged, young people, and children).

[0009] The speaker is someone who is having a conversation with a person who has hearing problems; Specifically, it refers to the sounds of people with normal hearing, other objects, or music being played.

[0010] In the sound acquisition and content extraction module: a multi-channel microphone array is used to acquire scene sound signals. Let the acquired original sound signal be... ,in , Number of microphone channels It is a time series; The preprocessing process includes noise reduction and signal normalization. Noise reduction uses the Wiener filtering algorithm, and the formula is: Let the target sound signal after noise reduction be denoised as The formula for the Wiener filtering algorithm is as follows: In the above formula, This is the target audio signal after noise reduction. This represents the optimal frequency response of the Wiener filter. Represents the original sound signal The result after Fourier transform; normalization is performed using the min-max normalization algorithm, the calculation formula of which is: In the above formula, The normalized sound signal , These are the original sound signals. The minimum and maximum values; through the above preprocessing steps, environmental noise and signal amplitude differences can be effectively eliminated, laying the foundation for subsequent multi-speaker separation and speech-to-text processing.

[0011] In the formula, The normalized sound signal , These are the original sound signals. The minimum and maximum values ​​are determined by preprocessing to eliminate environmental noise and signal amplitude differences, laying the foundation for subsequent processing.

[0012] In the sound acquisition and content extraction module: multi-person speech separation adopts a speaker separation algorithm based on Independent Component Analysis (ICA), and the acquired mixed sound signal is set to be composed of... The sound signal is a linear mixture of individual speaker signals, i.e., the normalized sound signal. The mixing model formula is: ; In the above formula, This is the preprocessed multi-channel mixed audio signal matrix, where The number of signal sampling points. This refers to the feature dimension size of each sample point's data; The mixture matrix is ​​unknown; Given a single speaker's voice signal matrix to be separated; the FastICA algorithm is used to solve for the separation matrix. The separated individual speaker signal can then be obtained, and its calculation formula is as follows: ; In the above formula, This is a matrix of individual speaker voice signals after separation, where each row vector corresponds to a speaker's voice signal. This enables effective separation of signals from multiple speakers; In the formula, This is the separated single-speaker voice signal matrix, where each row vector corresponds to the voice signal of one speaker, thus achieving effective separation of multi-speaker signals.

[0013] The speech-to-text and speaker identification components in the sound acquisition and content extraction module will separate the individual speaker voice signals. Input speech-to-text (ASR) model, , corresponding to the For each speaker, the decoding process using the Transformer-based ASR algorithm is as follows: In the above formula, The separated single-speaker speech feature sequence, The output is a sequence of text information. The length of the text sequence. For the first The conditional probability of each character is calculated; simultaneously, the Mel-frequency cepstral coefficients (MFCCs) of each speaker's voice signal are extracted as voiceprint features. Speaker identity matching is achieved through cosine similarity calculation. The formula for calculating the identity identifier is as follows: In the above formula, For the preset speaker voiceprint reference features, This represents the cosine similarity; when the cosine similarity is greater than a preset threshold... (in When the value ranges from 0.75 to 0.85, the speaker's identity can be determined, and the speaker's identity identifier can be bound to the corresponding text information and then output. In the formula, For the preset speaker voiceprint reference features, Cosine similarity is used when the similarity exceeds a preset threshold. ( When the value is between 0.75 and 0.85, the speaker's identity corresponding to the text information is determined, and the identity identifier is bound to the text information and output.

[0014] The wireless transmission and secondary gain module includes a wireless transmission unit mounted on the system host and a Bluetooth headset unit worn by the listener. The wireless transmission unit is communicatively connected to the audiogram adaptive TTS synthesis module and is used to transmit TTS audio files to the Bluetooth headset unit wirelessly via Bluetooth. The Bluetooth headset unit has a built-in gain adjustment module, which, after receiving the TTS audio file, performs secondary spectral gain adjustment based on audiogram parameters to ensure that the listener clearly receives the dialogue content. The secondary spectral gain adjustment is based on audiogram parameters pre-recorded by the listener. These individual audiograms are obtained through professional hearing testing equipment, and the parameters include hearing thresholds for each frequency band. (Unit: dBHL), hearing loss type (such as high-frequency steep drop type, low-frequency loss type, flat type) and hearing sensitivity compensation needs. The adjustment process takes the equalization of the target hearing threshold as the core principle to ensure that the actual perceived sound level of the TTS sound signal after gain adjustment tends to be consistent in each hearing frequency band of the listener. The basis and implementation method of the second-order spectrum gain adjustment are as follows: Step 1. Adjustment Basis: Based on the hearing threshold values ​​of each frequency band in the listener's audiogram. Based on the standard hearing threshold of people with normal hearing, (Refer to GB / T 16403-1996 standard, normal hearing threshold ≤25dB HL), calculate the gain compensation amount for each frequency band to ensure that the perceived sound level of the adjusted sound reaches the auditory perception level of people with normal hearing, while avoiding excessive gain that may cause auditory discomfort or secondary damage. Step 2. Specific Adjustment Calculation: The gain adjustment module has a built-in gain compensation algorithm. First, it performs spectrum analysis on the received TTS audio file to extract the frequency distribution of the audio signal. ( The sound frequency (unit: Hz, covering the key human vocal frequency band of 20Hz~8000Hz) is then used to calculate the target gain value for each frequency band based on audiogram parameters. The calculation formula is: In the above formula, For the corresponding frequency The second gain value (unit: dB) is non-negative (to avoid negative gain causing sound attenuation). For people with normal hearing at frequencies Standard hearing threshold at that location; For listeners on the frequency The actual hearing threshold at that location; This is a safety compensation factor (ranging from 1 to 3 dB), used to reserve a safety margin for hearing and prevent excessive gain from damaging hearing.

[0015] Step 3. Categorized Adaptive Adjustment: Differentiated gain optimization is performed based on the different types of hearing loss reflected in the audiogram: For high-frequency steep-drop hearing loss (significantly elevated hearing threshold in the high-frequency range), the gain value in the 1000Hz~8000Hz frequency range is increased to ensure that consonants (mostly distributed in the high-frequency range) in the TTS sound are clearly distinguishable; for low-frequency hearing loss, the 200Hz~1000Hz frequency range is compensated to ensure the integrity of vowel and intonation information; for flat hearing loss, the gain across the entire frequency range is adjusted according to a uniform compensation ratio to maintain the overall timbre of the sound. Step 4. Real-time calibration mechanism: The gain adjustment module collects feedback signals from the user during the wearing process in real time, and combines this with dynamic changes in the audiogram (such as hearing fluctuations after long-term wear) to adjust the gain value. Dynamic calibration is performed to ensure that the secondary spectrum gain adjustment always matches the listener's actual hearing condition, further improving listening clarity and comfort.

[0016] The sound acquisition and content extraction module includes a single-channel acquisition unit and a multi-channel acquisition unit. The single-channel acquisition unit is used for sound acquisition in simple point-to-point communication scenarios, while the multi-channel acquisition unit is used for sound acquisition in complex conference scenarios. The speech-to-text and speaker separation functions of the sound acquisition and content extraction module are implemented through an on-chip deep learning model or by calling a remote deep learning model.

[0017] The spectral gain calculation process of the audiogram adaptive TTS synthesis module is as follows: Based on the attenuation decibel values ​​of each frequency in the listener's audiogram, the gain value of the corresponding frequency is calculated in reverse to ensure that the synthesized TTS audio file has a higher gain value than the normal frequency range in the frequency range where the listener's hearing is significantly attenuated, and that the overall spectrum distribution conforms to human hearing habits, thus avoiding sound distortion. At the same time, a frequency shifting algorithm is added according to the listener's audiogram distribution to compress high-frequency signals that are not easily distinguishable into a specific low-frequency distribution. The specific parameters are selected based on the personalized audiogram calculation.

[0018] Specifically as follows: Step 1. Inverse calculation of frequency gain: Assume the audiogram spectrum is at a frequency... The measured hearing threshold at the location is (Unit: dBHL), the standard hearing threshold for normal hearing is... The formula for calculating hearing loss at this frequency is: in The formula for calculating the gain compensation value at the corresponding frequency point is: In the formula, The compensation coefficient ranges from 0.8 to 1.2. The base comfort gain is set at 3–6 dB. To avoid distortion and hearing damage, the gain is clamped at upper and lower limits using the following formula: in , .

[0019] Step 2. Frequency Shift Algorithm: Let the original high-frequency components be... The target audible low-frequency range is The frequency shift mapping formula is: In the formula, This is the output frequency after frequency shifting; The personalized frequency shift coefficient is dynamically calculated from the audiogram, using the following formula: in Frequency shifting only applies to the high-frequency, weakly perceived region, without making significant adjustments to the mid- and low-frequency vowel components.

[0020] The wireless transmission unit uses Bluetooth 5.2 or above, supports high-definition audio transmission, and has a transmission latency of ≤100ms, ensuring the real-time transmission of TTS audio files. The secondary gain adjustment of the Bluetooth headset unit adopts the LMS adaptive filtering algorithm, which adjusts the gain of each frequency band in real time according to the audiogram parameters to dynamically adapt to the listener's hearing status. The specific implementation process is as follows: The LMS adaptive filtering algorithm uses the audiogram parameters pre-recorded by the listener as a benchmark and aims to achieve a balanced perceived sound level across all frequency bands. It dynamically updates the filtering coefficients by collecting the sound signal output from the Bluetooth headset unit in real time and the listener's auditory feedback (indirectly through signal amplitude feedback), thereby achieving adaptive adjustment of the gain across all frequency bands. Its core principle is the minimum mean square error criterion, which optimizes the gain adjustment accuracy by minimizing the mean square error between the output signal and the target signal.

[0021] Let the frequency in the listener's audiogram be... The target gain value at that location is The TTS audio signal received by the Bluetooth headphone unit is output from the audiogram adaptive TTS synthesis module mentioned above. , Given a discrete-time series, the input vector of the LMS adaptive filter is: ,in The filter order is 8 to 16 (to suit the needs of human voice frequency band analysis), and the filter coefficient vector is... .

[0022] The output signal of the LMS adaptive filtering algorithm (the audio signal after double gain adjustment) is: The algorithm's error signal is defined as the difference between the output signal corresponding to the target gain and the actual output signal: In the formula, The target output signal is the TTS audio signal. With target gain value Multiplying them together yields the result, i.e. ; To minimize the mean square value of the error signal The filter coefficients are iteratively updated using the gradient descent method, and the coefficient update formula is as follows: In the formula, The step size factor (ranging from 0.001 to 0.01) is used to control the coefficient update speed, avoiding signal distortion due to excessively fast convergence or failure to adapt to hearing changes in a timely manner due to excessively slow convergence. This is the gain adjustment coefficient, ensuring the stability of the iteration process.

[0023] During real-time adjustment, the algorithm extracts the input signal every frame (20~30ms, consistent with the frame length of the audio preprocessing mentioned earlier). The spectral characteristics are matched with the corresponding frequency points in the audiogram, and the target gain value of that frequency point is called. The filter coefficients are updated using the above formula, thereby adjusting the gain of the corresponding frequency band. When the listener's hearing condition fluctuates dynamically (such as auditory fatigue after prolonged use or perceptual differences caused by changes in environmental noise), the algorithm uses error signals... The change in gain is corrected in real time, dynamically adapting to the hearing state, ensuring that the clarity perceived by the listener is consistent across all frequency bands after secondary gain adjustment, while avoiding hearing damage caused by excessive gain.

[0024] The beneficial effects of this invention are: 1. Through the preprocessing unit of the sound acquisition and content extraction module and the multi-speaker separation technology, it can accurately filter out interference signals such as Gaussian white noise and power frequency noise in the environment, separate the voices of different speakers, avoid noise interference with the listening content, and solve the problem of serious noise interference in traditional hearing aids.

[0025] 2. Based on the listener's audiogram parameters, the TTS synthesis module calculates the spectrum and gain to generate a suitable TTS sound file. Combined with the secondary gain adjustment of the Bluetooth headset, it can accurately compensate for the hearing defects (attenuation differences in different frequency bands) of different listeners. It supports the storage of multiple listener parameters and can dynamically switch between them. It is suitable for various types of hearing-impaired people and supports multiple voice styles such as elderly, middle-aged, and children to meet different usage needs.

[0026] 3. It adopts open-source hearing analysis software and lightweight on-chip deep learning models, eliminating the need for expensive professional equipment. The system structure is simple and the hardware cost is controllable. At the same time, it is easy to operate, and the hearing test and parameter configuration process is simple. Ordinary hearing-impaired people can operate it themselves, which is conducive to large-scale popularization and application.

[0027] 4. It features both single-channel and multi-channel acquisition units to adapt to various scenarios, from simple to complex; it supports both offline and online voice processing modes, with on-chip models capable of offline operation without network access, and remote models enhancing processing performance in complex scenarios; wireless transmission utilizes Bluetooth technology, making it convenient to wear and not interfering with the listener's daily activities.

[0028] 5. It enables the differentiation of multiple speakers, text labels, and voice timbre, allowing listeners to clearly distinguish the dialogue content of different speakers; TTS synthesis has strong real-time performance and low transmission latency, ensuring that the sound is synchronized with the scene dialogue, improving the user experience of hearing-impaired people in daily communication, meeting participation, and other scenarios. Attached Figure Description

[0029] Figure 1 This is a diagram of the overall system architecture of the present invention.

[0030] Figure 2 This is a block diagram of the sound acquisition and content extraction module.

[0031] Figure 3 This is an example of an audiogram from an audiogram testing tool.

[0032] Figure 4 This is a flowchart of the audiographic adaptive TTS synthesis module.

[0033] Figure 5 This is a block diagram of the wireless transmission and secondary gain module.

[0034] Figure 6 This is a flowchart of the system operation of the present invention.

[0035] Explanation of reference numerals in the attached figures: Figure 1Chinese: 1-Sound acquisition and content extraction module, 2-Audiograph measurement tool, 3-Audiograph adaptive TTS synthesis, 4-Spectrum adaptation and secondary gain, 5-Bluetooth wireless transmission, 6-Sound playback, 7-System host, 8-Bluetooth headset; Figure 2 In the middle: 11-Single-channel acquisition unit, 12-Multi-channel acquisition unit, 13-Preprocessing unit, 14-Speech separation and text conversion unit; Figure 3 In the middle: the horizontal axis is frequency (Hz), the vertical axis is decibel (dB), A is the air conduction curve of the left ear, B is the air conduction curve of the right ear, C is the bone conduction curve of the left ear, and D is the bone conduction curve of the right ear. Figure 4 In Chinese: S1 - Receive parameters, S2 - Spectrum and gain calculation, S3 - TTS synthesis, S4 - Output TTS file; Figure 5 In the middle: 41-Wireless transmission unit, 42-Bluetooth headset unit, 421-Audio decoding module, 422-Gain adjustment module; Figure 6 In the process: Step 1 - Hearing test and parameter storage, Step 2 - Sound acquisition and preprocessing, Step 3 - Multi-speaker separation and speech-to-text conversion, Step 4 - Personalized TTS synthesis, Step 5 - Bluetooth transmission, Step 6 - Secondary gain and playback. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings.

[0037] An audiogram-based adaptive speech extraction and TTS adaptation system includes a sound acquisition and content extraction module, an audiogram-based adaptive TTS synthesis module, a wireless transmission and secondary gain module, and an audiogram testing tool. The sound acquisition and content extraction module is used to acquire sound signals in different scenarios, preprocess the sound signals and separate multiple people speaking, convert the separated single speaker's sound signal into text information, and identify the corresponding speaker's identity for each piece of text information. The audiogram testing tool is used to call open-source hearing analysis software to conduct hearing tests on listeners and output the listener's audiogram parameters, which include various personalized information such as hearing spectral attenuation data at different frequencies. The audiogram adaptive TTS synthesis module is communicatively connected to the audiogram testing tool and the sound acquisition and content extraction module. It is used to receive the text information, identify the corresponding speaker identity and audiogram parameters, calculate the spectral distribution and gain of each spectrum of the TTS synthesis file adapted to the listener based on the spectral attenuation data in the audiogram parameters, and generate a personalized TTS sound file by combining the text information and the corresponding speaker identity.

[0038] The scenarios include simple scenarios and complex scenarios. Simple scenarios are partial indoor scenarios with a limited number of people, while complex scenarios are large indoor scenarios or outdoor scenarios with many people. The TTS audio files distinguish the voice styles of men and women, as well as people of different ages (including the elderly, middle-aged, young people, and children).

[0039] In the sound acquisition and content extraction module: a multi-channel microphone array is used to acquire scene sound signals. Let the acquired original sound signal be... ,in , Number of microphone channels It is a time series; The preprocessing process includes noise reduction and signal normalization. Noise reduction uses the Wiener filtering algorithm, and the formula is: Let the target sound signal after noise reduction be denoised as The formula for the Wiener filtering algorithm is as follows: In the above formula, This represents the optimal frequency response of the Wiener filter. Represents the original sound signal The result after Fourier transform; normalization is performed using the min-max normalization algorithm, the calculation formula of which is: In the above formula, The normalized sound signal , These are the original sound signals. The minimum and maximum values; through the above preprocessing steps, environmental noise and signal amplitude differences can be effectively eliminated, laying the foundation for subsequent multi-speaker separation and speech-to-text processing.

[0040] In the formula, The normalized sound signal , These are the original sound signals. The minimum and maximum values ​​are determined by preprocessing to eliminate environmental noise and signal amplitude differences, laying the foundation for subsequent processing.

[0041] In the sound acquisition and content extraction module: multi-person speech separation adopts a speaker separation algorithm based on Independent Component Analysis (ICA), and the acquired mixed sound signal is set to be composed of... The sound signal is a linear mixture of individual speaker signals, i.e., the normalized sound signal. The mixing model formula is: The mathematical expression for the hybrid model is: ; In the above formula, This is the preprocessed multi-channel mixed audio signal matrix, where This represents the number of signal sampling points. The mixture matrix is ​​unknown; Given a single speaker's voice signal matrix to be separated; the FastICA algorithm is used to solve for the separation matrix. The separated individual speaker signal can then be obtained, and its calculation formula is as follows: ; In the above formula, This is a matrix of individual speaker voice signals after separation, where each row vector corresponds to a speaker's voice signal. This enables effective separation of signals from multiple speakers; In the formula, This is the separated single-speaker voice signal matrix, where each row vector corresponds to the voice signal of one speaker, thus achieving effective separation of multi-speaker signals.

[0042] In the sound acquisition and content extraction module: speech-to-text and speaker identification are used to separate the individual speaker's voice signal. Input speech-to-text (ASR) model, , corresponding to the For each speaker, the decoding process using the Transformer-based ASR algorithm is as follows: The ASR algorithm based on Transformer has the following decoding process formula: In the above formula, The separated single-speaker speech feature sequence, The output is a sequence of text information. The length of the text sequence. For the first The conditional probability of each character is calculated; simultaneously, the Mel-frequency cepstral coefficients (MFCCs) of each speaker's voice signal are extracted as voiceprint features. Speaker identity matching is achieved through cosine similarity calculation. The formula for calculating the identity identifier is as follows: In the above formula, For the preset speaker voiceprint reference features, This represents the cosine similarity; when the cosine similarity is greater than a preset threshold... (in When the value ranges from 0.75 to 0.85, the speaker's identity can be determined, and the speaker's identity identifier can be bound to the corresponding text information and then output. In the formula, For the preset speaker voiceprint reference features, Cosine similarity is used when the similarity exceeds a preset threshold. ( When the value is between 0.75 and 0.85, the speaker's identity corresponding to the text information is determined, and the identity identifier is bound to the text information and output.

[0043] The wireless transmission and secondary gain module includes a wireless transmission unit mounted on the system host and a Bluetooth headset unit worn by the listener. The wireless transmission unit is communicatively connected to the audiogram adaptive TTS synthesis module and is used to transmit TTS audio files to the Bluetooth headset unit wirelessly via Bluetooth. The Bluetooth headset unit has a built-in gain adjustment module, which, after receiving the TTS audio file, performs secondary spectral gain adjustment based on audiogram parameters to ensure that the listener clearly receives the dialogue content. The secondary spectral gain adjustment is based on the individual audiogram pre-recorded by the listener. This individual audiogram is obtained through professional hearing testing equipment, and its parameters include hearing threshold values ​​for each frequency band. (Unit: dBHL), hearing loss type (such as high-frequency steep drop type, low-frequency loss type, flat type) and hearing sensitivity compensation needs. The adjustment process takes the equalization of the target hearing threshold as the core principle to ensure that the actual perceived sound level of the TTS sound signal after gain adjustment tends to be consistent in each hearing frequency band of the listener. The specific adjustment criteria and implementation methods are as follows: Step 1. Adjustment Basis: Based on the hearing threshold values ​​of each frequency band in the listener's audiogram. Based on the standard hearing threshold of people with normal hearing, (Refer to GB / T 16403-1996 standard, normal hearing threshold ≤25dB HL), calculate the gain compensation amount for each frequency band to ensure that the perceived sound level of the adjusted sound reaches the auditory perception level of people with normal hearing, while avoiding excessive gain that may cause auditory discomfort or secondary damage. Step 2. Specific Adjustment Calculation: The gain adjustment module has a built-in gain compensation algorithm. First, it performs spectrum analysis on the received TTS audio file to extract the frequency distribution of the audio signal. ( The sound frequency (unit: Hz, covering the key human vocal frequency band of 20Hz~8000Hz) is then used to calculate the target gain value for each frequency band based on audiogram parameters. The calculation formula is: In the above formula, For the corresponding frequency The second gain value (unit: dB) is non-negative (to avoid negative gain causing sound attenuation). For people with normal hearing at frequencies Standard hearing threshold at that location; For listeners on the frequency The actual hearing threshold at that location; This is a safety compensation factor (ranging from 1 to 3 dB), used to reserve a safety margin for hearing and prevent excessive gain from damaging hearing.

[0044] Step 3. Categorized Adaptive Adjustment: Differentiated gain optimization is performed based on the different types of hearing loss reflected in the audiogram: For high-frequency steep-drop hearing loss (significantly elevated hearing threshold in the high-frequency range), the gain value in the 1000Hz~8000Hz frequency range is increased to ensure that consonants (mostly distributed in the high-frequency range) in the TTS sound are clearly distinguishable; for low-frequency hearing loss, the 200Hz~1000Hz frequency range is compensated to ensure the integrity of vowel and intonation information; for flat hearing loss, the gain across the entire frequency range is adjusted according to a uniform compensation ratio to maintain the overall timbre of the sound. Step 4. Real-time calibration mechanism: The gain adjustment module collects feedback signals from the user during the wearing process in real time, and combines this with dynamic changes in the audiogram (such as hearing fluctuations after long-term wear) to adjust the gain value. Dynamic calibration is performed to ensure that the secondary spectrum gain adjustment always matches the listener's actual hearing condition, further improving listening clarity and comfort.

[0045] The sound acquisition and content extraction module includes a single-channel acquisition unit and a multi-channel acquisition unit. The single-channel acquisition unit is used for sound acquisition in simple point-to-point communication scenarios, while the multi-channel acquisition unit is used for sound acquisition in complex conference scenarios. The speech-to-text and speaker separation functions of the sound acquisition and content extraction module are implemented through an on-chip deep learning model or by calling a remote deep learning model.

[0046] The preprocessing process of the sound acquisition and content extraction module includes: framing the acquired sound signal, with each frame lasting 20-30ms and an overlap rate of 50%. Specifically, a sliding window method is used for framing, meaning the overlap between the previous and subsequent frames accounts for 50% of the frame duration, effectively preventing signal breaks between frames and ensuring signal continuity. After framing, a Hamming window is applied to each frame to reduce spectral truncation. Then, the time-domain sound signal is converted to a frequency-domain signal using FFT (Fast Fourier Transform). Finally, a targeted filtering algorithm is used to identify and filter out environmental interference signals such as Gaussian white noise and power frequency noise. The detailed process is as follows: 1. Hamming Window Processing: Because audio signals are non-stationary, truncation distortion occurs at the edges of each frame after frame-by-frame processing, leading to spectral leakage (spurious frequency components caused by abrupt changes in edge signals) during spectrum analysis. Therefore, a Hamming window needs to be applied to each frame for smoothing to suppress spectral leakage. The time-domain expression of the Hamming window is: In the above formula, For the Hanming window in the first Window function values ​​for each sampling point The number of sampling points for a single frame signal (calculated based on a sampling rate of 16kHz, a 20ms frame length corresponds to 320 sampling points, and a 30ms frame length corresponds to 480 sampling points); by analyzing the time-domain signal of each frame... With Hanming Window Perform point-by-point multiplication (i.e.) This allows the amplitude of the signal at the frame edge to decay smoothly to 0, effectively reducing the spectral edge truncation effect and improving the accuracy of subsequent frequency domain analysis.

[0047] 2. FFT Transform Processing: Even after Hamming window smoothing, the noise components in each frame of the time-domain signal cannot be directly identified. Therefore, an FFT transform is needed to convert it into a frequency-domain signal, achieving a time-domain to frequency-domain mapping, facilitating subsequent noise spectrum identification and filtering. The FFT transform is based on a fast algorithm for the Discrete Fourier Transform (DFT), which significantly reduces computational complexity. Its core transform formula is: In the above formula, For the first The frequency domain signal amplitude at each frequency point The time-domain signal after Hamming window processing. The imaginary unit, This refers to the number of sampling points per frame. For frequency point indexing; through FFT transformation, each frame of time-domain audio signal can be converted into a frequency-domain signal containing frequency and amplitude information, where the frequency resolution is... ( The sampling rate is set to 16kHz to ensure accurate capture of the difference between the human voice frequency band (200Hz~3400Hz) and the noise frequency band, providing a basis for noise filtering.

[0048] 3. Environmental Interference Signal Filtering: Based on the frequency domain signal obtained by FFT transform, and combined with the spectral characteristics of different types of noise, targeted filtering algorithms are used to identify and filter out interference signals. Specifically, this is divided into two types of noise filtering: (1) Gaussian white noise filtering: Gaussian white noise has a uniform distribution across the entire frequency band, with random amplitude and no fixed frequency peak. It is filtered out using the Wiener filtering algorithm. First, the power spectral density of Gaussian white noise is estimated by collecting signals during periods of silence. Then, combined with the power spectral density of the target sound signal Calculate the frequency response of the Wiener filter. frequency domain signal and Multiplying them yields the frequency domain signal after filtering out Gaussian white noise. This enables adaptive filtering of Gaussian white noise.

[0049] (2) Power frequency noise filtering: Power frequency noise mainly consists of 50Hz power frequency signals and their harmonics (100Hz, 150Hz, etc.), with a spectral characteristic of peak amplitude at fixed frequency points. Notch filters are used for targeted filtering. Notch filters with center frequencies of 50Hz, 100Hz, and 150Hz are designed, and their frequency response satisfies the requirement that the amplitude attenuation at the center frequency is ≥40dB and the amplitude attenuation in the human voice frequency band (200Hz~3400Hz) is ≤1dB to avoid damaging the target sound signal; the frequency domain signal is converted into a signal with a frequency domain frequency of ≥40dB. By using a notch filter to remove power frequency and harmonic interference, a clean frequency domain target sound signal is finally obtained, providing high-quality signal support for subsequent multi-speaker separation and speech-to-text processing.

[0050] The multi-speaker speech separation employs a deep learning model based on CNN / RNN, with DeepFilterNet 3 being the preferred choice. It fully integrates the FFT frequency domain analysis results from the preceding section to conduct phased processing. First, coarse frequency domain separation is used to initially segment the speakers. Then, a neural network is used to refine the processing of residual noise and signal aliasing, ultimately achieving accurate separation of the voices of different speakers. The specific process is as follows: (1) Model structure configuration: The CNN / RNN deep learning model consists of a feature extraction layer, a CNN frequency domain coarse separation layer, an RNN fine processing layer, and an output layer; wherein the feature extraction layer receives the clean frequency domain signal output after the preprocessing mentioned above, and converts it into a Mel-spectrum feature map (with dimensions [batch_size, time_step, ...). [mel_bins], where mel_bins is set to 128 and time_step corresponds to the frame sequence length), serves as the input feature for subsequent separation processing to ensure the effectiveness and specificity of the features; the CNN frequency domain coarse separation layer uses a 3-layer convolutional neural network with kernel sizes of 3×3, 3×3, and 2×2, and a stride of 1 for each layer. Padding is used to keep the feature map size unchanged to capture the frequency feature differences between different speakers in the frequency domain signal; the RNN fine processing layer uses a bidirectional long short-term memory network (Bi-LSTM) with 2 hidden layers, each with 256 hidden units, to capture the temporal dependencies of the signal and correct the signal aliasing problem that occurs during the coarse separation process; the output layer uses a fully connected layer combined with a sigmoid activation function to output the separation signal mask corresponding to each speaker.

[0051] (2) FFT coarse separation in the frequency domain: based on the frequency domain signal obtained by the FFT transformation mentioned above. First, calculate the power spectrum of the signal. And combined with the Mel scale conversion formula The frequency domain power spectrum is converted into a Mel power spectrum and input into the CNN frequency domain coarse separation layer. The CNN layer extracts features of different speakers in different frequency bands (such as the fundamental frequency and overtone frequency differences of different speakers) through convolution operations, generating a preliminary speaker separation mask. ( (For time frame index), compare the mask with the original frequency domain signal Multiplying them together yields the preliminary separated frequency domain signals of each speaker. ( For speaker indexing, For the first (coarse separation mask for each speaker) to complete the coarse separation process, initially dividing the signal range of different speakers and eliminating obvious signal aliasing.

[0052] (3) Refinement of DeepFilterNet model: The frequency domain signals of each speaker obtained from coarse separation are processed. The signal is converted back to the time domain and input into the DeepFilterNet 3 model for fine-tuning. This model adaptively corrects for the weak noise, signal aliasing, and spectral distortion remaining after coarse separation. Specifically, the DeepFilterNet 3 model includes a noise estimation branch and a signal enhancement branch. The noise estimation branch analyzes the temporal characteristics of the time domain signal through an RNN layer, accurately identifying residual environmental noise (such as weak background noise) and interference signals from other speakers after coarse separation, and outputs a noise estimate. The signal enhancement branch combines the frequency domain features extracted by CNN and the temporal features extracted by RNN to perform spectral compensation and amplitude calibration on the coarsely separated signal, suppressing residual noise while repairing the signal amplitude distortion caused by coarse separation.

[0053] (4) Precise separation output: After the fine processing is completed, the frequency domain signal of each speaker is converted back to the time domain signal by the inverse FFT transformation to obtain a pure single speaker voice signal. Meanwhile, the model output confidence threshold (set to 0.8) is used to filter valid signals and remove invalid interference segments, ensuring that the separated speaker's voice signal has no obvious aliasing or noise interference and has a stable amplitude, providing accurate signal input for subsequent speech-to-text conversion and speaker identification.

[0054] The frequency range of the audiogram parameters covers 125Hz-6000Hz, including 18 key frequency points: 125Hz, 150Hz, 200Hz, 250Hz, 315Hz, 400Hz, 500Hz, 630Hz, 800Hz, 1000Hz, 1250Hz, 1600Hz, 2000Hz, 2500Hz, 3150Hz, 4000Hz, 5000Hz, and 6000Hz. The decibel values ​​range from -10dB to 120dB, and the test is conducted in 5dB increments. The above 18 frequency points are designed around the needs of key speech frequency band coverage and refined frequency measurement, and are adapted to the core application scenarios of this system for speech interaction for hearing-impaired people. The specific explanation is as follows: The core function of this system is to realize efficient interaction between hearing-impaired people and external speech. The key frequency range of human speech is mainly concentrated in 125Hz-6000Hz. This frequency band covers all the characteristic frequencies of vowels (mainly distributed in 125Hz-1000Hz) and consonants (mainly distributed in 1000Hz-6000Hz), which is the core frequency band for hearing-impaired people to understand speech content. Compared to conventional audiometry (which typically uses 8-12 frequency points), this solution increases the number of frequency points to 18, focusing on encrypting the frequency bands with dense speech features. Specifically, 8 frequency points are set in the 125Hz-1000Hz primary frequency band (125Hz, 150Hz, 200Hz, 250Hz, 315Hz, 400Hz, 500Hz, 630Hz, 800Hz, 1000Hz, including 10 points), and 8 frequency points are set in the 1000Hz-6000Hz secondary frequency band (1250Hz, 1600Hz, 2000Hz, 2500Hz, 3150Hz, 4000Hz, 5000Hz, 6000Hz). By encrypting the frequency points, the differences in hearing thresholds of hearing-impaired individuals in different speech frequency bands can be captured more accurately, avoiding the fitting deviation of the hearing curve caused by excessively large frequency point intervals in conventional audiometry. Meanwhile, the setting of 18 frequency points does not exceed the reasonable range of conventional hearing tests, and the test accuracy of decibel values ​​in 5dB increments can be combined with the fine setting of frequency points. This can accurately obtain the individual audiograms of hearing-impaired individuals, providing more accurate parameter support for subsequent TTS synthesis and secondary spectrum gain adjustment, while also taking into account testing efficiency and avoiding a lengthy testing process due to too many frequency points. It is fully adapted to the actual application needs of this system.

[0055] The spectral gain calculation process of the audiogram adaptive TTS synthesis module is as follows: Based on the attenuation decibel values ​​of each frequency in the listener's audiogram, the gain value of the corresponding frequency is calculated in reverse to ensure that the synthesized TTS audio file has a higher gain value than the normal frequency range in the frequency range where the listener's hearing is significantly attenuated, and that the overall spectrum distribution conforms to human hearing habits, thus avoiding sound distortion. At the same time, a frequency shifting algorithm is added according to the listener's audiogram distribution to compress high-frequency signals that are not easily distinguishable into a specific low-frequency distribution. The specific parameters are selected based on the personalized audiogram calculation.

[0056] Specifically as follows: Step 1. Inverse calculation of frequency gain: Assume the audiogram spectrum is at a frequency... The measured hearing threshold at the location is (Unit: dBHL), the standard hearing threshold for normal hearing is... The formula for calculating hearing loss at this frequency is: in The formula for calculating the gain compensation value at the corresponding frequency point is: In the formula, The compensation coefficient ranges from 0.8 to 1.2. The base comfort gain is set at 3–6 dB. To avoid distortion and hearing damage, the gain is clamped at upper and lower limits using the following formula: in , .

[0057] Step 2. Frequency Shift Algorithm: Let the original high-frequency components be... The target audible low-frequency range is The frequency shift mapping formula is: In the formula, This is the output frequency after frequency shifting; The personalized frequency shift coefficient is dynamically calculated from the audiogram, using the following formula: in Frequency shifting only applies to the high-frequency, weakly perceived region, without making significant adjustments to the mid- and low-frequency vowel components.

[0058] The wireless transmission unit uses Bluetooth 5.2 or above, supports high-definition audio transmission, and has a transmission latency of ≤100ms, ensuring the real-time transmission of TTS audio files. The secondary gain adjustment of the Bluetooth headset unit adopts the LMS adaptive filtering algorithm, which adjusts the gain of each frequency band in real time according to the audiogram parameters to dynamically adapt to the listener's hearing status. The specific implementation process is as follows: The LMS adaptive filtering algorithm uses the audiogram parameters pre-recorded by the listener as a benchmark and aims to achieve a balanced perceived sound level across all frequency bands. It dynamically updates the filtering coefficients by collecting the sound signal output from the Bluetooth headset unit in real time and the listener's auditory feedback (indirectly through signal amplitude feedback), thereby achieving adaptive adjustment of the gain across all frequency bands. Its core principle is the minimum mean square error criterion, which optimizes the gain adjustment accuracy by minimizing the mean square error between the output signal and the target signal.

[0059] Let the frequency in the listener's audiogram be... The target gain value at that location is The TTS audio signal received by the Bluetooth headphone unit is output from the audiogram adaptive TTS synthesis module mentioned above. , Given a discrete-time series, the input vector of the LMS adaptive filter is: ,in The filter order is 8 to 16 (to suit the needs of human voice frequency band analysis), and the filter coefficient vector is... .

[0060] The output signal of the LMS adaptive filtering algorithm (the audio signal after double gain adjustment) is: The algorithm's error signal is defined as the difference between the output signal corresponding to the target gain and the actual output signal: In the formula, The target output signal is the TTS audio signal. With target gain value Multiplying them together yields the result, i.e. ; To minimize the mean square value of the error signal The filter coefficients are iteratively updated using the gradient descent method, and the coefficient update formula is as follows: In the formula, The step size factor (ranging from 0.001 to 0.01) is used to control the coefficient update speed, avoiding signal distortion due to excessively fast convergence or failure to adapt to hearing changes in a timely manner due to excessively slow convergence. This is the gain adjustment coefficient, ensuring the stability of the iteration process.

[0061] During real-time adjustment, the algorithm extracts the input signal every frame (20~30ms, consistent with the frame length of the audio preprocessing mentioned earlier). The spectral characteristics are matched with the corresponding frequency points in the audiogram, and the target gain value of that frequency point is called. The filter coefficients are updated using the above formula, thereby adjusting the gain of the corresponding frequency band. When the listener's hearing condition fluctuates dynamically (such as auditory fatigue after prolonged use or perceptual differences caused by changes in environmental noise), the algorithm uses error signals... The change in gain is corrected in real time, dynamically adapting to the hearing state, ensuring that the clarity perceived by the listener is consistent across all frequency bands after secondary gain adjustment, while avoiding hearing damage caused by excessive gain.

[0062] This system adopts a dual-model collaborative architecture that combines an on-chip lightweight deep learning model with a remote high-performance model. It automatically switches models based on scene complexity, network status, and hardware resource load, ensuring basic offline functions while meeting the requirements for high-precision speech recognition and speaker separation in complex scenarios.

[0063] The on-chip deep learning model is a lightweight speech recognition and speaker separation model. It is optimized for the computing power, storage and power consumption constraints of embedded hardware platforms. It adopts a lightweight convolutional neural network and recurrent neural network structure, and reduces the amount of computation and parameters through model quantization, weight pruning and operator fusion. It can independently complete speech feature extraction, speech-to-text and multi-speaker separation processing in offline state without network support, ensuring the availability of basic functions of the system in the absence of network environment.

[0064] The remote model is deployed in the cloud or on-chip computing nodes, establishing a data communication link with the system host through a network interface. It employs a high-capacity deep neural network structure, possessing stronger anti-interference capabilities, overlapping speech separation capabilities, and semantic parsing accuracy. It is suitable for complex application scenarios with high computing power requirements, such as multi-person overlapping speech, strong environmental noise, and long-distance sound pickup. The remote model and the on-chip model use a unified input / output interface and data format, achieving bidirectional compatibility between feature-level and result-level aspects.

[0065] The system features a scenario-adaptive scheduling unit to monitor network connectivity, transmission bandwidth, hardware load, and the complexity of the current speech scenario in real time. Based on preset strategies, it automatically switches model invocation methods: when the network is disconnected, the signal is weak, or the scenario is simple with few speakers, the scheduling unit uses the on-chip model for offline real-time processing; when the network is smooth, transmission is stable, and the scenario complexity is high, it automatically switches to a remote model, uploading pre-processed speech features to a remote end for high-precision inference calculations; in scenarios with network fluctuations, a collaborative mode of real-time output from the on-chip model and background correction from the remote model is adopted to correct and enhance the offline processing results, balancing processing latency and recognition accuracy.

[0066] Through the aforementioned dual-model collaborative working mechanism, the system not only has the ability to operate independently offline on the embedded platform, but also can achieve high-performance speech processing in complex scenarios with the help of remote computing power, thereby improving the system's adaptability, reliability and recognition accuracy in different application environments.

[0067] The audiogram testing tool supports separate testing for the left and right ears and can generate a visual audiogram. It is drawn using an electro-audiogram tool library (such as electro-audiogram) and clearly shows the hearing loss at each frequency point. The test data can be exported and stored for subsequent TTS synthesis and gain adjustment parameter calls.

[0068] Example 1: This embodiment provides a speech extraction and TTS adaptation system based on auditory spectrogram adaptive speech extraction, such as... Figure 1 As shown, it includes a sound acquisition and content extraction module 1, an audiogram measurement tool 2, an audiogram adaptive TTS synthesis 3, a spectrum adaptation and secondary gain 4, a Bluetooth wireless transmission 5, a sound playback 6, a system host 7, and a Bluetooth headset 8.

[0069] The sound acquisition and content extraction module 1, the audiogram measurement tool 2, the audiogram adaptive TTS synthesis 3, and the Bluetooth wireless transmission 5 are components of the system host 7. The spectrum adaptation and secondary gain 4, the Bluetooth wireless transmission 5, and the sound playback 6 are components of the Bluetooth headset 8.

[0070] 1. Sound Acquisition and Content Extraction Module 1: The single-channel acquisition unit 11 uses a high-sensitivity electret microphone with a sampling rate of 16kHz and a frequency response range of 20Hz-20kHz, suitable for simple scenarios such as family conversations and one-on-one communication; the multi-channel acquisition unit 12 uses a 4-channel array microphone group arranged in a square with a spacing of 10cm and a sampling rate of 48kHz, suitable for complex scenarios such as conference rooms and outdoor squares, and can achieve 360° sound acquisition, improving the sound acquisition coverage in complex scenarios.

[0071] The preprocessing unit 13 processes the acquired audio signal: it divides the continuous audio signal into frames, each frame lasting 25ms with an overlap rate of 50%, and applies a Hamming window; it converts the time-domain signal into a frequency-domain signal using FFT, analyzes the spectral characteristics, and identifies 50Hz power frequency noise and full-band Gaussian white noise; it uses an FIR linear phase filter with a low-pass cutoff frequency of 3.4kHz (corresponding to the main energy range of the speech signal) and a transition band width of 100Hz to filter out high-frequency noise and power frequency noise, while retaining the target speech signal.

[0072] The speech separation and text conversion unit 14 employs the DeepFilterNet deep learning model, combined with FFT frequency domain coarse separation, to separate the voices of multiple speakers and identify the fundamental frequency characteristics of different speakers (male voice 80-150Hz, female voice 150-300Hz), achieving accurate separation. The speech-to-text function uses a lightweight on-chip model (such as Whisper Tiny), adapted to embedded hardware (such as STM32H743), and can work offline with a recognition accuracy of ≥95%. Simultaneously, it identifies the speaker for each text segment, such as "Speaker 1: The weather is very nice today" and "Speaker 2: Yes, it's suitable for outdoor activities." This module also has an auxiliary memory function, which can record the conversational interaction process between the listener and different people in real time, sort out the communication events through built-in analysis algorithms, and set timed reminders to meet the needs of elderly users and help them remember important communication content. Simultaneously, it collects the voice characteristics of the communication objects and stores them on the system host's SD card, providing voice style data for TTS synthesis.

[0073] 2. Audiogram Testing Tool 2: The system utilizes open-source hearing analysis software and the electro-audiogram tool library to perform hearing tests on the display screen of the system host 5. During the test, the listener wears test headphones, and the system plays test sounds at different frequencies and decibels. The listener uses buttons to indicate whether they hear the sound, gradually adjusting the decibel value to obtain the hearing threshold at each frequency point. The test frequencies include 18 key frequency points: 125Hz, 150Hz, 200Hz, 250Hz, 315Hz, 400Hz, 500Hz, 630Hz, 800Hz, 1000Hz, 1250Hz, 1600Hz, 2000Hz, 2500Hz, 3150Hz, 4000Hz, 5000Hz, and 6000Hz. The decibel value starts from -10dB and increases in 5dB increments until the listener reports hearing the sound, at which point the attenuation decibel value is recorded.

[0074] After the test is completed, a visual audiogram is generated using the electro-audiogram toolkit, such as... Figure 3 As shown, the right ear uses a solid red line (air conduction) and a dashed red line (bone conduction), while the left ear uses a solid blue line (air conduction) and a dashed blue line (bone conduction). Test data is stored on the SD card of the system host, supporting parameter storage for multiple users (up to 10). When switching users, the audiogram parameters of the corresponding user can be selected through the buttons on the system host.

[0075] 3. Audiographic Adaptive TTS Synthesis Module 3: The system receives text information, speaker identification, and audiogram parameters output from the sound acquisition and content extraction module 1, and simultaneously retrieves the speech characteristic data of the communication subject stored in the module. Based on the audiogram parameters, a linear interpolation algorithm is used to calculate the gain value for each frequency band. For example, if the listener experiences a 25dB attenuation in the 1000Hz frequency band and a 30dB attenuation in the 2000Hz frequency band, then in the 1000-2000Hz frequency band, the gain value linearly transitions from 25dB to 30dB, ensuring a smooth spectrum and avoiding sound distortion. The TTS synthesis uses an open-source TTS engine (such as Coqui). The system (TTS) distinguishes between men and women and supports eight basic voice styles, including the elderly, middle-aged, young adults, and children. It can also adjust the speech rate of different voice styles according to the listener's habits. Furthermore, it can combine the voice characteristics of the communication partner to generate TTS voices that match the style of the communication partner. This means that it takes into account both the listener's audiogram adaptation and the speaker's voice style restoration. At the same time, it assigns corresponding timbres to different speakers (e.g., speaker 1 uses a middle-aged male voice, speaker 2 uses a middle-aged female voice), thereby improving the naturalness of communication.

[0076] 4. Bluetooth wireless transmission unit 5: The wireless transmission unit 41 uses a Bluetooth 5.2 module (such as nRF52840), supports the A2DP high-definition audio transmission protocol, and has a transmission latency of ≤80ms, ensuring that the TTS sound is synchronized with the scene dialogue; the Bluetooth headset unit 42 uses an in-ear Bluetooth headset, with a built-in CSR8675 Bluetooth chip and an audio decoding module 421, which supports audio formats such as MP3 and WAV. 5. Spectrum Adaptation and Secondary Gain Module 4: The gain adjustment module 422 uses the LMS adaptive filtering algorithm with a sampling frequency of 16kHz. Based on the audiogram parameters, it performs secondary gain adjustment on each frequency band of the TTS audio file with an adjustment accuracy of 1dB, further compensating for hearing loss and ensuring that the listener can clearly hear all the dialogue content. The Bluetooth headset also has volume adjustment buttons and voice style switching buttons for easy operation.

[0077] The system workflow of this embodiment is as follows: Figure 6 As shown: Step 1: The listener conducts a hearing test using the audiogram testing tool 2 on the system host 5, obtains and stores the audiogram parameters; Step 2: The sound acquisition and content extraction module 1 acquires scene sound using its acquisition unit (single-channel or multi-channel), and the preprocessing unit 13 filters out environmental noise; Step 3: Speech separation and text conversion unit 14 performs multi-speaker separation on the preprocessed audio and converts it into text information with speaker identifiers; Step 4: The audiogram adaptive TTS synthesis module 3 calculates the spectral distribution and gain based on the text information, speaker identification, and audiogram parameters to generate a personalized TTS sound file; Step 5: The wireless transmission unit 41 transmits the TTS audio file to the Bluetooth headset 6 via Bluetooth; Step 6: The audio decoding module 421 of the Bluetooth headset 6 decodes the TTS file, the gain adjustment module 422 performs secondary gain adjustment, and plays it to the listener to complete the hearing assistance.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech extraction and TTS adaptation system based on auditory spectrograms, characterized in that, It includes a sound acquisition and content extraction module, an audiogram adaptive TTS synthesis module, a wireless transmission and secondary gain module, and an audiogram testing tool; The sound acquisition and content extraction module is used to acquire sound signals in different scenarios, preprocess the sound signals and separate multiple people speaking, convert the separated single speaker's sound signal into text information, and identify the corresponding speaker's identity for each piece of text information. The audiogram testing tool is used to call open-source hearing analysis software to conduct hearing tests on the listener and output the listener's audiogram parameters. The audiogram parameters include a variety of personalized information, including hearing spectral attenuation data at different frequencies. The audiogram adaptive TTS synthesis module receives the text information, the speaker identity corresponding to the text information identifier, and audiogram parameters. Based on the audiogram parameters, it calculates the spectral distribution and gain of each spectrum of the TTS synthesis file adapted to the listener. Combining the text information and the speaker identity corresponding to the identifier, it generates a personalized TTS sound file. The wireless transmission and secondary gain module receives the TTS audio file and then performs secondary spectral gain adjustment based on the audiogram parameters.

2. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 1, characterized in that, The scenarios include simple scenarios and complex scenarios. Simple scenarios are partial indoor scenarios with a limited number of people, while complex scenarios are large indoor scenarios or outdoor scenarios with many people. The TTS audio files distinguish the voice styles of men and women as well as different age groups; The speaker is someone who is having a conversation with a person who has hearing problems; Specifically, it refers to the sounds of people with normal hearing, other objects, or music being played.

3. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 1, characterized in that, In the sound acquisition and content extraction module: a multi-channel microphone array is used to acquire scene sound signals. Let the acquired original sound signal be... ,in , Number of microphone channels It is a time series; The preprocessing process includes noise reduction and signal normalization. Noise reduction uses the Wiener filtering algorithm, and the formula is: In the above formula, This is the target audio signal after noise reduction. This represents the optimal frequency response of the Wiener filter. Represents the original sound signal The result after Fourier transform; normalization is performed using the min-max normalization algorithm, the calculation formula of which is: In the formula, The normalized sound signal , These are the original sound signals. The minimum and maximum values ​​are determined by preprocessing to eliminate environmental noise and signal amplitude differences.

4. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 3, characterized in that, In the sound acquisition and content extraction module: multi-person speech separation adopts a speaker separation algorithm based on Independent Component Analysis (ICA), and the acquired mixed sound signal is set to be... The sound signal is a linear mixture of individual speaker signals, i.e., the normalized sound signal. The mixing model formula is: ； In the above formula, This is the preprocessed multi-channel mixed audio signal matrix, where The number of signal sampling points. This refers to the feature dimension size of each sample point's data; The mixture matrix is ​​unknown; Given a single speaker's voice signal matrix to be separated; the FastICA algorithm is used to solve for the separation matrix. The separated individual speaker signal can then be obtained, and its calculation formula is as follows: ； In the above formula, This is a matrix of individual speaker voice signals after separation, where each row vector corresponds to a speaker's voice signal. This enables the effective separation of signals from multiple speakers.

5. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 4, characterized in that, The speech-to-text conversion and speaker identification in the sound acquisition and content extraction module are implemented by: separating the individual speaker voice signals... Input speech-to-text ASR model, , corresponding to the For each speaker, the decoding process using the Transformer-based ASR algorithm is as follows: In the above formula, The separated single-speaker speech feature sequence, The output is a sequence of text information. The length of the text sequence. For the first The conditional probability of each character is calculated; simultaneously, the Mel-frequency cepstral coefficients (MFCCs) of each speaker's voice signal are extracted as voiceprint features. Speaker identity matching is achieved through cosine similarity calculation. The formula for calculating the identity identifier is as follows: In the above formula, For the preset speaker voiceprint reference features, This represents the cosine similarity; when the cosine similarity is greater than a preset threshold... In this way, the speaker's identity can be determined, and the speaker's identity identifier can be bound to the corresponding text information before being output.

6. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 5, characterized in that, The wireless transmission and secondary gain module includes a wireless transmission unit set on the system host and a Bluetooth headset unit worn by the listener. The wireless transmission unit is communicatively connected to the audiogram adaptive TTS synthesis module and transmits TTS audio files to the Bluetooth headset unit wirelessly via Bluetooth. The Bluetooth headset unit has a built-in gain adjustment module, which is used to perform secondary spectral gain adjustment based on audiogram parameters after receiving the TTS audio file, to ensure that the listener can clearly obtain the content of the scene dialogue. The secondary spectral gain adjustment is based on audiogram parameters pre-recorded by the listener. These individual audiograms are obtained through professional hearing testing equipment, and the parameters include hearing thresholds for each frequency band. The adjustment process is based on the principle of equalizing the target hearing threshold, taking into account the type of hearing loss and the need for hearing sensitivity compensation. This ensures that the actual perceived sound level of the TTS sound signal after gain adjustment is consistent across the listener's hearing frequency range.

7. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 6, characterized in that, The basis and implementation method of the second-order spectrum gain adjustment are as follows: Step 1. Use the hearing threshold values ​​for each frequency band in the listener's audiogram. Based on the standard hearing threshold of people with normal hearing, Calculate the gain compensation for each frequency band to ensure that the perceived sound level of the adjusted sound reaches the auditory perception level of people with normal hearing, while avoiding excessive gain that could cause auditory discomfort or secondary damage. Step 2. The gain adjustment module has a built-in gain compensation algorithm. First, it performs spectrum analysis on the received TTS audio file to extract the frequency distribution of the audio signal. Then, the target gain value for each frequency band is calculated based on the audiogram parameters. The calculation formula is: In the above formula, For the corresponding frequency The second-order gain value is non-negative; For people with normal hearing at frequencies Standard hearing threshold at that location; For listeners on the frequency The actual hearing threshold at that location; This is a safety compensation factor used to reserve a safety margin for hearing and prevent excessive gain from damaging hearing. Step 3. Perform differentiated gain optimization based on the different types of hearing loss reflected in the audiogram: For high-frequency steep-drop hearing loss, focus on increasing the gain value in the 1000Hz~8000Hz frequency band to ensure that consonants in TTS sounds are clearly distinguishable; for low-frequency hearing loss, focus on compensating the 200Hz~1000Hz frequency band to ensure the integrity of vowel and intonation information. For flat-type hearing loss, the gain across the entire frequency band is adjusted according to a uniform compensation ratio to maintain the overall timbre of the sound. Step 4. The gain adjustment module collects feedback signals from the listener during the wearing process in real time, and adjusts the gain value based on the dynamic changes in the audiogram. Dynamic calibration is performed to ensure that the secondary spectrum gain adjustment always matches the listener's actual hearing condition, further improving listening clarity and comfort.

8. The speech extraction and TTS adaptation system based on auditory spectrogram adaptive speech extraction according to claim 7, characterized in that, The spectral gain calculation process of the audiogram adaptive TTS synthesis module is as follows: Step 1. Assume the audiogram at a frequency The measured hearing threshold at the location is The standard hearing threshold for normal hearing is The formula for calculating hearing loss at this frequency is: in The formula for calculating the gain compensation value at the corresponding frequency point is: In the formula, The compensation coefficient ranges from 0.8 to 1.

2. Based on a basic comfort gain of 3–6 dB, the gain is clamped at both upper and lower limits to avoid distortion and hearing damage. The formula is as follows: in , ; Step 2. Let the original high-frequency components be... The target audible low-frequency range is The frequency shift mapping formula is: In the formula, This is the output frequency after frequency shifting; The personalized frequency shift coefficient is dynamically calculated from the audiogram, using the following formula: in Frequency shifting only applies to the high-frequency, weakly perceived region, without making significant adjustments to the mid- and low-frequency vowel components.

9. A speech extraction and TTS adaptation system based on auditory spectrograms according to claim 8, characterized in that, The wireless transmission unit uses Bluetooth 5.2 or above, supports high-definition audio transmission, and has a transmission latency of ≤100ms, ensuring the real-time transmission of TTS audio files. The secondary gain adjustment of the Bluetooth headset unit adopts the LMS adaptive filtering algorithm, which adjusts the gain of each frequency band in real time according to the audiogram parameters to dynamically adapt to the listener's hearing status. The specific implementation process is as follows: The LMS adaptive filtering algorithm uses the audiogram parameters pre-recorded by the listener as a benchmark and aims to make the perceived sound level of each frequency band more balanced. It dynamically updates the filtering coefficients by collecting the sound signal output by the Bluetooth headset unit and the listener's auditory feedback in real time, so as to achieve adaptive adjustment of the gain of each frequency band. Let the frequency in the listener's audiogram be... The target gain value at that location is The TTS audio signal received by the Bluetooth headphone unit is output from the audiogram adaptive TTS synthesis module mentioned above. , For a discrete-time series, the input vector of the LMS adaptive filter is: ； in Let be the filter order, and let be the filter coefficient vector. ; The output signal of the LMS adaptive filtering algorithm is: The algorithm's error signal is defined as the difference between the output signal corresponding to the target gain and the actual output signal: In the formula, The target output signal is the TTS audio signal. With target gain value Multiplying them together yields the result, i.e. ; To minimize the mean square value of the error signal The filter coefficients are iteratively updated using the gradient descent method, and the coefficient update formula is as follows: In the formula, This is the step size factor, used to control the coefficient update speed, to avoid signal distortion due to too fast convergence or failure to adapt to hearing changes in a timely manner due to too slow convergence. This is used as a gain adjustment coefficient to ensure the stability of the iteration process; During real-time adjustment, the algorithm extracts the input signal every frame. The spectral characteristics are matched with the corresponding frequency points in the audiogram, and the target gain value of that frequency point is called. The filter coefficients are updated using the above formula, thereby adjusting the gain of the corresponding frequency band. When the listener's hearing condition fluctuates dynamically, the algorithm uses error signals... The change in gain is corrected in real time, dynamically adapting to the hearing state, ensuring that the clarity perceived by the listener is consistent across all frequency bands after secondary gain adjustment, while avoiding hearing damage caused by excessive gain.

10. The speech extraction and TTS adaptation system based on auditory spectrograms according to claim 1, characterized in that, The sound acquisition and content extraction module includes a single-channel acquisition unit and a multi-channel acquisition unit. The single-channel acquisition unit is used for sound acquisition in simple point-to-point communication scenarios, while the multi-channel acquisition unit is used for sound acquisition in complex conference scenarios. The speech-to-text and speaker separation functions of the sound acquisition and content extraction module are implemented through an on-chip deep learning model or by calling a remote deep learning model.

Images