Hearing-impaired voice conversion and generation method based on multi-language self-adaption

Abstract

The invention relates to a hearing-impaired voice conversion and generation method based on multi-language self-adaption, and belongs to the field of voice processing and artificial intelligence. According to the invention, hearing-impaired voices in multiple languages are converted into standard voices, so that normal communication with hearing-impaired people is facilitated. According to the method, acoustic features of hearing-impaired speech are extracted and aligned with target language phonemes, the acoustic features are input into a speech adaptation model, standard speech features are generated, multi-language training is carried out based on the standard speech features and target language corpus, and the training result is used for generating target language natural speech. According to the invention, the intelligibility of hearing-impaired voice can be improved, so that the voice is closer to natural pronunciation. According to the invention, rapid adaptation of multiple languages can be realized, and the cost of cross-language training is reduced. The system can be applied to education, medical rehabilitation, cross-border communication and other scenes, and provides effective voice communication assistance for hearing-impaired people.

Description

Technical Field

[0001] This invention belongs to the field of speech processing and artificial intelligence, specifically relating to a method for converting and generating hearing-impaired speech based on multilingual adaptation. Background Technology

[0002] Hearing-impaired speech refers to speech signals generated by individuals with hearing impairments in the absence of normal auditory feedback. Existing speech recognition and synthesis systems are mainly designed for hearing speech and do not adequately support the recognition and generation of speech from hearing-impaired individuals (who often have pronunciation deviations, missing phonemes, and abnormal intonation).

[0003] Furthermore, cross-language speech generation (such as Chinese to Vietnamese) requires a large amount of labeled data and high training costs, which cannot meet the actual needs for rapid adaptation.

[0004] Therefore, there is an urgent need for a method that can adapt hearing-impaired speech under small sample conditions and support multilingual output to improve the communication efficiency of hearing-impaired people in cross-language communication scenarios. Summary of the Invention

[0005] (a) Technical problems to be solved The technical problem to be solved by this invention is how to provide a multilingual adaptive method for speech conversion and generation for the hearing impaired, so as to solve the problem that existing speech recognition and synthesis systems are mainly designed for hearing speech and do not provide sufficient support for the recognition and generation of speech for the hearing impaired.

[0006] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a method for hearing-impaired speech conversion and generation based on multilingual adaptation, which includes the following steps: S1. Feature extraction is performed using the feature extraction module. The feature extraction module is used to acquire speech samples from hearing-impaired individuals and extract multi-dimensional acoustic features and phoneme sequences from them. By preprocessing the original speech signal, a stable acoustic feature sequence is obtained. The acoustic features are then forcibly aligned with the phoneme annotations of the target language to form an acoustic-phoneme correspondence, providing input for subsequent speech adaptation and multilingual training. S2. Perform voice adaptation via the voice adaptation module. The speech adaptation module uses an adaptive mapping model to map hearing-impaired speech features to standard speech features and utilizes a transfer learning mechanism to reduce dependence on large-scale data. S3. Multilingual training via the multilingual training module The multilingual training module is based on standard speech features, incorporates target language corpus, and achieves multilingual speech generation capability through a cross-language alignment mechanism; S4. Generate speech through the speech generation module. The speech generation module converts the adapted standard speech features into natural speech in the target language, supporting output in multiple languages.

[0007] (III) Beneficial Effects This invention proposes a multilingual adaptive method for converting and generating hearing-impaired speech. This method converts hearing-impaired speech in multiple languages ​​into standard speech, facilitating normal communication with hearing-impaired individuals. The invention extracts acoustic features from the hearing-impaired speech and aligns them with phonemes of the target language. These acoustic features are then input into a speech adaptation model to generate standard speech features. Multilingual training is performed based on these standard speech features and target language corpora. The training results are then used to generate natural speech in the target language.

[0008] This invention can improve the intelligibility of hearing-impaired speech, making the speech closer to natural pronunciation.

[0009] This invention enables rapid adaptation to multiple languages ​​and reduces the cost of cross-language training.

[0010] This invention can be applied to scenarios such as education, medical rehabilitation, and cross-border exchanges, providing effective voice communication assistance for people with hearing impairments. Attached Figure Description

[0011] Figure 1 Here is an example of a spectrogram. Detailed Implementation

[0012] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0013] This invention proposes a multilingual adaptive method for the conversion and generation of hearing-impaired speech. This method achieves the standardization and cross-lingual generation of hearing-impaired speech through four steps: feature extraction, speech adaptation, multilingual training, and speech generation.

[0014] This invention proposes a method for hearing-impaired speech conversion and generation based on multilingual adaptation, the method comprising: S1. Feature extraction is performed using the feature extraction module. The feature extraction module is used to acquire speech samples from hearing-impaired individuals and extract multi-dimensional acoustic features and phoneme sequences from them. By preprocessing the original speech signal, a stable acoustic feature sequence is obtained. The acoustic features are then forcibly aligned with the phoneme annotations of the target language to form an acoustic-phoneme correspondence, providing input for subsequent speech adaptation and multilingual training. S2. Perform voice adaptation via the voice adaptation module. The speech adaptation module uses an adaptive mapping model to map hearing-impaired speech features to standard speech features and utilizes a transfer learning mechanism to reduce dependence on large-scale data. S3. Multilingual training via the multilingual training module The multilingual training module is based on standard speech features, incorporates target language corpus, and achieves multilingual speech generation capability through a cross-language alignment mechanism; S4. Generate speech through the speech generation module. The speech generation module converts the adapted standard speech features into natural speech in the target language, supporting output in multiple languages.

[0015] Example 1: 1. Feature Extraction Module: The feature extraction module acquires speech samples from hearing-impaired individuals and extracts multi-dimensional acoustic features and phoneme sequences. These acoustic features include, but are not limited to, spectrograms, fundamental frequency, energy distribution, energy curves, formant parameters, and Mel-spectral coefficients. Stable acoustic feature sequences are obtained through preprocessing of the original speech signal, including denoising, normalization, frame windowing, and speech activity detection. Furthermore, the acoustic features are forcibly aligned with the phoneme annotations of the target language to form an acoustic-phoneme correspondence, providing input for subsequent speech adaptation and multilingual training.

[0016] In a specific embodiment, in response to issues such as abnormal fundamental frequency and low energy in hearing-impaired speech, this module also includes an anomaly detection and correction unit, which is used to smooth the fundamental frequency curve and compensate for energy distribution, thereby improving the stability and recognizability of feature representation.

[0017] S11. Data Acquisition: Collect speech data from hearing-impaired individuals. Data sources include, but are not limited to: recordings from rehabilitation centers, speech corpora from schools for the hearing impaired, and daily speech records. Data format can be WAV, FLAC, or other common audio formats. To ensure data quality, the sampling rate is generally set between 16kHz and 48kHz, with a bit depth of 16 bits or higher.

[0018] S12. Preprocessing steps: The recorded speech undergoes denoising, silence segment extraction, and energy normalization. The speech signal is then framed and windowed (e.g., using a Hamming window) to ensure the stability of subsequent feature calculations. Silent segments are removed through speech activity detection, retaining only valid speech.

[0019] S13. Acoustic Feature Extraction: Spectrogram: Perform a Fast Fourier Transform on each frame to obtain the spectrum, then stack them to form a time-frequency graph. See the following code (Python): import librosa.display import matplotlib.pyplot as plt plt.rcParams['font.sans-serif'] = ['SimHei'] plt.rcParams['axes.unicode_minus'] = False signal, sr = librosa.load("audio_file / example.wav", sr=16000) # Calculate STFT n_fft = 1024 hop_length = 256 stft_result=librosa.stft(signal,n_fft=n_fft,hop_length=hop_length) spectrogram_db = librosa.amplitude_to_db(abs(stft_result)) # Draw a spectrogram plt.figure(figsize=(10, 4)) librosa.display.specshow(spectrogram_db,sr=sr, hop_length=hop_length,x_axis="time", y_axis="hz", cmap="magma") plt.colorbar(format="%+2.0f dB") plt.xlabel("Time (seconds)") plt.ylabel("Frequency (Hz)") plt.title("Spectrogram of real speech") plt.tight_layout() output_path = "F: / / spectral graph example.png" plt.savefig(output_path, dpi=300, bbox_inches="tight") # Save the image print("Saving complete:", output_path) plt.show() plt.close() Preparing for input: Ensure you have a mono, 16kHz audio file. Calculation of features: Parameters are set: n_fft (the number of points in each frame for the Short-Time Fourier Transform), frame length, and frame shift. The complex spectrum is obtained from the audio data → amplitude is taken → converted to a logarithmic scale (dB), resulting in the spectrogram matrix.

[0020] Creating a canvas and drawing a spectrogram will yield... Figure 1 Similar images.

[0021] The fundamental frequency estimation unit includes a feature modeling submodule and a multi-task output submodule.

[0022] Feature Modeling Submodule: Used to model acoustic features in time and frequency patterns using convolutional neural networks, recurrent neural networks, Transformers or combinations thereof, to obtain frame-level hidden representations.

[0023] The multi-task output submodule is used to simultaneously perform sound / silent determination and fundamental frequency regression. The fundamental frequency is output in logarithmic form, and robustness is improved through a combination of bucket classification and regression refinement. Robustness refers to the ability of the method to maintain stable and reliable output when facing noise, interference, abnormal input, or uncertainty. The fundamental frequency trajectory of consecutive frames (a curve formed by concatenating the fundamental frequency values ​​of consecutive frames) is smoothed and outlier suppressed, and interpolation is performed to complete short-term discontinuous silent intervals.

[0024] The purpose of smoothing is to reduce jitter in the fundamental frequency trajectory. Moving average filtering can be used.

[0025] The average value is taken within the window, where k is the range of the serial port, to eliminate sharp jumps.

[0026] Outlier suppression aims to remove outliers that the prediction model may produce (such as a sudden jump from 100Hz to 800Hz). A threshold range is set (e.g., 50Hz–500Hz), and values ​​outside this range are marked as outliers and replaced with the nearest neighbor average. Neighbor-frame difference detection: If |F0(t)–F0(t-1)| exceeds a certain threshold (e.g., 80Hz), it is considered an outlier and replaced with an interpolated value or the value from the previous frame.

[0027] The purpose of interpolation completion is to maintain the continuity of the fundamental frequency curve during short periods of silence (when the model determines it to be unvoiced). This is achieved through linear interpolation:

[0028] t1 and t2 are the positions of the preceding and following sound frames, respectively. Interpolation is only performed during "short" silent segments (e.g., ≤100ms), while long silent segments remain empty.

[0029] Energy distribution: Calculate the short-time energy of each frame to reflect speech intensity and stress information.

[0030] Formants: Estimating vocal tract formant characteristics through linear predictive coding helps identify pronunciation deviations.

[0031] S14. Feature Correction of Hearing-Impaired Speech: Since hearing-impaired speech often suffers from phoneme loss and unstable fundamental frequency, an "anomaly detection mechanism" needs to be designed. The extracted fundamental frequency and energy curves are compared with the statistical model of healthy speech to mark abnormal segments. Abnormal parts are then normalized or corrected, for example, by smoothing the fundamental frequency curve and compensating for segments with low energy.

[0032] S15. Align with Target Language Phoneme Labels: Align the acoustic features with the phoneme labels of the target language (e.g., Vietnamese / Japanese) to generate an "Acoustic Feature – Phoneme" lookup table. Forced alignment can be used, such as with tools like Kaldi or Montreal Forced Aligner. The alignment results guide subsequent model training, enabling the model to correctly learn the mapping relationship between "hearing-impaired speech → standard speech → multilingual speech." Standard speech is the correct pronunciation of the target speech. For example, if the target speech is Chinese, the correct pronunciation is "nihao," but a hearing-impaired person might pronounce it as "ni ao." Multilingual speech involves translating this standard speech into multiple languages, such as Japanese or Vietnamese, so that people from multiple countries can understand the hearing-impaired Chinese speech.

[0033] 2. Speech Adaptation Module: This module uses an adaptive mapping model to map hearing-impaired speech features to standard speech features, leveraging transfer learning to reduce reliance on large-scale data. It addresses deviations in phoneme clarity, fundamental frequency stability, and energy distribution in hearing-impaired speech. By mapping "hearing-impaired speech features" to "standard speech features close to those of hearing-impaired individuals," it lays the foundation for subsequent multilingual training. Input includes: the acoustic feature sequence output from the feature extraction module, and the corresponding target language phoneme sequence.

[0034] S21. Input Features: Acoustic features from the feature extraction module: spectrogram, fundamental frequency, energy curve, formant parameters, Mel-frequency cepstral coefficients, etc. The energy curve can be directly obtained from the feature extraction module.

[0035] S22, Adaptive Mapping Model Design: Adaptive mapping networks: constructed using deep neural networks (DNN, CNN, RNN, or Transformer).

[0036] Input layer: Accepts a multidimensional acoustic feature vector from the feature extraction module. This vector is composed of features such as spectrogram, fundamental frequency, energy curve, formant parameters, and MFCC.

[0037] Adaptive layer: Introduces normalization and correction mechanisms to correct abnormal features of hearing-impaired speech.

[0038] a. Eliminate the differences in mean and variance among different speech samples by using batch normalization or layer normalization methods; b. By learning the difference function between the fundamental frequency of hearing impairment and the standard fundamental frequency through the residual mapping unit, the overall translation and fine-grained correction of the fundamental frequency curve are achieved; c. Energy features are redistributed through an attention weighting mechanism to enhance the expressive ability of weak energy segments; d. Dynamic smoothing and compensation are performed on phenomena such as phoneme boundary blurring and pitch instability in the temporal dimension through the context modeling unit. Output layer: Generates "standard speech features" that approximate the statistical distribution of sound speech.

[0039] S23. Transfer Learning Mechanism: Due to the limited number of hearing-impaired speech samples, directly training deep models can easily lead to overfitting. Therefore, this invention employs a transfer learning method, using a pre-trained acoustic model (e.g., Higgs audio, Tacotron, VITS model, etc.) on a large-scale healthy speech corpus as the base network. The learned speech time-frequency patterns and phoneme boundary alignment capabilities are used as initial weights. Based on this, only a small amount of hearing-impaired speech data is used for fine-tuning, fixing the underlying general feature extraction parameters and updating only the higher-level mapping layers, thereby rapidly expanding the features of hearing-impaired speech.

[0040] This transfer learning mechanism enables the acquisition of a stable mapping model from hearing-impaired speech to standard speech features under limited sample conditions, providing high-quality input for subsequent multilingual training modules.

[0041] S24. Alignment and Correction: Using a forced alignment tool, the input phoneme sequence is aligned with standard speech features. For detected "phoneme missing" or "pronunciation abnormal" regions, they are supplemented using pseudo-labels or data augmentation.

[0042] a. Input: Acoustic feature sequence and phoneme sequence provided by the feature extraction module.

[0043] b. Alignment: Use forced alignment tools (such as Kaldi, MFA) to map phoneme sequences to standard speech features frame by frame.

[0044] c. Correction: For missing phonemes: Use alignment information to mark "phoneme gaps" and generate reasonable phoneme patches or data augmentation (speech rate stretching, pitch perturbation) through pseudo-labels to fill the gaps. For pronunciation abnormalities: Use alignment results to detect abnormal segments and replace or correct them with standard features that are statistically close to sound speech (this can be achieved through average features or residual correction).

[0045] d. Output: Corrected "standard speech features" that are close to a healthy speech distribution in both temporal and phoneme aspects.

[0046] S25. Output Features: The mapped features possess the following attributes: improved phoneme clarity, smoothed fundamental frequency curve, balanced energy distribution, and increased similarity to standard speech. 3. Multilingual Training Module: The input is standard speech features from the speech adaptation module (obtained through hearing-impaired speech correction and mapping). Therefore, the multilingual training module further extends the speech generation to different target languages ​​based on the hearing-impaired speech adaptation results. Based on standard speech features, target language corpora are introduced, and multilingual speech generation capability is achieved through a cross-language alignment mechanism. This addresses the problem of how to further extend the adapted hearing-impaired speech to different languages ​​(e.g., Chinese → Vietnamese). Under small sample conditions, a mapping relationship of "standard speech features → multilingual output" is established to support cross-language speech synthesis.

[0047] S31. Input data: Standard speech features output by the speech adaptation module. Speech and text corpora of the target language, such as Vietnamese audio and corresponding phoneme or word annotations.

[0048] S32. Cross-language alignment mechanism: Phoneme-level alignment: Using a forced alignment method, the phonemes of the target language are aligned with the features of the standard speech.

[0049] Taking Japanese as an example: Converting Japanese writing systems to Romanization and then to phoneme sequences is commonly done using tools like Julius and OpenJTalk. For example: こんにちは→“ko-n-ni-chi-wa”→ / ko N ni ch iwa / . The Japanese speech is then segmented and windowed to extract MFCC or Mel-filter energy spectra. Input: Japanese phoneme sequences (e.g., [k][o][N][n][i][ch][i][w][a]) and a standard Japanese speech feature sequence. Output: the start and end frames for each phoneme.

[0050] Semantic alignment: Mapping the semantic spaces of different languages ​​using shared semantic representations (such as multilingual word vectors or Transformer encoders).

[0051] a. Cross - language word vectors: Use multilingual pre - trained word vectors (such as MUSE, fastText multilingual models) to map words or sub - words in different languages to the same vector space. For example, apple (en), 苹果 (zh), táo (vi) are close to each other in the vector space.

[0052] b. Transformer encoder (multilingual pre - trained model): Utilize multilingual Transformer models such as mBERT, XLM - R, mT5, etc. to encode texts in different languages into unified semantic vectors. These models share word embeddings and some parameters during training, thus aligning the semantic spaces of multiple languages.

[0053] c. Alignment method: Standard speech features → text (phonemes / words) → semantic vector, and target - language text → semantic vector. Use the shared vector space to map the semantics of the source language and the target language, ensuring that the expressions of the same semantics have similar vector representations in different languages.

[0054] d. Training strategies: Multi - task learning: Simultaneously train tasks from speech to semantics and from semantics to speech, forcing the model to learn cross - language alignment. Contrastive learning: Make different - language representations of the same semantics close to each other and representations of different semantics far from each other.

[0055] Temporal alignment: Use the attention mechanism to ensure the correspondence of standard speech features and target - language speech in the time dimension.

[0056] S33. Training method: Construct a multilingual speech generation network with standard speech features as the input and the acoustic feature sequence of the target language as the output. The model can adopt frameworks such as Seq2Seq, Transformer, VITS or Higgs Audio. Introduce a multi - task learning mechanism to enable the model to learn the feature distributions of multiple languages simultaneously and improve cross - language generalization ability. Adopt a parameter - sharing mechanism to reduce the training cost while ensuring multilingual generation.

[0057] S34. Optimization and enhancement: Use speech data augmentation (such as pitch perturbation, duration stretching) to improve the model's adaptability to different pronunciation differences. In the case where there is a lack of large - scale corpus in the target language, introduce transfer learning and pseudo - label generation to improve the training effect.

[0058] The specific implementation methods are as follows: Speech data augmentation: a. Pitch perturbation: Without changing the semantics, raise or lower the overall speech by a certain number of semitones (+2 / -2 semitones) to simulate the pitch ranges of different speakers.

[0059] b. Duration stretching: Speed ​​up or slow down the speech (±10%–20%), but keep the frequency unchanged to simulate different speech speeds.

[0060] c. Add noise: Superimpose background noise or white noise to improve robustness.

[0061] d. Spectral occlusion: Randomly occlude time periods or frequency bands of the Mel spectrum to allow the model to learn to ignore local missing data.

[0062] Optimization in small-corpus scenarios: When the target language lacks a large-scale corpus, a transfer learning method is adopted to transfer the parameters of the multilingual model trained on a large-corpus language to the target language, with only fine-tuning of the high-level parameters; at the same time, the automatic speech recognition model is used to generate pseudo-labels for unlabeled speech and train it together with real labeled corpus, thereby improving the training effect of the model on low-resource languages.

[0063] S35. Output Results: The multilingual training module, based on acoustic models such as Transformer or VITS, maps standard speech features to acoustic feature sequences of the target language (e.g., Mel spectrum or MFCC features). Subsequently, a neural vocoder (such as WaveNet, WaveGlow, or HiFi-GAN) is used to convert this acoustic feature sequence into a waveform signal, generating natural speech. Through the synergistic effect of the aforementioned acoustic model and vocoder, the final generated speech outperforms existing methods in terms of naturalness, clarity, and semantic accuracy.

[0064] 4. Speech generation module: Converts the adapted standard speech features into natural speech in the target language, supporting output in multiple languages ​​such as Chinese and Vietnamese.

[0065] Beneficial effects: This invention converts hearing-impaired speech in multiple languages ​​into standard speech, facilitating normal communication with hearing-impaired individuals. The invention extracts acoustic features from the hearing-impaired speech and aligns them with phonemes of the target language. These acoustic features are then input into a speech adaptation model to generate standard speech features. Multilingual training is performed based on these standard speech features and target language corpora, and the training results are used to generate natural speech in the target language.

[0066] This invention can improve the intelligibility of hearing-impaired speech, making the speech closer to natural pronunciation.

[0067] This invention enables rapid adaptation to multiple languages ​​and reduces the cost of cross-language training.

[0068] This invention can be applied to scenarios such as education, medical rehabilitation, and cross-border exchanges, providing effective voice communication assistance for people with hearing impairments.

[0069] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for hearing-impaired speech conversion and generation based on multilingual adaptive methods, characterized in that, The method includes the following steps: S1. Feature extraction is performed using the feature extraction module. The feature extraction module is used to acquire speech samples from hearing-impaired individuals and extract multi-dimensional acoustic features and phoneme sequences from them. By preprocessing the original speech signal, a stable acoustic feature sequence is obtained. The acoustic features are then forcibly aligned with the phoneme annotations of the target language to form an acoustic-phoneme correspondence, providing input for subsequent speech adaptation and multilingual training. S2. Perform voice adaptation via the voice adaptation module. The speech adaptation module uses an adaptive mapping model to map hearing-impaired speech features to standard speech features and utilizes a transfer learning mechanism to reduce dependence on large-scale data. S3. Multilingual training via the multilingual training module The multilingual training module is based on standard speech features, incorporates target language corpus, and achieves multilingual speech generation capability through a cross-language alignment mechanism; S4. Generate speech through the speech generation module. The speech generation module converts the adapted standard speech features into natural speech in the target language, supporting output in multiple languages.

2. The method for hearing-impaired speech conversion and generation based on multilingual adaptation as described in claim 1, characterized in that, In S1, the acoustic features include: spectrogram, fundamental frequency, energy distribution, energy curve, formant parameters, and Mel-frequency cepstral coefficients.

3. The method for hearing-impaired speech conversion and generation based on multilingual adaptive as described in claim 1, characterized in that, S1 specifically includes: S11. Data Acquisition: Collecting speech data from hearing-impaired individuals; S12. Preprocessing steps: Denoise the recorded speech, extract silent segments, and normalize the energy; perform frame-by-frame windowing on the speech signal, and remove silent segments through speech activity detection to retain valid speech; S13. Acoustic Feature Extraction: Extract spectrogram, fundamental frequency, energy distribution, energy curve, formant parameters, and Mel-frequency cepstral coefficients; S14. Feature correction of hearing-impaired speech: Compare the extracted fundamental frequency and energy curves with the statistical model of healthy speech, mark abnormal segments, and normalize or correct the abnormal parts. S15. Align with target language phoneme annotations: Align acoustic features with target language phoneme annotations to generate an "acoustic feature-phoneme" lookup table.

4. The method for hearing-impaired speech conversion and generation based on multilingual adaptive as described in claim 1, characterized in that, S2 includes: S21. Input acoustic features; S22, Adaptive Mapping Model Design Adaptive mapping network: constructed using a deep neural network; Input layer: Accepts a multi-dimensional acoustic feature vector from the feature extraction module, which is composed of concatenated acoustic features; Adaptive layer: Introduces normalization and correction mechanisms to correct abnormal features of hearing-impaired speech; Output layer: Generates standard speech features, approximating the statistical distribution of sound speech; S23. Transfer learning mechanism: The acoustic model trained on a large-scale sound speech corpus is used as the base network, and its learned speech time-frequency patterns and phoneme boundary alignment capabilities are used as initial weights. On this basis, only a small amount of hearing-impaired speech data is used for fine-tuning, the general feature extraction parameters of the lower layer are fixed, and only the mapping layer of the higher layer is updated, thereby rapidly expanding the features of hearing-impaired speech. S24. Alignment and Correction: Use a forced alignment tool to align the input phoneme sequence with standard speech features; for detected "phoneme missing" or "pronunciation abnormal" regions, fill them in with pseudo-labels or data augmentation; S25. Output features: The mapped features have the following attributes: improved phoneme clarity, smoothed fundamental frequency curve, balanced energy distribution, and increased similarity to standard speech.

5. The method for hearing-impaired speech conversion and generation based on multilingual adaptive as described in claim 4, characterized in that, The adaptive layer includes: By employing batch normalization or layer normalization methods, the differences in mean and variance among different speech samples are eliminated; by using residual mapping units to learn the difference function between the fundamental frequency of hearing impairment and the standard fundamental frequency, the overall translation and fine-grained correction of the fundamental frequency curve are achieved; by using an attention weighting mechanism to redistribute energy features, the expressive ability of weak energy segments is enhanced; and by using context modeling units, dynamic smoothing and compensation are performed on phoneme boundary ambiguity and pitch instability in the temporal dimension.

6. The method for hearing-impaired speech conversion and generation based on multilingual adaptive as described in claim 4, characterized in that, S24 includes: Input: Acoustic feature sequence and phoneme sequence provided by the feature extraction module; Alignment: Using a forced alignment tool, the phoneme sequence is mapped to the standard speech features frame by frame; Correction: "Phonetic gaps" are marked using alignment information, and reasonable phoneme patches or data enhancements are generated through pseudo-labels to fill them; for pronunciation abnormalities: abnormal segments are detected using alignment results and replaced or corrected to standard features that are statistically close to sound speech. Output: Corrected standard speech features, ensuring that they closely approximate a healthy speech distribution at both the temporal and phoneme levels.

7. The method for hearing-impaired speech conversion and generation based on multilingual adaptive as described in claim 1, characterized in that, S3 includes: S31. Input data: Standard speech features output by the speech adaptation module; S32. Cross-language alignment mechanism: Phoneme-level alignment: Using a forced alignment method, the phonemes of the target language are aligned with the features of standard speech. Semantic alignment: mapping the semantic spaces of different languages ​​through shared semantic representations; Temporal alignment: An attention mechanism is used to ensure that the features of the standard speech correspond to the speech of the target language in the time dimension; S33. Training Method: Construct a multilingual speech generation network with standard speech features as input and acoustic feature sequence of the target language as output; introduce a multi-task learning mechanism to learn the feature distribution of multiple languages ​​simultaneously, thereby improving cross-language generalization ability; adopt a parameter sharing mechanism to reduce training overhead while ensuring multilingual generation. S34. Optimization and Enhancement: Use speech data augmentation to improve the model's adaptability to differences in pronunciation. In the absence of a large-scale corpus of the target language, introduce transfer learning and pseudo-label generation to improve training performance. S35. Output: The multilingual training module is based on an acoustic model, which maps standard speech features to acoustic feature sequences of the target language; a neural vocoder is used to convert the acoustic feature sequences into waveform signals to generate natural speech.

8. The method for hearing-impaired speech conversion and generation based on multilingual adaptation as described in claim 7, characterized in that, The semantic level alignment includes: Cross-lingual word vectors: using multilingual pre-trained word vectors to map words or subwords from different languages ​​to the same vector space; Transformer encoders: These utilize multilingual Transformer models to encode text from different languages ​​into unified semantic vectors. These models share word embeddings and some parameters during training, enabling alignment of the semantic spaces of multiple languages. Alignment method: Convert standard speech features into semantic vectors and target language text into semantic vectors; use a shared vector space to map the semantics of the source language and the target language to ensure that the same semantic expression is represented by similar vectors in different languages; Training strategy: Simultaneously train speech-to-semantics and semantic-to-speech tasks, forcing the model to learn cross-language alignment; bring different language representations of the same semantics closer together, and move different semantic representations further apart.

9. The method for hearing-impaired speech conversion and generation based on multilingual adaptation as described in claim 7, characterized in that, The voice data enhancement includes: Pitch perturbation: Without changing the semantics, raise or lower the overall speech by a certain number of semitones to simulate the pitch range of different speakers; Duration stretching: Speeds up or slows down speech while keeping the frequency constant, simulating different speech rates; Add noise: Superimpose background noise or white noise to improve robustness; Spectral occlusion: Randomly occlude time periods or frequency bands of the Mel spectrum, allowing the model to learn to ignore local missing data.

10. The method for hearing-impaired speech conversion and generation based on multilingual adaptation as described in claim 9, characterized in that, Introducing transfer learning and pseudo-label generation to improve training performance includes: when the target language lacks a large-scale corpus, using transfer learning to transfer the parameters of a multilingual model trained on a large-scale corpus language to the target language, with only fine-tuning of high-level parameters; at the same time, using an automatic speech recognition model to generate pseudo-labels for unlabeled speech and training them together with real labeled corpus, thereby improving the training performance of the model on low-resource languages.

Images