Intelligent listening and speaking auxiliary system and method for mislistening error correction and misspeaking translation based on audio conversion

Abstract

The invention discloses an intelligent listening and speaking auxiliary system combining audio conversion, mislistening error correction and mislistening translation and an implementation method of the intelligent listening and speaking auxiliary system. The system is based on a machine learning technology, deeply analyzes a mislistening mode of a user, constructs a personalized audio conversion model, can optimize voice output in real time, and helps hearing-impaired people to more accurately understand voice information. The system can be widely applied to terminal equipment such as intelligent earphones, mobile phones, hearing-aid equipment and the like, and supports various scenes such as daily communication, music appreciation, audio reading, video watching and the like. Besides, the system can automatically translate the spoken content of the hearing-impaired person into standard voice and play the standard voice through the loudspeaker, so that the healthy hearing person can clearly understand the expression of the hearing-impaired person, and the communication ability and life quality of the hearing-impaired user are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and hearing aids, specifically to an intelligent hearing aid system and its implementation method that combines mishearing correction, misspeech translation, and audio conversion. This system can be applied to various terminals such as smart headphones, mobile phones, and hearing aids to help hearing-impaired individuals with mishearing or misspeech difficulties improve their speech comprehension, while enabling hearing individuals to accurately understand the expressions of the hearing-impaired. Background Technology

[0002] In modern society, the number of people with hearing impairments continues to rise. According to global health statistics, approximately 15% of adults experience hearing loss to varying degrees. A significant portion of these individuals not only experience decreased auditory sensitivity but also exhibit specific instances of "mishearing" or "misspeaking." For example, some users might mishear "tissue" as "pen refill," or "chili pepper" as "take away," or even mishear "weather forecast" as "Tianqi ointment" during a conversation. Furthermore, some hearing-impaired individuals may misspeak due to inaccurate pronunciation, speech signal processing difficulties, or unclear articulation, meaning their spoken content is difficult for hearing individuals to understand correctly.

[0003] Existing hearing aids typically employ traditional techniques such as volume amplification, frequency compensation, or noise reduction to improve the perception of environmental sounds by hearing-impaired individuals. However, these methods only enhance the clarity of sound signals and cannot effectively address mishearing and misspeaking caused by cognitive biases or pronunciation disorders. Some studies have attempted to use speech recognition technology to convert speech into text to assist hearing-impaired individuals in understanding speech information, but such technologies have not been optimized for individual users' mishearing patterns and misspeaking characteristics. Furthermore, in multilingual, dialectal, or accent-rich environments, existing hearing aids lack the flexibility to adapt and fail to meet the personalized needs of hearing-impaired users.

[0004] Therefore, there is an urgent need for a hearing assistance system that combines mishearing analysis, misspeech detection, personalized audio conversion, and intelligent learning optimization. This system would enable hearing-impaired individuals to obtain more accurate speech information in various scenarios, including daily communication, listening to music, watching videos, telephone calls, and business meetings. Simultaneously, based on timbre analysis technology, the system could translate the misspoken content of hearing-impaired individuals into speech, making their expression more standardized. This would allow hearing individuals to accurately understand the spoken content, improving communication efficiency for both parties. Summary of the Invention

[0005] (1) Purpose of the invention

[0006] The main objective of this invention is to address the shortcomings of existing hearing aid technologies in areas such as mishearing correction, misspeech translation, and personalized speech conversion. While existing technologies can amplify or reduce noise in sound, they cannot intelligently address users' mishearing and misspeech issues. Therefore, this invention provides an intelligent hearing aid system that can:

[0007] 1. Misunderstanding Correction: For words that users often mishear (such as mishearing "tissue" as "pen refill" or "chili pepper" as "take away"), the system automatically identifies the mishearing pattern and converts it into speech, allowing hearing-impaired people to have a more accurate auditory experience.

[0008] 2. Mispronouncing Translation: Utilizing timbre analysis technology, this feature detects whether users mispronounce words in their speech and automatically performs standardized translation, enabling hearing individuals to clearly understand the content of their speech.

[0009] 3. Multilingual adaptation: Suitable for Mandarin, dialects and multilingual environments, ensuring that users can obtain accurate voice information in different language communication scenarios.

[0010] 4. Personalized learning optimization: Combining an intelligent learning mechanism, the system can continuously optimize the mishearing and misspeaking models based on user feedback, gradually adapting them to the user's hearing characteristics and language expression habits, thereby improving the long-term user experience.

[0011] (2) Technical solution

[0012] The present invention achieves the above objectives through the following technical solutions:

[0013] 1. Voice input and takeover:

[0014] o Deploy software modules in smart headphones, mobile phones, or hearing aids to monitor and take over the audio stream at the operating system or application layer, ensuring the integrity of voice data collection.

[0015] The system employs real-time signal processing technology to improve the accuracy of audio input and avoid external noise interference.

[0016] 2. Misunderstanding Analysis:

[0017] o uses machine learning models such as deep neural networks (DNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), or transformers to train on users' historical mishearing data and extract personalized mishearing patterns.

[0018] o Combines language model analysis to identify mishearing phenomena and automatically matches and corrects the identified incorrect words.

[0019] 3. Misinterpretation Analysis and Translation:

[0020] o By using timbre analysis technology, the user's pronunciation characteristics are detected to determine whether there are any mispronunciations, such as unclear pronunciation or phoneme confusion.

[0021] The o uses a language conversion model to translate misspoken content into standard speech, and optimizes speech rate, intonation, and timbre to make the output speech more natural and fluent.

[0022] It is suitable for dialects, accents, and multilingual environments, ensuring the accuracy of mispronunciation translation.

[0023] 4. Real-time error correction and audio conversion:

[0024] After detecting misheard or misspoken content, the system maps it to the correct words or standard pronunciation and makes necessary speech style adjustments to the audio signal (such as amplification, speech rate adjustment, and timbre optimization).

[0025] o Employs dynamic audio processing technology to improve the naturalness of speech conversion and user experience.

[0026] 5. User feedback and intelligent learning:

[0027] During daily use, users can provide feedback and correction information through the APP or hardware interface. The system will automatically incorporate this information into the training set to optimize the mishearing and misspeaking models.

[0028] By combining incremental learning algorithms, the system can gradually improve the accuracy of mishearing correction and misspeech translation as the user's usage time increases.

[0029] 6. Audio output and adaptation:

[0030] o Employs high-definition voice coding technology to optimize audio output quality.

[0031] o Connects via Bluetooth, Wi-Fi, or wired to output corrected audio signals to headphones, speakers, or hearing aids, ensuring users receive clear, natural voice information.

[0032] (3) Comparison with existing technologies and advantages

[0033] 1. Precise mishearing correction: This system uses a personalized mishearing model to automatically detect and correct users' mishearing phenomena. Compared with traditional hearing aids that only rely on volume amplification, this technology can significantly improve the accuracy of speech comprehension.

[0034] 2. Innovative mispronunciation translation function: Combining timbre analysis technology, the system can identify the mispronunciations of hearing-impaired individuals and convert them into standard speech, making their expression clearer and easier for hearing individuals to understand.

[0035] 3. Multi-scenario adaptation: Supports various scenarios such as daily communication, media entertainment, and business meetings, and can adapt to voice needs in different environments.

[0036] 4. Dialect compatibility and multilingual support: The system can adapt to Mandarin, dialects and foreign language environments to ensure smooth cross-language communication.

[0037] 5. Intelligent learning and personalized optimization: Through user feedback mechanisms and combined with deep learning technology, the system can continuously optimize the mishearing and misspeaking models, gradually improving the accuracy of error correction and translation as users use it. Attached Figure Description Figure 1 It is a schematic diagram of the system architecture and model. Figure 2 This is a flowchart for analyzing mishearing / misspoken statements. Figure 3 This is a flowchart of audio conversion and real-time error correction. Figure 4 This is a schematic diagram of the APP user interface (adaptive learning). Figure 5 It refers to the audio transmission and signal output method. Detailed Implementation

[0038] The present invention will be further described below with reference to the implementation method and embodiments of this system.

[0039] System architecture ( Figure 1 )

[0040] This system mainly consists of the following modules:

[0041] 1. Audio input module:

[0042] o Listening to or taking over the audio signals of mobile phones, smart headphones, or hearing aids.

[0043] o Collects sound data via system-level hooks, Bluetooth audio input, or microphone.

[0044] 2. Speech recognition module:

[0045] o Use deep neural networks (such as RNN, Transformer) for speech-to-text processing.

[0046] The system generates preliminary text results, which are then further analyzed by the mishearing analysis module and the misspeaking analysis module.

[0047] 3. Misunderstanding / Misspokenness Analysis Module:

[0048] o Identifies common misheard words or phoneme deviations by users.

[0049] By combining speech recognition results with a user-personalized mishearing model, potential mishearing / misspeaking can be detected.

[0050] o Combines dialect or foreign language adaptation, and calls the corresponding language model to correct mishearing / misspeaking.

[0051] 4. Audio Conversion Module:

[0052] o Corrects misheard words and mispronounced phonemes, and adjusts timbre and enhances key frequency bands.

[0053] If users are prone to mishearing "tissue paper," then the clarity of the word should be enhanced or a more easily understood alternative should be provided.

[0054] If users are prone to mispronouncing "tissue," then convert it to standard language during output.

[0055] o Supports audio style adjustment for multiple languages ​​and dialects.

[0056] 5. Intelligent Learning Module:

[0057] o We continuously optimize the mishearing and misspokenness recognition model based on user feedback.

[0058] o Continuously optimizes using deep learning algorithms such as DNN, LSTM, and Transformer.

[0059] When a user marks a mishearing / misspoken phenomenon multiple times, the system automatically reinforces the corresponding correction rules.

[0060] 6. User Feedback Module:

[0061] Users can annotate the system's error correction effect on the app or hardware interface.

[0062] o Collects user feedback information and automatically updates the mishearing and misspeaking models.

[0063] Users can customize the corrected words, such as manually adding samples like "chili pepper → remove".

[0064] 7. Audio output module:

[0065] The corrected audio stream is output to headphones or speakers.

[0066] It can use Bluetooth in smart headphone mode, and can play via wired or wireless means in mobile phone or hearing aid mode.

[0067] Users can adjust parameters such as output tone, volume, and speech rate.

[0068] Analysis of Misunderstandings and Misspokennesses ( Figure 2 )

[0069] 1. Testing with a predefined corpus:

[0070] When a user first uses the system, they read a series of phrases or words aloud. The system records any mishearings and mispronunciations to form a basic model.

[0071] 2. Real-time conversation recording analysis:

[0072] When a user mishears or misspoke during daily conversations or media playback, the system will compare the pronunciation with the correct pronunciation.

[0073] If repeated mishearing or misspeaking is detected, the corresponding correction rules will be strengthened in the user model.

[0074] 3. User feedback data collection:

[0075] Users actively marked actual mishearing situations such as "tissue" → "pen refill" and "chili pepper" → "remove".

[0076] The pronunciation characteristics of users who mispronounce words are recorded and used to improve the accuracy of mispronunciation translation.

[0077] The system includes this sample in the training set and focuses on correcting such errors in subsequent audio conversions.

[0078] 4. Model Training:

[0079] o Use deep learning models such as DNN, RNN, LSTM or Transformer to train the collected data.

[0080] o Extract users' unique patterns of mishearing and misspeaking for real-time error correction and translation.

[0081] Real-time conversion and error correction Figure 3 )

[0082] 1. Monitor user environment:

[0083] When users are conversing, the voice analysis module processes user voice and non-user voice separately.

[0084] When users listen to music or watch videos using audio devices, the system obtains text through the speech recognition module and identifies potential errors through the mishearing analysis module.

[0085] 2. Detection and Error Correction:

[0086] If a non-user voice is detected, a mishearing correction analysis is performed, and the system infers the correct word or phoneme through a mapping table or neural network.

[0087] The system performs voice analysis. If it detects that the user misspoke, it will convert the user's spoken voice information into correct words and sentences.

[0088] Multilingual and dialect adaptation: If a dialect or foreign phoneme is detected, the corresponding language model is invoked to ensure the accuracy of error correction.

[0089] 3. Audio enhancement:

[0090] o can perform secondary processing on the corrected / transcribed audio, including speech rate adjustment, timbre correction, noise suppression, etc., so that users can obtain more understandable speech.

[0091] 4. Output:

[0092] Finally, the optimized audio is sent to headphones, speakers, or hearing aids to enable real-time hearing assistance and mispronunciation correction.

[0093] Adaptive learning and app interaction Figure 4 )

[0094] 1. User App Feedback:

[0095] When the system's error correction or translation results are inaccurate, users can "confirm" them within the app.

[0096] Or a "reject" flag.

[0097] The system records this feedback and incorporates it into the next training iteration.

[0098] 2. Dynamic model update:

[0099] The system periodically or in real-time incorporates user feedback into the training set and updates the mishearing and misspeaking models using online or offline training methods.

[0100] The system employs an incremental learning strategy to ensure that users can benefit from model improvements without having to reread large amounts of text.

[0101] 3. Continuous iterative optimization:

[0102] As usage time accumulates, the system becomes more accurate in grasping users' patterns of mishearing and misspeaking, thus improving speech comprehension.

[0103] If a user uses the system in a new scenario (such as switching to a dialect environment), the system will automatically collect corresponding mishearing and misspeaking samples and quickly conduct adaptive training.

[0104] Output method ( Figure 5 )

[0105] 1. Headphone output:

[0106] Suitable for smart headphones or hearing aids, transmitting via Bluetooth, 2.4G wireless, 4G / 5G or Wi-Fi.

[0107] It can maintain low latency and high fidelity even in noisy environments.

[0108] 2. Speaker playback:

[0109] Suitable for home or public environments, the system can transmit the corrected audio stream to speakers or television devices.

[0110] Users can adjust personalized parameters such as volume and tone in the app.

[0111] 3. Personalized settings:

[0112] Users can manually adjust the tone, speaking speed, or other voice style options in the app interface to meet the needs of different usage scenarios.

[0113] Example

[0114] Example 1: Correcting Misunderstandings in Daily Communication

[0115] Scenario description: Users experience communication barriers due to mishearing in daily interactions, such as mishearing "weather forecast" as "Tianqi ointment" or "contract" as "sweat."

[0116] Implementation steps:

[0117] 1. Voice input and recognition (in accordance with claims 1a and 3):

[0118] The speech recognition module processes the input audio into text and takes over the phone's voice system to ensure a seamless audio conversion experience.

[0119] 2. Misunderstanding analysis (in accordance with claims 1b and 2):

[0120] The mishearing analysis module detects and identifies errors based on personalized mishearing data, and combines dialect adaptation to improve applicability.

[0121] 3. Audio conversion and optimization (in accordance with claims 1c and 4):

[0122] The system corrects misheard words or phonemes and makes adaptive adjustments to timbre or frequency bands to make the converted speech more in line with the user's understanding habits.

[0123] 4. Intelligent learning (in accordance with claims 1d and 5):

[0124] o Optimize the mishearing correction algorithm by incorporating user feedback, and dynamically train the mishearing conversion model using machine learning.

[0125] 5. Audio output (in accordance with claims 1e and 6):

[0126] o Sends optimized audio to headphones, mobile phones, or hearing aids, supporting Bluetooth,

[0127] Multiple connection methods including Wi-Fi.

[0128] Effects: Improves users' speech comprehension, making communication smoother and reducing communication barriers caused by misunderstandings.

[0129] Example 2: Misinterpretation Facilitates Public Communication

[0130] Scenario description: Hearing-impaired individuals experience communication difficulties in public places (such as hospitals and shopping malls) due to mispronunciation, affecting normal communication.

[0131] Implementation steps:

[0132] 1. Voice input and recognition (in accordance with claim 1a):

[0133] The o-mispronunciation translation module combines timbre analysis to identify the content that the user mispronounces.

[0134] 2. Misrepresentation Analysis (compliant with claims 1b and 2):

[0135] The mishearing analysis module detects misspoken phenomena based on personalized user data, trains audio conversion models, and improves translation accuracy.

[0136] 3. Misrepresentation (in accordance with claims 1c and 4):

[0137] After mispronunciation detection, the system automatically matches the correct words and adjusts the tone to make the speech conform to normal communication habits.

[0138] 4. Intelligent learning (in accordance with claim 5):

[0139] o Optimize the mistranslation model by incorporating user feedback.

[0140] 5. Audio output (in accordance with claims 1e and 6):

[0141] o The converted speech is played through a speaker so that hearing people can understand it clearly.

[0142] Effect: Helps hearing-impaired individuals express their needs more fluently in public places and improves communication efficiency.

[0143] Example 3: Video Content Optimization

[0144] Scenario description: Users misunderstand a video due to mishearing, for example, mishearing "Important Announcement" as "Traditional Chinese Medicine College Entrance Examination".

[0145] Implementation steps:

[0146] 1. Voice input and recognition (in accordance with claims 1a and 3):

[0147] The system monitors video and audio streams, and the speech recognition module performs speech-to-text conversion.

[0148] 2. Misunderstanding analysis (in accordance with claims 1b and 2):

[0149] The mishearing analysis module compares common misheard words with those of users and incorporates dialect adaptation.

[0150] 3. Audio conversion (in accordance with claims 1c and 4):

[0151] o Corrects misheard words, enhances pronunciation clarity, and adjusts tone style.

[0152] 4. Intelligent learning (in accordance with claim 5):

[0153] Words that users repeatedly mishear will be reinforced and optimized by the system.

[0154] 5. Audio output (in accordance with claims 1e and 6):

[0155] The optimized voice signal is synchronized to headphones, speakers, or mobile phones for playback.

[0156] Effect: Improves users' understanding of video content, making information delivery more accurate.

[0157] Example 4: Multilingual Misunderstanding and Mispronouncing Adaptation

[0158] Scenario description: When users communicate across languages, they may experience communication barriers due to mishearing or mispronouncing, such as mishearing "apple" as "a-pao" in an English environment.

[0159] Implementation steps:

[0160] 1. Voice input and recognition (in accordance with claims 1a and 3):

[0161] The mishearing analysis module is equipped with multilingual adaptation functionality.

[0162] 2. Analysis of mishearing and misrepresentation (in accordance with claims 1b and 2):

[0163] o Combines different language models to detect and adapt to mishearing and misspeaking.

[0164] 3. Audio conversion (in accordance with claims 1c and 4):

[0165] o Adjust the pronunciation to make the speech fit the current language environment.

[0166] 4. Intelligent learning (in accordance with claim 5):

[0167] o We optimize the cross-language mishearing correction model by combining user feedback data.

[0168] 5. Audio output (in accordance with claims 1e and 6):

[0169] o Play the corrected audio through headphones, speakers, or a mobile phone.

[0170] Effect: Improves users' communication efficiency in multilingual environments.

[0171] Example 5: Facilitating Doctor Communication in Medical Scenarios

[0172] Scenario description: Patients with hearing or speech impairments have difficulty accurately describing their symptoms when visiting the hospital, which affects the doctor's diagnosis.

[0173] Implementation steps:

[0174] 1. Voice input and recognition (in accordance with claim 1a):

[0175] o The system listens to conversations between patients and doctors, and processes the speech using a speech recognition module.

[0176] 2. Analysis of mishearing and misrepresentation (in accordance with claims 1b and 2):

[0177] The mishearing analysis module optimizes the ability to recognize medical terminology.

[0178] 3. Audio conversion (in accordance with claims 1c and 4):

[0179] o Optimize audio for medical terminology to ensure key information is clear.

[0180] 4. Intelligent learning (in accordance with claim 5):

[0181] o By combining patients' historical medical records, we can improve our ability to correct mishearing.

[0182] 5. Audio output (in accordance with claims 1e and 6):

[0183] The corrected audio is played through a doctor's terminal or hearing aid.

[0184] Effects: Improves the accuracy of doctor-patient communication and reduces the risk of misdiagnosis.

[0185] Summarize

[0186] The embodiments of this invention cover a variety of practical application scenarios, including daily communication, public communication, video viewing, multilingual adaptation, and medical scenarios, fully covering all the contents of claims 1-6. Through mishearing analysis, misspeech translation, intelligent learning, and multi-mode audio output, this system effectively improves the speech comprehension ability of hearing-impaired individuals, while optimizing the quality of speech communication and enhancing the understanding of hearing-impaired individuals by hearing-able individuals.

Claims

1. An intelligent listening and speaking assistance system based on audio conversion for error correction and mispronunciation translation, comprising an audio input module, an error analysis module, an audio conversion module, an intelligent learning module, and an audio output module, characterized in that: a) The speech recognition module processes the input audio signal into text to generate a preliminary text result; b) The mishearing analysis module performs error detection on the identified text based on the user's personalized mishearing data and matches possible mishearing patterns; c) The audio conversion module corrects detected misheard words or phonemes, and adjusts the timbre, intonation or frequency band to make the output speech more in line with the user's listening habits. d) The intelligent learning module combines user feedback to continuously optimize the mishearing correction algorithm, thereby improving the accuracy of recognition and conversion; e) The audio output module outputs the corrected audio signal to headphones, speakers or other terminal devices for users to listen to; f) The system is suitable for a variety of application scenarios, including but not limited to smart headphones, mobile applications, hearing aids and other wearable audio devices.

2. The intelligent listening and speaking assistance system according to claim 1, characterized in that: The mishearing analysis module uses machine learning algorithms to train a personalized audio conversion model and combines it with dialect adaptation to improve system applicability. The machine learning algorithms include, but are not limited to: a) Deep Neural Networks (DNNs); b) Recurrent Neural Network (RNN); c) Long Short-Term Memory Network (LSTM); d) Support Vector Machine (SVM); e) Various models such as Transformer.

3. The intelligent listening and speaking assistance system according to claim 1, characterized in that: The audio input module can take over the voice system of the terminal device to ensure a complete audio conversion experience. The takeover methods include, but are not limited to: a) Operating system-level audio redirection; b) Telephone call audio hooking; c) Media playback stream monitoring; d) Other underlying audio acquisition methods of the system.

4. The intelligent listening and speaking assistance system according to claim 1, characterized in that: The audio conversion module can automatically adjust the voice style to make the converted voice more in line with the user's understanding habits. The voice style includes, but is not limited to: a) Standard Mandarin; b) Local dialects (such as Sichuanese, Cantonese, etc.); c) Automatic switching and adaptation in other multilingual scenarios (such as English, Japanese, etc.).

5. The intelligent listening and speaking assistance system according to claim 1, characterized in that: The system can dynamically adjust the conversion model through a user feedback mechanism to improve the accuracy of mishearing correction. This feedback mechanism includes, but is not limited to: a) Users submit error correction information through the APP or hardware interface; b) The system updates the model in real time after receiving a specific speech correction instruction; c) Automatically record user satisfaction with the error correction results and incorporate it into the model training data to continuously optimize system performance.

6. The intelligent listening and speaking assistance system according to claim 1, characterized in that: The system is applicable to smart headphones, mobile phones, hearing aids, and other terminal devices, wherein the terminal connection methods include, but are not limited to: a) Bluetooth connectivity (e.g., BLE, BT5.0); b) 2.4G wireless communication; c) 4G / 5G mobile communication networks; d) Wi-Fi (2.4GHz / 5GHz) network connection; e) Other wired or wireless transmission protocols.

Images