Speech processing method for hearing-impaired people, and speech processing device and system therefor

Abstract

The application discloses a speech processing method and system suitable for hearing-impaired people, and a speech processing device thereof. The speech processing method suitable for hearing-impaired people comprises the following steps: acquiring environmental audio signals around a user, identifying speech segments about human voices from the environmental audio signals; judging whether the speech segments correspond to a target speaker based on speech feature information of the speech segments, the target speaker and the user communicating with each other; if yes, analyzing the speech segments to obtain language elements, the language elements comprising semantic information of the speech segments, spectral energy distribution characteristics of each frame and relative time length relationships between each syllable; generating targeted digital speech signals based on the hearing loss degree of the user, and transmitting the digital speech signals to an audio output module of a hearing aid. The method can accurately identify the target speaker from a complex environment and assist the user in accurately understanding the conversation content, thereby improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of hearing aid technology, and in particular to a speech processing method, speech processing device and system suitable for people with hearing impairments. Background Technology

[0002] Current hearing aids primarily use microphones to collect ambient sounds and then amplify, reduce noise, or compensate for frequency bands. However, for people with severe hearing loss, simply amplifying ambient sounds often fails to effectively improve language comprehension. For example, severe hearing loss often affects the perception of specific phonemes, especially high-pitched consonants. Although these consonants are relatively weak, they are responsible for most of the semantic distinctions. Therefore, simply increasing the volume cannot restore these missing sound signals; instead, it may make the speech sound unclear or disorganized, making it difficult to understand the specific content. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, one objective of this invention is to propose a speech processing method suitable for people with hearing impairments. This method can accurately identify the target speaker in complex environments and assist users in accurately understanding dialogue content, thereby improving the user experience.

[0004] The second objective of this invention is to provide a speech processing device suitable for people with hearing impairments.

[0005] The third objective of this invention is to propose a speech processing system suitable for people with hearing impairments.

[0006] To address the aforementioned problems, a first aspect of this invention provides a speech processing method applicable to people with hearing impairments, comprising: acquiring ambient audio signals surrounding the user; identifying speech segments related to human voices from the ambient audio signals; determining, based on the speech feature information of the speech segments, whether the speech segments correspond to a target speaker, and having the target speaker communicate with the user via speech; if so, analyzing the speech segments to obtain language elements, the language elements including semantic information of the speech segments, spectral energy distribution characteristics of each frame, and relative duration relationships between syllables; and generating targeted digital speech signals based on the degree of hearing loss of the user.

[0007] According to the speech processing method for people with hearing impairments according to embodiments of the present invention, based on the fact that speech feature information can reflect the essential characteristics of the speaker's identity, language content, or emotional state, the speech feature information of the speech segment is used to determine whether the speech segment corresponds to the target speaker, thereby effectively and accurately identifying the target speaker from a complex environment. Then, the speech segment of the target speaker is converted into language elements. Because the form of language elements can accurately reflect the speech content and the physical characteristics of the original sound, especially accurately reflecting the semantic content that is less perceived by people with hearing impairments, a targeted digital speech signal is generated based on the language elements and transmitted to the user. Therefore, compared with the processing method of amplifying the original acoustic signal, the processing method of this application can help the user accurately understand the dialogue content and improve the user experience.

[0008] In some embodiments, the speech feature information includes at least one of voiceprint features, directional features, semantic context features, and interaction distance features.

[0009] In some embodiments, the semantic information is one or more of text information, semantic text, and phoneme sequences.

[0010] In some embodiments, generating targeted digital speech signals from the language elements includes: configuring a corresponding voice role for each target speaker when there are multiple target speakers; and generating a digital speech signal for each target speaker based on the voice role and the language elements.

[0011] In some embodiments, configuring a corresponding voice role for the at least one target speaker includes: configuring a corresponding voice role for the at least one target speaker based on a timbre template library, wherein the timbre template library is established based on at least one of voice type, pitch, speech rate and timbre; and / or, configuring a corresponding voice role for the at least one target speaker based on a timbre cloning model, wherein the timbre cloning model is trained based on speech samples corresponding to the target speaker.

[0012] In some embodiments, the degree of hearing loss of the user is determined based on the hearing aid configuration information of the hearing aid, and the method further includes: adjusting the digitized speech signal according to the user's hearing aid configuration information, the adjustment including at least one of frequency band enhancement, speech rate adjustment, pitch adjustment, frequency response adjustment, pause enhancement, or keyword repetition.

[0013] In some embodiments, the hearing aid configuration information includes at least one of local configuration information and / or network configuration information; the hearing aid configuration information includes at least one of the user's hearing loss curve, volume, timbre selection, and hearing aid mode.

[0014] In some embodiments, the hearing aid configuration information is determined based on hearing aid keywords extracted from speech segments about human voices.

[0015] In some embodiments, acquiring ambient audio signals around the user includes: acquiring ambient audio signals within the sound reception range.

[0016] In some embodiments, the pickup range is determined based on the hearing aid mode.

[0017] In some embodiments, the sound reception range is determined based on environmental feature information surrounding the user; the environmental feature information includes at least one of ambient noise intensity, sound source distance, number of target speakers, user posture, and signal-to-noise ratio.

[0018] In some embodiments, the method further includes: if a non-human voice segment and / or the user's own voice segment are identified from the ambient audio signal, prohibiting the transmission of the digital voice signal corresponding to the non-human voice segment and / or the digital voice signal corresponding to the user's own voice segment to the user; or, reducing the volume of the digital voice signal corresponding to the non-human voice segment and / or the volume of the digital voice signal corresponding to the user's own voice segment.

[0019] In some embodiments, the method further includes: if it is determined that there is no correspondence between the speech segment and the target speaker, determining whether the speech segment meets the reminder conditions; if so, providing a reminder based on the speech segment, wherein the reminder includes at least one of phrase speech, prompt tone, spatialized audio, and vibration feedback.

[0020] In some embodiments, the method is applied to a voice processing device, which includes a first audio channel and a second audio channel; the reminder is fed back to the user through the first audio channel. The digitized voice signal is transmitted to the user through the second audio channel.

[0021] A second aspect of the present invention provides a speech processing device suitable for people with hearing impairments, comprising: at least one processor; a memory communicatively connected to at least one processor; wherein the memory stores a computer program executable by at least one processor, and when at least one processor executes the computer program, it implements the speech processing method for people with hearing impairments described in the above embodiment.

[0022] According to embodiments of the present invention, a speech processing device suitable for people with hearing impairments can accurately identify target speakers in complex environments and assist users in accurately understanding dialogue content, thereby improving the user experience.

[0023] A third aspect of the present invention provides a speech processing system suitable for people with hearing impairments, including at least one speech processing device suitable for people with hearing impairments as described in the above embodiments.

[0024] The speech processing system for people with hearing impairments according to embodiments of the present invention can accurately identify the target speaker in complex environments and help users accurately understand the dialogue content, thereby improving the user experience.

[0025] In some embodiments, the at least one speech processing device suitable for people with hearing impairments includes two or three of the following: hearing aids, terminal devices, and cloud devices.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a speech processing method according to an embodiment of the present invention; Figure 2 This is a flowchart of a speech processing method according to another embodiment of the present invention; Figure 3 This is a flowchart of a speech processing method according to another embodiment of the present invention; Figure 4 This is a structural diagram of a voice processing device according to an embodiment of the present invention; Figure 5 This is a structural diagram of a speech processing system according to an embodiment of the present invention. Detailed Implementation

[0028] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0029] In related technologies, hearing aids employ processing methods centered on enhancing or selectively amplifying the original sound signal. However, for people with severe hearing impairment, simply amplifying ambient sounds often fails to effectively improve language comprehension. In particular, the effect is limited when the original sound is affected by noise, reverberation, distance, or the hearing loss curve of the hearing-impaired individual. Moreover, the difficulty in understanding conversations over a long period can lead to psychological burnout for users. This psychological aversion can reduce the willingness to communicate and even indirectly weaken the stress response of the auditory nerve.

[0030] To address the aforementioned issues, a first aspect of this invention provides a speech processing method suitable for people with hearing impairments. This method can accurately identify target speakers in complex environments and assist users in accurately understanding dialogue content, thereby improving the user experience.

[0031] The following is for reference. Figure 1 This invention describes a speech processing method applicable to people with hearing impairments, as described in embodiments of the present invention. Figure 1 As shown, the method includes at least steps S1-S4.

[0032] Step S1: Acquire ambient audio signals around the user and identify speech segments related to human voices from the ambient audio signals.

[0033] Specifically, after a user wears a hearing aid, the ambient audio signal around the user can be continuously collected by the sound acquisition module on the hearing aid and sent to the controller of the voice processing device. The sound acquisition module may include at least one microphone and / or a VAD (Voice Amplifier) ​​detector.

[0034] For example, considering that environmental audio signals contain a variety of different sounds, by classifying them, audio signals with strong correlation or high similarity can be classified into the same type of sound information. Then, based on the key features corresponding to each type of sound information, speech segments about human voices can be identified. The key features include at least one or more of the following: time domain features, frequency domain features, semantic features, and location features.

[0035] Among these features, temporal features can include sound volume, sound intensity, duration of phonation, and sound wave waveform; frequency domain features can include frequency, spectrum, and timbre; semantic features can include speech text content, language type, and intonation; and location features can include orientation and distance. Therefore, based on these key features, speech segments related to human voices can be identified independently or in combination, thus achieving effective separation of human voices for subsequent processing.

[0036] Step S2: Based on the speech feature information of the speech segment, determine whether the speech segment corresponds to a certain target speaker, and the target speaker and the user conduct voice communication.

[0037] Specifically, considering that the identified speech segments about human voices may contain speech segments from multiple speakers, speech feature information can reflect the essential characteristics of the speaker's identity, language content, or emotional state. Speaker separation is performed on the environmental audio signal to extract the speech feature information of each speech segment. Based on the speech feature information, it is determined whether each speaker is the target speaker communicating with the user.

[0038] Step S3: If yes, convert the speech fragments of the target speaker into language elements.

[0039] Among these, linguistic elements include the semantic information of a speech segment, the spectral energy distribution characteristics of each frame within the speech segment, and the relative duration relationships between syllables within the speech segment. In other words, linguistic elements represent not only the semantics of a speech segment but also the physical profile of the target speaker's voice. Thus, by analyzing the physical characteristics of a speech segment—the linguistic elements—as described above, it is beneficial to preserve as much of the speaker's original voice's fundamental frequency, prosody, and other non-semantic information as possible for the user. This allows the user to accurately understand the dialogue content and also perceive the speaker's natural acoustic characteristics, such as vocal emotions.

[0040] Furthermore, considering the varying levels of comprehension among different users, especially the elderly who often encounter vocabulary beyond their daily understanding or face dialect barriers, this application goes beyond simply converting speech fragments into language elements. Instead, it transforms these fragments into language elements tailored to the user's comprehension. For instance, addressing the communication habits and interaction delays of hearing-impaired users, after converting speech fragments into language elements that retain all semantic meaning, the converted language elements can be further simplified through sentence segmentation, colloquialization of unfamiliar terms, and dialect conversion. This results in standard language elements that allow hearing-impaired users to quickly understand the dialogue. Therefore, accurately converting speech fragments using these methods and subsequently providing the converted language elements to the user can better assist them in understanding the dialogue.

[0041] Step S4: Based on the user's degree of hearing loss, generate targeted digital speech signals from language elements and transmit them to the audio output module of the hearing aid.

[0042] Specifically, since the form based on language elements can accurately reflect the speech content, this application no longer simply adopts a processing method that amplifies the original acoustic signal. Instead, it converts the speech segments of the target speaker into language elements, and then generates a digital speech signal based on the language elements to transmit to the user. Therefore, compared with the processing method that amplifies the original acoustic signal, the processing method of this application can help users accurately understand the dialogue content, effectively improve users' language comprehension ability, stimulate users' interactive enthusiasm, and improve the user experience.

[0043] According to the speech processing method for people with hearing impairments according to embodiments of the present invention, speech feature information can reflect the essential characteristics of the speaker's identity, language content, or emotional state. Therefore, by using the speech feature information of speech segments, it is determined whether the speech segment corresponds to the target speaker, thereby effectively and accurately identifying the target speaker from a complex environment. Then, the speech segment of the target speaker is converted into language elements. Since the form of language elements can accurately reflect the speech content and the physical characteristics of the original sound, a digital speech signal is generated based on the language elements and transmitted to the user. Therefore, compared with the processing method of amplifying the original acoustic signal, the processing method of this application can help the user accurately understand the dialogue content and improve the user experience.

[0044] In some embodiments, the speech feature information includes at least one of voiceprint features, directional features, semantic context features, and interaction distance features. Therefore, by fusing multiple speech features from biological, spatial, linguistic logic, and physical distance dimensions, it is beneficial to achieve complementarity between different features, avoiding the problem of single features easily failing in noisy environments or when the speaker is far away, thus improving recognition accuracy and reliability.

[0045] Specifically, different speech feature information can be adaptively selected to identify the target speaker based on different scenarios. For example, in a quiet environment, only voiceprint features can be used; in a noisy multi-person environment, directional features and interaction distance features can be fused for identification; in a multi-person interruption scenario, voiceprint features, directional features, and semantic context features can be combined for identification.

[0046] For example, voice activity detection is performed on the environmental audio signal to identify time periods containing human voices. Speaker clustering is then performed within these time periods to separate speech segments from different speakers. Direction estimation is performed on each separated speaker's speech segment to determine the spatial location features of each speaker. These spatial location features are one or more of direction features and interaction distance features. Furthermore, based on the speaker's spatial location features, it is determined whether the speaker is within a predetermined angle range in front of the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. And / or, based on the semantic context features of the speaker's speech segments, it is determined whether they contain dialogue content related to the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. And / or, based on the speaker's historical interaction features with the user, it is determined whether the speaker is a preset target contact for the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. Here, historical interaction features refer to characteristics such as behavior, voice, operation data, and voiceprint features retained from previous interactions with the user.

[0047] In some embodiments, semantic information is one or more of textual information, semantic text, and phoneme sequences.

[0048] Among them, a phoneme sequence refers to a list of phonemes arranged in chronological order, used to accurately represent speech segments, and is the core intermediate representation form for technologies such as speech recognition and synthesis.

[0049] Specifically, the speech segment corresponding to the target speaker can be converted into text information, and then a digital speech signal can be generated based on the text information for transmission to the user. Alternatively, the speech segment corresponding to the target speaker can be converted into semantic text, and then a digital speech signal can be generated based on the semantic text for transmission to the user. Or, the speech segment corresponding to the target speaker can be converted into a phoneme sequence, and then a digital speech signal can be generated based on the phoneme sequence for transmission to the user. Alternatively, the speech segment corresponding to the target speaker can be simultaneously converted into semantic text and a phoneme sequence, and then a digital speech signal can be generated based on the semantic text and phoneme sequence for transmission to the user. Thus, by fusing semantic text and phoneme sequences, a mutual verification effect can be achieved between the two, which is beneficial for further improving language understanding capabilities.

[0050] In some embodiments, targeted digital speech signals will be generated based on language elements, including: configuring a corresponding voice role for each target speaker when there are multiple target speakers; and generating a digital speech signal for each target speaker based on the voice role and language elements.

[0051] The following section provides a detailed explanation of the multi-speaker digital voice role mapping process, which involves configuring a corresponding voice role for each target speaker. Figure 2 As shown, it specifically includes steps S11-S15.

[0052] Step S11: Acquire ambient audio signals around the user.

[0053] Step S12: Determine whether a target speaker exists based on the ambient audio signal.

[0054] Specifically, speaker separation of environmental audio signals involves identifying speech segments related to human voices, obtaining speech segments from at least two speakers, extracting speech feature information from each speaker, and determining whether each speaker is the target speaker communicating with the user based on the speech feature information. The speech feature information includes at least one of voiceprint features, directional features, semantic context features, and interaction distance features.

[0055] For example, voice activity detection is performed on the environmental audio signal to identify time periods containing human voices. Speaker clustering is then performed within these time periods to separate speech segments from different speakers. Direction estimation is performed on each separated speaker's speech segment to determine the spatial location features of each speaker. These spatial location features are one or more of direction features and interaction distance features. Furthermore, based on the speaker's spatial location features, it is determined whether the speaker is within a predetermined angle range in front of the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. And / or, based on the semantic context features of the speaker's speech segments, it is determined whether they contain dialogue content related to the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. And / or, based on the speaker's historical interaction features with the user, it is determined whether the speaker is a preset target contact for the user. If so, a target speaker is identified; otherwise, the speaker is not the target speaker for communication with the user. Here, historical interaction features refer to the behavioral, voice, and operational data characteristics retained from previous interactions with the user.

[0056] Step S13: In the presence of at least one target speaker, separate the speech segment corresponding to at least one target speaker based on the environmental audio signal.

[0057] Among them, speech segments refer to raw sound waves that have not been semantically interpreted.

[0058] Step S14: Configure a corresponding voice role for at least one target speaker. The voice role is defined by a set of role-based physical parameters, which are used to control the filtering characteristics of physical filtering of speech segments.

[0059] Specifically, this application no longer uses a uniform original timbre to provide feedback to users for different speakers. Instead, it assigns a corresponding voice role to at least one target speaker. This allows different voice roles to be used to provide feedback to users for different speech segments from different target speakers. As a result, users can clearly distinguish different target speakers through the corresponding voice roles. In a diverse and changing timbre environment, while maintaining speech clarity, this approach effectively stimulates user interaction, increases auditory interest, enhances auditory nerve sensitivity, and improves the user experience.

[0060] Step S15: Physically filter the speech segment according to the voice role to generate a digital speech signal, and transmit it to the audio output module of the hearing aid.

[0061] Specifically, considering that the environmental audio signal contains a variety of different sounds, in order to classify and identify the speech segment corresponding to at least one target speaker and combine it with the corresponding voice role, a time-varying filter, IIR (Infinite Impulse Response) filter, or convolutional filter is constructed using the role-specific physical parameters contained in the voice role. This filter directly applies physical filtering to the original speech segment, generating a role-specific enhanced signal that retains the original time structure and prosodic features of the speech segment, but whose spectral envelope is adjusted to match the role-specific physical parameters. This generates a digital speech signal, thereby achieving the goal of combining role-specific timbre with hearing aid functionality, effectively solving the problem of monotonous sound caused by different speakers using the same original timbre. For other sound segments in the environmental audio signal besides the speech segment corresponding to at least one target speaker, the digital speech signal corresponding to that speech segment can be reduced, ignored, or not output.

[0062] In some embodiments, for generating a digital speech signal based on voice roles and speech segments, the speech segments can be converted into language elements, which include text information, semantic text, semantic units, or one or more of phoneme sequences and phoneme sequences. Then, a digital speech signal is generated based on the voice roles and language elements. Next, the digital speech signal is subjected to frequency band enhancement, speech rate adjustment, or pitch adjustment according to the user's hearing loss curve. Finally, the adjusted digital speech signal is transmitted to the user. Thus, instead of simply amplifying the original speech segments to obtain a digital speech signal, a digital speech signal is reconstructed based on language elements, thereby improving speech clarity.

[0063] According to the speech processing method for people with hearing impairments according to embodiments of the present invention, when there is at least one target speaker, by configuring a corresponding voice role for the at least one target speaker, the user can clearly distinguish different target speakers by the corresponding voice role after the speech segments of the at least one target speaker are fed back to the user. This combines role-based timbre with hearing aid function, effectively solving the problem of monotonous sound caused by different speakers using the same original timbre, increasing the user's auditory interest while maintaining speech clarity, and improving the user experience.

[0064] In some embodiments, acquiring ambient audio signals around the user includes: acquiring ambient audio signals within the sound reception range, thereby reducing the influence of excessive and complex sounds and avoiding confusion for the user due to excessive sound.

[0065] In some embodiments, the pickup range is determined based on the hearing aid mode. The hearing aid mode may include quiet mode, outdoor mode, music mode, telephone mode, pickup range, etc., and is not limited thereto. Specifically, the pickup range differs in different hearing aid modes, so adjusting the pickup range based on the hearing aid mode can optimize the hearing aid effect in different modes.

[0066] In some embodiments, the reception range is determined based on environmental features around the user.

[0067] Specifically, the hearing aid effect varies depending on the environment. Therefore, to adapt to environmental changes, the sound reception range in this application is not fixed but can be adaptively adjusted based on the environmental characteristics of the user's surroundings. That is, the sound reception range of the environmental audio signal can be automatically narrowed or widened depending on the environment. This adaptive adjustment of the sound reception range can reduce audio superposition interference, facilitate accurate identification of the target speaker, reduce the burden of auditory discrimination, and the whole process can be automatically adjusted, reducing the difficulty of operation, especially eliminating usage barriers for the elderly and improving the convenience of use.

[0068] In some embodiments, environmental feature information includes at least one of ambient noise intensity, sound source distance, number of target speakers, user posture, and signal-to-noise ratio.

[0069] For example, three levels of ambient noise intensity can be set: low noise intensity, medium noise intensity, and high noise intensity. If the ambient noise intensity is low, then the ambient audio signal within the corresponding reception range is acquired; if the ambient noise intensity is medium, then the ambient audio signal within the corresponding reception range is acquired; if the ambient noise intensity is high, then the ambient audio signal within the corresponding reception range is acquired. The higher the ambient noise intensity, the narrower the reception range.

[0070] And / or, three levels of sound source distance can be set: near distance (e.g., 0.5m-1.5m), medium distance (e.g., 1.5m-3m), and far distance (e.g., greater than 3m). If the sound source distance is near, then the ambient audio signal within the corresponding sound reception range will be acquired; if the sound source distance is medium, then the ambient audio signal within the corresponding sound reception range will be acquired; if the sound source distance is far, then the ambient audio signal within the corresponding sound reception range will be acquired. The farther the sound source distance, the wider the sound reception range.

[0071] And / or, the more numerous and concentrated the target speakers, the narrower the pickup range. And / or, if the user is in a static posture, such as sitting or standing still, the sound field is relatively stable, so the pickup range can be narrowed; if the user is moving slowly, the sound source is constantly swaying slightly and changing direction, so the pickup range can be appropriately widened to avoid intermittent sound; of course, when determining the pickup range, the pickup direction can also be adaptively adjusted according to the user's posture. For example, if the user is turning their head or turning to the side, the pickup direction can be synchronously deflected with the user's posture while keeping the pickup range unchanged. And / or, if the signal-to-noise ratio is higher, it means that the human voice is significantly stronger than the noise, so the pickup range can be appropriately widened to retain more environmental information; if the signal-to-noise ratio is lower, it means that the noise is significantly stronger than the human voice, so the pickup range can be appropriately narrowed to accurately capture the speech segments of the target speaker.

[0072] In some embodiments, identifying at least one speech segment corresponding to a target speaker based on ambient audio signals includes: when the number of target speakers is less than a first preset number, identifying at least one speech segment corresponding to a target speaker within a first sound-receiving range based on ambient audio signals; when the number of target speakers is greater than or equal to the first preset number, identifying at least one speech segment corresponding to a target speaker within a second sound-receiving range based on ambient audio signals, wherein the upper limit of the second sound-receiving range is less than the upper limit of the first sound-receiving range. That is, when the number of target speakers is small, the sound-receiving range can be automatically widened, such as the first sound-receiving range being a circular area with the user as the center and a radius of 3m, thus providing the user with more environmental information while maintaining normal conversation and increasing auditory interest; when the number of target speakers is large, the sound-receiving range can be automatically narrowed, such as the second sound-receiving range being an area with the user as the center, a radius of 3m, and an angle of 60° to 90° directly in front of the user, thereby prioritizing the identification of speech segments from speakers to the user's side or opposite, thereby effectively assisting the user in distinguishing speakers based on voice roles, reducing the superposition interference of multiple voices, and further reducing the auditory discrimination burden.

[0073] In some embodiments, configuring a corresponding voice role for at least one target speaker includes: configuring a corresponding voice role for at least one target speaker based on a timbre template library, wherein the timbre template library is established based on at least one of voice type, pitch, speech rate and timbre; and / or, configuring a corresponding voice role for at least one target speaker based on a timbre cloning model, wherein the timbre cloning model is trained based on speech samples corresponding to the target speaker.

[0074] Specifically, a voice template library can be stored in the language processing device. This library can be pre-established at the factory based on at least one of voice type, pitch, speech rate, and timbre, and / or, during actual use, the user can customize at least one of voice type, pitch, speech rate, and timbre according to actual needs, thereby establishing the voice template library. Each voice role includes different voice types, pitches, speech rates, timbres, or combinations thereof, and each voice role is audibly distinguishable. Furthermore, in actual use, when there is at least one target speaker, a unique digital voice role is assigned to each target speaker from the voice template library, and the mapping relationship between the target speaker's voiceprint features and the digital voice role is recorded. This maintains the consistency of the mapping relationship in subsequent dialogues, ensuring that the same target speaker uses the same voice role in multiple dialogues, allowing the user to clearly distinguish different target speakers through the corresponding voice roles. In assigning a unique digital voice role to each target speaker from the timbre template library, the corresponding voice role can be determined by calculating cosine similarity or Euclidean distance based on the voiceprint features of the target speaker.

[0075] The types of human voices can include male voices, female voices, and children's voices, and there are no restrictions on this.

[0076] In some embodiments, for the timbre template library, the number of voice roles contained in the timbre template library stored by the speech processing device, such as the number of preset timbres, is greater than or equal to the number of voice roles that the speech processing device supports configuring. For example, taking the speech processing device as a terminal device such as a mobile terminal, the timbre template library stored in the mobile terminal application contains 10 to 30 preset timbres and supports user-customized timbres, while the mobile terminal simultaneously supports parallel speech separation and role mapping for 5 to 10 target speakers. As another example, the local timbre template library of a hearing aid may contain 4 to 8 preset timbres, while the hearing aid's NPU simultaneously supports parallel speech separation and role mapping for 2 to 4 target speakers. This configuration not only provides more choices but also improves the device's processing efficiency to meet different user needs.

[0077] Alternatively, voice roles can be configured for a target speaker through voice cloning. This involves generating a digital voice role that closely resembles the speaker's original voice based on the speaker's voiceprint characteristics. For example, an initial version of the voice cloning model is established in the speech processing device. During actual use, the model is progressively optimized based on multiple speech samples from the target speaker. This voice cloning model-based technology enables voice roles to achieve a high degree of similarity to the target speaker's voiceprint characteristics. Furthermore, this technology overcomes traditional limitations, no longer restricted by the number of voice roles, thus greatly enriching the voice role library and providing users with more diverse choices. It increases auditory enjoyment while maintaining speech clarity, improving the user experience.

[0078] The timbre cloning model can include an encoder, a synthesizer, and a vocoder. The encoder is configured to extract the speaker's timbre vector, the synthesizer is configured to generate a Mel spectrum, and the vocoder is configured to generate a waveform.

[0079] In some embodiments, the method further includes: storing the timbre cloning model locally when the similarity of the timbre cloning model reaches a preset threshold.

[0080] Specifically, the similarity of the timbre cloning model refers to the similarity between the voice character cloned by the timbre cloning model and the actual timbre of the target speaker. Therefore, when the similarity of the timbre cloning model reaches a preset threshold, it means that the voice character cloned by the timbre cloning model is very similar to the actual timbre of the target speaker. Thus, the timbre cloning model can be stored locally. As a result, when the user has a conversation with the target speaker again, there is no need to train again with the speech sample corresponding to the target speaker. The timbre cloning model corresponding to the target speaker can be directly retrieved to configure the voice character, thereby reducing the amount of computation.

[0081] In some embodiments, considering the large computational load required for the voice cloning model, the training step can be performed by a cloud server. When the similarity of the voice cloning model reaches a preset threshold, the model is distributed to a speech processing device, such as a mobile terminal or hearing aid, for subsequent voice role configuration. This reduces the computational load on the speech processing device and facilitates its lightweight design. For example, the cloud device supports real-time voice cloning, generating a voice role similar to the speaker's original voice timbre based on the target speaker's voiceprint features. The cloud device can flexibly expand its computing resources, supporting the simultaneous processing of parallel speech separation and role mapping for a certain number of target speakers, such as more than 20. The cloud device can also maintain a user's historical dialogue database, supporting consistent speaker identification and role mapping across devices and sessions.

[0082] In some embodiments, the method further includes: when there are multiple target speakers and the number of target speakers is greater than a preset number, configuring a corresponding voice role for each target speaker in sequence based on the scene feature information corresponding to each target speaker.

[0083] Scene feature information can be understood as attributes that are unique to the current hearing aid scenario and can be perceived / extracted, such as at least one of the following: person characteristics, vocalization sequence, interaction frequency, and interaction distance. Person characteristics can be understood as all inherent / temporary attribute information used to identify, describe, and distinguish a person, such as behavioral characteristics and identity characteristics.

[0084] Specifically, considering the limited processing capacity of voice processing devices, in this application, when there are multiple target speakers and the number of target speakers exceeds the preset number, it indicates that the number of detected target speakers exceeds the local processing capacity of the voice processing device. Therefore, the priority order of configuring voice roles is determined based on the scene feature information corresponding to each target speaker. Then, the corresponding voice role is configured for each target speaker in sequence. That is, the voice role is configured for each target speaker according to this priority order. In the current dialogue, the target speaker with higher priority is configured with voice roles first, while the target speaker with lower priority enters the waiting queue. Finally, after configuration in sequence, the digital voice signal corresponding to each target speaker is transmitted to the user in sequence.

[0085] The priority order for configuring voice roles based on scene feature information corresponding to each target speaker can be customized according to actual needs and is not restricted. For example, the priority order for configuring voice roles can be determined based on the vocalization order of each target speaker. Taking a hearing aid as an example, the hearing aid's NPU can simultaneously process parallel speech separation and role mapping for B target speakers. Therefore, when the number of detected target speakers exceeds B, voice roles are assigned to the speaker in the current dialogue first, and the remaining speakers enter a waiting queue.

[0086] In some embodiments, the method further includes: when multiple target speakers exist, performing spatial audio rendering on the digital speech signal corresponding to each target speaker. Spatial audio rendering refers to processing ordinary digital speech signals into 3D sound effects that are "directional, distance-based, and environmentally immersive." In other words, different spatial positions are assigned to different target speakers. This not only allows users to clearly distinguish between different target speakers but also enables them to perceive that each target speaker is coming from a different direction, thereby further stimulating user interaction, increasing auditory interest, enhancing auditory nerve sensitivity, and improving the user experience. The spatial position can be based on the relative position of the target speaker in the real environment or a preset fixed position in a virtual sound field.

[0087] For example, in a meeting, if speaker B is seated on the left and speaker C on the right, the speech processing device can use spatial audio rendering to assign digital sound from the left to B and digital sound from the right to C. Thus, the user can not only distinguish speakers by timbre but also perceive their location by sound direction. This can be achieved by varying the time difference between the arrival times of different speakers at both ears to present directional information, or by varying the time difference between the arrival times of different speakers at both ears and / or by differences in sound levels in the two ear canals to present directional / distance information.

[0088] In some embodiments, spatial audio rendering is performed on the digitized speech signal corresponding to each target speaker based on the sound source location of each target speaker.

[0089] Specifically, considering that the target speaker's posture may change during a conversation, if the spatial audio rendering remains constant, it will affect the user's ability to distinguish the speaker. For example, if the target speaker is on the user's left, its corresponding digital voice signal will be fed back to the user in the left direction. If the target speaker subsequently moves from the user's left to the user's right, and its corresponding digital voice signal is still fed back to the user in the left direction, it will create an auditory contrast for the user. Therefore, to accurately distinguish the target speaker, in addition to assisting the user in distinguishing the speaker based on the voice role, spatial audio rendering is performed on the digital voice signal corresponding to each target speaker based on the sound source position of each target speaker during the conversation. This avoids the situation where the user frequently turns their head to find the speaker due to the speaker's movement or sound source shift, avoids the problem of sound interruption and distortion after the sound source moves, and improves the auditory continuity in multi-person dynamic communication.

[0090] In some embodiments, differentiated gain processing is performed on the digital speech signal corresponding to each target speaker based on the sound source location of each target speaker. The gain processing includes at least one of frequency band adjustment and pitch adjustment. That is, the frequency band and / or pitch of the digital speech signal can be adjusted as the sound source location of the target speaker changes, so that the sound localization changes in real time with the speaker's location, allowing the user to perceive the speaker's changes simultaneously through hearing and vision, thereby enhancing the user's interactive immersion.

[0091] In some embodiments, when there are multiple target speakers, the digital voice signal corresponding to each target speaker can be configured with an independent audio channel for transmission, so as to achieve the effect of hierarchical management, independent audio channel output, and no interference between the two.

[0092] In some embodiments, the method further includes adjusting key information and non-key information in the digitized speech signal, with the key information adjusted to a first level of emphasis and the non-key information adjusted to a second level of emphasis, wherein the first level of emphasis is higher than the second level of emphasis. That is, when a speech segment of the target speaker contains key information, the emphasis on the key information is increased. For example, specific sound effects or speech adjustment methods can be used to emphasize the key information. This processing method not only makes the key information more prominent but also attracts the user's attention, enhances the user's immersion, and allows the user to become more engaged in the speech content, improving the user's understanding and memory of the content, and increasing the user's interest and participation in the content.

[0093] Key information may include numerical and identifying keywords such as numbers, time, location, names, phone numbers, or user-preset keywords of interest, and there are no restrictions on this.

[0094] For example, if the target speaker B says, "The meeting will be held in the conference room at 3 pm tomorrow," the speech processing device detects "3 pm" as a time keyword. Therefore, when outputting the digitized speech signal, a slightly slower speaking speed and a higher pitch can be used to emphasize this time information.

[0095] In some embodiments, the method of this application further includes: Since different users, such as the elderly and children, have different levels of response to reminders, a fixed processing method is used to process the digital voice signal, which cannot meet personalized needs. Therefore, before transmitting the signal to the audio output module, the digital voice signal is adjusted according to the user's hearing aid configuration information. This adjustment includes at least one of the following: frequency band enhancement, speech rate adjustment, pitch adjustment, frequency response adjustment such as loudness compression, pause enhancement, or repetition of numerical and identifier-type keywords. This allows for multiple options in processing the digital voice signal, resulting in a more diverse range of sounds that the user hears, thereby further stimulating user interaction and improving the user experience.

[0096] In some embodiments, the hearing aid configuration information is determined based on local configuration information and / or network configuration information; the hearing aid configuration information includes at least one of the user's hearing loss curve, volume, timbre selection, and hearing aid mode.

[0097] Local configuration information refers to personalized parameters and setting instructions stored on the voice processing device, which can be manually customized by relevant personnel. Network configuration information refers to configuration parameters and setting instructions provided by external devices, which can also be manually customized by relevant personnel. These external devices have a communication connection with the voice processing device; for example, the external device communicates with the voice processing device via Bluetooth, or via a network such as 3G, 4G, or 5G, without restriction.

[0098] The hearing aid configuration information may be determined solely based on local configuration information; or it may be determined solely based on network configuration information; or it may be determined based on both local and network configuration information.

[0099] In some embodiments, the hearing aid configuration information is determined based on local configuration information and network configuration information. Therefore, basic configuration information such as volume and hearing loss curve can be set on the voice processing device, while some / all configuration information such as tone selection and hearing aid mode can be set on an external device. This facilitates quick operation and response by the user and is conducive to the lightweight design of the voice processing device. On the other hand, it also facilitates remote operation, especially for users who are inconvenient or unfamiliar with the usage process, such as the elderly and children, who can be remotely operated by other personnel to assist the user in completing the hearing aid configuration.

[0100] For example, taking a hearing aid as the voice processing device and a mobile terminal as the external device, the hearing aid locally stores local configuration information such as a tone template library, voiceprint feature vector, and hearing loss curve. The hearing aid's processing latency is 30ms~100ms, resulting in a fast response speed. The mobile terminal connects to the hearing aid via Bluetooth and provides a parameter adjustment interface, including network configuration information such as volume adjustment, tone selection, hearing loss curve configuration, and hearing aid mode switching. This configuration is then provided to the hearing aid. When the mobile terminal disconnects via Bluetooth, the hearing aid can continue to operate independently using the most recent parameter configuration.

[0101] In some embodiments, the hearing aid configuration information is determined based on hearing aid keywords extracted from speech segments about human voices. That is, the voice processing device supports voice adjustment of the hearing aid configuration information, thereby allowing users who are unable or unwilling to operate manually, such as the elderly, to modify the configuration via voice, improving the user experience.

[0102] Among them, hearing aid keywords refer to voice segments related to hearing aid configuration, such as increasing hearing aid volume, increasing sound reception range, switching music modes, etc. There are no restrictions on these. For example, hearing aid keywords can be preset or customized. Based on this, when one of the above hearing aid keywords is identified from the environmental audio signal, the hearing aid configuration information required by the user is determined based on the hearing aid keyword, and the digital voice signal is adjusted according to the hearing aid configuration information, so that the feedback digital voice signal is more in line with the user's auditory experience.

[0103] In some embodiments, the method further includes: if non-human voice segments and / or the user's own voice segments are identified from the ambient audio signal, prohibiting the transmission of the digital audio signals corresponding to the non-human voice segments and / or the user's own voice segments to the user; or, reducing the volume of the digital audio signals corresponding to the non-human voice segments and / or the user's own voice segments. This helps to reduce the transmission of non-conversational content, thereby ensuring the user's focus on the conversational content.

[0104] In some embodiments, the method further includes: if it is determined that there is no correspondence between the speech segment and the target speaker, determining whether the speech segment meets the reminder conditions; if so, providing a reminder based on the speech segment, wherein the reminder includes at least one of phrase speech, prompt tone, spatialized audio, and vibration feedback.

[0105] The reminder process will be explained in detail below, such as... Figure 3 As shown, the process specifically includes steps S21-S24. The voice processing device includes a first audio channel and a second audio channel.

[0106] Step S21: Acquire ambient audio signals around the user.

[0107] Specifically, an ambient audio signal around the user can be continuously collected by a sound acquisition device and sent to the controller of a voice processing device. The sound acquisition device may include at least one microphone and / or a VAD (Voice Amplifier) ​​detector.

[0108] Step S22: Identify non-dialogue environment events and / or dialogue environment events based on the ambient audio signals.

[0109] Dialogue-based environmental events can be understood as environmental events involving real-time communication, such as current or remote conversations. Non-dialogue-based environmental events can be understood as environmental events involving one-way sound without requiring immediate interactive feedback. For example, non-dialogue-based environmental events can include transportation-related events, alarm-related events, natural environment-related events, and human activity-related events, without limitation. Transportation-related events can include sounds from cars, electric vehicles, motorcycles, trains, and airplanes; alarm-related events can include fire alarms, burglar alarms, car horns, and ambulance sirens; natural environment-related events can include thunder, rain, and wind; and human activity-related events can include knocking, footsteps, shouting, and the sound of breaking glass.

[0110] Step S23: When a non-dialogue environment event meets the preset reminder conditions, the non-dialogue environment event is processed through the first audio channel to remind the user.

[0111] Step S24: Process the dialogue environment events through the second audio channel to transmit them to the user.

[0112] Specifically, to avoid the problem of conflicts between dialogue and safety reminders affecting normal user communication, this application no longer uses the same audio channel to feed back non-dialogue environment events and dialogue environment events to the user. Instead, separate audio channels are established for non-dialogue environment events and dialogue environment events. Thus, by identifying non-dialogue environment events and / or dialogue environment events from the ambient audio signal, and then outputting them using the corresponding independent audio channels, the two are managed in a hierarchical manner and do not interfere with each other. Therefore, even in hearing aid scenarios where dialogue and safety reminders coexist, non-dialogue environment events can still be output through an independent reminder channel, i.e., the first audio channel, to remind the user. This process will not affect the user's normal dialogue, effectively solving the problem of conflicts between dialogue and safety reminders affecting normal user communication, achieving the goal of balancing normal dialogue and safety reminders, and improving the user experience.

[0113] It is understandable that if only non-dialogue environmental events are identified from the ambient audio signal, then only those non-dialogue environmental events need to be processed. If only dialogue environmental events are identified from the ambient audio signal, then only those dialogue environmental events need to be processed. If both non-dialogue environmental events and dialogue environmental events are identified from the ambient audio signal, then they can be processed synchronously, meaning that both non-dialogue environmental events and dialogue environmental events can be fed back to the user at the same time; or they can be processed asynchronously, meaning that they can be fed back to the user sequentially, for example, non-dialogue environmental events can be fed back to the user first to ensure the user's safety.

[0114] In some embodiments, the priority of the first audio channel may be set higher than that of the second audio channel, thereby ensuring that the user perceives the reminder information first.

[0115] In some embodiments, processing dialogue environment events includes: identifying a speech segment corresponding to the target speaker communicating with the user based on the dialogue environment event; converting the speech segment into language elements, which include text information, semantic text, semantic units, or one or more of phoneme sequences and phoneme sequences; and then generating a digital speech signal based on the language elements for transmission to the user. Thus, instead of simply amplifying the original speech segment to obtain a digital speech signal, a digital speech signal is reconstructed based on language elements, thereby improving speech clarity.

[0116] According to the speech processing method for people with hearing impairments according to embodiments of the present invention, based on the design of a first audio channel and a second audio channel, non-dialogue environment events and dialogue environment events are separated. The first audio channel is used to process non-dialogue environment events, and the second audio channel is used to process dialogue environment events. This achieves the effect of hierarchical management of the two, independent audio channel output, and no interference between them, so as to achieve the purpose of balancing normal dialogue communication and safety reminders and improve the user experience.

[0117] In some embodiments, identifying non-dialogue environment events and / or dialogue environment events based on ambient audio signals includes: classifying sound segments in the ambient audio signals to obtain multiple types of sound information. That is, considering that the ambient audio signals contain a variety of different sounds, by classifying them, audio signals with strong correlation or high similarity can be classified into the same type of sound information. Then, non-dialogue environment events and / or dialogue environment events are identified based on key features corresponding to each type of sound information. The key features include at least one or more of time-domain features, frequency-domain features, semantic features, and location features.

[0118] Among these features, temporal features can include sound volume, sound intensity, duration of sound production, and sound wave waveform; frequency domain features can include frequency, spectrum, and timbre; semantic features can include speech-text content, language type, and intonation; and location features can include orientation and distance. Based on these key features, non-dialogue environment events and dialogue environment events can be identified independently or in combination, thus achieving effective separation between the two. This allows for subsequent output of each event through an independent audio channel, achieving the goal of balancing normal conversation and safety alerts.

[0119] Preset alert conditions can be pre-set at the factory based on actual conditions, and / or can be customized by the user based on their own circumstances; there are no restrictions on this. In some embodiments, preset alert conditions include at least one or more of the following: the event type corresponding to the non-dialogue environment event belongs to a preset high-priority event type; the sound intensity corresponding to the non-dialogue environment event is greater than or equal to a preset threshold; the event direction corresponding to the non-dialogue environment event meets preset collision conditions; the event distance corresponding to the non-dialogue environment event is within a preset warning range. Therefore, based on the above preset alert conditions, hearing-impaired individuals can perceive specific sound events in the environment, especially dangerous sound events, to the greatest extent possible, which is beneficial for optimizing alert services for hearing-impaired individuals in different scenarios.

[0120] Among them, the preset threshold, preset collision conditions and preset warning range can be set based on actual conditions such as the user's hearing level, and there are no restrictions on them.

[0121] For example, preset high-priority event types may include alarm types, vehicle horn types, and siren types. In practical applications, it will be determined whether the event type corresponding to the non-dialogue environment event belongs to one of the above preset high-priority event types. If it does, the non-dialogue environment event meets the preset reminder conditions; otherwise, the non-dialogue environment event does not meet the preset reminder conditions. The above example of preset high-priority event types is only set from a security perspective. Of course, it can also be customized based on event types that users are interested in. For example, considering that users such as the elderly may experience communication, psychological, and physical problems due to hearing impairment, entertainment event types such as songs and operas can be customized. This can further stimulate users' interest in using the device, effectively enrich their life experience, and reduce their psychological and physiological burdens.

[0122] In some embodiments, processing non-dialogue environment events through a first audio channel includes: generating a first reminder message based on at least one of the event type, event direction, event distance, event duration, and event handling measures of the non-dialogue environment event; and transmitting the first reminder message through the first audio channel.

[0123] Among them, incident handling measures refer to the handling suggestions provided to users for incidents in non-dialogue environments, so that users can respond and take action in a timely manner through these suggestions.

[0124] For example, alert messages can be generated based on the event type and direction of a non-dialogue environment event. For instance, if a user is walking outdoors and the voice processing device detects the engine sound of a vehicle behind them, it generates a first alert message, "Car behind," which is played through a separate audio channel (the first audio channel). Simultaneously, the user can continue processing conversations with friends ahead without interrupting the dialogue. Alternatively, the first alert message can be generated based on the event type, direction, and distance of a non-dialogue environment event. For example, if a user is walking on a street and the voice processing device recognizes the engine sound of a car to their right rear, it generates an alert message, "Car to your right rear," which is played through a separate audio channel (the first audio channel). This allows the user to determine the direction of the approaching vehicle. Or, the first alert message can be generated based on the event type, direction, distance, and handling measures of a non-dialogue environment event. For example, if a user is at a busy intersection and the voice processing device recognizes an electric scooter rapidly approaching from about 5 meters to their right rear, and determines the safety level to be high based on real-time traffic information, it outputs an alert message, "Electric scooter rapidly approaching from your right rear, please be careful," which is played through a separate audio channel (the first audio channel). Therefore, this application provides users with reminders of different levels of precision from multiple perspectives, which helps to further enrich the reminder service.

[0125] In some embodiments, the method further includes: acquiring the status information of the hearing aid; generating second reminder information based on the status information when the status information meets the status reminder conditions; and transmitting the second reminder information through the first audio channel.

[0126] Among them, the hearing aid status information refers to the hearing aid's own condition reflected during the user's wearing and use process, such as whether it is worn correctly, whether it is malfunctioning, communication connection status, battery level, etc. In this way, when the status information meets the status reminder conditions, a second reminder message is generated based on the status information and transmitted through the first audio channel, so that the user can know the usage status of the hearing aid in a timely manner, so as to solve the problem in advance and avoid the psychological panic caused by not being able to hear.

[0127] In some embodiments, the status alert conditions include at least one or more of the following: the hearing aid's battery level is below a preset battery threshold; the hearing aid is detected as being worn ineffectively; the hearing aid's sound outlet is detected as blocked. Of course, the above are merely examples, and other status alert conditions can be set based on actual needs, provided that false alarms can be avoided; there are no restrictions on this.

[0128] The second reminder information may include a digitized voice signal indicating the status information and corresponding processing measures. This avoids the problem that users, especially the elderly, do not know how to handle abnormal hearing aid statuses. By providing processing measures, it is easier to guide users to resolve hearing difficulties in a timely manner.

[0129] In some embodiments, the method further includes: determining the reminder level corresponding to the first reminder message or the second reminder message; and determining the reminder method corresponding to the reminder message based on the reminder level, wherein the reminder method includes at least one of text display, phrase voice, prompt tone, spatialized audio, and vibration feedback.

[0130] For example, a voice processing device can be configured with multiple alert levels, each with different alert methods. For instance, three alert levels can be set: Level 1, Level 2, and Level 3. The alert methods at Level 1 include short speech phrases, vibration feedback, and spatialized audio; at Level 2, short speech phrases, prompts, and spatialized audio; and at Level 3, text display, short speech phrases, prompts, spatialized audio, and vibration feedback. Therefore, if the alert level for a given message is determined to be Level 1, then the alert methods for that message will include short speech phrases, vibration feedback, and spatialized audio. Similarly, the determination methods for other alert levels are the same as described above, and will not be illustrated further here. Thus, using multiple different alert methods can maximize user security.

[0131] For some reminder methods, other smart devices besides voice processing devices, such as smartwatches and smart glasses, can be used. For example, a user wearing a voice processing device such as a hearing aid and a smartwatch can trigger an alert from all devices simultaneously—the hearing aid and smartwatch vibrate when an emergency event, such as a car horn, is detected. When a non-emergency event, such as rain, only the hearing aid plays an alert. This multi-device collaborative reminder effect facilitates further optimization of the reminder service.

[0132] In addition, for some reminder methods, other channels besides the first and second audio channels can be used for transmission. This multi-channel setup facilitates synchronous feedback of different reminder methods. For example, the voice processing device can also include a vibration channel for transmitting reminder information through tactile feedback, a visual channel for displaying text reminders on mobile terminals or smart glasses, etc., without any restrictions.

[0133] In some embodiments, the method further includes: adjusting the first reminder information or the second reminder information based on environmental characteristics information around the user before transmitting the first reminder information or the second reminder information, the adjustment including at least one of frequency band adjustment, volume adjustment and frequency adjustment.

[0134] In some embodiments, environmental feature information includes at least one of time features, ambient noise intensity, and user posture.

[0135] Specifically, the hearing aid effect varies for users in different environments. At the same time, some users, such as the elderly, cannot effectively distinguish the reminder sounds, such as being unable to distinguish between alarms and noises, or being confused by similar reminder sounds. Therefore, this application adjusts the frequency band, volume and / or frequency of the first or second reminder information based on the environmental characteristics of the user's surroundings. This optimizes the prompt sound for different situations so that users can experience different levels of prompt effects, especially to adapt to the hearing of the elderly, achieving the goal of balancing recognition and comfort.

[0136] For example, at night, considering that the user's environment is mostly quiet, a loud alert could cause panic. Therefore, the frequency and volume of the first or second alert can be adjusted based on time characteristics, such as using a low-volume, low-frequency alert at night; or, the frequency and volume of the first or second alert can be adjusted based on the ambient noise level, such as using a low-volume, low-frequency alert in low-noise conditions; or, when the user is in a sleeping posture, a low-volume, low-frequency alert can also be used to avoid waking the user up with the alert.

[0137] In some embodiments, the method further includes: uploading the environmental audio signal to the cloud device when the recognition accuracy of non-dialogue environment events is lower than a preset accuracy and the voice processing device meets normal communication conditions; receiving the recognition result fed back by the cloud device to remind the user again based on the recognition result.

[0138] Normal communication conditions may include the successful establishment of a network connection between the voice processing device and the cloud device.

[0139] Specifically, if the recognition accuracy of non-dialogue environment events is lower than the preset accuracy, it means that the current voice processing device cannot support the accurate classification of non-dialogue environment events, especially for abnormal environment events. [A1.1] Therefore, under normal communication conditions, on the one hand, the voice processing device reminds users of the recognition results of non-dialogue environment events, and on the other hand, it uploads the environmental audio signal to the cloud device, which performs secondary fine classification of the environmental audio signal. Then, the cloud device feeds back the recognition results to the voice processing device, so that the voice processing device can remind the user again based on the recognition results. Thus, the reminder service can be further optimized through edge-cloud collaborative reminders.

[0140] For example, when a voice processing device such as a mobile phone is running, the mobile application software detects an abnormal sound that cannot be accurately classified. It first outputs a basic reminder "Attention to abnormal sound" and at the same time uploads the environmental audio signal, such as the audio clip of the event, to the cloud device. If the network is available, the cloud device can quickly provide feedback such as "Construction Alert" after 5 seconds. The mobile application software then adds a reminder of "Construction Alert Ahead".

[0141] In some embodiments, the method further includes: obtaining historical response behavior information of the user when reminded; and adjusting the handling of non-dialogue environment events based on the historical response behavior information.

[0142] Specifically, different users, such as the elderly and children, react differently to reminders, especially when there are safety risks. Some users react quickly, while others react slowly. Therefore, using a fixed processing method for non-dialogue environment events cannot meet personalized needs. To address this, the voice processing device of this application obtains the user's historical response behavior information when receiving reminders, such as confirmation of receipt, ignoring or adjusting reminder settings, response time, etc. Based on the historical response behavior information, it adjusts the processing method for non-dialogue environment events. This processing method can include preset reminder conditions when judging non-dialogue environment events and the reminder method after generating reminder information. In this way, by learning the user's reminder preferences and usage habits based on the user's historical response behavior information, the preset reminder conditions and reminder methods are dynamically adjusted, thereby improving the intelligence of the voice processing device and further optimizing the reminder service.

[0143] In some embodiments, adjusting the handling of non-dialogue environment events based on historical response behavior information includes: reducing the priority of the event type corresponding to the non-dialogue environment event, or reducing the reminder level of the non-dialogue environment event, when the historical response behavior information meets preset adjustment conditions. The preset adjustment conditions are pre-set based on actual needs, such as the number of times a user ignores a certain event type greater than a first preset number, or the number of times a user adjusts reminder settings for a certain event type less than a second preset number, etc., and are not limited thereto.

[0144] For example, the voice processing device records the user's historical response behavior information to different types of reminders. If the user ignores the rain sound reminder multiple times, the priority of the rain sound reminder can be reduced; if the user confirms the car horn reminder multiple times, the priority of the car horn reminder can be increased.

[0145] A second aspect of the present invention provides a speech processing device suitable for people with hearing impairments, such as... Figure 4 As shown, the voice processing device 10 includes at least one processor 1 and a memory 2 communicatively connected to at least one processor 1.

[0146] The memory 2 stores a computer program that can be executed by at least one processor 1. When the at least one processor 1 executes the computer program, it implements the speech processing method applicable to people with hearing impairments as described in the above embodiments.

[0147] In some embodiments, the voice processing device 10 can be a hearing aid to ensure user response speed and independence from external devices; alternatively, the voice processing device 10 can be a terminal device or a cloud device to improve voice processing accuracy. The terminal device can be a mobile phone, a smart home device, a smart speaker, or similar device.

[0148] In some embodiments, the voice processing device 10 can be a hearing aid. For example, the hearing aid may include at least one microphone, a local processor, a memory, a Bluetooth communication module, an audio output module, and a power supply module. The microphone is used to collect ambient audio signals. The local processor includes an NPU (Neural Processing Unit) or an AI acceleration chip, configured to independently perform voice detection, speaker separation, voiceprint extraction, target speaker recognition, digital voice role assignment, speech synthesis, and digital speech generation. The memory is configured to store a local timbre template library and speaker voiceprint-role mapping relationships. The Bluetooth communication module is configured to connect to a mobile terminal to receive control commands and parameter settings. The audio output module is configured to output digitized voice signals to headphones, bone conduction devices, or speakers. The power supply module supplies power to the hearing aid. By having the hearing aid independently perform all processing steps, it can assign voice roles to a preset number of speakers, ensuring fast user response and eliminating reliance on external devices. In this scenario, the mobile terminal connects to the hearing aid via Bluetooth to provide a control interface and parameter settings; when the mobile terminal is not connected to the hearing aid, the hearing aid can still independently complete all speech processing and output.

[0149] For example, user A, wearing a hearing aid with a built-in NPU chip, talks to friend B in an outdoor park. The hearing aid uses dual microphones to collect ambient sound, and the local NPU sequentially performs the following steps: VAD (Voice Activity Detection) voice detection → excluding user A's own voice → speaker recognition (identifying B as the target speaker) → ASR (Automatic Speech Recognition) → TTS (Text-to-Speech) → digital voice output. This processing has a latency of only 50ms, providing a fast response. The user's phone is in their bag and not connected; the hearing aid works completely independently. After returning home in the evening, the user can open the mobile app and adjust the tone and hearing loss curve settings via Bluetooth; these settings are saved locally on the hearing aid.

[0150] For example, user A is talking to three family members, B and C, at home. The hearing aid's NPU identifies the three speakers and assigns them three preset voice roles, such as a deep male voice, a clear female voice, and a neutral voice, all done locally with no delay. The user can distinguish between the speech of B and C by these voice roles.

[0151] Alternatively, the voice processing device 10 can be a terminal device such as a mobile terminal. For example, ambient audio signals are acquired by at least one microphone of the hearing aid, and the hearing aid's local processor is configured to perform voice detection, personal voice recognition, audio encoding, and playback control. The hearing aid's Bluetooth communication module is configured to send ambient audio signals to the mobile terminal's application and receive digitized voice signals returned by the mobile terminal. The hearing aid's audio output module is configured to output digitized voice signals to provide feedback to the user. Speaker separation, voiceprint extraction, speech recognition, and speech synthesis are performed by an application in the mobile terminal connected to the hearing aid via Bluetooth. The mobile terminal can store a rich timbre template library, allowing for the assignment of fine timbres to more speakers. Furthermore, the mobile terminal application can store the mapping relationship between the speaker's voiceprint features and voice roles in a local database, thereby ensuring that the mapping relationship remains consistent when the Bluetooth connection is reconnected after being lost.

[0152] For example, user A wears a lightweight hearing aid (i.e., one without an NPU), and their phone is in their pocket connected to the hearing aid via Bluetooth. The hearing aid collects ambient sound (ambient audio signal), performs VAD detection and excludes the user's own voice, and sends a segment of the user's voice via Bluetooth to a mobile app. The mobile app uses the phone's CPU / NPU to perform speaker recognition, ASR speech recognition, and TTS speech synthesis, returning the digitized speech signal to the hearing aid for playback via Bluetooth. This processing has a latency of approximately 120ms. Even if the user enters a subway area with no network coverage, the hearing aid will not be affected because all processing is completed locally on the phone.

[0153] For example, in a hearing aid scenario where user A is attending a small meeting with 5 people, the hearing aid collects the ambient audio signals of the multiple people and sends them to a mobile app via Bluetooth. The mobile app identifies the 5 speakers, assigns them 5 preset voice roles, processes them in parallel into digital speech signals, and feeds them back to the hearing aid in sequence. In this way, the user can identify each speaker through different voice roles.

[0154] Alternatively, the voice processing device 10 can be a cloud-based device. For example, it may include an ambient audio signal acquired by at least one microphone of the hearing aid, a local processor configured to perform voice detection, voice recognition, audio encoding, and playback control, a Bluetooth communication module configured to send the ambient audio signal to a mobile terminal application and receive a digitized voice signal returned by the mobile terminal, and an audio output module configured to output a digitized voice signal for feedback to the user. Speaker separation, voiceprint extraction, speech recognition, and speech synthesis are performed by the cloud-based device. The mobile terminal acts as a relay for the ambient audio signal; the hearing aid must cooperate with the mobile terminal, and the mobile terminal must maintain a network connection with the cloud-based device. When the network connection is interrupted, the mobile terminal uses a local lightweight model for degradation processing or suspends voice processing output until the network is restored. The cloud-based device maintains a global timbre template library and voiceprint database, supports timbre cloning, and can assign highly personalized digital voice roles to any number of speakers, thereby improving the accuracy of voice processing.

[0155] For example, User A wears a lightweight hearing aid, and their mobile phone connects to the hearing aid via Bluetooth while maintaining a 5G network connection. The hearing aid collects ambient audio signals, performs VAD detection and excludes the user's own voice, and sends a segment of the user's voice to a mobile app via Bluetooth. The mobile app uploads the voice segment to a cloud device, which performs high-precision speaker recognition → ASR speech recognition → TTS speech synthesis (including voice cloning), and returns the digitized voice signal to the mobile phone via 5G, and then forwards it to the hearing aid for playback via Bluetooth. This method helps improve the accuracy of speech processing.

[0156] For example, in a hearing aid scenario where user A attends a meeting with 20 people, the hearing aid collects ambient audio signals and sends them to the user's phone via Bluetooth. The phone then uploads the signals to a cloud device. The cloud device identifies the 20 speakers and uses a voice cloning model to clone the original voice of the key speaker, while using preset voices for the others. This cloning process makes the digital voice of the key speaker closer to their real voice. Furthermore, if the user has a long-term conversation with user B, the cloud device can gradually collect B's voice samples and optimize B's voice cloning model. When the cloning similarity reaches a certain value, such as 90%, the voice cloning model can be sent to the user's phone, allowing the phone to also use B's cloned voice.

[0157] According to embodiments of the present invention, a speech processing device suitable for people with hearing impairments can accurately identify target speakers in complex environments and assist users in accurately understanding dialogue content, thereby improving the user experience.

[0158] A third aspect of the present invention provides a speech processing system 20 suitable for people with hearing impairments, such as... Figure 5As shown, the speech processing system 20 includes at least one speech processing device 10 of the above embodiments suitable for people with hearing impairments.

[0159] The speech processing system 20 for people with hearing impairments according to embodiments of the present invention can accurately identify the target speaker in complex environments and help users accurately understand the dialogue content, thereby improving the user experience.

[0160] In some embodiments, at least one speech processing device 10 for people with hearing impairments includes two or three of the following: hearing aids, terminal devices, and cloud devices.

[0161] For example, the voice processing system 20 may include a hearing aid, which may include at least one microphone, a local processor, a storage module, a first audio channel and a second audio channel.

[0162] The microphone is used to collect ambient audio signals; the local processor is configured to run a lightweight CNN model to independently execute the speech processing method for hearing-impaired individuals described in the above embodiments; the storage module is configured to store the local environmental event classification model and user reminder preference settings. This allows the hearing aid to independently complete environmental event recognition and reminder output without relying on external devices, thus improving response speed. There are no restrictions on the type of hearing aid; it can be an in-ear hearing aid, behind-the-ear hearing aid, receiver-external hearing aid, neckband hearing aid, or eyeglass hearing aid.

[0163] Alternatively, the voice processing system 20 may include a hearing aid and a terminal device. The hearing aid may include at least one microphone, a local processor, a Bluetooth communication module, a first audio channel, and a second audio channel. The local processor is configured to run a lightweight CNN model and supports the voice processing method for people with hearing impairments described in the above embodiments. The terminal device is connected to the hearing aid via Bluetooth and supports the voice processing method for people with hearing impairments described in the above embodiments. In other words, both the hearing aid and the terminal device support the operation of the above voice processing method, thus enabling the method to switch or downgrade between different devices.

[0164] Alternatively, the voice processing system 20 may include a hearing aid, a terminal device, and a cloud device. The hearing aid may include at least one microphone, a local processor, a Bluetooth communication module, a first audio channel, and a second audio channel. The local processor is configured to run a lightweight CNN model, supporting the voice processing method for hearing-impaired individuals described in the above embodiments. The terminal device is connected to the hearing aid via Bluetooth and is used to forward communication information between the hearing aid and the cloud device. The terminal device also supports running the voice processing method for hearing-impaired individuals described in the above embodiments. The cloud device is connected to the terminal device via a network and supports running the voice processing method for hearing-impaired individuals described in the above embodiments. In other words, the hearing aid, the terminal device, and the cloud device all support the operation of the above voice processing method, thus enabling the method to switch or downgrade between different devices.

[0165] In some embodiments, when the voice processing system 20 includes multiple voice processing devices, since each voice processing device supports running the voice processing method applicable to people with hearing impairments as described in the above embodiments, the voice processing system 20 can support device switching when device switching trigger conditions, such as network interruption, are met. For example, when the network connection is interrupted while running on a cloud device, the system switches to the mobile terminal (i.e., mobile phone) processing mode, where the mobile terminal application executes the voice processing method using a local model. When the Bluetooth connection is interrupted while running on a mobile terminal, the system switches to the hearing aid independent processing mode based on the hearing aid's built-in NPU. The above switching process is transparent to the user. Thus, in the case of multi-device collaboration, the continuity of dialogue understanding is not affected, improving the system's flexibility.

[0166] For example, when operating as a cloud device, if the network is interrupted after entering the subway, the system can automatically switch from the cloud device to the processing mode of the terminal device, such as a mobile phone. After the user exits the subway and the network is restored, the terminal device automatically reverts to the cloud device's processing mode. Another example: If user A enters an elevator while operating as a cloud device and the 5G signal is interrupted, the mobile app detects the network unavailability and automatically switches to the mobile processing mode, using the phone's local lightweight model (rather than the large cloud model) to continue speaker recognition → ASR speech recognition → TTS speech synthesis. This ensures uninterrupted dialogue comprehension. After the user exits the elevator and the 5G network is restored, it automatically switches back to the cloud device's processing mode. Yet another example: If user A uses a multi-mode hearing aid with a built-in NPU, and the phone runs out of battery and shuts down, causing a Bluetooth connection loss, the hearing aid detects the Bluetooth disconnection and automatically switches to the hearing aid's independent processing mode, using the hearing aid's local NPU and lightweight model to continue all speech processing. After the phone is charged and Bluetooth is reconnected, it automatically reverts to the phone's processing mode.

[0167] It is understandable that the NPU built into the hearing aid could be configured with an independent switch to control its activation. This allows the hearing aid to operate independently when the NPU is active, and to function as a terminal / cloud device when the NPU is off, effectively reducing power consumption. Users can also control the NPU's activation via a mobile app or the hearing aid's physical buttons; there are no restrictions on this method.

[0168] In some embodiments, at least one voice processing device 10 for people with hearing impairments includes two or three of hearing aids, terminal devices, and cloud devices. For example, voice processing system 20 may include hearing aids and terminal devices, or voice processing system 20 may include hearing aids, terminal devices, and cloud devices.

[0169] When the device switching trigger condition is met, the switched voice processing device is used to reconfigure the corresponding voice role for at least one target speaker based on the timbre similarity to the voice role of the target speaker before the switch.

[0170] The device switching trigger conditions can be preset based on the actual situation. For example, they can be set based on the device's working status, such as whether it is faulty, the device's computing power status, and the device's network connection status, without any restrictions.

[0171] Specifically, taking the speech processing system 20, which includes a hearing aid, a terminal device, and a cloud device, as an example, when the device switching trigger condition is met, causing a switch from the cloud device to a terminal device such as a mobile device, if the target speaker has been assigned a cloned voice, the mobile terminal application can download the voice clone model from the cloud device, or automatically select a preset voice role with the highest similarity to the cloned voice role to replace it; when the device switching trigger condition is met, causing a switch from the terminal device to the hearing aid, the hearing aid can select a preset voice role with the highest similarity to the currently used voice role from the local voice template library. In this way, the user's perception of the target speaker's voice role is effectively maintained continuously, minimizing the confusion caused by device switching.

[0172] For example, a user first interacts with B through a cloud-based device, using B's voice as a cloned voice. When switching from the cloud device to a mobile phone, the mobile app automatically selects a preset voice that is most similar to the cloned voice. When switching from the mobile phone to the hearing aid, the hearing aid selects the most similar preset voice locally. This ensures the continuity of the user's perception of B's ​​voice.

[0173] Furthermore, when network access is available, mobile terminals or hearing aid devices can optionally upload speech segments to cloud devices for secondary, refined processing, including fine-grained environmental event classification, timbre cloning updates, and incremental model learning. It is understandable that when the aforementioned speech processing methods are executed locally on the hearing aid, all data is processed locally and not transmitted externally. Furthermore, regarding the data transmission between the hearing aid and the mobile phone, all data remains within Bluetooth range and is not uploaded to the cloud, thus effectively protecting user privacy. For data that needs to be uploaded to cloud devices, a privacy mode can be designed to protect user privacy. In this privacy mode, only the anonymized feature vector, rather than the original audio signal, needs to be uploaded. The data is deleted immediately after processing by the cloud device, thereby ensuring privacy.

[0174] In the description of this specification, any process or method described in the flowcharts or otherwise herein may be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0176] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0177] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0178] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0179] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0180] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0181] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A speech processing method suitable for people with hearing impairments, characterized in that, include: Acquire ambient audio signals around the user and identify speech segments related to human voices from the ambient audio signals; Based on the speech feature information of the speech segment, it is determined whether the speech segment corresponds to a target speaker, and the target speaker communicates with the user via voice. If so, the speech segment is analyzed to obtain language elements, which include the semantic information of the speech segment, the spectral energy distribution characteristics of each frame, and the relative duration relationship between each syllable; Based on the user's degree of hearing loss, the language elements are used to generate targeted digital speech signals.

2. The speech processing method for people with hearing impairments according to claim 1, characterized in that, Generating targeted digital speech signals from the language elements includes: In the case of multiple target speakers, a corresponding voice role is assigned to each target speaker; A digital speech signal corresponding to each target speaker is generated based on the voice role and the language elements.

3. The speech processing method for people with hearing impairments according to claim 2, characterized in that, Assigning a corresponding voice role to the at least one target speaker includes: The at least one target speaker is configured with a corresponding voice role based on a timbre template library, which is established based on at least one of voice type, pitch, speech rate and timbre. And / or, configure a corresponding voice role for the at least one target speaker based on a timbre cloning model, wherein the timbre cloning model is trained based on the speech samples corresponding to the target speaker.

4. The speech processing method for people with hearing impairments according to any one of claims 1-3, characterized in that, The degree of hearing loss of the user is determined based on the hearing aid configuration information of the hearing aid, and the method further includes: The digital speech signal is adjusted according to the hearing aid configuration information. The adjustment includes at least one of the following: frequency band enhancement, speech rate adjustment, pitch adjustment, frequency response adjustment, pause enhancement, or repetition of numerical values ​​and identifier keywords.

5. The speech processing method for people with hearing impairments according to claim 4, characterized in that, The hearing aid configuration information includes at least one of local configuration information and / or network configuration information; The hearing aid configuration information includes at least one of the user's hearing loss curve, volume, timbre selection, and hearing aid mode.

6. The speech processing method for people with hearing impairments according to claim 4, characterized in that, The hearing aid configuration information is determined based on hearing aid keywords extracted from speech segments about human voices.

7. The speech processing method for people with hearing impairments according to claim 1, characterized in that, The acquisition of ambient audio signals around the user includes: Acquire ambient audio signals within the reception range.

8. The speech processing method for people with hearing impairments according to claim 7, characterized in that, The pickup range is determined based on the hearing aid mode.

9. The speech processing method for people with hearing impairments according to claim 7, characterized in that, The sound reception range is determined based on the environmental characteristics of the user's surroundings; The environmental feature information includes at least one of the following: environmental noise intensity, sound source distance, number of target speakers, user posture, and signal-to-noise ratio.

10. The speech processing method for people with hearing impairments according to claim 1, characterized in that, The method further includes: In the event that non-human voice segments and / or segments of the user's own voice are identified from the ambient audio signal. The digital voice signal corresponding to the non-human voice segment and / or the digital voice signal corresponding to the user's own voice segment shall be prohibited from being transmitted to the user. Alternatively, reduce the volume of the digital voice signal corresponding to the non-human voice segment and / or the volume of the digital voice signal corresponding to the human voice segment.

11. The speech processing method for people with hearing impairments according to claim 1, characterized in that, The method further includes: If it is determined that there is no corresponding relationship between the voice segment and the target speaker, it is determined whether the voice segment meets the reminder conditions; If so, a reminder is given based on the voice segment, which includes at least one of phrase voice, prompt tone, spatialized audio, and vibration feedback.

12. The speech processing method for people with hearing impairments according to claim 11, characterized in that, The method is applied to a voice processing device, which includes a first audio channel and a second audio channel. The reminder is sent to the user via the first audio channel; The digitized voice signal is transmitted to the user through the second audio channel.

13. The speech processing method for people with hearing impairments according to claim 1, characterized in that, The speech feature information includes at least one of voiceprint features, directional features, semantic context features, and interaction distance features.

14. The speech processing method for people with hearing impairments according to claim 1, characterized in that, The semantic information includes one or more of textual information, semantic text, and phoneme sequences.

15. A speech processing device suitable for people with hearing impairments, characterized in that, include: At least one processor; A memory that is communicatively connected to at least one of the processors; The memory stores a computer program that can be executed by at least one of the processors, and when the at least one processor executes the computer program, it implements the speech processing method for people with hearing impairments as described in any one of claims 1-14.

16. A speech processing system suitable for people with hearing impairments, characterized in that, Includes at least one speech processing device for people with hearing impairments as described in claim 15.

17. The speech processing system for people with hearing impairments according to claim 16, characterized in that, The at least one speech processing device suitable for people with hearing impairments includes two or three of the following: hearing aids, terminal devices, and cloud devices.

Images