Alerting control method and alerting device and speech processing system suitable for people with hearing impairments

Abstract

The application discloses a reminding control method and reminding device and a speech processing system suitable for people with hearing impairment, the method is applied to a speech processing device, the speech processing device comprises a first audio channel and a second audio channel, and the method comprises the following steps: acquiring environmental audio signals around a user; identifying non-conversation environmental events and / or conversation environmental events according to the environmental audio signals; when the non-conversation environmental events meet preset reminding conditions, processing the non-conversation environmental events through the first audio channel to remind the user; and / or processing the conversation environmental events through the second audio channel to transmit to the user. The method can avoid the problem that normal communication of the user is affected due to the conflict between conversation and safety reminding, and effectively provides the user with better reminding service.

Description

Technical Field

[0001] This invention relates to the field of hearing aid technology, and in particular to a reminder control method, reminder device, and voice processing system suitable for people with hearing impairments. Background Technology

[0002] For hearing-impaired individuals, a significant safety risk in daily life is the inability to promptly perceive dangerous environmental sounds, such as car horns, alarms, and approaching vehicles. Existing products typically alert users by interrupting conversations, thus hindering normal communication. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, one objective of this invention is to provide a reminder control method suitable for people with hearing impairments. This method can avoid the problem of conflicts between dialogue and safety reminders affecting the user's normal communication, effectively providing users with a better reminder service.

[0004] The second objective of this invention is to provide a voice processing device.

[0005] The third objective of this invention is to propose a speech processing system.

[0006] To address the aforementioned problems, a first aspect of the present invention provides a reminder control method applicable to people with hearing impairments. The method is applied to a voice processing device, which includes a first audio channel and a second audio channel. The method includes: acquiring ambient audio signals surrounding the user; identifying non-dialogue environmental events and / or dialogue environmental events based on the ambient audio signals; processing the non-dialogue environmental events through the first audio channel to remind the user when the non-dialogue environmental events meet preset reminder conditions; and / or processing the dialogue environmental events through the second audio channel to transmit them to the user.

[0007] The reminder control method for people with hearing impairments according to embodiments of the present invention is based on the design of a first audio channel and a second audio channel. By separating non-dialogue environment events and dialogue environment events, the first audio channel is used to process non-dialogue environment events, and the second audio channel is used to process dialogue environment events. This achieves the effect of hierarchical management of the two, independent audio channel output, and no interference between them, so as to achieve the purpose of taking into account normal dialogue communication and safety reminders, and improve the user experience.

[0008] In some embodiments, identifying non-dialogue environment events and / or dialogue environment events based on the ambient audio signal includes: classifying sound segments in the ambient audio signal to obtain multiple types of sound information; and identifying non-dialogue environment events and / or dialogue environment events based on key features corresponding to each type of sound information, wherein the key features include at least one or more of time-domain features, frequency-domain features, semantic features, and location features.

[0009] In some embodiments, the preset alert conditions include at least one or more of the following: the event type corresponding to the non-dialogue environment event belongs to a preset high-priority event type; the sound intensity corresponding to the non-dialogue environment event is greater than or equal to a preset threshold; the event direction corresponding to the non-dialogue environment event meets preset collision conditions; and the event distance corresponding to the non-dialogue environment event is within a preset warning range.

[0010] In some embodiments, processing the non-dialogue environment event through the first audio channel includes: generating a first reminder message based on at least one of the event type, event direction, event distance, event duration, and event handling measures of the non-dialogue environment event; and transmitting the first reminder message through the first audio channel.

[0011] In some embodiments, the method further includes: acquiring the status information of the hearing aid; generating second reminder information based on the status information when the status information meets the status reminder conditions; and transmitting the second reminder information through the first audio channel.

[0012] In some embodiments, the status reminder conditions include at least one or more of the following: the hearing aid's battery level is lower than a preset battery threshold; the hearing aid is detected to be worn in an invalid state; the hearing aid's sound outlet is detected to be blocked.

[0013] In some embodiments, the method further includes: determining the reminder level corresponding to the first reminder information or the second reminder information; determining the reminder method corresponding to the reminder information based on the reminder level, wherein the reminder method includes at least one of text display, phrase voice, prompt tone, spatialized audio, and vibration feedback.

[0014] In some embodiments, the method further includes: adjusting the first reminder information or the second reminder information based on environmental feature information around the user before transmitting the first reminder information or the second reminder information, wherein the adjustment includes at least one of frequency band adjustment, volume adjustment and frequency adjustment.

[0015] In some embodiments, the environmental feature information includes at least one of time features, environmental noise intensity, and user posture. In some embodiments, the method further includes: uploading the environmental audio signal to a cloud device when the recognition accuracy of the non-dialogue environment event is lower than a preset accuracy and the voice processing device meets normal communication conditions; receiving the recognition result fed back by the cloud device, and reminding the user again based on the recognition result.

[0016] In some embodiments, the method further includes: obtaining historical response behavior information of the user when reminded; and adjusting the handling method of the non-dialogue environment event based on the historical response behavior information.

[0017] In some embodiments, adjusting the handling of non-dialogue environment events based on the historical response behavior information includes: reducing the priority of the event type corresponding to the non-dialogue environment event, or reducing the alert level corresponding to the non-dialogue environment event, when the historical response behavior information meets preset adjustment conditions.

[0018] A second aspect of the present invention provides a voice processing device, comprising: at least one processor; a memory communicatively connected to at least one processor; wherein the memory stores a computer program executable by at least one processor, and when at least one processor executes the computer program, it implements the reminder control method for people with hearing impairments described in the above embodiments.

[0019] The voice processing device according to embodiments of the present invention can avoid the problem of conflicts between dialogue and safety reminders affecting the user's normal communication, and effectively provide the user with better reminder services.

[0020] A third aspect of the present invention provides a speech processing system, including at least one speech processing device for hearing aid scenarios as described in the above embodiments.

[0021] The voice processing system according to embodiments of the present invention can avoid the problem of conflicts between dialogue and safety reminders affecting the user's normal communication, and effectively provide the user with better reminder services.

[0022] In some embodiments, the at least one voice processing device includes two or three of the following: a hearing aid, a terminal device, and a cloud device.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a speech processing method according to an embodiment of the present invention; Figure 2 This is a structural diagram of a voice processing device according to an embodiment of the present invention; Figure 3 This is a structural diagram of a speech processing system according to an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0026] The first aspect of this invention provides a reminder control method suitable for people with hearing impairments. This method can avoid the problem of conflicts between dialogue and safety reminders affecting the user's normal communication, and effectively provide the user with a better reminder service.

[0027] The following is for reference. Figure 1 This invention describes an alert control method applicable to people with hearing impairments, which is applied to a voice processing device. The voice processing device includes a first audio channel and a second audio channel. The method includes at least steps S1-S4.

[0028] Step S1: Acquire ambient audio signals around the user.

[0029] Specifically, an ambient audio signal around the user can be continuously acquired through a sound acquisition module and sent to the controller of the voice processing device. The sound acquisition module may include at least one microphone and / or a VAD (Voice Amplifier) ​​detector.

[0030] Step S2: Identify non-dialogue environment events and / or dialogue environment events based on the ambient audio signals.

[0031] Dialogue-based environmental events can be understood as environmental events involving real-time communication, such as current or remote conversations. Non-dialogue-based environmental events can be understood as environmental events involving one-way sound without requiring immediate interactive feedback. For example, non-dialogue-based environmental events can include transportation-related events, alarm-related events, natural environment-related events, and human activity-related events, without limitation. Transportation-related events can include sounds from cars, electric vehicles, motorcycles, trains, and airplanes; alarm-related events can include fire alarms, burglar alarms, car horns, and ambulance sirens; natural environment-related events can include thunder, rain, and wind; and human activity-related events can include knocking, footsteps, shouting, and the sound of breaking glass.

[0032] Step S3: When a non-dialogue environment event meets the preset reminder conditions, the non-dialogue environment event is processed through the first audio channel to remind the user.

[0033] Step S4: Process the dialogue environment events through the second audio channel to transmit them to the user.

[0034] Specifically, to avoid the problem of conflicts between dialogue and safety reminders affecting normal user communication, this application no longer uses the same audio channel to feed back non-dialogue environment events and dialogue environment events to the user. Instead, separate audio channels are established for non-dialogue environment events and dialogue environment events. Thus, by identifying non-dialogue environment events and / or dialogue environment events from the ambient audio signal, and then outputting them using the corresponding independent audio channels, the two are managed in a hierarchical manner and do not interfere with each other. Therefore, even in hearing aid scenarios where dialogue and safety reminders coexist, non-dialogue environment events can still be output through an independent reminder channel, i.e., the first audio channel, to remind the user. This process will not affect the user's normal dialogue, effectively solving the problem of conflicts between dialogue and safety reminders affecting normal user communication, achieving the goal of balancing normal dialogue and safety reminders, and improving the user experience.

[0035] It is understandable that if only non-dialogue environmental events are identified from the ambient audio signal, then only those non-dialogue environmental events need to be processed. If only dialogue environmental events are identified from the ambient audio signal, then only those dialogue environmental events need to be processed. If both non-dialogue environmental events and dialogue environmental events are identified from the ambient audio signal, then they can be processed synchronously, meaning that both non-dialogue environmental events and dialogue environmental events can be fed back to the user at the same time; or they can be processed asynchronously, meaning that they can be fed back to the user sequentially, for example, non-dialogue environmental events can be fed back to the user first to ensure the user's safety.

[0036] In some embodiments, the priority of the first audio channel may be set higher than that of the second audio channel, thereby ensuring that the user perceives the reminder information first.

[0037] In some embodiments, processing dialogue environment events includes: identifying a speech segment corresponding to the target speaker communicating with the user based on the dialogue environment event; converting the speech segment into language elements, which include text information, semantic text, semantic units, or one or more of phoneme sequences and phoneme sequences; and then generating a digital speech signal based on the language elements for transmission to the user. Thus, instead of simply amplifying the original speech segment to obtain a digital speech signal, a digital speech signal is reconstructed based on language elements, thereby improving speech clarity.

[0038] The reminder control method for people with hearing impairments according to embodiments of the present invention is based on the design of a first audio channel and a second audio channel. By separating non-dialogue environment events and dialogue environment events, the first audio channel is used to process non-dialogue environment events, and the second audio channel is used to process dialogue environment events. This achieves the effect of hierarchical management of the two, independent audio channel output, and no interference between them, so as to achieve the purpose of taking into account normal dialogue communication and safety reminders, and improve the user experience.

[0039] In some embodiments, identifying non-dialogue environment events and / or dialogue environment events based on ambient audio signals includes: classifying sound segments in the ambient audio signals to obtain multiple types of sound information. That is, considering that the ambient audio signals contain a variety of different sounds, by classifying them, audio signals with strong correlation or high similarity can be classified into the same type of sound information. Then, non-dialogue environment events and / or dialogue environment events are identified based on key features corresponding to each type of sound information. The key features include at least one or more of time-domain features, frequency-domain features, semantic features, and location features.

[0040] Among these features, temporal features can include sound volume, sound intensity, duration of sound production, and sound wave waveform; frequency domain features can include frequency, spectrum, and timbre; semantic features can include speech-text content, language type, and intonation; and location features can include orientation and distance. Based on these key features, non-dialogue environment events and dialogue environment events can be identified independently or in combination, thus achieving effective separation between the two. This allows for subsequent output of each event through an independent audio channel, achieving the goal of balancing normal conversation and safety alerts.

[0041] Preset alert conditions can be pre-set at the factory based on actual conditions, and / or can be customized by the user based on their own circumstances; there are no restrictions on this. In some embodiments, preset alert conditions include at least one or more of the following: the event type corresponding to the non-dialogue environment event belongs to a preset high-priority event type; the sound intensity corresponding to the non-dialogue environment event is greater than or equal to a preset threshold; the event direction corresponding to the non-dialogue environment event meets preset collision conditions; the event distance corresponding to the non-dialogue environment event is within a preset warning range. Therefore, based on the above preset alert conditions, hearing-impaired individuals can perceive specific sound events in the environment, especially dangerous sound events, to the greatest extent possible, which is beneficial for optimizing alert services for hearing-impaired individuals in different scenarios.

[0042] Among them, the preset threshold, preset collision conditions and preset warning range can be set based on actual conditions such as the user's hearing level, and there are no restrictions on them.

[0043] For example, preset high-priority event types may include alarm types, vehicle horn types, and siren types. In practical applications, it will be determined whether the event type corresponding to the non-dialogue environment event belongs to one of the above preset high-priority event types. If it does, the non-dialogue environment event meets the preset reminder conditions; otherwise, the non-dialogue environment event does not meet the preset reminder conditions. The above example of preset high-priority event types is only set from a security perspective. Of course, it can also be customized based on event types that users are interested in. For example, considering that users such as the elderly may experience communication, psychological, and physical problems due to hearing impairment, entertainment event types such as songs and operas can be customized. This can further stimulate users' interest in using the device, effectively enrich their life experience, and reduce their psychological and physiological burdens.

[0044] In some embodiments, processing non-dialogue environment events through a first audio channel includes: generating a first reminder message based on at least one of the event type, event direction, event distance, event duration, and event handling measures of the non-dialogue environment event; and transmitting the first reminder message through the first audio channel.

[0045] Among them, incident handling measures refer to the handling suggestions provided to users for incidents in non-dialogue environments, so that users can respond and take action in a timely manner through these suggestions.

[0046] For example, alert messages can be generated based on the event type and direction of a non-dialogue environment event. For instance, if a user is walking outdoors and the voice processing device detects the engine sound of a vehicle behind them, it generates a first alert message, "Car behind," which is played through a separate audio channel (the first audio channel). Simultaneously, the user can continue processing conversations with friends ahead without interrupting the dialogue. Alternatively, the first alert message can be generated based on the event type, direction, and distance of a non-dialogue environment event. For example, if a user is walking on a street and the voice processing device recognizes the engine sound of a car to their right rear, it generates an alert message, "Car to your right rear," which is played through a separate audio channel (the first audio channel). This allows the user to determine the direction of the approaching vehicle. Or, the first alert message can be generated based on the event type, direction, distance, and handling measures of a non-dialogue environment event. For example, if a user is at a busy intersection and the voice processing device recognizes an electric scooter rapidly approaching from about 5 meters to their right rear, and determines the safety level to be high based on real-time traffic information, it outputs an alert message, "Electric scooter rapidly approaching from your right rear, please be careful," which is played through a separate audio channel (the first audio channel). Therefore, this application provides users with reminders of different levels of precision from multiple perspectives, which helps to further enrich the reminder service.

[0047] In some embodiments, the method further includes: acquiring the status information of the hearing aid; generating second reminder information based on the status information when the status information meets the status reminder conditions; and transmitting the second reminder information through the first audio channel.

[0048] Among them, the hearing aid status information refers to the hearing aid's own condition reflected during the user's wearing and use process, such as whether it is worn correctly, whether it is malfunctioning, communication connection status, battery level, etc. In this way, when the status information meets the status reminder conditions, a second reminder message is generated based on the status information and transmitted through the first audio channel, so that the user can know the usage status of the hearing aid in a timely manner, so as to solve the problem in advance and avoid the psychological panic caused by not being able to hear.

[0049] In some embodiments, the status alert conditions include at least one or more of the following: the hearing aid's battery level is below a preset battery threshold; the hearing aid is detected as being worn ineffectively; the hearing aid's sound outlet is detected as blocked. Of course, the above are merely examples, and other status alert conditions can be set based on actual needs, provided that false alarms can be avoided; there are no restrictions on this.

[0050] The second reminder information may include a digitized voice signal indicating the status information and corresponding processing measures. This avoids the problem that users, especially the elderly, do not know how to handle abnormal hearing aid statuses. By providing processing measures, it is easier to guide users to resolve hearing difficulties in a timely manner.

[0051] In some embodiments, the method further includes: determining the reminder level corresponding to the first reminder message or the second reminder message; and determining the reminder method corresponding to the reminder message based on the reminder level, wherein the reminder method includes at least one of text display, phrase voice, prompt tone, spatialized audio, and vibration feedback.

[0052] For example, a voice processing device can be configured with multiple alert levels, each with different alert methods. For instance, three alert levels can be set: Level 1, Level 2, and Level 3. The alert methods at Level 1 include short speech phrases, vibration feedback, and spatialized audio; at Level 2, short speech phrases, prompts, and spatialized audio; and at Level 3, text display, short speech phrases, prompts, spatialized audio, and vibration feedback. Therefore, if the alert level for a given message is determined to be Level 1, then the alert methods for that message will include short speech phrases, vibration feedback, and spatialized audio. Similarly, the determination methods for other alert levels are the same as described above, and will not be illustrated further here. Thus, using multiple different alert methods can maximize user security.

[0053] For some reminder methods, other smart devices besides voice processing devices, such as smartwatches and smart glasses, can be used. For example, a user wearing a voice processing device such as a hearing aid and a smartwatch can trigger an alert from all devices simultaneously—the hearing aid and smartwatch vibrate when an emergency event, such as a car horn, is detected. When a non-emergency event, such as rain, only the hearing aid plays an alert. This multi-device collaborative reminder effect facilitates further optimization of the reminder service.

[0054] In addition, for some reminder methods, other channels besides the first and second audio channels can be used for transmission. This multi-channel setup facilitates synchronous feedback of different reminder methods. For example, the voice processing device can also include a vibration channel for transmitting reminder information through tactile feedback, a visual channel for displaying text reminders on mobile terminals or smart glasses, etc., without any restrictions.

[0055] In some embodiments, the method further includes: adjusting the first reminder information or the second reminder information based on environmental characteristics information around the user before transmitting the first reminder information or the second reminder information, the adjustment including at least one of frequency band adjustment, volume adjustment and frequency adjustment.

[0056] In some embodiments, environmental feature information includes at least one of time features, ambient noise intensity, and user posture.

[0057] Specifically, the hearing aid effect varies for users in different environments. At the same time, some users, such as the elderly, cannot effectively distinguish reminder sounds, such as being unable to distinguish between alarms and noises, or being confused by similar reminder sounds. Therefore, this application adjusts the frequency band, volume and / or frequency of the first or second reminder information based on the environmental characteristics of the user's surroundings, so as to form different prompt sounds for different situations, so that users can experience different levels of prompt effects.

[0058] For example, at night, considering that the user's environment is mostly quiet, a loud alert could cause panic. Therefore, the frequency and volume of the first or second alert can be adjusted based on time characteristics, such as using a low-volume, low-frequency alert at night; or, the frequency and volume of the first or second alert can be adjusted based on the ambient noise level, such as using a low-volume, low-frequency alert in low-noise conditions; or, when the user is in a sleeping posture, a low-volume, low-frequency alert can also be used to avoid waking the user up with the alert.

[0059] In some embodiments, the method further includes: uploading the environmental audio signal to the cloud device when the recognition accuracy of non-dialogue environment events is lower than a preset accuracy and the voice processing device meets normal communication conditions; receiving the recognition result fed back by the cloud device to remind the user again based on the recognition result.

[0060] Normal communication conditions may include the successful establishment of a network connection between the voice processing device and the cloud device.

[0061] Specifically, if the recognition accuracy of non-dialogue environment events is lower than the preset accuracy, it indicates that the current voice processing device cannot support accurate classification of non-dialogue environment events, especially for abnormal environment events. Therefore, under normal communication conditions, on the one hand, the voice processing device provides reminders based on the recognition results of non-dialogue environment events, and on the other hand, it uploads the environmental audio signal to the cloud device, which performs secondary fine classification of the environmental audio signal. Then, the cloud device feeds back the recognition results to the voice processing device, so that the voice processing device can remind the user again based on the recognition results. In this way, the reminder service can be further optimized through edge-cloud collaborative reminders.

[0062] For example, when a voice processing device such as a mobile phone is running, the mobile application software detects an abnormal sound that cannot be accurately classified. It first outputs a basic reminder "Attention to abnormal sound" and at the same time uploads the environmental audio signal, such as the audio clip of the event, to the cloud device. If the network is available, the cloud device can quickly provide feedback such as "Construction Alert" after 5 seconds. The mobile application software then adds a reminder of "Construction Alert Ahead".

[0063] In some embodiments, the method further includes: obtaining historical response behavior information of the user when reminded; and adjusting the handling of non-dialogue environment events based on the historical response behavior information.

[0064] Specifically, different users, such as the elderly and children, react differently to reminders, especially when there are safety risks. Some users react quickly, while others react slowly. Therefore, using a fixed processing method for non-dialogue environment events cannot meet personalized needs. To address this, the voice processing device of this application obtains the user's historical response behavior information when receiving reminders, such as confirmation of receipt, ignoring or adjusting reminder settings, response time, etc. Based on the historical response behavior information, it adjusts the processing method for non-dialogue environment events. This processing method can include preset reminder conditions when judging non-dialogue environment events and the reminder method after generating reminder information. In this way, by learning the user's reminder preferences and usage habits based on the user's historical response behavior information, the preset reminder conditions and reminder methods are dynamically adjusted, thereby improving the intelligence of the voice processing device and further optimizing the reminder service.

[0065] In some embodiments, adjusting the handling of non-dialogue environment events based on historical response behavior information includes: reducing the priority of the event type corresponding to the non-dialogue environment event, or reducing the reminder level of the non-dialogue environment event, when the historical response behavior information meets preset adjustment conditions. The preset adjustment conditions are pre-set based on actual needs, such as the number of times a user ignores a certain event type greater than a first preset number, or the number of times a user adjusts reminder settings for a certain event type less than a second preset number, etc., and are not limited thereto.

[0066] For example, the voice processing device records the user's historical response behavior information to different types of reminders. If the user ignores the rain sound reminder multiple times, the priority of the rain sound reminder can be reduced; if the user confirms the car horn reminder multiple times, the priority of the car horn reminder can be increased.

[0067] A second aspect of the present invention provides a voice processing device, such as... Figure 2 As shown, the voice processing device 10 includes at least one processor 1 and a memory 2 communicatively connected to at least one processor 1.

[0068] The memory 2 stores a computer program that can be executed by at least one processor 1. When the at least one processor 1 executes the computer program, it implements the reminder control method applicable to people with hearing impairments as described in the above embodiment.

[0069] In some embodiments, the voice processing device 10 can be a hearing aid to ensure user response speed and independence from external devices; or, the voice processing device 10 can be a terminal device or a cloud device to improve voice processing accuracy.

[0070] The voice processing device 10 according to the present invention can avoid the problem of the user's normal communication being affected by the conflict between dialogue and safety reminders, and effectively provide the user with better reminder services.

[0071] A third aspect of the present invention provides a voice processing system 20, such as... Figure 3 As shown, the voice processing system 20 includes at least one voice processing device 10 as described in the above embodiments.

[0072] The voice processing system 20 according to an embodiment of the present invention can avoid the problem of the user's normal communication being affected by the conflict between dialogue and safety reminders, and effectively provide the user with better reminder services.

[0073] In some embodiments, at least one voice processing device includes two or three of the following: a hearing aid, a terminal device, and a cloud device.

[0074] For example, the voice processing system 20 may include a hearing aid, which may include at least one microphone, a local processor, a storage module, a first audio channel and a second audio channel.

[0075] The system includes a microphone for acquiring ambient audio signals; a local processor configured to run a lightweight CNN model for independently executing the alert control method for hearing-impaired individuals described in the above embodiments; and a storage module configured to store a local environmental event classification model and user alert preference settings. This allows the hearing aid to independently complete environmental event recognition and alert output without relying on external devices, thus improving response speed. The type of hearing aid is not limited; it can be an in-ear hearing aid, behind-the-ear hearing aid, receiver-external hearing aid, neckband hearing aid, or eyeglass hearing aid.

[0076] Alternatively, the voice processing system 20 may include a hearing aid and a terminal device. The hearing aid may include at least one microphone, a local processor, a Bluetooth communication module, a first audio channel, and a second audio channel. The local processor is configured to run a lightweight CNN model and supports running the reminder control method for people with hearing impairments described in the above embodiments. The terminal device is connected to the hearing aid via Bluetooth and supports running the reminder control method for people with hearing impairments described in the above embodiments. In other words, both the hearing aid and the terminal device support the operation of the above-described voice processing method.

[0077] Alternatively, the voice processing system 20 may include a hearing aid, a terminal device, and a cloud device. The hearing aid may include at least one microphone, a local processor, a Bluetooth communication module, a first audio channel, and a second audio channel. The local processor is configured to run a lightweight CNN model, supporting the execution of the reminder control method for hearing-impaired individuals described in the above embodiments. The terminal device is connected to the hearing aid via Bluetooth and is used to forward communication information between the hearing aid and the cloud device. The terminal device also supports the execution of the reminder control method for hearing-impaired individuals described in the above embodiments. The cloud device is connected to the terminal device via a network and supports the execution of the reminder control method for hearing-impaired individuals described in the above embodiments. In other words, the hearing aid, the terminal device, and the cloud device all support the execution of the above-described voice processing method.

[0078] In some embodiments, when the voice processing system 20 includes multiple voice processing devices, since each voice processing device supports running the reminder control method for people with hearing impairments as described above, the voice processing system 20 can support switching devices when the device switching trigger condition, such as network interruption, is met, thereby improving the system's flexibility in the case of multiple devices working together.

[0079] For example, when operating as a cloud-based device, if the network is interrupted after entering the subway, the system can automatically switch from the cloud-based device to the processing mode of the terminal device, such as a mobile phone. Once the user exits the subway and the network is restored, the terminal device will automatically revert to the cloud-based device processing mode.

[0080] In the description of this specification, any process or method described in the flowcharts or otherwise herein may be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0082] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0083] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0084] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0085] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0086] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0087] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A reminder control method suitable for people with hearing impairments, characterized in that, The method is applied to a voice processing device, the voice processing device including a first audio channel and a second audio channel, the method comprising: Acquire ambient audio signals from the user's surroundings, wherein the ambient audio signals are provided by a sound acquisition module on the user's hearing aid; Based on the ambient audio signals, non-dialogue environment events and / or dialogue environment events are identified; When the non-dialogue environment event meets the preset reminder conditions, the non-dialogue environment event is processed through the first audio channel to remind the user; And / or, process the dialogue environment events through the second audio channel for transmission to the user.

2. The reminder control method for people with hearing impairments according to claim 1, characterized in that, Identifying non-dialogue environment events and / or dialogue environment events based on the ambient audio signals includes: The sound segments in the environmental audio signal are classified to obtain multiple types of sound information; Based on the key features corresponding to each type of sound information, non-dialogue environment events and / or dialogue environment events are identified, and the key features include at least one or more of the following: time domain features, frequency domain features, semantic features, and location features.

3. The reminder control method for people with hearing impairments according to claim 1, characterized in that, The preset reminder conditions include at least one or more of the following: The event type corresponding to the non-dialogue environment event belongs to a preset high-priority event type; The sound intensity corresponding to the non-dialogue environment event is greater than or equal to a preset threshold. The event direction corresponding to the non-dialogue environment event satisfies the preset collision condition; The distance to the non-dialogue environment event is within the preset warning range.

4. The reminder control method for people with hearing impairments according to claim 1, characterized in that, Processing the non-dialogue environment events through the first audio channel includes: A first reminder message is generated based on at least one of the event type, event direction, event distance, event duration, and event handling measures of the non-dialogue environment event; The first reminder information is transmitted through the first audio channel.

5. The reminder control method for people with hearing impairments according to claim 4, characterized in that, The method further includes: Obtain the status information of the hearing aid; When the status information meets the status reminder conditions, a second reminder message is generated based on the status information; The second reminder information is transmitted through the first audio channel.

6. The reminder control method for people with hearing impairments according to claim 5, characterized in that, The status alert conditions include at least one or more of the following: The hearing aid's battery level is below a preset battery threshold; The hearing aid was detected to be worn in an invalid state. The sound outlet of the hearing aid was detected to be blocked.

7. The reminder control method for people with hearing impairments according to claim 5, characterized in that, The method further includes: Determine the alert level corresponding to the first or second alert message; The reminder method corresponding to the reminder information is determined according to the reminder level, and the reminder method includes at least one of text display, short voice, prompt tone, spatialized audio and vibration feedback.

8. The reminder control method for people with hearing impairments according to claim 5, characterized in that, The method further includes: Before transmitting the first reminder information or the second reminder information, the first reminder information or the second reminder information is adjusted based on the environmental characteristics information around the user. The adjustment includes at least one of frequency band adjustment, volume adjustment and frequency adjustment.

9. The reminder control method for people with hearing impairments according to claim 8, characterized in that, The environmental feature information includes at least one of time features, environmental noise intensity, and user posture.

10. The reminder control method for people with hearing impairments according to claim 1, characterized in that, The method further includes: When the recognition accuracy of the non-dialogue environment event is lower than the preset accuracy and the voice processing device meets the normal communication conditions, the environmental audio signal is uploaded to the cloud device. The system receives the identification results from the cloud device and uses these results to remind the user again.

11. The reminder control method for people with hearing impairments according to claim 1, characterized in that, The method further includes: Obtain the user's historical response behavior information when reminded; Adjust the handling of non-dialogue environment events based on the historical response behavior information.

12. The reminder control method for people with hearing impairments according to claim 11, characterized in that, Adjusting the handling of non-dialogue environment events based on the historical response behavior information includes: If the historical response behavior information meets the preset adjustment conditions, reduce the priority of the event type corresponding to the non-dialogue environment event, or reduce the reminder level corresponding to the non-dialogue environment event.

13. A voice processing device, characterized in that, include: At least one processor; A memory that is communicatively connected to at least one of the processors; The memory stores a computer program that can be executed by at least one of the processors, and when the at least one processor executes the computer program, it implements the reminder control method for people with hearing impairments as described in any one of claims 1-12.

14. A voice processing system, characterized in that, Includes at least one of the voice processing devices as described in claim 13.

15. The speech processing system according to claim 14, characterized in that, The at least one voice processing device includes two or three of the following: hearing aids, terminal devices, and cloud devices.

Images