Sound signal conversion method, system, device and program product
By converting external sounds into tactile signals and transmitting them through wearable devices, the problem of visual dependence of traditional auxiliary hearing devices is solved, and the information acquisition ability and safety of deaf people in a variety of scenarios is improved.
Patent Information
- Application Number
- CN202510145075.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-10
- Publication Date
- 2025-05-16
AI Technical Summary
Traditional auxiliary hearing devices rely on visual cues, which have problems such as low information transmission efficiency, limited application scenarios, and inability to meet users' personalized needs.
Audio processing technology and wearable devices are used to convert external sound into specific tactile signals, and sensory signals are transmitted to users through wearable portable devices such as wristbands or smart glasses.
It improves the safety and convenience of deaf people in their daily lives, expands their ability to obtain information in complex environments, is suitable for a variety of scenarios, and provides a more comprehensive sensory compensation plan.
Smart Images
Figure CN120018042A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of hearing aid technology, and in particular to a sound-tactile signal conversion method, system, device and program product. Background Art
[0002] Due to hearing impairment, the deaf community cannot perceive the sound information in the surrounding environment, which brings many inconveniences to their daily lives. Especially in emergency situations, important sound information such as car horns and fire alarms is crucial to the safety of the deaf. The emergence of assistive hearing devices is to address the above problems and help the deaf recognize specific sound signals.
[0003] Traditional assistive hearing devices often rely on visual cues to provide feedback on sound information, such as flashing lights or screen displays. Such devices rely on visual attention, so when the line of sight is blocked or the attention is distracted, the information fed back by the device may not be received in a timely and accurate manner. At the same time, the properties of the visual feedback device itself often make it difficult to apply in mobile and sports scenes, and cannot meet the requirements of personalized settings.
[0004] Based on the above characteristics, traditional assistive hearing devices have limitations due to visual sensory characteristics, low information transmission efficiency, limited application scenarios, and inability to meet users' personalized needs, which need to be improved urgently. Summary of the invention
[0005] In view of the shortcomings of the existing technology, the present invention proposes an assisted hearing solution based on sensory feedback, which uses advanced audio processing technology and wearable devices to convert external sounds into specific sensory signals and feed them back to users, so that deaf people can identify key sound information such as car horns and alarms through the sense of the skin. This method not only improves the safety and convenience of deaf people in their daily lives, but also expands their ability to obtain information in complex environments. It is suitable for a variety of scenarios such as public safety, home life, and transportation, and provides a more comprehensive sensory compensation solution for the deaf community.
[0006] In a first aspect, the present invention provides a sound signal conversion system, comprising the following modules:
[0007] 1. Sound collection module
[0008] It includes a sound collection device, one or more highly sensitive microphones, responsible for collecting sound signals from the outside world. The microphone can be directional or omnidirectional to adapt to different application scenarios.
[0009] 2. Sound recognition and classification module
[0010] The collected sounds are analyzed in real time using a machine learning-based audio processing algorithm to identify important sound events (such as car horns, alarms, door knocks, etc.). The identified sound events are categorized according to a pre-set database, and a unique identifier is assigned to each type of sound event.
[0011] 3. Signal conversion module
[0012] Based on the results of sound recognition and classification, the signal conversion module converts each sound event into a specific sensory signal instruction. The sensory signal instruction includes but is not limited to a visual signal instruction and a tactile signal instruction.
[0013] Taking tactile signal instructions as an example, the instructions achieve different tactile styles through a combination of different vibration parameters. These parameters include but are not limited to vibration frequency, amplitude, vibration mode (such as continuous vibration, intermittent vibration) and vibration position, ensuring that different types of events can be distinguished through different tactile experiences.
[0014] 4. Signal feedback module
[0015] This module executes sensory signal instructions through a wearable portable device (such as a wristband, belt, glasses or other forms of wearable devices) to transmit sensory signals to the user.
[0016] Taking the tactile signal instruction as an example, the tactile signal is transmitted to the user according to the tactile pattern pointed to by the signal. The device generating the tactile signal includes a small vibration motor or other elements capable of generating tactile sensation.
[0017] 5. User control module
[0018] In order to facilitate user operation and settings, a user interface is provided in the user control module. Through this interface, users can start or stop the device, adjust the sensitivity of the device, select the type of sound events they want to receive, and customize certain sensory signal instructions, such as customizing certain tactile styles.
[0019] In the second aspect, the principle of the coordinated operation of the various modules of the sound signal conversion system, that is, the sound signal conversion method proposed by the present invention, includes:
[0020] After the user starts the device through the control interface, the sound collection module begins to continuously monitor the surrounding environment.
[0021] When the loudness of the detected sound exceeds the preset threshold, the sound recognition and classification module will immediately analyze and identify the sound event and determine its category.
[0022] For sound events that are identified as important, the signal conversion module will quickly convert the sound event into a corresponding sensory signal instruction and send it to the signal feedback module.
[0023] The signal feedback module executes the received sensory signal instructions, generates sensory signals, and transmits them to the user.
[0024] Through learning and memory, users can identify the sound events corresponding to different sensory signals, and thereby obtain information about the current situation and make corresponding responses.
[0025] In a third aspect, the present invention provides a set of hardware devices, including a wearable device, a sound collecting device, a sensory signal generating device, a processor and a memory, and the hardware devices work together to realize the functions of the system described in the first aspect.
[0026] In a fourth aspect, the present invention provides a computer program product, which implements the functions of the system described in the first aspect when the computer program is executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 The present invention discloses a flow chart of a sound signal conversion method. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models, simulation environments, and software described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0029] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.
[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.
[0031] Embodiment 1: Sound signal conversion device based on wristband
[0032] This embodiment provides a portable, wearable tactile feedback system, which is intended to help deaf people identify and understand important sound events in the surrounding environment.
[0033] The device includes the following modules: (“M” stands for module)
[0034] M1 sound collection module: It uses a highly sensitive omnidirectional microphone that can capture sound signals from all directions. The microphone is embedded inside the wristband to ensure that it is comfortable to wear and does not affect daily activities.
[0035] M2 sound recognition and classification module: built-in high-performance microprocessor and dedicated audio processing chip, based on the sound event detection technology SED-CRNN of convolutional recurrent neural network, real-time analysis of sound features, identification and classification of important sound events such as car horns, alarms, knocks, etc.
[0036] M3 signal conversion module: according to the sound recognition results, each type of sound event is converted into a specific tactile pattern. In this embodiment, the car horn corresponds to high-frequency continuous vibration, and the alarm corresponds to intermittent strong vibration. These patterns are managed and updated through a preset database.
[0037] M4 signal feedback module: includes multiple small vibration motors distributed in different positions of the wristband, which can generate vibrations of different frequencies, amplitudes and modes. Users can adjust the vibration intensity and position according to their personal preferences to obtain the best perception effect.
[0038] M5 User Control Module: This includes an interface on a smartphone app or wristband. Through the control interface, users can set the sensitivity of the device, select the types of sound events they want to receive, and customize the characteristics of certain haptic styles.
[0039] The workflow of the device includes: ("S" is used below to represent the workflow)
[0040] S1 Startup and initialization: The user turns on the device through the mobile phone application or the button on the wristband. After startup, the sound collection module begins to continuously monitor the surrounding environment and is ready to detect any possible important sound events.
[0041] S2 Sound Detection and Threshold Judgment: When the sound loudness exceeds the preset threshold, the sound collection module triggers the sound recognition and classification module for further analysis. The threshold can be dynamically adjusted according to the ambient noise level to reduce false alarms.
[0042] S3 Sound Recognition and Classification: The sound recognition and classification module uses a machine learning model to analyze the captured sound in real time to determine whether it belongs to a preset important sound event. If it is confirmed to be an important sound event, it will proceed to the next step; otherwise, it will continue to monitor the environment.
[0043] S4 tactile signal conversion: Once an important sound event is identified, the signal conversion module will immediately send its corresponding tactile pattern instruction to the signal feedback module. In this embodiment, the car horn event corresponds to a high-frequency continuous vibration instruction, and the fire alarm event corresponds to a strong intermittent vibration instruction.
[0044] S5 tactile feedback transmission: The signal feedback module gives the user tactile signals according to the received instructions. The user quickly recognizes these tactile signals based on previous learning and memory, and takes corresponding actions accordingly.
[0045] S6 User Interaction and Personalization: Users can adjust the device's parameters, such as sensitivity, vibration intensity, vibration position, etc., through the mobile phone application or the buttons on the wristband to meet their personal needs. In addition, users can also choose which types of sound events to pay attention to and customize the characteristics of certain tactile styles.
[0046] Embodiment 2: Multimodal feedback device based on smart glasses
[0047] The sensory feedback in this embodiment includes tactile feedback and visual feedback, which combines the two sensory channels of vision and touch to provide a richer way of conveying information.
[0048] The device includes the following modules: (“M” stands for module)
[0049] M1 sound collection module: A high-sensitivity omnidirectional microphone is integrated into the temples of smart glasses to ensure comfortable wearing without affecting the field of vision.
[0050] M2 sound recognition and classification module: Built-in high-performance microprocessor and dedicated audio processing chip, running advanced machine learning algorithms, real-time analysis of sound features, identification and classification of important sound events such as car horns, alarms, knocks, etc.
[0051] M3 signal conversion module: according to the sound recognition result, each type of sound event is converted into a specific tactile signal instruction and a visual signal instruction. In this embodiment, the car horn event is converted into a specific vibration pattern and a warning icon of a specific color.
[0052] M4 signal feedback module: includes small vibration motors distributed on the legs of the glasses, which can achieve multiple vibration modes; it also includes a micro-projection display on the lenses of the smart glasses, which can project short text prompts or symbols in the user's field of view to help users understand the current sound events more intuitively.
[0053] M5 User Interface: Users can easily set up and manage the various functions of the device through the mobile phone application or buttons on the smart glasses.
[0054] The working process of the equipment includes: ("S" stands for process)
[0055] S1 Startup and Initialization: The user turns on the device through the mobile phone application or the button on the smart glasses, and the sound collection module starts to monitor the surrounding environment.
[0056] S2 Sound Detection and Threshold Judgment: When the sound loudness exceeds the preset threshold, the sound recognition and classification module performs analysis. The threshold can be automatically adjusted according to the environment to adapt to different usage scenarios.
[0057] S3 Sound Recognition and Classification: After identifying important sound events, the system will simultaneously trigger tactile and visual signal feedback.
[0058] S4 multimodal feedback transmission: The signal feedback module generates vibrations on the glasses legs and displays corresponding prompt information on the lenses.
[0059] S5 User Interaction and Personalized Settings: Users can adjust device parameters such as sensitivity, vibration intensity, location and content of visual cues through mobile phone applications or buttons on smart glasses.
[0060] Example 3: Multimodal feedback device based on wristband and smart glasses
[0061] The sensory feedback in this embodiment includes tactile feedback and visual feedback, wherein the tactile feedback is transmitted through two devices, a wristband and smart glasses.
[0062] The device includes the following modules: (“M” stands for module)
[0063] M1 sound collection module: Integrate a high-sensitivity omnidirectional microphone on the temples or wristbands of smart glasses.
[0064] M2 sound recognition and classification module: Built-in high-performance microprocessor and dedicated audio processing chip, running advanced machine learning algorithms, real-time analysis of sound features, identification and classification of important sound events such as car horns, alarms, knocks, etc.
[0065] M3 signal conversion module: according to the sound recognition result, each type of sound event is converted into a specific tactile signal instruction and a visual signal instruction. In this embodiment, the car horn event is converted into a specific vibration pattern and a warning icon of a specific color.
[0066] M4 signal feedback module: includes small vibration motors distributed on the temples and wristbands, which can achieve multiple vibration modes; it also includes a micro-projection display on the lenses of smart glasses, which can project short text prompts or symbols in the user's field of view to help users more intuitively understand the current sound events.
[0067] M5 User Interface: Users can easily set up and manage the device’s functions through a mobile app or buttons on smart glasses or wristbands.
[0068] The working process of the equipment includes: ("S" stands for process)
[0069] S1 Startup and Initialization: The user turns on the device through the mobile phone application or the button on the smart glasses or wristband, and the sound collection module starts to monitor the surrounding environment.
[0070] S2 Sound Detection and Threshold Judgment: When the sound loudness exceeds the preset threshold, the sound recognition and classification module performs analysis. The threshold can be automatically adjusted according to the environment to adapt to different usage scenarios.
[0071] S3 Sound Recognition and Classification: After identifying important sound events, the sound recognition and classification module will simultaneously issue tactile signal instructions and visual signal instructions.
[0072] S4 multi-modal feedback transmission: The signal feedback module generates vibrations on the glasses legs and wristbands, and displays corresponding prompt information on the lenses. Users can choose which sensory channel to rely on first according to their personal habits.
[0073] S5 User Interaction and Personalized Settings: Users can adjust device parameters such as sensitivity, vibration intensity, location and content of visual cues through mobile phone applications or buttons on smart glasses or wristbands.
[0074] Compared with the prior art, the present invention has the following beneficial effects:
[0075] The present invention uses other sensory signals to provide sound information for the deaf, especially the more direct sensory channel of touch, with a highly sensitive sound collection module and a fast sound recognition algorithm, so that the deaf can quickly perceive external sound information in different scenarios, that is, the real-time nature of information feedback is improved; the accuracy and recognizability of information transmission are improved by using a variety of tactile styles; the wearable design is convenient for users to carry around, does not affect daily activities, and can be personalized through a simple user interface. The present invention is particularly suitable for early warning information feedback in emergency situations, such as traffic warnings, fire alarms, etc., which significantly improves the safety factor of the deaf in complex environments.
Claims
1. A sound signal conversion system, characterized in that: Includes the following modules: Sound collection module: including sound collection equipment; Sound recognition and classification module: used to analyze sound features and identify sound events; Signal conversion module: generating specific sensory signal instructions according to the sound event; Signal feedback module: executes sensory signal instructions and transmits sensory signals to users; User control module: allows users to customize the operating parameters of other modules in the system.
2. The system according to claim 1, characterized in that The signal feedback module is integrated in the portable wearable device to directly provide sensory signals to the user.
3. The system according to claim 1 or 2, characterized in that: The signal feedback module includes elements capable of generating sensory signals.
4. The system according to claim 1 or 2, characterized in that: The sound collection module may be integrated into the portable wearable device or may be independent of the wearable device.
5. The system according to claim 1 or 2, characterized in that: The user control module includes a mobile phone application and a button integrated on the wearable device.
6. The system according to claim 1, characterized in that The sound recognition and classification module uses an audio processing algorithm based on machine learning to recognize important sound events; and classifies the recognized sound events according to a pre-set database.
7. The system according to claim 6, characterized in that The machine learning includes the sound event detection technology SED-CRNN based on convolutional recurrent neural network.
8. The system according to claim 1, characterized in that The sensory signal instruction includes the presentation method and specific style of the sensory signal, as well as the content or details of the signal.
9. The system according to claim 8, characterized in that The sensory signal instructions include but are not limited to visual signal instructions and tactile signal instructions.
10. The system according to claim 9, characterized in that The tactile signal instruction realizes different tactile patterns through a combination of different vibration parameters, and the vibration parameters include vibration frequency, amplitude, mode, position, etc.
11. The system according to claim 9, characterized in that The visual signal instructions include displaying text prompts, displaying symbols and corresponding colors, etc.
12. The system according to claim 1, characterized in that The operating parameters of the other modules include: system switch, sound recognition sensitivity, type of sound event, and specific content of the sensory signal instruction.
13. A method for converting a sound signal using the system as claimed in claim 1, characterized in that: The following steps are involved: Use the sound collection module to monitor the surrounding environment; For sounds whose loudness exceeds a preset threshold, the sound recognition and classification module is used to identify sound events; The signal conversion module converts the recognized sound events into sensory signal instructions; The signal feedback module executes the sensory signal instruction, generates a sensory signal, and transmits it to the user.
14. The method according to claim 13, characterized in that Depending on the ambient noise level, the threshold is adjusted to reduce false alarms.
15. The method according to claim 13, characterized in that There is a corresponding relationship between the sensory signal and the sound event: Based on this correspondence, the user learns the sound event information conveyed by the sensory signal; The user can personalize the corresponding relationship; The user may personalize the content of the sensory signal.
16. A set of hardware devices, including a wearable device, a sound collecting device, a sensory signal generating device, a processor and a memory, wherein the hardware devices work together to implement the system or method described in any one of claims 1-15.
17. A computer program product, characterized in that When the computer program is executed by a processor, the system or method according to any one of claims 1 to 15 is implemented.