Speech enhancement method and device based on double sensors
Through the adaptive adjustment of voice enhancement method, the frequency and decibel values of air-guided voice and non-air-guided voice are used to determine the noise level, which solves the problem of poor voice enhancement effect in complex acoustic scenarios, and improves the accuracy of speech recognition and user experience.
Patent Information
- Application Number
- CN202310647205.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to adaptively adjust the speech enhancement intensity in complex acoustic scenarios, resulting in a large impact on noise, reducing the accuracy of speech recognition, and possibly overstimulating the user in simple scenarios.
By obtaining air-guided voice and non-air-guided voice, counting their frequency and decibel values, judging the noise level, and adjusting the decibel values of mapped voice and air-guided voice according to the noise level, and performing weighted fusion to obtain enhanced voice. The calculation formula is d=a *b*n^n/(n-1)^(n-1).
Improved voice enhancement effect in high-noise environments, making it easier for users to recognize voice, reduce overstimulation in low-noise environments, and improves user experience.
Smart Images

Figure CN119068897A_ABST
Abstract
Description
Technical Field
[0001] The patent of this invention relates to the field of speech enhancement technology, and specifically to a speech enhancement method and device based on dual sensors. Background Art
[0002] Speech enhancement involves a wide range of application areas, including voice calls, telephone conferences, scene recordings, hearing aids and speech recognition equipment, and has become a preprocessing module for many speech coding and recognition systems. In the voice assistant of a mobile phone, speech recognition is performed in a close distance (generally less than 1 meter) and noise-free environment, with a high speech recognition accuracy. However, if the acoustic scene becomes more complex, such as in exhibitions, streets and other scenes, the impact of noise will greatly reduce their speech recognition accuracy. Therefore, front-end noise reduction for speech recognition is very important. In addition, speech enhancement technology is also used in some hearing aid devices. Ordinary hearing aids only achieve basic amplification of a speech. More complex ones will perform sound pressure level compression to compensate for the patient's hearing range. However, if the auditory scene is more complex, the speech heard by the patient not only contains the amplified speech but also contains a lot of noise. Over time, it is bound to cause secondary damage to the patient's auditory system. Therefore, in high-end digital hearing aid devices, speech noise reduction has also become an important aspect that cannot be ignored. Chinese patent publication number CN105632512B discloses a dual-sensor speech enhancement method and device based on a statistical model, which adaptively weights and fuses the filtered speech with the mapped speech of the non-air-conducted speech, and can achieve a good speech enhancement effect. However, in the above technology, the relationship between the mapped speech of the non-air-conducted speech and the air-conducted speech is a constant. When the acoustic scene becomes more complex, it will increase the difficulty of user speech recognition. When the acoustic scene becomes simpler, it may also over-stimulate the user. For this reason, the present invention provides a dual-sensor based speech enhancement method for solving the above problems.
[0003] Invention patent content In view of the defects in the prior art, the present invention provides a speech enhancement method based on dual sensors to achieve adaptive adjustment of speech enhancement intensity.
[0004] According to a first aspect of the embodiments of the present disclosure, a preferred embodiment of the present invention provides a dual-sensor based speech enhancement method for a hearing aid terminal, comprising: Acquire air-conducted speech and non-air-conducted speech, and convert the non-air-conducted speech into air-conducted mapped speech; Collecting statistics on the frequency of the air-conducted speech and the decibel value of each frequency, and determining the noise level according to the statistical results; and The mapped speech and the air-conducted speech are adjusted according to the noise level, and the air-conducted speech is converted into enhanced speech.
[0005] In one embodiment, the frequency of the air-conducted speech and the decibel value of each frequency are counted, and the noise level is determined based on the statistical results, including: Performing segmented filtering on the air-conducted speech to obtain the number of frequencies of the air-conducted speech; Eliminate the air conduction speech frequencies whose decibel values are less than a preset value to obtain the number of effective air conduction speech frequencies that meet the conditions; The number of effective frequencies of the air-conducted speech is counted to determine the noise level thereof.
[0006] In one embodiment, adjusting the mapped speech and the air-conducted speech according to the noise level to obtain enhanced speech includes: The decibel value of the mapped speech is adjusted according to the noise level, and the calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; The adjusted mapped speech and air-conducted speech are weightedly fused to obtain fused speech; The fused speech is filtered to obtain the enhanced speech.
[0007] According to the second aspect of the embodiment of the present disclosure, the present invention provides a dual-sensor based speech enhancement device for a hearing aid terminal, comprising: An acquisition module, used for acquiring air-conducted speech and non-air-conducted speech, and converting the non-air-conducted speech into air-conducted mapping speech; A judgment module, used to collect statistics on the frequency of the air-conducted speech and the decibel value of each frequency, and judge the noise level according to the statistical results; and The integration module is used to adjust the mapped speech and the air-conducted speech according to the noise level, and convert the air-conducted speech into enhanced speech.
[0008] In one embodiment, the determination module includes: A resolution module, used for performing segmented filtering on the air-conducted speech to obtain the number of frequencies of the air-conducted speech; A screening module, used to remove the air conduction speech frequencies whose decibels are less than a preset value, so as to obtain the number of effective air conduction speech frequencies that meet the conditions; The determination module is used to count the number of effective frequencies of the air-conducted speech to determine the noise level thereof.
[0009] In one embodiment, the integration module includes: The calculation module is used to adjust the decibel value of the mapped speech according to the noise level, and the calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; A fusion module, used for weighted fusion of the adjusted mapped speech and air-conducted speech to obtain fused speech; The output module is used to filter the fused speech to obtain enhanced speech.
[0010] According to the third aspect of the embodiment of the present disclosure, the present invention provides a speech enhancement device based on dual sensors, comprising: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the steps of the above method.
[0011] According to the fourth aspect of the embodiments of the present disclosure, the present invention provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the above method.
[0012] It can be seen from the above technical solution that a dual-sensor speech enhancement method provided by the patent of the present invention may include the following beneficial effects: the present invention can obtain an enhanced speech effect by converting non-air-conducted speech into air-conducted mapped speech, and weightedly fusing the air-conducted mapped speech and the air-conducted speech. By judging the current noise level, the decibel value of the air-conducted mapped speech can be adjusted in advance. When the noise level is high, the enhanced speech effect can be improved, making it easy for users to distinguish the enhanced speech. When the noise level is low, the enhanced speech effect can be reduced, which can avoid excessive stimulation of the user and improve the user experience.
[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the specific implementation of the present invention, the following is a brief introduction to the drawings required for the specific implementation or prior art description. In all the drawings, the components or parts are not necessarily drawn according to the actual scale.
[0015] Figure 1 A flow chart of a dual-sensor based speech enhancement method provided by the present invention; Figure 2 A flowchart of step S2 in a dual-sensor based speech enhancement method provided by the present invention; Figure 3 A flowchart of step S3 in a dual-sensor based speech enhancement method provided by the present invention; Figure 4 A block diagram of a dual-sensor based speech enhancement device provided by the present invention; Figure 5 A block diagram of another dual-sensor based speech enhancement device provided by the patent of this invention. Implementation
[0016] The following is a detailed description of the embodiments of the technical solution of the present invention in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present invention.
[0017] Figure 1 The flowchart of a dual-sensor voice enhancement method provided by the present invention is applied to a hearing aid terminal, which can display pictures, videos, text messages, WeChat and other information. The terminal can be equipped with any terminal device with a display screen, such as a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc. This embodiment provides a dual-sensor voice enhancement method, such as Figure 1 As shown, the method, used in a hearing aid terminal, comprises the following steps S100-S300: In step S100, air-conducted speech and non-air-conducted speech are acquired, and the non-air-conducted speech is converted into air-conducted mapped speech; In this implementation, by establishing a joint statistical model for classifying non-air-conducted speech and air-conducted speech, the non-air-conducted speech can be converted into air-conducted mapped speech, and the air-conducted mapped speech has the same characteristics as the air-conducted speech but different intensities.
[0018] In step S200, the frequency of the air-conducted speech and the decibel value of each frequency are counted, and the noise level is determined according to the statistical results; and In step S300, the mapped speech and the air-conducted speech are adjusted according to the noise level, and the air-conducted speech is converted into enhanced speech; In this implementation, by judging the current noise level, the decibel value of the air conduction mapping voice can be adjusted in advance. When the noise level is high, the enhanced voice effect can be improved, making it easier for users to distinguish the enhanced voice. When the noise level is low, the enhanced voice effect can be reduced to avoid excessive stimulation of the user, thereby improving the user experience.
[0019] Among them, Figure 2As shown, in step S200, the frequency of the air-conducted speech and the decibel value of each frequency are counted, and the noise level is determined according to the statistical results, including the following steps S201-S203: In step S201, the air-conducted speech is subjected to segmented filtering to obtain the number of frequencies of the air-conducted speech; In this implementation, continuous signals are converted into discrete signals, which are convenient for storage and easy for computer processing. Analog signals and digital signals can be converted into each other. Continuous analog signals can be sampled, quantized, and encoded to obtain digital signals, which is convenient for analyzing and counting air-conducted speech.
[0020] In step S202, the air conduction speech frequencies whose decibel values are less than a preset value are eliminated to obtain the number of effective air conduction speech frequencies that meet the conditions; In step S203, the number of effective frequencies of the air-conducted speech is counted to determine the noise level thereof.
[0021] In this implementation, after the voices whose decibel values meet the interference requirement are counted, the noise level can be divided into binary levels. The more voices whose decibel values meet the interference requirement, the higher the noise level.
[0022] In one embodiment, if Figure 3 As shown, in step S300, mapping speech and air-conducted speech are adjusted according to the noise level to obtain enhanced speech, including the following steps S301-S303: In step S301, the decibel value of the mapped speech is adjusted according to the noise level, and the calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; In this implementation, the higher the noise level, the decibel value of air-conducted speech, and the constant value, the greater the decibel value of the mapped speech. Among them, air-conducted speech is directly affected by the ambient volume, and the constant value is affected by the user's hearing damage value. The higher the hearing damage value, the larger the constant value. The noise level is affected by the amount of noise in the current environment. The more noise there is, the larger the value of n^n / (n-1)^(n-1).
[0023] In step S302, the adjusted mapped speech and air-conducted speech are weightedly fused to obtain fused speech; In step S303, the fused speech is filtered to obtain enhanced speech; In this implementation, the mapped voice and the air-conducted voice are played synchronously without generating additional noise, and the non-air-conducted voice contained in the air-conducted voice is enhanced so that this part of the voice is deliberately amplified, and the sound of a specific frequency is filtered out, which is beneficial for users to distinguish the enhanced voice.
[0024] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.
[0025] Figure 4 This is a block diagram of a dual-sensor voice enhancement device provided by the present invention. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the device is used in a hearing aid terminal and comprises: An acquisition module 10 is used to acquire air-conducted speech and non-air-conducted speech, and convert the non-air-conducted speech into air-conducted mapping speech; A judgment module 20 is used to collect statistics on the frequency of the air-conducted speech and the decibel value of each frequency, and judge the noise level according to the statistical results; and The integration module 30 is used to adjust the mapped speech and the air-conducted speech according to the noise level, and convert the air-conducted speech into enhanced speech.
[0026] The present disclosure can obtain an enhanced speech effect by converting non-air-conducted speech into air-conducted mapped speech and weightedly fusing the air-conducted mapped speech and the air-conducted speech. By judging the current noise level, the decibel value of the air-conducted mapped speech can be adjusted in advance. When the noise level is high, the enhanced speech effect can be improved, making it easier for users to distinguish the enhanced speech. When the noise level is low, the enhanced speech effect can be reduced, avoiding excessive stimulation of users and improving the user experience.
[0027] In one embodiment, if Figure 4 As shown, the judging module 20 includes: A resolution module 21, configured to perform segmented filtering on the air-conducted speech to obtain the number of frequencies of the air-conducted speech; A screening module 22, used to remove the air conduction speech frequencies whose decibels are less than a preset value, so as to obtain the number of effective air conduction speech frequencies that meet the conditions; A determination module 23 is used to count the number of effective frequencies of the air-conducted speech to determine the noise level thereof; In one embodiment, if Figure 4 As shown, the integration module 30 includes: The calculation module 31 is used to adjust the decibel value of the mapped speech according to the noise level, and its calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; A fusion module 32, configured to weightedly fuse the adjusted mapped speech and the air-conducted speech to obtain a fused speech; An output module 33 is used to filter the fused speech to obtain enhanced speech; Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0028] The present disclosure also provides another dual-sensor based speech enhancement device: Figure 5 8 is a block diagram of a dual-sensor based speech enhancement device 800 according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0029] Reference Figure 5 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0030] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0031] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0032] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0033] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0034] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0035] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0036] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0037] The communication component 816 is configured to facilitate wired or wireless communication between the apparatus 800 and other devices. The apparatus 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof.
[0038] In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0039] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0040] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0041] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0042] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A dual-sensor based speech enhancement method for hearing aid terminals, It is characterized in that include: Acquire air-conducted speech and non-air-conducted speech, and convert the non-air-conducted speech into air-conducted mapped speech; Collecting statistics on the frequency of the air-conducted speech and the decibel value of each frequency, and determining the noise level according to the statistical results; and The mapped speech and the air-conducted speech are adjusted according to the noise level, and the air-conducted speech is converted into enhanced speech.
2. The method according to claim 1, It is characterized in that The frequency of the air-conducted speech and the decibel value of each frequency are counted, and the noise level is determined based on the statistical results, including: Performing segmented filtering on the air-conducted speech to obtain the number of frequencies of the air-conducted speech; Eliminate the air conduction speech frequencies whose decibel values are less than a preset value to obtain the number of effective air conduction speech frequencies that meet the conditions; The number of effective frequencies of the air-conducted speech is counted to determine the noise level thereof.
3. The method according to claim 1, It is characterized in that Adjusting the mapped speech and the air-conducted speech according to the noise level to obtain enhanced speech includes: The decibel value of the mapped speech is adjusted according to the noise level, and the calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; The adjusted mapped speech and air-conducted speech are weightedly fused to obtain fused speech; The fused speech is filtered to obtain the enhanced speech.
4. A speech enhancement device based on dual sensors, It is characterized in that Used in a hearing aid terminal, characterized by comprising: An acquisition module, used for acquiring air-conducted speech and non-air-conducted speech, and converting the non-air-conducted speech into air-conducted mapping speech; A judgment module, used to collect statistics on the frequency of the air-conducted speech and the decibel value of each frequency, and judge the noise level according to the statistical results; and The integration module is used to adjust the mapped speech and the air-conducted speech according to the noise level, and convert the air-conducted speech into enhanced speech.
5. The device according to claim 4, It is characterized in that The judging module comprises: A resolution module, used for performing segmented filtering on the air-conducted speech to obtain the number of frequencies of the air-conducted speech; A screening module, used to remove the air conduction speech frequencies whose decibels are less than a preset value, so as to obtain the number of effective air conduction speech frequencies that meet the conditions; The determination module is used to count the number of effective frequencies of the air-conducted speech to determine the noise level thereof.
6. The device according to claim 4, It is characterized in that The integration module comprises: The calculation module is used to adjust the decibel value of the mapped speech according to the noise level, and the calculation formula is: d=a*b*n^n / (n-1)^(n-1) Where a is the decibel value of air-conducted speech, b is a constant, n is the noise level, and d is the decibel value of the mapped speech; A fusion module, used for weighted fusion of the adjusted mapped speech and air-conducted speech to obtain fused speech; The output module is used to filter the fused speech to obtain enhanced speech.
7. A speech enhancement device based on dual sensors, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the steps of any one of the methods in claims 1 to 3.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of any one of the methods of claims 1 to 3 are implemented.
