Integrated cochlear implant sound processing method based on dual-mode signal input

Abstract

The application discloses a sound processing method of an integrated artificial cochlea based on a bimodal signal input, and the method comprises the following steps: synchronously acquiring two-way microphone time domain signals and one-way bone conduction time domain signals, and performing time-frequency conversion; generating a speech activity flag and an impact flag according to the bone conduction signals; performing adaptive beamforming on the microphone frequency domain signals based on a preset steering vector and a noise covariance matrix, and updating the noise covariance matrix when there is no speech and no impact; controlling the bone conduction compensation filter to be updated or frozen according to the speech activity flag and the impact flag, and compensating the amplitude of the bone conduction frequency domain signals; synthesizing the compensated bone conduction amplitude and the phase of the beamformed frequency domain signals to obtain an enhanced frequency domain signal, and performing inverse time-frequency conversion and overlap addition to obtain an output time domain signal. The application can reduce the interference of speech and impact on parameter updating, and improve the spontaneous sound enhancement effect and output stability.

Description

Technical Field

[0001] This application relates to the field of sound processing technology, specifically to an integrated cochlear implant sound processing method based on dual-modal signal input. Background Technology

[0002] Cochlear implants typically consist of an external sound processor and an implant. The external sound processor collects sound through a microphone, performs noise reduction, enhancement, and encoding on the sound signal, and sends the processing results to the implant via a radio frequency communication link. To improve sound pickup capabilities in complex environments, some sound processors are equipped with two or more microphones, utilizing the amplitude and phase differences of the signals received by different microphones to perform beamforming, thereby enhancing sound from the target direction and suppressing noise from other directions.

[0003] However, microphones capture airborne sound. When the wearer speaks, the captured speech is easily affected by environmental noise, reflected sound, wind noise, and changes in the device's position. It's difficult to reliably distinguish between speech and non-speech periods based solely on microphone signals. Furthermore, when speech or sudden interference is incorrectly used for noise statistics updates, it can cause shifts in beamforming parameters, affecting the preservation of the target speech. Bone conduction vibration sensors can capture vibration signals transmitted through the skull when the wearer speaks, providing auxiliary information for spontaneous speech detection. However, bone conduction signals and microphone signals differ in spectral amplitude, propagation delay, and phase characteristics. Direct superposition without proper compensation can easily cause spectral fluctuations and sound distortion.

[0004] Furthermore, integrated sound processors are typically attached magnetically to the scalp behind the ear. Mechanical vibrations from touch, friction, or button presses can simultaneously affect the microphone and bone conduction vibration sensor. Existing output limiting or compression methods usually only process signals after their amplitude has increased, making it difficult to promptly prevent sudden impacts from interfering with noise estimation, adaptive filtering, and the output signal. Therefore, it is necessary to improve the dual-modal signal processing of integrated cochlear implant sound processors to enhance the stability of speech enhancement, parameter updates, and transient interference suppression. Summary of the Invention

[0005] This application provides an integrated cochlear implant sound processing method based on dual-modal signal input, which at least solves some of the technical problems existing in the related technologies described above.

[0006] According to a first aspect of the embodiments of this application, an integrated cochlear implant sound processing method based on dual-modal signal input is provided, comprising: Simultaneously acquire two microphone time-domain signals and one bone conduction time-domain signal, and perform time-frequency transformation on the three time-domain signals using the same parameters to obtain the microphone frequency domain signal and the bone conduction frequency domain signal; Speech activity markers are generated based on bone conduction frequency domain signals, and impact markers are generated based on bone conduction time domain signals. Adaptive beamforming is performed on the microphone frequency domain signal based on the preset steering vector and noise covariance matrix to obtain the beamforming frequency domain signal. The noise covariance matrix is ​​updated only in the frames where the voice activity flag indicates a non-voice frame and the impact flag does not indicate an impact. The bone conduction compensation filter is updated within the speech frame and frozen during the impact according to the speech activity flag and the impact flag. The bone conduction compensation filter is updated according to the beamforming frequency domain signal and the bone conduction frequency domain signal. The bone conduction compensation filter is used to perform amplitude compensation on the bone conduction frequency domain signal to obtain the compensated bone conduction amplitude. The phase synthesis of the compensated bone conduction amplitude and the beamforming frequency domain signal is used to obtain the enhanced frequency domain signal. The enhanced frequency domain signal is then subjected to inverse time-frequency transformation and superposition to obtain the output time domain signal.

[0007] As an optional approach, the generation of the voice activity marker includes: summarizing the energy of the bone conduction frequency domain signal at each frequency point to obtain the bone conduction frame energy; comparing the bone conduction frame energy with a preset energy threshold, marking frames that reach the preset energy threshold as voice frames, marking frames that do not reach the preset energy threshold as non-voice frames, and applying a minimum duration limit to the marking results.

[0008] As an optional approach, the preset guidance vector is pre-calculated based on the distance between the two omnidirectional microphones, the speed of sound propagation, the direction of the target directly in front, and the physical frequency corresponding to each frequency point; the two omnidirectional microphones are arranged on opposite sides along the axis of the integrated sound processor.

[0009] As an optional approach, the adaptive beamforming is a minimum variance distortionless response beamforming, comprising: constructing receive vectors for the frequency domain signals of the two microphones on a frequency-by-frequency basis; recursively updating the noise covariance matrix based on the receive vectors of non-speech frames, and diagonally loading the noise covariance matrix; under the constraint of the preset steering vector, determining beamforming weights based on the diagonally loaded noise covariance matrix, and weighting the receive vectors using the beamforming weights to obtain the beamforming frequency domain signal.

[0010] As an optional approach, the update period of the bone conduction compensation filter is the frame in which the voice activity flag indicates a voice frame, the signal-to-noise ratio of the beamforming frequency domain signal reaches a preset signal-to-noise ratio threshold, and the impact flag does not indicate an impact. During the update period, the spectral mapping is estimated based on the amplitude correspondence between the beamforming frequency domain signal and the bone conduction frequency domain signal at each frequency point, and the bone conduction compensation filter is updated using a time-recursive smoothing method.

[0011] As an alternative, the initial state of the bone conduction compensation filter is either all-pass or a preset spectral mapping based on the average skull conduction to air conduction characteristics measured offline.

[0012] As an optional solution, the amplitude compensation includes: correcting the amplitude of the bone conduction frequency domain signal at each frequency point through the bone conduction compensation filter, and applying a preset amplitude lower limit to the corrected amplitude to obtain the compensated bone conduction amplitude; extracting the phase of the beamforming frequency domain signal at each frequency point, and synthesizing it with the compensated bone conduction amplitude at the corresponding frequency point.

[0013] As an optional approach, the method further includes: after obtaining the output time-domain signal, performing dual-path dynamic range control on the output time-domain signal; generating a feedforward gain based on the bone conduction time-domain signal, generating a feedback gain based on the output time-domain signal, performing minimum value selection on the feedforward gain and the feedback gain to obtain a control gain, and applying the control gain to the output time-domain signal to obtain the final output signal.

[0014] As an optional approach, the generation of the feedforward gain includes: performing high-pass filtering on the bone conduction time-domain signal, calculating the Teager energy envelope based on the high-pass filtering result, and smoothing the Teager energy envelope; generating the impact flag based on whether the smoothed Teager energy envelope simultaneously reaches the impact detection threshold and the rising edge threshold; determining the feedforward attenuation based on the level of the smoothed Teager energy envelope relative to the feedforward compression threshold and the feedforward compression ratio, and recursively generating the feedforward gain according to the attack time constant and the release time constant.

[0015] As an optional approach, the generation of the feedback gain includes: converting the output time-domain signal into a decibel level and smoothing it; comparing the smoothed decibel level with a feedback compression threshold; determining the feedback attenuation based on the feedback compression ratio; and recursively generating the feedback gain according to the attack time constant and the release time constant; both the feedforward gain and the feedback gain are decibel gains, and the one with the smallest value is selected and converted into a linear control gain.

[0016] This application simultaneously acquires time-domain signals from two microphones and one bone conduction time-domain signal. It generates speech activity markers by utilizing the correlation between the bone conduction signal and the wearer's spontaneous voice, and identifies impact interference such as shell contact by combining the transient energy changes. This ensures that the noise covariance matrix is ​​updated only during non-speech and impact-free periods, and the bone conduction compensation filter is updated only during periods when speech quality meets requirements and there is no impact. This reduces the contamination of adaptive parameter estimation by target speech and mechanical impact, improving the stability of beamforming and spectral compensation. By learning the amplitude correspondence between the bone conduction spectrum and the beamforming spectrum, adaptive amplitude compensation is performed on the bone conduction signal. The compensated amplitude is then synthesized with the phase of the beamforming signal, avoiding phase cancellation and spectral fluctuations caused by the direct superposition of the air conduction path and the bone conduction path. This improves the clarity of the wearer's spontaneous voice while maintaining the timing information of the target sound. Furthermore, the bone conduction feedforward gain and output feedback gain are used to jointly control the final output signal. This can establish attenuation before the sudden impact reaches the output of the main processing path and supplement and limit the abnormal peak after processing, thereby reducing the impact of transient overload on auditory perception and implant stimulation output, and improving the reliability of the integrated cochlear implant sound processor in complex acoustic environments and daily wearing conditions.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Furthermore, no embodiment in this disclosure is required to achieve all the effects described above. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a schematic diagram illustrating the wearing method of the integrated sound processor provided in an embodiment of this disclosure.

[0020] Figure 2 This is a schematic diagram showing the relative positional relationship of the cochlear implant provided in the embodiments of this disclosure when magnetically coupled.

[0021] Figure 3 A schematic diagram illustrating the basic working principle of the processing apparatus provided in the embodiments of this disclosure.

[0022] Figure 4 This is a schematic diagram showing two omnidirectional microphones arranged opposite each other along the axis of an integrated sound processor, as provided in an embodiment of this disclosure.

[0023] Figure 5 This is a flowchart illustrating an integrated cochlear implant sound processing method based on dual-modal signal input, provided as an embodiment of this disclosure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] This embodiment applies to an integrated sound processor used in conjunction with a cochlear implant. The cochlear implant and the integrated sound processor together form a processing device. The cochlear implant includes at least an RF receiving coil and an implanted chip; the integrated sound processor includes two omnidirectional microphones, a bone conduction vibration sensor, a signal processing chip, an RF transmitting chip, an RF transmitting antenna, buttons, a power supply, an external magnet, and a housing. The two omnidirectional microphones collect sound transmitted through the air, the bone conduction vibration sensor collects vibrations transmitted through the skull when the wearer speaks, the signal processing chip processes the three signals according to a unified sampling clock, and sends the processing results to the cochlear implant via the RF transmitting chip and the RF transmitting antenna.

[0026] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the wearing method of the integrated sound processor provided in this embodiment of the disclosure. The integrated sound processor is worn on the outer side of the scalp behind the ear, as shown below. Figure 2 As shown, Figure 2 This is a schematic diagram showing the relative positional relationship of the cochlear implant provided in this embodiment when magnetically coupled. The external magnet built into the sound processor unit is coupled to the internal magnet in the radio frequency receiving coil implanted under the patient's skin through magnetic attraction, thereby causing the processor to be attracted to the scalp outside the receiving coil implantation site, which is usually located behind the ear; the external magnet is mechanically connected to the outer shell structure of the unit.

[0027] Specifically, the external sound processor is worn on the outer side of the scalp, and contains an external magnet. The cochlear implant is located below the scalp, containing an internal magnet, a radio frequency receiving coil, and implant processing circuitry. The external and internal magnets are magnetically coupled along the thickness of the scalp, ensuring the sound processor is stably attached to the implantation site. Simultaneously, the external radio frequency transmitting antenna and the implant's radio frequency receiving coil maintain corresponding positions, guaranteeing the stability of wireless radio frequency energy and data transmission. This magnetic coupling structure achieves reliable positioning of the external device and the implant without mechanical connection. While ensuring wearing comfort, it also helps maintain the coupling efficiency of the radio frequency link, improving communication reliability and sound processing stability during long-term use of the cochlear implant system.

[0028] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the basic working principle of the processing device provided in this embodiment. The device should consist of at least two omnidirectional microphones, a bone conduction vibration sensor, a signal processing chip, a radio frequency (RF) transmitter chip, an RF transmitter antenna, buttons, a power supply, and other components. In this embodiment, the aforementioned integrated sound processor first acquires sound signals through the two microphones and skull vibration signals through the bone conduction vibration sensor. The signal processing chip performs filtering, noise reduction, STFT, and ISTFT operations on the three input signals. The processing results are then transmitted wirelessly to the implanted device under the patient's skin via the RF transmitter chip and RF transmitter antenna. The two omnidirectional microphones and the bone conduction vibration sensor are respectively connected to the signal processing chip. The output of the signal processing chip is connected to the RF transmitter chip, and the RF transmitter chip transmits the processing results to the cochlear implant via the RF transmitter antenna.

[0029] Specifically, two microphones and a bone conduction vibration sensor serve as acquisition units for air-conducted sound signals and bone conduction vibration signals, respectively, and their outputs are all connected to a signal processing chip. After completing dual-modal signal synchronous sampling, sound enhancement, bone conduction compensation, and dynamic range control, the signal processing chip sends the final output data to the radio frequency (RF) transmitter chip. The RF transmitter chip establishes an RF wireless communication link with the implant via an RF transmitting antenna, sending the encoded sound information to the cochlear implant. Buttons are used for user operation and working mode control, and a power supply provides power to each functional module, thus forming a complete sound processing link consisting of dual-modal signal acquisition, digital signal processing, and wireless RF transmission, enabling collaborative work between the external sound processor and the cochlear implant.

[0030] like Figure 4 As shown, Figure 4 This is a schematic diagram of two omnidirectional microphones arranged opposite each other along the axis of an integrated sound processor, with an example spacing of 12 mm. The bone conduction vibration sensor is arranged on the axis of the device, and its printed circuit board is rigidly connected to the housing and has an additional counterweight. The two omnidirectional microphones and their corresponding acoustic cavity components are suspended inside the housing by an elastomer, and the mass, stiffness, damping and center of mass position of the two suspension components are kept paired.

[0031] Specifically, two microphones are mounted on opposite sides of the outer casing and isolated from it via elastic connections and vibration-damping silicone sleeves to reduce structural noise transmitted from device vibrations to the microphones. A bone conduction vibration sensor is positioned at the center of the device, rigidly connected to the printed circuit board, and forms a pre-defined inertial mass vibration unit through counterweights to improve its response sensitivity to skull vibration acceleration. A magnet is located in the center of the processor for magnetic coupling and fixation with the implant magnet. This structure mechanically separates the air conduction signal acquisition unit from the bone conduction signal acquisition unit, reducing interference from casing vibrations on the microphone signals and improving the stability of the bone conduction vibration sensor's measurement of the wearer's vocal vibrations, thus providing a stable and reliable input signal for subsequent dual-modal sound processing algorithms.

[0032] In one embodiment, the bone conduction vibration sensor may be a high-sensitivity accelerometer. The sensor, printed circuit board, and counterweight constitute the overall mass, and the contact area between the scalp tissue and the outer shell is equivalent to a spring and damper. Along a direction perpendicular to the skull surface, the motion of the overall mass relative to the skull satisfies: in, The total mass of the sensor, printed circuit board, and counterweight is expressed in kilograms. Contact damping, measured in Newton-seconds per meter; Contact stiffness, measured in Newtons per meter; The displacement of the overall mass relative to the skull is expressed in meters. Let be the acceleration due to skull vibration, measured in meters per second squared. Both sides of the equation represent force. The contact resonant frequency is: in, The unit is Hertz. To ensure that the contact resonant frequency is below the lower boundary of the main speech frequency band, the lower limit of the total mass can be determined by the following formula: in, The maximum permissible contact resonant frequency is expressed in Hertz (Hz). As an example, with a measured contact stiffness of 10,000 Newtons per meter, setting the maximum permissible contact resonant frequency to 80 Hz yields a calculated minimum total mass of approximately 39.6 grams, which can be configured to exceed 40 grams in engineering applications. This setup ensures that the sensor housing and skull move approximately synchronously within the speech band. If the vibration transmission relationship is represented by the frequency domain ratio of the sensor-measured acceleration to the skull acceleration, then within the speech band, the following conditions are met... ,in, and The units are all meters per second squared. Dimensionless, thereby reducing the additional phase change of the bone conduction spectrum at the contact interface.

[0033] The acoustic cavity components of the two omnidirectional microphones are each suspended by an elastic body. Let the total mass of a single acoustic cavity component and its counterweight be... The stiffness of the elastic body is The damping coefficient is The outer shell vibration displacement is The displacement of the acoustic cavity component is Then the vibration transmission relationship of the foundation can be written as: in, and These are the frequency domain representations of the corresponding displacements, and their ratio is the dimensionless transmissibility. Frequency, measured in Hertz; The natural frequency of the suspension assembly. For the damping ratio, the two satisfy the following: in, The unit is kilogram. The unit is Newtons per meter. The unit is Newton-second per meter. The frequency is significantly higher than... At that time, the displacement transmissibility is approximately based on Decrease. If a displacement attenuation of not less than 30 dB is required from 300 Hz, the natural frequency can be defined by the following formula: In one example, the natural frequency is configured to 40 Hz, and the total mass is calculated using the following formula: The damping ratio was then adjusted to be between 0.1 and 0.3 by adjusting the counterweight. The parameters of both suspension components were measured individually using elastomers from the same batch, with mass differences controlled within assembly tolerances, and the relative positions of the center of mass to the center of each microphone diaphragm were identical. The displacement difference between the two components and the outer shell is expressed as: in, and All units are meters. After assembly, a swept-frequency vibration test is performed for inspection. Based on this, the counterweight is adjusted to ensure that the relative distance between the two omnidirectional microphones remains stable during device movement. The signal processing chip loads processing parameters upon startup.

[0034] For example, the system sampling rate The frequency was set to 12.8 kHz, the frame length of the short-time Fourier transform was set to 20 milliseconds, corresponding to 256 sampling points; the frame shift was set to 10 milliseconds, corresponding to 128 sampling points; the window function used was the Hanning window, and the fast Fourier transform length was set to 256. The signal processing chip also loaded preset steering vectors, initial values ​​of the noise covariance matrix, noise covariance matrix smoothing factors, diagonal loading, preset energy thresholds, initial values ​​of the bone conduction compensation filter, filter update smoothing factors, preset amplitude lower limits, preset signal-to-noise ratio thresholds, impact detection thresholds, rising edge thresholds, hold time, feedforward compression thresholds, feedback compression thresholds, feedforward compression ratio, feedback compression ratio, attack time constant, release time constant, and digital delay of the bone conduction signal. These parameters were stored in non-volatile memory and written to the signal processing chip's runtime memory after power-on.

[0035] The preset energy threshold is calibrated using bone conduction noise floor under quiet conditions. In practical applications, the bone conduction frame energy is calculated frame by frame within one second while the wearer remains still and silent, yielding the mean and standard deviation of the noise floor. The preset energy threshold is the sum of the mean noise floor and six times the standard deviation. in, The preset energy threshold is expressed in squared normalized digital amplitude. This represents the average energy of the low-noise frame, expressed as the square of the normalized digital amplitude. The standard deviation of the noise floor frame energy is expressed as the square of the normalized digital amplitude. If the mean of the noise floor frame energy is... The standard deviation is The preset energy threshold is When no quiet calibration data is available, the threshold obtained from factory statistics on the same model of sensor can be used, and replaced with the individual calibration results after the sensor first enters a quiet state. The noise covariance matrix smoothing factor can be configured from 0.95 to 0.995, and the filter update smoothing factor can be configured from 0.90 to 0.99; these values ​​are selected based on the noise change rate and the stability of the spectral mapping, respectively, with values ​​closer to 1 used when the change is slow.

[0036] After installation, the two omnidirectional microphones underwent channel consistency calibration. The calibration sound source was placed at an equidistant position directly in front of the wearer, and a sweep signal from 300 Hz to 3400 Hz was played. The amplitude and phase of each microphone at each frequency were recorded. Using the first microphone as a reference, the amplitude correction factor and fixed phase correction amount for the second microphone were calculated. in, For the second microphone at frequency The amplitude correction coefficient at the specified point is dimensionless. For the first microphone at frequency The range at that point, For the second microphone at frequency For the amplitude at that point, both use the same normalized digital amplitude benchmark; For the second microphone at frequency The fixed phase correction value at the location, in radians; and The first and second microphones are respectively located at the frequency of the two microphones. The phase at a given point is expressed in radians. After generating the first microphone frequency domain signal and the second microphone frequency domain signal, the amplitude correction coefficient and the fixed phase correction amount are applied to the second microphone frequency domain signal. The amplitude correction coefficient is the reciprocal of the ratio of the two amplitudes, and the fixed phase correction amount is the inverse of the phase difference between the two phases.

[0037] The preset guide vector is generated according to the actual installation direction of the equipment and stored in the parameter table. Taking a 12 mm pitch, a sound velocity of 343 m / s, and a sampling rate of 12.8 kHz as an example, the parameter table contains complex vectors with 129 non-negative frequency points; negative frequency points are recovered from conjugate symmetry and are not stored separately. When there is no temperature sensor, the preset sound velocity is used, and the processing flow does not depend on real-time temperature updates.

[0038] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0039] Please see Figure 5 , Figure 5 This is a flowchart of an integrated cochlear implant sound processing method based on dual-modal signal input according to an embodiment of the present invention, as shown below. Figure 5 As shown, the method includes: S1, synchronously acquire two microphone time-domain signals and one bone conduction time-domain signal, and perform time-frequency transformation on the three time-domain signals using the same parameters to obtain the microphone frequency domain signal and the bone conduction frequency domain signal; S2, generate speech activity markers based on bone conduction frequency domain signals and generate impact markers based on bone conduction time domain signals; S3, based on the preset steering vector and noise covariance matrix, adaptive beamforming is performed on the microphone frequency domain signal to obtain the beamforming frequency domain signal, and the noise covariance matrix is ​​updated only in the frame where the voice activity flag indicates a non-voice frame and the impact flag does not indicate an impact. S4, according to the voice activity flag and the impact flag, the bone conduction compensation filter is updated within the voice frame and frozen during the impact. The bone conduction compensation filter is updated according to the beamforming frequency domain signal and the bone conduction frequency domain signal. The bone conduction compensation filter is used to perform amplitude compensation on the bone conduction frequency domain signal to obtain the compensated bone conduction amplitude. S5 combines the phase of the compensated bone conduction amplitude and the beamforming frequency domain signal to obtain the enhanced frequency domain signal. The enhanced frequency domain signal is then subjected to inverse time-frequency transformation and superposition to obtain the output time domain signal.

[0040] In some embodiments, for step S1, the three signals are sampled synchronously. The two omnidirectional microphones respectively output the time-domain signal of the first microphone. Second microphone time domain signal The bone conduction vibration sensor outputs bone conduction time-domain signals. The three analog-to-digital conversion channels use the same sampling clock and are written to three equal-length buffers within the same sampling interrupt. The bone conduction time-domain signal first undergoes a fixed digital delay so that the three signals correspond to the same acquisition period at the frame boundary; this fixed digital delay only corrects the fixed delay of the acquisition channel and the pre-filter, and does not replace the digital delay of the bone conduction signal in the dual-path dynamic range control described later.

[0041] Short-time Fourier transforms are performed on the three time-domain signals using the same parameters. Using any one of the time-domain signals... For example, its frequency domain result is: in, Indicates the first microphone, the second microphone, or the bone conduction channel; Frequency index; For frame indexing; The number of sampling points is the frame length. This represents the number of frame shift sampling points. The length of the Fast Fourier Transform; The coefficients for the Hanning window are dimensionless. The same normalized amplitude reference is used as the input digital sample value. After processing, the first microphone frequency domain signal is obtained. Second microphone frequency domain signal and bone conduction frequency domain signals For example, when the sampling rate is 12.8 kHz, the center frequency of the 20th frequency point is: The frequency calculation results are then processed by speech activity detection, adaptive beamforming, and bone conduction compensation filter, respectively.

[0042] In some embodiments, for step S2, a voice activity marker is generated based on the bone conduction frequency domain signal. The signal processing chip sums the effective frequency point energy of each frame of bone conduction frequency domain signal to obtain the bone conduction frame energy: in, For the first Bone conduction frame energy, expressed as normalized digital amplitude squared; and Corresponding to 300 Hz and 3400 Hz respectively, this eliminates DC components and high-frequency noise outside the speech band. When the bone conduction frame energy reaches a preset energy threshold, the current frame is marked as a speech frame; when it does not reach the preset energy threshold, the current frame is marked as a non-speech frame. The speech activity flag is represented as follows: in, To preset the energy threshold, the unit is... The same applies. To limit repeated switching between frames, the voice activity flag is set to 1 only after three consecutive frames reach the preset energy threshold, and to 0 only after five consecutive frames fail to reach the preset energy threshold; the flag of the previous frame is maintained until the corresponding number of consecutive frames has been met. For example, the bone conduction frame energies of three consecutive frames are respectively... , and The preset energy threshold is Only the last two frames reach the threshold, and the speech activity marker remains unchanged; the energy in the next frame is... When three consecutive frames are formed and the threshold is reached, the voice activity flag is set to 1.

[0043] Impact markers are generated based on bone conduction time-domain signals. The bone conduction time-domain signal is first passed through a second-order high-pass filter with a cutoff frequency of 100 Hz to obtain a high-pass bone conduction signal. Then calculate the Teager energy: in, and All values ​​are represented using the normalized squared digital amplitude. Considering that the discrete Teager energy may be negative under low-amplitude noise, it is first non-negatively processed: Then recursively smooth the non-negative Teager energy: in, The smoothed Teager energy envelope, This is a dimensionless smoothing coefficient. The smoothing coefficient is determined based on the envelope time constant. in, This represents the envelope time constant, in seconds; in the example with a sampling rate of 12.8 kHz and an envelope time constant of 0.5 milliseconds, It is approximately 0.855. For example, if three consecutive high-pass bone conduction sampling values ​​are 0.10, 0.40 and 0.20, the Teager energy of the current sampling point is 0.14, which remains 0.14 after nonnegation; when the smooth envelope of the previous sampling point is 0.02, the current smooth envelope is approximately 0.037.

[0044] Impact detection thresholds and rise time thresholds were calibrated using normal sound-emitting samples and samples lightly touching the outer shell: the results of the two types of samples were statistically analyzed separately. The difference between adjacent sample points is used as the midpoint between the 99th percentile of the normal vocalization sample and the 5th percentile of the light touch sample as the corresponding threshold.

[0045] When the smoothed Teager energy envelope reaches the impact detection threshold, and the envelope increment of adjacent samples reaches the rising edge threshold, the impact flag is set to 1. in, The impact detection threshold, Both the rising edge threshold and the normalized squared digital amplitude are used to represent the threshold. The impulse flag is set to 1 and held for 50 milliseconds, corresponding to 640 sampling points. It is not repeatedly extended during the holding period. If the above two conditions are met again after the holding period ends, the flag is reset to 1 and a new holding period begins. The voice activity flag is generated per frame, and the impulse flag is generated per sample point. Setting the impulse flag to 1 for any sample point covered by a frame indicates that the frame is in an impulse period, and the frame-level impulse state is recorded. This indicates that when the frame does not cover the sample point with the impact flag set to 1, .

[0046] In some embodiments, for step S3, adaptive beamforming is performed on the microphone frequency domain signal based on a preset steering vector and noise covariance matrix. When establishing the coordinate system, the line connecting the two omnidirectional microphones is set as... The axis, the coordinates of the first microphone are The coordinates of the second microphone are The direction directly in front of the wearer should be perpendicular to the microphone line and pointing outwards from the head. Azimuth angle. relatively If the positive axis is defined, then the preset steering vector of the far-field plane wave is: in, The distance between the two omnidirectional microphones is in meters. The speed of sound in air, measured in meters per second; For the first The angular frequency of each frequency point is expressed in radians per second. (The text then abruptly shifts to a seemingly unrelated topic: "Corresponding to the front.") At this point, the theoretical propagation time delay of the far-field sound from directly in front of the two omnidirectional microphones is the same, and the preset directional vectorization is... For predetermined target directions that are not directly ahead, then proceed according to that direction. Calculate the phase difference. This coordinate definition avoids mistakenly writing "directly in front" as the direction along the microphone line.

[0047] The frequency domain signals from the two microphones are combined to form a receiving vector: The noise covariance matrix is ​​updated only when the speech activity flag indicates a non-speech frame and the current frame is not in an impulse period: in, It is a second-order complex matrix, and the unit of each element is the square of the normalized numerical amplitude; The smoothing factor for the noise covariance matrix; superscript This indicates the conjugate transpose. If the speech activity flag indicates a speech frame, or the current frame is in the impulse phase, then let... The initial noise covariance matrix can be set as the identity matrix multiplied by the average noise power of the power-on silent segment; if there is no silent segment, the initial factory noise power value is used first, and then updated according to the above formula after a non-speech frame is detected.

[0048] To limit abrupt weight changes when the matrix approaches singularity, the noise covariance matrix is ​​diagonally loaded: in, For diagonal loading, Represents the trace of a matrix. It is a second-order identity matrix. The diagonal loading amount is determined based on the noise power; for example, when the loading level is -30 dB, the power ratio is... The minimum variance distortionless response beamforming weight is set to 0.001. Beamforming weights satisfy and reduce output noise power Take the minimum value. The beamforming frequency domain signal is: As a calculation example, at a certain frequency, the noise covariance matrix after diagonal loading is: The preset guide vector directly in front is Substituting into the above formula, we get the weights as follows: If the frequency domain signals of the two microphones are respectively and The beamforming frequency domain signal is In actual operation, the weight of each frequency point is calculated independently, and the weight changes gradually with the noise statistics in non-speech frames.

[0049] In some embodiments, for step S4, the initial value of the output noise power at the current frequency point is first calculated from the beamforming weights and the noise covariance matrix: During non-speech frames and not in the impact period, the output noise power is recursively updated according to the actual power of the beamforming frequency domain signal; during speech frames or impact periods, the value of the previous frame is maintained. in, This is a dimensionless noise power smoothing factor, which can be the same value as the noise covariance matrix smoothing factor. The signal-to-noise ratio (SNR) estimate for the current frame is obtained by summing the beamforming signal power and output noise power at each frequency point. in, The unit is decibel; To prevent positive numbers with a denominator of 0, its value can be set to the full-scale power. The preset signal-to-noise ratio threshold can be determined in the quiet spontaneous sound calibration, specifically by taking the 10th percentile of the effective spontaneous sound frame signal-to-noise ratio and limiting it to the range of 3 dB to 10 dB; for example, if the 10th percentile obtained from the calibration is 6 dB, then the preset signal-to-noise ratio threshold is set to 6 dB.

[0050] The bone conduction compensation filter is updated only when the speech activity marker indicates that the signal-to-noise ratio (SNR) of the beamforming frequency domain signal reaches a preset SNR threshold and the current frame is not in an impact period. The instantaneous spectral mapping used for online updates is as follows: in, The amplitude ratio is dimensionless. To prevent the preset positive value of 0 for the bone conduction frequency domain signal amplitude, a full-range amplitude can be used. The bone conduction compensation filter is updated using a time-recursive method. in, For bone conduction compensation filters in the first Amplitude coefficient at each frequency point The smoothing factor is updated for the filter. If any update condition is not met, the bone conduction compensation filter of the previous frame is retained. The impact flag has higher priority than the speech activity flag, that is, during the impact period, regardless of the value of the speech activity flag, the noise covariance matrix and the bone conduction compensation filter are not updated; during the non-impact period, non-speech frames only update the noise covariance matrix, and speech frames that reach the preset signal-to-noise ratio threshold only update the bone conduction compensation filter.

[0051] The initial state of the bone conduction compensation filter can be set to an all-pass state where the amplitude coefficient at all frequencies is 1, or it can be loaded with the average spectrum mapping measured offline. During offline measurement, the wearer's self-generated microphone signal and bone conduction signal are simultaneously acquired in a quiet environment. First, a frontal beamforming is performed on the microphone signal, and then the average amplitude ratio at each frequency is calculated using the following formula: in, To calibrate the set of speech frames; both the numerator and denominator are the sum of the squares of normalized digital amplitudes. For example, at a certain frequency point, the bone conduction compensation filter in one frame is 2.70, the beamforming amplitude in the current frame is 0.24, and the bone conduction amplitude is 0.08, ignoring extremely small values. The instantaneous spectrum mapping is 3.00; the filter update smoothing factor is 0.95, and the update result is 2.715. This result is directly used for bone conduction amplitude compensation at the same frequency point.

[0052] To avoid amplifying the bone conduction compensation filter by abnormally high instantaneous amplitude ratios at individual frequencies, the instantaneous spectrum mapping can be limited to the offline calibration range before online updates. Specifically, the 5th and 95th percentiles of the amplitude ratios at each frequency point are calculated from multiple spontaneous sound calibration data segments of the same wearer, serving as the lower and upper limits for that frequency point, respectively; when individual data is unavailable, the statistical range of a group of devices of the same model is used. This limitation process only applies to... The values ​​do not change the judgment logic of the voice activity flag, the preset signal-to-noise ratio threshold, and the impact flag.

[0053] In some embodiments, for step S5, amplitude compensation is performed on the bone conduction frequency domain signal using a bone conduction compensation filter. The compensated bone conduction amplitude is: in, To compensate for bone conduction amplitude, normalized numerical amplitude is used for representation; The preset lower limit of amplitude can be determined based on the quantization noise of the bone conduction channel, for example, by taking the full-range amplitude. Times. Extracting the phase of the beamforming frequency domain signal: The compensated bone conduction amplitude is synthesized with the phase at the same frequency and within the same frame to obtain the enhanced frequency domain signal: For example, if the bone conduction compensation filter at a certain frequency is 3.00 and the bone conduction amplitude is 0.08, the compensated bone conduction amplitude will be 0.24; if the phase of the beamforming frequency domain signal is 0.60 radians, then the enhanced frequency domain signal will be approximately... During synthesis, the complex spectrum of bone conduction is not directly added to the complex spectrum of the microphone. Therefore, the time delay difference between the bone conduction path and the air propagation path will not form a comb-like change with alternating cancellation and continuation in the synthesized spectrum; the microphone phase still retains the natural timing information of the sound propagating through the air.

[0054] The enhanced frequency domain signal is subjected to an inverse short-time Fourier transform, and the output time domain signal is obtained by overlapping and adding the results. The inverse transform result of the frame is: Using the same Hanning window as the short-time Fourier transform, adjacent frames are added according to their frame shift positions: in, To output the time-domain signal, a normalized digital amplitude representation is used; To prevent positive numbers where the sum of squares of the window functions equals zero, a 20-millisecond frame length and a 10-millisecond frame shift result in a 50% overlap. The Hanning window performs amplitude normalization in the overlapping region to continuously reconstruct the time-domain waveform. For example, if the windowing contributions of two adjacent frames to the same output sample are 0.12 and 0.08, respectively, and the sum of squares of the corresponding window functions is 1, the reconstructed sample will be 0.20.

[0055] Dual-path dynamic range control is performed on the output time-domain signal. The impulse flag uses an undelayed, smoothed Teager energy envelope to immediately freeze adaptive updates; the feedforward path of the dual-path dynamic range control delays the same envelope. 1 sampling point, obtained Then convert it to decibel level: in, The unit is decibel full scale; To normalize the full-scale power, we take 1; To prevent positive logarithmic inputs of 0, the feedforward compression threshold can be configured to -10 dB full scale, and the feedforward compression ratio can be configured to 20:1. The positive feedforward attenuation is: in, The feedforward compression ratio is dimensionless. This is the feedforward compression threshold, measured in decibels at full scale. The unit is decibels. The target feedforward gain is... Its value is not greater than 0. The attack coefficient and release coefficient are respectively: in, and These are the attack time constant and the release time constant, respectively, in seconds. The feedforward gain is recursively calculated using the following formula: The feedback path takes the output time-domain signal as input, converts its instantaneous amplitude to decibel level, and smooths it. in, and All units are decibels (full scale). This is a dimensionless smoothing coefficient, calculated using a 0.5 millisecond time constant. The feedback compression threshold can be configured to -3 dB full scale, and the feedback compression ratio can be configured to 100:1. The feedback attenuation and target feedback gain are as follows: in, To provide feedback on the compression ratio, The feedback compression threshold is used. The feedback gain is recursively generated using the same attack time constant and release time constant as the feedforward gain. Both the feedforward gain and the feedback gain are decibel gains not greater than 0; the smaller value is selected as the control gain. The control gain is converted into a linear control gain and applied to the output time-domain signal: ,in, Between 0 and 1, This is the final output signal.

[0056] The bone conduction time-domain signal used in the bone conduction feedforward pathway is digitally delayed before being mapped to the output time-domain signal. The digital delay of the bone conduction signal is determined according to the fixed delay of the main processing path. in, This represents the digital delay of bone conduction signals, measured in sampling points. , , and These represent the delays introduced by short-time Fourier transform, adaptive beamforming, inverse short-time Fourier transform, buffering, and computation, respectively, with each delay measured in sampling points. During device calibration, pulses are simultaneously applied to the three inputs, and the peak position difference between the bone conduction feedforward path and the output time-domain signal path is recorded. The total delay calculated is then used to verify the measurement results.

[0057] The thresholds for dual-path dynamic range control are determined using two types of calibration data. The feedforward compression threshold uses the 5th percentile of the smoothed Teager energy level of a sample touching the outer shell, compared to the 99th percentile of a normal sound sample; if there is an interval between the two, the median value is used. The feedback compression threshold uses the 99.5th percentile of the peak value of the impact-free speech output, plus a 1 dB margin. The attack time constant controls the attenuation build-up speed, and the release time constant controls the recovery speed after the impact ends; for example, these are set to 0.5 ms and 50 ms respectively. The coefficients are calculated from the sampling rate according to the aforementioned exponential relationship, without directly using coefficients from other sampling rates. If the normal sound peak value is statistically -4.2 dB full scale, the feedback compression threshold can be configured to -3.2 dB full scale, and a -3 dB full scale is chosen for ease of fixed-point implementation.

[0058] When implemented on a fixed-point digital signal processing chip, the three time-domain signals can use signed 16-bit fixed-point numbers, while the frequency-domain signals and matrix operations use 32-bit fixed-point numbers with guard bits or single-precision floating-point numbers. When using fixed-point numbers, the short-time Fourier transform output is scaled according to each stage of butterfly operation, and saturation upper limits are set for the noise covariance matrix, bone conduction compensation filter, and gain. Logarithmic and exponential operations can be performed using lookup tables and linear interpolation. The lookup table inputs and outputs are generated according to the normalized range of the corresponding physical quantities. The interpolation error is compared with the floating-point reference result during factory testing to avoid fixed-point overflow altering the threshold judgment result.

[0059] After the signal processing chip completes the final output signal, it can execute acoustic coding or stimulus command encapsulation according to the communication protocol of the cochlear implant, and then transmit it by the radio frequency transmitting chip and radio frequency transmitting antenna. In specific applications, short-time Fourier transform, matrix operation, bone conduction compensation filter update and inverse short-time Fourier transform are executed frame by frame, while impact detection and dual-path dynamic range control are executed sample by sample. Taking 256-point complex transformation and second-order matrix operation as an example, the runtime storage area can store three time-domain frames, three frequency-domain frames, second-order noise covariance matrices of each frequency point, beamforming weights and bone conduction compensation filters, which is suitable for real-time execution by a low-power digital signal processing chip with hardware multiply-accumulate unit and fast Fourier transform instructions.

[0060] Therefore, airborne sound first forms a directional frequency domain signal through two omnidirectional microphones, while the wearer's spontaneous skull vibrations provide a reference for speech activity and amplitude. The noise covariance matrix is ​​updated only during non-speech and impact-free periods, and the bone conduction compensation filter is updated only during speech periods when the signal-to-noise ratio reaches a preset threshold and there are no impacts. The compensated bone conduction amplitude and beamforming phase are synthesized in the frequency domain to avoid spectral fluctuations caused by the direct superposition of the two propagation paths. The sample-level impact flag simultaneously freezes both adaptive updates, and the instantaneous amplitude is limited by both feedforward and feedback gains. The final processed output signal is sent to the cochlear implant via the radio frequency path.

[0061] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The above descriptions are merely preferred embodiments of this application and explanations of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the inventive concept.

Claims

1. A method for integrated cochlear implant sound processing based on dual-modal signal input, characterized in that, include: Simultaneously acquire two microphone time-domain signals and one bone conduction time-domain signal, and perform time-frequency transformation on the three time-domain signals using the same parameters to obtain the microphone frequency domain signal and the bone conduction frequency domain signal; Speech activity markers are generated based on bone conduction frequency domain signals, and impact markers are generated based on bone conduction time domain signals. Adaptive beamforming is performed on the microphone frequency domain signal based on the preset steering vector and noise covariance matrix to obtain the beamforming frequency domain signal. The noise covariance matrix is ​​updated only in frames where the voice activity flag indicates a non-voice frame and the impact flag does not indicate an impact. The bone conduction compensation filter is updated within the speech frame and frozen during the impact according to the speech activity flag and the impact flag. The bone conduction compensation filter is updated according to the beamforming frequency domain signal and the bone conduction frequency domain signal. The bone conduction compensation filter is used to perform amplitude compensation on the bone conduction frequency domain signal to obtain the compensated bone conduction amplitude. The phase synthesis of the compensated bone conduction amplitude and the beamforming frequency domain signal is used to obtain the enhanced frequency domain signal. The enhanced frequency domain signal is then subjected to inverse time-frequency transformation and superposition to obtain the output time domain signal.

2. The method according to claim 1, characterized in that, The generation of the speech activity marker includes: summarizing the energy of the bone conduction frequency domain signal at each frequency point to obtain the bone conduction frame energy; comparing the bone conduction frame energy with a preset energy threshold, marking frames that reach the preset energy threshold as speech frames, marking frames that do not reach the preset energy threshold as non-speech frames, and applying a minimum duration limit to the marking results.

3. The method according to claim 2, characterized in that, The preset guidance vector is pre-calculated based on the distance between the two omnidirectional microphones, the speed of sound propagation, the direction of the target directly in front, and the physical frequency corresponding to each frequency point; the two omnidirectional microphones are arranged on opposite sides along the axis of the integrated sound processor.

4. The method according to claim 3, characterized in that, The adaptive beamforming is a minimum variance distortionless response beamforming, comprising: constructing receive vectors for the frequency domain signals of the two microphones on a frequency-point-by-frequency basis; recursively updating the noise covariance matrix based on the receive vectors of non-speech frames, and diagonally loading the noise covariance matrix; under the constraint of the preset steering vector, determining beamforming weights based on the diagonally loaded noise covariance matrix, and weighting the receive vectors using the beamforming weights to obtain the beamforming frequency domain signal.

5. The method according to claim 4, characterized in that, The update period of the bone conduction compensation filter is the frame in which the voice activity flag indicates a voice frame, the signal-to-noise ratio of the beamforming frequency domain signal reaches a preset signal-to-noise ratio threshold, and the impact flag does not indicate an impact. During the update period, the spectrum mapping is estimated based on the amplitude correspondence between the beamforming frequency domain signal and the bone conduction frequency domain signal at each frequency point, and the bone conduction compensation filter is updated using a time recursive smoothing method.

6. The method according to claim 5, characterized in that, The initial state of the bone conduction compensation filter is either all-pass characteristic or a preset spectral mapping based on the average skull conduction to air conduction characteristics measured offline.

7. The method according to claim 6, characterized in that, The amplitude compensation includes: correcting the amplitude of the bone conduction frequency domain signal at each frequency point through the bone conduction compensation filter, and applying a preset lower limit to the corrected amplitude to obtain the compensated bone conduction amplitude; extracting the phase of the beamforming frequency domain signal at each frequency point, and synthesizing it with the compensated bone conduction amplitude at the corresponding frequency point.

8. The method according to claim 7, characterized in that, Also includes: After obtaining the output time-domain signal, dual-path dynamic range control is performed on the output time-domain signal; a feedforward gain is generated based on the bone conduction time-domain signal, and a feedback gain is generated based on the output time-domain signal. The minimum value selection is performed on the feedforward gain and the feedback gain to obtain the control gain. The control gain is applied to the output time-domain signal to obtain the final output signal.

9. The method according to claim 8, characterized in that, The generation of the feedforward gain includes: performing high-pass filtering on the bone conduction time-domain signal, calculating the Teager energy envelope based on the high-pass filtering result, and smoothing the Teager energy envelope; generating the impact flag based on whether the smoothed Teager energy envelope simultaneously reaches the impact detection threshold and the rising edge threshold; determining the feedforward attenuation based on the level of the smoothed Teager energy envelope relative to the feedforward compression threshold and the feedforward compression ratio, and recursively generating the feedforward gain according to the attack time constant and the release time constant.

10. The method according to claim 9, characterized in that, The generation of the feedback gain includes: converting the output time-domain signal into a decibel level and smoothing it; comparing the smoothed decibel level with a feedback compression threshold; determining the feedback attenuation based on the feedback compression ratio; and recursively generating the feedback gain according to the attack time constant and the release time constant; both the feedforward gain and the feedback gain are decibel gains, and the one with the smallest value is selected and converted into a linear control gain.

Images