Multi-modal fusion speech noise reduction method and device, electronic equipment and storage medium

Abstract

The application provides a multi-modal fusion voice noise reduction method and device, electronic equipment and storage medium, comprising: collecting multi-modal signals and aligning the multi-modal signals to obtain synchronous data streams, wherein the multi-modal signals include AC microphone signals, BC microphone signals and IMU signals; performing air conduction difference noise reduction on two AC microphone signals to obtain AC difference synchronous signals; determining whether it is valid speech through a preset voice activity detection mechanism; in response to valid speech, performing fusion processing on the AC difference synchronous signals, BC synchronous signals and IMU synchronous signals based on a constructed neural network to generate a pure speech signal. Through the collection and fusion processing of multi-modal data, the application has stronger robustness than the traditional single-modal or AC+BC noise reduction method, can reduce non-speech interference, and is suitable for extreme noise environments with low signal-to-noise ratio.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of voice noise reduction, and particularly relates to a multi-modal fusion voice noise reduction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Current traditional voice noise reduction technology is mostly based on air conduction (AC) collection or bone conduction (BC) collection. Among them, the principle of air conduction microphone is that sound waves are transmitted to the air conduction sensor through the air, and then the sound is converted into an electrical signal output. Its advantages are low cost and high fidelity of high-frequency voice. However, due to the easy interference of environmental noise, especially in high-noise (such as greater than 90dB) environment, the traditional air conduction microphone is easy to fail. Bone conduction is a technology that transmits sound through bone vibration. When using a bone conduction microphone, the microphone is usually attached to the zygomatic bone or behind the ear. When speaking, the bone vibration of the head caused by the vocal cords is detected, and then the sound is converted into an electrical signal output. Since environmental noise (such as wind noise and background human voice) is difficult to cause bone vibration, in recent years, bone conduction technology has been gradually used for voice enhancement and noise reduction, and has shown unique advantages in noisy environments. However, bone conduction has the characteristics of low frequency strong and high frequency weak, which can easily cause voice distortion and make the sound sound dull.

[0003] Therefore, some noise reduction schemes combining air conduction and bone conduction have emerged, which effectively make up for the problems of air conduction microphone failure in high-noise environment and high-frequency voice distortion of bone conduction microphone. However, there are still the following shortcomings in the use of industrial high-noise environment:

[0004] (1) The traditional method combines one air conduction microphone and one bone conduction microphone. For non-stationary noise that is not strong in regularity, the noise reduction effect is weak, and it relies too much on post-processing algorithms;

[0005] (2) Due to the unique principle of bone conduction, non-voice activities of people during movement (such as chewing, swallowing, and walking) may be miscollected, which makes it difficult for traditional noise reduction methods to meet the requirements in the scene of frequent movement of users in the industrial field;

[0006] (3) The traditional noise reduction method is difficult to balance performance and power consumption. Some current deep neural network technologies have significantly improved the voice noise reduction effect, but most of them ignore the control of power consumption, which limits the practicality in mobile scenarios with battery life. SUMMARY

[0007] Therefore, the present application aims to provide a multi-modal fusion voice noise reduction method, device, electronic equipment and storage medium to solve at least one of the above problems.

[0008] To achieve the above purpose, the technical scheme of the present application is as follows:

[0009] In a first aspect, the application provides a multi-modal fusion speech noise reduction method, comprising:

[0010] collecting multi-modal signals and performing data alignment on the multi-modal signals to obtain a synchronous data stream, wherein the multi-modal signals include two AC microphone signals, one BC microphone signal, and a group of IMU signals;

[0011] performing air conduction differential noise reduction on the two AC microphone signals to obtain AC differential synchronous signals;

[0012] determining whether it is valid speech through a preset voice activity detection mechanism, wherein the voice activity detection mechanism includes signal preprocessing, speech / non-speech classification, and post-smoothing processing, the speech / non-speech classification includes noise level dynamic estimation, cross-modal signal-to-noise ratio fusion, energy threshold adaptive adjustment, and multi-modal decision fusion to classify and determine whether it is speech or noise;

[0013] in response to valid speech, performing fusion processing on the AC differential synchronous signals, the BC synchronous signals, and the IMU synchronous signals based on a constructed neural network to generate a pure speech signal.

[0014] In a second aspect, based on the same inventive concept, the application further provides a multi-modal fusion speech noise reduction device, comprising:

[0015] a signal collection module configured to collect multi-modal signals and perform data alignment on the multi-modal signals to obtain a synchronous data stream, wherein the multi-modal signals include two AC microphone signals, one BC microphone signal, and a group of IMU signals;

[0016] an air conduction differential processing module configured to perform air conduction differential noise reduction on the two AC microphone signals to obtain AC differential synchronous signals;

[0017] a voice activity detection module configured to determine whether it is valid speech through a preset voice activity detection mechanism, wherein the voice activity detection mechanism includes signal preprocessing, speech / non-speech classification, and post-smoothing processing, the speech / non-speech classification includes noise level dynamic estimation, cross-modal signal-to-noise ratio fusion, energy threshold adaptive adjustment, and multi-modal decision fusion to classify and determine whether it is speech or noise;

[0018] a signal fusion module configured to, in response to valid speech, perform fusion processing on the AC differential synchronous signals, the BC synchronous signals, and the IMU synchronous signals based on a constructed neural network to generate a pure speech signal.

[0019] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method according to the first aspect.

[0020] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions for causing the computer to execute the method according to the first aspect.

[0021] Compared with the prior art, the multi-modal fusion speech noise reduction method, device, electronic device and storage medium provided by the present application have the following beneficial effects:

[0022] The multi-modal fusion speech noise reduction method provided by the present application solves the problems that the traditional air conduction sensor is easily affected by environmental noise, the bone conduction sensor has serious high-frequency loss and is easily affected by vibration such as running, and has stronger robustness than the traditional single-mode or AC+BC noise reduction method, can effectively reduce non-speech interference, and can be applied to an extreme noise environment with low signal-to-noise ratio. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein for a purpose of explanations and are not intended as an improper limitation to the present application. In the drawings:

[0024] Figure 1 The multi-modal fusion speech noise reduction method flowchart described in the embodiments of the present application;

[0025] Figure 2 The multi-modal signal acquisition sensor schematic diagram described in the embodiments of the present application;

[0026] Figure 3 The data alignment processing flowchart described in the embodiments of the present application;

[0027] Figure 4 The VAD detection flowchart described in the embodiments of the present application;

[0028] Figure 5 The multi-modal data fusion processing flowchart described in the embodiments of the present application;

[0029] Figure 6 The multi-modal fusion speech noise reduction device structure schematic diagram described in the embodiments of the present application;

[0030] Figure 7The electronic device hardware structure schematic diagram described in the embodiments of the present application. DETAILED DESCRIPTION

[0031] For the purpose, technical solutions and advantages of the present application to be more clearly and obviously, the present application is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0032] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood as the general meaning understood by those skilled in the art to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Include" or "contain" and similar words mean that the elements or objects before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connected" or "connected" and similar words are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationship, when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] The AC mentioned in the present solution refers to Air Conduction, i.e. air conduction; BC refers to Bone Conduction, i.e. bone conduction; IMU refers to Inertial Measurement Unit, i.e. inertial measurement unit.

[0034] The embodiments of the present application are described in detail below in combination with the drawings.

[0035] Please refer to Figure 1 The present embodiment provides a multi-modal fusion speech noise reduction method, which specifically includes the following steps:

[0036] Step S1, collecting multi-modal signals, and performing data alignment on the multi-modal signals to obtain a synchronous data stream, wherein the multi-modal signals include two AC microphone signals, one BC microphone signal and a group of IMU signals.

[0037] Specifically, in the present embodiment, as shown in Figure 2As shown, the acquired multimodal signals include two AC microphone signals, one BC microphone signal, and one IMU signal. Both AC microphone signals are digital signals. One signal is generated by AC microphone 1, placed close to the mouth, primarily capturing noisy air-conducted speech. The other signal is generated by AC microphone 2, placed further away from the mouth, mainly capturing air-conducted noise. The distance between the two AC microphones is controlled at approximately 5 cm. The BC microphone signal is a digital signal generated by a bone conduction microphone placed against the cheekbone or behind the ear, primarily capturing bone conduction speech. The IMU signal is generated by an IMU inertial measurement unit, which includes an accelerometer and a gyroscope. The accelerometer detects minute reciprocating head movements caused by vocal cord vibration (such as vertical larynx vibration), i.e., acquiring acceleration values. The gyroscope detects head rotation or tilt caused by jaw opening and closing, tongue movement, etc., i.e., acquiring angular velocity values. Both the acceleration and angular velocity signals are digital signals. The IMU is typically worn on the larynx or jaw.

[0038] It should be noted that both AC and BC microphones have devices that output analog signals and devices that output digital signals. Microphones that output analog signals need to have their signals amplified and converted from analog to digital (AD) before subsequent processing, and they generally output signals continuously and uncontrollably after power-on. In this embodiment, to facilitate the control and data alignment of the microphone's data acquisition, a device that outputs digital signals is used.

[0039] In some implementations, such as Figure 3 As shown, data alignment includes hardware-level time synchronization, offline calibration of fixed offset, and robust alignment to prevent failures.

[0040] Hardware-level time synchronization includes a unified clock source, synchronous triggering of sampling, and timestamp marking.

[0041] Offline calibration of the fixed offset involves playing simulated voice to trigger a reference time and recording the response time of each sensor. The static delay of each sensor is calculated based on the trigger reference time and response time. The delay data is stored as calibration parameters in the device firmware and calibration is performed based on the timestamp data.

[0042] Failure-resistant robust alignment includes multi-level residual detection, sensor health scoring, observation noise matrix adjustment, state update logic judgment, and Kalman filtering.

[0043] Specifically, in this embodiment, all sensors (two AC microphones, one BC microphone, and one IMU inertial measurement unit) are connected to the same main control chip. The main control chip sends GPIO trigger pulses to synchronously start all sensors for sampling. A high-precision timer tag with an accuracy within ±1µs is attached to each data packet. The timestamps of the data packets from the four sensors are defined as follows: , , , .

[0044] It needs further explanation that hardware-level synchronization can only guarantee simultaneous sampling by all sensors. However, because the time it takes for speech to travel through the air to the AC microphone and through the body to the BC microphone, as well as the IMU's response time when a person performs a movement, are not the same—for example, the time difference between the sound wave measured by the AC microphone traveling through the air and the sound wave measured by the BC microphone traveling through the bones is typically between 0.1ms and 0.15ms, and the mechanical delay in the IMU sensor's response to movement is around 0.2ms—meaning that even if the AC, BC, and IMU sensors sample simultaneously at the hardware level, the data they collect are not the speech or movement information generated at the same moment. Therefore, to align the data from multiple sensors, in addition to synchronously triggering sampling, offline calibration is also required. The goal is to establish a reference offset by measuring a fixed hardware delay.

[0045] Offline calibration for fixed offset refers to the calibration work performed to avoid data misalignment caused by differences in the transmission and physiological delay characteristics of various sensors. This includes exciting a reference event, calculating the static delay, and storing calibration parameters. Specifically, it involves exciting the reference event by playing a sine wave and synchronous vibration (analog voice), recording the response time of each sensor in the AC / BC / IMU using an oscilloscope, and calculating the static delay of each sensor based on the exciting reference time and response time. For example, the moment when the sine wave and synchronous vibration begin playing is recorded as... The times when the two AC sensors have outputs are recorded as follows: and The moment when the BC sensor outputs data is recorded as... The moment when the IMU sensor outputs data is recorded as... The static delays of the four sensors are respectively , , , .

[0046] These four delay data points are stored as calibration parameters in the device firmware, and calibration is performed based on the timestamp data. , , , ,Right now , , , That is the timestamp that truly represents the corresponding data.

[0047] Furthermore, during use, sensors experience dynamic drift due to factors such as temperature, which has a cumulative effect. As time accumulates, this error increases. Therefore, static calibration... , , , Further dynamic adjustments are needed during use. The robust alignment in this step mainly addresses the issue of dynamic drift of sensors due to factors such as temperature during use.

[0048] Kalman filtering can achieve data alignment from multiple sensors and eliminate error accumulation caused by clock drift through an iterative prediction-update mechanism. However, traditional algorithms are based on the premise that all sensors are working properly. When the data from a certain sensor is abnormal, if Kalman filtering directly fuses the data from the failed sensor, it will lead to divergence in state estimation.

[0049] This embodiment employs a robust alignment algorithm, an improvement on the traditional Kalman filter algorithm, to address this problem. Specifically, it uses multi-level residual detection to score sensor health, and adaptively adjusts the covariance of the Kalman filter model based on this score. This dynamically identifies failed sensors and reduces their weights. The algorithm then determines whether to use a Kalman filter prediction or update mode based on the sensor's health score, achieving a balance between accuracy and robustness. This ensures system stability while providing high-quality synchronous data streams. The specific solution is as follows:

[0050] Step S11: Multi-level residual detection. For each sensor Calculate its short-term observation residuals using the following formula:

[0051]

[0052] In the formula, For the first Short-term observation residuals of individual sensors, For sensors The timestamp observations can be detected using event alignment methods (such as zero-crossing points of speech signals) or energy thresholding methods. For sensors Related observation vectors, Possible values , Representing the The state variable is the sensor clock drift. Let the deviation between the sensor clock and the reference clock be denoted as . Clock drift rate ,but It can be expressed in the following form:

[0053]

[0054] Maintain a sliding window of length L (e.g., L=10) to store the most recent L residuals. Calculate its root mean square based on the observation residuals of L times. .

[0055]

[0056] Step S12: Health Score. Calculate the health score for each sensor based on the root mean square value obtained from multi-level residual detection.

[0057]

[0058] In the formula, For the first Health rating of each sensor A value close to 0 indicates failure, and 1 indicates health. In the formula... This is an empirical threshold (e.g., 1ms).

[0059] Step S13: Adjust the observation noise matrix. Dynamically amplify the observation noise variance of the failed sensor based on the health score. The specific formula is as follows:

[0060]

[0061] In the formula, For the first The standard noise variance of each sensor, when it fails This is equivalent to ignoring the sensor.

[0062] Step S14: State update logic judgment. When the health scores of all sensors are below 0.5, the Kalman filter prediction mode is triggered, and the state equation is extrapolated only, with the pause time for synchronous update. When the health score of at least one sensor reaches 0.5, normal update is performed through Kalman filtering.

[0063] Step S15: Kalman filtering. Based on the judgment logic in step S14, determine whether to activate the Kalman filter prediction mode or the Kalman filter update mode.

[0064] In this model, the Kalman filter prediction mode uses only the state equation extrapolation, meaning the system relies solely on historical states to predict the current time synchronization parameters without fusing any observation data. The state equation is as follows:

[0065]

[0066]

[0067] In the formula, for The state vector at time t, Here is the state transition matrix. for The transpose of the matrix, for The state covariance matrix at time t, Let be the process noise covariance.

[0068] To facilitate understanding, the above parameters are further explained as follows:

[0069] State vector Includes the clock offset of each sensor relative to the reference clock. and drift rate It can be expressed in the following form:

[0070]

[0071] Initially, due to static offset calibration, the initial state is as follows:

[0072]

[0073] State transition matrix This section primarily describes the evolution of the system state. Considering that the sensor's clock deviation is the time deviation from the previous moment plus the drift rate multiplied by the time interval between the two moments, let this time interval be... (i.e., the Kalman filter step size is) Then the state transition matrix It can be represented as:

[0074]

[0075] When the Kalman filter step size is 0.1s (i.e., the filter frequency is 10Hz), the state transition matrix can be expressed as:

[0076]

[0077] Let be the state covariance matrix, representing the uncertainty of the state estimate; it is typically a diagonal matrix. If the initial covariance matrix... The value is relatively large, such as If the initial state of the filter is extremely unreliable, it will quickly trust early observations, resulting in faster convergence. However, if the observation noise is high... (Used in normal update mode) A large initial covariance matrix may cause oscillations in the initial estimate. Smaller values ​​result in a smoother steady state, but the filter converges more slowly. Furthermore, large initial value deviations can lead to prolonged convergence failure. Therefore... The recommended value is moderate, such as .

[0078] Process noise covariance Primarily determined by the stability of the sensor clock, it is typically a diagonal matrix, with the diagonal elements corresponding to the noise variance of each state variable. Clock deviation is generally considered to be... The noise is entirely due to the clock drift rate. Since it has no independent noise, the variance is set to 0, and the clock drift rate is... The variance is ,in For the first The standard deviation of the sensor's drift per second can be obtained through a sensor clock stability test. The step size for the Kalman filter is given by the following formula:

[0079] .

[0080] The Kalman filter update mode, specifically includes the following steps:

[0081] Step S151, Kalman filter prediction:

[0082]

[0083]

[0084] In the formula, In time step Prior state estimation, i.e., based only on the previous time step The information predicts the current state, and its state vector elements are similar to those in the prediction pattern mentioned above. same. The state transition matrix describes how the state variables change from one state to the next. Evolving to time, for The transpose of . To estimate the covariance matrix a priori, for The state covariance matrix at time t, Let be the process noise covariance matrix.

[0085] Step S152: Calculate the Kalman gain :

[0086]

[0087] In the formula, Kalman gain We need to weigh the confidence levels of the predicted state against the observed values, and consider the observation noise. Small (reliable observation), then Increase the number of observations and trust them more.

[0088] Observation noise covariance matrix These are parameters characterizing sensor observation errors in Kalman filtering. This matrix is ​​a diagonal matrix, and its diagonal elements... Indicates sensor The variance of observation noise, The smaller the value, the more the filter trusts the sensor; conversely, the larger the value, the more the filter ignores the sensor. Its value is determined by the sensor's... Standard noise variance The ratio of the health score to the standard noise variance is determined. The value is usually obtained from the clock jitter data in the device datasheet or through experimental calibration.

[0089] Observation matrix It maps state variables to the observation space, used to obtain information from the state vector. Extracting from sensors Related predicted values, for The transpose matrix. In this scheme The value can be:

[0090]

[0091] Step S153: Merge observations To update the status:

[0092]

[0093] In the formula, The posterior state estimate combines the predicted and observed optimal state.

[0094] Step S154: Update covariance:

[0095]

[0096] In the formula, This represents the identity matrix, which is a square matrix with all elements on the main diagonal being 1 and all other elements being 0. The updated state uncertainty is used to estimate the covariance for the posterior time, reflecting the degree of convergence of the filter.

[0097] After multiple prediction-update cycles, the state variables of the Kalman filter... In It will gradually converge to the actual clock deviation, thereby achieving data alignment and outputting a synchronous data stream.

[0098] In step S1 of this embodiment, data time alignment is crucial to ensuring the effectiveness of the algorithm. This embodiment employs hardware-level time synchronization, offline calibration of fixed offset to solve static offset compensation, and robust alignment to resist failure. Through multi-level residual detection, sensor health scoring, and observation noise matrix adjustment, the Kalman filter is improved to solve the dynamic drift problem of the sensor and maximize the time synchronization of AC microphone signals, BC microphone signals, and IMU signals, providing a set of high-quality signals for air conduction differential noise reduction, voice activity detection, and multimodal data fusion processing.

[0099] Step S2: Obtain AC differential synchronization signal by performing air conduction differential noise reduction on the two AC microphone signals.

[0100] Specifically, in this embodiment, air conduction differential noise reduction outputs an air conduction differential signal after AC microphone signal alignment and differential amplification. Since AC microphone 1 is close to the mouth, it collects a speech signal with background noise, which is named for ease of description. , ,in, Useful human voice signals, The signal is a noise signal, and since AC microphone 2 is far from the mouth, the signal it collects is named... , ,in, Useful human voice signals, This is a noise signal.

[0101] During a normal call, AC microphone 1 is close to the mouth, therefore, The signal amplitude is relatively large, and AC microphone 2 is far from the mouth. The signal amplitude is relatively small. Since ambient noise is not sensitive to the distance between the two AC microphones, therefore... and They are almost equal. Further differential processing of the two signals yields... ,Right now The signal is a useful human voice signal, but since the signal amplitude is smaller after subtraction, it needs to be further amplified to obtain the required air conduction differential signal. The amplification factor can be set according to the distance between the two AC microphones, and it is recommended to select between 2 and 5.

[0102] In step S12 of this embodiment, differential noise reduction is adopted. Differential noise reduction is the method of subtracting the signals from the two AC microphones. The signal-to-noise ratio of the signal obtained by differential noise reduction is significantly improved, which is beneficial to the next processing work. It is more effective than the traditional method in the processing of non-stationary noise with large noise. Compared with the traditional single AC microphone noise reduction method, it is less affected by non-stationary noise environment.

[0103] Step S3: Determine whether the speech is valid by using a preset speech activity detection (VAD) mechanism. The speech activity detection mechanism includes signal preprocessing, speech / non-speech classification and post-smoothing. The speech / non-speech classification includes dynamic noise level estimation, cross-modal signal-to-noise ratio fusion, adaptive energy threshold adjustment and multimodal decision fusion, to classify it as speech or noise.

[0104] Specifically, in this embodiment, VAD detection adapts to different noise scenarios by adopting a dynamic noise level estimation method, and is more reasonable than traditional fixed threshold comparison by using an adaptive energy threshold adjustment method. Furthermore, it adopts a hierarchical decision-making strategy through a multimodal decision fusion mechanism, selecting sensor voting weights according to environmental noise, which is more accurate than single-modal decision-making.

[0105] like Figure 4 As shown, signal preprocessing includes preprocessing of the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal. AC differential synchronization signal preprocessing includes high-pass filtering and frame energy calculation. The cutoff frequency of the high-pass filter can be set to 100Hz. Energy calculation can be performed by dividing the data into frames, such as 20ms / frame, and calculating the energy of each frame. BC synchronization signal preprocessing includes band-pass filtering and frame energy calculation. The frequency of the band-pass filter can be set to 100Hz~1000kHz to preserve the frequency bands where speech energy is concentrated. IMU synchronization signal preprocessing includes low-pass filtering of the IMU data and calculation of the three-axis vector sum of the accelerometer. A cutoff frequency of 5Hz is recommended for the low-pass filter.

[0106] The frame energy of the AC differential synchronization signal and the BC synchronization signal is calculated using the squared short-time average amplitude, where N is the frame length. The formula for the squared short-time average amplitude is as follows:

[0107]

[0108]

[0109] In the formula, This represents the size of the nth data point in a frame of AC signal data; This represents the size of the nth data point in a frame of BC signal data; This represents the square of the average amplitude of the AC differential synchronization signal. This represents the square of the average amplitude of the BC synchronization signal;

[0110] The IMU synchronization signal is calculated using the sum of three-axis acceleration vectors, where N is the frame length. The specific formula is as follows:

[0111] .

[0112] In the formula, This represents the square of the x-axis acceleration value of the IMU sensor. This represents the square of the y-axis acceleration value of the IMU sensor. This represents the square of the z-axis acceleration value of the IMU sensor. This represents the square of the average amplitude of the IMU synchronization signal.

[0113] Furthermore, dynamic noise level estimation includes initial noise statistics and dynamic noise updates. Initial noise statistics involve collecting the first M frames as pure noise upon system startup, calculating the initial noise energy mean and noise energy standard deviation; where, , , These are the average noise energy values ​​of the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal, respectively. , , These are the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal, respectively. Frame energy, The standard deviation of AC signal noise energy.

[0114]

[0115]

[0116]

[0117]

[0118] The noise dynamic update method for AC differential synchronization signals is as follows:

[0119]

[0120] In the formula, This is the first smoothing factor, which can be set to 0.8~0.9 to avoid sudden changes in noise values. The noise energy of the current frame. This is a noise estimation based on accumulated historical data, reflecting long-term trends. It provides the latest noise estimates and adapts to noise changes in real time.

[0121] The noise dynamic update method, for BC synchronization signals, is less affected by environmental noise and adopts initial noise. constant.

[0122] The noise dynamic update method, for the IMU synchronization signal, is as follows:

[0123]

[0124] In the formula, This is the second smoothing factor, which can be set between 0.9 and 0.95 to avoid sudden changes in noise values. The noise energy of the current frame. This is a noise estimation based on accumulated historical data, reflecting long-term trends. It provides the latest noise estimates and adapts to noise changes in real time.

[0125] Furthermore, the noise dynamic update method includes a protection mechanism that pauses noise updates when voice activity is detected. Additionally, when the AC differential synchronization signal frame energy suddenly increases (e.g., exceeding...),... This is considered speech or burst noise; the noise is not updated. The IMU synchronization signal angular velocity exceeds the threshold (i.e., (At that time), it does not update its noise.

[0126] angular velocity threshold It is based on the average angular velocity of the gyroscope when the user is sitting still. and standard deviation Determined, generally taken .

[0127] Cross-modal SNR (signal-to-noise ratio) fusion includes SNR calculation, weight allocation calculation, and cross-modal SNR fusion. The formula for calculating the SNR of the three sensors is as follows:

[0128]

[0129] The formula for calculating the weight allocation is as follows:

[0130]

[0131]

[0132]

[0133] in, , , The weights assigned to the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal are respectively. , , The confidence levels for the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal are respectively calculated using the following formulas:

[0134]

[0135]

[0136]

[0137] The confidence level of the AC differential synchronization signal is calculated using an exponential decay function. The increase approaches 1, which is characterized by greater sensitivity to low signal-to-noise ratios and rapid deweighting in complex environmental noise conditions. In the formula... The slope coefficient controls the rate of increase in confidence level, and can be set to around 0.1.

[0138] The confidence level of the BC synchronization signal is calculated using the sigmoid function, which is characterized by the maximum slope at SNR=b and symmetrical saturation on both sides. In the formula, the parameter m is the slope, which can be taken as about 0.2, and b is the offset parameter, which can be taken as about 10.

[0139] The confidence level of the IMU synchronization signal is calculated using the sigmoid function in conjunction with angular velocity. When the angular velocity approaches the threshold, the confidence level is forcibly reduced.

[0140] Cross-modal SNR fusion is calculated using the assigned weights and signal-to-noise ratios of each sensor, as shown in the following formula:

[0141]

[0142] In the formula, This indicates the signal-to-noise ratio of the AC differential synchronization signal. Indicates the signal-to-noise ratio of the BC synchronization signal. Indicates the signal-to-noise ratio of the IMU synchronization signal. Indicates the signal-to-noise ratio of cross-modal fusion;

[0143] Adaptive adjustment of energy threshold:

[0144]

[0145] In the formula, This is a dynamic coefficient, and its value varies with... The magnitude varies and is determined by a piecewise linear function. hour, , hour, The intermediate interval is a linear transition, that is ;

[0146] This indicates the energy threshold of the AC differential synchronization signal; Indicates the energy threshold of the BC synchronization signal; Indicates the IMU synchronization signal energy threshold;

[0147] To correct the bone conduction fixation deviation, experimental calibration was performed. Specifically, the user spoke at different volumes in a laboratory environment, and the energy distribution of the BC synchronization signal was recorded to determine the lower limit of speech energy. , .

[0148] Furthermore, the BC synchronization signal threshold formula uses the hyperbolic tangent function (tanh) to dynamically adjust the offset and selects 10dB as the transition midpoint to achieve non-linear matching between the noise level and the threshold, avoiding speech truncation (misjudged as noise) at low SNR or noise leakage (misjudged as speech) at high SNR.

[0149] Furthermore, considering that IMU noise typically follows a Rayleigh or Gaussian distribution, its standard deviation is related to its mean as follows: Therefore, the IMU synchronization signal threshold formula includes a term representing the square root of the noise mean and a weighted inverse adjustment term. The introduction is because when IMU weights When the threshold is high, reduce the threshold multiplier to enhance sensitivity; when the IMU weights are high... When the threshold is low, increase the threshold multiplier to avoid false triggering.

[0150] Multimodal decision fusion includes independent decisions and a final decision. The rules for independent decisions are as follows:

[0151]

[0152]

[0153]

[0154] In the formula, This indicates the speech activity result determined by the AC differential synchronization signal, where 1 represents speech and 0 represents non-speech. This indicates the speech activity result determined by the BC synchronization signal, with 1 indicating speech and 0 indicating non-speech. This indicates the speech activity result determined by the IMU synchronization signal, with 1 for speech and 0 for non-speech.

[0155] The final results are classified and determined based on the angular velocity threshold and the signal-to-noise ratio, as follows:

[0156]

[0157] In the above formula, a value of V of 1 indicates that the detection result is speech activity; a value of V of 0 indicates that the detection result is noise.

[0158] Post-smoothing processing includes duration filtering and state machine smoothing. Duration filtering means that signals identified as speech segments are only considered valid speech after at least 60ms to avoid transient false triggering. State machine smoothing means that, in order to avoid frequent switching, the possible speech state is maintained for 200ms after the speech ends.

[0159] It should be further explained that multimodal data fusion processing is only initiated when the acquired signal is determined to be valid speech, which helps to save power consumption and resource consumption.

[0160] In step S3 of this embodiment, a balance is found between high performance and low power consumption through VAD detection. No further data fusion processing is performed when there is no voice behavior, reducing system overhead and power consumption, which is more advantageous in mobile operation scenarios powered by batteries. Through dynamic estimation of noise level and adaptive adjustment of threshold, the accuracy of VAD in complex scenarios is significantly improved.

[0161] Step S4: In response to the presence of valid speech, the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal are fused based on the constructed neural network to generate a clean speech signal.

[0162] Specifically, in this embodiment, a noise reduction method using AC+BC+IMU multimodal data fusion is adopted. This method combines the advantages of AC microphone's good high-frequency response and BC microphone's strong noise immunity. Furthermore, by combining the IMU's acquisition of motion features, it avoids the occurrence of false triggering of the BC microphone. Compared with traditional single-modal or AC+BC noise reduction methods, this method is more robust, reduces non-speech interference, and is suitable for extreme noise environments with low signal-to-noise ratios.

[0163] In some implementations, the neural network is built on the Wave-U-Net architecture, which includes interconnected preprocessing layers, encoders, hierarchical fusion layers, and decoders.

[0164] The preprocessing layer is configured to perform normalization and channel-independent mapping processing on the AC differential synchronization signal, BC synchronization signal and IMU synchronization signal.

[0165] The encoder includes multiple layers of processing, each layer consisting of one-dimensional convolution, activation functions, and downsampling;

[0166] The hierarchical fusion layer is configured to concatenate features after downsampling at each layer of the encoder and pass the features of each layer to the decoder via skip connections.

[0167] The decoder includes multi-layer processing, with each layer consisting of upsampling, skip connections, and one-dimensional convolution.

[0168] Specifically, in this embodiment, such as Figure 5 As shown, the preprocessing layer includes normalization and channel-independent mapping. Normalization normalizes the amplitudes of the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal. Channel-independent mapping maps these signals to the same channel dimension using one-dimensional convolution. Since the AC differential synchronization signal, BC synchronization signal, and IMU synchronization signal are measured from different dimensions and output by different sensors, their amplitudes are related to the power supply voltage and sensor characteristics, resulting in significant differences in data scale between different modalities. Direct fusion without normalization leads to the model being dominated by large-scale data, ignoring small-scale but important signals. For example, if the amplitude of the AC differential synchronization signal is 10 times that of the BC synchronization signal, the fused features will be biased towards the AC differential synchronization signal, suppressing the effective features of the BC synchronization signal. Normalization typically involves independently performing z-score standardization (mean = 0, variance = 1) or Min-Max scaling on each modality, such as scaling to the [-1, 1] interval.

[0169] An encoder can contain multiple layers of processing, each layer including one-dimensional convolution (conv1D), activation functions, and downsampling. One-dimensional convolution is used to extract local features of the signal and capture waveform patterns within a local time window; activation functions are used to introduce nonlinear transformations, enabling the network to fit complex audio features; downsampling reduces temporal resolution, expands the receptive field of subsequent layers, and allows the network to focus more on global features.

[0170] The decoder can contain multiple layers of processing, each layer including upsampling, skip connections, and one-dimensional convolution (conv1D). The purpose of upsampling is to gradually expand the low-resolution feature map to the length of the original input, reconstructing the temporal structure of the audio signal; skip connections directly pass the features of the corresponding layer of the encoder to the decoder, supplementing local information such as transients and high-frequency signals lost during upsampling; one-dimensional convolution performs convolution operations on the features from the skip connections, refines the features, and gradually approaches the output target; the upsampling operation of the decoder is symmetrical to the downsampling operation of the encoder, and the number of layers is equal.

[0171] Layered fusion refers to concatenating features after downsampling at each layer of the encoder, and then directly passing the features from each layer to the corresponding layer of the decoder through skip connections.

[0172] Outputting clean speech refers to the clean speech digital signal obtained after multimodal fusion processing of AC data, BC data, and IMU data.

[0173] It should be further noted that the number of layers in the encoder and decoder, as well as the parameter selection for the one-dimensional convolution, can be determined based on the actual usage and the availability of hardware resources. For ease of description, the following example with a 4-layer encoder and decoder will be used to further illustrate multimodal data fusion processing.

[0174] Preprocessing layer: The neural network input signals are synchronized (based on a sampling rate of 16kHz, 0.5s of data, i.e., 8000 sampling points) one-dimensional AC differential signal, one-dimensional BC signal, and one-dimensional IMU signal. The amplitudes of the three input signals are normalized to the [-1, 1] interval, and all inputs are mapped to the same number of channels through independent one-dimensional convolution conv1D (k=7, s=1, ch=16). Here, k is the convolution kernel width, representing the size of the sliding window in the time dimension; s is the stride of the convolution kernel, representing the number of time steps for each movement; and ch is the number of output channels of the convolutional layer, i.e., the number of convolution kernels. Each channel corresponds to an independent convolution kernel, extracting different types of features.

[0175] Encoder: As shown in Table 1, the operations of each layer include convolution processing, nonlinear transformation introduced by the activation function (ReLU activation function is used in this embodiment), and downsampling. The convolution kernel width is k=5, the convolution stride s is 1, the number of channels ch doubles layer by layer from 36 to 256 in the L4 layer, and the number of sampling points halve layer by layer from T / 2 to T / 16 in the L4 layer. The three parameters in the output size represent the batch size, the number of channels, and the number of sampling points, respectively. For example, the output size of the first layer is [B, 32, T / 2], where B is the batch size, representing the number of samples processed in parallel (e.g., 16), 32 represents the number of channels, and T / 2 is the number of output sampling points, which is selected as 8000 in this embodiment.

[0176] Table 1. Encoder layer structure

[0177]

[0178] Fusion Strategy Layer: It adopts a layered fusion approach. The first layer concatenates AC information with BC information, the second layer concatenates IMU feature information, and the third and fourth layers perform AC+BC+IMU full feature fusion.

[0179] Decoder: As shown in Table 2, the decoder adopts a symmetrical structure with the encoder. The resolution is doubled by upsampling in each layer operation, outputting T / 8 from L4 layer to T in L1 layer. At the same time, the number of channels in each layer is halved, from 128 channels in L4 layer to 16 channels in L1 layer. Jump connections are made to the same layer feature values ​​from the corresponding encoder to supplement the local information lost during the upsampling process.

[0180] Table 2 Decoder Layer Structure

[0181]

[0182] Outputting clean speech: After passing through the L1 layer of the decoder to obtain the output data [B, 16, T], it is then passed through a one-dimensional convolution conv1D (k=1, s=1, ch=1) and a hyperbolic tangent function to convert the output into one-dimensional clean speech data with amplitude between [-1, 1].

[0183] It should be further noted that this embodiment is an example given for the convenience of describing the implementation principle. The number of layers selected in the model, the layer fusion strategy, the selection of one-dimensional convolution parameters, and the selection of related functions can all be determined according to different needs and are not limited to this embodiment.

[0184] The clean speech post-processing and output involves converting the digital signal after multimodal data fusion processing into an analog signal through a DA converter, then amplifying the signal through a power amplifier, and finally outputting a transmittable analog clean speech signal.

[0185] This embodiment of a multimodal fusion speech noise reduction method uses air conduction (AC) microphones, bone conduction (BC) microphones, and inertial measurement units (IMUs) to fuse multimodal data. This method solves the problems of traditional air conduction sensors being susceptible to environmental noise, bone conduction sensors suffering from severe high-frequency loss, and being affected by vibrations such as running. It is particularly suitable for application scenarios in high-noise dynamic environments such as sports and industry.

[0186] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0187] Based on the same inventive concept, corresponding to any of the above embodiments, the embodiments of this application also provide a multimodal fusion speech noise reduction device.

[0188] like Figure 6 As shown, the multimodal fusion speech noise reduction device includes:

[0189] The signal acquisition module 11 is configured to acquire multimodal signals and perform data alignment on the multimodal signals to obtain a synchronous data stream. The multimodal signals include two AC microphone signals, one BC microphone signal, and a set of IMU signals.

[0190] The air conduction differential processing module 12 is configured to obtain an AC differential synchronization signal by performing air conduction differential noise reduction on the two AC microphone signals.

[0191] The speech activity detection module 13 is configured to determine whether the speech is valid through a preset speech activity detection mechanism. The speech activity detection mechanism includes signal preprocessing, speech / non-speech classification and post-smoothing processing. The speech / non-speech classification includes dynamic noise level estimation, cross-modal signal-to-noise ratio fusion, adaptive adjustment of energy threshold and multimodal decision fusion, to classify it as speech or noise.

[0192] The signal fusion module 14 is configured to, in response to valid speech, fuse the AC differential synchronization signal, BC synchronization signal and IMU synchronization signal based on the constructed neural network to generate a clean speech signal.

[0193] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0194] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0195] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0196] Figure 7 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0197] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0198] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0199] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0200] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0201] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0202] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0203] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0204] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0205] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0206] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0207] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0208] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0209] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A multi-modal fusion speech denoising method, characterized in that, The method comprises: collecting multi-modal signals and performing data alignment on the multi-modal signals to obtain a synchronous data stream, wherein the multi-modal signals include two AC microphone signals, one BC microphone signal, and a group of IMU signals; the data alignment includes hardware-level time synchronization, offline calibration of fixed offset, and failure-resistant robust alignment, wherein the offline calibration of fixed offset is to play a simulated voice excitation reference time, record the response time of each sensor, calculate the static delay of each sensor according to the excitation reference time and the response time, store the delay data as a calibration parameter into the device firmware, and calibrate according to the timestamp data; the sensors include two AC microphones, a BC microphone, and an IMU inertial measurement unit; the failure-resistant robust alignment includes multi-level residual detection, sensor health score, observation noise matrix adjustment, state update logic judgment, and Kalman filtering processing, comprising: wherein the multi-level residual detection is to determine the root mean square value according to the short-term observation residual of each sensor; the sensor health score is determined according to the root mean square value obtained by the multi-level residual detection; the observation noise matrix adjustment is to dynamically amplify the observation noise variance of the failed sensor according to the health score; the state update logic judgment is to trigger the Kalman filtering prediction mode when all sensor health scores are lower than 0.5, and to suspend the time synchronization update, and to normally update through Kalman filtering when at least one sensor health score reaches 0.5; performing air conduction differential noise reduction on the two AC microphone signals to obtain an AC differential synchronous signal; determining whether it is valid speech through a preset voice activity detection mechanism, wherein the voice activity detection mechanism includes signal preprocessing, speech / non-speech classification, and post-smoothing processing, the speech / non-speech classification includes noise level dynamic estimation, cross-modal SNR fusion, energy threshold adaptive adjustment, and multi-modal decision fusion to classify and determine whether it is speech or noise; comprising: the noise level dynamic estimation includes initialization noise statistics and dynamic noise update, wherein the initialization noise statistics collects the first M frames as pure noise when the system starts, and calculates the initial noise energy mean and noise energy standard deviation; the dynamic noise update dynamically updates the noise energy using a smoothing factor and has a protection mechanism; the cross-modal SNR fusion is obtained by calculating the assigned weight and the signal-to-noise ratio of each sensor; the multi-modal decision fusion includes independent decision and final decision, and the final decision result is classified and determined based on the joint signal-to-noise ratio and the angular velocity threshold; wherein: ； wherein, denotes the voice activity result of the AC differential synchronization signal decision; denotes the voice activity result of the BC synchronization signal decision; denotes the voice activity result of the IMU synchronization signal decision; denotes the cross-modal fusion signal-to-noise ratio, denotes the angular velocity, denotes the angular velocity threshold; wherein, a value of 1 indicates that the result of the detection is speech activity; a value of 0 indicates that the detection is noise. in response to valid speech, performing fusion processing on the AC differential synchronous signal, the BC synchronous signal, and the IMU synchronous signal based on the constructed neural network to generate a pure speech signal.

2. The method of claim 1, wherein: wherein, the hardware-level time synchronization includes a unified clock source, synchronous trigger start sampling, and timestamp marking.

3. The method of claim 2, wherein: a short-term observation residual of each sensor is calculated by a set residual formula, and the root mean square of the short-term observation residual is calculated. The root mean square value obtained by the multi-stage residual detection is used to calculate the health score of each sensor, and the observation noise variance of the failed sensor is dynamically amplified based on the health score; The health score is compared with a preset threshold to trigger the Kalman filter prediction mode or the Kalman filter update mode.

4. The method of claim 1, wherein, The signal preprocessing includes: AC differential synchronous signal preprocessing, which performs high-pass filtering and frame energy calculation on the AC differential synchronous signal; BC synchronous signal preprocessing, which performs band-pass filtering and frame energy calculation on the BC synchronous signal; IMU synchronous signal preprocessing, which performs low-pass filtering and three-axis vector sum calculation on the IMU synchronous signal.

5. The method of claim 1, wherein: In response to the classification discrimination being speech, the continuous duration filtering is used to determine valid speech or invalid speech, and the state machine smoothing is performed on the valid speech.

6. The method of claim 1, wherein: The neural network is constructed based on a Wave-U-Net architecture, which includes a preprocessing layer, an encoder, a hierarchical fusion layer and a decoder connected in series; The preprocessing layer is configured to perform normalization and channel independent mapping on the AC differential synchronous signal, the BC synchronous signal and the IMU synchronous signal; The encoder includes multiple layers of processing, each layer including one-dimensional convolution, activation function and down-sampling; The hierarchical fusion layer is configured to perform feature splicing after down-sampling of each layer of the encoder, and to pass the features of each layer to the decoder through a jump connection; The decoder includes multiple layers of processing, each layer including up-sampling, jump connection and one-dimensional convolution.

7. A multi-modal fusion speech noise reduction device, characterized in that, It includes: A signal acquisition module configured to acquire multi-modal signals and perform data alignment on the multi-modal signals to obtain synchronous data streams, wherein the multi-modal signals include two AC microphone signals, one BC microphone signal and a set of IMU signals; The data alignment includes hardware-level time synchronization, offline calibration of fixed offset and failure-resistant robust alignment, wherein the offline calibration of fixed offset is to play an analog voice excitation reference time and record the response time of each sensor, calculate the static delay of each sensor according to the excitation reference time and the response time, store the delay data as a calibration parameter in the device firmware, and calibrate according to the timestamp data; The sensors include two AC microphones, a BC microphone and an IMU inertial measurement unit; The failure-resistant robust alignment includes multi-stage residual detection, sensor health score, observation noise matrix adjustment, state update logic judgment and Kalman filter processing, including: The multi-stage residual detection is to determine the root mean square value according to the short-term observation residual of each sensor; The sensor health score is determined according to the root mean square value obtained by the multi-stage residual detection; The observation noise matrix adjustment is to dynamically amplify the observation noise variance of the failed sensor according to the health score; The observation noise matrix adjustment is to dynamically amplify the observation noise variance of the failed sensor according to the health score; The state update logic determines that when all sensor health scores are lower than 0.5, a Kalman filter prediction mode is triggered, and time synchronization update is suspended; when at least one sensor health score reaches 0.5, normal update is performed through Kalman filtering; The air conduction difference processing module is configured to obtain an AC differential synchronization signal by performing air conduction difference noise reduction on the two AC microphone signals; The voice activity detection module is configured to determine whether it is valid speech through a preset voice activity detection mechanism, wherein the voice activity detection mechanism includes signal preprocessing, voice / non-speech classification, and post-smoothing processing, the voice / non-speech classification includes noise level dynamic estimation, cross-modal signal-to-noise ratio fusion, energy threshold adaptive adjustment, and multi-modal decision fusion to classify and determine whether it is voice or noise; comprising: The noise level dynamic estimation includes initialization noise statistics and dynamic noise update, wherein the initialization noise statistics collects the first M frames as pure noise when the system starts, and calculates the initial noise energy mean and noise energy standard deviation; the dynamic noise update dynamically updates the noise energy using a smoothing factor, and has a protection mechanism; The cross-modal SNR fusion is obtained by calculating the distribution weight and signal-to-noise ratio of each sensor; The multi-modal decision fusion includes independent decision and final decision, and the final decision result is classified and determined based on the joint signal-to-noise ratio and the angular velocity threshold; wherein, comprising: ； wherein, denotes the voice activity result of the AC differential synchronization signal decision; denotes the voice activity result of the BC synchronization signal decision; denotes the voice activity result of the IMU synchronization signal decision; denotes the cross-modal fusion signal-to-noise ratio, denotes the angular velocity, denotes the angular velocity threshold; wherein, a value of 1 indicates that the result of the detection is speech activity; a value of 0 indicates that the detection is noise; The signal fusion module is configured to, in response to valid speech, perform fusion processing on the AC differential synchronization signal, the BC synchronization signal and the IMU synchronization signal based on the constructed neural network to generate a pure speech signal. 8.An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1-6 when executing the program.

9. A non-transitory computer-readable storage medium, comprising: In which, The non-transitory computer readable storage medium stores computer instructions for causing a computer to execute the method of any one of claims 1-6.

Images