An enhanced speech recovery method and device based on multi-modal motion sensor fusion
By using multimodal fusion of accelerometers and gyroscopes and the CSMambaGAN generative adversarial network model, the robustness and fidelity issues of speech signal reconstruction on heterogeneous devices are solved, achieving efficient speech signal reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-05-15
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies struggle to achieve high-quality speech signal reconstruction on heterogeneous devices, especially in scenarios with unstable sampling rates and heterogeneous devices. The robustness and fidelity of single-sensor methods are insufficient, making them unsuitable for complex and diverse real-world application scenarios.
A multimodal fusion strategy using accelerometer z-axis signals and gyroscope x-axis signals is employed. Through hierarchical time-frequency fusion and a generative adversarial network (CSMambaGAN) model, high-fidelity unconstrained speech signals are reconstructed. Specific steps include signal preprocessing, time-domain interleaving, frequency-domain characteristic analysis, and fundamental frequency sensing domain fusion. The differences in timestamp offset and spectral response between the accelerometer and gyroscope are utilized to compensate for energy attenuation in the dominant speech frequency band.
It significantly improves the robustness and fidelity of speech recovery in low sampling rate scenarios, realizes high-fidelity speech signal reconstruction under heterogeneous sampling configurations, breaks through the bottleneck of single-sensor methods, and adapts to complex and diverse scenarios.
Smart Images
Figure CN122290615B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mobile device and software security technology, specifically to an enhanced voice recovery method and apparatus based on multimodal motion sensor fusion. Background Technology
[0002] In mainstream mobile devices such as Android and iOS, motion sensors like accelerometers and gyroscopes are typically classified as zero-permission sensors, meaning third-party applications can access their raw data without explicit user authorization. This zero-permission characteristic also creates possibilities for the recovery and enhancement of multi-source signals. Existing research has confirmed that motion sensors can serve as side-channel carriers to acquire micro-vibration signals during device operation, and previous studies have verified their feasibility in parsing device interaction signals. Based on this, motion sensors can be used to enhance voice signal recovery: when a device's built-in speaker plays audio, the mechanical vibrations caused by the sound are transmitted to the built-in motion sensor through the device's casing, motherboard, and other structures. The device can legally collect these vibration signals and then use signal processing algorithms to reconstruct and enhance the voice signal. This technology can serve as a voice backup and restoration solution for mobile devices. When the original voice file captured by the microphone is damaged, lost, or distorted, the vibration data recorded by the motion sensor can be used to reconstruct the core voice content, achieving the recovery of key voice information. Simultaneously, this technology can be applied to cross-device voice data repair, adapting to different brands and models of mobile terminals, ensuring stable reconstruction and reuse of voice signals in multi-device scenarios. However, speech reconstruction requires the recovery of continuous, high-frequency signals with high information density. This task places higher demands on the effective perceptual bandwidth and fine-grained time-frequency resolution under hardware sampling constraints, making it a more challenging task.
[0003] Some studies attempt to recover user speech signals using a single motion sensor. For example, paper 1, "Gyrophone: recognizing speech from gyroscope signals," uses a gyroscope to capture vibrations propagating through a solid medium. Other studies utilize accelerometers for speech signal recovery. Paper 2, "Learning-based Practical Smartphone Eavesdropping with Built-in Accelerometer," can recover low-frequency speech components within a limited vocabulary; paper 3, "Audio Keyword Reconstruction from On-Device Motion Sensor Signals via Neural Frequency Unfolding," uses aliasing extension technology to improve the bandwidth of recoverable speech; and paper 4, "AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary," employs adversarial learning to recover arbitrary speech signals. However, single-sensor methods are always limited by insufficient effective bandwidth and a single sensing dimension. Furthermore, in scenarios with heterogeneous devices, changing placement, and fluctuating sampling rates, the fidelity and robustness of the recovered speech signal significantly decrease, making it difficult to meet practical application requirements.
[0004] To overcome the limitations of single sensors, recent research has begun to explore multi-motion sensor fusion techniques. For example, paper 5, "Device-Independent Smartphone Eavesdropping Jointly Using Accelerometer and Gyroscope," integrates accelerometer and gyroscope data to achieve device-independent speech signal recovery; paper 6, "Glitchin Time: Exploiting Temporal Misalignment of IMU for Eavesdropping," employs sampling time misalignment technology to effectively improve the sampling rate and optimize recovery results; and paper 7, "The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds," extends fusion technology to the headphone's inertial measurement unit (IMU) to achieve cross-device speech signal recovery. Although multi-sensor fusion research has made some progress, existing fusion methods often rely on specific sampling assumptions such as a fixed sampling rate and regular timestamps. Therefore, they are highly sensitive to sampling configurations, making it difficult to stably achieve high-quality reconstruction of unconstrained speech signals on heterogeneous devices and unable to adapt to complex and diverse real-world application scenarios. Summary of the Invention
[0005] To address the problems existing in the prior art, the present invention provides an enhanced speech recovery method and apparatus based on multimodal motion sensor fusion, which can reconstruct high-fidelity and unconstrained speech signals under heterogeneous sampling configurations, thereby improving the robustness of speech signal recovery.
[0006] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0007] An enhanced speech recovery method based on multimodal motion sensor fusion includes the following steps: Step S1: Collect the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio. Preprocess the collected z-axis signal and x-axis signal respectively to obtain the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis. Step S2: The effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis are fused using a hierarchical time-frequency fusion strategy to obtain the fundamental frequency sensing domain fused spectrum; Step S3: Construct a CSMambaGAN speech reconstruction model based on generative adversarial networks and integrating the CSMamba structure. Perform Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate the speech reconstruction signal.
[0008] Furthermore, the preprocessing of the acquired z-axis and x-axis signals in step S1 is as follows: i: Perform Min-Max normalization on the acquired z-axis and x-axis signals respectively, and map them to... interval; ii: Upsample the normalized z-axis and x-axis signals to a sampling rate of 500Hz to obtain time-aligned z-axis and x-axis signals; iii: Filter and denoise the time-domain aligned z-axis and x-axis signals to obtain denoised z-axis and x-axis signals; iv: Calculate the absolute amplitude sequence of the denoised z-axis signal, and set the vibration segmentation threshold according to the maximum and minimum values in the absolute amplitude sequence of the denoised z-axis signal. Segment the signal above the vibration segmentation threshold from the denoised z-axis signal as the effective vibration signal segment of the accelerometer z-axis, and segment the signal above the vibration segmentation threshold from the denoised x-axis signal as the effective vibration signal segment of the gyroscope x-axis.
[0009] Furthermore, the vibration segmentation threshold The setup process is as follows:
[0010] in, This represents the maximum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients, This represents the minimum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients.
[0011] Further, step S2 includes the following sub-steps: Step S2.1: Based on the timestamp, perform time-domain interleaving and fusion of the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis to obtain a time-domain interleaved signal; Step S2.2: Perform frequency domain characteristic analysis on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis respectively. Based on the fundamental frequency sensing domain fusion strategy, obtain the fundamental frequency sensing domain fused spectrum.
[0012] Furthermore, the process of acquiring the time-domain interleaved signal is as follows:
[0013] in, express The time-domain interleaved signal at that moment, express The effective vibration signal of the accelerometer z-axis at time t is express The effective vibration signal of the gyroscope's x-axis at that moment. express The periodic sampling pulse function of the accelerometer at a given time. , Indicates the sampling point index. Indicates the equivalent sampling interval. Represents the unit impulse function. express The periodic sampling pulse function of the gyroscope at that time. .
[0014] Further, step S2.2 includes the following sub-steps: Step S2.2.1: Perform short-time Fourier transform on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis, respectively, to obtain the time-domain interleaved spectrum signal, the accelerometer z-axis vibration spectrum signal, and the gyroscope x-axis vibration spectrum signal; Step S2.2.2: Within the human speech frequency range of 85~255Hz, calculate the total harmonic energy of the accelerometer and the total harmonic energy of the gyroscope at each frequency using the accelerometer z-axis vibration spectrum signal and the gyroscope x-axis vibration spectrum signal, respectively, and estimate the fundamental frequency of the accelerometer z-axis signal and the fundamental frequency of the gyroscope x-axis signal; Step S2.2.3: If the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the accelerometer z-axis signal, record the amplitude of the accelerometer z-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the gyroscope x-axis signal, record the amplitude of the gyroscope x-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is greater than 250Hz, record the amplitude of the time-domain interleaved spectrum signal; otherwise, record the maximum value of the amplitude of the accelerometer z-axis vibration spectrum signal, the amplitude of the gyroscope x-axis vibration spectrum signal, or the amplitude of the time-domain interleaved spectrum signal at the corresponding frequency point. Step S2.2.4: Sort the amplitudes of the recorded accelerometer z-axis vibration spectrum signal, the gyroscope x-axis vibration spectrum signal, and the time-domain interleaved spectrum signal according to the frequency points collected to obtain the fundamental frequency sensing domain fused spectrum.
[0015] Furthermore, the CSMambaGAN speech reconstruction model that integrates the CSMamba structure in step S3 includes: a generator for mapping the fundamental frequency sensing domain fused spectrum to a Mel spectrum and a HiFiGAN vocoder for generating a speech reconstruction signal based on the Mel spectrum. in: The generator includes an image embedding layer, a CSMamba bottleneck layer, and an upsampled output layer; The image embedding layer is used to project the fundamental frequency sensing domain fused spectrum, which introduces random noise vectors, onto a low-resolution feature space. The CSMamba bottleneck layer adopts a symmetrical encoder-decoder structure, wherein the encoder layer extracts deep features step by step from the spectral feature map of the low-resolution feature space through multi-level downsampling; the latent layer is used to compress the deep features; the decoder layer fuses the multi-scale features of the encoder through skip connections and combines symmetrical upsampling to gradually restore the feature resolution of the compressed deep features. The upsampling output layer is used to restore the feature map with restored feature resolution to the target size, generating a Mel spectrum.
[0016] Furthermore, each downsampling stage comprises two cascaded CSMamba blocks and a block merging layer; Each CSMamba block contains a global branch and a local branch, wherein the global branch utilizes two-dimensional selective scanning of spectral features. Figure 4 The global context in each direction is modeled to obtain a spectral feature map with enhanced local features; the local branch uses a cascaded channel and spatial attention mechanism to extract the spectral feature map that highlights the dominant speech frequency band from the spectral feature map; The block merging layer is used to stitch together the spectral feature map with enhanced local features and the spectral feature map highlighting the dominant speech frequency band to obtain a stitched spectral feature map.
[0017] Furthermore, the CSMambaGAN speech reconstruction model with the fused CSMamba structure is trained by the PatchGAN discriminator until the spectral consistency and adversarial fusion loss of the generator and the adversarial loss of the PatchGAN discriminator converge, thus completing the training of the CSMambaGAN speech reconstruction model. The generator's spectral consistency and adversarial fusion loss ,in, This represents the spectral consistency loss of the generator. , Represents the true Mel spectrum. This represents the generated Mel spectrum. express Norm, Describe the Frobenius norm. express The weighting coefficients, This represents the adversarial loss of the generator. , This indicates the fusion of spectrum in the baseband sensing domain. Introducing random noise vectors noise samples, Represents a generator. Indicates the discriminator, This represents the expected value of the noisy sample. express Weighting coefficients; The adversarial loss of the PatchGAN discriminator ,in, The expected value of a sample without introducing random noise vectors.
[0018] Furthermore, the present invention also provides an enhanced speech recovery system based on multimodal motion sensor fusion, comprising: a vibration signal acquisition module, a vibration signal preprocessing module, a hierarchical time-frequency fusion module, and a speech reconstruction module; The vibration signal acquisition module is used to acquire the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio; The vibration signal preprocessing module is used to preprocess the acquired z-axis signal and x-axis signal respectively to obtain the effective vibration signal segment of the accelerometer z-axis and the effective vibration signal segment of the gyroscope x-axis; The hierarchical time-frequency fusion module is used to fuse the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis using a hierarchical time-frequency fusion strategy to obtain the fundamental frequency sensing domain fused spectrum. The speech reconstruction module employs a CSMambaGAN speech reconstruction model based on a generative adversarial network and a fused CSMamba structure. This model is used to perform Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate a speech reconstruction signal.
[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention presents an enhanced speech recovery method and apparatus based on multimodal motion sensor fusion. It fuses effective vibration signal segments from the accelerometer's z-axis and the gyroscope's x-axis using a hierarchical time-frequency fusion strategy. By leveraging the differences in timestamp offset and spectral response between the accelerometer and gyroscope, it effectively combats sampling rate instability and compensates for energy attenuation in the dominant speech frequency band. This overcomes the bottleneck of existing technologies, which are limited by specific sampling rates and cannot reconstruct arbitrary speech content. It significantly improves the robustness and fidelity of speech recovery in low-sampling-rate scenarios, providing an efficient and practical solution for speech enhancement signal recovery based on zero-authority motion sensors. It achieves high-fidelity, unconstrained speech signal recovery under heterogeneous sampling configurations. Attached Figure Description
[0020] Figure 1 This is a flowchart of the enhanced speech recovery method based on multimodal motion sensor fusion according to the present invention; Figure 2 This is a schematic diagram illustrating the Min-Max normalization process for the z-axis and x-axis signals. Figure 3 This is a schematic diagram showing the spectral envelope signals of the accelerometer's z-axis and the gyroscope's x-axis, along with their corresponding spectra. Figure 4 This is a schematic diagram of a time-domain interleaved signal; Figure 5 This is a schematic diagram of the fused spectrum in the fundamental frequency sensing domain. Figure 6 Network architecture diagram of the CSMambaGAN speech reconstruction model; Figure 7 This is a schematic diagram illustrating the performance of the enhanced speech recovery method of the present invention in a single-speaker scenario, wherein, Figure 7 (a) in the diagram is a schematic of Mel-frequency cepstral distortion (MCD). Figure 7 (b) in the diagram is a schematic diagram of Short-Term Objective Intelligibility (STOI). Figure 7 (c) in the figure is a schematic diagram of the speech-to-reverberation modulation energy ratio (SRMR). Figure 8 This is a schematic diagram illustrating the generalization performance of the speech recovery method of the present invention in the scenario of unknown speaker. Figure 9 This diagram illustrates speech recovery performance under various real-world environmental constraints. Figure 9 (a) in the diagram is a schematic diagram of speech recovery performance under playback volume constraints. Figure 9 (b) in the diagram is a schematic diagram of speech recovery performance under device placement constraints. Figure 9 (c) in the figure is a schematic diagram of speech recovery performance under sampling rate constraints; Figure 10 This is a schematic diagram illustrating the generalization performance of the enhanced speech recovery method of the present invention on different mobile devices; Figure 11 This is a schematic diagram comparing the speech recovery performance of single-sensor and multi-sensor fusion, where... Figure 11 (a) in the figure is a comparison chart of Short-Term Objective Intelligibility (STOI). Figure 11 (b) in the figure is a comparison diagram of Mel-Cepstral Distortion MCD; Figure 12 A diagram illustrating the performance comparison of speech recovery using different fusion strategies; Figure 13 A diagram showing the comparison of speech recovery performance of different speech reconstruction models; Figure 14 This is a schematic diagram comparing the speech recovery performance of the present invention and the AccEar conditional generative adversarial network architecture at different sampling rates; Figure 15 This is a schematic diagram of the enhanced speech recovery device based on multimodal motion sensor fusion according to the present invention. Detailed Implementation
[0021] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.
[0022] Because the diaphragm of the built-in speaker in a mobile device is fixed relative to the motion sensor's direction of motion (motion sensors include accelerometers and gyroscopes), vibration leakage exhibits a clear directionality. To rigorously identify the most sensitive axis for each motion sensor and avoid bias from single-frequency testing, a linear frequency modulation (LFM) experiment was conducted on a Redmi K60 phone: a LFM signal of 50-500Hz was played, covering the full frequency range of the target speech, and the speaker volume was adjusted from 20% to 100%. The signal responses of the three axes of the accelerometer and the three axes of the gyroscope were recorded, and the average signal-to-noise ratio (SNR) was calculated throughout the entire LFM process. The SNR is... , This refers to the signal power when the FM signal is played. This represents the background noise power.
[0023] Since the z-axis of the accelerometer is aligned with the main vibration vector of the mobile device body, experimental results show that the z-axis signal of the accelerometer exhibits the highest sensitivity at all volume levels. Although the overall sensitivity of the gyroscope is lower than that of the accelerometer, its x-axis can effectively capture the significant rotational vibration caused by the resonance of the mobile device body. Therefore, this invention utilizes the z-axis signal of the accelerometer and the x-axis signal of the gyroscope to enhance the speech recovery process.
[0024] like Figure 1 This is a flowchart of the enhanced speech recovery method based on multimodal motion sensor fusion according to the present invention. The enhanced speech recovery method includes the following steps: Step S1: Collect the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio. The sampling rate of the accelerometer and the gyroscope is 200~500Hz. The collected z-axis signal and x-axis signal are preprocessed to solve the problems of signal dimensional heterogeneity, time misalignment and noise interference, so as to obtain the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis.
[0025] The preprocessing process of the acquired z-axis signal and x-axis signal in this invention is as follows: i: Due to inherent differences in the signal reference standards and response characteristics of accelerometers and gyroscopes, direct fusion will lead to bias. For example, the accelerometer's z-axis reference value is approximately 9.8 due to gravity, while the gyroscope's x-axis reference value is close to 0. To mitigate the impact of these data range differences, the acquired z-axis and x-axis signals are respectively processed using Min-Max normalization and mapped to... This allows for the conversion of the accelerometer's z-axis signal and the gyroscope's x-axis signal to a unified numerical scale, providing a consistent foundation for subsequent multi-motion sensor fusion.
[0026] The Min-Max normalization process in this invention is as follows:
[0027] in, This represents the first axis of the accelerometer's z-axis or the first axis of the gyroscope's x- Each sampling point signal, and These represent the minimum and maximum values of the accelerometer's z-axis signal or the gyroscope's x-axis signal, respectively.
[0028] ii: After unifying the signal dimensions, it is necessary to address the signal sequence misalignment problem caused by the unstable sampling interval of the motion sensor. On the Android platform, the sampling interval of motion sensor data fluctuates at the microsecond level, causing the acquisition signals from the accelerometer and gyroscope to be unable to be directly aligned. To alleviate this problem, the millisecond-level timestamps provided by the motion sensor are used to upsample the normalized z-axis and x-axis signals to a sampling rate of 500Hz through linear interpolation, resulting in time-domain aligned z-axis and x-axis signals.
[0029] iii: The time-domain aligned z-axis and x-axis signals still contain a significant amount of non-speech interference, including low-frequency interference caused by human motion concentrated below 20Hz and additive white noise interference inherent in the motion sensor. Therefore, the time-domain aligned z-axis and x-axis signals are filtered and denoised to obtain denoised z-axis and x-axis signals. Specifically: Low-frequency interference is removed by high-pass filtering: an 8th-order Butterworth high-pass filter with a cutoff frequency of 20Hz is used to retain the human speech frequency range of 85~255Hz. Wiener filtering for white noise removal: For inherent white noise with a flat frequency distribution that cannot be eliminated by traditional filters, Wiener filtering is used for adaptive frequency domain denoising.
[0030] iv: To extract signal segments containing valid speech, vibration region segmentation is performed on the denoised z-axis and x-axis signals. Specifically: Calculate the absolute amplitude sequence of the denoised z-axis signal and quantize the signal strength; Two rounds of exponential moving averages with window sizes of 100 and 20 were used to smooth the absolute amplitude sequence, highlighting the envelope of speech activity. Vibration segmentation thresholds are set based on the maximum and minimum values in the smoothed absolute amplitude sequence. Signals above the vibration segmentation threshold are segmented from the denoised z-axis signal as effective vibration signal segments for the accelerometer z-axis, and signals above the vibration segmentation threshold are segmented from the denoised x-axis signal as effective vibration signal segments for the gyroscope x-axis. Simultaneously, to fully capture the speech segment, the boundary of each detected region is extended forward and backward by 50 sampling points.
[0031] Vibration segmentation threshold in this invention The setup process is as follows:
[0032] in, This represents the maximum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients, This represents the minimum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients.
[0033] Accelerometers typically use a spring-suspended sensing mass to measure linear acceleration, resulting in a relatively flat frequency response, but they are susceptible to structural background noise. Gyroscopes, on the other hand, often utilize a vibrating tuning fork structure to measure angular velocity through the Coriolis effect. This structure acts like a mechanical resonator, exhibiting a strong resonance peak at a specific frequency while suppressing other frequencies. These hardware-level differences between accelerometers and gyroscopes lead to differences in their spectral responses. Figure 3 This study visually demonstrates the difference in spectral response between the accelerometer's z-axis and the gyroscope's x-axis: Within the 50–250 Hz range, the accelerometer captures strong energy signals in the 100–110 Hz band, corresponding to the low-order harmonics of male speech in the experiment; while the gyroscope has lower sensitivity in this low-frequency band, but exhibits a significant response peak in the 115–150 Hz band, capable of capturing high-order harmonics that the accelerometer fails to detect. Because the accelerometer lacks high-frequency details and the gyroscope is insensitive to low-frequency fundamental frequencies, a single motion sensor cannot fully cover the harmonic structure of speech. Only by fusing the two can a continuous spectral representation from low to high harmonics be formed, thus fully representing the characteristics of speech.
[0034] Secondly, because accelerometers and gyroscopes are typically driven by independent internal clocks, or by a driver that reads data sequentially, their timestamps are difficult to align perfectly. Figure 2 It can be observed that the sampling times of the accelerometer and gyroscope do not overlap, but rather exhibit continuous and random sampling offsets. This misalignment provides a unique opportunity for time-domain interleaving: if both sensors operate at their nominal sampling rates... With a sampling rate of 500Hz, perfect time alignment would produce redundant time-domain information; however, the actual sampling offset allows gyroscope sampling to fill the gaps between accelerometer samples. By interleaving the two sensor data streams, the sampling density can be effectively doubled, achieving a pseudo-sampling rate of approximately 1000Hz. This "super-resolution" effect is crucial, enabling the Nyquist frequency limit to be increased from 250Hz to 500Hz, thereby capturing high-frequency speech details that would otherwise be aliased.
[0035] Step S2: This invention fuses the effective vibration signal segments of the accelerometer's z-axis and the gyroscope's x-axis using a layered time-frequency fusion strategy to obtain a fundamental frequency sensing domain fused spectrum. This layered time-frequency fusion strategy effectively integrates the advantages of all signal sources, restoring the strong fundamental frequency energy unique to the physical resonance of each motion sensor while preserving the broadband details brought about by time-domain interleaving. It includes the following sub-steps: Step S2.1: As Figure 4 Utilizing the inherent sampling offset of the two motion sensors, it is assumed that both operate at the nominal sampling rate. =500 Hz sampling, accelerometer sampling and gyroscope sampling Usually present The time interval is calculated. Based on the timestamp, the effective vibration signal segments of the accelerometer z-axis and the gyroscope x-axis are interleaved and fused in the time domain to obtain a time-domain interleaved signal with an effective sampling rate of 1000Hz. Since the accelerometer and gyroscope signals have been normalized and scaled to the same amplitude range in the preprocessing stage, the signals achieve a smooth transition at the intersection, effectively avoiding pseudo-high-frequency noise that may be caused by abrupt amplitude changes. In addition, the sampling rate increase brought about by time-domain interleaving and fusion expands the effective spectrum. According to the Nyquist theorem, additional spectral information in the range of 250~500Hz can be captured, providing a more complete characterization of vibration characteristics.
[0036] The process of obtaining the time-domain interleaved signal in this invention is as follows:
[0037] in, express The time-domain interleaved signal at that moment, express The effective vibration signal of the accelerometer z-axis at time t is express The effective vibration signal of the gyroscope's x-axis at that moment. express The periodic sampling pulse function of the accelerometer at a given time. , Indicates the sampling point index. Indicates the equivalent sampling interval. =1 / 1000 second, Represents the unit impulse function. express The periodic sampling pulse function of the gyroscope at that time. .
[0038] Because the gyroscope pulse sequence is delayed by one sampling period compared to the accelerometer pulse sequence. To analyze the frequency domain characteristics caused by this fixed time delay, a fast Fourier transform is performed on the time-interleaved signal: ,in, Represents frequency The effective vibration frequency domain signal of the z-axis of the lower accelerometer Represents frequency Effective vibration frequency domain signal of the x-axis of the lower gyroscope. This represents the spectrum corresponding to the time-domain sampled pulse sequence of the accelerometer. , Indicates the harmonic index, This represents the spectrum corresponding to the time-domain sampling pulse sequence of the gyroscope. . and It is a comb-like spectrum, with intervals consisting of... While time-domain interleaving and fusion expands the effective bandwidth, it also presents key challenges in the frequency domain: comb spectrum analysis shows that sampling offset... A phase rotation term was introduced. In the low-frequency speech band of 20–250 Hz, this phase shift often causes destructive interference between accelerometer and gyroscope signals. Therefore, simple time-domain interleaved signals often exhibit significant energy attenuation at the fundamental frequency because the signals from the two motion sensors may be out of phase at this point.
[0039] Step S2.2: Perform frequency domain characteristic analysis on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis respectively. Based on the fundamental frequency sensing domain fusion strategy, obtain the fundamental frequency sensing domain fused spectrum, including the following sub-steps: Step S2.2.1: Perform short-time Fourier transform on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis, respectively, to obtain the time-domain interleaved spectrum signal, the accelerometer z-axis vibration spectrum signal, and the gyroscope x-axis vibration spectrum signal. The window length of the short-time Fourier transform is 128 sampling points, and the step size is 120 sampling points.
[0040] Step S2.2.2: Within the human speech frequency range of 85~255Hz, calculate the total harmonic energy of the accelerometer and the total harmonic energy of the gyroscope at each frequency using the accelerometer's z-axis vibration spectrum signal and the gyroscope's x-axis vibration spectrum signal, respectively, thereby estimating the fundamental frequency of the accelerometer's z-axis signal. and the fundamental frequency of the gyroscope x-axis signal .
[0041] The calculation process for the total harmonic energy of the accelerometer and the total harmonic energy of the gyroscope at each frequency in this invention is as follows:
[0042] in, The cutoff frequency is 500Hz. Indicates frequency, This indicates the number of frequency points acquired by the short-time Fourier transform. This indicates rounding down. Harmonic index The fundamental frequency of the accelerometer's z-axis vibration spectrum signal or the gyroscope's x-axis vibration spectrum signal is determined by the following formula: .
[0043] Step S2.2.3: To ensure robustness against phase cancellation, a max-pooling strategy is used for non-fundamental frequencies, rather than blindly relying on interleaved signals: If the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the accelerometer z-axis signal, record the amplitude of the accelerometer z-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the gyroscope x-axis signal, record the amplitude of the gyroscope x-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is greater than 250Hz, record the amplitude of the time-domain interleaved spectrum signal, which contains effective super Nyquist information; otherwise, record the maximum value of the amplitude of the accelerometer z-axis vibration spectrum signal, the amplitude of the gyroscope x-axis vibration spectrum signal, or the amplitude of the time-domain interleaved spectrum signal at the corresponding frequency point to avoid spectral energy attenuation caused by phase cancellation.
[0044] Step S2.2.4: Sort the amplitudes of the recorded accelerometer z-axis vibration spectrum signal, the gyroscope x-axis vibration spectrum signal, and the time-domain interleaved spectrum signal according to the frequency points collected to obtain the fundamental frequency sensing domain fused spectrum.
[0045] Figure 5The image shows the fused spectrum envelope of the accelerometer and gyroscope in the baseband sensing domain of the Redmi K60 phone. It can be seen that a single motion sensor can only capture fragmented spectral features, while the fused spectrum in the baseband sensing domain enhances the energy of the 0~250Hz low frequency band by combining the resonance peaks of the two motion sensors. In addition, through time-domain interleaving fusion, the details of the 250~500Hz high frequency band are expanded.
[0046] Step S3: Construct a CSMambaGAN speech reconstruction model based on generative adversarial networks and integrating the CSMamba structure. Perform Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate the speech reconstruction signal.
[0047] like Figure 6 While the fused spectrum in the fundamental frequency sensing domain achieves bandwidth expansion, it still contains residual noise and spectral artifacts. To map this noisy spectrum into high-fidelity speech, the CSMambaGAN speech reconstruction model in this invention, which incorporates the CSMamba architecture, includes a generator and a HiFiGAN vocoder. The generator, based on U-Net, learns the mapping from the fused spectrum in the fundamental frequency sensing domain to the target Mel spectrum. The HiFiGAN vocoder is used to generate the reconstructed speech signal based on the Mel spectrum.
[0048] in: The generator consists of an image embedding layer, a CSMamba bottleneck layer, and an upsampled output layer; To introduce randomness and enhance generation diversity, a random noise vector is spatially copied to the same dimension as the fused spectrum y of the fundamental frequency sensing domain and spliced along the channel axis. This is used as the input to the image embedding layer in the generator. The image embedding layer is used to project the fused spectrum of the fundamental frequency sensing domain with introduced random noise vectors onto a low-resolution feature space, thereby improving feature representation while reducing computational complexity. The CSMamba bottleneck layer employs a symmetrical encoder-decoder structure. The encoder layer extracts deep features stepwise from the spectral feature map of the low-resolution feature space through multi-level downsampling. The latent layer is used to compress the deep features. The decoder layer fuses the multi-scale features of the encoder through skip connections and combines symmetrical upsampling to gradually restore the feature resolution of the compressed deep features. The upsampled output layer is used to restore the feature map with restored feature resolution to the target size by passing a one-dimensional convolution followed by two deconvolution layers, generating the Mel spectrum.
[0049] In one technical solution of the present invention, each downsampling stage includes two cascaded CSMamba blocks and a block merging layer. Each CSMamba block includes a global branch and a local branch, wherein the global branch utilizes two-dimensional selective scanning 2D-SSM to sample spectral features. Figure 4The global context in each direction is modeled to obtain locally enhanced spectral feature maps. Local branches employ cascaded channel and spatial attention mechanisms to extract spectral feature maps highlighting the dominant speech frequency bands. A block merging layer concatenates the locally enhanced spectral feature maps and the spectral feature maps highlighting the dominant speech frequency bands, resulting in a concatenated spectral feature map. This allows the model to simultaneously capture fine-grained acoustic texture and long-range semantic dependencies of speech. In this invention, the latent layer uses two cascaded CSMamba blocks.
[0050] In one technical solution of the present invention, the constructed CSMambaGAN speech reconstruction model is trained by the PatchGAN discriminator. Specifically, the PatchGAN discriminator distinguishes between the real Mel spectrum and the generated Mel spectrum for authenticity until the spectral consistency and adversarial fusion loss of the generator and the adversarial loss of the PatchGAN discriminator converge, thereby completing the training of the CSMambaGAN speech reconstruction model and improving the perceptual quality while ensuring spectral accuracy.
[0051] Spectral consistency and adversarial fusion loss of the generator in this invention ,in, This represents the spectral consistency loss of the generator. , Represents the true Mel spectrum. This represents the generated Mel spectrum. express Norm, Describe the Frobenius norm. express The weighting coefficients, This represents the adversarial loss of the generator. , This indicates the fusion of spectrum in the baseband sensing domain. Introducing random noise vectors noise samples, Represents a generator. Indicates the discriminator, This represents the expected value of the noisy sample. express Weighting coefficients; Adversarial loss of PatchGAN discriminator ,in, The expected value of a sample without introducing random noise vectors.
[0052] Figure 7This diagram illustrates the speech recovery performance of the present invention in a single-speaker scenario. Due to differences in gender, language background, and pronunciation habits, the speech characteristics of different speakers vary significantly. To analyze the influence of these factors, eight representative public figures were selected as research subjects, such as... Figure 7 In (a), the average Mel-frequency cepstral distortion (MCD) of the eight subjects was below 4, significantly lower than the speech intelligibility threshold of 8.0, indicating that the hierarchical time-frequency fusion strategy of this invention successfully recovered the fine-grained spectral structure of speech; Figure 7 In (b), the average Short-Time Objective Intelligibility (STOI) of the eight subjects was 0.59, indicating that the speech reconstruction signal generated by this invention preserved clear speech content and speaker identity; Figure 7 In (c), the speech-to-reverberation modulation energy ratio (SRMR) of the eight subjects consistently exceeded 4.0, indicating that the speech restoration enhancement method of this invention effectively suppressed motion sensor noise and motion artifacts.
[0053] Figure 8 This diagram illustrates the generalization performance of speech recovery in scenarios with unknown speakers. The results show that the CSMambaGAN speech reconstruction model in this invention has good generalization ability, with the average MCD value remaining below 5.0 and highly concentrated. Only the data variance of female speaker number 16 is slightly higher, which is physically reasonable: the fundamental frequency of female speech is usually over 250Hz, exceeding the Nyquist limit of 500Hz sampling rate for a single sensor. Although the hierarchical time-frequency fusion strategy expands the bandwidth, the energy attenuation in the high-frequency band is naturally more severe due to the weak structural coupling.
[0054] Figure 9 This diagram illustrates speech recovery performance under various real-world environmental constraints. Figure 9 (a) Sensor data was collected at playback volumes of 60%, 80%, and 100%, and the CSMambaGAN speech reconstruction model was trained using data from the 100% playback volume. The model was then tested at all playback volumes. The results show that even when the playback volume is reduced to 60%, the enhanced speech recovery performance of this invention remains stable, with MCD ≈ 3.8. Figure 9 In (b), three common placement scenarios were evaluated: static desktop, handheld still, and handheld walking. Sensor data collected during desktop stillness was used to train the CSMambaGAN speech reconstruction model, which was then tested in all scenarios. The MCD value remained consistently below 4 in all scenarios, indicating that the enhanced speech recovery method of this invention has high robustness to mobile device placement and user movement. Figure 9(c) In the Android system, the performance at sampling rates of 200Hz, 300Hz, 400Hz and 500Hz was evaluated. To ensure the validity of the experiment, the CSMambaGAN speech reconstruction model was trained separately for each sampling rate. The results showed that the MCD value naturally improved with the increase of the sampling rate, but even at the low sampling rate of 200Hz, the present invention can still maintain a stable MCD value of about 3.5.
[0055] Figure 10 This diagram illustrates the generalization performance of the speech recovery method of the present invention on different mobile devices. The cross-device generalization of the enhanced speech recovery method of the present invention was evaluated on three commercial smartphones. The results are as follows: When trained on one device such as Redmi K60 and tested on other devices such as Xiaomi 13, the MCD value only showed a slight decrease, indicating that the normalization and baseband sensing fusion strategy of the present invention effectively mitigated the impact of hardware heterogeneity.
[0056] Figure 11 This diagram illustrates the performance comparison between single-sensor and multi-sensor fusion methods for speech recovery. It shows that the performance of the multi-motion sensor fusion method in this invention consistently outperforms the single-sensor baseline method, and the fusion of complementary perceptual modalities effectively compensates for the hardware limitations of a single sensor. Figure 11 In (a), for user 1, the present invention improves the STOI value by 13%, indicating that the present invention effectively improves speech intelligibility; as Figure 11 In (b) of this invention, for user 2, the MCD value is reduced by 13.6% and the spectral fidelity is significantly improved.
[0057] Figure 12 A schematic diagram comparing the speech recovery performance of different fusion strategies is shown. The hierarchical time-frequency fusion strategy of this invention is compared with two mainstream fusion strategies: (1) Time-domain splicing InertiEAR: ignoring phase alignment; (2) Multi-axis interleaving SwanTrumpet: focusing on bandwidth expansion and ignoring spectral energy attenuation. To ensure fairness, all methods use the same CSMambaGAN speech reconstruction model and dataset. The results show that: Time-domain splicing InertiEAR often causes destructive interference at the fundamental frequency due to phase mismatch; Although multi-axis interleaving SwanTrumpet achieves bandwidth expansion, it does not consider the phase relationship between different axes, resulting in spectral energy attenuation; The hierarchical time-frequency fusion strategy proposed in this invention effectively solves this contradiction by explicitly processing the fundamental frequency resonance in the frequency domain and achieving bandwidth expansion in the time domain, achieving the lowest MCD value and more stable speech reconstruction.
[0058] Figure 13This diagram illustrates the performance comparison of different speech reconstruction models. To evaluate the performance of the CSMambaGAN speech reconstruction model in this invention, it is compared with the encoder-decoder AccelEve and the conditional generative adversarial network AccEar architecture. The results show that the CSMamba block in the CSMambaGAN speech reconstruction model of this invention can model the long-range temporal dependencies of speech with linear complexity. This is crucial for capturing prosodic features in high-resolution spectrograms, thus achieving an optimal MCD value of 3.05, which is 1.55 lower than the MCD of the encoder-decoder AccelEve and 1.23 lower than the MCD of the conditional generative adversarial network AccEar architecture.
[0059] Figure 14 This diagram illustrates the performance comparison of the present invention and the AccEar conditional generative adversarial network architecture at different sampling rates. At 500Hz, both methods perform well, but the present invention has a lower MCD due to the fusion of information from two motion sensors. At 200Hz, the performance of the AccEar conditional generative adversarial network architecture drops sharply because the 200Hz sampling rate of a single motion sensor cannot effectively capture the fundamental frequency range of the speech. The present invention, however, maintains a low MCD even at a 200Hz sampling rate.
[0060] like Figure 15 In one technical solution of the present invention, an enhanced speech recovery system based on multimodal motion sensor fusion is also provided, including: a vibration signal acquisition module, a vibration signal preprocessing module, a hierarchical time-frequency fusion module, and a speech reconstruction module; The vibration signal acquisition module is used to acquire the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio; The vibration signal preprocessing module is used to preprocess the acquired z-axis and x-axis signals respectively to obtain the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis. The hierarchical time-frequency fusion module is used to fuse the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis using a hierarchical time-frequency fusion strategy to obtain the fundamental frequency sensing domain fused spectrum. The speech reconstruction module employs a CSMambaGAN speech reconstruction model based on generative adversarial networks and incorporating the CSMamba structure. This model is used to perform Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate the reconstructed speech signal.
[0061] In one embodiment of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the enhanced speech recovery method based on multimodal motion sensor fusion of the present invention.
[0062] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the enhanced speech recovery method based on multimodal motion sensor fusion of the present invention.
[0063] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0065] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. An enhanced speech recovery method based on multi-modal motion sensor fusion, characterized in that, Includes the following steps: Step S1: Collect the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio. Preprocess the collected z-axis signal and x-axis signal respectively to obtain the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis. Step S2: The effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis are fused using a hierarchical time-frequency fusion strategy to obtain the fundamental frequency sensing domain fused spectrum; including the following sub-steps: Step S2.1: Based on the timestamp, perform time-domain interleaving and fusion of the effective vibration signal segments of the accelerometer z-axis and the gyroscope x-axis to obtain the time-domain interleaved signal: in, express The time-domain interleaved signal at that moment, express The effective vibration signal of the accelerometer z-axis at time t is express The effective vibration signal of the gyroscope's x-axis at that moment. express The periodic sampling pulse function of the accelerometer at a given time. , Indicates the sampling point index. Indicates the equivalent sampling interval. Represents the unit impulse function. express The periodic sampling pulse function of the gyroscope at that moment. ; Step S2.2: Perform frequency domain characteristic analysis on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis respectively. Based on the fundamental frequency sensing domain fusion strategy, obtain the fundamental frequency sensing domain fused spectrum; including the following sub-steps: Step S2.2.1: Perform short-time Fourier transform on the time-domain interleaved signal, the effective vibration signal segment of the accelerometer z-axis, and the effective vibration signal segment of the gyroscope x-axis, respectively, to obtain the time-domain interleaved spectrum signal, the accelerometer z-axis vibration spectrum signal, and the gyroscope x-axis vibration spectrum signal; Step S2.2.2: Within the human speech frequency range of 85~255Hz, calculate the total harmonic energy of the accelerometer and the total harmonic energy of the gyroscope at each frequency using the accelerometer z-axis vibration spectrum signal and the gyroscope x-axis vibration spectrum signal, respectively, and estimate the fundamental frequency of the accelerometer z-axis signal and the fundamental frequency of the gyroscope x-axis signal; Step S2.2.3: If the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the accelerometer z-axis signal, record the amplitude of the accelerometer z-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is aligned with the fundamental frequency of the gyroscope x-axis signal, record the amplitude of the gyroscope x-axis vibration spectrum signal at the corresponding frequency point; if the frequency point acquired by the short-time Fourier transform is greater than 250Hz, record the amplitude of the time-domain interleaved spectrum signal; otherwise, record the maximum value of the amplitude of the accelerometer z-axis vibration spectrum signal, the amplitude of the gyroscope x-axis vibration spectrum signal, or the amplitude of the time-domain interleaved spectrum signal at the corresponding frequency point. Step S2.2.4: Sort the amplitudes of the recorded accelerometer z-axis vibration spectrum signal, the gyroscope x-axis vibration spectrum signal, and the time-domain interleaved spectrum signal according to the frequency points collected to obtain the fundamental frequency sensing domain fused spectrum; Step S3: Construct a CSMambaGAN speech reconstruction model based on generative adversarial networks and integrating the CSMamba structure. Perform Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate the speech reconstruction signal.
2. The enhanced speech recovery method based on multimodal motion sensor fusion according to claim 1, characterized in that, The preprocessing of the acquired z-axis and x-axis signals in step S1 is as follows: i: Perform Min-Max normalization on the acquired z-axis and x-axis signals respectively, and map them to... interval; ii: Upsample the normalized z-axis and x-axis signals to a sampling rate of 500Hz to obtain time-aligned z-axis and x-axis signals; iii: Filter and denoise the time-domain aligned z-axis and x-axis signals to obtain denoised z-axis and x-axis signals; iv: Calculate the absolute amplitude sequence of the denoised z-axis signal, and set the vibration segmentation threshold according to the maximum and minimum values in the absolute amplitude sequence of the denoised z-axis signal. Segment the signal above the vibration segmentation threshold from the denoised z-axis signal as the effective vibration signal segment of the accelerometer z-axis, and segment the signal above the vibration segmentation threshold from the denoised x-axis signal as the effective vibration signal segment of the gyroscope x-axis.
3. The enhanced speech recovery method based on multimodal motion sensor fusion according to claim 2, characterized in that, The vibration segmentation threshold The setup process is as follows: in, This represents the maximum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients, This represents the minimum value in the absolute amplitude sequence of the denoised z-axis signal. express The weighting coefficients.
4. The enhanced speech recovery method based on multimodal motion sensor fusion according to claim 1, characterized in that, The CSMambaGAN speech reconstruction model that integrates the CSMamba structure in step S3 includes: a generator for mapping the fundamental frequency sensing domain fused spectrum to a Mel spectrum and a HiFiGAN vocoder for generating a speech reconstruction signal based on the Mel spectrum. in: The generator includes an image embedding layer, a CSMamba bottleneck layer, and an upsampled output layer; The image embedding layer is used to project the fundamental frequency sensing domain fused spectrum, which introduces random noise vectors, onto a low-resolution feature space. The CSMamba bottleneck layer adopts a symmetrical encoder-decoder structure, wherein the encoder layer extracts deep features step by step from the spectral feature map of the low-resolution feature space through multi-level downsampling; the latent layer is used to compress the deep features; the decoder layer fuses the multi-scale features of the encoder through skip connections and combines symmetrical upsampling to gradually restore the feature resolution of the compressed deep features. The upsampling output layer is used to restore the feature map with restored feature resolution to the target size, generating a Mel spectrum.
5. The enhanced speech recovery method based on multimodal motion sensor fusion according to claim 4, characterized in that, Each downsampling level consists of two cascaded CSMamba blocks and a block merging layer; Each CSMamba block contains a global branch and a local branch. The global branch uses two-dimensional selective scanning to model the global context of the spectral feature map in four directions to obtain a spectral feature map with enhanced local features. The local branch uses a cascaded channel and spatial attention mechanism to extract the spectral feature map that highlights the dominant speech frequency band from the spectral feature map. The block merging layer is used to stitch together the spectral feature map with enhanced local features and the spectral feature map highlighting the dominant speech frequency band to obtain a stitched spectral feature map.
6. The enhanced speech recovery method based on multimodal motion sensor fusion according to claim 4, characterized in that, The CSMambaGAN speech reconstruction model with the CSMamba structure is trained by the PatchGAN discriminator until the spectral consistency of the generator, the adversarial fusion loss, and the adversarial loss of the PatchGAN discriminator all converge, thus completing the training of the CSMambaGAN speech reconstruction model. The generator's spectral consistency and adversarial fusion loss ,in, This represents the spectral consistency loss of the generator. , Represents the true Mel spectrum. This represents the generated Mel spectrum. express Norm, Describe the Frobenius norm. express The weighting coefficients, This represents the adversarial loss of the generator. , This indicates the fusion of spectrum in the baseband sensing domain. Introducing random noise vectors noise samples, Represents a generator. Indicates the discriminator, This represents the expected value of the noise sample. express Weighting coefficients; The adversarial loss of the PatchGAN discriminator ,in, This represents the expected value of a sample vector without introduced random noise.
7. A system applying the enhanced speech recovery method based on multimodal motion sensor fusion as described in claim 1, characterized in that, include: Vibration signal acquisition module, vibration signal preprocessing module, hierarchical time-frequency fusion module, and speech reconstruction module; The vibration signal acquisition module is used to acquire the z-axis signal of the accelerometer and the x-axis signal of the gyroscope when the built-in speaker of the mobile device plays audio; The vibration signal preprocessing module is used to preprocess the acquired z-axis signal and x-axis signal respectively to obtain the effective vibration signal segment of the accelerometer z-axis and the effective vibration signal segment of the gyroscope x-axis; The hierarchical time-frequency fusion module is used to fuse the effective vibration signal segments of the accelerometer z-axis and the effective vibration signal segments of the gyroscope x-axis using a hierarchical time-frequency fusion strategy to obtain the fundamental frequency sensing domain fused spectrum. The speech reconstruction module employs a CSMambaGAN speech reconstruction model based on a generative adversarial network that integrates the CSMamba structure. It performs Mel-spectrum mapping on the fused spectrum of the fundamental frequency sensing domain to generate the speech reconstruction signal.