Voice collection method, device and storage medium

Abstract

The application discloses a voice collection method and device and a storage medium, relates to the technical field of voice interaction, and comprises the following steps: acquiring voice signals collected by at least one microphone channel; acquiring vibration reference signals collected by at least one vibration reference channel; determining a coupling relationship between the voice signals and the vibration reference signals; correcting a collection strategy of the voice signals according to the coupling relationship; and obtaining a collection result of the voice signals according to the corrected collection strategy. The application can reduce the influence of device self-noise on voice collection quality, make the collection result more accurately reflect target voice, effectively distinguish external environmental noise from device self-structure vibration noise, and improve the collection quality of voice signals.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and in particular to a voice acquisition method, device and storage medium. Background Technology

[0002] With the widespread adoption of voice interaction technology in various devices, users have raised higher requirements for the reliability of voice acquisition under complex operating conditions. When performing voice acquisition, devices with mobility or moving parts (such as robots, smart home devices, etc.) generate self-noise due to their own mechanical movement, structural vibration, or component operation. This self-noise can be coupled to the microphone channel through the structural conduction path and superimposed on the target voice signal, resulting in a decrease in voice acquisition quality.

[0003] In existing voice acquisition solutions, general noise reduction techniques mainly rely on the statistical characteristics of the audio signal itself or the spatial differences of multiple microphones to suppress noise, making it difficult to distinguish whether the noise comes from the external environment or the vibration of the equipment's own structure; echo cancellation technology mainly targets the air propagation path of the sound played by the speaker, and cannot effectively deal with mechanical noise transmitted through the equipment structure; and simply reducing the vibration level of the hardware is also difficult to eliminate the self-noise during equipment operation. Summary of the Invention

[0004] The main purpose of this application is to provide a voice acquisition method, device and storage medium, which aims to solve the technical problem that the voice acquisition quality is poor due to the influence of device self-noise.

[0005] To achieve the above objectives, this application proposes a voice acquisition method, which includes: Acquire the voice signal from at least one microphone channel; Acquire vibration reference signals from at least one vibration reference channel, wherein the vibration reference signals are used to characterize the self-noise of the equipment; Determine the coupling relationship between the speech signal and the vibration reference signal; Based on the coupling relationship, the acquisition strategy for the speech signal is modified, and the acquisition result of the speech signal is obtained according to the modified acquisition strategy.

[0006] In one embodiment, the step of acquiring the vibration reference signal collected by at least one vibration sensor channel includes: Acquire candidate reference signals, wherein the candidate reference signals include at least one physical vibration sensing signal and / or at least one device status signal; The candidate reference signal is evaluated for reference validity to determine whether the candidate reference signal is related to the device's self-noise. Candidate reference signals that are associated with the self-noise of the device are identified as vibration reference signals.

[0007] In one embodiment, before the step of determining the coupling relationship between the speech signal and the vibration reference signal, the method further includes: The speech signal and the vibration reference signal are time-aligned, and the step of determining the coupling relationship between the speech signal and the vibration reference signal is performed based on the time-aligned speech signal and vibration reference signal.

[0008] In one embodiment, the step of time-aligning the speech signal and the vibration reference signal includes: The time alignment offset is determined based on the timestamp of the voice signal, the timestamp of the vibration reference signal, and the preset vibration transmission delay. Based on the time alignment offset, the speech signal and the vibration reference signal are synchronized by a time window to obtain the time-aligned speech signal and vibration reference signal.

[0009] In one embodiment, the coupling relationship includes a self-noise coupling component and / or a self-noise confidence level, and the step of determining the coupling relationship between the speech signal and the vibration reference signal includes: Within the synchronized time window, the coupling characteristics of the speech signal and the vibration reference signal in multiple frequency bands are determined, wherein the coupling characteristics include coherence, correlation and / or synchronization strength. When the coupling characteristics meet the preset reference validity conditions, the coupling parameters of the microphone channel and each frequency band are determined, wherein the coupling parameters include coupling weights and / or transfer functions; Based on the coupling parameters, the self-noise coupling component and / or the self-noise confidence level are generated.

[0010] In one embodiment, the step of determining the coupling characteristics of the speech signal and the vibration reference signal across multiple frequency bands within a synchronized time window includes: Within the time window, the speech signal and the vibration reference signal are processed by framing or windowing respectively to obtain the speech sub-signal and vibration sub-signal within each time sub-window; Each of the speech sub-signals and each of the vibration sub-signals are subjected to frequency domain transformation to obtain the frequency domain components corresponding to each frequency band; The coupling characteristics are determined based on each of the frequency domain components.

[0011] In one embodiment, the step of modifying the acquisition strategy of the speech signal according to the coupling relationship includes: Based on the coupling relationship, at least one of self-noise coupling component cancellation, frequency band suppression, and beamforming weight adjustment is performed on the speech signal to obtain a compensated speech signal; and / or, Based on the coupling relationship, at least one of the following is corrected: the voice activity detection threshold, the voice wake-up confidence, the azimuth offset of the sound source localization result, and the near-far field judgment result.

[0012] In one embodiment, the step of obtaining the acquisition result of the speech signal according to the modified acquisition strategy includes: Based on the compensated speech signal, the target speech is determined, and / or, based on at least one of the corrected speech activity detection threshold, speech wake-up confidence, directional offset of sound source localization result, and near-far field judgment result, the speech recognition confidence is determined; The acquisition result includes at least one of the target speech, the corrected speech activity detection threshold, the speech wake-up confidence, the azimuth offset of the sound source localization result, the near and far field judgment result, and the speech recognition confidence.

[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the voice acquisition method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the voice acquisition method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application acquires a speech signal from at least one microphone channel and a vibration reference signal from at least one vibration reference channel to characterize the device's self-noise. It then determines the coupling relationship between the speech signal and the vibration reference signal, and uses this coupling relationship to modify the speech signal acquisition strategy, obtaining a modified acquisition result. Since the vibration reference signal can characterize the self-noise generated by the device's mechanical motion, structural vibration, or component operation, it carries the structural transmission characteristics of the device's self-noise. When the coupling relationship between the speech signal and the vibration reference signal is determined, the self-noise components mixed into the speech signal due to the structural transmission path can be identified. Modifying the speech signal acquisition strategy based on this coupling relationship allows the acquisition strategy to no longer rely solely on the statistical characteristics of the microphone audio signal itself, but to actively intervene in the interference of device self-noise on the speech signal, thereby effectively suppressing the interference components caused by device self-noise in the acquisition result. This reduces the impact of device self-noise on the speech acquisition quality, makes the acquisition result more accurately reflect the target speech, effectively distinguishes between external environmental noise and the device's own structural vibration noise, and improves the quality of speech signal acquisition. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating Embodiment 1 of the voice acquisition method of this application; Figure 2 This is a schematic diagram of the vibration reference signal source for the voice acquisition method provided in Embodiment 1 of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the voice acquisition method of this application; Figure 4 This is a schematic diagram of the system architecture of the voice acquisition method provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the coupling estimation process of the speech acquisition method provided in Embodiment 2 of this application; Figure 6 This is a schematic diagram of the compensation and correction process for the voice acquisition method provided in Embodiment 2 of this application; Figure 7This is a schematic diagram illustrating an application scenario of the voice acquisition method provided in Embodiment 2 of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice acquisition method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is: to acquire a voice signal collected by at least one microphone channel; to acquire a vibration reference signal collected by at least one vibration reference channel, wherein the vibration reference signal is used to characterize the device self-noise; to determine the coupling relationship between the voice signal and the vibration reference signal; to modify the acquisition strategy of the voice signal according to the coupling relationship; and to obtain the acquisition result of the voice signal according to the modified acquisition strategy.

[0023] Because existing voice acquisition solutions rely mainly on the statistical characteristics of the audio signal itself or the spatial differences of multiple microphones to suppress noise, it is difficult to distinguish whether the noise comes from the external environment or the vibration of the equipment itself; echo cancellation technology mainly targets the air propagation path of the sound played by the speaker, and cannot effectively deal with mechanical noise transmitted through the equipment structure; and simply reducing the vibration level of the hardware is also difficult to eliminate the self-noise of the equipment during operation.

[0024] This application provides a solution whereby a vibration reference signal, which characterizes the self-noise generated by the mechanical motion, structural vibration, or component operation of a device, carries the structural transmission characteristics of the device's self-noise. When the coupling relationship between the speech signal and the vibration reference signal is determined, the self-noise components mixed into the speech signal due to the structural transmission path can be identified. Based on this coupling relationship, the speech signal acquisition strategy is modified so that it no longer relies solely on the statistical characteristics of the microphone audio signal itself, but actively intervenes against the interference of device self-noise on the speech signal, thereby effectively suppressing the interference components caused by device self-noise in the acquisition results. This reduces the impact of device self-noise on the speech acquisition quality, making the acquisition results more accurately reflect the target speech, effectively distinguishing between external environmental noise and the device's own structural vibration noise, and improving the acquisition quality of the speech signal.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses an electronic device as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, the embodiments of this application provide a voice acquisition method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the voice acquisition method of this application.

[0027] In this embodiment, the voice acquisition method includes steps S10 to S40: Step S10: Acquire the voice signal collected by at least one microphone channel; It should be noted that the voice signal is the acoustic signal containing the target voice components that the device acquires through at least one microphone channel during the voice acquisition task.

[0028] It is understandable that the speech signal is not the pure target speech, but a mixture of the target speech and various interference components, which may include external environmental noise and the equipment's own noise. When the equipment itself is in mechanical motion, structural vibration, or component operation, the speech signal will inevitably be mixed with self-noise components coupled in through the structural conduction path, thereby reducing the quality of the speech signal.

[0029] Step S20: Obtain vibration reference signals acquired from at least one vibration reference channel, wherein the vibration reference signals are used to characterize the self-noise of the equipment; It should be noted that the vibration reference signal is a signal used to characterize the self-noise of the equipment, acquired through at least one vibration reference channel. The vibration reference signal and the speech signal originate from different acquisition channels, and the acquisition focuses on physical quantities related to the equipment's mechanical motion, structural vibration, or component operation. The vibration reference signal can reflect the characteristics of the equipment's self-noise along the structural transmission path, such as the amplitude, frequency, and phase of the vibration, thus providing a reference for identifying the self-noise components in the speech signal.

[0030] Equipment self-noise refers to the noise generated by equipment during its own operation, such as movement, rotation, joint movements, fan operation, or speaker playback, due to its mechanical motion, structural vibration, or component operation. The conduction path of equipment self-noise includes not only airborne paths but also solid-borne paths through the equipment's structural components. Therefore, equipment self-noise can be directly coupled to the microphone channel through the equipment's casing, support, or internal structural components, superimposed on the target speech, and thus contaminate the speech acquisition results.

[0031] Reference Figure 2 In this diagram, the robot body serves as the equipment carrier, and its various parts and components can all be used as sources of vibration reference signals: an IMU (Inertial Measurement Unit) (such as an accelerometer or gyroscope) can be installed at the head or gimbal to sense vibrations caused by rotation and attitude changes; a piezoelectric or contact vibration pickup can be installed on the microphone bracket to directly sense structural vibrations transmitted to the vicinity of the microphone channel; strain gauges, force sensors, or structural acoustic sensors can be installed at the joints or shell to sense mechanical noise caused by joint movements and shell resonance; a vibration sensor can be installed at the speaker bracket to sense structural coupling vibrations caused by playback; in addition, actuator status signals (such as motor current, encoder, and control commands) and fan and chassis status signals (such as rotational speed, speed, and braking status) can also serve as signal sources that indirectly characterize the equipment's self-noise. After multi-source fusion processing, these various signals can form a virtual vibration reference quantity covering different vibration sources and different structural parts, providing a comprehensive reference for subsequently determining the coupling relationship between the voice signal and the vibration reference signal.

[0032] Step S30: Determine the coupling relationship between the voice signal and the vibration reference signal; It should be noted that coupling relationship refers to the correlation between the speech signal and the vibration reference signal in the time domain or frequency domain.

[0033] When device self-noise couples to the microphone channel through the structural conduction path, the self-noise components in the speech signal and the vibration reference signal will exhibit consistency or correlation in dimensions such as vibration amplitude, frequency, or phase. This consistency or correlation is the coupling relationship. The purpose of determining the coupling relationship is to quantify the extent to which the self-noise components in the speech signal originate from the device self-noise represented by the vibration reference signal, thereby providing a basis for subsequent acquisition strategy correction.

[0034] Step S40: Based on the coupling relationship, the acquisition strategy of the speech signal is modified, and the acquisition result of the speech signal is obtained according to the modified acquisition strategy.

[0035] It should be noted that the acquisition strategy refers to the processing rules or methods adopted by the device in processing the speech signal to obtain the acquisition results. The acquisition strategy covers multiple stages of post-acquisition processing of the speech signal, including but not limited to: the signal processing methods for the speech signal itself, and the judgment criteria for tasks such as speech activity detection, voice wake-up, sound source localization, near and far field judgment, and speech recognition based on the speech signal.

[0036] Once the coupling relationship is determined, the device can make targeted modifications to the acquisition strategy based on the coupling relationship in order to suppress the impact of device self-noise on the speech signal and subsequent speech processing tasks.

[0037] The voice signal acquisition result is the output information obtained by the device after processing the voice signal according to the corrected acquisition strategy. The acquisition result can be reflected as the final output data of the voice acquisition task that can be used or presented later, such as the target voice after noise compensation, the corrected sound source direction or near-far field judgment result, and the voice recognition confidence level.

[0038] The acquisition results of the voice signal are different from the voice signal itself. The acquisition results are output data that have been processed and corrected by the acquisition strategy, and can more accurately reflect the target voice and its related attribute information. They can be used by the device to perform voice interaction or task decision-making in the future.

[0039] Understandably, existing general noise reduction technologies rely on the statistical characteristics of the audio signal itself or the spatial differences of multiple microphones to suppress noise, making it difficult to distinguish whether the noise comes from the external environment or the vibration of the device's own structure. Echo cancellation technology mainly targets the air propagation path of the sound played by the speaker and cannot effectively deal with mechanical noise transmitted through the device structure. Simply reducing the level of hardware vibration is also difficult to eliminate the residual self-noise during device operation. These defects lead to a decrease in the quality of voice acquisition in scenarios where self-noise exists.

[0040] Therefore, since device self-noise can couple to the microphone channel and superimpose with the target speech through the structural conduction path, this embodiment obtains the speech signal acquired by at least one microphone channel and the vibration reference signal acquired by at least one vibration reference channel to characterize the device self-noise, determines the coupling relationship between the speech signal and the vibration reference signal, and uses this coupling relationship to correct the speech signal acquisition strategy. Based on the corrected acquisition strategy, the acquisition result of the speech signal is obtained. This avoids the problem that the acquisition strategy cannot actively intervene in the device self-noise because it cannot identify the self-noise component mixed in by the structural conduction path in the speech signal. This reduces the impact of device self-noise on the speech acquisition quality and makes the acquisition result more accurately reflect the target speech.

[0041] For example, when performing a voice acquisition task, the robotic device first acquires a voice signal containing the target voice and possible device self-noise through at least one microphone channel, and simultaneously acquires a vibration reference signal characterizing the device self-noise through at least one vibration reference channel. This vibration reference signal reflects the vibration characteristics of the device itself caused by mechanical movement, structural vibration, or component operation. Next, the device analyzes in the time or frequency domain whether there is a correlation between the voice signal and the vibration reference signal, that is, determines the coupling relationship between the two, thereby identifying which components in the voice signal are device self-noise components coupled in through structural transmission paths. Subsequently, the device modifies the voice signal acquisition strategy according to the determined coupling relationship. This modification includes, but is not limited to, adjusting the processing method of the voice signal itself or adjusting the judgment criteria for subsequent processing tasks such as voice activity detection and sound source localization. Finally, the device processes the voice signal according to the modified acquisition strategy to obtain acquisition results that more accurately reflect the target voice and its related attribute information.

[0042] This embodiment provides a voice acquisition method. Since a vibration reference signal can characterize the self-noise generated by the mechanical motion, structural vibration, or component operation of a device, it carries the structural transmission characteristics of the device's self-noise. When the coupling relationship between the voice signal and the vibration reference signal is determined, the self-noise components mixed into the voice signal due to the structural transmission path can be identified. Based on this coupling relationship, the voice signal acquisition strategy is modified so that it no longer relies solely on the statistical characteristics of the microphone audio signal itself, but actively intervenes against the interference of device self-noise on the voice signal, thereby effectively suppressing the interference components caused by device self-noise in the acquisition results. This reduces the impact of device self-noise on the voice acquisition quality, making the acquisition results more accurately reflect the target voice, effectively distinguishing between external environmental noise and the device's own structural vibration noise, and improving the voice signal acquisition quality.

[0043] In one feasible implementation, step S20 may include steps S21 to S23: Step S21: Obtain candidate reference signals, wherein the candidate reference signals include at least one physical vibration sensing signal and / or at least one device status signal; It should be noted that candidate reference signals are all the alternative signals initially acquired by the equipment during the process of acquiring vibration reference signals, which have not yet been validated. These include various signal sources that the equipment can acquire and that may be related to the equipment's self-noise. These signal sources can include sensor signals that directly sense physical vibration (i.e., physical vibration sensor signals) or signals that indirectly reflect the equipment's operating status (i.e., equipment status signals).

[0044] Physical vibration sensing signals refer to signals that reflect the mechanical or structural vibrations of equipment itself, directly acquired by physical vibration sensors. Physical vibration sensors include, but are not limited to, piezoelectric vibration sensors, MEMS (Micro-Electro-Mechanical System) accelerometers, inertial measurement units, structural acoustic sensors, strain gauges, or force sensors. These sensors are typically attached to or installed on structural components near the equipment's housing, joints, supports, or microphone channels, and can directly sense physical quantities such as the amplitude, frequency, and phase of vibrations generated during the equipment's movement or operation.

[0045] Equipment status signals refer to signals acquired by equipment during operation through its internal control, drive, or monitoring systems, which indirectly characterize the equipment's own movement or operating status. Equipment status signals originate from electrical control or status data generated when the equipment performs actions such as movement, rotation, joint movements, fan operation, or speaker playback, such as motor current signals, encoder signals, joint movement command signals, fan speed signals, chassis speed signals, or speaker playback status signals.

[0046] Step S22: Perform a reference validity judgment on the candidate reference signal to determine whether the candidate reference signal is related to the self-noise of the device; Step S23: The candidate reference signal that is associated with the self-noise of the device is determined as the vibration reference signal.

[0047] It is understandable that, since the sources of equipment self-noise are diverse, they may arise from various operating behaviors such as the equipment's own mechanical movement, structural vibration, or component operation. Self-noise from different sources differs in structural transmission paths and vibration characteristics. If any signal that can reflect the operating state of the equipment is directly used as a vibration reference signal for subsequent coupling relationship determination, invalid reference components unrelated to the equipment self-noise may be introduced, which may mislead the correction of the voice signal acquisition strategy.

[0048] To address this, this embodiment acquires candidate reference signals, which include at least one physical vibration sensing signal and / or at least one device status signal. The reference validity of these candidate reference signals is then assessed to determine whether they are correlated with the device's self-noise. Only candidate reference signals correlated with the device's self-noise are used as vibration reference signals. This avoids inaccurate coupling relationship determination due to the use of signals uncorrelated with the device's self-noise, which could lead to deviations in the acquisition strategy correction. This ensures that the vibration reference signals participating in subsequent steps accurately reflect the characteristics of the device's self-noise, thereby improving the accuracy of coupling relationship determination and the reliability of acquisition strategy correction.

[0049] For example, when acquiring a vibration reference signal, the device first acquires a candidate reference signal, which includes at least one physical vibration sensing signal and / or at least one device status signal. For instance, the device can acquire a physical vibration sensing signal through a piezoelectric vibration sensor installed near the housing or microphone channel, and simultaneously acquire a device status signal through a motor current sensor or encoder inside the device. Next, the device performs a reference validity judgment on the candidate reference signal to determine whether the candidate reference signal is associated with the device's self-noise. Specifically, this can be done as follows: the device acquires the candidate reference signal while acquiring the voice signal, and calculates the coherence or correlation between the candidate reference signal and the voice signal acquired by the microphone channel in each frequency band. When the coherence between a candidate reference signal and the voice signal in a specific frequency band exceeds a preset threshold, it is determined that the candidate reference signal is associated with the device's self-noise; otherwise, it is determined that the candidate reference signal is not associated with the device's self-noise and is discarded. Finally, the device uses the candidate reference signal determined to be associated with the device's self-noise as the vibration reference signal for subsequent coupling relationship determination steps.

[0050] In one feasible implementation, the step of modifying the acquisition strategy of the speech signal according to the coupling relationship in step S40 may include steps S41-S42: Step S41: Based on the coupling relationship, perform at least one of self-noise coupling component cancellation, frequency band suppression, and beamforming weight adjustment on the speech signal to obtain a compensated speech signal; and / or, Step S42: Based on the coupling relationship, correct at least one of the following: voice activity detection threshold, voice wake-up confidence, azimuth offset of sound source localization result, and near / far field judgment result.

[0051] It should be noted that self-noise coupling component cancellation involves subtracting or eliminating the portion of the signal contributed by the device's self-noise from the speech signal based on a determined coupling relationship. Once the coupling relationship is determined, the device can determine which components in the speech signal originate from the device's self-noise represented by the vibration reference signal. Self-noise coupling component cancellation utilizes this information to generate a cancellation signal with the same amplitude but opposite phase as the self-noise component, and then superimposes this cancellation signal onto the original speech signal, thereby eliminating the self-noise coupling component in the speech signal.

[0052] Frequency band suppression is a technique that reduces the energy of specific frequency bands in a speech signal that are heavily contaminated by device self-noise, based on established coupling relationships. Device self-noise is often concentrated within certain frequency ranges, and different self-noise sources may correspond to different frequency band characteristics. Frequency band suppression identifies which frequency bands are significantly contaminated by device self-noise based on coupling relationships, and then attenuates the signal components within those bands while preserving the speech components in other frequency bands. This allows for targeted reduction of the impact of self-noise without completely canceling out the entire speech signal.

[0053] Beamforming weight adjustment modifies the weight coefficients of each microphone channel in a microphone array during beamforming, based on the established coupling relationship. The microphone array enhances the speech signal from the target direction and suppresses interference from other directions by assigning different weights to different channels. When a microphone channel is severely contaminated by self-noise due to the device's structural conduction path, beamforming weight adjustment reduces the weight contribution of that channel while increasing the weight contribution of less contaminated channels, thereby suppressing the impact of device self-noise on the acquisition results at the spatial filtering level.

[0054] The compensated speech signal refers to the speech signal obtained after performing at least one of the following processing steps on the original speech signal: self-noise coupling component cancellation, frequency band suppression, or beamforming weight adjustment. The difference between the compensated speech signal and the original speech signal is that the device self-noise components mixed in through the structural transmission path have been partially or completely eliminated. Therefore, the compensated speech signal can more realistically reflect the content of the target speech, providing higher quality input for subsequent processing tasks such as speech recognition and sound source localization.

[0055] The speech activity detection threshold is a decision threshold used by the device to determine whether human voice activity exists in the current speech signal when performing speech activity detection. Speech activity detection typically determines whether the current signal segment is valid speech by comparing certain characteristics of the speech signal (such as energy, zero-crossing rate, or spectral characteristics) with a preset threshold. When the device has high self-noise, this threshold needs to be increased accordingly to avoid misjudging self-noise as target speech; conversely, when the self-noise is low, this threshold can be appropriately decreased to improve detection sensitivity.

[0056] Voice wake-up confidence is the degree to which a device, when performing a voice wake-up task, determines whether the currently acquired voice signal contains a wake-up word. Voice wake-up confidence is usually presented as a probability value or a score, with the value reflecting the reliability of the device's wake-up judgment. When device self-noise degrades the voice signal quality, even if a wake-up word is detected, the confidence will decrease due to noise interference. Correcting this confidence score aims to make it more accurately reflect the true reliability of the wake-up judgment.

[0057] The azimuth offset of the sound source localization result is the deviation between the calculated sound source direction and the actual sound source direction caused by interference from the device's own noise when determining the sound source direction. When the device's own noise couples to each microphone channel through the structural conduction path, it changes the phase difference or amplitude difference between the channels, thus affecting the accuracy of sound source localization based on these differences. Correcting the azimuth offset of the sound source localization result means eliminating or reducing this deviation according to the coupling relationship, so that the device can output a more accurate target direction.

[0058] The near-field / far-field judgment result is a conclusion drawn by the device based on the characteristics of the speech signal to determine whether the sound source is in the near or far field. Near-field sound sources have higher signal energy and lower reverberation, while far-field sound sources have the opposite. Device self-noise, originating from its own structure, may have energy characteristics similar to near-field speech, thus interfering with the near-field / far-field judgment result and biasing it towards the near field. Correcting the near-field / far-field judgment result involves reducing the interference of self-noise on the judgment basis based on the coupling relationship, enabling the device to more accurately reflect the true distance of the sound source.

[0059] It is understandable that the interference of device self-noise on voice acquisition is not only reflected in the voice signal itself, but also further transmitted to downstream processing links based on the voice signal. For example, self-noise may trigger misjudgment of voice activity detection, reduce the reliability of wake-up judgment, introduce positioning deviation of sound source direction, and interfere with the judgment of distance status between near and far fields. If the above links are not specifically processed, the accuracy of the acquisition results will still be indirectly affected by self-noise.

[0060] To address this, this embodiment performs at least one of self-noise coupling component cancellation, frequency band suppression, and beamforming weight adjustment on the speech signal according to the coupling relationship to obtain a compensated speech signal, and / or corrects at least one of the following according to the coupling relationship: speech activity detection threshold, speech wake-up confidence, directional offset of sound source localization result, and near-far field judgment result. This avoids the problem that downstream processing results such as false triggering of speech activity detection, unreliable wake-up confidence, sound source direction deviation, and inaccurate near-far field judgment are still contaminated by self-noise due to the failure to suppress specific interference forms of self-noise at different stages. It achieves self-noise suppression from two levels: the speech signal itself and the judgment criteria based on the speech signal.

[0061] For example, after determining the coupling relationship between the speech signal and the vibration reference signal, the device modifies the speech signal acquisition strategy based on this coupling relationship. Specifically, the device obtains the degree of self-noise contamination of each frequency band or microphone channel in the speech signal based on the coupling relationship. If the coupling relationship of a certain frequency band is strong, it indicates that the frequency band is severely contaminated by self-noise, and the device performs frequency band suppression to reduce the energy of that frequency band. If the coupling relationship between a certain microphone channel and the vibration reference signal is strong, it indicates that the microphone channel is greatly affected by structure-conducted self-noise, and the device adjusts the weight of that microphone channel in beamforming to reduce the contribution of that channel to the acquisition results. At the same time, the device also increases the speech activity detection threshold accordingly based on the self-noise confidence determined by the coupling relationship to avoid misjudging self-noise as target speech, decreases the speech wake-up confidence to reflect the impact of self-noise on the reliability of wake-up judgment, corrects the azimuth offset of the sound source localization result to eliminate the directional deviation caused by self-noise, and corrects the near-field judgment result to avoid self-noise being misjudged as near-field speech features. Through the above processing, the device obtains the compensated speech signal, as well as the corrected speech activity detection threshold, speech wake-up confidence, directional offset of the sound source localization result, and near-far field judgment result. These corrected data will serve as the basis for obtaining subsequent acquisition results.

[0062] In one feasible implementation, the step of obtaining the acquisition result of the speech signal according to the modified acquisition strategy in step S40 may include steps S43-S44: Step S43: Determine the target speech based on the compensated speech signal, and / or determine the speech recognition confidence based on at least one of the corrected speech activity detection threshold, speech wake-up confidence, directional offset of the sound source localization result, and near-far field judgment result. It should be noted that the target speech is the human voice content spoken by the user that the device intends to collect. When the device performs a speech acquisition task, the speech signal captured by the microphone channel is not the pure target speech, but rather a mixture of the target speech, device self-noise, and any ambient noise. The device self-noise is coupled to the microphone channel through the structural conduction path and superimposed on the target speech, thus contaminating the target speech component in the speech signal. By processing the speech signal according to a corrected acquisition strategy, the device can extract the self-noise-suppressed speech component from the original speech signal; this extracted speech component is the target speech.

[0063] Speech recognition confidence is a quantitative indicator of the reliability of a device's recognition results when performing speech recognition tasks. After acquiring a speech signal or a compensated speech signal, the device converts it into text or semantic information through a speech recognition model. Speech recognition confidence reflects the reliability of this conversion result. This confidence can be presented in the form of probability values ​​or scores, but is not limited to these. The higher the value, the more reliable the device is in terms of the recognition result.

[0064] Speech recognition confidence is affected by the quality of the speech signal. When the device has a lot of self-noise, the target speech component in the speech signal is contaminated, and the speech recognition model has difficulty in accurately extracting the speech content. Even if the recognition process is completed, the confidence of the output recognition result will be low due to the decrease in input quality.

[0065] When the device corrects the acquisition strategy based on the coupling relationship, it can correct the speech recognition confidence. For example, it can recalculate the confidence based on the compensated speech signal, so that the corrected speech recognition confidence can more accurately reflect the actual credibility of the speech recognition result under the current self-noise suppression conditions. This provides a basis for the device to decide whether to adopt the recognition result or whether to ask the user to confirm again.

[0066] Step S44: At least one of the target speech, the corrected speech activity detection threshold, the speech wake-up confidence, the directional offset of the sound source localization result, the near and far field judgment result, and the speech recognition confidence is used as the acquisition result.

[0067] Understandably, by determining the target speech based on the compensated speech signal, and / or determining the speech recognition confidence based on at least one of the following: the corrected speech activity detection threshold, speech wake-up confidence, azimuth offset of the sound source localization result, and near-far field judgment result, and using at least one of the target speech, the corrected speech activity detection threshold, speech wake-up confidence, azimuth offset of the sound source localization result, near-far field judgment result, and speech recognition confidence as the acquisition result, it is possible to avoid the situation where the corrected data of each dimension is not integrated into a clearly usable acquisition result, which would prevent the corrected information from being unable to directly serve subsequent voice interaction decisions. This achieves the output of multi-dimensional information after self-noise suppression and correction of target direction, distance status, and confidence in the form of structured acquisition results, providing a directly usable data foundation for subsequent voice interaction or task decision-making.

[0068] For example, after the device performs at least one of self-noise coupling component cancellation, frequency band suppression and beamforming weight adjustment on the speech signal according to the coupling relationship to obtain a compensated speech signal, and / or corrects at least one of the speech activity detection threshold, speech wake-up confidence, azimuth offset of sound source localization result and near-far field judgment result, the device determines the target speech based on the compensated speech signal. Specifically, the device outputs the compensated speech signal as the target speech after self-noise suppression. Compared with the original speech signal, the device self-noise mixed in through the structural conduction path has been partially or completely eliminated. Simultaneously, the device determines the speech recognition confidence based on at least one of the following: a corrected speech activity detection threshold, speech wake-up confidence, azimuth offset of the sound source localization result, and near / far field judgment result. Specifically, the device uses the corrected judgment results as input features and inputs them into a speech recognition confidence evaluation model. This model outputs a speech recognition confidence reflecting the reliability of the current recognition result. If the corrected speech wake-up confidence is high and the azimuth offset of the sound source localization result is small, the speech recognition confidence is correspondingly high; conversely, it is correspondingly low. Finally, the device outputs the target speech and the speech recognition confidence together as the acquisition result, or the target speech and at least one of the corrected speech activity detection threshold, speech wake-up confidence, azimuth offset of the sound source localization result, and near / far field judgment result together as the acquisition result. This acquisition result can be directly used by the device for subsequent voice interaction or task decision-making.

[0069] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before step S30, the voice acquisition method further includes step S01: Step S01: Time-align the speech signal and the vibration reference signal, and perform the step of determining the coupling relationship between the speech signal and the vibration reference signal based on the time-aligned speech signal and vibration reference signal.

[0070] It should be noted that time alignment refers to the process of adjusting the speech signal and the vibration reference signal to be in a consistent state in the time dimension.

[0071] It is understandable that, since the speech signal is acquired through the microphone channel and the vibration reference signal through the vibration reference channel, the two may have different sampling rates, data transmission delays, and response delays during the acquisition process. Furthermore, the device self-noise requires a certain propagation time to travel from the vibration location to the microphone channel via the structural conduction path. This means that the device self-noise generated at the same moment may appear at different times in the speech signal and the vibration reference signal. If the unaligned speech signal and vibration reference signal are directly used to determine the coupling relationship, the calculated coupling relationship will fail to accurately reflect the correspondence between the self-noise components in the speech signal and the vibration reference signal due to time misalignment. Therefore, time alignment aims to compensate for these time differences, ensuring that the speech signal and vibration reference signal correspond on the same time base. This guarantees that when determining the coupling relationship later, the speech signal at the current moment and the vibration reference signal at the same moment can be matched in time, thereby accurately identifying which components in the speech signal originate from the device self-noise represented by the vibration reference signal. This provides a correct time correspondence basis for subsequent steps to determine the coupling relationship based on the time-aligned signal.

[0072] For example, after acquiring a speech signal from at least one microphone channel and a vibration reference signal from at least one vibration reference channel, and before performing the step of determining the coupling relationship between the speech signal and the vibration reference signal, the device performs time alignment on the speech signal and the vibration reference signal. Specifically, the device identifies the acquisition time information corresponding to each of the speech signal and the vibration reference signal, determines the overall time deviation between them due to differences in sampling rate, data transmission delay, and the propagation time required for self-noise to travel from the vibration location through the structural conduction path to the microphone channel, and then adjusts the time axis of at least one of the speech signal or the vibration reference signal according to the time deviation, so that the speech signal and the vibration reference signal correspond on the same time reference, ensuring that the speech signal at the current moment matches the vibration reference signal at the same moment in time. After completing the time alignment, the device performs the step of determining the coupling relationship between the speech signal and the vibration reference signal based on the time-aligned speech signal and the time-aligned vibration reference signal.

[0073] In this embodiment, the speech signal and vibration reference signal are time-aligned. The step of determining the coupling relationship between the speech signal and the vibration reference signal is then performed based on the time-aligned speech signal and vibration reference signal. Since the device's self-noise requires a certain propagation time to travel from the vibration location to the microphone channel through the structural conduction path, and the speech signal and vibration reference signal may have different sampling rates, data transmission delays, and response delays, the unaligned speech signal and vibration reference signal are out of sync in time. Directly using them to determine the coupling relationship would result in calculation results that cannot accurately reflect the correspondence between the self-noise components in the speech signal and the vibration reference signal. This avoids the problem of inaccurate coupling relationships determined later due to time misalignment, which in turn makes the basis for correcting the acquisition strategy unreliable. It achieves the correspondence between the speech signal and the vibration reference signal on the same time base, ensuring that the speech signal at the current moment matches the vibration reference signal at the same moment in time. This provides a correct time correspondence basis for accurately identifying which components in the speech signal originate from the device's self-noise represented by the vibration reference signal.

[0074] In one feasible implementation, step S01 may include steps S011-S012: Step S011: Determine the time alignment offset based on the timestamp of the voice signal, the timestamp of the vibration reference signal, and the preset vibration transmission delay; It should be noted that vibration transmission delay is the time required for the device's self-noise to travel from the vibration location to the microphone channel through the device's structural conduction path. When the device performs actions such as moving, rotating, jointing, or fan operation, the self-noise generated by its mechanical motion or structural vibration does not arrive at the microphone channel instantaneously at the same time as the voice signal is collected. Instead, it needs to propagate through solid media such as the device's shell, support, and internal structural components. Factors such as the density, elastic modulus, and structural shape of the solid medium will affect the propagation speed of the vibration within it. In addition, the structural conduction path length between different vibration locations and the microphone channel is different, so the propagation time of the device's self-noise generated at the same moment may also be different. Vibration transmission delay is used to quantify this propagation time. Its specific value can be obtained through pre-calibration methods, such as measuring the transmission time from each vibration location to the microphone channel through impact tests or frequency sweep tests before the device leaves the factory, or it can be dynamically estimated based on historical data during actual operation.

[0075] The existence of vibration transmission delay causes the device self-noise generated at the same moment to appear at different times in the vibration reference signal and the speech signal. That is, the time when the vibration is collected by the vibration reference channel at the vibration location is different from the time when the self-noise reaches the microphone channel through the structural conduction path and is collected by the microphone channel. Therefore, when aligning the speech signal and the vibration reference signal in time, the vibration transmission delay needs to be used as one of the compensation criteria.

[0076] The time alignment offset is a quantized value of the deviation between the speech signal and the vibration reference signal on the time axis. This quantized value characterizes the time difference between when the same device self-noise event is acquired in the speech signal and when it is acquired in the vibration reference signal. The time alignment offset is not a fixed constant; its value is determined by multiple factors, including the difference in the acquisition start time of the speech signal and the vibration reference signal, the difference in their sampling rates, the difference in the latency of the data transmission link, and the vibration transmission delay required for the device self-noise to travel from the vibration location to the microphone channel through the structural conduction path.

[0077] When determining the time alignment offset, the device performs a comprehensive calculation based on the timestamp of the voice signal, the timestamp of the vibration reference signal, and a preset vibration propagation delay. For example, the timestamp of the vibration reference signal can be added to the preset vibration propagation delay and then compared with the timestamp of the voice signal; the difference between the two can be calculated as the time alignment offset. Once the time alignment offset is determined, it serves as the basis for subsequent time axis adjustments to the voice signal or vibration reference signal. This offset quantifies the specific degree of temporal misalignment between the voice signal and the vibration reference signal.

[0078] Step S012: Synchronize the speech signal and the vibration reference signal with a time window according to the time alignment offset to obtain the time-aligned speech signal and vibration reference signal.

[0079] It should be noted that time window synchronization is the process of adjusting the voice signal and vibration reference signal to the same time reference based on the time alignment offset, so that the two are matched in time.

[0080] The specific operation method of time window synchronization can be as follows: according to the time alignment offset, the time axis of the speech signal or vibration reference signal is shifted or resampled so that the sampling time point of the speech signal and the sampling time point of the vibration reference signal correspond one-to-one within the same time window, thereby ensuring that within the current time window, the acoustic information collected by the microphone channel reflected in the speech signal and the vibration information collected by the vibration reference channel reflected in the vibration reference signal come from the same physical moment.

[0081] The audio signal and vibration reference signal obtained after synchronization through the time window are time-aligned audio signal and vibration reference signal. The synchronized signal can accurately reflect the correspondence between the device's self-noise in the audio signal and vibration reference signal at the same moment.

[0082] It is understandable that the audio signal and the vibration reference signal may have different sampling rates, acquisition start times, and data transmission delays during the acquisition process, and the device's self-noise needs a certain propagation time to travel from the vibration location to the microphone channel through the structural conduction path.

[0083] To address this, this embodiment determines the time alignment offset based on the timestamp of the speech signal, the timestamp of the vibration reference signal, and a preset vibration transmission delay. It then synchronizes the speech signal and vibration reference signal using a time window based on this time alignment offset, resulting in time-aligned speech and vibration reference signals. This avoids residual time misalignment between the speech signal and vibration reference signal, which could reduce the accuracy of subsequent coupling relationship determination. The embodiment achieves precise quantification of the time deviation between the speech signal and vibration reference signal using timestamps and a preset vibration transmission delay, and synchronizes them via a time window to ensure a one-to-one correspondence on the same time reference, providing a high-precision time alignment signal for subsequent coupling relationship determination.

[0084] For example, after acquiring the voice signal and the vibration reference signal, the device first acquires the timestamp of the voice signal, the timestamp of the vibration reference signal, and a preset vibration propagation delay. The timestamp of the voice signal records the time information of each sampling point of the voice signal acquired by the microphone channel, the timestamp of the vibration reference signal records the time information of each sampling point of the vibration reference signal acquired by the vibration reference channel, and the preset vibration propagation delay reflects the propagation time required for noise to travel from the vibration location to the microphone channel through the structural conduction path. This preset vibration propagation delay can be a fixed value obtained through calibration measurement and stored in the device before it leaves the factory, or it can be an estimated value dynamically estimated based on historical data during device operation. Then, the device uses the timestamp of the voice signal, the timestamp of the vibration reference signal, and the preset vibration propagation delay... The device determines the time alignment offset by adding a preset vibration propagation delay to the timestamp of the vibration reference signal to obtain an estimated time for the self-noise corresponding to the vibration reference signal to reach the microphone channel. This estimated time is then compared with the timestamp of the speech signal, and the difference between the two is calculated as the time alignment offset. Finally, the device synchronizes the speech signal and vibration reference signal within a time window based on this time alignment offset. Specifically, the device keeps the time axis of the speech signal unchanged and shifts or resamples the time axis of the vibration reference signal according to the time alignment offset, ensuring that the sampling time points of the vibration reference signal and the speech signal correspond one-to-one within the same time window. This results in time-aligned speech and vibration reference signals, which will be used in subsequent steps to determine the coupling relationship.

[0085] In one feasible implementation, the coupling relationship includes a self-noise coupling component and / or a self-noise confidence level, and step S30 may include steps S31 to S33: Step S31: Within the synchronized time window, determine the coupling characteristics of the speech signal and the vibration reference signal in multiple frequency bands, wherein the coupling characteristics include coherence, correlation and / or synchronization strength. It should be noted that coupling characteristics are a general term for the degree of correlation between a speech signal and a vibration reference signal in the time or frequency domain. When the device's self-noise is coupled to the microphone channel through the structural conduction path, the self-noise component in the speech signal and the vibration reference signal will exhibit consistency or correlation in dimensions such as vibration amplitude, frequency, or phase. Coupling characteristics are used to quantify the strength of this consistency or correlation.

[0086] Coupling characteristics are not limited to a single metric; they can be characterized by various mathematical measures, such as coherence, correlation, or synchronization strength. These measures reflect the degree of correlation between the speech signal and the vibration reference signal from different perspectives. The purpose of determining coupling characteristics is to identify which components of the speech signal are associated with the device self-noise represented by the vibration reference signal, and the strength of this association, thereby providing a basis for subsequently determining the coupling parameters for each frequency band and each microphone channel.

[0087] Coherence, in this context, refers to the degree of linear correlation between a speech signal and a vibration reference signal across different frequency bands in the frequency domain. It measures the stability of the phase consistency and amplitude ratio between two signals at a specific frequency. Coherence can be obtained by calculating the ratio of the cross-power spectral density of the speech signal and the vibration reference signal to their respective self-power spectral densities. The value ranges from 0 to 1. A value closer to 1 indicates a stronger linear correlation between the speech signal and the vibration reference signal in the corresponding frequency band, meaning that the speech signal components in that band are likely contributed by the equipment self-noise represented by the vibration reference signal. A value closer to 0 indicates no significant linear correlation between the two in the corresponding frequency band, meaning that the speech signal components in that band are unlikely to originate from the equipment self-noise represented by the vibration reference signal. Coherence can accurately locate frequency bands contaminated by self-noise in the frequency domain, providing a basis for subsequently determining coupling weights or transfer functions by frequency band.

[0088] Correlation is the degree of similarity between the amplitude variation trends of a speech signal and a vibration reference signal in the time or frequency domain. Correlation can be obtained by calculating the Pearson correlation coefficient, cross-correlation function, or other correlation measures between the speech signal and the vibration reference signal. It reflects whether the two signals have a consistent trend in amplitude variation; that is, whether the component contributed by self-noise in the speech signal also increases accordingly when the amplitude of the vibration reference signal increases at a certain moment. Unlike coherence, which focuses on the stability of the phase and amplitude relationship in the frequency domain, correlation focuses more on the consistency of the amplitude variation trend of the signals. Both characterize the coupling relationship between the speech signal and the vibration reference signal from different perspectives and can be used complementaryly.

[0089] Synchronization strength is a measure of the temporal consistency of events occurring between the speech signal and the vibration reference signal. Since device self-noise takes a certain amount of time to propagate from the vibration location to the microphone channel via the structural conduction path, there is a fixed time difference between the occurrence of the same self-noise event in the vibration reference signal and the speech signal. Synchronization strength measures whether, after compensating for this time difference, the self-noise component in the speech signal and the vibration reference signal exhibit synchronous changes along the time axis. Synchronization strength can be evaluated through event trigger analysis, peak detection, or envelope alignment. For example, detecting the moment when a large vibration peak occurs in the vibration reference signal and determining whether a corresponding energy peak occurs in the speech signal at the corresponding delay time indicates a high degree of synchronization between the speech signal and the vibration reference signal. Synchronization strength, coherence, and correlation reflect the coupling relationship between the speech signal and the vibration reference signal from different dimensions; these three can be used independently or in combination.

[0090] Step S32: If the coupling characteristics meet the preset reference validity conditions, determine the coupling parameters of the microphone channel and each frequency band, wherein the coupling parameters include coupling weights and / or transfer functions; It should be noted that the reference valid condition is a preset judgment condition for determining whether the coupling characteristics meet the usability requirements.

[0091] When the equipment is stationary and there is no self-noise, the vibration reference signal may only contain environmental vibration or sensor noise floor. In this case, the calculated coupling characteristics may be very low and cannot reflect the effective coupling relationship. When the vibration reference signal is saturated or distorted, the calculated coupling characteristics cannot truly reflect the actual situation.

[0092] The reference validity condition is a threshold set to avoid the above situations. Its specific forms may include, but are not limited to: whether the average coherence value across all frequency bands exceeds a preset coherence threshold, whether the correlation coefficient exceeds a preset correlation threshold, whether the synchronization strength meets preset synchronization criteria, whether the amplitude of the vibration reference signal is within its effective range and not saturated, and whether the time alignment offset between the speech signal and the vibration reference signal is within a reasonable range. Only when the coupling characteristic meets the reference validity condition is the calculation result of the coupling characteristic considered reliable and used for subsequent determination of coupling parameters.

[0093] The coupling parameter is a specific numerical index used to quantify the coupling relationship between the speech signal and the vibration reference signal. It describes the specific way in which the device's self-noise is coupled from the vibration source represented by the vibration reference channel to each microphone channel and each frequency band through the structural conduction path.

[0094] The coupling parameters are not a single value, but a set of parameters determined for different microphone channels and frequency bands. The specific form of the coupling parameters includes at least one of coupling weights and transfer functions, which are determined under the condition that the coupling characteristics satisfy the reference validity condition. The determined coupling parameters are used to generate the self-noise coupling components and self-noise confidence levels in the speech signal. The coupling parameters convert the device self-noise characteristics reflected in the vibration reference signal into specific values ​​of the self-noise components in the speech signal, that is, mapping the amplitude, frequency, and phase information of the vibration reference signal to the amplitude, frequency, and phase information of the self-noise components in the speech signal.

[0095] The coupling weight is a quantification coefficient used to measure the degree of self-noise contamination in the speech signal of each microphone channel and frequency band. The coupling weight is a value calculated based on coupling characteristics, reflecting the proportion of self-noise components in the speech signal relative to the vibration reference signal at a specific microphone channel and frequency band. A larger coupling weight indicates more severe self-noise contamination in the corresponding microphone channel and frequency band, requiring stronger suppression processing when subsequently modifying the acquisition strategy. Conversely, a smaller coupling weight indicates less self-noise contamination, allowing for the retention of a higher weight.

[0096] A transfer function is a mathematical model describing the frequency response characteristics of a device's self-noise transmitted from a vibration source (represented by a vibration reference channel) through a structural conduction path to each microphone channel. The transfer function can include not only amplitude proportionality relationships but also phase change relationships, comprehensively describing the amplitude and phase changes of the vibration reference signal as it travels through the structural conduction path to the microphone channel. When determining the transfer function, the frequency domain transmission characteristics between the speech signal and the vibration reference signal can be calculated; for example, the transfer function can be estimated by dividing the cross-power spectral density by the self-power spectral density. Once the transfer function is determined, the vibration reference signal at any given time can be input into it to obtain the predicted value of the device's self-noise coupled to the microphone channel through the structural conduction path at that moment.

[0097] Step S33: Generate the self-noise coupling component and / or the self-noise confidence level based on the coupling parameters.

[0098] It should be noted that the self-noise coupling component is the portion of the speech signal contributed by the device's self-noise coupled to the microphone channel through the structural conduction path. A speech signal is not a single-component signal, but a mixture of multiple components including the target speech, device self-noise, and potential environmental noise. The self-noise coupling component specifically refers to the component originating from the device self-noise.

[0099] The self-noise coupling component is generated based on coupling parameters and a vibration reference signal. Specifically, by mapping the amplitude and phase of the vibration reference signal according to a determined coupling weight or transfer function, the predicted signal component in the speech signal contributed by the device's self-noise can be obtained. Determining the self-noise coupling component allows the device to clearly identify which specific components in the speech signal originate from the device's self-noise, thus providing specific signal content for subsequently canceling or suppressing these components from the speech signal.

[0100] Self-noise confidence is a comprehensive quantitative indicator of the reliability of the coupling relationship and the degree to which the speech signal is affected by device self-noise. Self-noise confidence differs from self-noise coupling component. Self-noise coupling component quantifies the specific signal content of self-noise components in the speech signal, while self-noise confidence quantifies the credibility of the currently determined coupling relationship itself and the reliability of judging the extent to which the speech signal is affected by self-noise based on the coupling relationship.

[0101] The self-noise confidence level is determined comprehensively based on factors such as the degree to which coupling characteristics satisfy the reference validity conditions, the consistency and stability of coupling parameters, etc. For example, if the coupling characteristics of multiple frequency bands all satisfy the reference validity conditions and the determined coupling parameters remain stable within different time windows, the self-noise confidence level is high, indicating that the currently determined coupling relationship is reliable. Conversely, if the coupling characteristics of only a few frequency bands satisfy the reference validity conditions or the coupling parameters fluctuate greatly within different time windows, the self-noise confidence level is low, indicating that the currently determined coupling relationship is not reliable enough. The self-noise confidence level can be used to judge the strength of compensation operations when correcting subsequent acquisition strategies. When the self-noise confidence level is high, stronger compensation processing can be performed; when the self-noise confidence level is low, compensation processing needs to be performed cautiously to avoid incorrect compensation that could damage the target speech.

[0102] Understandably, by determining the coupling characteristics of the speech signal and the vibration reference signal in multiple frequency bands within the synchronized time window, and determining the coupling parameters of the microphone channel and each frequency band when the coupling characteristics meet the preset reference validity conditions, and then generating self-noise coupling components and / or self-noise confidence based on the coupling parameters, the correlation between the time-aligned speech signal and the vibration reference signal is transformed into specific parameters that can be used for subsequent acquisition strategy correction, namely self-noise coupling components and / or self-noise confidence, providing a reliable quantitative basis for the accurate correction of the acquisition strategy.

[0103] For example, after obtaining the time-aligned speech signal and vibration reference signal, the device calculates the coupling characteristics of the speech signal and vibration reference signal of each microphone channel across multiple frequency bands within a synchronized time window. These coupling characteristics include at least one of coherence, correlation, and synchronization strength. Specifically, the device performs a frequency domain transformation on the synchronized speech signal and vibration reference signal, decomposing the signal into multiple frequency bands. Then, for each frequency band, it calculates the coherence value between the speech signal and the vibration reference signal. This coherence value reflects the degree of linear correlation between the speech signal and the vibration reference signal in that frequency band. After obtaining the coherence values ​​for each frequency band, the device compares these values ​​with a preset reference validity condition. This reference validity condition can be set as a coherence threshold. For example, if the coherence value calculated for a certain frequency band exceeds 0.6, then the coupling characteristics of that frequency band are determined to meet the reference validity condition; conversely, if the coherence value does not exceed 0.6, then the coupling characteristics of that frequency band are determined not to meet the reference validity condition.

[0104] For each frequency band and each microphone channel that meets the reference validity conditions, the device determines the corresponding coupling parameters, which include at least one of coupling weight and transfer function. For example, the amplitude ratio of the speech signal and the vibration reference signal in the frequency band can be calculated as the coupling weight, or the frequency domain transfer characteristics of the frequency band can be calculated as the transfer function.

[0105] Finally, the device generates self-noise coupling components and self-noise confidence based on the determined coupling parameters. For example, the vibration reference signal is mapped in amplitude and phase according to the coupling weight or transfer function to generate the predicted signal component contributed by the device's self-noise in the speech signal as the self-noise coupling component. At the same time, the self-noise confidence is determined according to the degree to which the coupling characteristics meet the reference validity conditions. If multiple frequency bands meet the reference validity conditions and the determined coupling parameters remain stable in different time windows, the self-noise confidence is high; otherwise, the self-noise confidence is low.

[0106] In the specific implementation process, step S31 may include steps S311 to S313: Step S311: Within the time window, the speech signal and the vibration reference signal are processed by framing or windowing respectively to obtain the speech sub-signal and vibration sub-signal within each time sub-window; It should be noted that a speech sub-signal is a segment of the speech signal obtained within each time window after the speech signal has been framed or windowed. The speech signal is a continuously changing acoustic signal over time; its frequency components and amplitude characteristics may dynamically change over time. If the speech signal within the entire time window is analyzed as a whole, the dynamic changes in the coupling relationship between the speech signal and the vibration reference signal at different time segments cannot be captured. Therefore, within the synchronized time window, the device divides the speech signal into multiple consecutive time windows in chronological order; the segment of speech signal within each time window is the speech sub-signal.

[0107] The difference between the speech sub-signal and the original speech signal lies in the fact that the speech sub-signal is a truncated portion of the original speech signal at a specific time segment. Its duration is relatively short, for example, set to tens to hundreds of milliseconds. Within this short time range, the speech signal can be approximated as stationary, facilitating subsequent frequency domain transformation. The speech sub-signal corresponding to each time window will be paired with the vibration sub-signal within the same time window for analysis to determine the coupling characteristics between the speech signal and the vibration reference signal at each time segment.

[0108] A vibration sub-signal is a segment of the vibration reference signal obtained within each time window after framing or windowing the vibration reference signal. The vibration reference signal and the speech signal at the same moment together reflect the propagation and coupling process of the equipment's self-noise along the structural conduction path. While framing or windowing the speech signal, the equipment also segments the vibration reference signal using the same framing or windowing parameters, dividing the vibration reference signal into multiple vibration sub-signals that correspond one-to-one with the speech sub-signals in time.

[0109] The relationship between the vibration sub-signal and the vibration reference signal is similar to that between the speech sub-signal and the speech signal. That is, the vibration sub-signal is a segment of the vibration reference signal within a specific time window, and its duration is exactly the same as the corresponding speech sub-signal. Acquiring the vibration sub-signal allows the device to compare the local features of the vibration reference signal with the local features of the speech signal within each time window, thereby determining the coupling characteristics between the two signals at different time segments.

[0110] Step S312: Perform frequency domain transformation on each of the speech sub-signals and each of the vibration sub-signals to obtain the frequency domain components corresponding to each frequency band; It should be noted that frequency domain components are the signal components at various frequency points obtained after performing a frequency domain transformation on the speech or vibration sub-signal. In the time domain, the speech and vibration sub-signals appear as waveforms whose amplitude varies with time. These waveforms contain the superposition of multiple sinusoidal components of different frequencies, but the time-domain waveform itself cannot directly reflect the energy distribution of each frequency component. The device converts the time-domain waveform into a frequency-domain representation by performing frequency domain transformations on each speech and vibration sub-signal, such as Fast Fourier Transform or Discrete Fourier Transform. In the resulting frequency-domain representation, the complex values ​​corresponding to different frequency points are the frequency components at those frequencies. Each frequency component contains amplitude and phase information for the corresponding frequency component. The amplitude reflects the energy level of the frequency component, and the phase reflects the time offset of the frequency component.

[0111] Step S313: Determine the coupling characteristics based on each of the frequency domain components.

[0112] It is understood that this embodiment performs frame-by-frame or window-by-window processing on the speech signal and vibration reference signal within a time window to obtain the speech sub-signal and vibration sub-signal within each time sub-window. Then, it performs frequency domain transformation on each speech sub-signal and each vibration sub-signal to obtain the frequency domain components corresponding to each frequency band. The coupling characteristics are then determined based on each frequency domain component. This avoids the problem that the dynamic changes of coupling characteristics in different time segments cannot be captured due to the uniform analysis of coupling relationships throughout the entire time window, which leads to the coupling characteristics failing to reflect the differences in coupling characteristics in local time segments and thus making the determination of coupling parameters inaccurate. This embodiment achieves the determination of the coupling characteristics between the speech signal and the vibration reference signal segment by segment at a finer time granularity, so that the determined coupling characteristics can reflect the dynamic changes of coupling relationships in each time segment, providing a more accurate coupling characteristic input for subsequent determination of coupling parameters.

[0113] For example, within the synchronized time window, the device first performs frame-segmentation or windowing processing on the speech signal and vibration reference signal respectively. Specifically, the device divides the speech signal within the entire time window into multiple continuous and partially overlapping speech sub-signals according to a preset frame length and frame shift. At the same time, the vibration reference signal is divided using the same frame length and frame shift to obtain vibration sub-signals that correspond one-to-one with each speech sub-signal in time. Each speech sub-signal and its corresponding vibration sub-signal have exactly the same time start position and time length, ensuring precise time matching. Next, the device performs frequency domain transformation on each speech sub-signal and each vibration sub-signal, for example, by applying a Fast Fourier Transform to each sub-signal to transform each sub-signal from a time-domain waveform. The frequency domain representation is used, and the complex values ​​corresponding to each discrete frequency point obtained after conversion are the frequency domain components of that frequency point. Each frequency domain component contains the amplitude and phase information of the corresponding frequency component. Finally, the device determines the coupling characteristics based on each frequency domain component. That is, in each frequency band, the coherence, correlation or synchronization strength between the frequency domain components of the speech sub-signal and the vibration sub-signal are calculated as the coupling characteristics of that time sub-window and that frequency band, thereby obtaining the coupling characteristic matrix corresponding to each time sub-window and each frequency band. This coupling characteristic matrix reflects the degree of correlation between the speech signal and the vibration reference signal in different time segments and different frequency bands, providing fine-grained coupling characteristic data for subsequent determination of the coupling parameters of each microphone channel and each frequency band.

[0114] In the specific implementation process, after step S33, steps S301 to S303 may also be included: Step S301: Obtain the speech confidence of the target speech; It should be noted that speech confidence refers to a quantitative indicator of the degree to which a device can confidently determine that a currently acquired speech signal contains valid target speech, or a quantitative indicator of the reliability of speech recognition results or speech wake-up judgment results obtained based on that speech signal. The specific calculation method for speech confidence can be based on dimensions such as the signal-to-noise ratio of the target speech, the degree of matching between the energy characteristics of the target speech and historical speech templates, and the recognition score output by the speech recognition model or the wake-up score output by the wake-up model. A higher score indicates higher reliability of the currently acquired target speech, allowing the device greater confidence in performing subsequent voice interaction tasks based on that target speech; a lower score indicates that the currently acquired target speech is significantly affected by device self-noise or other interference factors, resulting in insufficient reliability of the target speech.

[0115] Step S302: If the self-noise confidence level is greater than a first preset threshold and the speech confidence level is less than a second preset threshold, and / or the vibration reference signal is an invalid signal, a preset interactive degradation strategy is executed.

[0116] It should be noted that the self-noise confidence level reflects the reliability of the estimation of the degree of contamination of the speech signal by the device's self-noise, while the speech confidence level reflects the reliability of the target speech itself. When the self-noise confidence level is higher than the first preset threshold and the speech confidence level is lower than the second preset threshold, it indicates that the degree of contamination of the speech signal by the device's self-noise has been reliably estimated, but the degree of contamination is so severe that a reliable target speech cannot be obtained. At this time, the device needs to activate the degradation interaction strategy.

[0117] Interactive degradation strategy is a set of preset coping measures that the device proactively changes its behavior mode to cope with the low confidence state when it determines that the current voice acquisition conditions cannot obtain reliable acquisition results. Interactive degradation strategy is a backup coping plan that the device activates under extreme abnormal conditions such as excessively high self-noise confidence and excessively low voice confidence, and / or failure of vibration reference signal.

[0118] Specific forms of interactive degradation strategies include, but are not limited to: pausing or reducing the speed of the device to reduce the amount of self-noise generated at the current moment, thereby improving voice acquisition conditions; requesting the user to repeat the voice command to reacquire the target voice in a lower self-noise environment; delaying recognition, i.e., temporarily refraining from recognizing the currently acquired voice signal and waiting for the self-noise conditions to improve before performing recognition; re-acquiring voice, i.e. discarding the current acquisition result and starting a new round of voice acquisition process; and issuing low-confidence confirmation prompts, i.e. providing feedback to the user through voice, screen display, or light to indicate that the current acquisition conditions are poor and the recognition results may be unreliable, requesting the user to confirm or re-enter.

[0119] For example, after generating the self-noise coupling component and / or self-noise confidence score based on the coupling parameters, the device continues to perform handling processes for abnormal or low-confidence states. Specifically, the device first acquires the speech confidence score of the target speech, which reflects the reliability of the currently acquired target speech and can be determined by the signal-to-noise ratio of the target speech or the recognition score output by the speech recognition model. Then, the device compares the self-noise confidence score with a first preset threshold and the speech confidence score with a second preset threshold, while checking whether the vibration reference signal is valid. When the device determines that the self-noise confidence score is greater than the first preset threshold and the speech confidence score is less than the second preset threshold, it indicates that the device's self-noise contamination of the speech signal has been reliably estimated, but the contamination level is so severe that reliable target speech cannot be obtained. When the vibration reference signal is invalid, it indicates that the vibration reference signal cannot provide a reliable basis for determining the coupling relationship due to sensor saturation, signal loss, or time alignment failure. If any of the above conditions are met, the device executes a preset interaction degradation strategy. For example, the device issues a voice prompt through the speaker to request the user to repeat the voice command, while pausing the current movement to reduce the amount of self-noise generated, and delaying the recognition and processing of the currently acquired voice signal, waiting for the self-noise conditions to improve or for the user to re-enter before resuming the normal voice acquisition and recognition process.

[0120] For example, to help understand the implementation flow of the voice acquisition method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A system architecture diagram of a voice acquisition method is provided, specifically: The microphone acquisition module acquires multi-channel speech signals, while the vibration reference acquisition module acquires vibration reference signals, including vibration signals, structural acoustic signals, and equipment status signals. Simultaneously, the motion state input module acquires motion or operational status information such as chassis, joints, fans, and playback. The multi-channel speech signals and vibration reference signals first enter the time alignment module, which achieves precise temporal alignment through delay estimation, window synchronization, and sampling rate renormalization. Subsequently, the self-noise coupling estimation module estimates the coupling components or self-noise confidence in the speech signal based on the time-aligned signal. This estimation result is sent to the adaptive compensation module to perform self-noise coupling component cancellation, frequency band suppression, or beamforming weight adjustment on the multi-channel speech signal. It is also sent to the acquisition result correction module to correct acquisition results such as sound source direction, distance status, and confidence. The corrected result is finally passed to the degradation interaction module, which triggers degradation interaction strategies such as requesting repetition or pausing motion when self-noise is too strong or confidence is insufficient. Simultaneously, the feedback update module feeds the processing results back to the coupled model to achieve dynamic updates of the model parameters.

[0121] Further, please refer to Figure 5 , Figure 5 A schematic diagram of the coupling estimation process for a speech acquisition method is provided, specifically: The microphone signal (a mixture of target speech and device self-noise) and vibration reference signal are first time-aligned, eliminating time misalignment through delay compensation and synchronization windows. After time alignment, the system can estimate coupling relationships using three methods: first, transfer function estimation, which estimates the transfer function of the structural conduction path by analyzing the amplitude, phase, and frequency band characteristics of the vibration reference signal and microphone signal at various frequency bands; second, adaptive filtering estimation, which uses the vibration reference signal as the reference input and the residual noise in the microphone signal as the target signal, estimating the coupling components based on the convergence result of the adaptive filter; and third, model or calibration table estimation, which uses a table lookup based on the device's operating state and historical calibration data, or generates self-noise prediction values ​​through a model. These three methods can be used individually or in combination, ultimately outputting the self-noise coupling components and self-noise confidence levels for each microphone channel and frequency band.

[0122] Further, please refer to Figure 6 , Figure 6 A schematic diagram of the compensation and correction process for a voice acquisition method is provided, specifically: The self-noise coupling component and self-noise confidence level are used as inputs to drive two separate processing paths: one is a compensation and correction process, which performs self-noise coupling component cancellation, frequency band suppression, beamforming weight adjustment, or feature correction on the original microphone speech signal, outputting the compensated target speech while retaining residual noise for subsequent evaluation; the other is a acquisition result correction process, which corrects the sound source directional stability, near-far field judgment confidence, wake-up confidence, and recognition confidence based on the self-noise confidence level. The corrected target speech and the corrected judgment results are combined as the acquisition result output for subsequent voice interaction tasks. Simultaneously, when the compensation reliability is insufficient, this process also triggers a degraded interaction strategy, such as requesting the user to repeat the command or pausing device movement. Furthermore, a feedback update mechanism feeds back the compensated and corrected residual noise, directional stability, and recognition feedback information to the coupling model to update the coupling weights and transfer function parameters.

[0123] Further, please refer to Figure 7 , Figure 7 A schematic diagram illustrating an application scenario for a voice acquisition method is provided, specifically: Figure 7 In the middle (a), the baseline acquisition scene is static. The reference recognition rate and orientation stability of the device are recorded in this scene as a comparison reference for other motion scenes. Figure 7(b) is a scene of head turning or gimbal movement. In this scene, we verify whether the directional jitter caused by gimbal rotation is effectively reduced and whether the wake-up sensitivity is kept stable. Figure 7 (c) represents the chassis walking scenario, in which the stability of the device's near and far field judgment during movement is verified. Figure 7 In the middle (d), a joint motion or robotic arm motion scenario is used to verify whether the mechanical noise generated by the joint motion is effectively compensated. Figure 7 In the middle (e), the scenario is fan operation and casing impact, under which the periodic fan noise and impact casing vibration are effectively suppressed. Figure 7 In scenario (f), voice recording is performed while the speaker plays a prompt tone. In this scenario, the effectiveness of the structural vibration feedback compensation caused by the playback is verified. The above six scenarios only provide examples of six typical sources of self-noise that may occur during equipment operation, and are not limited to applying the solution of this embodiment to other operating conditions.

[0124] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the voice acquisition method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0125] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the voice acquisition method in Embodiment 1 above.

[0126] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0127] like Figure 8As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0128] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0129] The electronic device provided in this application, employing the voice acquisition method described in the above embodiments, can solve the technical problem of poor voice acquisition quality due to the influence of device self-noise. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the voice acquisition method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0130] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0132] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the voice acquisition method in the above embodiments.

[0133] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0135] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: acquire a speech signal acquired by at least one microphone channel; acquire a vibration reference signal acquired by at least one vibration reference channel, wherein the vibration reference signal is used to characterize device self-noise; determine the coupling relationship between the speech signal and the vibration reference signal; modify the speech signal acquisition strategy according to the coupling relationship; and obtain the speech signal acquisition result according to the modified acquisition strategy.

[0136] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0139] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described voice acquisition method, which can solve the technical problem of poor voice acquisition quality due to device self-noise. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the voice acquisition method provided in the above embodiments, and will not be repeated here.

[0140] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A voice acquisition method, characterized in that, The voice acquisition method includes: Acquire the voice signal from at least one microphone channel; Acquire vibration reference signals from at least one vibration reference channel, wherein the vibration reference signals are used to characterize the self-noise of the equipment; Determine the coupling relationship between the speech signal and the vibration reference signal; Based on the coupling relationship, the acquisition strategy for the speech signal is modified, and the acquisition result of the speech signal is obtained according to the modified acquisition strategy.

2. The voice acquisition method as described in claim 1, characterized in that, The step of acquiring the vibration reference signal collected by at least one vibration sensor channel includes: Acquire candidate reference signals, wherein the candidate reference signals include at least one physical vibration sensing signal and / or at least one device status signal; The candidate reference signal is evaluated for reference validity to determine whether the candidate reference signal is related to the device's self-noise. Candidate reference signals that are associated with the self-noise of the device are identified as vibration reference signals.

3. The voice acquisition method as described in claim 1, characterized in that, Before the step of determining the coupling relationship between the speech signal and the vibration reference signal, the method further includes: The speech signal and the vibration reference signal are time-aligned, and the step of determining the coupling relationship between the speech signal and the vibration reference signal is performed based on the time-aligned speech signal and vibration reference signal.

4. The voice acquisition method as described in claim 3, characterized in that, The step of time-aligning the speech signal and the vibration reference signal includes: The time alignment offset is determined based on the timestamp of the voice signal, the timestamp of the vibration reference signal, and the preset vibration transmission delay. Based on the time alignment offset, the speech signal and the vibration reference signal are synchronized by a time window to obtain the time-aligned speech signal and vibration reference signal.

5. The voice acquisition method as described in claim 4, characterized in that, The coupling relationship includes a self-noise coupling component and / or a self-noise confidence level, and the step of determining the coupling relationship between the speech signal and the vibration reference signal includes: Within the synchronized time window, the coupling characteristics of the speech signal and the vibration reference signal in multiple frequency bands are determined, wherein the coupling characteristics include coherence, correlation and / or synchronization strength. When the coupling characteristics meet the preset reference validity conditions, the coupling parameters of the microphone channel and each frequency band are determined, wherein the coupling parameters include coupling weights and / or transfer functions; Based on the coupling parameters, the self-noise coupling component and / or the self-noise confidence level are generated.

6. The voice acquisition method as described in claim 5, characterized in that, The step of determining the coupling characteristics of the speech signal and the vibration reference signal in multiple frequency bands within the synchronized time window includes: Within the time window, the speech signal and the vibration reference signal are processed by framing or windowing respectively to obtain the speech sub-signal and vibration sub-signal within each time sub-window; Each of the speech sub-signals and each of the vibration sub-signals are subjected to frequency domain transformation to obtain the frequency domain components corresponding to each frequency band; The coupling characteristics are determined based on each of the frequency domain components.

7. The voice acquisition method as described in claim 1, characterized in that, The step of modifying the acquisition strategy of the speech signal based on the coupling relationship includes: Based on the coupling relationship, at least one of self-noise coupling component cancellation, frequency band suppression, and beamforming weight adjustment is performed on the speech signal to obtain a compensated speech signal; and / or, Based on the coupling relationship, at least one of the following is corrected: the voice activity detection threshold, the voice wake-up confidence, the azimuth offset of the sound source localization result, and the near-far field judgment result.

8. The voice acquisition method as described in claim 7, characterized in that, The step of obtaining the acquisition result of the speech signal according to the modified acquisition strategy includes: Based on the compensated speech signal, the target speech is determined, and / or, based on at least one of the corrected speech activity detection threshold, speech wake-up confidence, directional offset of sound source localization result, and near-far field judgment result, the speech recognition confidence is determined; The acquisition result includes at least one of the target speech, the corrected speech activity detection threshold, the speech wake-up confidence, the azimuth offset of the sound source localization result, the near and far field judgment result, and the speech recognition confidence.

9. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the voice acquisition method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the voice acquisition method as described in any one of claims 1 to 8.

Images