Intelligent earphone multi-channel noise reduction audio processing method and system

Abstract

The embodiment of the application provides a kind of intelligent earphone multi-channel noise reduction audio processing method and system, it is related to multi-channel noise reduction audio processing technical field, method includes obtaining inertial physical signal and multi-channel audio signal and carries out time synchronization processing, obtains the synchronized inertial physical data and the synchronized multi-channel audio data;Relative rotation information is determined based on the synchronized inertial physical data;Based on the synchronized multi-channel audio data, the instantaneous sound source direction observation information of intelligent earphone array is obtained;Analysis direction offset and residual direction offset;The coordinate compensation mapping of instantaneous sound source direction observation information is obtained to obtain the corrected target direction information;According to inertial physical signal, adjusted separation matrix update strategy is obtained;Based on the corrected target direction information and adjusted separation matrix update strategy, multi-channel audio signal is processed to extract target speech.The application can improve the accuracy and robustness of noise reduction processing.

Description

Technical Field

[0001] This application relates to the field of multi-channel noise reduction frequency processing technology, and more specifically, to a method and system for multi-channel noise reduction frequency processing of smart headphones. Background Technology

[0002] In the field of multi-channel noise reduction audio processing, smart headphones typically utilize built-in multi-microphone arrays to establish a spatial sound field model. This model estimates the direction of the target sound source and separates the target speech in complex acoustic environments to achieve noise reduction. In practical use, wearers often converse with a speaker opposite them in places like cafes. If the wearer naturally looks down at their phone on the table, their head will tilt downwards, shifting their gaze, but the relative physical position of their ears and the target speaker remains largely unchanged. This action directly affects the direction estimation of the spatial sound field model and the update of the sound source separation matrix after the head rotation. Specifically, when the wearer tilts their head down, all microphones tilt downwards along with their head, changing the incident angle of the target speaker's sound waves from directly in front to obliquely above in the array coordinate system. However, existing noise reduction methods typically assume the wearer's head remains relatively stable, making it difficult to distinguish between the physical movement of the target sound source and the relative orientation change caused by the wearer's head rotation. Therefore, the spatial sound field model often fails to synchronously perceive this posture change, still assuming the target sound source is in the old directly in front direction, and the separation matrix continues to extract acoustic components based on the old direction parameters. At this point, the actual direction corresponds to the target speaker's legs or the background environment, while the actual target speech from the upper diagonal is treated as lateral interference and suppressed by the model. This existing method is insensitive to minute head movements, which leads to situations like conversations in a coffee shop where the wearer is looking down at their phone. When the head tilts or turns but the spatial sound field model does not perceive this synchronously, the separation matrix continues to lock onto the old direction before the turn, resulting in suppression of the actual target speech and retention of the incorrect direction. Ultimately, this causes significant distortion or loss of the target speech in the noise reduction output. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a multi-channel noise reduction audio processing method and system for smart headphones, aiming to solve the technical problem in the prior art where smart headphones struggle to accurately distinguish between the relative positional changes caused by the movement of the sound source itself and head rotation when the wearer's head posture changes, leading to distortion or loss of the target speech during noise reduction processing.

[0004] In a first aspect, embodiments of this application provide a multi-channel noise reduction frequency processing method for smart headphones, including: Acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and perform time synchronization processing on the inertial physical signals and the multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. Based on the synchronized inertial physical data, the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system is determined; Based on the synchronized multi-channel audio data, the transient sound source direction observation information of the smart headphone array is obtained; Combining the relative rotation information and the transient sound source direction observation information, analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information; When the residual directional offset is less than a preset threshold, coordinate compensation mapping is performed on the transient sound source directional observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target directional information; Based on the inertial physical signal, the update strategy of the separation matrix used to separate the multi-channel audio signal is adjusted to obtain the adjusted separation matrix update strategy; Based on the corrected target direction information and the adjusted separation matrix update strategy, the multi-channel audio signal is processed to extract the target speech.

[0005] According to some embodiments of this application, the step of performing time synchronization processing on the inertial physical signal and the multi-channel audio signal to obtain synchronized inertial physical data and synchronized multi-channel audio data includes: Extract the center absolute timestamp of the multi-channel audio signal; Based on the central absolute timestamp, a backtracking search is performed in the preset inertial data buffer queue to locate the two inertial data samples adjacent to the central absolute timestamp. Based on the linear interpolation algorithm, the transient triaxial acceleration and angular velocity synchronized with the multi-channel audio signal are calculated from the two inertial data samples. When the synchronized transient triaxial acceleration and angular velocity drifts with the underlying clock of the transient triaxial acceleration and angular velocity of the inertial physical signal and the time difference of the drift is greater than a preset synchronization tolerance threshold, the inertial physical signal and the multi-channel audio signal are time-synchronized to obtain synchronized inertial physical data and synchronized multi-channel audio data.

[0006] According to some embodiments of this application, determining the relative rotation information of the coordinate system of the smart earphone array relative to a preset reference coordinate system based on the synchronized inertial physical data includes: Based on the synchronized inertial physics data, the synchronized triaxial acceleration and synchronized triaxial angular velocity are extracted; Based on the synchronous triaxial acceleration, the synchronous triaxial angular velocity, the preset extended Kalman filter, and the preset reference coordinate system, the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system is determined.

[0007] According to some embodiments of this application, determining the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system based on the synchronous triaxial acceleration, the synchronous triaxial angular velocity, the preset extended Kalman filter, and the preset reference coordinate system includes: The synchronous triaxial acceleration and the synchronous triaxial angular velocity are input into a preset extended Kalman filter to obtain real-time attitude quaternions, wherein the real-time attitude quaternions include pitch angle, yaw angle and roll angle; Calculate the relative rotation quaternion between the real-time attitude quaternion and the reference attitude quaternion of the preset reference coordinate system; The relative rotation quaternion is converted into a three-dimensional spatial relative rotation matrix according to the formula of the square sum and cross product of the quaternion components to obtain the relative rotation information.

[0008] According to some embodiments of this application, obtaining transient sound source direction observation information of the smart headphone array based on the synchronized multi-channel audio data includes: The synchronized multi-channel audio data is converted using Fast Fourier Transform to obtain frequency domain data; The cross-power spectral density between the multi-channel audio data is calculated based on the frequency domain data to construct a spatial covariance matrix; A subspace-based multiple signal classification algorithm is used to divide the eigenvectors of the spatial covariance matrix into a signal subspace and a noise subspace. Construct a spatial spectrum function based on the signal subspace and the noise subspace; The global maximum point is determined based on the spatial spectrum function to obtain the transient sound source direction observation information of the smart earphone array, wherein the transient sound source direction observation information of the smart earphone array includes the azimuth and elevation angles corresponding to the global maximum point.

[0009] According to some embodiments of this application, the step of combining the relative rotation information and the transient sound source direction observation information to analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself includes: Obtain the target reference orientation that reflects the smart earphone array after historical smoothing; The expected theoretical direction is obtained by performing matrix multiplication based on the three-dimensional spatial relative rotation matrix in the relative rotation information and the target reference direction. Convert the azimuth and elevation angles corresponding to the global maximum points in the transient sound source direction observation information into three-dimensional unit direction vectors; Calculate the first directional difference between the three-dimensional unit direction vector and the target reference direction, and calculate the second directional difference between the three-dimensional unit direction vector and the expected theoretical direction; The difference between the first directional difference and the second directional difference is determined as the directional offset caused by the attitude change of the smart earphone array in the transient sound source direction observation information; Calculate the dot product between the three-dimensional unit direction vector and the expected theoretical direction, and calculate the inverse cosine value to obtain the angle between the three-dimensional unit direction vector and the expected theoretical direction, so as to obtain the residual direction offset caused by the movement of the sound source itself.

[0010] According to some embodiments of this application, when the residual directional offset is less than a preset threshold, coordinate compensation mapping is performed on the transient sound source direction observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target direction information, including: Based on the directional offset caused by the attitude change of the smart earphone array, the three-dimensional unit directional vector corresponding to the transient sound source directional observation information is rotated in reverse to compensate for the directional offset, and the compensated directional vector is obtained. Extract the compensated azimuth and compensated pitch angles from the compensated direction vector; Based on the compensated azimuth and the compensated pitch angle, coordinate compensation mapping is performed on the transient sound source direction observation information to obtain the corrected target direction information.

[0011] According to some embodiments of this application, adjusting the update strategy of the separation matrix used to separate the multi-channel audio signals based on the inertial physical signal to obtain the adjusted separation matrix update strategy includes: Real-time angular velocity data is extracted based on the inertial physical signal, and the head attitude change rate is calculated based on the real-time angular velocity data; The head posture change rate is input into a preset update control mechanism for processing, and the smooth update step size coefficient of the separation matrix is ​​output. The update strategy of the separation matrix of the multi-channel audio signal is adjusted according to the head pose change rate and the smooth update step size coefficient of the separation matrix to obtain the adjusted separation matrix update strategy.

[0012] According to some embodiments of this application, the step of processing the multi-channel audio signal to extract target speech based on the corrected target direction information and the adjusted separation matrix update strategy includes: The inverse of the spatial covariance matrix after smoothing control update is calculated based on the adjusted separation matrix update strategy. Based on the inverse matrix and the corrected target direction information, minimum variance distortionless response beamforming is performed on the multi-channel audio signal of the current frame to obtain a single-channel target speech signal; The single-channel target speech signal is subjected to post-single-channel noise reduction processing to obtain an enhanced single-channel frequency domain signal; The enhanced single-channel frequency domain signal is used to extract the target speech through inverse fast Fourier transform and overlap addition.

[0013] Secondly, this application also discloses a multi-channel noise reduction frequency processing system for smart headphones, comprising: The processing module is used to acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and to perform time synchronization processing on the inertial physical signals and the multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. The determination module is used to determine the relative rotation information of the coordinate system of the smart earphone array relative to a preset reference coordinate system based on the synchronized inertial physical data. The acquisition module is used to acquire transient sound source direction observation information of the smart headphone array based on the synchronized multi-channel audio data; The analysis module is used to combine the relative rotation information and the transient sound source direction observation information to analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information. The compensation mapping module is used to perform coordinate compensation mapping on the transient sound source direction observation information based on the direction offset caused by the attitude change of the smart earphone array when the residual direction offset is less than a preset threshold, so as to obtain the corrected target direction information. The adjustment module is used to adjust the update strategy of the separation matrix used to separate the multi-channel audio signals according to the inertial physical signal, so as to obtain the adjusted separation matrix update strategy. The generation and extraction module is used to process the multi-channel audio signal to extract the target speech based on the corrected target direction information and the adjusted separation matrix update strategy.

[0014] The technical solution according to the embodiments of this application has at least the following beneficial effects: The multi-channel noise reduction audio processing method for smart headphones disclosed in this application ensures data accuracy by acquiring inertial physical signals of the smart headphone array's attitude change and multi-channel audio signals, and performing time synchronization processing. Based on this, the method can determine the relative rotation information of the smart headphone array based on synchronized inertial physical data, and acquire transient sound source direction observation information based on synchronized multi-channel audio data. Crucially, this application combines relative rotation information and transient sound source direction observation information to accurately analyze the directional offset caused by the smart headphone array's attitude change and the residual directional offset caused by the sound source's own movement in the transient sound source direction observation information. When the residual directional offset is less than a preset threshold, the method performs coordinate compensation mapping on the transient sound source direction observation information based on the directional offset caused by the smart headphone array's attitude change to obtain corrected target direction information. Simultaneously, the update strategy of the separation matrix is ​​adjusted according to the inertial physical signals. Finally, based on the corrected target direction information and the adjusted separation matrix update strategy, the multi-channel audio signals are processed to extract target speech. This application effectively solves the problem in existing technologies where smart headphones struggle to distinguish between the physical movement of the target sound source and the relative orientation change caused by the wearer's head rotation when the wearer's head posture changes. This leads to the spatial sound field model's inability to synchronously perceive posture changes, resulting in the suppression of the true target speech and the retention of incorrect directions, ultimately causing significant distortion or loss of the target speech in the noise reduction output. This application precisely distinguishes between head posture changes and the sound source's own movement, and performs targeted direction compensation and separation matrix update strategies. This enables smart headphones to accurately identify the direction of the target sound source and effectively extract the target speech even when the wearer's head posture changes (such as looking down at a phone). This significantly improves the accuracy and robustness of noise reduction processing, avoids distortion or loss of the target speech, and thus provides a superior audio experience.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0017] Figure 1 This is a flowchart illustrating a multi-channel noise reduction frequency processing method for smart headphones provided in one embodiment of this application; Figure 2 This is a schematic flowchart illustrating the process of determining relative rotation information according to one embodiment of this application; Figure 3 A flowchart illustrating the determination of relative rotation information is provided for another embodiment of this application; Figure 4 This is a schematic diagram of a multi-channel noise reduction frequency processing system for smart headphones provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical methods, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that the meaning of "multiple" (or "more than") in the description of the embodiments of this application refers to two or more, and "greater than," "less than," "exceeding," etc. are understood to exclude the number itself, while "above," "below," "within," etc. are understood to include the number itself. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0020] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: the existence of a alone, the existence of b alone, the existence of c alone, the simultaneous existence of a and b, the simultaneous existence of a and c, the simultaneous existence of b and c, or the simultaneous existence of a, b, and c, where a, b, and c can be single or multiple.

[0021] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0022] The multi-channel noise reduction frequency processing method for smart headphones provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application that implements the multi-channel noise reduction frequency processing method for smart headphones, etc., but is not limited to the above forms.

[0023] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0024] See Figure 1 , Figure 1 This is a flowchart illustrating a multi-channel noise reduction frequency processing method for smart headphones according to an embodiment of this application. The multi-channel noise reduction frequency processing method for smart headphones provided in this embodiment includes, but is not limited to, steps S110 to S170, which will be described in detail below.

[0025] Step S110: Acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and perform time synchronization processing on the inertial physical signals and multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. Step S120: Based on synchronized inertial physical data, determine the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system; Step S130: Based on synchronized multi-channel audio data, obtain transient sound source direction observation information of the smart headphone array; Step S140: Combining relative rotation information and transient sound source direction observation information, analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information. Step S150: When the residual directional offset is less than the preset threshold, coordinate compensation mapping is performed on the transient sound source directional observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target directional information. Step S160: Based on the inertial physical signal, adjust the update strategy of the separation matrix used to separate multi-channel audio signals to obtain the adjusted separation matrix update strategy; Step S170: Based on the corrected target direction information and the adjusted separation matrix update strategy, process the multi-channel audio signal to extract the target speech.

[0026] It should be noted that inertial physical signals refer to data collected by the inertial measurement unit (IMU) built into the smart headphones, such as three-axis acceleration and three-axis angular velocity. These signals reflect the attitude and motion state of the smart headphone array in space. Multi-channel audio signals refer to audio data collected simultaneously by multiple microphones in the smart headphone array. This data includes target speech and ambient noise. The smart headphone array refers to the geometric arrangement of multiple microphones configured on the smart headphones, and its coordinate system changes with wearing the headphones and head movement. The preset reference coordinate system is a relatively stable reference coordinate system, such as the coordinate system when the user's head is facing forward when initially wearing the headphones, or a global fixed coordinate system.

[0027] In one embodiment, it is first necessary to acquire inertial physical signals reflecting the attitude changes of the smart earphone array and multi-channel audio signals. The inertial physical signals can be acquired in real time by an IMU sensor integrated within the smart earphones; for example, the IMU can output triaxial acceleration and triaxial angular velocity data at a high sampling rate (e.g., 100Hz or higher). The multi-channel audio signals are synchronously acquired by multiple microphones in the smart earphone array; for example, the microphone array can acquire audio data at a sampling rate of 44.1kHz or 48kHz. To ensure the accuracy of subsequent processing, the acquired inertial physical signals and multi-channel audio signals need to be time-synchronized to obtain synchronized inertial physical data and synchronized multi-channel audio data. One implementation is to timestamp the inertial physical signals and multi-channel audio signals during data acquisition, and then align them according to these timestamps during processing. The frame center timestamp of the audio signal can be used as a reference to find the inertial data sample closest to that timestamp in the inertial data stream, and interpolation calculations can be performed to obtain inertial data precisely synchronized with the audio frame. Synchronized triaxial acceleration and triaxial angular velocity data can be input into an attitude fusion algorithm, such as an extended Kalman filter (EKF) or a complementary filter, to estimate the attitude quaternions of the smart earphone array in real time. Then, by calculating the relative rotation quaternions between the real-time attitude quaternions and the reference attitude quaternions in a preset reference coordinate system, and converting this into a three-dimensional spatial relative rotation matrix, the relative rotation information can be obtained. Another approach is to utilize a machine learning model, training a large amount of inertial data and corresponding attitude labels to directly predict the relative rotation information from synchronized inertial physical data. A Fast Fourier Transform (FFT) can be performed on synchronized multi-channel audio data to obtain frequency domain data, and then the cross-power spectral density between the multi-channel audio data can be calculated to construct a spatial covariance matrix. Next, methods such as subspace-based multiple signal classification (MUSIC) or generalized sidelobe cancellers (GSC) are used to analyze the eigenvectors of the spatial covariance matrix, thereby constructing a spatial spectral function. By finding the global maximum point of the spatial spectral function, the direction of the transient sound source, such as azimuth and pitch angles, can be determined. As another approach, a deep learning-based sound source localization model can be used to directly predict the direction of the sound source from the frequency domain data.

[0028] In one embodiment, the target reference direction maintained by the smart earphone array after historical smoothing is first obtained. This can be a relatively stable sound source direction estimate over a period of time. Then, the three-dimensional spatial relative rotation matrix in the relative rotation information is applied to the target reference direction to obtain a expected theoretical direction, which reflects the change in sound source direction caused only by head rotation if the sound source is stationary. Next, the azimuth and pitch angles in the transient sound source direction observation information are converted into three-dimensional unit direction vectors. By calculating the first directional difference between this three-dimensional unit direction vector and the target reference direction, and the second directional difference between it and the expected theoretical direction, the difference can be determined as the directional offset caused by the attitude change of the smart earphone array. Simultaneously, by calculating the dot product between the three-dimensional unit direction vector and the expected theoretical direction and calculating the inverse cosine value, the angle between the two can be obtained. This angle is the residual directional offset caused by the movement of the sound source itself. When the residual directional offset is less than a preset threshold, it indicates that the sound source itself has not moved significantly, and the direction change at this time is mainly caused by the attitude change of the smart earphone array. Therefore, it is necessary to perform coordinate compensation mapping on the transient sound source direction observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target direction information. The three-dimensional unit direction vector corresponding to the transient sound source direction observation information can be reverse-rotated to compensate for the influence of head attitude changes, resulting in a compensated direction vector. Then, the compensated azimuth and pitch angles are extracted from the compensated direction vector, and coordinate compensation mapping is performed on the transient sound source direction observation information accordingly to obtain the corrected target direction information. Real-time angular velocity data can be extracted from inertial physics signals, and the head attitude change rate can be calculated. When the head attitude change rate is large, it indicates that the head is moving rapidly. In this case, the update step size of the separation matrix should be appropriately reduced to avoid instability of the separation matrix due to rapid attitude changes. Conversely, when the head attitude change rate is small, the update step size of the separation matrix can be appropriately increased to adapt to environmental changes more quickly. The head attitude change rate is input into a preset update control mechanism for processing, which outputs the smooth update step size coefficient of the separation matrix, and the update strategy of the separation matrix is ​​adjusted accordingly. The inverse of the spatial covariance matrix after smoothing control is calculated based on the adjusted separation matrix update strategy. Then, based on this inverse matrix and the corrected target direction information, minimum variance distortionless response (MVDR) beamforming is performed on the multi-channel audio signal of the current frame to obtain a single-channel target speech signal.

[0029] It should be noted that, in order to improve speech quality, post-processing single-channel noise reduction is performed on the single-channel target speech signal using spectral subtraction or a deep learning noise reduction model to obtain an enhanced single-channel frequency domain signal. Finally, the enhanced single-channel frequency domain signal is used to extract the target speech through inverse fast Fourier transform (IFFT) and overlapping addition.

[0030] It should be noted that this application introduces inertial physical signals and performs time synchronization processing with multi-channel audio signals. This enables the system to simultaneously acquire accurate head posture information and acoustic information, laying the foundation for subsequent posture compensation and direction analysis. This application can more comprehensively perceive the dynamic environment in which the smart headphone array is located. Secondly, based on synchronized inertial physical data, this application can accurately determine the relative rotation information of the smart headphone array's coordinate system relative to a preset reference coordinate system. It can understand the precise posture of the smart headphone array in space in real time, providing a key reference for subsequent direction offset analysis. Furthermore, by combining relative rotation information and transient sound source direction observation information, this application can finely analyze the direction offset caused by the posture change of the smart headphone array and the residual direction offset caused by the movement of the sound source itself in the transient sound source direction observation information. It can intelligently determine whether the current direction change is due to "false movement" caused by the user's head rotation or the real movement of the sound source itself. When the residual direction offset is less than a preset threshold, the system can identify that the head posture change is the main factor, and accordingly perform coordinate compensation mapping on the transient sound source direction observation information to obtain the corrected target direction information. This compensation mechanism ensures that the direction estimation of the target sound source remains accurate even when the head posture changes.

[0031] Furthermore, this application adjusts the update strategy of the separation matrix used to separate multi-channel audio signals based on inertial physical signals. When head posture changes drastically, the update step size of the separation matrix is ​​appropriately adjusted to avoid instability and erroneous updates caused by rapid posture changes. Finally, based on the corrected target direction information and the adjusted separation matrix update strategy, the multi-channel audio signals are processed to extract target speech. With accurate direction information and a stable separation matrix, this application can more effectively perform beamforming and noise reduction processing on multi-channel audio signals, thereby extracting high-quality target speech.

[0032] In some embodiments of this application, to ensure time consistency between inertial physical signals and multi-channel audio signals, precise time synchronization processing is required for these two different data sources. Specifically, the aforementioned time synchronization processing of inertial physical signals and multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data may include the following steps: Extract the center absolute timestamp of the multi-channel audio signal; Based on the center absolute timestamp, a backtracking search is performed in the preset inertial data buffer queue to locate the two inertial data samples adjacent to the center absolute timestamp. The transient triaxial acceleration and angular velocity, synchronized with the multi-channel audio signal, are calculated based on two inertial data samples using a linear interpolation algorithm. When there is a drift between the underlying clock of the synchronized transient triaxial acceleration and angular velocity and the transient triaxial acceleration and angular velocity of the inertial physical signal, and the time difference of the drift is greater than the preset synchronization tolerance threshold, time synchronization processing is performed on the inertial physical signal and the multi-channel audio signal to obtain synchronized inertial physical data and synchronized multi-channel audio data.

[0033] Specifically, when processing multi-channel audio signals, the center absolute timestamp is first extracted. This timestamp represents the precise position of the current audio frame on the global timeline. Then, using this center absolute timestamp, a backtracking search is performed in a pre-defined inertial data buffer queue. The purpose of this backtracking search is to locate the two inertial data samples closest to the audio timestamp—one time point earlier than the audio timestamp and the other later. This ensures that a time window containing inertial data related to the audio signal's time point is found. Once these two adjacent inertial data samples are located, calculations can be performed based on a linear interpolation algorithm. Linear interpolation is a commonly used data interpolation method that estimates the value of unknown data points based on the trends between known data points. Here, by performing linear interpolation on these two inertial data samples, the transient triaxial acceleration and angular velocity synchronized with the center absolute timestamp of the multi-channel audio signal can be accurately calculated. This interpolation method effectively compensates for inconsistencies in sampling rates or sampling times between different sensors, thereby obtaining high-precision time synchronization data. When a drift is detected between the synchronized transient triaxial acceleration and angular velocity and the underlying clock of the transient triaxial acceleration and angular velocity of the inertial physical signal, and the time difference of this drift is greater than a preset synchronization tolerance threshold, the system triggers further time synchronization processing. This processing aims to correct the clock drift and ensure that the inertial physical data and multi-channel audio data maintain a high degree of time consistency throughout the processing, thereby obtaining more accurate synchronized inertial physical data and synchronized multi-channel audio data.

[0034] It should be noted that the solution in this application effectively aligns inertial physical data and audio data in time by using the absolute timestamp of the center of the multi-channel audio signal as a reference, performing precise backtracking search in the inertial data buffer queue, and combining it with a linear interpolation algorithm. This timestamp- and interpolation-based method overcomes the synchronization problem caused by inconsistencies in sampling frequencies and sampling times of different sensors. Furthermore, by introducing a clock drift detection and compensation mechanism, this application can dynamically identify and correct underlying clock differences, ensuring that the inertial physical signal and multi-channel audio signal maintain high-precision synchronization even during long-term operation or in complex environments, providing reliable basic data for subsequent attitude determination and sound source direction analysis.

[0035] See Figure 2 , Figure 2This is a flowchart illustrating the process of determining relative rotation information according to an embodiment of this application. The steps described above, based on synchronized inertial physical data, determine the relative rotation information of the smart earphone array's coordinate system relative to a preset reference coordinate system, including but not limited to steps S210 to S220. Each step will be described in turn below.

[0036] Step S210: Extract synchronous triaxial acceleration and synchronous triaxial angular velocity based on synchronous inertial physics data; Step S220: Determine the relative rotation information of the smart earphone array's coordinate system relative to the preset reference coordinate system based on the synchronous triaxial acceleration, synchronous triaxial angular velocity, preset extended Kalman filter, and preset reference coordinate system.

[0037] The synchronous triaxial acceleration refers to the acceleration data measured in real time in the X, Y, and Z orthogonal directions by accelerometers during the movement of the smart headphone array, reflecting its linear motion. The synchronous triaxial angular velocity refers to the angular velocity data measured in real time in the X, Y, and Z orthogonal directions by gyroscopes, reflecting its rotational motion. This data is time-synchronized inertial physics data, ensuring temporal consistency with the audio data. The preset extended Kalman filter is a nonlinear state estimation algorithm capable of handling nonlinear system models and observation models. Through iterative prediction and update processes, it provides high-precision attitude estimation of the smart headphone array. This filter fuses accelerometer and gyroscope data, effectively suppressing noise and drift, thus providing more stable and accurate attitude information. The preset reference coordinate system can be understood as a fixed reference coordinate system pre-set before the smart headphone array is worn or used; it can be the Earth coordinate system or the initial coordinate system when the user's head is stationary. The attitude changes of the smart headphone array will be described and calculated relative to this reference coordinate system. The proposed solution first extracts synchronous triaxial acceleration and angular velocity from synchronized inertial physical data. These data are direct physical quantities reflecting the attitude changes of a smart headphone array. Subsequently, these extracted acceleration and angular velocity data are input into a preset extended Kalman filter. The extended Kalman filter, leveraging its powerful nonlinear state estimation capabilities, fuses these inertial measurement data and combines them with a preset reference coordinate system to estimate the current attitude of the smart headphone array in real time. By comparing the real-time estimated coordinate system of the smart headphone array with the preset reference coordinate system, the relative rotation information of the smart headphone array's coordinate system relative to the preset reference coordinate system can be accurately determined. This method effectively overcomes the noise and drift problems inherent in single-sensor data, providing stable and high-precision attitude estimation.

[0038] See Figure 3 , Figure 3This is a flowchart illustrating the determination of relative rotation information according to another embodiment of this application. The steps described above, which determine the relative rotation information of the smart earphone array's coordinate system relative to the preset reference coordinate system based on synchronous triaxial acceleration, synchronous triaxial angular velocity, a preset extended Kalman filter, and a preset reference coordinate system, include, but are not limited to, steps S310 to S330. Each step will be described in detail below.

[0039] Step S310: Input the synchronous triaxial acceleration and synchronous triaxial angular velocity into the preset extended Kalman filter to obtain the real-time attitude quaternion, wherein the real-time attitude quaternion includes pitch angle, yaw angle and roll angle; Step S320: Calculate the relative rotation quaternion between the real-time attitude quaternion and the reference attitude quaternion of the preset reference coordinate system; Step S330: Convert the relative rotation quaternion into a three-dimensional relative rotation matrix according to the formula for the sum of squares and cross product of quaternion components to obtain relative rotation information.

[0040] The synchronous triaxial acceleration and angular velocity are raw data collected by the inertial measurement unit (IMU) sensors during the head movement of the smart earphone array wearer, reflecting the instantaneous motion state of the smart earphone array. The preset extended Kalman filter is a nonlinear state estimation algorithm designed to provide a robust and accurate estimate of the smart earphone array's attitude by fusing noise measurement data from the accelerometer and gyroscope. The real-time attitude quaternion is a compact and singularity-free rotation representation that effectively describes the instantaneous attitude of the smart earphone array in three-dimensional space. Its components include pitch, yaw, and roll angles, corresponding to rotations about different axes. Furthermore, the relative rotation quaternion between the real-time attitude quaternion and the reference attitude quaternion in the preset reference coordinate system is calculated to quantify the deviation of the smart earphone array's current attitude from a fixed or known reference attitude. The preset reference coordinate system can be understood as the reference orientation of the smart earphone array during initial wearing or calibration; for example, it can be set as the attitude when the user's head is facing forward. By calculating the relative rotation quaternion, the complex Euler angle transformation problem can be avoided. The relative rotation quaternion is converted into a three-dimensional spatial relative rotation matrix based on the formula for the sum of squares and cross product of quaternion components. The purpose is to transform the abstract quaternion representation into a more intuitive matrix form that facilitates subsequent geometric transformations. The three-dimensional spatial relative rotation matrix can be directly applied to the rotation of three-dimensional vectors, for example, to compensate for offsets caused by attitude changes in sound source direction observation information.

[0041] Specifically, the aforementioned acquisition of transient sound source direction observation information for the smart headphone array based on synchronized multi-channel audio data includes: The synchronous multi-channel audio data is converted using Fast Fourier Transform to obtain frequency domain data; The cross-power spectral density between multi-channel audio data is calculated based on frequency domain data to construct a spatial covariance matrix; A subspace-based multi-signal classification algorithm is adopted to divide the eigenvectors of the spatial covariance matrix into signal subspace and noise subspace; Construct a spatial spectrum function based on the signal subspace and noise subspace; The global maximum point is determined based on the spatial spectrum function to obtain the transient sound source direction observation information of the smart headphone array. The transient sound source direction observation information of the smart headphone array includes the azimuth and elevation angles corresponding to the global maximum point.

[0042] Specifically, the Fast Fourier Transform (FFT) is an algorithm that converts time-domain signals into frequency-domain signals. Its purpose is to transform raw multi-channel audio data from the time dimension to the frequency dimension to facilitate subsequent frequency domain analysis and processing. Through the Fast Fourier Transform, the amplitude and phase information of each channel's audio signal at different frequencies can be obtained, i.e., frequency domain data.

[0043] Cross-Power Spectral Density (CPSD) is an indicator that measures the correlation between two signals in the frequency domain. By calculating the CPSD between multi-channel audio data, a spatial covariance matrix can be constructed. This spatial covariance matrix reflects the spatial correlation between sound wave signals received by different microphone channels and is the basis for sound source localization. Subspace-based Multiple Signal Classification (MUSIC) is a high-resolution sound source localization algorithm. This algorithm performs eigenvalue decomposition on the spatial covariance matrix, dividing its eigenvectors into a signal subspace and a noise subspace. The signal subspace consists of eigenvectors related to the sound source signal, while the noise subspace consists of eigenvectors related to the noise. The spatial spectral function exhibits sharp peaks in the direction of the sound source. By searching for the global maximum point in the spatial spectral function, the azimuth and elevation angles of the sound source can be accurately determined, thus obtaining transient sound source direction observation information for the smart earphone array. The transient sound source direction observation information specifically includes the azimuth and elevation angles corresponding to the global maximum point. These angles characterize the instantaneous position of the sound source in three-dimensional space. Through the aforementioned technical solution, a series of processing steps, including Fast Fourier Transform, cross-power spectral density calculation, and subspace-based multiple signal classification algorithms, can achieve high-precision, high-resolution sound source direction estimation of multi-channel audio signals received by a smart earphone array. This method not only effectively distinguishes between signal and noise but also provides transient sound source azimuth and elevation angle information, offering precise spatial localization for subsequent sound source tracking, noise reduction, and target speech extraction, significantly improving the performance of smart earphones in complex acoustic environments.

[0044] In response, this application further proposes a specific method for analyzing the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information by combining relative rotation information and transient sound source direction observation information, including: Obtain the target reference orientation that reflects the smart earphone array after historical smoothing; The expected theoretical direction is obtained by performing matrix multiplication based on the three-dimensional spatial relative rotation matrix and the target reference direction from the relative rotation information; Convert the azimuth and elevation angles corresponding to the global maximum points in the transient sound source direction observation information into three-dimensional unit direction vectors; Calculate the first directional difference between the three-dimensional unit direction vector and the target reference direction, and calculate the second directional difference between the three-dimensional unit direction vector and the expected theoretical direction; The difference between the first directional difference and the second directional difference is determined as the directional offset caused by the attitude change of the smart earphone array in the transient sound source directional observation information; Calculate the dot product between the three-dimensional unit direction vector and the expected theoretical direction, and calculate the inverse cosine value to obtain the angle between the three-dimensional unit direction vector and the expected theoretical direction, so as to obtain the residual direction offset caused by the movement of the sound source itself.

[0045] Specifically, the analysis first requires obtaining the target reference direction that the smart headphone array maintains after historical smoothing. This target reference direction can be understood as the stable direction obtained after smoothing the sound source direction over a period of time; it represents the expected position of the target sound source when there is no drastic movement or sound source movement. Further, matrix multiplication is performed between the three-dimensional spatial relative rotation matrix in the relative rotation information and the target reference direction to obtain the expected theoretical direction. The three-dimensional spatial relative rotation matrix characterizes the attitude change of the smart headphone array from a preset reference coordinate system to the current moment. By applying it to the target reference direction, the expected position of the sound source in the current smart headphone array coordinate system can be predicted if the sound source is stationary relative to the reference coordinate system but the smart headphone array undergoes an attitude change. Simultaneously, the azimuth and pitch angles corresponding to the global maxima in the transient sound source direction observation information are converted into three-dimensional unit direction vectors. The transient sound source direction observation information is the sound source direction acquired in real time based on multi-channel audio data, usually expressed in the form of azimuth and pitch angles. Converting it to a three-dimensional unit direction vector facilitates vector operations, such as dot product and direction difference calculations. Based on this, the first directional difference between the three-dimensional unit direction vector and the target reference direction is calculated, as well as the second directional difference between the three-dimensional unit direction vector and the expected theoretical direction. The first directional difference reflects the total deviation between the currently observed sound source direction and the historically smoothed target reference direction, including the effects of both the smart earphone array's attitude change and the sound source's own movement. The second directional difference reflects the deviation between the currently observed sound source direction and the position the sound source should be in after the smart earphone array's attitude change (expected theoretical direction), which is mainly caused by the sound source's own movement. Subsequently, the difference between the first and second directional differences is determined as the directional offset caused by the smart earphone array's attitude change in the transient sound source direction observation information. Through this difference calculation, the directional offset caused solely by the smart earphone array's attitude change can be effectively isolated. This angle directly quantifies the angular deviation between the current observed direction and the expected direction of the sound source after the smart earphone array's attitude change, thus accurately reflecting the directional offset caused by the sound source's own movement.

[0046] Through the above technical solution, this application can accurately distinguish between the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information. This accurate distinction avoids the problem of traditional methods conflating all directional changes, significantly improving the accuracy of direction analysis. Therefore, targeted compensation can be made for the attitude change of the smart earphone array, while simultaneously identifying the actual movement of the sound source. This provides more reliable and refined data support for subsequent coordinate compensation mapping and separation matrix update strategies, ultimately improving the overall performance of the smart earphone's multi-channel noise reduction audio processing and the clarity of target speech extraction.

[0047] Specifically, when the residual directional offset is less than a preset threshold, coordinate compensation mapping is performed on the transient sound source direction observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target direction information, including: Based on the directional offset caused by the attitude change of the smart earphone array, the three-dimensional unit directional vector corresponding to the transient sound source directional observation information is rotated in reverse to compensate for the directional offset, and the compensated directional vector is obtained. Extract the compensated azimuth and compensated pitch angles from the compensated direction vector; The transient sound source direction observation information is mapped using coordinate compensation based on the compensated azimuth and pitch angles to obtain the corrected target direction information. The direction offset caused by the attitude change of the smart earphone array refers to the direction offset determined in the above embodiment by calculating the difference between the first and second direction differences. This offset reflects the deviation of the sound source direction observation information caused by the attitude change of the smart earphone array itself. Reverse rotation compensation is performed on the three-dimensional unit direction vector corresponding to the transient sound source direction observation information. This involves using the aforementioned direction offset caused by the attitude change of the smart earphone array to rotate the currently observed sound source direction vector in the opposite direction and angle. The purpose is to eliminate or counteract the influence of the smart earphone array attitude change on the sound source direction observation information, making the compensated direction vector closer to the true direction of the sound source. The compensated direction vector is the three-dimensional unit direction vector after reverse rotation compensation processing; it represents the corrected direction of the sound source in three-dimensional space after excluding the influence of the smart earphone array attitude change. The compensated azimuth and pitch angles refer to the angle information on the horizontal and vertical planes resolved from the compensated direction vector. These angular information represent the corrected target direction information. Coordinate compensation mapping refers to applying the compensated azimuth and pitch angles corresponding to the direction vector after reverse rotation compensation to the transient sound source direction observation information, thereby generating corrected target direction information. This mapping process ensures that the final output target direction information is a more accurate sound source direction corrected for attitude changes.

[0048] This application's solution focuses on compensating for directional shifts caused by changes in the attitude of the smart earphone array when the residual directional offset is less than a preset threshold, i.e., when the sound source's own movement is insignificant. Specifically, firstly, the directional offset caused by the attitude change of the smart earphone array is used to perform reverse rotation compensation on the three-dimensional unit direction vector corresponding to the transient sound source direction observation information. The core of this step is to reverse the observation error introduced by the motion of the smart earphone array, thereby correcting the observed direction of the sound source in three-dimensional space. Subsequently, the compensated azimuth and pitch angles are extracted from the compensated three-dimensional direction vector. These angles directly reflect the sound source's corrected spatial position. Finally, based on these compensated angle information, coordinate compensation mapping is performed on the transient sound source direction observation information, thereby obtaining the corrected target direction information. The entire process ensures that when the sound source is relatively stationary or moving slowly, the attitude change of the smart earphone array does not cause significant errors in sound source direction recognition, thus improving the accuracy and stability of direction estimation.

[0049] In response, this application further proposes the aforementioned update strategy for adjusting the separation matrix used to separate multi-channel audio signals based on inertial physical signals, specifically including: Real-time angular velocity data is extracted based on inertial physics signals, and the head attitude change rate is calculated based on the real-time angular velocity data. The head pose change rate is input into a preset update control mechanism for processing, and the smooth update step size coefficient of the separation matrix is ​​output. The update strategy for the separation matrix of the multi-channel audio signal is adjusted based on the head pose change rate and the smooth update step size coefficient of the separation matrix, resulting in the adjusted separation matrix update strategy.

[0050] Specifically, inertial physical signals are typically acquired by the inertial measurement unit integrated within the smart headphones, containing data such as three-axis acceleration and three-axis angular velocity. Real-time angular velocity data refers to the speed information of the smart headphone array's rotation in space within the current time frame, usually expressed in radians per second or degrees per second. The head posture change rate is calculated based on this real-time angular velocity data, reflecting the dynamic movement degree and speed of the smart headphone array, i.e., the user's head. For example, the head posture change rate can be estimated by integrating the real-time angular velocity data or averaging it over a certain time window. The preset update control mechanism can be understood as an algorithm or model designed to dynamically adjust the update behavior of the separation matrix based on the input head posture change rate. This mechanism outputs a smooth update step size coefficient for the separation matrix based on the magnitude of the head posture change rate. The smooth update step size coefficient is a value between 0 and 1, used to control the magnitude of change of the separation matrix in each iteration update. For example, when the head pose change rate is high, this coefficient might be set to a smaller value to slow down the update speed of the separation matrix and avoid transient errors caused by rapid movement; conversely, when the head pose change rate is low, this coefficient might be set to a larger value to accelerate the convergence of the separation matrix. Adjusting the update strategy for the separation matrix of multi-channel audio signals refers to dynamically modifying the algorithm parameters used to update the separation matrix based on the head pose change rate and the smooth update step size coefficient of the separation matrix. For example, in separation algorithms based on adaptive beamforming or independent component analysis, the update of the separation matrix typically involves a learning rate or step size parameter. By combining the smooth update step size coefficient with this learning rate, fine control over the update speed of the separation matrix can be achieved, making it better adaptable to the dynamic pose of the smart headphone array.

[0051] It should be noted that the solution proposed in this application effectively addresses the problem of untimely or unstable updates to the separation matrix of intelligent headphone arrays in dynamic environments by introducing the head posture change rate as a key parameter for adjusting the separation matrix update strategy. Specifically, when the intelligent headphone array, i.e., the user's head, undergoes rapid rotation or posture changes, the real-time angular velocity data in the inertial physical signal increases significantly, thereby increasing the calculated head posture change rate. This head posture change rate is input into a preset update control mechanism, which intelligently outputs a smooth update step size coefficient based on the severity of the head posture change. For example, when the head posture change is drastic, this coefficient is set to a smaller value to slow down the update speed of the separation matrix, avoiding inaccurate sound source direction estimation due to rapid head movement, which could lead to erroneous updates of the separation matrix and distortion of the target speech. Conversely, when the head posture change is gradual or static, this coefficient is set to a larger value, enabling the separation matrix to converge to the optimal state more quickly, thereby improving the efficiency and accuracy of target speech extraction. Through this dynamic adjustment mechanism, the separation matrix update strategy can adapt to the motion state of the smart headphone array in real time, ensuring stable noise reduction and voice separation performance in various motion scenarios.

[0052] Specifically, the steps for processing multi-channel audio signals to extract target speech include: The inverse of the spatial covariance matrix after smoothing control is calculated based on the adjusted separation matrix update strategy. Based on the inverse matrix and the corrected target direction information, minimum variance distortionless response beamforming is performed on the multi-channel audio signal of the current frame to obtain a single-channel target speech signal. The single-channel target speech signal is subjected to post-single-channel noise reduction processing to obtain an enhanced single-channel frequency domain signal. The enhanced single-channel frequency domain signal is used to extract the target speech through inverse fast Fourier transform and overlap addition.

[0053] The calculation of the inverse of the spatial covariance matrix after smoothing control involves constructing a spatial covariance matrix reflecting the correlation between channels based on the multi-channel audio signals received by the smart earphone array. This matrix update is not instantaneous but is performed through smoothing control based on an adjusted separation matrix update strategy to ensure its stability and adaptability in dynamic environments. The inverse of the smoothed-updated spatial covariance matrix is ​​a key input to the subsequent minimum variance distortionless response beamforming algorithm, used to calculate the optimal beamforming weights. Minimum variance distortionless response beamforming (MVDR) is an adaptive spatial filtering technique. Its core idea is to minimize the total output power of the beamformer while ensuring distortion-free response in the target direction, thereby maximally suppressing noise and interference from non-target directions. The corrected target direction information provides the MVDR algorithm with the precise spatial location of the target sound source, enabling the beamformer to accurately point at the target sound source and form nulls to suppress interference. Thus, the multi-channel input signals are effectively combined into a single-channel signal, in which the target speech is significantly enhanced.

[0054] It should be noted that post-processing single-channel denoising is performed on the single-channel target speech signal to obtain an enhanced single-channel frequency domain signal. Although MVDR beamforming can effectively suppress spatial noise, non-spatial noise, residual noise, or reverberation may still exist. Therefore, further denoising processing of the obtained single-channel signal after beamforming is necessary. This post-processing is usually performed in the frequency domain and can employ various denoising algorithms, such as spectral subtraction, Wiener filtering, or deep learning-based denoising methods, to further improve the signal-to-noise ratio and clarity of the speech. The enhanced single-channel frequency domain signal is then used to extract the target speech through inverse fast Fourier transform (IFFT) and overlap-addition (OLA). Inverse fast Fourier transform (IFFT) is a standard method for converting a frequency-domain processed signal back to the time domain. Since audio signals are usually processed in frames, to avoid discontinuities between frames and ensure the smoothness of the output speech, overlap-addition (OLA) is used to smoothly stitch adjacent processed time-domain frames together, thereby reconstructing a continuous and complete time-domain target speech signal.

[0055] See Figure 4 , Figure 4 This is a schematic diagram of a multi-channel noise reduction frequency processing system for smart headphones according to an embodiment of this application. The multi-channel noise reduction frequency processing system 400 for smart headphones includes: The processing module 410 is used to acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and to perform time synchronization processing on the inertial physical signals and multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. The determination module 420 is used to determine the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system based on synchronized inertial physical data. The acquisition module 430 is used to acquire transient sound source direction observation information of the smart headphone array based on synchronized multi-channel audio data; Analysis module 440 is used to combine relative rotation information and transient sound source direction observation information to analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information. The compensation mapping module 450 is used to perform coordinate compensation mapping on the transient sound source direction observation information based on the direction offset caused by the attitude change of the smart earphone array when the residual direction offset is less than a preset threshold, so as to obtain the corrected target direction information. The adjustment module 460 is used to adjust the update strategy of the separation matrix used to separate multi-channel audio signals according to the inertial physical signal, so as to obtain the adjusted separation matrix update strategy. The generation and extraction module 470 is used to process the multi-channel audio signal to extract the target speech based on the corrected target direction information and the adjusted separation matrix update strategy.

[0056] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0057] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0058] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined in this application.

Claims

1. A multi-channel noise reduction frequency processing method for smart headphones, characterized in that, include: Acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and perform time synchronization processing on the inertial physical signals and the multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. Based on the synchronized inertial physical data, the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system is determined; Based on the synchronized multi-channel audio data, the transient sound source direction observation information of the smart headphone array is obtained; Combining the relative rotation information and the transient sound source direction observation information, analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information; When the residual directional offset is less than a preset threshold, coordinate compensation mapping is performed on the transient sound source directional observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target directional information; Based on the inertial physical signal, the update strategy of the separation matrix used to separate the multi-channel audio signal is adjusted to obtain the adjusted separation matrix update strategy; Based on the corrected target direction information and the adjusted separation matrix update strategy, the multi-channel audio signal is processed to extract the target speech.

2. The method according to claim 1, characterized in that, The step of performing time synchronization processing on the inertial physical signal and the multi-channel audio signal to obtain synchronized inertial physical data and synchronized multi-channel audio data includes: Extract the center absolute timestamp of the multi-channel audio signal; Based on the central absolute timestamp, a backtracking search is performed in the preset inertial data buffer queue to locate the two inertial data samples adjacent to the central absolute timestamp. Based on the linear interpolation algorithm, the transient triaxial acceleration and angular velocity synchronized with the multi-channel audio signal are calculated from the two inertial data samples. When the synchronized transient triaxial acceleration and angular velocity drifts with the underlying clock of the transient triaxial acceleration and angular velocity of the inertial physical signal and the time difference of the drift is greater than a preset synchronization tolerance threshold, the inertial physical signal and the multi-channel audio signal are time-synchronized to obtain synchronized inertial physical data and synchronized multi-channel audio data.

3. The method according to claim 1, characterized in that, The determination of the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system based on the synchronized inertial physical data includes: Based on the synchronized inertial physics data, the synchronized triaxial acceleration and synchronized triaxial angular velocity are extracted; Based on the synchronous triaxial acceleration, the synchronous triaxial angular velocity, the preset extended Kalman filter, and the preset reference coordinate system, the relative rotation information of the coordinate system of the smart earphone array relative to the preset reference coordinate system is determined.

4. The method according to claim 3, characterized in that, The step of determining the relative rotation information of the smart earphone array's coordinate system relative to the preset reference coordinate system based on the synchronous triaxial acceleration, the synchronous triaxial angular velocity, the preset extended Kalman filter, and the preset reference coordinate system includes: The synchronous triaxial acceleration and the synchronous triaxial angular velocity are input into a preset extended Kalman filter to obtain real-time attitude quaternions, wherein the real-time attitude quaternions include pitch angle, yaw angle and roll angle; Calculate the relative rotation quaternion between the real-time attitude quaternion and the reference attitude quaternion of the preset reference coordinate system; The relative rotation quaternion is converted into a three-dimensional spatial relative rotation matrix according to the formula of the square sum and cross product of the quaternion components to obtain the relative rotation information.

5. The method according to claim 1, characterized in that, The acquisition of transient sound source direction observation information of the smart headphone array based on the synchronized multi-channel audio data includes: The synchronized multi-channel audio data is converted using Fast Fourier Transform to obtain frequency domain data; The cross-power spectral density between the multi-channel audio data is calculated based on the frequency domain data to construct a spatial covariance matrix; A subspace-based multiple signal classification algorithm is used to divide the eigenvectors of the spatial covariance matrix into a signal subspace and a noise subspace. Construct a spatial spectrum function based on the signal subspace and the noise subspace; The global maximum point is determined based on the spatial spectrum function to obtain the transient sound source direction observation information of the smart earphone array, wherein the transient sound source direction observation information of the smart earphone array includes the azimuth and elevation angles corresponding to the global maximum point.

6. The method according to claim 5, characterized in that, The step of combining the relative rotation information and the transient sound source direction observation information to analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information includes: Obtain the target reference orientation that reflects the smart earphone array after historical smoothing; The expected theoretical direction is obtained by performing matrix multiplication based on the three-dimensional spatial relative rotation matrix in the relative rotation information and the target reference direction. Convert the azimuth and elevation angles corresponding to the global maximum points in the transient sound source direction observation information into three-dimensional unit direction vectors; Calculate the first directional difference between the three-dimensional unit direction vector and the target reference direction, and calculate the second directional difference between the three-dimensional unit direction vector and the expected theoretical direction; The difference between the first directional difference and the second directional difference is determined as the directional offset caused by the attitude change of the smart earphone array in the transient sound source direction observation information; Calculate the dot product between the three-dimensional unit direction vector and the expected theoretical direction, and calculate the inverse cosine value to obtain the angle between the three-dimensional unit direction vector and the expected theoretical direction, so as to obtain the residual direction offset caused by the movement of the sound source itself.

7. The method according to claim 6, characterized in that, When the residual directional offset is less than a preset threshold, coordinate compensation mapping is performed on the transient sound source direction observation information based on the directional offset caused by the attitude change of the smart earphone array to obtain the corrected target direction information, including: Based on the directional offset caused by the attitude change of the smart earphone array, the three-dimensional unit directional vector corresponding to the transient sound source directional observation information is rotated in reverse to compensate for the directional offset, and the compensated directional vector is obtained. Extract the compensated azimuth and compensated pitch angles from the compensated direction vector; Based on the compensated azimuth and the compensated pitch angle, coordinate compensation mapping is performed on the transient sound source direction observation information to obtain the corrected target direction information.

8. The method according to claim 1, characterized in that, The step of adjusting the update strategy of the separation matrix used to separate the multi-channel audio signals based on the inertial physical signal to obtain the adjusted separation matrix update strategy includes: Real-time angular velocity data is extracted based on the inertial physical signal, and the head attitude change rate is calculated based on the real-time angular velocity data; The head posture change rate is input into a preset update control mechanism for processing, and the smooth update step size coefficient of the separation matrix is ​​output. The update strategy of the separation matrix of the multi-channel audio signal is adjusted according to the head pose change rate and the smooth update step size coefficient of the separation matrix to obtain the adjusted separation matrix update strategy.

9. The method according to claim 1, characterized in that, The step of processing the multi-channel audio signal to extract target speech based on the corrected target direction information and the adjusted separation matrix update strategy includes: The inverse of the spatial covariance matrix after smoothing control update is calculated based on the adjusted separation matrix update strategy. Based on the inverse matrix and the corrected target direction information, minimum variance distortionless response beamforming is performed on the multi-channel audio signal of the current frame to obtain a single-channel target speech signal; The single-channel target speech signal is subjected to post-single-channel noise reduction processing to obtain an enhanced single-channel frequency domain signal; The enhanced single-channel frequency domain signal is used to extract the target speech through inverse fast Fourier transform and overlap addition.

10. A multi-channel noise reduction frequency processing system for intelligent headphones, characterized in that, include: The processing module is used to acquire inertial physical signals and multi-channel audio signals that reflect the attitude changes of the smart earphone array, and to perform time synchronization processing on the inertial physical signals and the multi-channel audio signals to obtain synchronized inertial physical data and synchronized multi-channel audio data. The determination module is used to determine the relative rotation information of the coordinate system of the smart earphone array relative to a preset reference coordinate system based on the synchronized inertial physical data. The acquisition module is used to acquire transient sound source direction observation information of the smart headphone array based on the synchronized multi-channel audio data; The analysis module is used to combine the relative rotation information and the transient sound source direction observation information to analyze the directional offset caused by the attitude change of the smart earphone array and the residual directional offset caused by the movement of the sound source itself in the transient sound source direction observation information. The compensation mapping module is used to perform coordinate compensation mapping on the transient sound source direction observation information based on the direction offset caused by the attitude change of the smart earphone array when the residual direction offset is less than a preset threshold, so as to obtain the corrected target direction information. The adjustment module is used to adjust the update strategy of the separation matrix used to separate the multi-channel audio signals according to the inertial physical signal, so as to obtain the adjusted separation matrix update strategy. The generation and extraction module is used to process the multi-channel audio signal to extract the target speech based on the corrected target direction information and the adjusted separation matrix update strategy.

Images