Sound processing method, wearable device and readable storage medium

Abstract

The invention discloses a sound processing method, wearable equipment and a readable storage medium, and the method comprises the steps: obtaining an original audio signal received by a microphone array, and the original audio signal comprises an audio signal corresponding to at least one sound source; if it is detected that the user turns the head, acquiring head posture information of the user and position information of the microphone array, and performing directional speech enhancement on the original audio signal according to the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, the target sound source is a sound source closest to the front of the head of the user; and playing the target audio signal. According to the sound processing method, directional speech enhancement can be carried out according to the head turning of the user, other sound sources and environmental noise are inhibited, and the auditory experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a sound processing method, a wearable device, and a computer-readable storage medium. Background Art

[0002] At present, most of the existing smart wearable devices (e.g., smart glasses) lack effective sound enhancement functions, while traditional wearable devices (e.g., Bluetooth headsets) are limited by space and hardware processing capabilities and cannot implement advanced audio separation and enhancement algorithms. As a result, wearable devices cannot achieve sound enhancement in noisy environments, meetings, multi-person conversations and other scenarios, reducing the user's auditory experience. Summary of the invention

[0003] The present application provides a sound processing method, a wearable device and a computer-readable storage medium, which can solve the problem that existing wearable devices cannot achieve sound enhancement.

[0004] In the first aspect, the present application also provides a sound processing method, which includes: obtaining an original audio signal received by the microphone array, the original audio signal including an audio signal corresponding to at least one sound source; if it is detected that the user turns his head, obtaining the user's head posture information and the position information of the microphone array, and performing directionally enhanced speech on the original audio signal based on the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, the target sound source being the sound source closest to the front of the user's head; and playing the target audio signal.

[0005] In a second aspect, the present application provides a sound processing method, which is applied to a wearable device, wherein the wearable device includes a microphone array; the method includes: Acquire the user's head posture information, the original audio signal received by the microphone array, and the position information of the microphone array; Performing sound source localization according to the original audio signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source; Performing beam synthesis according to the original audio signal, the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; Performing blind source separation on the original audio signal to obtain an independent audio signal corresponding to each sound source; Sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0006] In a third aspect, the present application further provides a wearable device, the wearable device comprising a memory, a processor, a microphone array and an inertial measurement unit; The memory is used to store computer programs; The microphone array is used to receive audio signals; The inertial measurement unit is used to collect the user's head posture information; The processor is used to execute the computer program and implement the above-mentioned sound processing method when executing the computer program.

[0007] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the sound processing method as described above.

[0008] The present application discloses a sound processing method, a wearable device and a computer-readable storage medium. When detecting that the user turns his head, the user's head posture information and the position information of the microphone array are obtained, and the original audio signal is directional voice enhanced according to the head posture information and the position information. This method can achieve directional voice enhancement based on the user's head rotation behavior and suppress other sound sources and environmental noise. It can be applicable to various application scenarios, can effectively filter out noise, and improve the user's auditory experience. At the same time, it can play a hearing aid role and help users with hearing impairments improve their hearing. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 is a schematic block diagram of the structure of a wearable device provided in an embodiment of the present application; Figure 2 is a schematic flow chart of a sound processing method provided in an embodiment of the present application; Figure 3 is a schematic scene diagram of directional speech enhancement provided in an embodiment of the present application; Figure 4 is a software flow chart of a sound processing method provided in an embodiment of the present application; Figure 5 is a schematic block diagram of a hardware system of a sound processing method provided in an embodiment of the present application; Figure 6is a schematic flow chart of preprocessing an audio signal provided by an embodiment of the present application; Figure 7 is a schematic flow chart of a second sound processing method provided in an embodiment of the present application; Figure 8 is a schematic flow chart of directional speech enhancement provided by an embodiment of the present application; Fig. 9 is a schematic flow chart of a sound source localization provided in an embodiment of the present application; Fig.10 is a schematic flow chart of beamforming provided in an embodiment of the present application; Fig.11 is a schematic flow chart of blind source separation provided in an embodiment of the present application; Fig.12 is a schematic structural diagram of a convolutional neural network model provided in an embodiment of the present application; Fig.13 is a schematic flow chart of sound enhancement provided by an embodiment of the present application; Fig.14 It is a schematic flow chart of the third sound processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0013] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0014] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0015] At present, most of the existing smart wearable devices (e.g., smart glasses) lack effective sound enhancement functions, while traditional wearable devices (e.g., Bluetooth headsets) are limited by space and hardware processing capabilities and cannot implement advanced audio separation and enhancement algorithms. As a result, wearable devices cannot achieve sound enhancement in noisy environments, meetings, multi-person conversations and other scenarios, reducing the user's auditory experience.

[0016] To this end, the embodiments of the present application provide a sound processing method, a wearable device, and a computer-readable storage medium. The sound processing method can be applied to a wearable device, by obtaining the user's head posture information and the position information of the microphone array when detecting that the user turns his head, and performing directional speech enhancement on the original audio signal according to the head posture information and the position information, and obtaining the target audio signal corresponding to the target sound source, which can realize directional speech enhancement based on the user's head rotation behavior and suppress other sound sources and environmental noise, can be applied to various application scenarios, can effectively filter out noise, improve the user's auditory experience, and can also play a hearing aid role, helping users with hearing impairments improve their hearing.

[0017] Exemplarily, wearable devices may include, but are not limited to, smart glasses such as AR (Augmented Reality) glasses and VR (Virtual Reality) glasses, and may also include smart electronic devices such as smart helmets.

[0018] See also Figure 1 , Figure 1 1 is a schematic diagram of a wearable device provided in an embodiment of the present application. The wearable device may include a processor 1001, a memory 1002, a microphone array 1003, and an inertial measurement unit 1004, wherein the processor 1001, the memory 1002, the microphone array 1003, and the inertial measurement unit 1004 may be connected via a bus, and the bus may be any applicable bus such as an Inter-integrated Circuit (I2C) bus.

[0019] The memory 1002 may include a storage medium and an internal memory. The storage medium may be a non-volatile storage medium or a volatile storage medium. The storage medium may store an operating system and a computer program, and the internal memory provides an environment for the operation of the computer program in the storage medium. The computer program includes program instructions, and when the program instructions are executed, the processor may execute the sound processing method described in any embodiment.

[0020] The microphone array 1003 is used to receive audio signals. The microphone array may include microphones, wherein (the present application embodiment takes three microphones as an example for description), the original audio signal collected by each microphone in the microphone array 1003 can be expressed as: ,in Indicates Microphones at all times The signal collected has a sampling frequency of .

[0021] The geometric information of the microphone array 1003 in the world coordinate system (assuming that the center of the head is the origin of the world coordinate system) can be expressed as: ,in Indicates the number of the microphone; the microphone spacing can be expressed as: , Indicates microphone and microphone The distance between.

[0022] Inertial Measurement Unit 1004 (IMU) is used to collect the user's head posture information. It should be noted that the inertial measurement unit 1004 is a device for measuring the three-axis posture angle (or angular velocity) and acceleration of an object. In the embodiment of the present application, the head posture information can be expressed as: ,in, represents the yaw angle, represents the pitch angle, Indicates the roll angle.

[0023] The processor 1001 is used to provide computing and control capabilities to support the operation of the entire wearable device 10.

[0024] The processor 1001 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other types of processors such as programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. A general-purpose processor may be a microprocessor, or a general-purpose processor may be any conventional processor.

[0025] In one embodiment, the processor 1001 is used to run a computer program stored in the memory to implement the following steps: Acquire an original audio signal received by the microphone array, the original audio signal including an audio signal corresponding to at least one sound source; if it is detected that the user turns his head, obtain the user's head posture information and the position information of the microphone array, and perform directionally enhanced speech on the original audio signal based on the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, the target sound source being the sound source closest to the front of the user's head; play the target audio signal.

[0026] In some embodiments, the original audio signal includes an audio signal corresponding to at least one sound source received by at least one microphone channel; before the processor 1001 performs directional speech enhancement on the original audio signal according to the head posture information and the position information to obtain the target audio signal corresponding to the target sound source, it is also used to implement: Perform frequency domain transformation on the audio signal received by each microphone channel to obtain the time-frequency signal corresponding to each microphone channel; combine the time-frequency signals corresponding to each microphone channel to obtain a multi-channel time-frequency signal; perform covariance matrix estimation on the multi-channel time-frequency signal to obtain the covariance matrix estimation result corresponding to the multi-channel time-frequency signal.

[0027] In some embodiments, after the processor 1001 estimates the covariance matrix of the multi-channel time-frequency signal and obtains the covariance matrix estimation result corresponding to the multi-channel time-frequency signal, it is further used to implement: The number of sound sources is estimated according to the covariance matrix estimation result to obtain an estimated value of the number of sound sources.

[0028] In some embodiments, when the processor 1001 performs directional speech enhancement on the original audio signal according to the head posture information and the position information to obtain the target audio signal corresponding to the target sound source, it is used to implement: Sound source localization is performed according to multi-channel time-frequency signals, head posture information and position information to obtain a sound source position estimation value corresponding to at least one sound source; beam synthesis is performed on the multi-channel time-frequency signals according to the head posture information, position information and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; blind source separation is performed on the multi-channel time-frequency signals to obtain an independent audio signal corresponding to each sound source; sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0029] In some embodiments, when the processor 1001 performs sound source localization according to the multi-channel time-frequency signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source, it is used to implement: Determine the time delay estimate corresponding to the arrival time difference of the multi-channel time-frequency signal; based on the Kalman filter, perform sound source positioning according to the time delay estimate, head posture information and position information to obtain the sound source position estimate corresponding to each sound source.

[0030] In some embodiments, when determining the delay estimation value corresponding to the arrival time difference of the multi-channel time-frequency signal according to the multi-channel time-frequency signal, the processor 1001 is used to implement: Construct a cross-correlation function corresponding to the multi-channel time-frequency signal; construct a generalized cross-correlation function according to the Fourier transform of the cross-correlation function; and determine a delay estimation value according to the peak position of the generalized cross-correlation function.

[0031] In some embodiments, the position information includes the first position of the microphone array; when the processor 1001 performs sound source localization based on the Kalman filter according to the delay estimation value, the head posture information and the position information to obtain the sound source position estimation value corresponding to each sound source, it is used to implement: Determine the rotation matrix of the head coordinate system used for head posture information relative to the world coordinate system; according to the rotation matrix, convert the first position from the world coordinate system to the head coordinate system to obtain the second position; input the delay estimate and the second position into the Kalman filter for loop filtering to obtain the state estimate corresponding to each sound source; perform coordinate system conversion on the state estimate corresponding to each sound source to obtain the sound source position estimate corresponding to each sound source.

[0032] In some embodiments, when the processor 1001 performs coordinate system conversion on the state estimation value corresponding to each sound source to obtain the sound source position estimation value corresponding to each sound source, it is used to implement: According to the state estimation value corresponding to each sound source, the first position estimation value of each sound source in the head coordinate system is determined; according to the rotation matrix, the first position estimation value corresponding to each sound source is converted from the head coordinate system to the world coordinate system to obtain the second position estimation value corresponding to each sound source; according to the second position estimation value corresponding to each sound source, the sound source position estimation value corresponding to each sound source is determined.

[0033] In some embodiments, when the processor 1001 performs beam synthesis on the multi-channel time-frequency signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain the beam synthesis signal, it is used to implement: The sound source direction is calculated based on the head posture information and the estimated sound source position corresponding to each sound source to obtain the target sound source direction corresponding to each sound source; beam calculation is performed on the multi-channel time-frequency signal based on the target sound source direction and position information corresponding to each sound source and the covariance matrix estimation result to obtain the beam synthesis signal.

[0034] In some embodiments, the sound source position estimate includes a second position estimate corresponding to each sound source; when the processor 1001 calculates the sound source direction according to the head posture information and the sound source position estimate to obtain the target sound source direction, it is used to implement: The head posture information is used to calculate the direction in front of the head to obtain the direction in front of the head; a target sound source is determined based on the second position estimation value corresponding to each sound source and the direction in front of the head, where the target sound source is a sound source whose angle between the second position estimation value and the direction in front of the head is less than a preset angle; the direction of the target sound source is determined based on the direction of the target sound source relative to the microphone array.

[0035] In some embodiments, when the processor 1001 performs beam calculation on the multi-channel time-frequency signal according to the target sound source direction, position information and covariance matrix estimation result to obtain the beam synthesis signal, it is used to implement: According to the direction and position information of the target sound source, the steering vector is calculated; according to the steering vector and the covariance matrix estimation result, the weight calculation is performed to obtain the beam synthesis weight value; according to the beam synthesis weight value and the multi-channel time-frequency signal, the matrix calculation is performed to obtain the output signal matrix; according to the output signal matrix, the beam synthesis signal is determined.

[0036] In some embodiments, when performing blind source separation on a multi-channel time-frequency signal to obtain an independent audio signal corresponding to each sound source, the processor 1001 is used to implement: Feature extraction is performed on the multi-channel time-frequency signal to obtain Mel-frequency cepstral coefficients; the Mel-frequency cepstral coefficients are input into the trained deep learning model for masking estimation to obtain a first masking estimation value corresponding to each sound source; based on the estimated value of the number of sound sources, the multi-channel time-frequency signal is separated according to the first masking estimation value corresponding to each sound source to obtain an independent audio signal corresponding to each sound source.

[0037] In some embodiments, when the processor 1001 performs signal separation on the multi-channel time-frequency signal based on the estimated value of the number of sound sources and the first masking estimated value corresponding to each sound source to obtain an independent audio signal corresponding to each sound source, it is used to implement: Perform masking expansion on the first masking estimation value corresponding to each sound source to obtain the second masking estimation value after masking expansion of each sound source; superimpose the second masking estimation value corresponding to each sound source on the multi-channel time-frequency signal to obtain the frequency domain estimation signal of each sound source on each microphone channel; perform time domain transformation on the frequency domain estimation signal of each sound source on each microphone channel to obtain the multi-channel time domain signal corresponding to each sound source; determine the independent audio signal corresponding to each sound source according to the multi-channel time domain signal corresponding to each sound source.

[0038] In some embodiments, when the processor 1001 performs sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain the target audio signal, it is configured to implement: A target sound source is determined from at least one sound source according to head posture information and a sound source position estimation value corresponding to at least one sound source; a frequency domain transformation is performed on the independent audio signal corresponding to each sound source to obtain a time-frequency signal of each sound source; a gain calculation is performed according to the beam synthesis signal and the time-frequency signal of each sound source to obtain a soft masking value of each sound source on each microphone channel; a signal synthesis is performed on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal; and a time domain transformation is performed on the single-channel frequency domain total signal to obtain a target audio signal.

[0039] In some embodiments, when the processor 1001 performs gain calculation according to the beam synthesis signal and the time-frequency signal of each sound source to obtain the soft masking value of each sound source on each microphone channel, it is used to implement: The beam synthesis signal and the time-frequency signal of each sound source are input into the trained deep learning model for masking estimation to obtain the time-frequency masking value of each sound source on each microphone channel; based on the multi-channel time-frequency signal and time-frequency masking value corresponding to each sound source signal, the soft masking value of each sound source on each microphone channel is determined.

[0040] In some embodiments, when the processor 1001 performs signal synthesis on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal, it is used to implement: The soft masking values ​​of each sound source on each microphone channel are summed to obtain a single-channel frequency domain signal corresponding to each sound source; the single-channel frequency domain signals corresponding to all sound sources are added to obtain a single-channel total frequency domain signal.

[0041] In some embodiments, when playing the target audio signal, the processor 1001 is used to implement: According to the head posture information and the estimated value of the sound source position corresponding to each sound source, the target audio signal is allocated to the left channel and the right channel of the wearable device; the audio signal on the left channel and the audio signal on the right channel are output to the sound unit of the wearable device respectively.

[0042] In some embodiments, the target audio signal includes an audio signal of at least one sound source; when the processor 1001 allocates the target audio signal to the left channel and the right channel of the wearable device according to the head posture information and the estimated value of the sound source position corresponding to each sound source, it is used to implement: Calculate the azimuth of the sound source position estimation value corresponding to each sound source in the head coordinate system; determine the first gain of each sound source on the left channel and the second gain of each sound source on the right channel according to the azimuth corresponding to each sound source; multiply the audio signal of each sound source by the corresponding first gain of each sound source and add them to the left channel, and multiply the audio signal of each sound source by the corresponding second gain of each sound source and add them to the right channel.

[0043] In some embodiments, the processor 1001 is further configured to implement: The user's head posture information, the original audio signal received by the microphone array, and the position information of the microphone array are obtained; sound source positioning is performed according to the original audio signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source; beam synthesis is performed according to the original audio signal, the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; blind source separation is performed on the original audio signal to obtain an independent audio signal corresponding to each sound source; sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0044] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features of the embodiments can be combined with each other. Figure 2 , Figure 2 is a schematic flow chart of a sound processing method provided in an embodiment of the present application, such as Figure 2 As shown, the sound processing method includes steps S101 to S103.

[0045] Step S101: Acquire an original audio signal received by a microphone array, where the original audio signal includes an audio signal corresponding to at least one sound source.

[0046] Exemplarily, an external audio signal may be received as an original audio signal through a microphone array, wherein the microphone array includes a plurality of microphone channels, and one microphone corresponds to one microphone channel. The original audio signal may include an audio signal corresponding to at least one sound source received by at least one microphone channel. A sound source refers to the source of a sound, such as an object or person that emits a sound.

[0047] Step S102: If it is detected that the user turns his head, the user's head posture information and the position information of the microphone array are obtained, and the original audio signal is directionally enhanced according to the head posture information and the position information to obtain a target audio signal corresponding to the target sound source, where the target sound source is the sound source closest to the front of the user's head.

[0048] For example, an inertial measurement unit may be used to detect whether the user turns his head. When the inertial measurement unit detects a change in the rotation angle of the user's head, it can be confirmed that the user turns his head. For another example, the position information of the microphone array may be detected to determine whether the user turns his head.

[0049] For example, if it is detected that the user turns his head, the user's head posture information and the position information of the microphone array are obtained, and the original audio signal is directional voice enhanced according to the head posture information and the position information to obtain the target audio signal corresponding to the target sound source. The target audio signal can be expressed as Among them, the user's head posture information can be collected through an inertial measurement unit; since the position of the microphone array in the wearable device (for example, a glasses frame) is fixed, if the center of the head is assumed as the origin, the X, Y, and Z coordinates of each microphone in the microphone array are fixed, so the position information of the microphone array can be determined based on the X, Y, and Z coordinates of each microphone in the microphone array.

[0050] See also Figure 3 , Figure 3 is a schematic scene diagram of directional speech enhancement provided by an embodiment of the present application, such as Figure 3 When a user is watching a concert, due to the noisy environment, the original audio signal received by the wearable device contains audio signals from multiple sound sources. At this time, the user can turn his head to listen to the target sound source he wants to hear, for example, the user turns his head to face the actor who is singing or speaking. When the wearable device detects that the user turns his head, it can perform directionally enhanced speech on the original audio signal based on the head posture information and position information, obtain the target audio signal corresponding to the target sound source, and play the target audio signal. In this way, the wearable device can perform directionally enhanced speech on the original audio signal according to the user's head direction, and suppress other sound sources and environmental noise, thereby improving the user's auditory experience.

[0051] Step S103: play the target audio signal.

[0052] Exemplarily, after obtaining the target audio signal corresponding to the target sound source, the target audio signal can be played. For example, the target audio signal can be played through a speaker in the wearable device. In some embodiments, the target audio signal can be amplified by a power amplifier, and then the amplified target audio signal can be played through the speaker.

[0053] The above embodiment obtains the user's head posture information and the position information of the microphone array when detecting that the user turns his head, and performs directionally voice enhancement on the original audio signal according to the head posture information and the position information to obtain the target audio signal corresponding to the target sound source. This can achieve directional voice enhancement based on the user's head rotation behavior and suppress other sound sources and environmental noise. It can be suitable for various application scenarios, can effectively filter out noise, and improve the user's auditory experience. At the same time, it can play a hearing aid role and help users with hearing impairments improve their hearing.

[0054] See also Figure 4 , Figure 4 is a software flow chart of a sound processing method provided in an embodiment of the present application, such as Figure 4 As shown in the figure, the sound processing method mainly includes four parts: sound source localization, beamforming, blind source separation, and sound enhancement and suppression. The following will respectively describe the four parts in detail. In the process of system software design, the low power consumption requirements in actual application scenarios are fully considered. At the same time, the software itself has strong scalability. Some links can be tailored according to the actual hardware performance and application scenarios, and the application scenarios are more flexible and diverse.

[0055] See also Figure 5 , Figure 5 is a schematic block diagram of a hardware system of a sound processing method provided in an embodiment of the present application, such as Figure 5 As shown, the hardware system of the sound processing method may include a microphone array, an inertial measurement unit, a system-on-chip, a digital signal processor, a neural processing unit, and a power amplifier.

[0056] Among them, the microphone array can be a PDM (Pulse Density Modulation) miniature digital microphone, including two or more microphones, for capturing audio signals. The inertial measurement unit contains sensors such as gyroscopes and accelerometers to capture and estimate the user's head movement. The system-on-chip (SoC) is responsible for collecting sensor signals and performing algorithm processing. The digital signal processor (DSP) is mainly responsible for processing various digital signals with a fast processing speed. In the embodiment of the present application, the speed of processing sound signals can be effectively improved by using a digital signal processor to process various digital signals of sound. The digital signal processor can be an external device or a subunit integrated in the SoC. The neural processing unit (NPU) is mainly used to accelerate neural network calculations; the power amplifier (PA) is used to drive the open speaker to sound.

[0057] In the embodiments of the present application, Figure 4 As shown, since the original audio signal is an audio signal in the time domain, in order to facilitate the subsequent algorithm to process the original audio signal, the original audio signal can also be preprocessed before the original audio signal is subjected to directional speech enhancement. The preprocessing mainly includes frequency domain transformation and covariance matrix estimation. Among them, the frequency domain transformation refers to the short-time Fourier transform of the original audio signal to obtain a time-frequency signal, taking into account the characteristics of the time domain and frequency domain. The covariance matrix is ​​used to describe the correlation between the signals received by the microphone array in different channels, reflecting the spatial distribution of the sound source and noise. Exemplarily, the frequency domain transformation mainly adopts the short-time Fourier transform. The frequency domain transformation of the original audio signal will be described in detail below.

[0058] See also Figure 6 , Figure 6 is a schematic flow chart of preprocessing an audio signal provided by an embodiment of the present application, such as Figure 6 As shown, it may include steps S201 to S203.

[0059] Step S201: Perform frequency domain transformation on the audio signal received by each microphone channel to obtain a time-frequency signal corresponding to each microphone channel.

[0060] Exemplarily, when performing frequency domain transformation on the audio signal received by each microphone channel, the audio signal corresponding to each microphone channel may be framed, windowed, and Fourier transformed in sequence to obtain a time-frequency signal corresponding to each microphone channel. The specific steps are as follows: (1) Initialize a preset window function. The preset window function may be a Hanning window function or other types of window functions, which is not limited in this application. This embodiment of the application takes the Hanning window function as an example for explanation. For example, the Hanning window function is expressed as , where the window length is , the frame length is , the frame shift is .

[0061] (2) Divide the audio signal into frames. The audio signal of each microphone channel can be divided into According to the frame length and frame shift Split into multiple overlapping short-time frame signals ,in, represents the timeframe index, represents the sample index within the frame, then .

[0062] (3) Window each short-time frame signal. Multiply by the window function , and get the windowed signal .

[0063] (4) Perform a short-time Fourier transform (STFT) on each frame of the windowed signal to obtain the time-frequency signal corresponding to each microphone channel, which can be expressed as ,in, .

[0064] Step S202: Combine the time-frequency signals corresponding to each microphone channel to obtain a multi-channel time-frequency signal.

[0065] Exemplarily, the time-frequency signals corresponding to each microphone channel can be combined to obtain a multi-channel time-frequency signal. The multi-channel time-frequency signal can be expressed as: .

[0066] In the formula, It indicates the total number of microphone channels. It can be understood that the multi-channel time-frequency signal includes time-frequency signals corresponding to multiple microphone channels.

[0067] Step S203: perform covariance matrix estimation on the multi-channel time-frequency signal to obtain a covariance matrix estimation result corresponding to the multi-channel time-frequency signal.

[0068] Exemplarily, performing covariance matrix estimation on a multi-channel time-frequency signal may include the following steps: (1) Frequency analysis of multi-channel time-frequency signals Averaging to obtain multi-channel time-frequency average signal ， in, is the number of frequency points after STFT transformation.

[0069] (2) Based on the multi-channel time-frequency average signal Calculate the instantaneous covariance matrix, where the instantaneous covariance matrix ,in, represents the conjugate transpose, for The matrix of .

[0070] (3) Based on the recursive averaging method, the instantaneous covariance matrix is ​​estimated to obtain the covariance matrix estimation result. The covariance matrix estimation result can be expressed as ,in for The matrix of For the forgetting factor.

[0071] See also Figure 7 , Figure 7 is a schematic flow chart of a second sound processing method provided in an embodiment of the present application, such as Figure 7 As shown, it may include steps S301 to S304.

[0072] Step S301: Perform frequency domain transformation on the audio signal received by each microphone channel to obtain a time-frequency signal corresponding to each microphone channel.

[0073] Step S302: Combine the time-frequency signals corresponding to each microphone channel to obtain a multi-channel time-frequency signal.

[0074] Step S303: perform covariance matrix estimation on the multi-channel time-frequency signal to obtain a covariance matrix estimation result corresponding to the multi-channel time-frequency signal.

[0075] It can be understood that steps S301 to S303 are the same as the above-mentioned steps S201 to S203, and are not described in detail here.

[0076] Step S304: estimate the number of sound sources according to the covariance matrix estimation result to obtain an estimated value of the number of sound sources.

[0077] In the embodiment of the present application, since there may be multiple sound sources, it is necessary to subsequently perform signal separation on the multiple sound sources, and therefore the number of sound sources needs to be determined.

[0078] like Figure 4 As shown, estimating the number of sound sources based on the covariance matrix estimation result may include the following steps: (1) Estimation results of the covariance matrix Perform eigenvalue decomposition and obtain the eigenvalue . Among them, the eigenvector matrix obtained by decomposition is discarded.

[0079] (2) Calculate the MDL (Minimum description length) criterion value for each possible number of sound sources k. The MDL criterion value is: , In the formula, is the likelihood function, indicating that the hypothesis is When there are multiple sound sources, observe the multi-channel time-frequency signal possibility. is the number of samples, that is, the number of STFT frames.

[0080] in,

[0081] is the degree of freedom of the model, indicating that there is an assumption When there are multiple sound sources, the number of parameters that need to be estimated in the model. .

[0082] (3) Estimation of the number of sound sources. For each frame , choose so that Smallest As an estimate of the number of sound sources in this frame , for multiple frames Perform smoothing to obtain the final estimate of the number of sound sources , follow-up Represents an estimate of the number of sound sources.

[0083] In the above embodiment, by estimating the number of sound sources according to the covariance matrix estimation result, the number of sound sources contained in the audio signal received by the microphone array can be estimated. By estimating the number of sound sources by the MDL criterion based on eigenvalue decomposition, the power consumption is lower and the real-time performance is higher.

[0084] See also Figure 8 , Figure 8 is a schematic flow chart of a directional speech enhancement provided by an embodiment of the present application, such as Figure 8 As shown, in step S102, directional speech enhancement is performed on the original audio signal according to the head posture information and the position information to obtain a target audio signal corresponding to the target sound source, which may include steps S401 to S404.

[0085] Step S401: localize a sound source according to the multi-channel time-frequency signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source.

[0086] In the embodiments of the present application, Figure 4 As shown, by localizing the sound source based on multi-channel time-frequency signals, head posture information, and position information, the position of each sound source in space can be estimated, and combined with the Kalman filter algorithm to provide more accurate direction information for the subsequent beam synthesis step.

[0087] See also Fig. 9 , Fig. 9 is a schematic flow chart of a sound source localization provided by an embodiment of the present application, such as Fig. 9 As shown, the sound source localization is performed according to the multi-channel time-frequency signal, the head posture information and the position information in step S401, which may include step S4011 and step S4012.

[0088] Step S4011: Determine a delay estimation value corresponding to an arrival time difference of a multi-channel time-frequency signal.

[0089] Exemplarily, based on Generalized Cross Correlation Phase Transformation (GCC-PHAT), a delay estimation value corresponding to a time difference of arrival (TDOA) of the multi-channel time-frequency signal may be calculated according to the multi-channel time-frequency signal.

[0090] In some embodiments, determining the delay estimate corresponding to the arrival time difference of the multi-channel time-frequency signal may include: constructing a cross-correlation function corresponding to the multi-channel time-frequency signal; constructing a generalized cross-correlation function based on the Fourier transform of the cross-correlation function; and determining the delay estimate based on the peak position of the generalized cross-correlation function.

[0091] Exemplarily, a multi-channel time-frequency signal can be constructed The time-frequency signal corresponding to each microphone channel in The cross-correlation function can be expressed as: ,in express The complex conjugate of .

[0092] Calculate the cross-correlation function The Fourier transform of : , and construct the generalized cross-correlation function, which can be expressed as: .

[0093] According to the generalized cross-correlation function The peak position (i.e., the maximum value) of is used to determine the delay estimate, where the delay estimate can be expressed as For example, the generalized cross-correlation function The value corresponding to the peak position of is determined as the delay estimation value .

[0094] Step S4012: Based on the Kalman filter, sound source localization is performed according to the time delay estimation value, the head posture information and the position information to obtain the sound source position estimation value corresponding to each sound source.

[0095] Exemplarily, the position information may include the first position of the microphone array. It should be noted that the position information of the microphone array needs to be transformed. In order to distinguish the position information before and after the coordinate transformation, the position of the microphone array before the coordinate transformation can be recorded as the first position. .

[0096] In some embodiments, based on a Kalman filter, sound source positioning is performed according to time delay estimation, head posture information and position information to obtain a sound source position estimation value corresponding to each sound source, which may include: determining a rotation matrix of a head coordinate system adopted by the head posture information relative to a world coordinate system; according to the rotation matrix, converting a first position from a world coordinate system to a head coordinate system to obtain a second position; inputting the time delay estimation value and the second position into a Kalman filter for cyclic filtering processing to obtain a state estimation value corresponding to each sound source; and performing a coordinate system conversion on the state estimation value corresponding to each sound source to obtain a sound source position estimation value corresponding to each sound source.

[0097] Exemplarily, the Kalman filter may be initialized first, and the specific operations include: (1) Construct a state vector for each sound source , both have: ,in is the sound source position, is the sound source velocity (initial value is 0).

[0098] (2) Establish the state transfer matrix based on the sound source motion model: The sound source motion model may include but is not limited to a uniform motion model, a uniformly accelerated motion model, and a random walk model.

[0099] (3) Construct the process noise covariance matrix: .

[0100] (4) Construct the observation matrix: , the covariance matrix of the observation noise: .

[0101] (5) Initialize state estimate And the error covariance matrix .

[0102] For example, the head posture information can be used , calculate head posture information The rotation matrix of the head coordinate system relative to the world coordinate system , the first position Convert from the world coordinate system to the head coordinate system and get the second position as: .

[0103] The delay estimation value and the second position are input into the Kalman filter for loop filtering. , each microphone and each sound source ,The specific process of loop filtering includes prediction calculation and ,result updating.

[0104] The prediction calculation may include the following steps: (1) State prediction: ; (2) Error covariance prediction: ; (3) Calculate the observed value of arrival time difference: ; (4) Calculate the Kalman gain: .

[0105] The result update may include the following steps: (1) Status update: ;in, Represents the state estimate corresponding to the sound source.

[0106] (2) Error covariance update: , ( is the identity matrix).

[0107] Exemplarily, the state estimation value corresponding to each sound source can be calculated through the above calculation formula, and then the state estimation value corresponding to each sound source is transformed into a coordinate system to obtain the sound source position estimation value corresponding to each sound source.

[0108] In some embodiments, a coordinate system transformation is performed on the state estimation value corresponding to each sound source to obtain a sound source position estimation value corresponding to each sound source, which may include: determining a first position estimation value of each sound source in the head coordinate system according to the state estimation value corresponding to each sound source; transforming the first position estimation value corresponding to each sound source from the head coordinate system to the world coordinate system according to the rotation matrix to obtain a second position estimation value corresponding to each sound source; and determining a sound source position estimation value corresponding to each sound source according to the second position estimation value corresponding to each sound source.

[0109] For example, the state estimation value corresponding to each sound source can be Extract the first position estimate of each corresponding sound source in the head coordinate system According to the rotation matrix , the first position estimate corresponding to each sound source The second position estimate corresponding to each sound source is obtained by converting from the head coordinate system to the world coordinate system. The second position estimate can be expressed as Finally, the second position estimate corresponding to each sound source is , is determined as the estimated value of the sound source position corresponding to each sound source, that is, the estimated value of the sound source position can be expressed as .

[0110] In some implementations, the second position estimation values ​​corresponding to each sound source may be fused to obtain the sound source position estimation values ​​corresponding to all sound sources. ,in Indicates A second position estimate of a sound source.

[0111] In the above embodiment, by performing sound source localization according to multi-channel time-frequency signals, head posture information and position information, a sound source position estimation value corresponding to the sound source can be obtained, thereby providing more accurate direction information for subsequent beam synthesis.

[0112] Step S402: beamsynthesize the multi-channel time-frequency signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beamsynthesized signal.

[0113] It should be noted that the main function of beamforming is to perform spatial filtering, optimize the directivity of the array receiving system, and ensure that the beam direction is consistent with the design direction to maximize the signal amplification in the observation direction. In the embodiment of the present application, beamforming can enhance the sound in the direction of the target sound source.

[0114] See also Fig.10 , Fig.10 is a schematic flow chart of beamforming provided in an embodiment of the present application, such as Fig.10 As shown, in step S402, beam synthesis is performed on the multi-channel time-frequency signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source, which may include step S4021 and step S4022.

[0115] Step S4021: Calculate the sound source direction according to the head posture information and the estimated sound source position value corresponding to each sound source to obtain the target sound source direction corresponding to each sound source.

[0116] In some embodiments, the sound source position estimate includes a second position estimate corresponding to each sound source; calculating the sound source direction based on the head posture information and the sound source position estimate to obtain the target sound source direction may include: calculating the front direction of the head based on the head posture information to obtain the front direction of the head; determining the target sound source based on the second position estimate corresponding to each sound source and the front direction of the head, the target sound source being a sound source for which the angle between the second position estimate and the front direction of the head is less than a preset angle; determining the target sound source direction based on the direction of the target sound source relative to the microphone array.

[0117] Exemplarily, the calculation of the front direction of the head may include the following steps: (1) Based on head posture information , calculate the rotation matrix of the head coordinate system in the world coordinate system .

[0118] (2) The direction of the front of the head in the head coordinate system is .

[0119] (3) Convert the forward direction of the head from the head coordinate system to the world coordinate system: .

[0120] Exemplarily, determining the target sound source according to the second position estimation value corresponding to each sound source and the front direction of the head may include: calculating each sound source With the head forward Angle: , Find the sound source with the smallest angle, which is the target sound source .

[0121] Calculate the direction of the target sound source relative to the microphone array to obtain the target sound source direction. For example, when the target sound source is the first microphone in the microphone array, the target sound source direction can be expressed as For another example, when the target sound source is the second microphone of the microphone array, the target sound source direction can be expressed as .

[0122] Step S4022: Perform beamforming calculation on the multi-channel time-frequency signal according to the target sound source direction, position information and covariance matrix estimation result to obtain a beamforming signal.

[0123] In the embodiments of the present application, Figure 4 As shown in the figure, after the sound source direction is calculated based on the head posture information and the estimated value of the sound source position to obtain the target sound source direction, the beamforming weight value can be calculated based on the MVDR (Minimum Variance Distortionless Response) beamformer according to the target sound source direction, position information and covariance matrix estimation result, and the beamforming signal is determined according to the beamforming weight value and the multi-channel time-frequency signal. It should be noted that the beamforming weight value refers to the weight value corresponding to the multi-channel time-frequency signal during beamforming.

[0124] In some embodiments, beam calculation is performed on a multi-channel time-frequency signal according to the target sound source direction, position information and covariance matrix estimation results to obtain a beam synthesis signal, which may include: calculating a steering vector according to the target sound source direction and position information; performing weight calculation according to the steering vector and the covariance matrix estimation results to obtain a beam synthesis weight value; performing matrix calculation according to the beam synthesis weight value and the multi-channel time-frequency signal to obtain an output signal matrix; and determining the beam synthesis signal according to the output signal matrix.

[0125] For example, for each frame and each frequency , according to the direction of the target sound source and the position information of the microphone array , calculate the steering vector : .

[0126] According to the steering vector And the covariance matrix estimation results Perform weight calculation to obtain the beamforming weight value as shown below:

[0127] In the formula, Represents the steering vector The transposed matrix of Represents the covariance matrix estimation result The inverse matrix of .

[0128] According to the beamforming weight value Perform matrix calculation with multi-channel time-frequency signals to obtain the output signal matrix, which can be expressed as ; According to the output signal matrix , which is determined as the beam synthesis signal. The beam synthesis signal can be expressed as .

[0129] The above embodiment, by performing beam synthesis according to the multi-channel time-frequency signal, the head posture information, the position information and the sound source position estimation value corresponding to each sound source, can realize spatial filtering of the multi-channel time-frequency signal, optimize the directivity of the array receiving system, and ensure that the beam direction is consistent with the design direction to maximize the amplification of the signal in the observation direction, thereby enhancing the sound in the direction of the target sound source.

[0130] Step S403: perform blind source separation on the multi-channel time-frequency signal to obtain an independent audio signal corresponding to each sound source.

[0131] It should be noted that the role of blind source separation is to analyze the unobserved original signal from multiple observed mixed signals. Usually, the observed mixed signal comes from the output of multiple sensors, and the output signals of the sensors are independent (linearly uncorrelated). In an embodiment of the present application, by performing blind source separation on a multi-channel time-frequency signal, it is possible to separate the interfering speech in the multi-channel time-frequency signal and suppress the background noise, thereby improving the speech quality.

[0132] See also Fig.11 , Fig.11 is a schematic flow chart of a blind source separation provided by an embodiment of the present application, such as Fig.11As shown, in step S403, blind source separation is performed on the multi-channel time-frequency signal to obtain an independent audio signal corresponding to each sound source, which may include step S4031 and step S4033.

[0133] Step S4031: extract features from the multi-channel time-frequency signal to obtain Mel-frequency cepstrum coefficients.

[0134] In the embodiment of the present application, considering the auditory characteristics of the human ear, Mel-scale Frequency Cepstral Coefficients (MFCC) are used for feature extraction.

[0135] Exemplarily, the specific process of extracting features from multi-channel time-frequency signals is as follows: (1) Calculate the energy spectrum of multi-channel time-frequency signals. Each frame and each audio channel Take the square of the modulus and get the energy spectrum .

[0136] (2) Mel filter preprocessing: a) The number of filters to be constructed is , minimum frequency , Maximum frequency ; b) Convert the lowest and highest frequency points to Mel frequencies : ; c) Divide the Mel frequency range of each filter into points;

[0137]

[0138] d) Convert the Mel frequency points back to linear frequency points:

[0139] (3) Based on , , , design Mel filter bank , indicating the The filter at frequency Response from .

[0140] (4) Constructing Mel filter bank: a) Initialize the filter bank is a zero matrix with shape ,in is the number of STFT points (usually the frame length).

[0141] b) For each filter : i. Select three frequency points ; ii. For each frequency point : 1. If :but ; 2. If :but ; 3. Otherwise: .

[0142] (5) Mel filter: For each time frame and each microphone channel , calculate the Mel filter bank energy .

[0143] .

[0144] (6) Logarithmic transformation: The result after Mel filtering is logarithmically transformed to compress the dynamic range, making the features less sensitive to energy changes and improving the robustness of subsequent processing.

[0145] .

[0146] (7) DCT Discrete Cosine Transform: For each time frame and each audio channel , perform DCT (Discrete Cosine Transform) on the result of logarithmic transformation, and finally obtain Mel frequency cepstrum coefficients ,in , is the number of required Mel-frequency cepstral coefficients (valued according to the feature dimension and the degree of information retention).

[0147] .

[0148] Step S4032: input the Mel-frequency cepstral coefficients into the trained deep learning model for masking estimation to obtain a first masking estimation value corresponding to each sound source.

[0149] In the embodiment of the present application, after feature extraction is performed on the multi-channel time-frequency signal to obtain the Mel-frequency cepstral coefficients, it is also necessary to perform masking estimation based on the Mel-frequency cepstral coefficients.

[0150] It should be noted that the deep learning model can learn complex noise patterns through a large amount of data training, thereby showing better robustness in noisy environments, which is particularly important for the application of smart glasses in noisy environments such as outdoors and public places. Among them, the deep learning model may include but is not limited to CNN (Convolution Neural Network), RNN (Recurrent Neural Networks), BM (Boltzmann machines) and other models. The embodiment of the present application takes the deep learning model as a convolutional neural network model as an example to illustrate how to perform masking estimation.

[0151] See also Fig.12 , Fig.12 is a schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present application. Fig.12 As shown in the figure, the structure of the convolutional neural network model includes: Input layer: using MFCC features as model input.

[0152] CNN layers (the number of layers is determined based on actual effects and computational complexity) can include convolutional layers and pooling layers.

[0153] Convolutional layers: Multiple convolutional layers are used to extract local features in the time and frequency domains. Each convolutional layer is usually followed by a batch normalization layer and a ReLU activation function.

[0154] Pooling layer: After some convolutional layers, pooling layers are used to reduce the dimension of feature maps and increase the robustness of the model.

[0155] Output layer: Dense fully connected layer: maps the features extracted by the CNN layer to Time-frequency masking of a sound source.

[0156] Sigmoid activation function: limits the output value to between 0 and 1 to obtain soft masking.

[0157] The Reshape layer adjusts the output matrix dimension to , in order to communicate with multi-channel audio signals correspond.

[0158] The loss function of the convolutional neural network model is used to calculate the mean square error between the estimated mask output by the model and the true mask. The loss function is as follows: .

[0159] In the formula, is the number of sound sources, is the number of time frames for time-frequency analysis, is the number of filters in the Mel filter bank, is the number of audio channels (this embodiment of the application takes 3 audio channels as an example), It is The sound source in the time frame , frequency point ,aisle The real mask on is the estimated mask output by the convolutional neural network model, denoted as the first mask estimate.

[0160] In the embodiment of the present application, the convolutional neural network model can be iteratively trained in advance to obtain a trained convolutional neural network model. When preparing training data, a large amount of multi-channel mixed speech and clean speech data corresponding to each sound source can be collected according to the actual application scenario; then, the mixed speech and clean speech data are processed in the above steps to obtain the MFCC feature extraction result. , the masked result of the model's output , Feature extraction results of clean speech ;right Perform STFT transformation and get ;according to and Calculate the true mask After obtaining the training data, the convolutional neural network model can be iteratively trained according to the training data until the loss value of the loss function corresponding to the convolutional neural network model no longer decreases or is less than a preset threshold, thereby obtaining a trained convolutional neural network model.

[0161] For example, the Mel frequency cepstral coefficients can be input into the trained convolutional neural network model for masking estimation to obtain a first masking estimation value corresponding to each sound source. The specific process of masking estimation can be found in the above description and will not be described in detail here.

[0162] Step S4033: Based on the estimated value of the number of sound sources, the multi-channel time-frequency signal is separated according to the first masking estimated value corresponding to each sound source to obtain an independent audio signal corresponding to each sound source.

[0163] In an embodiment of the present application, after the Mel-frequency cepstral coefficients are input into the trained deep learning model for masking estimation and the first masking estimation value corresponding to each sound source is obtained, it is necessary to perform signal separation on the multi-channel time-frequency signal according to the first masking estimation value corresponding to each sound source to obtain an independent audio signal corresponding to each sound source.

[0164] In some embodiments, based on an estimated value of the number of sound sources, a multi-channel time-frequency signal is separated according to a first masking estimation value corresponding to each sound source to obtain an independent audio signal corresponding to each sound source, which may include: performing masking expansion on the first masking estimation value corresponding to each sound source to obtain a second masking estimation value after masking expansion for each sound source; superimposing the second masking estimation value corresponding to each sound source onto the multi-channel time-frequency signal to obtain a frequency domain estimation signal of each sound source on each microphone channel; performing a time domain transformation on the frequency domain estimation signal of each sound source on each microphone channel to obtain a multi-channel time domain signal corresponding to each sound source; and determining the independent audio signal corresponding to each sound source based on the multi-channel time domain signal corresponding to each sound source.

[0165] It should be noted that since the input of the convolutional neural network model is MFCC features, its frequency dimension is the number of Mel filter banks. , and the multi-channel time-frequency signal and beamformed signal The frequency dimension is the number of frequency points after STFT transformation , so it is necessary to convert the first masked estimate output by the convolutional neural network model By linear interpolation, it can be expanded to and The same dimension, the second mask estimate after mask expansion is obtained Among them, the specific process of linear interpolation dimension expansion can be found in the relevant technology and will not be described in detail here.

[0166] Exemplarily, for each sound source to be separated and each microphone channel Perform the following processing: Use the soft masking method to mask the expanded time-frequency Superimpose on multi-channel time-frequency signals The frequency domain estimation signal of the sound source on the corresponding microphone channel is obtained. , ,in, Represents element-wise multiplication.

[0167] Exemplarily, the frequency domain estimation signal of each sound source on each microphone channel can be Perform ISTFT (inverse short-time Fourier transform) to obtain the multi-channel time domain signal corresponding to each sound source ; According to the multi-channel time domain signal corresponding to each sound source , determine the independent audio signal corresponding to each sound source, and the independent audio signal can be expressed as .

[0168] The above embodiment can separate the interfering speech in the multi-channel time-frequency signal and suppress the background noise by performing blind source separation on the multi-channel time-frequency signal, thereby improving the speech quality.

[0169] Step S404: Perform sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0170] In the embodiments of the present application, Figure 4 As shown in the figure, although the multi-channel time-frequency signals are processed to a certain extent through beamforming and blind source separation, in order to obtain better speech clarity and intelligibility, sound enhancement and suppression are still required to solve the interference from other directions that the beamforming algorithm cannot completely eliminate, as well as the noise residual and signal distortion problems existing in the blind source separation algorithm.

[0171] See also Fig.13 , Fig.13 is a schematic flow chart of a sound enhancement method provided in an embodiment of the present application, such as Fig.13 As shown, performing sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source in step S404 may include step S4041 and step S4045.

[0172] Step S4041: Determine a target sound source from at least one sound source based on the head posture information and the estimated value of the sound source position corresponding to at least one sound source.

[0173] In some embodiments, the sound source direction can be calculated based on the head posture information and the sound source position estimation value corresponding to each sound source to obtain the target sound source direction corresponding to each sound source, and the target sound source can be determined based on the target sound source direction corresponding to each sound source and the sound source position estimation value corresponding to each sound source.

[0174] For example, when the user wants to hear the sound source in front of the head, the target sound source is the sound source closest to the front of the head. The direction of the front of the head in the world coordinate system can be calculated first. , find the direction in front of the head The estimated value of the sound source position with the smallest angle , the estimated value of the sound source position Corresponding sound source As the target sound source , wherein the specific calculation method is the same as the calculation method for sound source selection in the above step S4021.

[0175] Step S4042: Perform frequency domain transformation on the independent audio signal corresponding to each sound source to obtain a time-frequency signal of each sound source.

[0176] For example, the independent audio signal corresponding to each sound source may be Perform frequency domain transformation to obtain the time-frequency signal of each sound source The frequency domain transformation refers to STFT transformation, and the parameters used in the frequency domain transformation are the same as those used in the frequency domain transformation in the above step S201, and the specific process is not described here.

[0177] Step S4043: Perform gain calculation according to the beam synthesis signal and the time-frequency signal of each sound source to obtain a soft masking value of each sound source on each microphone channel.

[0178] In some embodiments, gain calculation is performed based on the beam synthesis signal and the time-frequency signal of each sound source to obtain the soft masking value of each sound source on each microphone channel, which may include: inputting the beam synthesis signal and the time-frequency signal of each sound source into a trained deep learning model for masking estimation to obtain the time-frequency masking value of each sound source on each microphone channel; determining the soft masking value of each sound source on each microphone channel based on the multi-channel time-frequency signal and the time-frequency masking value corresponding to each sound source signal.

[0179] For example, for the time frame ,frequency and microphone channels , a simplified deep learning model can be used to synthesize the beam signal and the time-frequency signal of each sound source As input, predict the time-frequency masking value of each sound source on each microphone channel Among them, the model structure and training method of the deep learning model are similar to the CNN model used in the blind source segmentation in steps S4031 to S4033, and the parameters affecting power consumption such as the number of layers and the number of convolutions can be simplified according to actual applications.

[0180] Predict the time-frequency masking value of each sound source on each microphone channel Afterwards, the soft masking value of each sound source on each microphone channel can be determined according to the multi-channel time-frequency signal and the time-frequency masking value corresponding to each sound source signal, where the soft masking value can be expressed as , Represents element-wise multiplication.

[0181] Step S4044: perform signal synthesis on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal.

[0182] Exemplarily, after obtaining the soft masking value of each sound source on each microphone channel, signal synthesis may be performed on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal.

[0183] In some embodiments, performing signal synthesis on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal may include: summing the soft masking values ​​of each sound source on each microphone channel to obtain a single-channel frequency domain signal corresponding to each sound source; adding the single-channel frequency domain signals corresponding to all sound sources to obtain a single-channel frequency domain total signal.

[0184] For example, for each sound source , the soft mask value In microphone channel dimensions Sum the above to get the single-channel frequency domain signal of the sound source: The single-channel frequency domain signals of each sound source are added together to obtain the final single-channel frequency domain total signal, where the single-channel frequency domain total signal can be expressed as: .

[0185] Step S4045: perform time domain transformation on the single-channel frequency domain total signal to obtain a target audio signal.

[0186] Exemplarily, a time domain transform is performed on each frame of the single-channel frequency domain total signal, wherein the time domain transform may be an inverse Fourier transform, and the inverse Fourier transform may be performed using the same parameters as the frequency domain transform in step 201 to obtain a windowed time domain signal: .

[0187] In the formula, Indicates the sample index within the frame. The windowed time domain signal is superimposed on the output signal The target audio signal is obtained at the corresponding position of ,in .

[0188] The above-mentioned embodiment can achieve sound enhancement and suppression of audio signals by performing sound enhancement based on the beamforming signal and the independent audio signal corresponding to each sound source, thereby obtaining better speech clarity and intelligibility. At the same time, it can also solve the interference in other directions that cannot be completely eliminated by the beamforming algorithm, as well as the problems of noise residue and signal distortion existing in the blind source separation algorithm.

[0189] In an embodiment of the present application, when the target audio signal is played in step S103, in order to achieve stereo playback, the capacity of the target audio signal can be allocated to the left and right channels of the wearable device according to the sound source position estimation value and the head posture information.

[0190] In some embodiments, playing the target audio signal may include: allocating the target audio signal to the left channel and the right channel of the wearable device according to the head posture information and the estimated value of the sound source position corresponding to each sound source; and outputting the audio signal on the left channel and the audio signal on the right channel to the sound unit of the wearable device respectively.

[0191] Exemplarily, the target audio signal may include an audio signal of at least one sound source. In some embodiments, according to the head posture information and the sound source position estimation value corresponding to each sound source, the target audio signal is allocated to the left channel and the right channel of the wearable device, which may include: calculating the azimuth of the sound source position estimation value corresponding to each sound source in the head coordinate system; according to the azimuth corresponding to each sound source, determining the first gain of each sound source on the left channel and the second gain of each sound source on the right channel; respectively multiplying the audio signal of each sound source by the corresponding first gain of each sound source and superimposing them to the left channel, and respectively multiplying the audio signal of each sound source by the corresponding second gain of each sound source and superimposing them to the right channel.

[0192] Exemplarily, for each sound source corresponding to the sound source position estimate , respectively calculate its azimuth in the head coordinate system When , first estimate the position of each sound source Convert from world coordinate system to head coordinate system: , in is the rotation matrix of the head coordinate system relative to the world coordinate system.

[0193] Exemplarily, for each sound source, the position of the sound source position estimate corresponding to the sound source in the head coordinate system is , calculate the azimuth ,in It is the four-quadrant inverse tangent function, and the return value range is , can be converted to other ranges according to actual application needs, the unit is radian. As shown below: .

[0194] According to the azimuth The calculation result is used to calculate the first gain of each sound source on the left channel. , and a second gain on the right channel , as shown below: ; The gain calculation method may be a linear method or a segmented method. The embodiment of the present application may adopt a segmented method with higher flexibility. It should be noted that the angle of the sound source to the left or right may be determined according to the actual application situation (not necessarily ), the gain value settings of the left and right channels in the same piecewise function should be selected according to the actual situation (users can also be allowed to set it by themselves). Here is just an example.

[0195] Exemplarily, in step S4045, after performing a time domain transformation on the single-channel frequency domain total signal to obtain the target audio signal, the target audio signal can be Multiply by the first gain corresponding to the left channel to obtain the first audio signal and the target audio signal Multiply by the second gain corresponding to the right channel to obtain the second audio signal ; Then the first audio signal and the second audio signal are superimposed on the left and right channels respectively, and are output to the sound generating unit or the power amplifier unit corresponding to the sound generating unit respectively. Figure 5 As shown, the sound-generating unit may include a sound-generating device such as a speaker, and the power amplifier unit may include a power amplifier.

[0196] See also Fig.14 , Fig.14 is a schematic flow chart of a third sound processing method provided in an embodiment of the present application, such as Fig.14 As shown, it may include steps S501 to S505.

[0197] Step S501: Acquire the user's head posture information, the original audio signal received by the microphone array, and the position information of the microphone array.

[0198] Exemplarily, the user's head posture information may be collected through an inertial measurement unit, and an external audio signal may be received through a microphone array as an original audio signal.

[0199] Step S502: localize the sound source according to the original audio signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source.

[0200] Step S503: beamsynthesize the original audio signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beamsynthesized signal.

[0201] Step S504: perform blind source separation on the original audio signal to obtain an independent audio signal corresponding to each sound source.

[0202] Step S505: Perform sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0203] It should be noted that, before step S502, the original audio signal may be preprocessed to obtain a multi-channel time-frequency signal, wherein the specific process of the preprocessing may refer to the above-mentioned steps S201 to S202, which will not be described in detail here. Subsequent steps S502, S503, and S504 may respectively perform sound source localization, beam synthesis, and blind source separation on the multi-channel time-frequency signal, and it can be understood that steps S502 to S504 are the same as the above-mentioned steps S401 to S404, and the specific process may refer to the detailed description of the above-mentioned embodiment, which will not be described in detail here.

[0204] In the above embodiment, by locating the sound source according to the multi-channel time-frequency signal, the head posture information and the position information, the sound source position estimation value corresponding to the sound source can be obtained, and more accurate direction information can be provided for the subsequent beam synthesis. By performing beam synthesis according to the multi-channel time-frequency signal, the head posture information, the position information and the sound source position estimation value corresponding to each sound source, the multi-channel time-frequency signal can be spatially filtered, the directivity of the array receiving system can be optimized, and the beam direction can be guaranteed to be consistent with the design direction to maximize the signal in the observation direction, so that the sound in the direction of the target sound source can be enhanced. By performing blind source separation on the multi-channel time-frequency signal, the interference speech in the multi-channel time-frequency signal can be separated and the background noise can be suppressed, so that the voice quality can be improved. By performing sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source, the sound enhancement and suppression of the audio signal can be achieved, and better speech clarity and intelligibility can be obtained. At the same time, the interference in other directions that cannot be completely eliminated by the beam synthesis algorithm, as well as the noise residual and signal distortion existing in the blind source separation algorithm can also be solved.

[0205] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, which includes program instructions, and the processor executes the program instructions to implement any of the sound processing methods provided in the embodiments of the present application. For example, the computer program is loaded by the processor and can execute the following steps: Acquire an original audio signal received by the microphone array, the original audio signal including an audio signal corresponding to at least one sound source; if it is detected that the user turns his head, obtain the user's head posture information and the position information of the microphone array, and perform directionally enhanced speech on the original audio signal based on the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, the target sound source being the sound source closest to the front of the user's head; play the target audio signal.

[0206] For another example, the computer program is loaded by the processor and may execute the following steps: The user's head posture information, the original audio signal received by the microphone array, and the position information of the microphone array are obtained; sound source positioning is performed according to the original audio signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source; beam synthesis is performed according to the original audio signal, the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; blind source separation is performed on the original audio signal to obtain an independent audio signal corresponding to each sound source; sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

[0207] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0208] The computer-readable storage medium may be an internal storage unit of the wearable device of the aforementioned embodiment, such as a hard disk or memory of the wearable device. The computer-readable storage medium may also be an external storage device of the wearable device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital, SD), a flash card (Flash Card), etc., equipped on the wearable device.

[0209] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A sound processing method, characterized in that: Applied to a wearable device, the wearable device includes a microphone array; the method includes: Acquire an original audio signal received by the microphone array, wherein the original audio signal includes an audio signal corresponding to at least one sound source; If it is detected that the user turns his head, the head posture information of the user and the position information of the microphone array are obtained, and the original audio signal is subjected to directional speech enhancement according to the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, where the target sound source is the sound source closest to the front of the user's head; The target audio signal is played.

2. The sound processing method according to claim 1, characterized in that: The original audio signal includes an audio signal corresponding to at least one sound source received by at least one microphone channel; before performing directional speech enhancement on the original audio signal according to the head posture information and the position information to obtain a target audio signal corresponding to a target sound source, the method further includes: Performing frequency domain transformation on the audio signal received by each microphone channel to obtain a time-frequency signal corresponding to each microphone channel; Combining the time-frequency signals corresponding to each of the microphone channels to obtain a multi-channel time-frequency signal; Perform covariance matrix estimation on the multi-channel time-frequency signal to obtain a covariance matrix estimation result corresponding to the multi-channel time-frequency signal.

3. The sound processing method according to claim 2, characterized in that: The performing directional speech enhancement on the original audio signal according to the head posture information and the position information to obtain a target audio signal corresponding to a target sound source includes: Performing sound source localization according to the multi-channel time-frequency signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source; Performing beam synthesis on the multi-channel time-frequency signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; Performing blind source separation on the multi-channel time-frequency signal to obtain an independent audio signal corresponding to each sound source; Sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

4. The sound processing method according to claim 3, characterized in that: The performing sound source localization according to the multi-channel time-frequency signal, the head posture information and the position information to obtain a sound source position estimation value corresponding to at least one sound source includes: Determine a delay estimation value corresponding to the arrival time difference of the multi-channel time-frequency signal; Based on the Kalman filter, sound source localization is performed according to the time delay estimation value, the head posture information and the position information to obtain the sound source position estimation value corresponding to each sound source.

5. The sound processing method according to claim 4, characterized in that: The step of determining, according to the multi-channel time-frequency signal, a delay estimation value corresponding to the arrival time difference of the multi-channel time-frequency signal comprises: Constructing a cross-correlation function corresponding to the multi-channel time-frequency signal; constructing a generalized cross-correlation function according to the Fourier transform of the cross-correlation function; The delay estimation value is determined according to the peak position of the generalized cross-correlation function.

6. The sound processing method according to claim 4, characterized in that: The position information includes a first position of a microphone array; the method of performing sound source localization based on a Kalman filter according to the time delay estimation value, the head posture information, and the position information to obtain the sound source position estimation value corresponding to each sound source includes: Determine a rotation matrix of a head coordinate system relative to a world coordinate system used by the head posture information; According to the rotation matrix, converting the first position from the world coordinate system to the head coordinate system to obtain a second position; Inputting the time delay estimation value and the second position into the Kalman filter for loop filtering to obtain a state estimation value corresponding to each of the sound sources; The state estimation value corresponding to each of the sound sources is converted into a coordinate system to obtain the sound source position estimation value corresponding to each of the sound sources.

7. The sound processing method according to claim 6, characterized in that: The performing coordinate system conversion on the state estimation value corresponding to each of the sound sources to obtain the sound source position estimation value corresponding to each of the sound sources includes: Determine a first position estimation value of each sound source in the head coordinate system according to the state estimation value corresponding to each sound source; According to the rotation matrix, the first position estimation value corresponding to each of the sound sources is converted from the head coordinate system to the world coordinate system to obtain a second position estimation value corresponding to each of the sound sources; The sound source position estimation value corresponding to each sound source is determined according to the second position estimation value corresponding to each sound source.

8. The sound processing method according to claim 3, characterized in that: The performing beam synthesis on the multi-channel time-frequency signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal includes: Calculate the sound source direction according to the head posture information and the sound source position estimation value corresponding to each sound source to obtain the target sound source direction corresponding to each sound source; The beamforming signal is obtained by performing beamforming calculation on the multi-channel time-frequency signal according to the target sound source direction, the position information and the covariance matrix estimation result corresponding to each sound source.

9. The sound processing method according to claim 8, characterized in that: The sound source position estimation value includes a second position estimation value corresponding to each sound source; The calculating the sound source direction according to the head posture information and the sound source position estimation value to obtain the target sound source direction includes: Calculating the front direction of the head based on the head posture information to obtain the front direction of the head; Determine the target sound source according to the second position estimation value corresponding to each sound source and the direction in front of the head, the target sound source being a sound source for which the angle between the second position estimation value and the direction in front of the head is less than a preset angle; The target sound source direction is determined according to the direction of the target sound source relative to the microphone array.

10. The sound processing method according to claim 8, characterized in that: The performing beam calculation on the multi-channel time-frequency signal according to the target sound source direction, the position information and the covariance matrix estimation result to obtain the beam synthesis signal includes: Calculating a steering vector according to the target sound source direction and the position information; Performing weight calculation according to the steering vector and the covariance matrix estimation result to obtain a beamforming weight value; Performing matrix calculation based on the beamforming weight value and the multi-channel time-frequency signal to obtain an output signal matrix; The beam synthesis signal is determined according to the output signal matrix.

11. The sound processing method according to claim 3, characterized in that: After performing covariance matrix estimation on the multi-channel time-frequency signal to obtain a covariance matrix estimation result corresponding to the multi-channel time-frequency signal, the method further includes: The number of sound sources is estimated according to the covariance matrix estimation result to obtain an estimated value of the number of sound sources.

12. The sound processing method according to claim 11, characterized in that: The step of performing blind source separation on the multi-channel time-frequency signal to obtain an independent audio signal corresponding to each sound source includes: Extracting features of the multi-channel time-frequency signal to obtain Mel-frequency cepstrum coefficients; Inputting the Mel-frequency cepstral coefficients into the trained deep learning model for masking estimation to obtain a first masking estimation value corresponding to each of the sound sources; Based on the estimated value of the number of sound sources, the multi-channel time-frequency signal is subjected to signal separation according to the first masking estimated value corresponding to each of the sound sources to obtain an independent audio signal corresponding to each of the sound sources.

13. The sound processing method according to claim 12, characterized in that: The step of performing signal separation on the multi-channel time-frequency signal based on the estimated value of the number of sound sources and according to the first masking estimated value corresponding to each of the sound sources to obtain an independent audio signal corresponding to each of the sound sources includes: Performing masking expansion on the first masking estimation value corresponding to each of the sound sources to obtain a second masking estimation value after masking expansion of each of the sound sources; Superimposing the second masking estimation value corresponding to each of the sound sources to the multi-channel time-frequency signal to obtain a frequency domain estimation signal of each of the sound sources on each microphone channel; Performing a time domain transformation on the frequency domain estimation signal of each sound source on each microphone channel to obtain a multi-channel time domain signal corresponding to each sound source; According to the multi-channel time domain signal corresponding to each of the sound sources, an independent audio signal corresponding to each of the sound sources is determined.

14. The sound processing method according to claim 4, characterized in that: The step of performing sound enhancement according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal includes: Determine the target sound source from at least one of the sound sources according to the head posture information and a sound source position estimation value corresponding to at least one of the sound sources; Performing frequency domain transformation on the independent audio signal corresponding to each sound source to obtain a time-frequency signal of each sound source; Performing gain calculation according to the beam synthesis signal and the time-frequency signal of each sound source to obtain a soft masking value of each sound source on each microphone channel; Performing signal synthesis on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal; Performing a time domain transform on the single-channel frequency domain total signal to obtain the target audio signal.

15. The sound processing method according to claim 14, characterized in that: The step of performing gain calculation according to the beam synthesis signal and the time-frequency signal of each sound source to obtain a soft masking value of each sound source on each microphone channel includes: Input the beam synthesis signal and the time-frequency signal of each of the sound sources into the trained deep learning model for masking estimation, so as to obtain the time-frequency masking value of each of the sound sources on each of the microphone channels; The soft masking value of each sound source on each microphone channel is determined according to the multi-channel time-frequency signal corresponding to each sound source signal and the time-frequency masking value.

16. The sound processing method according to claim 14, characterized in that: The step of performing signal synthesis on the soft masking value of each sound source on each microphone channel to obtain a single-channel frequency domain total signal includes: Sum the soft masking values ​​of each sound source on each microphone channel to obtain a single-channel frequency domain signal corresponding to each sound source; The single-channel frequency domain signals corresponding to all the sound sources are added together to obtain the single-channel frequency domain total signal.

17. The sound processing method according to claim 1, characterized in that: The playing of the target audio signal comprises: Allocate the target audio signal to a left channel and a right channel of the wearable device according to the head posture information and the estimated value of the sound source position corresponding to each of the sound sources; The audio signal on the left channel and the audio signal on the right channel are respectively output to the sound-emitting unit of the wearable device.

18. The sound processing method according to claim 17, characterized in that: The target audio signal includes an audio signal of at least one sound source; and the step of allocating the target audio signal to a left channel and a right channel of the wearable device according to the head posture information and the sound source position estimation value corresponding to each sound source includes: Calculate the azimuth of the sound source position estimation value corresponding to each sound source in the head coordinate system; Determine, according to the azimuth angle corresponding to each of the sound sources, a first gain of each of the sound sources on the left channel and a second gain of each of the sound sources on the right channel; The audio signal of each sound source is multiplied by the first gain of each corresponding sound source and then added to the left channel, and the audio signal of each sound source is multiplied by the second gain of each corresponding sound source and then added to the right channel.

19. A sound processing method, characterized in that: Applied to a wearable device, the wearable device includes a microphone array; the method includes: Acquire the user's head posture information, the original audio signal received by the microphone array, and the position information of the microphone array; Performing sound source localization according to the original audio signal, the head posture information, and the position information to obtain a sound source position estimation value corresponding to at least one sound source; Performing beam synthesis on the original audio signal according to the head posture information, the position information, and the sound source position estimation value corresponding to each sound source to obtain a beam synthesis signal; Performing blind source separation on the original audio signal to obtain an independent audio signal corresponding to each sound source; Sound enhancement is performed according to the beam synthesis signal and the independent audio signal corresponding to each sound source to obtain a target audio signal.

20. A wearable device, characterized in that: The wearable device includes a memory, a processor, a microphone array, and an inertial measurement unit; The memory is used to store computer programs; The microphone array is used to receive audio signals; The inertial measurement unit is used to collect the user's head posture information; The processor is used to implement the sound processing method according to any one of claims 1 to 18 or the sound processing method according to claim 19 when executing the computer program.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the sound processing method according to any one of claims 1 to 18 or the sound processing method according to claim 19 is implemented.

Images