Beam forming method, apparatus, device, and computer readable storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GOERTEK INC
- Filing Date
- 2023-03-30
- Publication Date
- 2026-06-09
Smart Images
Figure CN116320896B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio processing technology, and in particular to a beamforming method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] Microphone arrays are a type of sensor array processing system, and beamforming algorithms based on microphone arrays are a core part of front-end processing in far-field voice interaction. In adaptive beamforming algorithms, such as MVDR (Minimum Variance Distortionless Response), the noise covariance matrix is a very important variable.
[0003] Existing technologies primarily estimate the noise covariance matrix by modeling the noise field, such as using the sinc (Singer model) for scattered noise. While this method is simple, the significant difference between the noise field model and the actual sound field leads to low accuracy in the estimated covariance matrix, thus affecting beamforming performance. Currently, another approach is to calculate the noise covariance matrix based on the assumption that the first few frames of the input audio are noise. However, this assumption is often not met, resulting in low accuracy of the obtained noise covariance matrix and further impacting beamforming performance. Summary of the Invention
[0004] The main objective of this invention is to provide a beamforming method, apparatus, device, and computer-readable storage medium, which aims to improve the accuracy of the noise covariance matrix, thereby improving the beamforming effect.
[0005] To achieve the above objectives, the present invention provides a beamforming method, the beamforming method comprising the following steps:
[0006] The target sound signal of the surrounding external environment is collected through a microphone array;
[0007] Identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene;
[0008] The covariance matrix corresponding to the target sound signal is calculated using a preset calculation model corresponding to the target sound scene;
[0009] The target sound signal is processed using the covariance matrix to obtain the output beam.
[0010] Optionally, when the target sound scene is a quiet scene, the preset calculation model corresponding to the quiet scene is the Singer model, and the step of calculating the covariance matrix corresponding to the target sound signal through the preset calculation model corresponding to the target sound scene includes:
[0011] Determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array;
[0012] The angle matrix is input into the Singer model to calculate the covariance matrix corresponding to the target sound signal.
[0013] Optionally, when the target sound scene is a noisy scene, the preset calculation model corresponding to the noisy scene is a covariance model, and the step of calculating the covariance matrix corresponding to the target sound signal through the preset calculation model corresponding to the target sound scene includes:
[0014] The signal matrix composed of the sound signals collected by each microphone in the microphone array is input into the covariance model to calculate the covariance matrix corresponding to the target sound signal. The covariance model is the product of the sound signal and the transpose and conjugate of the sound signal.
[0015] Optionally, before the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal, the method further includes:
[0016] Detect whether the target sound signal is a speech signal;
[0017] If the target sound signal is not a speech signal, then the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal is performed.
[0018] If the target sound signal is a speech signal, then the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, and the step of processing the target sound signal through the covariance matrix to obtain the output beam is performed.
[0019] Optionally, the step of identifying the target sound scene of the target sound signal includes:
[0020] Calculate the noise estimate of the target sound signal, and calculate the noise level of the target sound signal based on the noise estimate;
[0021] Detect whether the noise level is greater than a preset noise level threshold;
[0022] If the noise level is greater than the noise level threshold, then the target sound scene of the target sound signal is identified as a noisy scene;
[0023] If the noise level is less than or equal to the noise level threshold, then the target sound scene is determined to be a quiet scene.
[0024] Optionally, before the step of detecting whether the noise level is greater than a preset noise level threshold, the method further includes:
[0025] When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, a preset low noise threshold is used as the noise level threshold.
[0026] When the sound scene of the previous frame of the target sound signal is a quiet scene, a preset high noise threshold is used as the noise level threshold.
[0027] Optionally, the step of acquiring the target sound signal of the surrounding external environment through a microphone array includes:
[0028] Audio data of the surrounding environment is collected through a microphone array;
[0029] The audio data is processed into frames to obtain multiple frames of ambient sound signals, and each frame of the ambient sound signal is used as the target sound signal.
[0030] After the step of processing the target sound signal using the covariance matrix to obtain the output beam, the method further includes:
[0031] The final output target beam is obtained by superimposing the output beams corresponding to the target sound signals of each frame.
[0032] To achieve the above objectives, the present invention also provides a beamforming apparatus, the beamforming apparatus comprising:
[0033] The acquisition module is used to acquire target sound signals from the surrounding external environment through a microphone array;
[0034] The determining module is used to identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene;
[0035] The calculation module is used to calculate the covariance matrix corresponding to the target sound signal through a preset calculation model corresponding to the target sound scene;
[0036] The processing module is used to process the target sound signal through the covariance matrix to obtain the output beam.
[0037] To achieve the above objectives, the present invention also provides a beamforming apparatus, the beamforming apparatus comprising: a memory, a processor, and a beamforming program stored in the memory and executable on the processor, wherein the beamforming program, when executed by the processor, implements the steps of the beamforming method as described above.
[0038] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a beamforming program, which, when executed by a processor, implements the steps of the beamforming method as described above.
[0039] In this invention, target sound signals from the surrounding environment are acquired using a microphone array; the target sound scene of the target sound signal is identified, wherein the target sound scene is a quiet scene or a noisy scene; the covariance matrix corresponding to the target sound signal is calculated using a preset calculation model corresponding to the target sound scene; and the output beam is obtained by processing the target sound signal using the covariance matrix.
[0040] Compared to noise field modeling to estimate the covariance matrix and calculating the covariance matrix based on assumed noise, this invention achieves the determination of the noise covariance matrix based on the actual sound scene of the target sound signal, making the obtained noise covariance matrix more accurate, thereby improving the beamforming effect. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;
[0042] Figure 2 This is a schematic flowchart of the first embodiment of the beamforming method of the present invention;
[0043] Figure 3 This is a schematic diagram of scene recognition results according to one embodiment of the present invention;
[0044] Figure 4 This is a comparison diagram of beamforming results according to one embodiment of the present invention;
[0045] Figure 5 This is a schematic diagram of the functional modules of a preferred embodiment of the beamforming apparatus of the present invention.
[0046] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0047] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0048] like Figure 1 As shown, Figure 1This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.
[0049] It should be noted that the beamforming device in the embodiments of the present invention can be a device with a microphone array, such as headphones, head-mounted display devices, speakers, etc., or a device that establishes a communication connection with the device with the microphone array, such as a personal computer, server, etc., without specific limitations.
[0050] like Figure 1 As shown, the beamforming device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0051] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the beamforming device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0052] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a beamforming program. The operating system is a program that manages and controls the device's hardware and software resources, supporting the operation of the beamforming program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the beamforming program stored in the memory 1005 and perform the following operations:
[0053] The target sound signal of the surrounding external environment is collected through a microphone array;
[0054] Identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene;
[0055] The covariance matrix corresponding to the target sound signal is calculated using a preset calculation model corresponding to the target sound scene;
[0056] The target sound signal is processed using the covariance matrix to obtain the output beam.
[0057] Furthermore, when the target sound scene is a quiet scene, the preset calculation model corresponding to the quiet scene is the Singer model, and the operation of calculating the covariance matrix corresponding to the target sound signal through the preset calculation model corresponding to the target sound scene includes:
[0058] Determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array;
[0059] The angle matrix is input into the Singer model to calculate the covariance matrix corresponding to the target sound signal.
[0060] Furthermore, when the target sound scene is a noisy scene, the preset calculation model corresponding to the noisy scene is a covariance model, and the operation of calculating the covariance matrix corresponding to the target sound signal through the preset calculation model corresponding to the target sound scene includes:
[0061] The signal matrix composed of the sound signals collected by each microphone in the microphone array is input into the covariance model to calculate the covariance matrix corresponding to the target sound signal. The covariance model is the product of the sound signal and the transpose and conjugate of the sound signal.
[0062] Furthermore, before the operation of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal, the processor 1001 can also call the beamforming program stored in the memory 1005 to perform the following operations:
[0063] Detect whether the target sound signal is a speech signal;
[0064] If the target sound signal is not a speech signal, then the operation of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal is performed.
[0065] If the target sound signal is a speech signal, then the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, and the operation of processing the target sound signal through the covariance matrix to obtain the output beam is performed.
[0066] Furthermore, the operation of identifying the target sound scene of the target sound signal includes:
[0067] Calculate the noise estimate of the target sound signal, and calculate the noise level of the target sound signal based on the noise estimate;
[0068] Detect whether the noise level is greater than a preset noise level threshold;
[0069] If the noise level is greater than the noise level threshold, then the target sound scene of the target sound signal is identified as a noisy scene;
[0070] If the noise level is less than or equal to the noise level threshold, then the target sound scene is determined to be a quiet scene.
[0071] Furthermore, before the operation of detecting whether the noise level is greater than a preset noise level threshold, the processor 1001 can also call the beamforming program stored in the memory 1005 and perform the following operations:
[0072] When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, a preset low noise threshold is used as the noise level threshold.
[0073] When the sound scene of the previous frame of the target sound signal is a quiet scene, a preset high noise threshold is used as the noise level threshold.
[0074] Furthermore, the operation of acquiring target sound signals from the surrounding external environment via a microphone array includes:
[0075] Audio data of the surrounding environment is collected through a microphone array;
[0076] The audio data is processed into frames to obtain multiple frames of ambient sound signals, and each frame of the ambient sound signal is used as the target sound signal.
[0077] After processing the target sound signal using the covariance matrix to obtain the output beam, the processor 1001 can also call the beamforming program stored in the memory 1005 to perform the following operations:
[0078] The final output target beam is obtained by superimposing the output beams corresponding to the target sound signals of each frame.
[0079] Based on the above structure, various embodiments of the beamforming method are proposed.
[0080] Reference Figure 2 , Figure 2 This is a schematic flowchart of the first embodiment of the beamforming method of the present invention.
[0081] This invention provides embodiments of a beamforming method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order. In this embodiment, the executing entity of the beamforming method can be a device that sets up a microphone array, such as headphones, a head-mounted display device, or a speaker; or it can be a device that establishes a communication connection with the device that sets up the microphone array, such as a personal computer or a server. No limitation is made in this embodiment. For ease of description, the executing entity is omitted from the description of each embodiment. In this embodiment, the beamforming method includes:
[0082] Step S10: Collect target sound signals from the surrounding external environment using a microphone array;
[0083] In this embodiment, sound signals from the external environment where the microphone array is located are collected through a microphone array. The collected sound signals will be referred to as target sound signals for distinction.
[0084] In a specific implementation, the acquired audio data can be used as the target sound signal; or the acquired audio data can be processed into frames, and each frame signal can be used as the target sound signal. The specific settings can be made according to actual needs, and there are no restrictions here.
[0085] Step S20: Identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene;
[0086] In this embodiment, after obtaining the target sound signal, the target sound scene of the target sound signal is identified to determine whether the target sound signal is a noisy scene or a quiet scene.
[0087] Specifically, in one embodiment, scene recognition of the target sound signal can be based on ASC (Acoustic Scenes Classification) and AED (Acoustic Events Detection); in another embodiment, scene recognition can be based on the noise level of the target sound signal. There are no specific limitations here, and the settings can be made according to actual needs.
[0088] Step S30: Calculate the covariance matrix corresponding to the target sound signal using a preset calculation model corresponding to the target sound scene;
[0089] In this embodiment, different sound scenarios correspond to different preset covariance matrix calculation models. This allows for the calculation of the covariance matrix corresponding to the target sound signal based on the specific sound scenario, ensuring the obtained covariance matrix adapts to the actual scene and thus improving the accuracy of the noise covariance matrix and the beamforming processing effect. Specifically, in this embodiment, the covariance matrix corresponding to the target sound signal is calculated using a preset calculation model corresponding to the target sound scenario. Specifically, the covariance matrix is an m*m matrix, where m represents the number of microphones in the microphone array. The diagonal elements of the covariance matrix represent the variance of each microphone itself, and the off-diagonal elements represent the covariance between any two microphones.
[0090] Step S40: Process the target sound signal using the covariance matrix to obtain the output beam.
[0091] In this embodiment, after determining the covariance matrix, the target sound signal is processed using the covariance matrix to obtain a noise-suppressed sound signal, i.e., the output beam. Specifically, in a feasible implementation, when using the minimum variance distortionless response algorithm for beamforming, the specific processing steps can be: calculating the optimal weight vector and the covariance matrix; using the covariance matrix and the optimal weight vector to process the target sound signal to obtain an enhanced and noise-suppressed sound signal in the target direction, i.e., the output beam. The specific process will not be elaborated here.
[0092] Further, in one feasible embodiment, step S10 includes:
[0093] Step S101: Acquire audio data of the surrounding external environment through a microphone array;
[0094] In this embodiment, after the collected audio data of the external environment is processed by frame segmentation, the covariance matrix of the sound signal of each frame is calculated. Compared with calculating the covariance matrix by using the collected audio data as the target sound signal, this embodiment can calculate the covariance matrix that conforms to the actual sound scene in real time, thereby improving the beamforming effect of each frame of sound signal.
[0095] Specifically, in this embodiment, audio data of the surrounding external environment is collected through a microphone array.
[0096] Step S102: Perform frame segmentation processing on the audio data to obtain multiple frames of ambient sound signals, and use each frame of the ambient sound signal as the target sound signal.
[0097] The audio data is processed into frames to obtain multiple frames of sound signals (hereinafter referred to as ambient sound signals for distinction), and each frame of ambient sound signal is used as the target sound signal.
[0098] In a specific implementation, the length of each frame of ambient sound signal and the overlap length between each frame of ambient sound signal can be set as needed. For example, in a feasible implementation, the duration of each frame of ambient sound signal can be set to 7.5ms to 15ms.
[0099] In this embodiment, after step S40, the method further includes:
[0100] Step S50: Superimpose the output beams corresponding to the target sound signals of each frame to obtain the final output target beam.
[0101] In this embodiment, the final output target beam is obtained by superimposing the output beams corresponding to the target sound signals of each frame.
[0102] It should be noted that in this embodiment, each frame of ambient sound signal obtained by frame division is used as the target sound signal, and the covariance matrix of each frame of target sound signal is calculated. Compared with using the collected audio data as the target sound signal to calculate the covariance matrix, this embodiment can calculate the covariance matrix that conforms to the actual sound scene in real time, thereby improving the beamforming effect of each frame of sound signal.
[0103] Furthermore, compared to the current method of not updating the calculated noise covariance matrix, this implementation method updates the covariance matrix in real time during beamforming, making the obtained covariance matrix more accurate and thus improving the beamforming effect.
[0104] In this embodiment, a target sound signal from the surrounding environment is acquired through a microphone array; the target sound scene of the target sound signal is identified, wherein the target sound scene is a quiet scene or a noisy scene; the covariance matrix corresponding to the target sound signal is calculated through a preset calculation model corresponding to the target sound scene; and the target sound signal is processed through the covariance matrix to obtain the output beam.
[0105] Compared to noise field modeling to estimate the covariance matrix and calculating the covariance matrix based on assumed noise, this embodiment determines the noise covariance matrix based on the actual sound scene of the target sound signal, making the obtained noise covariance matrix more accurate and thus improving the beamforming effect.
[0106] Furthermore, based on the first embodiment described above, a second embodiment of the beamforming algorithm of the present invention is proposed. In this embodiment, when the target sound scene is a quiet scene, the preset calculation model corresponding to the quiet scene is the Singer model, and step S30 includes:
[0107] Step S301: Determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array;
[0108] In this embodiment, the target sound scene is determined to be a quiet scene, and the covariance matrix is calculated using a preset calculation model corresponding to the quiet scene.
[0109] Specifically, when the target sound scene is a quiet scene, fixed beamforming is used. To reduce the computational load, this embodiment uses the Singh model as the computational model. The specific formula of the Singh model is as follows:
[0110]
[0111] in, This is a matrix (hereinafter referred to as the angle matrix) composed of the incident angles of the target sound signal into each microphone in the microphone array. Therefore, in this embodiment, it is necessary to determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array. The specific formula for calculating the incident angle is as follows:
[0112]
[0113] Where f is the frequency and c is the wave speed. In one feasible embodiment, d can be the distance between each microphone in the microphone array and a preset reference microphone in the microphone array. In this embodiment, it can be an angle matrix composed of the incident angles after calculating each incident angle. In another feasible embodiment, d can be a distance matrix, where the elements of the distance matrix are the distances between each microphone and the preset reference microphone in the microphone array.
[0114] Step S302: Input the angle matrix into the Singer model to calculate the covariance matrix corresponding to the target sound signal.
[0115] In this embodiment, the angle matrix is input into the Singer model to calculate the covariance matrix corresponding to the target sound signal. The specific formula of the Singer model is as follows:
[0116]
[0117] Furthermore, in one feasible implementation, when the target sound scene is a noisy scene, the preset calculation model corresponding to the noisy scene is a covariance model, and step S30 includes:
[0118] Step S303: Input the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal, wherein the covariance model is the product of the sound signal and the transpose and conjugate of the sound signal;
[0119] In this embodiment, the target sound scene is a noisy scene. The covariance matrix is calculated using a preset covariance model corresponding to the noisy scene. The covariance model is the product of the sound signal and the transpose and conjugate of the sound signal, specifically:
[0120]
[0121] Among them, X N Represents sound signals. This represents the transpose and conjugate of a sound signal.
[0122] Specifically, in this embodiment, the matrix composed of the sound signals collected by each microphone in the microphone array (hereinafter referred to as the signal matrix for distinction) is used as X. N The covariance matrix corresponding to the target sound signal is calculated by inputting the covariance model.
[0123] Furthermore, in one feasible embodiment, before step S303, the method further includes:
[0124] Step S304: Detect whether the target sound signal is a speech signal;
[0125] In this embodiment, when the current sound signal corresponds to a noisy scene, the covariance matrix is updated based on whether the current sound signal is a speech signal, so as to reduce the calculation process and improve the efficiency of beamforming.
[0126] Specifically, the detection process determines whether the target sound signal is a speech signal. In a specific implementation, the detection of whether the target sound signal is a speech signal is based on VAD (Voice Activity Detection). This can be done based on a threshold, a classifier, or a model; no specific limitation is imposed here.
[0127] Furthermore, in a feasible implementation, when performing voice endpoint detection based on a threshold, the detection can be based on the signal variance of the target audio signal. Specifically, the detection process can be: calculating the signal variance of the target audio signal and detecting whether the signal variance is greater than a preset variance threshold. Specifically, the formula for calculating the signal variance of the target audio signal is:
[0128]
[0129] Where N is the number of signal sampling points, y represents the input signal, and μ represents the signal mean;
[0130] If the signal variance is greater than the preset variance threshold, the target sound signal is determined to be unstable, and the signal amplitude consistency within the frame is low. The target sound signal is determined to be a speech signal, that is, vadFlag = 1. If the signal variance is less than or equal to the preset variance threshold, the target sound signal is determined to be stable, and the signal amplitude within the frame is consistent. The target sound signal is determined not to be a speech signal, that is, vadFlag = 1.
[0131] Furthermore, in one feasible implementation, in order to avoid frequent jumps in the speech detection results, a speech hold rule can be set, that is, when the target sound signal is a speech signal (i.e., vadFlag=1), the detection result continues for a preset number of frames.
[0132] It is understandable that when performing speech endpoint detection based on thresholds, speech recognition can also be performed based on other features of the target sound signal, such as signal energy, without any restrictions.
[0133] Step S305: If the target sound signal is not a speech signal, then the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal is executed.
[0134] In this embodiment, if the target sound signal is not a speech signal, it is determined that the covariance matrix needs to be updated. In this case, the signal matrix composed of the sound signals collected by each microphone in the microphone array is input into the covariance model to calculate the covariance matrix corresponding to the target sound signal, thereby updating the covariance matrix. The specific covariance model is as follows:
[0135]
[0136] In this embodiment, X N =Y |hnFlag＝1,vadFlag＝0 That is, X N This is a signal matrix composed of sound signals collected by the microphone in a noisy environment. At this time, the signals in the signal matrix are not speech signals.
[0137] Step S306: If the target sound signal is a speech signal, then the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, and the step of processing the target sound signal through the covariance matrix to obtain the output beam is executed.
[0138] In this embodiment, if the target sound signal is a speech signal, it is determined that there is no need to update the covariance matrix. In this case, the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, so as to process the target sound signal to obtain the output beam.
[0139] It should be noted that updating the covariance matrix when the current sound signal corresponds to a noisy scene and the target sound signal is determined not to be a speech signal can reduce the calculation process and improve the efficiency of beamforming.
[0140] In this embodiment, when the target sound scene is a quiet scene, the angle matrix composed of the incident angles of each microphone in the microphone array is determined; the angle matrix is input into the Singer model to calculate the covariance matrix. This embodiment reduces the calculation process of the covariance matrix and improves the processing efficiency of beamforming.
[0141] Furthermore, based on the first and / or second embodiments described above, a third embodiment of the beamforming algorithm of the present invention is proposed. In this embodiment, step S20 includes:
[0142] Step S201: Calculate the noise estimate of the target sound signal, and calculate the noise level of the target sound signal based on the noise estimate;
[0143] In this embodiment, the sound scene of the target sound signal is determined based on the noise level of the target sound signal. Specifically, this embodiment proposes a noise level factor based on noise estimation, and the specific calculation model for the noise level factor is as follows:
[0144]
[0145] Where w is the summation weight, which is related to the frequency. Since the low-frequency component is more abundant for most noise, w is non-uniformly distributed, and λ is the noise estimate.
[0146] Specifically, the calculation model for the summation weight w is as follows:
[0147]
[0148] Where c1 and c2 are constant coefficients, and K is the number of frequency points.
[0149] Therefore, in this embodiment, a noise estimate of the target sound signal is calculated, and the noise level of the target sound signal is calculated based on the noise estimate. The specific formula for the noise estimate is as follows:
[0150] λ d (k,l+1)=α d (k,l)·λ d (k,l)+(1-α d (k,l))·|Y(k,l)| 2
[0151] Where, α d (k,l)=αd0 +(1-α d0 )·p(k,l),
[0152] α d0 As a constant coefficient, in a feasible implementation, α d0 It can be taken as 0.85; p(k,l) is the probability of speech presence, k is the frequency point number, l is the frame number, and Y(k,l) is the frequency domain amplitude of the input signal.
[0153] Step S202: Detect whether the noise level is greater than a preset noise level threshold;
[0154] Based on the specific calculation model of the noise level factor, it is known that the larger the noise estimate, the larger the noise level factor, and the greater the noise in the corresponding sound signal. Therefore, in this embodiment, a threshold for the noise level that can distinguish between noisy scenes and quiet scenes (hereinafter referred to as the noise level threshold for distinction) is preset, and it is detected whether the noise level is greater than the preset noise level threshold.
[0155] Specifically, in one embodiment, the noise level threshold can be a fixed value; in another embodiment, the noise level threshold can also be a variable value, for example, it can be based on the sound scene changes of the previous frame of the target sound signal. The specific value can be set according to actual needs and is not limited here.
[0156] Step S203: If the noise level is greater than the noise level threshold, then the target sound scene of the target sound signal is identified as a noise scene;
[0157] In this embodiment, if the noise level is greater than the noise level threshold, it is considered that the noise level in the target sound signal is relatively high, and therefore the target sound scene of the target sound signal is determined to be a noise scene.
[0158] Step S204: If the noise level is less than or equal to the noise level threshold, then the target sound scene is determined to be a quiet scene.
[0159] If the noise level is greater than the noise level threshold, it is considered that the noise level in the target sound signal is relatively low, and therefore the target sound scene is determined to be a quiet scene.
[0160] Specifically, refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the scene recognition result according to an embodiment of the present invention. Figure 3 As shown in the figure, this embodiment can accurately identify quiet scenes and noisy scenes.
[0161] In this embodiment, the target sound scene is determined based on the noise level of the target sound signal. Compared with scene recognition of the target sound signal based on acoustic scene classification and sound event detection, the scene recognition result obtained in this embodiment is more in line with the actual situation, thereby making the covariance matrix determined based on the target sound scene more accurate.
[0162] Furthermore, in one feasible embodiment, before step S202, the method further includes:
[0163] Step S205: When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, a preset low noise threshold is used as the noise level threshold.
[0164] In this embodiment, the noise level threshold can vary according to the previous frame of the target sound signal. Specifically, this embodiment sets a low threshold (hereinafter referred to as the low noise threshold for distinction) and a high threshold (hereinafter referred to as the high noise threshold for distinction).
[0165] When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, it is determined that the target sound signal enters from the noisy scene. At this time, the target sound signal contains residual noise signal from the previous frame of the sound signal. Therefore, a preset low noise threshold is used as the noise level threshold to avoid the noise in the previous frame of the sound signal affecting the scene recognition of the current frame of the sound signal.
[0166] Step S206: When the sound scene of the previous frame of the target sound signal is detected to be a quiet scene, a preset high noise threshold is used as the noise level threshold.
[0167] In this embodiment, when the sound scene of the previous frame of the target sound signal is detected to be a quiet scene, since there is no residual noise from the previous frame of the sound signal in the current sound signal, a preset high noise threshold is used as the noise level threshold to accurately distinguish between noisy scenes and quiet scenes.
[0168] It should be noted that, compared to setting a fixed noise level threshold, this embodiment determines the noise level threshold for detecting the target sound scene based on the sound scene of the previous frame of the current sound signal. This can prevent noise in the previous frame from affecting the scene recognition of the current sound signal when the previous frame of the sound signal corresponds to a noisy scene, and accurately distinguish between noisy and quiet scenes when the previous frame of the sound signal corresponds to a quiet scene by using a lower threshold. This embodiment achieves accurate identification of the target sound scene, thereby improving the accuracy of the covariance matrix and the noise suppression effect during beamforming.
[0169] In this embodiment, the target sound scene is determined based on the noise level of the target sound signal. Compared with scene recognition of the target sound signal based on acoustic scene classification and sound event detection, the scene recognition result obtained in this embodiment is more in line with the actual situation, thereby making the covariance matrix determined based on the target sound scene more accurate.
[0170] Furthermore, in one feasible embodiment, audio data of the external environment is collected through a microphone array, and the audio data is processed into frames to obtain multiple frames of environmental sound signals. Each frame of environmental sound signal is used as a target sound signal. Specifically, the length of each frame of environmental sound signal and the overlap length between frames of environmental sound signals can be set as needed. For example, in one feasible embodiment, the duration of each frame of environmental sound signal can be set to 7.5ms to 15ms.
[0171] After frame segmentation, the target sound signal is converted into a frequency domain signal using a Fast Fourier Transform. Scene recognition is then performed on the target sound signal within the frame-segmented target sound signal. The specific process is as follows:
[0172] The noise estimate of the target sound signal is calculated using the following formula:
[0173] λ d (k,l+1)=α d (k,l)·λ d (k,l)+(1-α d (k,l))·|Y(k,l)| 2
[0174] Where, α d (k,l)=α d0 +(1-α d0 )·p(k,l);
[0175] α d0 is a constant coefficient, which is taken as 0.85 in a feasible implementation; p(k,l) is the probability of speech presence, k is the frequency point number, l is the frame number, and Y(k,l) is the frequency domain amplitude of the input signal.
[0176] The noise level of the target sound signal is calculated based on noise estimation. The specific formula for calculating the noise level is as follows:
[0177]
[0178] Where w is the summation weight, and the specific values of w are:
[0179]
[0180] Where c1 and c2 are constant coefficients, and K is the number of frequency points.
[0181] If NL>Th NL (Where, NL is the noise level of the target sound signal, Th) NL If NL ≤ Th (where NL is the noise level threshold), then the target sound signal is determined to correspond to a noisy scene. In this case, the noise flag hnFlag can be updated to 1; if NL ≤ Th NL If the target sound signal corresponds to a quiet scene, then hnFlag can be updated to 0. Specifically, if the current hnFlag is 0 (meaning the previous frame of the sound signal corresponds to a quiet scene), then Th... NL Updated to ( (This is a preset high noise threshold); if the current hnFlag is 1, then Th NL Updated to ( (This is a preset low-noise threshold).
[0182] After identifying the target sound scene, the covariance matrix is calculated using different computational models based on the scene recognition results. Specifically:
[0183] If the target sound signal is determined to correspond to a quiet scene, the covariance matrix is calculated using the Singer model:
[0184]
[0185] in, f is the frequency, d is the distance matrix composed of the distances between each microphone in the microphone array and the preset reference microphone in the microphone array, and c is the wave speed.
[0186] If the target sound signal is determined to correspond to a noisy scene, the covariance matrix is updated when the target sound signal is determined to be a non-speech signal to achieve better noise suppression. The specific calculation formula is as follows:
[0187]
[0188] In this embodiment, the sound signals collected by each microphone in the microphone array are processed by frame segmentation. The number of frames, the length of each frame, and the overlap length between frames are all consistent with the frame segmentation processing of the audio data. In this embodiment, X N This represents a signal matrix composed of sound signals. Each element in the signal matrix is a frame of sound signal from each microphone, processed by framing, that corresponds to the target sound signal within the same frame. Specifically, X... N =Y |hnFlag＝1,vadFlag＝0 , This represents the transpose and conjugate of the target sound signal.
[0189] After obtaining the covariance matrix, the target sound signal is processed using the covariance matrix to obtain the output beam. The output beams corresponding to each frame of the target sound signal are superimposed to obtain the final output target beam.
[0190] Specifically, refer to Figure 4 , Figure 4 This is a comparison diagram of beamforming results according to one embodiment of the present invention, wherein, Figure 4 (1) The fixed beamforming result for the Singh model Figure 4 (2) The beamforming result according to one embodiment of the present invention is compared with... Figure 4 (1) and Figure 4 (2) The signal shows that this embodiment has better noise suppression effect compared to fixed beamforming.
[0191] Furthermore, embodiments of the present invention also propose a beamforming apparatus, referring to... Figure 5 The beamforming apparatus includes:
[0192] Acquisition module 10 is used to acquire target sound signals from the surrounding external environment through a microphone array;
[0193] The determining module 20 is used to identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene;
[0194] The calculation module 30 is used to calculate the covariance matrix corresponding to the target sound signal through a preset calculation model corresponding to the target sound scene;
[0195] The processing module 40 is used to process the target sound signal through the covariance matrix to obtain the output beam.
[0196] Furthermore, when the target sound scene is a quiet scene, the preset calculation model corresponding to the quiet scene is the Singer model, and the calculation module 30 is also used for:
[0197] Determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array;
[0198] The angle matrix is input into the Singer model to calculate the covariance matrix corresponding to the target sound signal.
[0199] Furthermore, when the target sound scene is a noisy scene, the preset calculation model corresponding to the noisy scene is a covariance model, and the calculation module 30 is further used for:
[0200] The signal matrix composed of the sound signals collected by each microphone in the microphone array is input into the covariance model to calculate the covariance matrix corresponding to the target sound signal. The covariance model is the product of the sound signal and the transpose and conjugate of the sound signal.
[0201] Furthermore, the beamforming apparatus further includes a detection module, which is used for:
[0202] Detect whether the target sound signal is a speech signal;
[0203] If the target sound signal is not a speech signal, then the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal is performed.
[0204] If the target sound signal is a speech signal, then the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, and the step of processing the target sound signal through the covariance matrix to obtain the output beam is performed.
[0205] Furthermore, the determining module 20 is also used for:
[0206] Calculate the noise estimate of the target sound signal, and calculate the noise level of the target sound signal based on the noise estimate;
[0207] Detect whether the noise level is greater than a preset noise level threshold;
[0208] If the noise level is greater than the noise level threshold, then the target sound scene of the target sound signal is identified as a noisy scene;
[0209] If the noise level is less than or equal to the noise level threshold, then the target sound scene is determined to be a quiet scene.
[0210] Furthermore, the determining module 20 is also used for:
[0211] When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, a preset low noise threshold is used as the noise level threshold.
[0212] When the sound scene of the previous frame of the target sound signal is a quiet scene, a preset high noise threshold is used as the noise level threshold.
[0213] Furthermore, the beamforming apparatus further includes a framing module, which is used for:
[0214] Audio data of the surrounding environment is collected through a microphone array;
[0215] The audio data is processed into frames to obtain multiple frames of ambient sound signals, and each frame of the ambient sound signal is used as the target sound signal.
[0216] The processing module 40 is further configured to:
[0217] The final output target beam is obtained by superimposing the output beams corresponding to the target sound signals of each frame.
[0218] The various embodiments of the beamforming apparatus of the present invention can be referred to the various embodiments of the beamforming method of the present invention, and will not be repeated here.
[0219] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a beamforming program, which, when executed by a processor, implements the steps of the beamforming method described below.
[0220] The various embodiments of the beamforming apparatus and computer-readable storage medium of the present invention can be referred to the various embodiments of the beamforming method of the present invention, and will not be repeated here.
[0221] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0222] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0224] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A beamforming method, characterized in that, The beamforming method includes the following steps: The target sound signal of the surrounding external environment is collected through a microphone array; Identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene; The covariance matrix corresponding to the target sound signal is calculated using a preset calculation model corresponding to the target sound scene; wherein, when the target sound scene is a quiet scene, the calculation model is a Singer model; when the target sound scene is a noisy scene, the calculation model is a covariance model. The target sound signal is processed using the covariance matrix to obtain the output beam; Wherein, when the target sound scene is a quiet scene, the step of calculating the covariance matrix corresponding to the target sound signal using a preset calculation model corresponding to the target sound scene includes: Determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array; The angle matrix is input into the Singh model to calculate the covariance matrix corresponding to the target sound signal.
2. The beamforming method as described in claim 1, characterized in that, When the target sound scene is a noisy scene, the step of calculating the covariance matrix corresponding to the target sound signal using a preset calculation model corresponding to the target sound scene includes: The signal matrix composed of the sound signals collected by each microphone in the microphone array is input into the covariance model to calculate the covariance matrix corresponding to the target sound signal. The covariance model is the product of the sound signal and the transpose and conjugate of the sound signal.
3. The beamforming method as described in claim 2, characterized in that, Before the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal, the method further includes: Detect whether the target sound signal is a speech signal; If the target sound signal is not a speech signal, then the step of inputting the signal matrix composed of the sound signals collected by each microphone in the microphone array into the covariance model to calculate the covariance matrix corresponding to the target sound signal is performed. If the target sound signal is a speech signal, then the covariance matrix corresponding to the previous frame of the target sound signal is used as the covariance matrix corresponding to the target sound signal, and the step of processing the target sound signal through the covariance matrix to obtain the output beam is performed.
4. The beamforming method as described in claim 1, characterized in that, The step of identifying the target sound scene of the target sound signal includes: Calculate the noise estimate of the target sound signal, and calculate the noise level of the target sound signal based on the noise estimate; Detect whether the noise level is greater than a preset noise level threshold; If the noise level is greater than the noise level threshold, then the target sound scene of the target sound signal is identified as a noisy scene; If the noise level is less than or equal to the noise level threshold, then the target sound scene is determined to be a quiet scene.
5. The beamforming method as described in claim 4, characterized in that, Before the step of detecting whether the noise level is greater than a preset noise level threshold, the method further includes: When the sound scene of the previous frame of the target sound signal is detected to be a noisy scene, a preset low noise threshold is used as the noise level threshold. When the sound scene of the previous frame of the target sound signal is a quiet scene, a preset high noise threshold is used as the noise level threshold.
6. The beamforming method according to any one of claims 1 to 5, characterized in that, The step of acquiring target sound signals from the surrounding external environment via a microphone array includes: Audio data of the surrounding environment is collected through a microphone array; The audio data is processed into frames to obtain multiple frames of ambient sound signals, and each frame of the ambient sound signal is used as the target sound signal. After the step of processing the target sound signal using the covariance matrix to obtain the output beam, the method further includes: The final output target beam is obtained by superimposing the output beams corresponding to the target sound signals of each frame.
7. A beamforming apparatus, characterized in that, The beamforming apparatus includes: The acquisition module is used to acquire target sound signals from the surrounding external environment through a microphone array; The determining module is used to identify the target sound scene of the target sound signal, wherein the target sound scene is a quiet scene or a noisy scene; The calculation module is used to calculate the covariance matrix corresponding to the target sound signal using a preset calculation model corresponding to the target sound scene; wherein, when the target sound scene is a quiet scene, the calculation model is a Singer model; when the target sound scene is a noisy scene, the calculation model is a covariance model. The processing module is used to process the target sound signal through the covariance matrix to obtain the output beam; When the target sound scene is a quiet scene, the calculation module is further configured to: determine the angle matrix composed of the incident angles of the target sound signal into each microphone in the microphone array; and input the angle matrix into the Singer model to calculate the covariance matrix corresponding to the target sound signal.
8. A beamforming device, characterized in that, The beamforming apparatus includes: a memory, a processor, and a beamforming program stored in the memory and executable on the processor, wherein the beamforming program, when executed by the processor, implements the steps of the beamforming method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a beamforming program that, when executed by a processor, implements the steps of the beamforming method as described in any one of claims 1 to 6.