Beam forming method and apparatus, electronic device, and storage medium

Abstract

The application discloses a beam forming method and device, electronic equipment and storage medium, the method comprises the following steps: pre-processing the received voice signal to obtain the corresponding frequency domain signal; input the frequency domain signal into the pre-trained neural network model; utilize N convolution channels in the neural network model to process the frequency domain signal, obtain N fixed beams corresponding to N preset directions, N is an integer greater than 1; utilize the attention mechanism layer in the neural network model to calculate the weight of each fixed beam in the N fixed beams, and judge the sound source direction of the voice signal according to the calculation result; output the fixed beam corresponding to the sound source direction. The application forms the fixed beam of the voice signal based on the neural network model and judges the sound source direction, which can accurately extract the speaker direction beam without DOA estimation and in a noisy environment, and has higher accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signal processing, and in particular to a beamforming method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Speech signal processing technology is a key technology in the field of human-computer interaction today. Microphone arrays have been researched for hands-free telephones, telephone conferences, hearing aids, speech recognition and speech enhancement. Microphone array beamforming can utilize spatial domain information and can better enhance speech signals, and has an extremely important role for back-end speech recognition, and is a key technology for speech signal processing.

[0003] Conventional microphone array beamforming methods first perform sound source direction estimation, perform phase compensation and other operations on the input multi-channel signal according to the estimated sound source direction, and then perform beamforming using methods such as minimum variance distortionless response (MVDR), and further suppress side lobe interference. For example, adaptive beamforming methods, which are based on adaptive beamforming such as generalized sidelobe canceller (GSC) and minimum variance distortionless response (MVDR), can achieve high anti-interference performance by arranging a small number of microphones in a small space. Although the directional diagram of the microphone array is omnidirectional, the adaptive beamformer can adjust the receiving direction of the array to the direction of arrival (DOA), and concentrate the gain on the direction of arrival, which is equivalent to forming a beam, and obtaining the maximum output power of the desired signal. However, for a classic adaptive beamformer based on GSC, such as the Griffiths-Jim beamformer (GJBF), target signal cancellation will occur in the presence of steering vector errors. Because the actual microphone array has defects, the array defects include microphone position, microphone gain and target DOA error. In addition, in a noisy environment, the DOA is also prone to estimation errors, and DOA estimation errors will cause the following adaptive beamformer to have zero gain or even negative gain, thereby causing poor accuracy of sound source positioning and affecting the beamforming result.

[0004] Therefore, it is necessary to provide an improved technical solution to overcome the above technical problems in the prior art. SUMMARY

[0005] To solve the above technical problems, the application provides a beam forming method, device, electronic equipment and storage medium, which can accurately extract the speaker direction beam in a noisy environment without DOA estimation and with higher accuracy and robustness.

[0006] According to the first aspect of the application, a beam forming method is provided, comprising: pre-processing a received speech signal to obtain a corresponding frequency domain signal;

[0007] inputting the frequency domain signal into a pre-trained neural network model;

[0008] processing the frequency domain signal by using N convolution channels in the neural network model to obtain N fixed beams corresponding to N preset directions, N being an integer greater than 1;

[0009] calculating the weight of each fixed beam in the N fixed beams by using an attention mechanism layer in the neural network model, and judging the sound source direction of the speech signal according to the calculation result;

[0010] outputting the fixed beam corresponding to the sound source direction.

[0011] Optionally, the step of pre-processing the received speech signal comprises:

[0012] frame and window conversion of the continuously input speech signal into a short-time stationary signal;

[0013] converting the short-time stationary signal into the frequency domain signal by short-time Fourier transform.

[0014] Optionally, the training method of the neural network model comprises:

[0015] determining the initial parameters of the neural network model based on the weight vector of the super-directive beam former in the corresponding direction;

[0016] taking the frequency domain signal corresponding to the speech signal of different sound source directions as input and taking the direction label as output, and training the initial parameters of the neural network model.

[0017] Optionally, calculating the weight of each fixed beam in the N fixed beams by using the attention mechanism layer in the neural network model comprises:

[0018] converting each fixed beam into an embedding vector;

[0019] calculating the Query value, Key value and Value value corresponding to each fixed beam according to the embedding vector and the corresponding coefficient matrix obtained by training;

[0020] According to the Query value and the Key value corresponding to each fixed beam, a weight coefficient is configured for each fixed beam, and a weighted sum of the Value values is performed according to the weight coefficients to obtain the weight of each fixed beam.

[0021] Optionally, the N preset directions include at least one of 0°, 30°, 60°, 90°, 120°, 150° and 180°.

[0022] Optionally, the neural network model is an end-to-end model.

[0023] Optionally, a direction corresponding to a fixed beam with the largest weight is taken as the sound source direction.

[0024] According to the second aspect of the present application, a beam forming device is provided, comprising:

[0025] A beam forming module is configured to process a frequency domain signal corresponding to a received voice signal by using N convolution channels in a pre-trained neural network model, to obtain N fixed beams corresponding to N preset directions.

[0026] A judging module is configured to calculate the weight of each fixed beam in the N fixed beams by using an attention mechanism layer in the pre-trained neural network model, and to judge the sound source direction of the voice signal according to the calculation result.

[0027] An output module is configured to output a fixed beam corresponding to the sound source direction according to the judgment result of the judging module.

[0028] Optionally, the device further comprises:

[0029] A time-frequency conversion module is configured to convert a continuously input voice signal into a short-time stationary signal by framing and windowing, and to convert the short-time stationary signal into the frequency domain signal by short-time Fourier transform.

[0030] A training module is configured to take the frequency domain signal corresponding to a voice signal of different sound source directions as input, take a direction label as output, and train initial parameters of the neural network model.

[0031] The initial parameters of the neural network model are determined based on a weight vector of a hyperdirectivity beam former in a corresponding direction.

[0032] According to the third aspect of the present application, an electronic device is provided, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program is executed by the processor to implement the steps in the beam forming method as described above.

[0033] According to a fourth aspect of the present application, a storage medium is provided, comprising: a computer program or instructions stored on the storage medium, which, when executed by a processor, implement the steps of the beamforming method as described above.

[0034] The technical solution of the present application can accurately extract the speaker direction beam without DOA estimation and in a noisy environment, and has higher accuracy and robustness.

[0035] It should be noted that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of a beamforming method according to an embodiment of the present application is shown;

[0037] Figure 2 A structural diagram of a beamforming device according to an embodiment of the present application is shown;

[0038] Figure 3 An input-output structural diagram of a neural network model according to an embodiment of the present application is shown;

[0039] Figure 4 A structural diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0041] The present application discloses a beamforming method, which realizes fixed beamforming of a speech signal in a preset direction and sound source direction judgment based on a neural network model, can accurately extract the speaker direction beam without DOA estimation and in a noisy environment, has higher execution efficiency, and has higher accuracy and robustness. Figure 1 and Figure 3 As shown, the method comprises the following steps:

[0042] In step S1, the received speech signal is preprocessed to obtain a corresponding frequency domain signal.

[0043] In step S2, the frequency domain signal is input into a pre-trained neural network model.

[0044] Optionally, the voice signal received by the microphone array in the embodiment is received by a microphone array including at least three microphones not arranged on a single one-dimensional line. As known to those skilled in the art, the microphone array can be coupled to a receiving circuit of the microphone signal, and the receiving circuit can be configured to amplify, filter, and digitize the voice signal.

[0045] In the embodiment, the preprocessing step includes: converting the continuously input voice signal into a short-time stationary signal by framing and windowing; and converting the short-time stationary signal into a frequency domain signal by short-time Fourier transform. That is, before the voice signal is input into the pre-trained neural network model 200, signal processing from time domain to frequency domain is performed on the voice signal to facilitate subsequent further processing operations.

[0046] Further, the training method of the neural network model 200 includes: determining initial parameters of the neural network model based on the weight vector of the hyperdirectivity beamformer in the corresponding direction; training the initial parameters of the neural network model by taking the frequency domain signal corresponding to the voice signal of different sound source directions as input and taking the direction label as output. The frequency domain signal corresponding to the voice signal of different sound source directions as the training sample at least includes the frequency domain signal corresponding to the voice signal of the N preset directions mentioned later.

[0047] The neural network model 200 used in the embodiment is an end-to-end (E2E) model, and the neural network model 200 as a whole is implemented by using an E2E algorithm. During the training process of the neural network model, a prediction result is obtained from the input end (input data, i.e., the frequency domain signal corresponding to the voice signal of different sound source directions) to the output end. By comparing the prediction result with the true result (i.e., the direction label), an error is obtained. This error is transmitted in the form of gradient back propagation in each layer of the model, and then the parameter representation of each layer of the model is iteratively adjusted according to the error until the model converges or reaches the expected effect, at which time the loss function of the model is minimized, and the parameters corresponding to each layer of the model are optimal. All operations during the running of the neural network model 200 are contained in the neural network, i.e., from the input of the original data to the output of the result, the neural network from the input end to the output end is self-contained (which can also be regarded as a black box), which is end-to-end. The end-to-end learning eliminates the data labeling performed before each independent learning task is executed, and reduces the cost and error rate caused by labeling the samples. Therefore, the neural network model 200 based on the E2E algorithm is used to process the frequency domain signal corresponding to the voice signal received by the microphone array in the present application, which is lower in cost and higher in accuracy.

[0048] Further, reference is made to Figure 3The neural network module 200 in the embodiment includes N convolution channels (including a first convolution channel 211, a second convolution channel 212, a third convolution channel 213,..., and an Nth convolution channel 21N, N being an integer greater than 1) and an attention mechanism layer 221. When the neural network module 200 is trained, the respective parameters of each of the N convolution channels and the respective parameters of the attention mechanism layer 221 are collectively trained.

[0049] In step S3, the N convolution channels in the neural network model are used to process the frequency domain signal to obtain N fixed beams corresponding to N preset directions, N being an integer greater than 1.

[0050] In the embodiment, each of the N convolution channels is used to simultaneously perform feature extraction, feature mapping, and the like on the input frequency domain signal based on the respective parameters to simultaneously obtain N fixed beams corresponding to N preset directions, wherein each convolution channel corresponds to a fixed beam of one of the N predetermined directions. Optionally, the N preset directions include at least one of 0°, 30°, 60°, 90°, 120°, 150°, and 180°. For example, in the embodiment, 7 convolution channels are used to simultaneously output 7 fixed beams corresponding to 0°, 30°, 60°, 90°, 120°, 150°, and 180°. However, it should be understood that in other embodiments of the present application, any number of direction angles (based on the same reference direction) can be selected as the preset direction according to actual needs, the neural network model 200 is trained based on the preset direction, and the fixed beam corresponding to the direction is obtained based on the trained neural network model 200, and the present application does not limit this. Optionally, when multiple preset directions are set, the angle difference between two adjacent directions arranged in descending or ascending order of the numerical value of the direction angle can be the same or different, and can be reasonably selected according to actual conditions. In some embodiments, the angle difference between adjacent beams is determined by the shape, configuration of the microphone, and the number of convolution channels.

[0051] Further, each of the N predetermined directions output by the N convolution channels corresponds to a sequence number (or label) indicating the corresponding relationship between the fixed beam of each predetermined direction and the convolution channel outputting the same.

[0052] In this embodiment, the N convolutional channels are the first part of the neural network model 200, which can obtain and output multiple fixed beams according to the corresponding parameters after training. Functionally, it is equivalent to a multi-path beam forming module. However, unlike the fixed beam former using traditional algorithms, the parameters of each network layer of the N convolutional channels in this application are initialized using the parameters of the traditional fixed beam former. Then, during the neural network training, these parameters can be iterated through back propagation according to the loss function of the neural network model 200, so as to obtain a more optimal multi-path neural network fixed beam former. Therefore, the parameters obtained after training are obviously more robust than traditional fixed beams.

[0053] In step S4, the weight of each fixed beam in the N fixed beams is calculated using the attention mechanism layer in the neural network model, and the sound source direction of the voice signal is determined according to the calculation result.

[0054] In step S5, the fixed beam corresponding to the sound source direction is output.

[0055] The attention mechanism is essentially to filter out a small amount of important information from a large amount of information and focus on these important information, ignoring a large amount of unimportant information. That is, the attention mechanism can automatically learn a set of weight coefficients through the network, and use the "dynamic weighting" mechanism to emphasize the area of interest of the user while suppressing the irrelevant background area.

[0056] In this embodiment, the attention mechanism layer 221 in the neural network model 200 receives the N fixed beams output by the N convolutional channels and their sequence numbers at the same time, and configures a corresponding weight for each fixed beam through parallel processing of the N fixed beams. The weight represents the probability that the corresponding fixed beam belongs to the sound source direction, such as the speaker beam. Then, by comparing the weights corresponding to the N fixed beams, one of the N preset directions can be selected as the sound source direction according to the comparison result, for example, the direction corresponding to the fixed beam with the largest weight can be selected as the sound source direction, and the fixed beam and sequence number corresponding to the selected sound source direction are output as the target.

[0057] For example, the weight of each fixed beam in the N fixed beams is calculated using the attention mechanism layer 221 in the neural network model 200, which includes: converting each fixed beam into an embedding vector; calculating the Query value, Key value and Value value (i.e. q value, k value and v value) corresponding to each fixed beam according to the embedding vector and the corresponding coefficient matrix obtained after training; configuring a weight coefficient for each fixed beam according to the q value and k value corresponding to each fixed beam, and performing weighted summation on the v value according to the weight coefficient to obtain the corresponding weight of each fixed beam.

[0058] The attention mechanism layer 221 in the neural network model 200 is used to process the multiple fixed beams in the scheme disclosed in the application, and based on the operation principle of the attention mechanism, the neural network based on the attention mechanism can compare the results of each fixed beam formation, and then give a weight to the output of each fixed beam formation, and finally select the fixed beam direction with the maximum attention weight as the output. Looking at the technical scheme disclosed in the embodiment of the application, the whole process is executed in parallel, and the transverse comparison between each output is also performed to find the optimal solution, which can realize the correlation between each fixed beam in the N fixed beams, is conducive to improving the accuracy and robustness of the output result, and has lower time complexity.

[0059] On the other hand, in the application, an E2E algorithm is used to form corresponding N fixed beams in N predetermined directions for the input speech signal, and the weight configuration and comparison of the N fixed beams, the whole algorithm is composed of a neural network model from input to output, and the corresponding parameters of the N convolution channels 211-21N and the attention mechanism layer 221 in the neural network model 200 are optimized by using the gradient back propagation method in the model training through a loss function, which is an optimal system, and there is no cascade of two different architectures, the loss is smaller, and the accuracy is higher.

[0060] Further, the application also discloses a beam forming device for realizing the beam forming method. Figure 2 and Figure 3 The beam forming device disclosed in the embodiment of the application comprises a time-frequency conversion module 100, a beam forming module 210, a judgment module 220, an output module 300 and a training module 400.

[0061] The time-frequency conversion module 100 is used to convert the continuous input speech signal into a short-time stationary signal by framing and windowing, and convert the short-time stationary signal into a frequency domain signal by short-time Fourier transform.

[0062] The beam forming module 210 is used to process the frequency domain signal corresponding to the received speech signal by using the N convolution channels (including the first convolution channel 211, the second convolution channel 212, the third convolution channel 213,..., the Nth convolution channel 21N, N is an integer greater than 1) in the pre-trained neural network model 200, to obtain N fixed beams corresponding to N preset directions.

[0063] The judgment module 220 is used to calculate the weight of each fixed beam in the N fixed beams by using the attention mechanism layer 221 in the pre-trained neural network model 200, and judge the sound source direction of the speech signal according to the calculation result.

[0064] The output module 300 is configured to output the fixed beam corresponding to the sound source direction according to the judgment result of the judgment module 220.

[0065] The training module 400 is configured to input the frequency domain signals corresponding to the speech signals of different sound source directions and output the direction labels, and iteratively train the initial parameters of the neural network model 200 in a gradient back propagation manner, wherein the initial parameters of the neural network model 200 are determined based on the weight vectors of the hyperdirectivity beamformer in the corresponding directions.

[0066] In specific implementation, each module / unit in the beamforming device can be implemented as an independent entity, or can be combined as the same or several entities. Meanwhile, the specific implementation of each module / unit in the beamforming device described above can refer to the foregoing beamforming method embodiments, which will not be repeated here.

[0067] The embodiment of the present application also provides an electronic device 500, as shown in the figure, comprising a memory 520, a processor 510, and a program stored in the memory 520 and executable on the processor 510, wherein the program is executed by the processor 510 to implement each process of each embodiment of the foregoing beamforming method and achieve the same technical effects. To avoid repetition, details will not be repeated here. Figure 4

[0068] Those skilled in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, which can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, the embodiment of the present application also provides a storage medium having a computer program or instructions stored thereon, which can implement each process of each embodiment of the foregoing beamforming method when executed by a processor. The storage medium includes, for example, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0069] Since the instructions stored in the storage medium can execute the steps of the beamforming method provided by the embodiment of the present application, the beneficial effects of the beamforming method provided by the embodiment of the present application can be achieved. Details are described in the foregoing embodiments, which will not be repeated here. The specific implementation of each operation can refer to the foregoing embodiments, which will not be repeated here.

[0070] ​To sum up, the application realizes fixed beam forming and sound source direction judgment of the speech signal in the preset direction based on N convolution channels and attention mechanism layers in a neural network model, removes the DOA estimation module by the powerful capacity of the neural network, and makes the network pay more attention to the speaker direction beam and exclude the interference of the non-speaker direction beam by the attention mechanism, so that the speaker direction beam can be accurately extracted without DOA estimation and in a noisy environment, the execution efficiency is higher, and the accuracy and robustness are stronger.

[0071] Finally, it should be noted that: obviously, the above embodiments are only examples for clearly illustrating the application, and are not limitations on the embodiments. Based on the above description, other different forms of changes or variations can be made by those of ordinary skill in the art. Here, all the embodiments cannot be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the application.

Claims

1. A beamforming method, wherein, The method comprises: preprocessing a received voice signal to obtain a corresponding frequency domain signal; inputting the frequency domain signal into a pre-trained neural network model, the neural network model being an end-to-end neural network model; processing the frequency domain signal by using N convolution channels in the neural network model to obtain N fixed beams corresponding to N preset directions, N being an integer greater than 1; calculating the weight of each fixed beam in the N fixed beams by using an attention mechanism layer in the neural network model, and judging the sound source direction of the voice signal according to the calculation result; outputting the fixed beam corresponding to the sound source direction.

2. The beam forming method according to claim 1, wherein the preprocessing of the received voice signal comprises: frame segmentation and windowing of the continuously input voice signal to convert the voice signal into a short-time stationary signal; conversion of the short-time stationary signal into the frequency domain signal by short-time Fourier transform.

3. The beamforming method of claim 1, wherein, The training method of the neural network model comprises: determining initial parameters of the neural network model based on a weight vector of a super-directive beamformer in a corresponding direction; training the initial parameters of the neural network model by taking the frequency domain signals corresponding to voice signals of different sound source directions as input and taking direction labels as output.

4. The beamforming method of claim 1, wherein, The calculation of the weight of each fixed beam in the N fixed beams by using the attention mechanism layer in the neural network model comprises: conversion of each fixed beam into an embedding vector; calculation of a Query value, a Key value and a Value value corresponding to each fixed beam according to the embedding vector and a corresponding coefficient matrix obtained by training; configuration of a weight coefficient for each fixed beam according to the Query value and the Key value corresponding to the fixed beam, and weighted summation of the Value value according to the weight coefficient, so as to obtain the weight of each fixed beam.

5. The beamforming method according to any one of claims 1-4, wherein, The N preset directions comprise at least one of 0°, 30°, 60°, 90°, 120°, 150° and 180°.

6. The beamforming method according to any one of claims 1-4, wherein, The direction corresponding to the fixed beam with the largest weight is taken as the sound source direction.

7. A beam forming apparatus, wherein, The method comprises: a beam forming module for processing a frequency domain signal corresponding to a received voice signal by using N convolution channels in a pre-trained neural network model, the neural network model being an end-to-end neural network model, to obtain N fixed beams corresponding to N preset directions; a judging module for calculating the weight of each fixed beam in the N fixed beams by using an attention mechanism layer in the pre-trained neural network model, and judging the sound source direction of the voice signal according to the calculation result; an output module for outputting the fixed beam corresponding to the sound source direction according to the judgment result of the judging module.

8. The beamforming apparatus of claim 7, wherein, The method further comprises: a time-frequency conversion module for frame segmentation and windowing of the continuously input voice signal to convert the voice signal into a short-time stationary signal, and conversion of the short-time stationary signal into the frequency domain signal by short-time Fourier transform; a training module for training initial parameters of the neural network model by taking the frequency domain signals corresponding to voice signals of different sound source directions as input and taking direction labels as output. The initial parameters of the neural network model are determined based on a weight vector of a hyperdirective beamformer in a corresponding direction.

9. An electronic device, comprising: Comprise: A processor, a memory, and a program stored on the memory and executable on the processor, the program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A storage medium, characterized by The storage medium stores a computer program or instructions, and the computer program or instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 6.

Images