A speech data training method and recognition method based on distributed array

Abstract

The present invention discloses a speech data training method and recognition method based on a distributed array, wherein the training method includes: determining each microphone array speech data and its optimal microphone array label signal; inputting each microphone array speech data and the optimal microphone array label signal into a neural network model for training, extracting microphone phase features of two different channels and energy features of microphone speech data of any channel for splicing; and gradually optimizing the neural network model until it outputs the optimal speech signal. The present invention uses a distributed array speech data training method, utilizes a neural network model to train and output the optimal speech signal of the distributed microphone array speech data, can accurately identify the speech signal of the optimal sound source, and recognizes the training results based on a multi-channel attention feature fusion network model, significantly improves the speech recognition performance, and ultimately ensures the accuracy of speech recognition, meeting the speech recognition requirements in a far-field sound pickup environment.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a speech data training method and a recognition method based on a distributed array. Background Art

[0002] With the development of big data, artificial intelligence, the Internet of Things, and 5G technology, intelligent voice has become an important way of interaction in people's lives today. Among them, the application of far-field voice recognition is also increasing. Far-field voice recognition is mainly used in the field of smart home, for example: it can be used in smart speakers, smart TVs, and conference scenes. However, for far-field voice recognition, there is a large distance between the sound source and the pickup microphone. The energy dissipation of the sound during the propagation process makes the voice energy collected by the microphone low, and the voice quality is easily affected by background noise, interfering voice, reverberation and other factors. Therefore, it is necessary to further improve the recognition performance of voice in complex sound field environments.

[0003] In the related technology, the target sound source is generally located, and a beamforming algorithm is used to collect only the sound from the direction of the target sound source, suppressing various interfering sounds from other directions. However, the current sound source localization or beamforming method has a limited positioning range in a complex sound field environment, especially in an environment with strong noise and strong reverberation, resulting in a significant decrease in its positioning and directional sound pickup performance. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the sound source localization or beamforming method for speech recognition in the prior art easily causes a significant decrease in its positioning and directional sound pickup performance, thereby providing a speech data training method and recognition method based on a distributed array.

[0005] According to the first aspect, the present invention discloses a distributed array-based speech data training method, comprising the following steps:

[0006] Determine the voice data of each microphone array from different directions and its optimal microphone array label signal;

[0007] Inputting the microphone array speech data and the optimal microphone array label signal into the neural network model training, wherein microphone phase features of two different channels and energy features of microphone speech data of any channel are extracted from the microphone array speech data for splicing;

[0008] The neural network model is gradually optimized until it outputs an optimal speech signal for the distributed microphone array speech data.

[0009] In one embodiment, the optimal microphone array tag signal is determined by the following steps, including:

[0010] In the target area, voice data from each microphone array is obtained from different directions;

[0011] The speech data of each microphone array is simulated by an IMAGE model, thereby generating an impulse response signal of the target area;

[0012] Calculating the signal-to-noise ratios corresponding to the microphone array speech data according to the impulse response signal and the noise signal and the reverberation signal in the microphone array speech data;

[0013] The optimal microphone array tag signal is determined among the signal-to-noise ratios corresponding to the microphone array voice data.

[0014] In one implementation, the signal-to-noise ratios corresponding to the microphone array speech data are calculated based on the impulse response signal and the noise signal and the reverberation signal in the microphone array speech data, and the calculation is performed by the following formula:

[0015]

[0016] Among them, SNR true is the signal-to-noise ratio corresponding to the voice data of each microphone array, Y noisy The noisy speech signal of the microphone array speech data;

[0017] Y noisy =Y target +Y reverb +Y noise , Y target The target speech signal and its early reverberation signal collected by each microphone, Y reverb is the late reverberation signal collected by each microphone, Y noise is the noise signal collected by each microphone.

[0018] In one implementation, extracting microphone phase features of two different channels from each microphone array voice data is performed by the following formula:

[0019]

[0020] in, is the microphone phase characteristic of channel i, is the microphone phase characteristic of channel j, and IPD is the cosine value of the difference between the microphone phase characteristic of channel i and the microphone phase characteristic of channel j.

[0021] In one embodiment, the step of gradually optimizing the neural network model until it outputs an optimal digital speech signal corresponding to each microphone array speech data includes:

[0022] Based on a preset activation function, the network parameters of the input layer, the intermediate layer and the output layer of the neural network model are gradually adjusted to optimize the neural network model so that it outputs the optimal speech signal of the distributed microphone array speech data.

[0023] According to a second aspect, the present invention discloses a distributed array-based voice data recognition method for identifying an optimal voice signal of the distributed microphone array voice data in the first aspect or any embodiment of the first aspect, comprising:

[0024] Obtaining an optimal voice signal of the distributed microphone array voice data;

[0025] The optimal speech signal of the distributed microphone array speech data is input into the multi-channel attention feature fusion network model training to further identify its accuracy.

[0026] In one embodiment, inputting the optimal speech signal of the distributed microphone array speech data to a multi-channel attention feature fusion network model training to thereby identify its accuracy includes:

[0027] Based on the standard threshold of the microphone array speech data, determine whether the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is the target value; and / or, based on the standard microphone array speech data, predict whether the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is the target value.

[0028] In one embodiment, the determining whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is a target value based on the microphone array speech data standard threshold includes:

[0029] If the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is less than or equal to the standard threshold of the microphone array speech data, then the optimal speech signal of the distributed microphone array speech data is accurate;

[0030] If the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is greater than the standard threshold of the microphone array speech data, re-training is performed using the distributed array-based speech data training method described in the first aspect or any embodiment of the first aspect.

[0031] According to the third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the distributed array-based speech data training method described in the first aspect or any embodiment of the first aspect; or, to execute the distributed array-based speech data recognition method described in the second aspect or any embodiment of the second aspect.

[0032] According to the fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the distributed array-based speech data training method described in the first aspect or any one of the embodiments of the first aspect; or, executing the distributed array-based speech data recognition method described in the second aspect or any one of the embodiments of the second aspect.

[0033] The technical solution of the present invention has the following advantages:

[0034] The present invention discloses a speech data training method and recognition method based on a distributed array, wherein the training method includes: determining each microphone array speech data from different directions and its optimal microphone array label signal; inputting each microphone array speech data and the optimal microphone array label signal into a neural network model for training, wherein the microphone phase features of two different channels and the energy features of the microphone speech data of any channel are extracted from each microphone array speech data for splicing; and the neural network model is gradually optimized until it outputs the optimal speech signal of the distributed microphone array speech data. The present invention uses a distributed array speech data training method, and utilizes the optimal training results output by the neural network model training to recognize each microphone array speech data, and can accurately identify the speech signal of the optimal sound source, and recognizes the training results based on the multi-channel attention feature fusion network model, which significantly improves the speech recognition performance, and ultimately ensures the accuracy of speech recognition, meeting the speech recognition needs in a far-field sound pickup environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flowchart of a specific example of a distributed array-based speech data training method in an embodiment of the present invention;

[0037] Figure 2 is a flowchart of another specific example of a distributed array-based speech data training method in an embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of indoor area simulation in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a neural network model in an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of any microphone array in an embodiment of the present invention;

[0041] Figure 6 Schematic diagram of a distributed array voice data recognition method according to an embodiment of the present invention;

[0042] Figure 7 It is a schematic diagram of a multi-channel attention feature fusion network model in an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of the structure of a multi-channel attention feature fusion network model in an embodiment of the present invention;

[0044] Fig. 9 Schematic diagram of the entire process of speech training and speech recognition in an embodiment of the present invention;

[0045] Fig.10 It is a structural block diagram of a distributed array-based speech data training device in an embodiment of the present invention;

[0046] Fig.11 is another structural block diagram of a distributed array-based speech data training device in an embodiment of the present invention;

[0047] Fig.12 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0050] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0052] In the field of speech recognition technology, based on the far-field sound pickup environment, due to the large distance between the sound source and the pickup microphone, the energy dissipation of the sound during propagation makes the voice energy collected by the microphone low, and the voice quality is also easily affected by background noise, interfering voices, reverberation and other factors. In related technologies, the target sound source is generally located, and a beamforming algorithm is used to collect only the sound from the direction of the target sound source, suppressing various interfering sounds from other directions. However, the current sound source localization or beamforming method has a limited positioning range in a complex sound field environment, especially in an environment with strong noise and strong reverberation, resulting in a significant decrease in its positioning and directional sound pickup performance.

[0053] Example 1

[0054] In view of this, an embodiment of the present invention provides a distributed array-based voice data training method, which can be preferably applied to far-field sound pickup environments, such as smart home scenarios, and can also be applied to other voice pickup environments. Figure 1 As shown, the following steps are included:

[0055] Step S11: Determine the voice data of each microphone array from different directions and its optimal microphone array label signal.

[0056] As smart homes and microphone arrays become more and more popular in the current far-field sound pickup environment, the layout of distributed microphone arrays is increasingly being used. Multi-channel array microphones are arranged on microphone arrays in different directions, and the sound source signals in the target area are picked up through the layout of distributed microphone arrays, which can improve the sound pickup efficiency, which is much better than the sound pickup effect of multi-channel microphones or microphone arrays in a single direction, and ultimately helps to improve speech recognition performance.

[0057] The microphones in the microphone arrays in the embodiments of the present invention may be arranged in one or more of a planar array, a three-dimensional array, or a topological array.

[0058] In one embodiment, Figure 2 As shown, the above step S11 determines the voice data of each microphone array from different directions, including:

[0059] Step S21: In the target area, voice data from each microphone array is acquired from different directions.

[0060] Here, each microphone array voice data includes a target sound source signal and a noise signal in a target area. For example, the target area may be a conference room area or an indoor area of ​​a home.

[0061] Step S22: simulating the speech data of each microphone array through the IMAGE model to generate an impulse response signal of the target area.

[0062] like Figure 3 As shown in the figure, it is a schematic diagram of indoor area simulation. The simulated audio data is generated based on the IMAGE model. Considering the positions of the microphone arrays in different directions and the complexity of the overall room model, the Figure 3 As shown in the schematic diagram, the dotted circle represents the microphone position, the * represents the target sound source position, and the x represents the noise source position.

[0063] exist Figure 3 In the example, the audio signals received by each microphone array can be represented as the direct signal from the sound source, the reverberation signal formed by the sound source signal reflected by the wall, and the received noise signal. Therefore, the i-th microphone m i The signal received at time t can be expressed as follows:

[0064]

[0065] Among them, α i s(t-τ i ) is the direct signal from the sound source received by the i-th microphone, α i The attenuation coefficient of the sound source reaching microphone i through the direct path, s(t-τi ) represents the direct signal, K represents the number of reflection paths, α ik s(t-τ ik ) is a signal containing reverberation, α ik represents the attenuation coefficient of the sound source reaching the i-th microphone through the k-th path, τ ik represents the propagation time of the path, n i (t) represents the noise received by microphone i. In order to simplify the IMAGE model, the direct sound and reflected sound can be regarded as the result of the convolution of the sound source signal and the room impulse response (RIR), so the signal received by each microphone can be expressed as,

[0066] m i (t) = h si (t)*s(t)+n i (t); (2)

[0067] Among them, h si is the room impulse response of the target sound source reaching the i-th microphone. When the noise is diffuse noise, n i (t) = n(t); when the noise is a point noise source, n i (t) = h ni (t)*n(t),h ni (t) represents the room impulse response of the point noise source reaching the i-th microphone.

[0068] According to the actual application scenario, given any room size, the room impulse response h from the target sound source and noise source at any position in the room to the microphone can be calculated based on the IMAGE model. si (t), h ni (t). The audio signals collected by each microphone on the distributed array are simulated and generated according to the above formula (2). Here, the speech noise source data of the target sound source can be selected from an existing public database according to the specific recognition task.

[0069] Generate data labels for the initial signals of speech data from microphone arrays in different directions. For example, in the case of a distributed array, the data of a sound source plus a noise source will form multiple array collection versions, and there must be one that is the best sound pickup among all the collected samples, which depends on the target sound source, the noise source, and the relative position between the arrays, and is also related to the reverberation conditions of the room. Therefore, it is necessary to select the sample with the best sound pickup quality from multiple sound pickup samples.

[0070] Step S23: Calculate the signal-to-noise ratios corresponding to the microphone array speech data according to the impulse response signal and the noise signal and the reverberation signal in the microphone array speech data.

[0071] Speech propagates in the room and reaches the human ear after multiple reflections in the medium, which includes early reverberation and late reverberation. It is generally believed that early reverberation helps to understand the speech content, while late reverberation is not conducive to speech perception.

[0072] Therefore, the above step S23 calculates the signal-to-noise ratios corresponding to the voice data of each microphone array according to the impulse response signal and the noise signal and reverberation signal in the voice data of each microphone array, and is performed by the following formula (3):

[0073]

[0074] Among them, SNR true is the signal-to-noise ratio corresponding to the voice data of each microphone array, Y noisy is the noisy speech signal of each microphone array speech data, and

[0075] Y noisy =Y target +Y everb +Y noise , Y target The target speech signal and its early reverberation signal collected by each microphone, Y reverb is the late reverberation signal collected by each microphone, Y noise is the noise signal collected by each microphone.

[0076] Step S24: determining the optimal microphone array tag signal among the signal-to-noise ratios corresponding to the microphone array speech data.

[0077] In practical applications, since the size of each microphone array is small and the distance between microphones in an array is much smaller than the array distance, it is preferred to use the true signal-to-noise ratio of channel 1 as the signal-to-noise ratio of the entire microphone array, and take the microphone array corresponding to the highest true signal-to-noise ratio as the optimal microphone array label.

[0078] Executing the above steps S21 and S23 by the embodiment of the present invention is beneficial to improving the audio quality of voice data collected from each microphone array in different directions, thereby achieving the purpose of optimizing training and ultimately improving speech recognition performance.

[0079] Step S12: input each microphone array speech data and the optimal microphone array label signal into the neural network model training, wherein the microphone phase features of two different channels and the energy features of the microphone speech data of any channel are extracted from each microphone array speech data and spliced.

[0080] like Figure 4 As shown, in each microphone array speech data, the phase characteristics of the microphone speech data of two different channels and the energy characteristics of the microphone speech data of any channel are determined.

[0081] For example: Figure 3 In the embodiment of the present invention, each microphone array is arranged in a circular shape, and among the four points on the upper, lower, left and right sides of the circular edge symmetrical about the center point, the upper point and the lower point are on a straight line, and the left point and the right point are on a straight line. In the embodiment of the present invention, two points on the same straight line are preferably used as microphone phase characteristics of two different channels. For example: Figure 5 As shown, one of the microphone array shapes is shown, in which the microphone array is circular. Figure 5 In the figure, the microphone of channel 1 and the microphone of channel 5 are located in a straight line, the microphone of channel 2 and the microphone of channel 6 are located in a straight line, the microphone of channel 3 and the microphone of channel 7 are located in a straight line, and the microphone of channel 4 and the microphone of channel 8 are located in a straight line. Therefore, the microphones of channels 1 and 5 can be selected, or the microphones of channels 3 and 7 can be selected, or the microphones of channels 4 and 8 can be selected. Further, the phase cosine values ​​of the microphones of the two channels located in the same straight line can be calculated.

[0082] In one embodiment, the above step S12 extracts microphone phase features of two different channels from each microphone array speech data.

[0083] It is implemented by the following formula (4):

[0084]

[0085] in, is the microphone phase characteristic of channel i, is the microphone phase characteristic of channel j, and IPD is the cosine value of the difference between the microphone phase characteristic of channel i and the microphone phase characteristic of channel j.

[0086] In a preferred embodiment, phase characteristics of microphones of two channels located on the same straight line are extracted.

[0087] For example, the microphones of channel 1 and channel 5 are spliced ​​together. Substituting into the above formula (4), it can be expressed as in, For the microphone of channel 1, This is the microphone for channel 5.

[0088] At the same time, in the speech data of each microphone array, extracting the energy characteristics of the microphone speech data of any channel and splicing them with the microphone phase characteristics of the above two different channels can help improve the determination performance of the optimal speech signal.

[0089] In a preferred embodiment, the energy features of the microphone speech data of channel 1 are preferentially extracted.

[0090] Step S13: gradually optimizing the neural network model until it outputs the optimal speech signal of the distributed microphone array speech data.

[0091] In the above training process, the neural network structure composed of a LSTM layer and a DNN layer can play the role of a small model with high efficiency. The features are output through the above neural network model to obtain the optimal predicted posterior probability of the sound pickup network.

[0092] In one embodiment, the above step S13, gradually optimizing the neural network model until it outputs an optimal speech signal of the distributed microphone array speech data, includes:

[0093] Based on the preset activation function, the network parameters of the input layer, intermediate layer and output layer of the neural network model are gradually adjusted to optimize the neural network model so that it outputs the optimal speech signal of the optimal distributed microphone array speech data. The preset activation function here is the LogSoftmax activation function.

[0094] The optimal array label is obtained through the LogSoftmax activation function The loss function uses the cross entropy function and continuously updates the network parameters through the Adam optimizer. Figure 4 In , the overall input features and network structure are shown.

[0095] In one implementation, in order to speed up the network operation and maintain numerical stability, the activation function uses the LogSoftmax function, which is specifically shown in the following formula (5):

[0096]

[0097] Among them, x i is the network parameter output by the i-th neuron in the current layer, exp(x i ) is the index of the network parameter output by the i-th neuron, j represents the number of neurons in the current layer, Logsoftmax(x i ) is the probability of taking the logarithm of the normalized index of the ith neuron.

[0098] The distributed array-based speech data training method in the embodiment of the present invention collects speech data from each microphone array in different directions, calculates the signal-to-noise ratios corresponding to the speech data of each microphone array, and further extracts the microphone phase characteristics of two different channels and the energy characteristics of the microphone speech data of any channel in the neural network model for splicing. This can not only improve the speech pickup efficiency, but also facilitate the output of the optimal speech signal of the distributed microphone array speech data.

[0099] Example 2

[0100] The embodiment of the present invention also provides a distributed array-based voice data recognition method, such as Figure 6 As shown, it can be preferably applied in a far-field sound pickup environment, such as a smart home scene, and of course it can also be applied in other voice pickup environments. The optimal voice signal for identifying the distributed microphone array voice data in Example 1 includes:

[0101] Step S61: Obtaining the optimal speech signal of the distributed microphone array speech data.

[0102] Here is the optimal training result of outputting the speech data of each microphone array after the neural network model training in Example 1.

[0103] Step S62: Input the optimal speech signal of the distributed microphone array speech data to the multi-channel attention feature fusion network model training, and then identify its accuracy.

[0104] like Figure 7 As shown in the figure, it is a multi-channel attention feature fusion network model. On the basis of the general end-to-end neural network model, feature fusion based on the multi-channel attention mechanism is added to realize multi-channel adaptive speech training. Taking transformer end-to-end speech recognition as an example, Figure 7 It is a transformer neural network model based on multi-channel attention mechanism feature fusion. The overall architecture of the transformer end-to-end neural network model is Sequence to Sequence (Seq2Seq). Figure 7 In the figure (except the dotted box), there are two parts, the encoder and the decoder. The end-to-end speech recognition network based on Transformer uses a multi-head attention mechanism and a feed-forward network to replace the RNN in the encoder and decoder, and introduces a masked multi-head attention mechanism to enable the transformer to support parallel training and accelerate the overall training process.

[0105] exist Figure 7In the process, the acoustic features are first encoded. For example, the audio is pre-emphasized, framed, and windowed to extract fbank (80 dimensions) and pitch (3 dimensions) features, which are 83 dimensions in total. Then, the time-frequency domain features can be extracted more effectively through two-dimensional dilated convolution. Since Transformer does not contain RNN and Conv, it is relatively insensitive to the position information of the sequence. Therefore, the above features are positionally encoded to enhance the relative position and absolute position information of the sequence. The features with position encoding information are input into a series of encoders in series, where the encoder includes layer normalization, multi-head attention network, residual network, and forward transmission network. At this point, the output of the encoder is an embedding with important speech recognition features. The multi-head attention mechanism makes the embedding contain context-related information. Then, the text features are compiled. The text features are speech text labels with start symbols. The start symbol S (Start) is similar to the silent segment before the start of the audio speech, indicating the position of the first text character. The true label of the network is a text feature with an end symbol, where the end symbol E (End) is located at the last position of the text and is used to indicate the position of the last text character.

[0106] During the training process, the decoder includes a masked multi-head attention network, layer normalization, a multi-head attention network, and a forward transfer network. The position encoding of the training text and the embedding output by the encoder are used as decoder input features. After a series of the above decoders, the maximum posterior probability of the text corresponding to the current feature is obtained through the DNN and softmax activation function. The text with the maximum probability of the current feature is found through the one-hot encoded dictionary. The maximum posterior probability of the text and the true label are used as the cross entropy as the loss function to continuously update the network to achieve end-to-end speech recognition training. The following Table 1 is an example of the input features and output features of the transformer end-to-end speech recognition network.

[0107] Table 1

[0108]

[0109] A neural network model based on multi-channel attention feature fusion transformer, Figure 6 In the figure (including the dotted box), the difference from the transformer end-to-end neural network model is that the front end introduces multi-channel attention feature fusion. Figure 8As shown, first, the maximum channel pooling and average channel pooling are performed on different channels, and the corresponding channel features are added after inputting the three-layer full connection respectively, and then the sigmoid activation function is used as the channel attention weight. The multi-channel features (83 dimensions * N channels) are multiplied by the channel attention weights and then added to achieve feature fusion, and then trained through the above-mentioned transformer end-to-end speech recognition network. The training process is consistent with 1-C. It can be seen that the multi-channel feature fusion method in the embodiment of the present invention is suitable for training various speech.

[0110] During the test phase, the distributed array signal is input, and the optimal array signal is obtained through the optimal sound pickup array judgment network. The optimal array signal is fused into a single channel feature through multi-channel features, and then input into the recognition network to achieve efficient far-field speech recognition. The overall system structure is as follows: Fig. 9 shown.

[0111] In one embodiment, the optimal speech signal of the distributed microphone array speech data is input to the multi-channel attention feature fusion network model training, and then its accuracy is identified, including:

[0112] Based on the standard threshold of microphone array speech data, determine whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is the target value; and / or, based on the standard microphone array speech data, predict whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is the target value.

[0113] In one embodiment, determining whether the optimal value of each microphone array speech data is an expected value based on the microphone array speech data standard threshold includes:

[0114] Step 1: If the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is less than or equal to the standard threshold of the microphone array speech data, then the optimal speech signal of the distributed microphone array speech data is accurate.

[0115] Step 2: If the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is greater than the standard threshold of the microphone array speech data, re-training is performed using the distributed array-based speech data training method in Example 1.

[0116] In one embodiment, based on standard microphone array speech data, it is predicted whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is the target value.

[0117] For example: a large amount of historical information of each microphone array voice data is collected through experience, and these large amounts of each microphone array voice data are all subjected to voice recognition and corresponding recognition results. Based on these recognition results, the output result of this training of the neural network model can be predicted. If the recognition result of the optimal voice signal of the distributed microphone array voice data is consistent with the predicted recognition result, it means that the optimal value of each microphone array voice data meets the expected requirements. Otherwise, it does not meet the requirements and it is still necessary to re-collect the voice data of each microphone array and re-train based on the method in Example 1. Fig. 9 The figure shows a schematic diagram of the entire process of speech training and speech recognition.

[0118] Therefore, the distributed array-based speech data recognition method in the embodiment of the present invention, based on the distributed array-based speech data training method in Example 1, uses the optimal training results output by the neural network model training to recognize the speech data of each microphone array, and can accurately identify the speech signal of the target sound source, significantly improve the speech recognition performance, and ultimately ensure the accuracy of speech recognition, meeting the speech recognition needs in the far-field sound pickup environment.

[0119] Example 3

[0120] The embodiment of the present invention provides a speech data training device based on a distributed array, such as Fig.10 As shown, the following steps are included:

[0121] The voice data determination module 101 is used to determine the voice data of each microphone array from different directions and its optimal microphone array label signal;

[0122] The speech data training module 102 is used to input the speech data of each microphone array and the optimal microphone array label signal to the neural network model training, wherein the microphone phase features of two different channels and the energy features of the microphone speech data of any channel are extracted from the speech data of each microphone array for splicing;

[0123] The speech data optimization module 103 is used to gradually optimize the neural network model until it outputs the optimal speech signal of the distributed microphone array speech data.

[0124] In one implementation, the voice data determination module 101 includes:

[0125] The speech signal acquisition submodule is used to acquire speech data from each microphone array in different directions in the target area.

[0126] The signal simulation submodule is used to simulate the speech data of each microphone array through the IMAGE model, and then generate an impulse response signal of the target area.

[0127] The signal-to-noise ratio calculation submodule is used to calculate the signal-to-noise ratios corresponding to the voice data of each microphone array according to the impulse response signal and the noise signal and the reverberation signal in the voice data of each microphone array.

[0128] The optimal label determination submodule is used to determine the optimal microphone array label signal in each signal-to-noise ratio corresponding to each microphone array voice data.

[0129] In one implementation, the signal-to-noise ratios corresponding to the voice data of each microphone array are calculated by using the above formula (3).

[0130] In one implementation, microphone phase features of two different channels are extracted from each microphone array speech data, which is performed using the above formula (4).

[0131] In one implementation, the voice data optimization module 103 includes:

[0132] The network parameter adjustment submodule is used to gradually adjust the network parameters of the input layer, intermediate layer and output layer of the neural network model based on a preset activation function to optimize the neural network model so that it outputs the optimal speech signal of the distributed microphone array speech data.

[0133] The embodiment of the present invention also provides a distributed array-based voice data recognition device for identifying the optimal numerical voice signal corresponding to the voice data of each microphone array, such as Fig.11 As shown, it includes the following modules:

[0134] An optimal speech signal acquisition module 111 is used to acquire an optimal speech signal of distributed microphone array speech data;

[0135] The optimal speech signal recognition module 112 is used to input the optimal speech signal of the distributed microphone array speech data to the multi-channel attention feature fusion network model training, and then identify its accuracy.

[0136] In one embodiment, the optimal speech signal recognition module 112 inputs the optimal speech signal of the distributed microphone array speech data to the multi-channel attention feature fusion network model training, and then recognizes its accuracy, including:

[0137] The optimal speech signal recognition submodule is used to determine whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is the target value based on the standard threshold of the microphone array speech data; and / or, based on the standard microphone array speech data, predict whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is the target value.

[0138] In one implementation, the optimal speech signal recognition module 112 includes:

[0139] A first determination submodule is used to determine that if the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is less than or equal to the microphone array speech data standard threshold, then the optimal speech signal of the distributed microphone array speech data is accurate;

[0140] The second determination submodule is used to retrain using the distributed array-based speech data training method in Example 1 if the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is greater than the standard threshold of the microphone array speech data.

[0141] Example 4

[0142] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, the electronic device may include a processor 121 and a memory 122, wherein the processor 121 and the memory 122 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0143] The processor 121 may be a central processing unit (CPU). The processor 121 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0144] The memory 122 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules. The processor 121 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 122, that is, to implement the distributed array-based voice data training method and recognition method in the above method embodiment.

[0145] The memory 122 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 121, etc. In addition, the memory 122 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 122 may optionally include a memory remotely arranged relative to the processor 121, and these remote memories may be connected to the processor 121 via a network. Examples of the above-mentioned network include, but are not limited to, a power grid, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] One or more modules are stored in the memory 122, and when executed by the processor 121, the distributed array-based speech data training method and recognition method in the embodiments shown in the drawings are executed.

[0147] The specific details of the above electronic device can be understood by referring to the corresponding related descriptions and effects in the embodiments shown in the accompanying drawings, and will not be repeated here.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0149] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A distributed array-based speech data training method, characterized in that: The steps include: Determine the voice data of each microphone array from different directions and its optimal microphone array label signal; Inputting the microphone array speech data and the optimal microphone array label signal into the neural network model training, wherein microphone phase features of two different channels and energy features of microphone speech data of any channel are extracted from the microphone array speech data for splicing; Gradually optimizing the neural network model until it outputs an optimal speech signal for the distributed microphone array speech data; The optimal microphone array tag signal is determined by the following steps: In the target area, voice data from each microphone array is obtained from different directions; The speech data of each microphone array is simulated by an IMAGE model, thereby generating an impulse response signal of the target area; Calculating the signal-to-noise ratios corresponding to the microphone array speech data according to the impulse response signal and the noise signal and the reverberation signal in the microphone array speech data; Determining the optimal microphone array tag signal among the signal-to-noise ratios corresponding to the microphone array voice data; According to the impulse response signal and the noise signal and the reverberation signal in the microphone array voice data, the signal-to-noise ratios corresponding to the microphone array voice data are calculated, and the calculation is performed by the following formula: Among them, SNR true is the signal-to-noise ratio corresponding to the voice data of each microphone array, Y noisy is the noisy speech signal of the microphone array speech data, Y noisy =Y target +Y rever b+Y noise , Y target The target speech signal and its early reverberation signal collected by each microphone, Y reverb is the late reverberation signal collected by each microphone, Y noise is the noise signal collected by each microphone; Extracting microphone phase features of two different channels from each microphone array voice data is performed by the following formula: in, is the microphone phase characteristic of channel i, is the microphone phase characteristic of channel j, and IPD is the cosine value of the difference between the microphone phase characteristic of channel i and the microphone phase characteristic of channel j.

2. The distributed array-based speech data training method according to claim 1, characterized in that: The step of gradually optimizing the neural network model until it outputs an optimal speech signal of the distributed microphone array speech data comprises: Based on a preset activation function, the network parameters of the input layer, the intermediate layer and the output layer of the neural network model are gradually adjusted to optimize the neural network model so that it outputs the optimal speech signal of the distributed microphone array speech data.

3. A distributed array-based speech data recognition method for identifying the optimal speech signal of the distributed microphone array speech data according to any one of claims 1 to 2, characterized in that: include: Obtaining an optimal voice signal of the distributed microphone array voice data; The optimal speech signal of the distributed microphone array speech data is input into the multi-channel attention feature fusion network model training to further identify its accuracy.

4. The distributed array-based speech data recognition method according to claim 3, characterized in that: Inputting the optimal speech signal of the distributed microphone array speech data to the multi-channel attention feature fusion network model training, and then identifying its accuracy, includes: Based on the standard threshold of the microphone array speech data, determine whether the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is the target value; and / or, based on the standard microphone array speech data, predict whether the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is the target value.

5. The distributed array voice data recognition method according to claim 4, characterized in that: The method of determining whether the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is a target value based on the standard threshold of the microphone array speech data includes: If the recognition result of the optimal speech signal of the distributed microphone array speech data output by the multi-channel attention feature fusion network model is less than or equal to the standard threshold of the microphone array speech data, then the optimal speech signal of the distributed microphone array speech data is accurate; If the recognition result of the optimal speech signal output by the multi-channel attention feature fusion network model for the distributed microphone array speech data is greater than the standard threshold of the microphone array speech data, the distributed array-based speech data training method described in any one of claims 1 to 2 is used for retraining.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the distributed array-based speech data training method described in any one of claims 1 to 2; or, to execute the distributed array-based speech data recognition method described in any one of claims 3 to 5.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the distributed array-based speech data training method according to any one of claims 1 to 2 by executing the computer instructions; Or, execute the distributed array-based speech data recognition method described in any one of claims 3 to 5.

Images