Voice processing method, electronic equipment and storage medium

Abstract

The invention discloses a voice processing method, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-channel voice data; processing the multi-channel voice data to obtain spatial features and spectral features; performing time-frequency modeling on the spatial features to obtain spatial time-frequency features, and performing time-frequency modeling on the spectral features to obtain spectral time-frequency features; and fusing and decoding the spatial time-frequency features and the spectral time-frequency features to obtain mask information which is used for separating target voice data from the multi-channel voice data. The recognition efficiency of the target voice is improved.

Description

Technical Field

[0001] The embodiments disclosed in the present application relate to the field of voice interaction technology, and more specifically, to a voice processing method, an electronic device and a storage medium. Background Art

[0002] In many application scenarios of smart cockpits, when a specific passenger wakes up the in-vehicle voice assistant and conducts voice interaction, other non-target speech components in the microphone signal need to be removed to ensure that the voice assistant can communicate normally with the specific passenger. Among them, body noise, road driving noise, and interference from non-target speakers in other sound areas will greatly affect the voice assistant's recognition accuracy of the target voice, which will cause it to fail to respond normally and affect the user experience. Summary of the invention

[0003] According to the embodiments of the present application, the present application proposes a speech processing method, an electronic device and a storage medium to solve the above problems.

[0004] The first aspect of the present application discloses a speech processing method, including: acquiring multi-channel speech data; processing the multi-channel speech data to obtain spatial features and spectral features; performing time-frequency modeling on the spatial features to obtain spatial time-frequency features, and performing time-frequency modeling on the spectral features to obtain spectral time-frequency features; fusing and decoding the spatial time-frequency features and the spectral time-frequency features to obtain mask information, wherein the mask information is used to separate target speech data from the multi-channel speech data.

[0005] In some embodiments, the time-frequency modeling of the spatial features to obtain spatial time-frequency features includes: performing time modeling on the spatial features to obtain spatial time features; performing frequency modeling on the spatial time features to obtain the spatial time-frequency features; the time-frequency modeling of the spectral features to obtain spectral time-frequency features includes: performing time modeling on the spectral features to obtain spectral time features; performing frequency modeling on the spectral time features to obtain the spectral time-frequency features.

[0006] In some embodiments, the temporal modeling of the spatial features to obtain the spatial-temporal features includes: using a space-time network to perform temporal modeling on the spatial features to obtain the space-time features, wherein the space-time network includes a multi-head self-attention module; performing frequency modeling on the space-time features to obtain the space-time-frequency features, including: using a space-frequency network to perform frequency modeling on the space-time features to obtain the space-time-frequency features, wherein the space-frequency network includes a first gated recurrent unit module.

[0007] In some embodiments, the space-time network also includes a first normalization layer and a first fully connected layer; the space-time network is used to perform time modeling on the spatial features to obtain the space-time features, including: passing the spatial features through the first normalization layer, the multi-head self-attention module and the first fully connected layer in sequence to obtain initial space-time features; superimposing the initial space-time features with the spatial features to obtain the space-time features; the space-frequency network also includes a second normalization layer and a second fully connected layer; the space-frequency network is used to perform frequency modeling on the space-time features to obtain the space-time-frequency features, including: passing the space-time features through the second normalization layer, the first gated recurrent unit module and the second fully connected layer in sequence to obtain initial space-time-frequency features; superimposing the initial space-time-frequency features with the space-time features to obtain the space-time-frequency features.

[0008] In some embodiments, the time modeling of the spectral features to obtain the spectral time features includes: using a spectral time network to perform time modeling on the spectral features to obtain the spectral time features, wherein the spectral time network includes a second gated recurrent unit module; the frequency modeling of the spectral time features to obtain the spectral time-frequency features includes: using the spectral frequency network to perform frequency modeling on the spectral time features to obtain the spectral time-frequency features, wherein the spectral frequency network includes a third gated recurrent unit module.

[0009] In some embodiments, the spectrum-time network also includes a third normalization layer and a third fully connected layer; the spectrum-time network is used to perform time modeling on the spectrum feature to obtain the spectrum-time feature, including: passing the spectrum feature through the third normalization layer, the second gated recurrent unit module and the third fully connected layer in sequence to obtain an initial spectrum-time feature; superimposing the initial spectrum-time feature with the spectrum feature to obtain the spectrum-time feature; the spectrum-frequency network also includes a fourth normalization layer and a fourth fully connected layer; the spectrum-frequency network is used to perform frequency modeling on the spectrum-time feature to obtain the spectrum-time-frequency feature, including: passing the spectrum-time feature through the fourth normalization layer, the third gated recurrent unit module and the fourth fully connected layer in sequence to obtain an initial spectrum-time-frequency feature; superimposing the initial spectrum-time-frequency feature with the spectrum-time feature to obtain the spectrum-time-frequency feature.

[0010] In some embodiments, the multi-channel speech data is processed to obtain the spatial features, including: obtaining the time-frequency domain phase difference features and the time-frequency domain sound level difference features of the multi-channel speech data, and splicing the time-frequency domain phase difference features and the time-frequency domain sound level difference features to obtain initial spatial features; splitting and splicing the initial spatial features to obtain the spatial features; processing the multi-channel speech data to obtain the spectral features, including: obtaining the logarithmic power spectrum features of the multi-channel speech data as the initial spectral features; splitting and splicing the initial spectral features to obtain the spatial features.

[0011] In some embodiments, the initial spatial features are split and concatenated to obtain the spatial features, including: splitting the initial spatial features in frequency according to a predetermined distribution to obtain multiple initial spatial feature sub-bands; passing the multiple initial spatial feature sub-bands through a normalization layer and a fully connected layer in sequence to obtain multiple corresponding spatial sub-band features; passing the multiple sub-band features in the order of the predetermined distribution to obtain the spatial features; splitting and concatenating the initial spectral features to obtain the spectral features, including: splitting the initial spectral features in frequency according to the predetermined distribution to obtain multiple initial spectral feature sub-bands; passing the multiple initial spectral feature sub-bands through a normalization layer and a fully connected layer in sequence to obtain multiple corresponding spectral sub-band features; passing the multiple sub-band features in the order of the predetermined distribution to obtain the spectral features.

[0012] In some embodiments, target channel voice data is determined from the multi-channel voice data; the mask information is applied to the target channel voice data to obtain initial voice data; and the initial voice data is inversely transformed to obtain the target voice data.

[0013] The second aspect of the present application discloses an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech processing method described in the first aspect.

[0014] The third aspect of the present application discloses a non-volatile computer-readable storage medium having program instructions stored thereon, and the program instructions, when executed by a processor, implement the speech processing method described in the first aspect.

[0015] The beneficial effects of the present application include: acquiring multi-channel speech data, processing the multi-channel speech data to obtain spatial features and spectral features, performing time-frequency modeling on the spatial features to obtain spatial-time-frequency features, and performing time-frequency modeling on the spectral features to obtain spectral-time-frequency features, and further, fusing and decoding the spatial-time-frequency features and the spectral-time-frequency features to obtain mask information, which is used to separate target speech data from the multi-channel speech data, thereby improving the recognition efficiency of the target speech and improving the user experience of multi-channel speech interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present application will be further described below in conjunction with the accompanying drawings and implementation methods, in which:

[0017] Figure 1 is a flow chart of a speech processing method according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of sub-band splitting in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a model structure of a speech processing method according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of spatial time-frequency characteristics and spectral time-frequency characteristics of an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a speech signal of a speech processing method according to an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0023] Figure 7 It is a schematic diagram of the structure of the non-volatile computer-readable storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, and can mean including any one or more elements selected from the set consisting of A, B, and C. In addition, the terms "first", "second", and "third" in this application are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0026] At present, traditional speech separation methods mainly rely on signal processing technology and corresponding mathematical models. For example, Independent Component Analysis (ICA) is a blind source separation algorithm based on statistical learning technology. It assumes that the mixed signal is a linear superposition of independent sources. The goal is to separate the individual source components from the mixed signal. Beamforming (BF) is a spatial filtering technology that uses a microphone array to capture sound from a specific direction while attenuating sounds from other directions. The signals from the microphones are combined to create a directional sensitivity pattern, effectively focusing on the sound source of interest. The speech separation method based on deep learning uses a deep neural network (DNN) to learn the complex patterns of audio signals. Through training on a large amount of labeled data, it can automatically extract features and perform classification or regression tasks. DNN is usually used to estimate the ideal ratio masking or ideal binary masking to achieve the separation of the target speech in the mixed audio. Among them, speech separation technology based on traditional signal processing algorithms is often limited by its assumptions about the statistical characteristics of the signal, resulting in obvious performance fluctuations in different application scenarios and poor adaptability; speech separation algorithms based on deep learning all regard the model as a black box, often paying less attention to the characteristics of the speech signal, and the speech separation effect is poor.

[0027] To this end, the present application provides a speech processing method, an electronic device and a storage medium to solve the above problems.

[0028] In order to enable those skilled in the art to better understand the technical solution of the present application, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] See also Figure 1 , Figure 1The figure is a flow chart of the speech processing method of the embodiment of the present application. The execution subject of the method can be an electronic device with computing function, for example, a microcomputer, a server, and a mobile device such as a notebook computer and a tablet computer.

[0030] It should be noted that if there are substantially the same results, the method of the present application is not limited to Figure 1 The process sequence shown is limited.

[0031] In some possible implementations, the method may be implemented by a processor calling computer-readable instructions stored in a memory, such as Figure 1 As shown, the method may include the following steps:

[0032] S11: Acquire multi-channel voice data.

[0033] For example, in a car smart cockpit scenario, multi-channel voice data is obtained. Taking a four-seater car as an example, two microphones are usually installed in the front and back seats of the vehicle to pick up the main speaker signals in four sound zones, and the obtained four-channel voice data.

[0034] S12: Process the multi-channel speech data to obtain spatial features and spectral features.

[0035] Process the multi-channel speech data, for example, by calculating the multi-channel speech signal to obtain the spatial features and spectral features corresponding to the multi-channel speech signal, wherein the spatial features can be used to help identify and separate sounds from different directions, for example, the spatial features can include the time delay information of different sound source signals reaching different microphones, the energy difference information of the same sound source signals reaching different microphones, etc. The spectral features can be used to estimate the target speech signal for enhancement or separation, for example, the spectral features can include the relationship between the power of a single-channel signal and the frequency, etc.

[0036] S13: Perform time-frequency modeling on the spatial features to obtain spatial time-frequency features, and perform time-frequency modeling on the spectral features to obtain spectral time-frequency features.

[0037] The spatial features are modeled in time and frequency to obtain the spatial time and frequency features, that is, to obtain the time information and frequency information of the spatial features. And the spectral features are modeled in time and frequency to obtain the spectral time and frequency features, that is, to obtain the time information and frequency information of the spectral features. For example, the spatial features and the spectral features are modeled separately in a decoupled manner so that information can be continuously exchanged during the spatial feature modeling and the spectral feature modeling process.

[0038] S14: Fusing and decoding the spatial time-frequency features and the spectral time-frequency features to obtain mask information, wherein the mask information is used to separate the target speech data from the multi-channel speech data.

[0039] The spatial time-frequency features and the spectral time-frequency features are fused and decoded to obtain mask information, where the mask information is used to separate the target speech data from the multi-channel speech data. For example, the obtained mask information can be used to identify the target speech signal from the multi-channel speech signal with interference, and the target speech signal can be used in subsequent interactive tasks such as speech recognition.

[0040] In this embodiment, multi-channel speech data is acquired, and the multi-channel speech data is processed to obtain spatial features and spectral features. Time-frequency modeling is performed on the spatial features to obtain spatial-time-frequency features, and time-frequency modeling is performed on the spectral features to obtain spectral-time-frequency features. Furthermore, mask information is obtained by fusing and decoding the spatial-time-frequency features and the spectral-time-frequency features. The mask information is used to separate target speech data from the multi-channel speech data, thereby improving the recognition efficiency of the target speech and enhancing the user experience of multi-channel voice interaction.

[0041] In some embodiments, multi-channel speech data is processed to obtain spatial features, including: obtaining time-frequency domain phase difference (IPD) features and time-frequency domain level difference (ILD) features of the multi-channel speech data, and splicing the time-frequency domain phase difference features and the time-frequency domain level difference features to obtain initial spatial features; splitting and splicing the initial spatial features to obtain spatial features.

[0042] The time-frequency domain phase difference feature is used to characterize the time delay information of different sound source signals reaching different microphones. When the target sound source is given, the phase difference of the corresponding speech segment received by each microphone is basically fixed, that is, considering the situation where the speaker moves in a small range, the activity range of each area in the car is limited. The calculation method of the time-frequency domain phase difference feature is as follows:

[0043] IPD(t,f)=angle(STFT(s 1 ))-angle(STFT(s 2 ))

[0044] Among them, S 1 represents the first channel speech signal, S 2 represents the second channel speech signal, STFT(·) is used to calculate the short-time Fourier spectrum characteristics of the signal, and angle(·) is used to calculate the phase angle information at each time-frequency point.

[0045] The time-frequency domain sound level difference feature mainly represents the energy difference information of different sound source signals reaching different microphones. When the target sound source is given, the sound pressure level difference of the corresponding speech segment in each microphone is relatively constant. The calculation method of the time-frequency domain sound level difference feature is as follows:

[0046] ILD(t,f)=abs(STFT(s 1 )) / (abs(STFT(s 2 ))+ε))

[0047] Among them, S 1 represents the first channel speech signal, S 2 represents the second channel speech signal, STFT(·) is used to calculate the short-time Fourier spectrum features of the signal, and abs(·) is used to calculate the amplitude information at each time-frequency point.

[0048] The time-frequency domain phase difference feature and the time-frequency domain sound level difference feature are spliced ​​to obtain the initial spatial feature. For example, the IPD feature and ILD feature between the four-channel microphone signals are calculated and spliced ​​to obtain 7-channel features as the initial spatial feature. For example, the four-channel microphone signal includes the driver's microphone signal a, the passenger's microphone signal b, the main rear microphone signal c, and the passenger rear microphone signal d. Taking the driver's seat as the target area, the four-channel microphone signals {a, b, c, d} are spliced ​​to obtain 4 IPD features and 3 ILD features, where the 4 IPD features include {IPD[a, b], IPD[c, d], IPD[a, c], IPD[b, d]}, and the 3 ILD features include {ILD[a, b], ILD[a, c], ILD[a, d]}. Among them, IPD[S 1 ,S 2 ] represents the microphone signal S 1 With microphone signal S 2 The corresponding IPD feature, ILD[S 1 ,S 2 ] represents the microphone signal S 1 With microphone signal S 2 Corresponding ILD characteristics.

[0049] Furthermore, the initial spatial features are split and concatenated to obtain spatial features, including: splitting the initial spatial features in frequency according to a predetermined distribution to obtain multiple initial spatial feature sub-bands; passing the multiple initial spatial feature sub-bands through a normalization layer and a fully connected layer in sequence to obtain corresponding multiple spatial sub-band features; and arranging the multiple sub-band features in a predetermined distribution order to obtain spatial features.

[0050] For example, the initial spatial feature obtained is X 1 ∈R F×T , where F and T represent the frequency and time dimensions in the time-frequency domain, respectively. Figure 2 , Figure 2 is a schematic diagram of sub-band splitting in an embodiment of the present application, where the initial spatial feature X 1The frequency is split according to a predetermined distribution to obtain multiple initial spatial feature sub-bands. For example, the initial spatial feature X 1 Split into K subbands i=1,2,……K,where {G i} is a predefined subband splitting and grouping strategy, ∑G i = F. Multiple initial spatial feature subbands are passed through the normalization layer and the fully connected layer in sequence to obtain the corresponding multiple spatial subband features, that is, each subband B i It passes through a normalization (LayerNorm) layer and a fully connected (FC) layer in turn, and finally generates K sub-band features. Multiple sub-band features are distributed in a predetermined order to obtain spatial features, that is, K sub-band features Spatial features are obtained after splicing Right now Figure 2 As shown in (a) in .

[0051] In some embodiments, multi-channel speech data is processed to obtain spectral features, including: obtaining logarithmic power spectrum (LPS) features of the multi-channel speech data as initial spectral features; and splitting and splicing the initial spectral features to obtain spatial features.

[0052] The logarithmic power spectrum feature is the main feature of the auxiliary model to complete the target speech separation. Different from the spatial clues provided by the time-frequency domain phase difference feature and the time-frequency domain sound level difference feature, the logarithmic power spectrum feature contains the effective spectral information of the target speech and is used to participate in the output of the target speech mask information. The calculation method of the logarithmic power spectrum feature is as follows:

[0053] LPS(t,f)=log(power(abs(STFT(s)),2)+1)

[0054] Among them, S represents a single-channel speech signal, STFT(·) is used to calculate the short-time Fourier spectrum characteristics of the signal, abs(·) is used to calculate the amplitude information at each time-frequency point, power(·) is used to calculate the power, and log(·) is used for logarithmic calculation to compress the dynamic range of the power spectrum characteristics.

[0055] The logarithmic power spectrum features of the multi-channel speech data are obtained as the initial spectrum features. For example, the LPS features of the four-channel microphone signals are calculated, and the microphone signal in the area where the target speech is located is used as the first channel speech signal. Among all the four-channel signals, the signal-to-noise ratio of the target speech in the first channel speech signal is the highest, thereby reducing the difficulty of the model in estimating the target speech mask.

[0056] Furthermore, the initial spectral features are split and concatenated to obtain spectral features, including: splitting the initial spectral features in frequency according to a predetermined distribution to obtain multiple initial spectral feature sub-bands; passing the multiple initial spectral feature sub-bands through a normalization layer and a fully connected layer in sequence to obtain corresponding multiple spectral sub-band features; and arranging the multiple sub-band features in a predetermined distribution order to obtain the spectral features.

[0057] For example, the initial spectrum feature obtained is X 2 ∈R F×T , where F and T represent the frequency and time dimensions in the time-frequency domain, respectively. Figure 2 , the initial spectrum feature X 2 The frequency is split according to a predetermined distribution to obtain multiple initial spectrum feature sub-bands. For example, the initial spectrum feature X 2 Split into K subbands i=1,2,……K,where {G i} is a predefined subband splitting and grouping strategy, ∑G i = F. Multiple initial spectral feature sub-bands are passed through the normalization layer and the fully connected layer in sequence to obtain the corresponding multiple spectral sub-band features, that is, each sub-band B i It passes through a normalization (LayerNorm) layer and a fully connected (FC) layer in turn, and finally generates K sub-band features. The multiple sub-band features are distributed in a predetermined order to obtain the spectral features, that is, K sub-band features After splicing, we get the spectral feature Z α ∈R N×T×K ,Right now Figure 2 As shown in (b) in .

[0058] In some embodiments, time-frequency modeling is performed on spatial features to obtain spatial time-frequency features, including: performing time modeling on spatial features to obtain spatial time features; performing frequency modeling on spatial time features to obtain spatial time-frequency features; performing time-frequency modeling on spectral features to obtain spectral time-frequency features, including: performing time modeling on spectral features to obtain spectral time features; performing frequency modeling on spectral time features to obtain spectral time-frequency features.

[0059] See also Figure 3-Figure 4 , Figure 3 is a schematic diagram of a model structure of a speech processing method according to an embodiment of the present application. Figure 4 It is a schematic diagram of the spatial time-frequency characteristics and spectral time-frequency characteristics of an embodiment of the present application.

[0060] For spatial feature Z s Perform time modeling to obtain spatial and temporal features Spatial-temporal features Perform frequency modeling to obtain the spatial time-frequency feature Q s . And, for the spectral feature Z α Perform time modeling to obtain spectral time characteristics Time characteristics of spectrum Perform frequency modeling to obtain the spectral time-frequency characteristics Q α .

[0061] It can be understood that in this embodiment, the modeling of spatial features and spectral features is used as two branches of the model, so that in the actual modeling process, information flow and interaction will continue to occur between the two branches, thereby making full use of spatial information and spectral information and improving the speech separation effect of the model.

[0062] In some embodiments, spatial features are temporally modeled to obtain spatial-temporal features, including: using a space-time network to perform temporal modeling on spatial features to obtain space-time features, wherein the space-time network includes a multi-head self-attention module; spatial-temporal features are frequency modeled to obtain space-time-frequency features, including: using a space-frequency network to perform frequency modeling on space-time features to obtain space-time-frequency features, wherein the space-frequency network includes a first gated recurrent unit module.

[0063] Please continue reading Figure 3 , the space-time network can include a multi-head self-attention (MHSA) module, a first normalization layer and a first fully connected layer, and the multi-head self-attention module connects the first normalization layer and the first fully connected layer. The space-time network can be used to capture long-term information in the time dimension T. Among them, long-term information refers to information with weak time-varying properties in the time dimension of the entire input. Taking phase difference information as an example, the phase difference of a single speaker between different channels is basically constant.

[0064] Among them, the multi-head self-attention module is expressed as follows:

[0065]

[0066] o=Av

[0067] Among them, A is the weight matrix of the multi-head self-attention mechanism, o is the output feature of the multi-head self-attention module, q, k, v are the three key values ​​in the multi-head self-attention mechanism, softmax(·) is the activation function in deep learning, and D h is the dimension of the hidden layer features.

[0068] In this embodiment, the multi-head self-attention module aggregates features with high similarity in the time dimension without relying on the temporal relationship, and both the IPD feature and the ILD feature explicitly contain relevant information, thereby significantly improving the effect of spatial information modeling and assisting the separation task, that is, achieving the purpose of identifying different speakers through clustering, thereby assisting the separation task.

[0069] In some embodiments, a spatial feature Z is processed using a spatial-temporal network. s Perform temporal modeling to obtain spatial and temporal features Among them, the spatial feature Z s ∈R N×T×K After passing through the first normalization layer, the multi-head self-attention module and the first fully connected layer, the initial space-time features are obtained. The initial space-time features are combined with the spatial features Z s ∈R N×T×K Perform superposition processing to obtain spatial and temporal characteristics

[0070] Please continue reading Figure 3 In some embodiments, the spatial frequency network includes a first gated recurrent unit (GRU) module, a second normalization layer, and a second fully connected layer, wherein the first gated recurrent unit module is connected to the second normalization layer and the second fully connected layer. The spatial frequency network can be used to capture the distribution characteristics of the spectrum in the frequency F dimension.

[0071] Using the space-frequency network, the spatial and temporal features Perform frequency modeling to obtain the spatial time-frequency feature Q s Among them, the spatial and temporal features After passing through the second normalization layer, the first gated recurrent unit module and the second fully connected layer, the initial space-time-frequency features are obtained. The initial space-time-frequency features are combined with the space-time features. Perform superposition processing to obtain the spatial time-frequency feature Q s ∈R N×T×K ,like Figure 4 As shown in (a), Figure 4 (a) shows the spatial time-frequency features obtained by time-frequency modeling of spatial features.

[0072] In some embodiments, the spectral features are temporally modeled to obtain spectral time features, including: using a spectral time network to perform time modeling on the spectral features to obtain spectral time features, wherein the spectral time network includes a second gated recurrent unit module; the spectral time features are frequency modeled to obtain spectral time-frequency features, including: using a spectral frequency network to perform frequency modeling on the spectral time features to obtain spectral time-frequency features, wherein the spectral frequency network includes a third gated recurrent unit module.

[0073] Please continue reading Figure 3 In some embodiments, the spectral time network may include a second gated recurrent unit module, a third normalization layer, and a third fully connected layer, wherein the second gated recurrent unit module connects the third normalization layer and the third fully connected layer. The spectral time network can capture the duration distribution characteristics in the time T dimension.

[0074] Using the spectral time network, the spectral feature Z α Perform time modeling to obtain spectral time characteristics Among them, the spectral feature Z α ∈R N×T×K After passing through the third normalization layer, the second gated recurrent unit module and the third fully connected layer, the initial spectral time feature is obtained. The initial spectral time feature is combined with the spectral feature Z α ∈R N×T×K Perform superposition processing to obtain the spectrum time characteristics

[0075] Further, in some embodiments, the spectrum frequency network includes a third gated recurrent unit module, a fourth normalization layer and a fourth fully connected layer, wherein the third gated recurrent unit module is connected to the fourth normalization layer and the fourth fully connected layer. The spectrum frequency network can be used to capture the distribution characteristics of the spectrum in the frequency F dimension.

[0076] Using spectrum frequency network, the spectrum time characteristics Perform frequency modeling to obtain the spectral time-frequency characteristics Q α Among them, the spectral time feature After passing through the fourth normalization layer, the third gated recurrent unit module and the fourth fully connected layer, the initial spectrum time-frequency features are obtained. The initial spectrum time-frequency features are combined with the spectrum time features. Perform superposition processing to obtain the spectrum time-frequency characteristics Q α ∈R N×T×K ,like Figure 4 As shown in (b), Figure 4 (b) shows the spectral time-frequency features obtained by performing time-frequency modeling on the spectral features.

[0077] Furthermore, the spatial time-frequency feature Q s Sum spectrum time-frequency characteristics Q α Fusion and decoding are performed to obtain mask information, such as the spatial time-frequency feature Q s Sum spectrum time-frequency characteristics Q α The mask value is obtained by fusion and then through the narrowband mask decoding module. The mask information is used to separate the target speech data from the multi-channel speech data. For example, the spectrogram of the target speech signal can be obtained by multiplying the mask value by the spectrogram corresponding to the multi-channel speech data, and then the speech signal can be restored using the spectrogram of the target speech signal.

[0078] In some embodiments, target channel speech data is determined from multi-channel speech data; mask information is applied to the target channel speech data to obtain initial speech data; and the initial speech data is inversely transformed to obtain target speech data.

[0079] Determine the target channel speech data from the multi-channel speech data, for example, determine the speech signal of the target sound zone from the multi-channel speech signal. Apply the mask information (msak) to the target channel speech data to obtain the initial speech data, and perform an inverse transformation on the initial speech data to obtain the target speech data, for example, by performing a dot multiplication of the mask value of the target speech signal and the amplitude spectrum of the multi-channel speech, and finally perform an inverse Fourier transform (ISTFT) to obtain the enhanced speech signal, such as Figure 5 As shown, Figure 5 is a schematic diagram of a speech signal of a speech processing method according to an embodiment of the present application, Figure 5 (a) shows the input multi-channel speech data, Figure 5 (b) in the figure shows the target speech data which is finally output.

[0080] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0081] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of an electronic device of an embodiment of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of the above-mentioned speech processing method embodiment. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, which is not limited here.

[0082] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of the above-mentioned speech processing method embodiment. The processor 62 can also be called a CPU (Central Processing Unit), and the processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0083] See also Figure 7 , Figure 7 Schematic diagram of the structure of a non-volatile computer-readable storage medium according to an embodiment of the present application. The non-volatile computer-readable storage medium 70 is used to store a computer program 701. When the computer program 701 is executed by a processor, for example, Figure 6 When the processor 62 in the embodiment is executed, it is used to implement the steps of the above-mentioned embodiment of the speech processing method.

[0084] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0085] In the several embodiments provided in the present application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the above-described related device implementations are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication disconnection shown or discussed can be through some interfaces, indirect coupling or communication disconnection of devices or units, which can be electrical, mechanical or other forms.

[0086] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0088] It is easy for a person skilled in the art to know that many modifications and changes can be made to the device and method while maintaining the teaching content of the present application. Therefore, the above disclosure should be considered as limited only by the scope of the appended claims.

Claims

1. A speech processing method, characterized in that: include: Get multi-channel voice data; Processing the multi-channel speech data to obtain spatial features and spectral features; Performing time-frequency modeling on the spatial features to obtain spatial time-frequency features, and performing time-frequency modeling on the spectral features to obtain spectral time-frequency features; The spatial time-frequency features and the spectral time-frequency features are fused and decoded to obtain mask information, wherein the mask information is used to separate the target speech data from the multi-channel speech data.

2. The method according to claim 1, characterized in that The performing time-frequency modeling on the spatial features to obtain spatial time-frequency features includes: Performing temporal modeling on the spatial features to obtain spatial-temporal features; Performing frequency modeling on the space-time feature to obtain the space-time-frequency feature; The performing time-frequency modeling on the spectral features to obtain spectral time-frequency features includes: Performing time modeling on the spectrum feature to obtain a spectrum time feature; Frequency modeling is performed on the spectral time feature to obtain the spectral time-frequency feature.

3. The method according to claim 2, characterized in that The performing time modeling on the spatial features to obtain spatial time features includes: Using a space-time network, temporally modeling the spatial features to obtain the space-time features, wherein the space-time network includes a multi-head self-attention module; Performing frequency modeling on the space-time feature to obtain the space-time-frequency feature includes: The spatial-temporal features are frequency modeled by using a spatial-frequency network to obtain the spatial-temporal-frequency features, wherein the spatial-frequency network includes a first gated cyclic unit module.

4. The method according to claim 3, characterized in that The spatial-temporal network also includes a first normalization layer and a first fully connected layer; The using of the space-time network to perform time modeling on the space feature to obtain the space-time feature includes: Passing the spatial features through the first normalization layer, the multi-head self-attention module and the first fully connected layer in sequence to obtain initial spatial-temporal features; Superimposing the initial space-time feature with the space feature to obtain the space-time feature; The spatial frequency network also includes a second normalization layer and a second fully connected layer; The spatial-temporal features are frequency modeled using a spatial-frequency network to obtain the spatial-temporal-frequency features, including: Passing the spatial-temporal features through the second normalization layer, the first gated recurrent unit module, and the second fully connected layer in sequence to obtain initial spatial-temporal-frequency features; The initial space-time-frequency feature is superimposed with the space-time feature to obtain the space-time-frequency feature.

5. The method according to claim 2, characterized in that: The performing time modeling on the spectrum feature to obtain the spectrum time feature includes: Using a spectro-temporal network, temporally modeling the spectral feature to obtain a spectro-temporal feature, wherein the spectro-temporal network includes a second gated recurrent unit module; The performing frequency modeling on the spectrum time feature to obtain the spectrum time-frequency feature includes: The spectrum time feature is frequency modeled by using a spectrum frequency network to obtain the spectrum time-frequency feature, wherein the spectrum frequency network includes a third gated cycle unit module.

6. The method according to claim 5, characterized in that The spectro-temporal network further includes a third normalization layer and a third fully connected layer; The using of the spectrum-time network to perform time modeling on the spectrum feature to obtain the spectrum-time feature includes: Passing the spectral features through the third normalization layer, the second gated recurrent unit module, and the third fully connected layer in sequence to obtain initial spectral time features; Superimposing the initial spectrum-time feature with the spectrum feature to obtain the spectrum-time feature; The spectrum frequency network also includes a fourth normalization layer and a fourth fully connected layer; The spectrum time feature is frequency modeled by using a spectrum frequency network to obtain the spectrum time-frequency feature, including: Passing the spectrum time feature through the fourth normalization layer, the third gated recurrent unit module and the fourth fully connected layer in sequence to obtain an initial spectrum time-frequency feature; The initial spectrum time-frequency feature is superimposed with the spectrum time feature to obtain the spectrum time-frequency feature.

7. The method according to any one of claims 1 to 6, characterized in that Processing the multi-channel speech data to obtain the spatial features includes: Acquire the time-frequency domain phase difference feature and the time-frequency domain sound level difference feature of the multi-channel speech data, and concatenate the time-frequency domain phase difference feature and the time-frequency domain sound level difference feature to obtain an initial spatial feature; Splitting and concatenating the initial spatial features to obtain the spatial features; Processing the multi-channel speech data to obtain the spectral features includes: Acquire logarithmic power spectrum features of the multi-channel speech data as initial spectrum features; The initial spectral features are split and concatenated to obtain the spatial features.

8. The method according to claim 7, characterized in that The initial spatial features are split and concatenated to obtain the spatial features, including: Splitting the initial spatial feature in frequency according to a predetermined distribution to obtain a plurality of initial spatial feature sub-bands; Passing the multiple initial spatial feature subbands through a normalization layer and a fully connected layer in sequence to obtain corresponding multiple spatial subband features; The plurality of sub-band features are arranged in the order of the predetermined distribution to obtain the spatial feature; The initial spectrum features are split and concatenated to obtain the spectrum features, including: Splitting the initial spectrum feature in frequency according to the predetermined distribution to obtain a plurality of initial spectrum feature sub-bands; Passing the multiple initial spectral feature sub-bands through a normalization layer and a fully connected layer in sequence to obtain corresponding multiple spectral sub-band features; The spectral feature is obtained by arranging the multiple sub-band features in the order of the predetermined distribution.

9. The method according to claim 1, characterized in that: Also includes: Determining target channel voice data from the multi-channel voice data; Applying the mask information to the target channel speech data to obtain initial speech data; The initial voice data is inversely transformed to obtain the target voice data.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech processing method according to any one of claims 1 to 9.

11. A non-volatile computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the speech processing method according to any one of claims 1 to 9 is implemented.

Images