A generalizable method for speech generation from SQL

Abstract

The application discloses a generalizable voice generation SQL method. The technical scheme adopted by the application is as follows: step 1), acoustic information diversification processing and speaker information standardization processing are performed on original voice input by using voice reprogramming; step 2), semantic information rich audio representation is extracted by using a self-supervised audio representation model; step 3), the audio representation obtained in step 2) is classified by using a reverse gradient and a classifier on audio of different speakers, and gradient reversal is used during reasoning to learn speaker-independent audio representation; and step 4), the audio representation obtained in step 2) is input into a relation-aware transformer together with text database features, and the output is randomly input into an SQL decoder to obtain predicted SQL, and MLE loss is calculated with real SQL. The application adopts voice reprogramming and gradient reversal classifier technology, solves direct voice to SQL parsing, reduces acoustic method and style information in audio while retaining semantic information.

Description

TECHNICAL FIELD

[0001] The present application relates to SQL sentence generation and speech understanding, in particular to a generalizable speech-to-SQL method. BACKGROUND

[0002] Speech question generation database query sentence aims to convert spoken language questions into SQL queries for a given relational database, and the traditional method uses a cascading approach, that is, through an automatic speech recognition model and a text question generation database query sentence model. But this way also faces the following challenges: 1) the main problem of model training is data scarcity, and the available parallel data is limited; 2) the system should be robust enough to handle various out-of-domain speech samples different from the source data. SUMMARY

[0003] In order to overcome the limitations of multi-accent and multi-speaker on speech-to-SQL parsing, the present application provides a generalizable speech-to-SQL method, which uses speech reprogramming and gradient reversal classifier technology to solve direct speech-to-SQL parsing, reduces style information in acoustics and audio while retaining semantic information.

[0004] To this end, the present application adopts the following technical solution: a generalizable speech-to-SQL method, comprising:

[0005] Step 1), using speech reprogramming to perform acoustic information diversification processing and speaker information standardization processing on the original speech input, to obtain new audio composed of acoustic information-rich audio, speaker information-standardized audio, and original audio;

[0006] Step 2), using a self-supervised audio representation model to extract semantic information-rich audio representations;

[0007] Step 3), using the audio representations obtained in step 2) to classify audio of different speakers through backpropagation and a classifier, and using gradient reversal during inference to learn speaker-independent audio representations;

[0008] Step 4), inputting the audio representations obtained in step 2) into a relation-aware transformer (RAT) along with text database features, and randomly inputting the output into a SQL decoder to obtain predicted SQL, and calculating MLE loss with the true SQL.

[0009] The speech-to-SQL method is traditionally implemented in a cascaded manner, which faces the following challenges: 1) the model training faces the major problem of data scarcity, and the available parallel data is limited; 2) the system should be robust enough to handle various out-of-domain speech samples different from the source data. In the present invention, the first direct speech-to-SQL parsing is proposed, which avoids the error composition of the cross-level system. Specifically, 1) the recent achievements in large-scale pre-training are utilized, which shows that it solves the data scarcity problem and allows direct speech-to-SQL parsing; 2) the speech reprogramming and gradient reversal classifier techniques are introduced to reduce acoustic variance and learn style-independent representations, and improve the model's generalization ability to unseen out-of-domain custom data.

[0010] Further, in step 1), in order to perturb the acoustic features, the following functions are used: a) random resampling RR, b) formant shift fs, c) pitch randomization pr, d) random frequency shaping using a parametric equalizer peq.

[0011] Further, in step 1), for RR, random resampling is used to modify the rhythm, the original waveform is divided into several segments, the length of which is randomly and uniformly extracted from 19 frames to 32 frames, each segment is resampled using linear interpolation, and the resampling factor is randomly extracted from 0.5 to 1.5.

[0012] Further, in step 1), for fs, the formant shift ratio is uniformly sampled from Uniform(1, 1.4); after sampling the ratio, it is randomly decided whether to take the inverse of the sampling ratio.

[0013] Further, in step 1), for pr, the pitch shift ratio and the pitch range ratio are uniformly sampled from Uniform(1, 2) and Uniform(1, 1.5), respectively; after sampling, it is randomly decided whether to take the inverse of the sampling ratio.

[0014] Further, in step 1), peq represents a series of combined low-shelf, peak, and high-shelf filters, using one low-shelf HLS, one high-shelf HHS, and eight peak filters HPeak.

[0015] Further, the specific content of step 2) is:

[0016] To alleviate the problem of data scarcity and learn language content from raw waveforms, a self-supervised speech representation model Hubert is used, which uses a multi-layer convolutional waveform encoder to generate audio features, and then constructs a context representation through a transformer context encoder;

[0017] The Hubert-Base model is used as the speech representation, which is pre-trained on 960 hours of LibriSpeech. It is worth noting that the speech representation can not only incorporate rich acoustic information, but also incorporate acoustic attributes related to accent and speaker.

[0018] Further, the specific content of step 3) is:

[0019] To eliminate the speaker identity in the speech representation, a gradient reversal layer GRL is introduced in the speaker classifier, which regards the speaker variation as a classification problem and directly maximizes the loss of the domain by reversing its gradient to optimize the classifier; in back propagation, GRL obtains the gradient from the subsequent layer and changes its sign by multiplying by-1 before passing it to the previous layer:

[0020] R(x)=x,

[0021] Where I represents the unit matrix; R represents the operation performed by the layer; x represents the input into the GRL layer.

[0022] Further, step 4) includes:

[0023] Step 41), the audio representation obtained in step 2) is cross-modally fused with the schema of the text to obtain an encoder output feature containing question and schema information, and the calculation formula is as follows:

[0024]

[0025] Where R ij is the pre-acquired edge relationship, representing the predefined cross relationship between the question and the schema itself and each other; H is the head number; are linear layer parameter matrices of query, key and value respectively; d k represents the dimension of the key; d model represents the model dimension; q i represents the i-th question; s j represents the j-th schema; e ij represents e(q i ,s j ), e(q i ,s j ) represents the intermediate obtained value; z(q i ,s i ) represents the alignment relationship of each q and s;

[0026] Step 42), input the encoder output obtained in step 41) into a SQL decoder, the SQL decoder follows a syntax-based architecture, which generates SQL as an abstract syntax tree AST in a depth-first traversal order; the generation process of the SQL AST is decomposed into sequential actions, which are divided into two cases: 1) APPLYRULE, which extends the last generated node or completes a leaf node according to the grammar rule, 2) SELECCTCOLUMN and SELECTTABLE, which represent selecting a column or a table item from the architecture, respectively.

[0027] Further, step 4) further comprises:

[0028] Step 43), the probability of generating a SQL y is defined as:

[0029]

[0030] where x is the encoded question, column and table, a t is the action label at time step t, a<t is the sequential action before time step t; y represents SQL;

[0031] In the tree-structured LSTM decoder, the hidden state at each time step t is updated as mt, h t = LSTM([a t-1 ; p t ; c t ; n t ], m t-1 , h t-1 ), where m t is the cell state at time step t, h t is the hidden state, a t-1 is the previous action embedding, p t is the parent information of the current node, c t is the context vector, n t is the embedding of the current node type, the parent information contains the hidden state and embedding of the parent action, and the context vector is calculated using multi-head attention on x and h t-1 ; finally, how to calculate the action probability p(a t |x, a<t) is explained as follows:

[0032] For the APPLYRULE operation,

[0033] p(a t = AR[r] | x, a ＜t ) = softmaxR(g(h t ))

[0034] where AR is the APPLYRULE action, g(·) is a feed-forward network consisting of two linear layers and a tanh activation function;

[0035] For the SELECTTABLE operation,

[0036]

[0037]

[0038] where x j represents the jth column of x; ST represents the SELECTABLE operation, and the calculation of the SELECTCOLUMN operation is consistent therewith.

[0039] The present application has the beneficial effects as follows: avoid the error composition of cross-level contact system, specifically, 1) in order to accelerate the speech-driven SQL analysis research in the community, a large-scale and multi-speaker data set MASpider is released; 2) the data scarcity problem is solved and direct speech-to-SQL analysis is allowed; 3) speech reprogramming and gradient reversal classifier technology are introduced, acoustic variance and learning style-independent representation are reduced, and the generalization ability to invisible out-of-domain custom data is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is an overview diagram of an end-to-end network system used by the present application, including a self-supervised speech representation model Hubert, a language model GloVe, a bidirectional LSTM, a gradient reversal layer, a domain classifier, a RAT and a SQL decoder. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] As Figure 1 shown, the present application provides a generalizable speech-to-SQL method, comprising the following steps:

[0043] Step 1), acoustic information diversification processing and speaker information standardization processing are performed on the original speech input by speech reprogramming, to obtain new audio composed of audio containing rich acoustic information and audio after speaker information planning and original audio;

[0044] Step 2), extracting audio representation with rich semantic information by using self-supervised audio representation model;

[0045] Step 3), classifying audio of different speakers by using the audio representation obtained in step 2) through reverse gradient and classifier, and using gradient inversion during inference to learn speaker-independent audio representation;

[0046] Step 4), inputting the audio representation obtained in step 2) into a relation-aware transformer (RAT) together with text database features, and then inputting the output into a SQL decoder to obtain predicted SQL and calculate MLE loss with real SQL.

[0047] The step 1) uses speech reprogramming to obtain acoustic information enhanced audio input, and the specific steps are as follows:

[0048] In order to disturb the acoustic features, the following functions are used, namely a) random resampling RR, and b) formant shift fs, and c) pitch randomization pr, and d) random frequency shaping using a parametric equalizer peq.

[0049] For RR, random resampling is used to modify the rhythm. The original waveform is divided into several segments, and the length is randomly and uniformly extracted from 19 frames to 32 frames. Linear interpolation is used to resample each segment, and the resampling factor is randomly extracted from 0.5 to 1.5.

[0050] For fs, the formant shift ratio is uniformly sampled from Uniform(1, 1.4). After sampling the ratio, it is randomly determined whether to take the reciprocal of the sampling ratio.

[0051] For pr, the pitch shift ratio and the pitch range ratio are uniformly sampled from Uniform(1, 2) and Uniform(1, 1.5), respectively. Similarly, it is randomly determined whether to take the reciprocal of the sampling ratio.

[0052] Peq represents a series of combined low tilt, peak and high tilt filters. The present application uses one low shelf HLS, one high shelf HHS and eight peak filters HPeak.

[0053] The step 2) is specifically as follows:

[0054] In order to alleviate the problem of data scarcity and learn language content from the original waveform, the present application uses a self-supervised speech representation model Hubert to generate audio features using a multi-layer convolutional waveform encoder, and then constructs a context representation through a transformer context encoder.

[0055] The Hubert-Base model is used as the speech representation, which is pre-trained on 960 hours of LibriSpeech. It is worth noting that it is found that the speech representation can not only incorporate rich acoustic information, but also incorporate acoustic attributes related to accent and speaker.

[0056] The step 3) is specifically:

[0057] In order to eliminate the speaker identity in the speech representation, a gradient reversal layer (GRL) is introduced in the speaker classifier, which regards the speaker variation as a classification problem and directly maximizes the loss of the domain by reversing its gradient to optimize the classifier. In back propagation, GRL obtains the gradient from the subsequent layer and changes its sign by multiplying by-1 before passing it to the previous layer:

[0058] R(x) = x,

[0059] Wherein, I represents a unit matrix; R represents the operation performed by the layer; x represents the input of the GRL layer.

[0060] The step 4) is specifically:

[0061] 4.1) Cross-modal fusion of the audio representation obtained in step 2) and the schema of the text to obtain an encoder output feature containing question and schema information, and the calculation formula is as follows:

[0062]

[0063] Wherein, R ij is the pre-acquired edge relationship, indicating the predefined cross relationship between the question and the schema itself and each other; H is the head number; are linear layer parameter matrices of query, key and value respectively; d k represents the dimension of key; d model represents the model dimension; q i represents the i-th question; s j represents the j-th schema; e ij represents e(q i ,s j ), e(q i ,s j ) represents the intermediate obtained value; z(q i ,s i ) represents the alignment relationship of each q and s.

[0064] 4.2) The encoder output obtained from step 4.1) is input into a SQL decoder, which follows a syntax-based architecture that generates SQL as an Abstract Syntax Tree (AST) in a depth-first traversal order. The generation process of the SQL AST is decomposed into sequential actions, which are categorized into two cases: (1) APPLYRULE, which extends the last generated node or completes a leaf node according to the grammar rules, (2) SELECCTCOLUMN and SELECTTABLE, which represent the selection of a column or a table item from the schema, respectively.

[0065] 4.3.1) The probability of generating a SQL y is defined as:

[0066]

[0067] where x is the encoded memory of questions, columns, and tables, a t is the action label at time step t, a<t is the sequential action before time step t.

[0068] 4.3.2) In the tree-structured LSTM decoder, the hidden state at each time step t is updated as mt, h t = LSTM([a t-1 ; p t ; c t ; n t ], m t-1 , h t-1 ), where m t is the cell state at time step t, h t is the hidden state, a t-1 is the previous action embedding, p t is the parent information of the current node, c t is the context vector, n t is the embedding of the current node type, the parent information contains the hidden state and embedding of the parent action, and the context vector is calculated using multi-head attention over x and h t-1 ; finally, how to calculate the action probability p(a t | x, a<t) is explained as follows:

[0069] For the APPLYRULE operation,

[0070] p(a t = AR[r] | X, a ＜t ) = softmaxR(g(h t ))

[0071] where AR is the APPLYRULE action, and g(·) is a feed-forward network composed of two linear layers and a tanh activation function;

[0072] For the SELECT TABLE operation,

[0073]

[0074]

[0075] where x j represents the jth column of x; ST represents the SELECTABLE operation, and the calculation of the SELECT COLUMN operation is consistent therewith.

[0076] Application example

[0077] The method described in the present application is tested on the labeled MASpider, and the experimental results of training based on the TTS dataset are compared between the cascade system and the base model (a model other than the method proposed in the present application). In order to objectively evaluate the performance of the algorithm of the present application, the component matching accuracy and the exact match accuracy are used as evaluation criteria to evaluate the effect of the present application in the selected test set. The experiment is completed under two settings, 1) testing the effect of the model, in which setting the accent and speaker of the test set are visible in the training set, but the database of the test set is still invisible; 2) generalization to custom data, in which setting the accent, speaker and database of the test set are all invisible.

[0078] According to the steps described in the specific embodiment, the obtained experimental results are shown in Table 1 and Table 2, and the method of the present application is represented as Wav2SQL.

[0079] It can be found from Table 1 that Wav2SQL is higher than the cascade model in all component matching accuracy and exact match accuracy, and achieves a competitive result compared with the upper bound model TTS+S2SQL, which proves the effectiveness of the method of the present application. For the generalization of custom data, Wav2SQL is also higher than the cascade system in most component matching accuracy, which proves that Wav2SQL has stronger generalization compared with the cascade model, and proves that the method proposed in the present application can effectively alleviate the restriction of multiple accents and multiple speakers on the speech generation SQL task.

[0080] It can also be found from Table 2 that Wav2SQL is better than the cascade model, so the conclusion obtained is similar.

[0081] Table 1 Test results of component matching and exact match accuracy obtained by the present application on MASpider

[0082]

[0083] Table 2 Test results according to difficulty on MASpider for the present application

[0084]

[0085] Finally, it should be noted that the above examples are only intended to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A generalizable method for generating SQL from speech, characterized in that: include: Step 1) using voice reprogramming to perform acoustic information diversification processing and speaker information normalization on the original voice input, thereby obtaining a new audio composed of audio rich in acoustic information, audio normalized by speaker information, and the original audio; Step 2) using a self-supervised audio representation model to extract semantically rich audio representations; Step 3) The audio representation obtained in step 2) is used to classify the audio of different speakers through reverse gradient and classifier, and gradient reversal is used during inference to learn speaker-independent audio representation; Step 4) Input the audio representation obtained in step 2) and the text database features into the relation-aware transformer, and then input the output into the SQL decoder to obtain the predicted SQL, and calculate the MLE loss with the actual SQL; Step 4) specifically includes: In step 41), the audio representation obtained in step 2) is cross-modally fused with the text schema to obtain the encoder output features containing question and schema information. The calculation formula is as follows: z(q i ,s i )=softmax(e ij )(s i W V +R ij ) Among them, R ij is the pre-obtained edge relationship, which represents the predefined cross-relationship between the question and the pattern itself and each other; H is the number of heads; are the linear layer parameter matrices for query, key, and value, respectively; d k Indicates the dimension of the key; d model Indicates the model dimension; q i Indicates the i-th question; s j Indicates the jth schema; e ij represents e(q i ,s j ), e(q i ,s j ) represents the intermediate value; z(q i ,s i ) represents the alignment relationship between each q and s; In step 42), the encoder output obtained in step 41) is input into the SQL decoder. The SQL decoder follows a grammar-based architecture and generates SQL into an abstract syntax tree (AST) in a depth-first traversal order. The SQL AST generation process is decomposed into sequential actions, which are divided into two cases: 1) APPLYRULE, which expands the last generated node or completes a leaf node according to the grammar rules; 2) SELECCTCOLUM and SELECTTABLE, which respectively represent selecting columns or table items from the schema.

2. A generalizable speech-to-SQL method according to claim 1, characterized in that: In step 1), in order to perturb the acoustic features, the following functions are used: a) random resampling RR, b) formant shift fs, c) pitch randomization pr, d) random frequency shaping using a parametric equalizer peq.

3. A generalizable speech-to-SQL method according to claim 2, characterized in that: In step 1), for RR, random resampling is used to modify the rhythm. The original waveform is divided into several segments, whose lengths are randomly and uniformly sampled from 19 frames to 32 frames. Each segment is resampled using linear interpolation, and the resampling factor is randomly sampled from 0.5 to 1.

5.

4. A generalizable speech-to-SQL method according to claim 2, characterized in that: In step 1), for fs, the resonance peak shift ratio is uniformly sampled from Uniform(1,1.4); after sampling the ratio, it is randomly decided again whether to take the inverse of the sampling ratio.

5. The generalizable speech-to-SQL method according to claim 2, wherein: In step 1), for pr, the pitch deviation rate and pitch range rate are uniformly sampled from Uniform(1,2) and Uniform(1,1.5), respectively; after sampling, it is randomly decided whether to take the inverse of the sampling ratio.

6. The generalizable speech-to-SQL method according to claim 2, wherein: In step 1), peq represents a series of combined low-shelving, peaking, and high-shelving filters, using one low shelf HLS, one high shelf HHS, and eight peaking filters HPeak.

7. The generalizable speech-to-SQL method according to claim 1, characterized in that: The specific content of step 2) is: To alleviate the data scarcity problem and learn language content from raw waveforms, we use the self-supervised speech representation model Hubert, which uses a multi-layer convolutional waveform encoder to generate audio features, and then constructs a contextual representation through a transformer context encoder. The Hubert-Base model is used as the speech representation, which is pre-trained on 960 hours of LibriSpeech.

8. The generalizable speech-to-SQL method according to claim 1, wherein: The specific content of step 3) is: To eliminate the speaker identity in the speech representation, a gradient reversal layer GRL is introduced in the speaker classifier, which regards the change of the speaker as a classification problem and directly maximizes the domain loss to optimize the classifier by reversing its gradient; in backpropagation, GRL obtains the gradient from the subsequent layer and changes its sign by multiplying it by -1 before passing it to the previous layer: Here, I represents the identity matrix; R represents the operation performed by the layer; and x represents the input passed into the GRL layer.

9. The generalizable speech-to-SQL method according to claim 1, wherein: Step 4) also includes: Step 43), the probability of generating an SQL y is defined as: where x is the encoded question, column, and table, and a t is the action token at time step t, and a<t is the sequential action before time step t; y represents SQL; In the tree-structured LSTM decoder, the hidden state at each time step t is updated to mt, h t = LSTM([a t-1 ; p t ; c t ; n t , m t-1 , h t-1 ), where m t is the cell state at time step t, h t is the hidden state, a t-1 is the previous action embedding, p t is the parent information of the current node, c t is the context vector, n t is the embedding of the current node type, the parent information includes the hidden state and embedding of the parent action, the context vector is calculated using multi-head attention on x and h t-1 ; finally, how to calculate the action probability p(a t |x, a<t) is explained as follows: For the APPLYRULE operation, p(a t =AR[r]|x,a <t )=softmax R (g(h t )) Where AR is the APPLYRULE action, g(·) is a feed-forward network consisting of two linear layers and a tanh activation function; For SELECTTABLE operations, Among them, x j Represents the jth column of x; ST represents the SELECTABLE operation, and the calculation of the SELECTCOLUMN operation is consistent with it.

Images