A language information dynamic detection method based on a deep neural network

Abstract

The application provides a kind of language information dynamic detection method based on deep neural network, it is related to speech processing technical field, the mixed language speech stream of the method is acquired and original acoustic feature is extracted;Attention vector of current time step is generated based on attention mechanism, and the speech feature frame sequence window of current time step is dynamically selected by combining one-way time selection and the restriction of specific time span;Local acoustic feature for detecting language information of current time step is generated by multiplying dynamic window signal with original acoustic feature signal;Arbitrary length local acoustic feature is regularized into fixed dimension feature, fixed dimension feature is input into deep neural network classifier, and the language probability value corresponding to the speech feature of current time step is output;Based on dynamic window information and language probability value, the start and end time of each speech segment in mixed speech stream and corresponding language label are output, and the time point of language switching is determined, and it is especially suitable for language switching identification in bilingual mixed speech stream.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and particularly relates to a language information dynamic detection method based on a deep neural network. BACKGROUND

[0002] The bilingual mixed speech stream is very difficult to recognize due to unknown language starting and ending positions. For the language recognition problem of the bilingual mixed speech stream switching between words, the second language vocabulary is usually inserted into the first language communication to replace the first language vocabulary. In the case of switching between words, on the one hand, the language switching time point position is not known in advance, and on the other hand, due to frequent language information switching and short time of most second language speech stream information, the traditional language classification method based on the entire test speech stream is also not applicable. For the bilingual mixed speech stream, it is not feasible to construct a classifier to directly distinguish the language category of each frame of input speech features. Because the frame-level language classifier will have a large error when distinguishing, if it is for a single language stream, a post-processing smoothing operation can be performed on the sequence of distinguishing results to obtain a language recognition result of the single language stream. However, when the speech stream is mixed with two languages, and the time point position of the language switching point is not known, the output result sequence of the frame-level classifier with large error will present the case of continuous jumping of language labels. In this case, the smoothing post-processing cannot solve the problem, and the result cannot obtain an ideal language classification result.

[0003] In order to solve the problem that other languages are mixed in a speech and cannot be effectively classified and recognized. The bilingual mixed speech stream can be divided into bilingual mixed speech stream switching between sentences and bilingual mixed speech stream switching between words according to the existence mode of information. For the language recognition problem of the bilingual mixed speech stream switching between sentences, it generally occurs in the application scene of bilingual recognition, and usually a conversation contains several sentences, each sentence can freely select a language. In this case, since the position of language switching is not known in advance, the starting and ending time point positions of the language are also not known in advance, and the traditional language classification is based on the assumption that the entire test speech stream is the same language, so the traditional language classification method is not applicable to the case of language switching in the same speech stream. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a language information dynamic detection method based on deep neural network, which is particularly suitable for language switching recognition in bilingual mixed speech stream, and by combining attention mechanism, multi-frame speech information, dynamic window selection, long-time information statistics and recurrent neural network, the detection accuracy of language switching point in bilingual mixed speech stream and the accuracy of language recognition are improved, and the problem that the traditional language classification method cannot effectively handle frequent and short language switching is solved.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] Based on the above purpose, in the first aspect, the present application provides a language information dynamic detection method based on deep neural network, comprising the following steps:

[0007] Obtain the mixed language speech stream to be detected and extract the original acoustic features;

[0008] Generate an attention vector of the current time step based on the attention mechanism, and dynamically select a speech feature frame sequence window of the current time step by combining one-way time selection and the restriction of a specific time span;

[0009] Multiply the dynamic window signal with the original acoustic feature signal to generate the local acoustic features of the current time step for detecting language information;

[0010] Regularize the local acoustic features of arbitrary length into fixed dimension features, and use the fixed dimension features as the input of neural network language classification;

[0011] Input the fixed dimension features into a deep neural network classifier to output the language probability value corresponding to the speech feature of the current time step;

[0012] Based on the dynamic window information and the language probability value, output the start and end time of each speech segment in the mixed speech stream and the corresponding language label, and determine the time point of language switching.

[0013] As a further scheme of the present application, the generation of the attention vector of the current time step based on the attention mechanism is:

[0014] a i = [a i ,...,a i ,...,a i ]

[0015] Generate the selected speech feature frame sequence window w i of the current time step based on the attention vector a i-1 , wherein:

[0016] w i = f(a i-1 )

[0017] Introducing the one-way time selection restriction, the time point position of the speech feature frame sequence window meets the monotone increasing property, in the calculation of the current window w i position information, the window w i-1 position information of the last time step is added, so that w i is later than w i-1 :

[0018] w i =f(a i ,w i-1 )

[0019] And add the restriction of limited time span, in the calculation of the current window w i position information, the window length limit parameter L w is added, so that the window shape does not appear scattered state:

[0020] w i =f(a i ,w i-1 ,L w ).

[0021] As a further scheme of the application, the dynamic window signal is multiplied by the original acoustic feature signal to generate the local acoustic feature for detecting the language information of the current time step, wherein the local acoustic feature is:

[0022]

[0023] In the formula, X is the original acoustic feature input, and X' is the acoustic feature sequence selected in real time as the feature input for language recognition of the current time step.

[0024] As a further scheme of the application, the local acoustic feature of any length is normalized to a fixed dimension feature, and the normalization method includes two ways of long-time information statistics and recurrent neural network; wherein the normalization method of long-time information statistics includes:

[0025] Calculate the mean and variance statistics of the frame sequence selected in real time;

[0026] The mean and variance statistics are combined with the frame sequence selected in real time as a fixed dimension feature.

[0027] As a further scheme of the application, the calculation formula of the language feature vector composed of long-time information statistics is as follows:

[0028] v t =f(Append(μ t ,σ t ))

[0029]

[0030] where X' i represents the i-th frame acoustic feature in the real-time selected frame sequence, T represents the length of the real-time selected frame sequence, μ t and σ t represent the mean and variance statistics respectively. Finally, X" t =[μ t ,σ t ] is combined as the language classifier input feature of the t-th time step; v t represents the language feature vector of the long-time information statistics group; T represents the length of the real-time selected frame sequence; t+T represents the cutoff time step position of the T frames from the t-th time step.

[0031] As a further scheme of the present application, the regularization method of the recurrent neural network comprises:

[0032] generating the fixed-dimension feature based on the recurrent neural network;

[0033] taking the output of the recurrent neural network as the fixed-dimension feature;

[0034] wherein the calculation formula of the fixed-dimension feature is as follows

[0035] s t =f(UX′ t +Ws t-1 )

[0036] X" t =softmax(Vs t )

[0037] wherein U, W and V are input mapping matrix, hidden layer mapping matrix and output mapping matrix respectively. Finally, X" t is taken as the language classifier input feature of the t-th time step; s t represents the hidden layer state of the t-th time step recurrent neural network; X' t represents the input feature of the t-th time step; s t-1 represents the hidden layer state of the t-1-th time step recurrent neural network, which is used to participate in the calculation of the current time step hidden layer state; Vs t represents the hidden layer state s t after linear transformation by the output mapping matrix V, which is the intermediate representation before output.

[0038] As a further scheme of the present application, when the fixed-dimension feature is input to the deep neural network classifier and the probability value of the language corresponding to the current time step speech feature is output, the probability value that the current time step speech feature is determined to be the corresponding language is determined according to the fixed-dimension language feature; wherein the deep neural network is used as the language classifier, the output of the neural network language classifier has two output endpoints, and the output results respectively represent the probability value that the current time step speech feature is determined to be the corresponding language.

[0039] As a further scheme of the present application, the deep neural network classifier is a multi-level structure, comprising:

[0040] an input layer for receiving the fixed-dimension feature;

[0041] a hidden layer for extracting high-level language features by using a feedforward neural network or a bidirectional LSTM;

[0042] an output layer for outputting the probability value of the language corresponding to the current time step speech feature by using Softmax.

[0043] As a further scheme of the present application, the language classification process of the neural network language classifier is as follows:

[0044] h' t =W i X” t

[0045] h t =f(h′ t )

[0046]

[0047] wherein X" t is the fixed-dimension language feature generated by a long-time information statistical method or a recurrent neural network method, and is the input feature of the language classifier at the tth time step, W i is the input layer mapping matrix, W o is the output layer mapping matrix, p(l t |X" t ) is the probability that the corresponding language is determined, p(l t |X" t ) is normalized by using spft max, so that the sum of the probabilities of two languages is 1; wherein, represents the probability that the corresponding language is determined; represents the probability that the corresponding language is determined; f' t represents the intermediate variable after the input layer transformation; h tis the output vector of the last hidden layer, is the highest level feature of language information, and can be transmitted to the decoder as a language information vector to assist in bilingual mixed speech content recognition; f() is a hidden layer function, which is different according to the type of neural network of the hidden layer, and a feedforward neural network or a recurrent neural network type is adopted.

[0048] As a further scheme of the present application, the language information dynamic detection method further comprises an adjusting step of the window length limit parameter:

[0049] Obtain a mixed data set and corresponding labeled text and language labels;

[0050] According to the window length limit parameter, a dynamic window is generated and the recognition accuracy is calculated;

[0051] According to the recognition accuracy, the optimization interval of the window length limit parameter is determined.

[0052] As a further scheme of the present application, the adjusting step of the window length limit parameter comprises:

[0053] In real-time time domain selection, a dynamic window length limit parameter is adopted;

[0054] According to the real-time confidence feedback, the window length limit parameter is dynamically adjusted.

[0055] Compared with the prior art, the language information dynamic detection method based on a deep neural network has the following beneficial effects:

[0056] 1. Precise language switching detection is realized: the present application can accurately locate the language switching point in a bilingual mixed speech stream by introducing an attention mechanism and dynamic window selection, jointly uses multiple frames of speech as the language classification input of the current time step, outputs the language discrimination of the current time step, and can better predict the language of the current speech stream according to historical speech information. The present application can dynamically adjust the feature window, focus on the key time step of language switching, and effectively improve the detection accuracy of language switching.

[0057] 2. In order to make the window have time concentration, a limited time span is added, that is, when calculating the current window w i Position information, a window length limit parameter L w is added to ensure that the window shape does not present a scattered state; through training, the window length limit parameter L w is set.

[0058] 3、The output of the neural network language classifier adopted has two output endpoints, and the output results respectively represent the probability value of the speech feature at the current time step being determined as the corresponding language. The application is not only suitable for bilingual mixed speech streams, but also can be extended and applied in multi-language speech streams. By adjusting the output layer of the classifier, the method can support the recognition of more languages, adapt to the switching between different languages, has high flexibility and scalability, and can be widely applied in the fields of speech recognition, speech translation, voice assistants and the like.

[0059] These aspects or other aspects of the present application will be more apparent in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the following will briefly introduce the drawings needed to be used in the exemplary embodiments or the related art description. The drawings are used to provide further understanding of the application, and constitute a part of the specification, and are used to explain the application together with the embodiments of the application, and do not constitute a limitation of the application. In the drawings:

[0061] Figure 1 A flow chart of a language information dynamic detection method based on a deep neural network according to an embodiment of the application.

[0062] Figure 2 A working principle diagram of long-time information statistics in a language information dynamic detection method based on a deep neural network according to an embodiment of the application.

[0063] Figure 3 A working principle diagram of generating fixed-dimension features based on a recurrent neural network in a language information dynamic detection method based on a deep neural network according to an embodiment of the application.

[0064] Figure 4 A neural network language classifier diagram in a language information dynamic detection method based on a deep neural network according to an embodiment of the application. DETAILED DESCRIPTION

[0065] In the following, the application will be further described in combination with the drawings and the specific embodiments. It should be noted that, under the premise of not conflicting, the following described embodiments or technical features can be combined to form new embodiments.

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the embodiments of the present application with reference to the accompanying drawings and in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0067] It should be noted that all the expressions of "first" and "second" in the embodiments of the present application are used to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of description and should not be understood as a limitation of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, other steps or units inherent to the process, method, system, product or device that includes a series of steps or units.

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] The flowchart shown in the accompanying drawings is only an example and does not necessarily include all the contents and operations / steps, nor does it necessarily execute in the order described. For example, some operations / steps can be further divided, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0070] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0071] In order to solve the problem that the traditional language classification method cannot effectively handle frequent and short language switching, the present application proposes a language information dynamic detection method based on deep neural network, which is particularly suitable for language switching recognition in bilingual mixed speech stream. By combining attention mechanism, multi-frame speech information, dynamic window selection, long-time information statistics and recurrent neural network, the detection accuracy of language switching points in bilingual mixed speech stream and the accuracy of language recognition are improved, and the problem that the traditional language classification method cannot effectively handle frequent and short language switching is solved.

[0072] Referring to Figure 1 The embodiments of the present application provide a language information dynamic detection method based on deep neural network, which comprises the following steps:

[0073] Step S10, acquiring a mixed language speech stream to be detected and extracting original acoustic features.

[0074] Step S20: Generate the attention vector for the current time step based on the attention mechanism, and dynamically select the speech feature frame sequence window for the current time step by combining unidirectional time selection and the limitation of a specific time span.

[0075] In this step, the attention vector for the current time step is generated based on the attention mechanism as follows:

[0076] a i =[a i,1 ,...,a i,j ,...,a i,T ]

[0077] Based on attention vector a i Generate a window of speech feature frame sequence selected at the current time step. i ,in:

[0078] w i =f(a i )

[0079] Introducing a one-way time selection constraint, the time point position of the speech feature frame sequence window conforms to the monotonically increasing property, when calculating the current window w i When providing location information, add the window from the previous time step. i-1 Location information, enabling w i Later than w i-1 Appear:

[0080] w i =f(a i ,w i-1 )

[0081] And add a finite time span constraint when calculating the current window w i When providing location information, add the window length limit parameter L. w This prevents the window shape from appearing fragmented.

[0082] w i =f(a i ,w i-1 ,L w ).

[0083] Step S30: Multiply the dynamic window signal with the original acoustic feature signal to generate local acoustic features of the language information detected at the current time step.

[0084] In this step, the dynamic window signal is multiplied by the original acoustic feature signal to generate local acoustic features for the language information detected at the current time step. These local acoustic features are:

[0085]

[0086] In the formula, X is the original acoustic feature input, and X' is the acoustic feature sequence selected in real time as the feature input for language recognition at the current time step.

[0087] In step S40, the local acoustic feature of an arbitrary length is regularized into a fixed-dimension feature, so as to take the fixed-dimension feature as the input of neural network language classification.

[0088] In this step, the local acoustic feature of an arbitrary length is regularized into a fixed-dimension feature, and the regularization method includes long-time information statistics and a recurrent neural network. Referring to FIG. 4, the regularization method of long-time information statistics includes: Figure 2

[0089] The mean and variance statistics of the frame sequence selected in real time are calculated.

[0090] The mean and variance statistics are combined with the frame sequence selected in real time as the fixed-dimension feature.

[0091] The calculation formula of the language feature vector constructed by long-time information statistics is as follows:

[0092] v t =f(Append(μ t ,σ t ))

[0093]

[0094]

[0095] In the formula, X' represents the i-th frame acoustic feature in the frame sequence selected in real time, the length of the frame sequence selected in real time is T, μ i and σ t represent the mean and variance statistics respectively. Finally, X" t =[μ t ,σ t ] are combined as the language classifier input feature at the t-th time step; v t represents the language feature vector constructed by long-time information statistics; T represents the length of the frame sequence selected in real time; t+T represents the cutoff time step position of T frames from the t-th time step. t

[0096] In this embodiment, referring to FIG. 5, the regularization method of the recurrent neural network includes: Figure 3

[0097] The fixed-dimension feature is generated based on the recurrent neural network.

[0098] The output of the recurrent neural network is taken as the fixed-dimension feature.

[0099] ​​​The calculation formula of the fixed dimension feature is as follows

[0100] s t = f(UX' t +Ws t-1 )

[0101] X″ t = softmax(Vs t )

[0102] Wherein, U, W and V are input mapping matrix, hidden layer mapping matrix and output mapping matrix respectively. Finally, X" t is taken as the language classifier input feature of the tthtime step; s t represents the hidden layer state of the tthtime step recurrent neural network; X' t represents the input feature of the tthtime step; s t-1 represents the hidden layer state of the t-1thtime step recurrent neural network, which is used to participate in the calculation of the current time step hidden layer state; Vs t represents the hidden layer state s t After linear transformation by the output mapping matrix V, the result is the intermediate representation before output.

[0103] Step S50, input the fixed dimension feature to the deep neural network classifier, and output the language probability value corresponding to the current time step speech feature.

[0104] In this step, when the fixed dimension feature is input to the deep neural network classifier and the language probability value corresponding to the current time step speech feature is output, the probability value that the current time step speech feature is determined to be the corresponding language is determined according to the fixed dimension language feature; wherein the deep neural network is used as the language classifier, and the output of the neural network language classifier has two output endpoints, and the output results respectively represent the probability value that the current time step speech feature is determined to be the corresponding language.

[0105] Wherein, referring to Figure 4 , the deep neural network classifier is a multi-level structure, including:

[0106] The input layer is used to receive the fixed dimension feature;

[0107] The hidden layer uses a feedforward neural network or a bidirectional LSTM to extract high-level language features;

[0108] The output layer outputs the probability value of the language corresponding to the current time step speech feature through Softmax.

[0109] In this embodiment, the language classification process of the neural network language classifier is as follows:

[0110] h't = W i X" t

[0111] h t = f(h' t )

[0112]

[0113] wherein X" t is a fixed dimension language feature generated by a long-time information statistical method or a recurrent neural network method, as an input feature of a language classifier at the tth time step, W i is an input layer mapping matrix, W o is an output layer mapping matrix, p(l t | X" t ) is a probability of being judged as a corresponding language, p(l t | X" t ) is normalized by a softmax, ensuring that the sum of the probabilities of two languages is 1; wherein, represents the probability of being judged as a corresponding language ; represents the probability of being judged as a corresponding language ; h' t represents an intermediate variable after input layer transformation; h t is an output vector of the last hidden layer, which is the highest level feature of language information, and can be transmitted to the decoder as a language information vector to assist in bilingual mixed speech content recognition; f() is a hidden layer function, which is different according to the type of neural network of the hidden layer, and adopts a feedforward neural network or a recurrent neural network type.

[0114] Step S60, based on the dynamic window information and the language probability value, outputting the start and end time of each speech segment in the mixed speech stream and the corresponding language label, and determining the time point of language switching.

[0115] In some embodiments, the language information dynamic detection method further comprises an adjusting step of a window length limit parameter:

[0116] obtaining a mixed data set and corresponding labeled text and language labels;

[0117] generating a dynamic window according to the window length limit parameter and calculating the recognition accuracy;

[0118] determining the optimization interval of the window length limit parameter according to the recognition accuracy.

[0119] In this embodiment, the adjusting step of the window length limit parameter comprises:

[0120] In real-time time domain selection, a dynamic window length limit parameter is adopted;

[0121] The window length limit parameter is dynamically adjusted according to real-time confidence feedback.

[0122] Specifically, the language information dynamic detection workflow includes the following steps:

[0123] A dynamic window is selected, the window signal is multiplied by the original feature signal, and a feature input for language recognition at a current time step is obtained;

[0124] The feature input of any length is regularized into a fixed-dimension language feature;

[0125] A deep neural network is used as a language classifier to perform speech recognition on the fixed-dimension language feature to obtain a probability of recognized language.

[0126] Determine an appropriate window length limit parameter L w Interval:

[0127] Obtain a mixed data set and labeled text and language labels of speech data in the mixed data set, wherein the mixed data set includes first language speech data and second language speech data;

[0128] Obtain an original feature signal of the mixed data set;

[0129] According to the window length limit parameter L w , a dynamic window is selected, and the window signal is multiplied by the original feature signal to obtain a feature input for language recognition at a current time step;

[0130] The feature input of any length is regularized into a fixed-dimension language feature;

[0131] A deep neural network is used as a language classifier to perform speech recognition on the fixed-dimension language feature to obtain a probability of recognized language, and to obtain a recognition result corresponding to the window length limit parameter L w ;

[0132] The recognition result is compared with the mixed data set to obtain a recognition accuracy corresponding to the window length limit parameter L w ;

[0133] According to the recognition accuracy corresponding to the window length limit parameter L w , the window length limit parameter L w interval is determined.

[0134] Further, the window length limit parameter L w is adjusted, and the window length limit parameter L wThe window can ensure time concentration, but can also reduce the language recognition accuracy, so it is necessary to determine the appropriate window length limit parameter L w Interval.

[0135] Obtain a mixed data set and the labeled text and language label of the speech data in the mixed data set, wherein the mixed data set includes first language speech data and second language speech data;

[0136] Obtain the original feature signal of the mixed data set;

[0137] According to the window length limit parameter L w , select a dynamic window, multiply the window signal and the original feature signal to obtain the feature input of the current time step language recognition;

[0138] Regularize the feature input of any length into a fixed dimension language feature;

[0139] Use a deep neural network as a language classifier to perform speech recognition on the fixed dimension language feature to obtain the probability of the recognized language, and obtain the recognition result corresponding to the window length limit parameter L w ;

[0140] Compare the recognition result with the mixed data set to obtain the recognition accuracy corresponding to the window length limit parameter L w ;

[0141] According to the recognition accuracy corresponding to the window length limit parameter L w , determine the window length limit parameter L w interval.

[0142] In real-time time domain selection, within the window length limit parameter L w interval, a dynamic length limit parameter L w is adopted.

[0143] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0144] It should be understood that, although the steps are described in a certain order, the steps are not necessarily executed in the order described above. Unless otherwise explicitly stated herein, the execution of the steps has no strict order limitation, and the steps can be executed in other orders. Moreover, some steps of the present embodiment can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0145] The above is the exemplary embodiment disclosed by the present application, but it should be noted that various changes and modifications can be made without departing from the scope of the embodiments disclosed by the present application defined by the claims. The functions, steps and / or acts of the method claims described herein need not be performed in any particular order. Furthermore, although the elements of the embodiments disclosed by the present application can be described or claimed in individual form or in a singular form, they can also be understood as plural unless explicitly limited as singular.

[0146] It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0147] Those skilled in the art will understand that the above discussion of any embodiment is merely exemplary and is not intended to suggest that the scope of the embodiments disclosed by the present application (including the claims) is limited to these examples; under the concept of the embodiments of the present application, the technical features of the above embodiments or different embodiments can also be combined, and there are many other variations of the above aspects of the embodiments of the present application. In order to be brief, they are not provided in detail. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the scope of protection of the embodiments of the present application.

Claims

1.A method for detecting language information dynamically based on a deep neural network, characterized in that, The method comprises the following steps: Obtain the mixed language speech stream to be detected and extract the original acoustic features; Generate an attention vector for the current time step based on an attention mechanism, and dynamically select a speech feature frame sequence window for the current time step by combining one-way time selection and a specific time span limit; Multiply the dynamic window signal with the original acoustic feature signal to generate local acoustic features for detecting language information at the current time step; Regularize the local acoustic features of arbitrary length into fixed-dimension features, and use the fixed-dimension features as inputs of a neural network language classification; Input the fixed-dimension features into a deep neural network classifier to output a language probability value corresponding to the speech features at the current time step; Based on the dynamic window information and the language probability value, output the start and end times of each speech segment in the mixed speech stream and the corresponding language label, and determine the time point of language switching. 2.The deep neural network-based language information dynamic detection method of claim 1, wherein, Generating an attention vector for the current time step based on an attention mechanism further comprises: Generating a speech feature frame sequence window for the current time step according to the attention vector; Introducing a one-way time selection limit, the time point position of the speech feature frame sequence window meets the monotonic increasing property, and when calculating the current window position information, the window position information of the previous time step is added, so that the current window position information appears later than the window position information of the previous time step; And adding a limit of a limited time span, when calculating the current window position information, a window length limit parameter is added, so that the window shape does not present a scattered state. 3.The deep neural network-based language information dynamic detection method of claim 2, wherein, Multiplying the dynamic window signal with the original acoustic feature signal to generate local acoustic features for detecting language information at the current time step, wherein the local acoustic features are: In the formula, X is the original acoustic feature input, X' is the acoustic feature sequence selected in real time as the feature input for language recognition at the current time step, w i is the current time step voice feature frame sequence window. 4.The method of claim 1-3, wherein, Regularizing the local acoustic features of arbitrary length into fixed-dimension features, and the regularization method includes two ways of long-time information statistics and recurrent neural network; wherein the regularization method of long-time information statistics includes: Calculating the mean and variance statistics of the real-time selected frame sequence; Combining the mean and variance statistics with the real-time selected frame sequence as fixed-dimension features. 5.The deep neural network-based language information dynamic detection method according to claim 4, wherein, The calculation formula of the language feature vector formed by long-time information statistics is as follows: v i = f(Append(μ t , σ t )) where X i represents the i-th frame acoustic feature in the real-time selected frame sequence with length T, μ t and σ t represent the mean and variance statistics, respectively; finally, X t t , σ t ] are jointly taken as the language classifier input features at the t-th time step; v t represents the language feature vector constructed from the long-time information statistics; T represents the length of the real-time selected frame sequence; t+T represents the cutoff time step position T frames after the t-th time step.​ 6.The deep neural network-based language information dynamic detection method according to claim 4, wherein, The regularization method of recurrent neural network includes: Generating fixed-dimension features based on a recurrent neural network; Taking the output of the recurrent neural network as the fixed-dimension features; Wherein, the calculation formula of the fixed-dimension features is as follows s t = f(UX' + Ws t + Ws t-1 ) X" t = soft max(Vs t ) where U, W and V are input mapping matrix, hidden layer mapping matrix and output mapping matrix respectively; finally, X t language classifier input feature at the t-th time step; s t represents the hidden layer state of the recurrent neural network at the t-th time step; X' t represents the input feature at the t-th time step; s t-1 represents the hidden layer state of the recurrent neural network at the t-1-th time step, which is used to participate in the calculation of the current time step hidden layer state; Vs t represents the hidden layer state s t the result after linear transformation by the output mapping matrix V, which is the intermediate representation before output. 7.The deep neural network-based language information dynamic detection method of claim 1, wherein, When inputting the fixed-dimension features into a deep neural network classifier to output a language probability value corresponding to the speech features at the current time step, the probability value that the speech features at the current time step are judged as the corresponding language is determined according to the fixed-dimension language features; wherein a deep neural network is used as a language classifier, and the output of the neural network language classifier has two output endpoints, and the output results respectively represent the probability value that the speech features at the current time step are judged as the corresponding language. 8.The deep neural network-based language information dynamic detection method of claim 7, wherein, The deep neural network classifier is a multi-level structure, which comprises: An input layer for receiving fixed-dimension features; A hidden layer using a feedforward neural network or a bidirectional LSTM to extract high-level language features; An output layer outputting a probability value of the corresponding language of the speech features at the current time step through Softmax. 9.The deep neural network-based language information dynamic detection method of claim 1, wherein, The language information dynamic detection method further comprises an adjustment step of a window length limit parameter: Obtain a mixed data set and corresponding labeled text and language labels; According to the window length limit parameter, a dynamic window is generated and a recognition accuracy is calculated; According to the recognition accuracy, an optimization interval of the window length limit parameter is determined. 10.The deep neural network-based language information dynamic detection method of claim 9, wherein, The adjusting step of the window length limit parameter comprises: In real-time time domain selection, a dynamic window length limit parameter is adopted; According to real-time confidence, the window length limit parameter is dynamically adjusted.

Images