A language recognition method and device

Abstract

The present invention discloses a language recognition method and device, wherein the above-mentioned language recognition method comprises the following steps: S1) extracting fbank features and mfcc features from a speech signal to be recognized frame by frame; S2) judging whether each frame belongs to silence or speech; S3) splicing the fbank features of the frames belonging to speech in time sequence; S4) taking a fixed length of frame number T, inputting the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features; S5) passing the T×C features through a first fully connected layer, a pooling layer, a second fully connected layer and a third fully connected layer to obtain the probability of each language. The present invention can automatically recognize different languages, thereby helping a foreign language automatic scoring system to handle the situation where a candidate can still get high scores when answering in Chinese, thereby improving the accuracy of the subsequent scoring system.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and in particular relates to a language recognition method and device. Background Art

[0002] As oral ability is increasingly valued, oral foreign language tests are being introduced in exams in all regions across the country. Computer-assisted scoring technology is gaining more and more attention because it can alleviate the shortage of teachers and provide flexible time arrangements.

[0003] Although computer automatic scoring technology is becoming more and more mature, there are still some technical problems. This is because the computer-assisted scoring system will take into account the relevant phonetic characteristics of the candidates, but some candidates will use Chinese to answer questions. At this time, the computer-assisted scoring system will still give a certain score based on the pronunciation information of the test, and the score is uncontrollable, which ultimately leads to the accuracy of the scoring and brings many inconveniences to actual use. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a language recognition method and device that can automatically identify language categories, thereby avoiding the problem that candidates still get high scores when answering questions in Chinese when using a computer-assisted scoring system, and improving the accuracy of scoring.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a language recognition method, comprising the following steps:

[0006] S1) extracting fbank features and mfcc features respectively according to the frame of the speech signal to be recognized;

[0007] S2) determining whether each frame is silence or speech;

[0008] S3) concatenating the fbank features of those frames belonging to speech in time order;

[0009] S4) Take a fixed length of frames T, input the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features;

[0010] S5) Pass the features of T×C through the first fully connected layer, the pooling layer, the second fully connected layer, and the third fully connected layer to obtain the probability of each language.

[0011] Furthermore, 64-dimensional fbank features and 40-dimensional mfcc features are extracted from the speech signal to be recognized frame by frame, with a window length of 25ms and a frame shift of 10ms.

[0012] Furthermore, in step S2, the mfcc features of each frame are input into a trained domain-adaptive speech activity detection module to determine whether each frame belongs to silence or speech.

[0013] Furthermore, in step S3, each frame needs to be normalized: the mean of the fbank features of all frames within 3 seconds before the current frame is calculated, and then the mean is subtracted from the fbank features of the current frame to obtain the normalized result of the current frame.

[0014] Furthermore, the TDNN block includes a TDNN layer, a RELU layer and a BatchNorm layer; the number of the TDNN blocks is 3.

[0015] Further, step S5 comprises the following steps:

[0016] S50) inputs the features of T×C into the first fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs the abstract features of T×C;

[0017] S51) inputs the T×C abstract features into the pooling layer and outputs 1×(2×C) pooling features;

[0018] S52) The 1×(2×C) pooled features are input into the second fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs 1*H language features;

[0019] S53) The 1*H language features are input into the third fully connected layer, which uses softmax as the activation function and outputs the probability of each language.

[0020] Furthermore, the pooling layer calculates the mean and variance of T frames, and concatenates the mean and standard deviation as the pooling feature. The input feature is denoted as F, where F(i, j) represents the j-th dimension feature of the i-th frame, a(i, j) represents the importance weight of the j-th dimension feature of the i-th frame, and M = N × C, i∈ [1, T], j∈ [1, M];

[0021] The output mean m' is 1×M dimensional, where

[0022]

[0023] The output variance v' is 1×M dimensions; where

[0024]

[0025] Furthermore, the importance weight of the pooling layer is calculated as follows: the feature F is input into a fully connected layer and a softmax layer, and the importance weight a(i, j) of the j-dimensional feature of the i-th frame is output, i∈[1,T], j∈[1,M].

[0026] Furthermore, after step S4, the output features T×C of the N TDNN blocks are concatenated to obtain T×(N×C) features.

[0027] A language recognition device, comprising:

[0028] An extraction module is used to extract fbank features and mfcc features from the speech signal to be recognized frame by frame;

[0029] A judgment module is used to judge whether each frame belongs to silence or speech;

[0030] A sequential concatenation module is used to concatenate the fbank features of those frames belonging to speech in time order;

[0031] The output module is used to take a fixed length of frames T, input the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features;

[0032] The language analysis module is used to obtain the probability of each language by passing the T×C features through the first fully connected layer, the pooling layer, the second fully connected layer, and the third fully connected layer.

[0033] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0034] The language identification method and device of the solution of the present invention first splices the fbank features of those frames belonging to speech in chronological order, and then inputs the fbank features of T frames into N TDNN blocks for calculation. Each TDNN block outputs T×C features, and then the T×C features are passed through the first fully connected layer, the pooling layer, the second fully connected layer and the third fully connected layer to obtain the probability of each language, thereby providing strong assistance to the subsequent automatic scoring system, improving the accuracy of the scoring system, and meeting actual usage needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0036] Attached Figure 1 Schematic diagram of the flow of the language identification method in an embodiment of the present invention;

[0037] Attached Figure 2 FIG. 4 is a schematic diagram of a flow chart of step S5 in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0039] Please refer to the attached Figure 1 A language recognition method described in an embodiment of the present invention comprises the following steps: S1) extracting fbank features and mfcc features from the speech signal to be recognized frame by frame; S2) judging whether each frame belongs to silence or speech; S3) concatenating the fbank features of the frames belonging to speech in chronological order; S4) taking a fixed length of frames T, inputting the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features; S5) passing the T×C features through the first fully connected layer, the pooling layer, the second fully connected layer and the third fully connected layer to obtain the probability of each language.

[0040] Specifically, in step S1, firstly, a speech signal to be recognized is input, and then 64-dimensional fbank features and 40-dimensional mfcc features are extracted from the input speech signal to be recognized frame by frame, with a window length of 25ms and a frame shift of 10m;

[0041] In step S2, the mfcc features of each frame are input into a trained domain-adaptive speech activity detection module to determine whether each frame belongs to silence or speech. The speech activity detection module is a common speech detection module.

[0042] In step S3, each frame needs to be normalized: that is, the mean of the fbank features of all frames within 3s before the current frame is calculated, and then the mean is subtracted from the fbank features of the current frame to obtain the normalized result of the current frame.

[0043] In step S4, a fixed length of frames T is taken, that is, a frame number of 1 s is taken.

[0044] In addition, the TDNN block includes a TDNN layer, a RELU layer and a BatchNorm layer; the number of the TDNN blocks is 3.

[0045] In this embodiment, a fixed length of frames T is taken, and the fbank features of T frames are input into N TDNN blocks for calculation. Each TDNN block outputs T×C features. The process of step S5 is as follows:

[0046] S50) inputs the features of T×C into the first fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs the abstract features of T×C;

[0047] S51) Input T×C abstract features into the pooling layer, output 1×(2×C) pooling features, the pooling layer calculates the mean and variance of T frames, and concatenates the mean and standard deviation as the pooling feature, and denotes the input feature as F, where F(i, j) represents the j-th dimension feature of the i-th frame, a(i, j) represents the importance weight of the j-th dimension feature of the i-th frame, and M=N×C, i∈[1,T], j∈[1,M]; the output mean m' is 1×M dimensional, where The output variance v' is 1×M dimensions;

[0048] in,

[0049] The importance weight of the pooling layer is calculated as follows: input the feature F into a fully connected layer and a softmax layer, and output the importance weight a(i, j) of the j-dimensional feature of the i-th frame, i∈[1,T], j∈[1,M].

[0050] S52) The 1×(2×C) pooled features are input into the second fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs 1*H language features;

[0051] S53) The 1*H language features are input into the third fully connected layer, which uses softmax as the activation function and outputs the probability of each language.

[0052] In addition, considering that shallow features still have a certain contribution to the subsequent language feature extraction, the output features T×C of N TDNN blocks are concatenated. In this way, after step S4, the output features T×C of N TDNN blocks are concatenated to obtain T×(N×C) features.

[0053] The process of step S5 becomes:

[0054] S500: Input the T×(N×C) features into a fully connected layer, ReLU layer, and BatchNorm layer, and output T×(N×C) abstract features;

[0055] S501: Input T×(N×C) abstract features into the pooling layer and output 1×(2×N×C) pooling features;

[0056] S502: Input the 1×(2×N×C) pooled features into a fully connected layer, a ReLU layer, and a BatchNorm layer, and output a 1*H language feature;

[0057] S503: Input the 1*H language features into a fully connected layer and a softmax layer, and output the probability of each language.

[0058] See attached Figure 2 , is a specific flow chart of step 5 in one embodiment of the present invention, wherein the number of TDNN blocks is 3 and the number of languages ​​is 2; the specific steps are as follows:

[0059] S510: Input the T×(3×C) features into a fully connected layer, a ReLU layer, and a BatchNorm layer, and output T×(3×C) abstract features;

[0060] S511: Input T×(3×C) abstract features into the pooling layer and output 1×(2×3×C) pooling features;

[0061] S512: Input the 1×(2×3×C) pooled features into a fully connected layer, a ReLU layer, and a BatchNorm layer, and output a 1*H language feature;

[0062] S513: Input the 1*H language features into a fully connected layer and a softmax layer, and output the probability of each language.

[0063] The present invention also discloses a language recognition device, characterized in that it includes an extraction module, a judgment module, a sequential splicing module, an output module and a language analysis module; the extraction module is used to extract fbank features and mfcc features of the speech signal to be recognized frame by frame; the judgment module is used to judge whether each frame belongs to silence or speech; the sequential splicing module is used to splice the fbank features of those frames belonging to speech in chronological order; the output module is used to take a fixed length of frame number T, input the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features; the language analysis module is used to obtain the probability of each language after passing the T×C features through the first fully connected layer, the pooling layer, the second fully connected layer and the third fully connected layer.

[0064] The language identification method and device of the present invention first splices the fbank features of those frames belonging to speech in chronological order, and then inputs the fbank features of T frames into N TDNN blocks for calculation. Each TDNN block outputs T×C features, and then the T×C features are passed through the first fully connected layer, the pooling layer, the second fully connected layer and the third fully connected layer to obtain the probability of each language, thereby providing powerful assistance to the subsequent automatic scoring system, improving the accuracy of the scoring system, and meeting actual usage needs.

[0065] The above are only specific application examples of the present invention and do not constitute any limitation on the protection scope of the present invention. Any technical solution formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention.

Claims

1. A language recognition method, characterized in that: The steps include: S1) extracting fbank features and mfcc features respectively according to the frame of the speech signal to be recognized; S2) determining whether each frame is silence or speech; S3) concatenating the fbank features of those frames belonging to speech in time order; S4) Take a fixed length of frames T, input the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features; S5) Passing the features of T×C through the first fully connected layer, the pooling layer, the second fully connected layer, and the third fully connected layer to obtain the probability of each language; Wherein, step S5 specifically comprises the following steps: S50) inputs the features of T×C into the first fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs the abstract features of T×C; S51) inputs the T×C abstract features into the pooling layer and outputs 1×(2×C) pooling features; S52) Input the 1×(2×C) pooled features into the second fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs 1*H language features; S53) The 1*H language feature is input into the third fully connected layer, which uses softmax as the activation function and outputs the probability of each language.

2. The language identification method according to claim 1, wherein: For the speech signal to be recognized, 64-dimensional fbank features and 40-dimensional mfcc features are extracted frame by frame, with a window length of 25ms and a frame shift of 10ms.

3. The language identification method according to claim 1, wherein: In step S2, the mfcc features of each frame are input into the trained domain-adaptive speech activity detection module to determine whether each frame belongs to silence or speech.

4. The language identification method according to claim 1, wherein: In step S3, each frame needs to be normalized: the mean of the fbank features of all frames within 3s before the current frame is calculated, and then the mean is subtracted from the fbank features of the current frame to obtain the normalized result of the current frame.

5. The language identification method according to claim 1, wherein: The TDNN block includes a TDNN layer, a RELU layer and a BatchNorm layer; the number of the TDNN blocks is 3.

6. The language identification method according to claim 1, wherein: The pooling layer calculates the mean and variance of T frames, and concatenates the mean and standard deviation as the pooling feature. The input feature is denoted as F, where F(i, j) represents the j-th dimension feature of the i-th frame, a(i, j) represents the importance weight of the j-th dimension feature of the i-th frame, and M=N×C. ; The output mean m' is 1×M dimensional, where ; The output variance v' is 1×M dimensions; where .

7. The language identification method according to claim 6, characterized in that: The importance weight of the pooling layer is calculated as follows: the feature F is input into a fully connected layer and a softmax layer, and the importance weight a(i, j) of the j-th dimension feature of the i-th frame is output, i∈[1,T], j∈[1,M].

8. The language recognition method according to claim 1, characterized in that: After step S4, the output features T×C of the N TDNN blocks are concatenated to obtain T×(N×C) features.

9. A language recognition device, characterized in that: include: An extraction module is used to extract fbank features and mfcc features from the speech signal to be recognized frame by frame; A judgment module is used to judge whether each frame belongs to silence or speech; A sequential concatenation module is used to concatenate the fbank features of those frames belonging to speech in time order; The output module is used to take a fixed length of frames T, input the fbank features of T frames into N TDNN blocks for calculation, and each TDNN block outputs T×C features; The language analysis module is used to obtain the probability of each language by passing the features of T×C through the first fully connected layer, the pooling layer, the second fully connected layer, and the third fully connected layer. It specifically includes the following steps: The features of T×C are input to the first fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function to output the abstract features of T×C; Input the T×C abstract features into the pooling layer and output 1×(2×C) pooling features; The 1×(2×C) pooled features are input to the second fully connected layer, which uses ReLU as the activation function and BatchNorm as the normalization function, and outputs 1*H language features; The 1*H language features are input into the third fully connected layer, which uses softmax as the activation function and outputs the probability of each language.

Images