Speech recognition method and related device, electronic device, and storage medium

Abstract

The present application discloses a speech recognition method and related devices, electronic devices, and storage media, wherein the speech recognition method includes: obtaining a spectrogram of the speech to be recognized; extracting a re-recognized spectrum segment from the spectrogram based on a first recognition result of language recognition performed window by window on the spectrogram by a first sliding window; analyzing and obtaining the target language involved in the re-recognized spectrum segment based on a second recognition result of language recognition performed window by window on the re-recognized spectrum segment by a plurality of second sliding windows; wherein the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window. The above scheme can improve the accuracy of recognizing the language involved in the speech.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a speech recognition method and related devices, electronic equipment, and storage media. Background Art

[0002] As globalization becomes increasingly international, how to communicate without barriers across the country and even the world has become an issue that cannot be ignored. Faced with such a huge language system, the number of languages ​​that an individual can master is very limited. Therefore, the importance and status of automatic language recognition technology has become increasingly important.

[0003] However, language information is weak information in speech information. Although many language recognition technologies have been proposed, such as Total Variability (TV) factor analysis technology and end-to-end language recognition framework, it is still difficult to guarantee the accuracy of language recognition. In view of this, how to improve the accuracy of language recognition has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech recognition method and related devices, electronic devices, and storage media, which can improve the accuracy of recognizing the languages ​​involved in the speech.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech recognition method, including: obtaining a spectrogram of a speech to be recognized; based on a first recognition result of performing language recognition window by window on the spectrogram by a first sliding window, extracting a re-recognized spectrogram segment from the spectrogram; based on a second recognition result of performing language recognition window by window on the re-recognized spectrogram segment by a plurality of second sliding windows, respectively, analyzing and obtaining a target language involved in the re-recognized spectrogram segment; wherein the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech recognition device, including: a spectrum acquisition module, a first recognition module and a second recognition module, the spectrum acquisition module is used to obtain a spectrum graph of a speech to be recognized; the first recognition module is used to extract a re-recognized spectrum segment from the spectrum graph based on a first recognition result of language recognition performed window by window on the spectrum graph by a first sliding window; the second recognition module is used to analyze and obtain a target language involved in the re-recognized spectrum segment based on a second recognition result of language recognition performed window by window on the re-recognized spectrum segment by a plurality of second sliding windows; wherein the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the speech recognition method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the speech recognition method in the above first aspect.

[0009] The above scheme obtains the spectrogram of the speech to be recognized, and obtains the first recognition result of language recognition based on the first sliding window in the spectrogram window by window, extracts the re-recognized spectrogram segment from the spectrogram, and on this basis, performs the second recognition result of language recognition based on a plurality of second sliding windows in the re-recognized spectrogram segment window by window, analyzes and obtains the target language involved in the re-recognized spectrogram segment, and the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window. Therefore, on the one hand, in the whole recognition process, at most two language recognitions can be performed, namely, the first language recognition based on the first sliding window and the second language recognition based on the second sliding window. Therefore, the double recognition can be beneficial to improve the recognition accuracy. On the other hand, in the second recognition process, a plurality of second sliding windows of different sizes are used, which can fuse the multi-scale recognition results, which is beneficial to improve the recognition accuracy again. On the other hand, the size of the second sliding window used in the second recognition process is smaller than the size of the first sliding window, that is, the second recognition has a finer scale than the first recognition, which is beneficial to further improve the recognition accuracy. Therefore, the accuracy of the language involved in the recognized speech can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the speech recognition method of the present application;

[0011] Figure 2 This is a process diagram of an embodiment of the speech recognition method of the present application;

[0012] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0013] Figure 4 is a flow chart of an embodiment of language recognition;

[0014] Figure 5 It is a schematic diagram of the framework of an embodiment of a language recognition model;

[0015] Figure 6 It is a flowchart of an embodiment of training a language recognition model;

[0016] Figure 7It is a schematic diagram of the framework of an embodiment of the speech recognition device of the present application;

[0017] Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0018] Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0022] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech recognition method of the present application. Specifically, it may include the following steps:

[0023] Step S11: Obtain a spectrogram of the speech to be recognized.

[0024] In an implementation scenario, the speech to be recognized can be windowed and Fourier transformed to obtain acoustic feature parameters (such as FilterBank, etc.). The dimensions of the acoustic feature parameters can be set according to actual needs, such as 128 dimensions, 256 dimensions, etc., which are not limited here. For the convenience of description, the dimension of the acoustic feature parameters can be recorded as d dimensions. On this basis, the acoustic feature parameters obtained by Fourier transforming the speech frame obtained by windowing the speech to be recognized can be composed of a spectrogram. For the convenience of description, the spectrogram can be recorded as x spec For a more specific spectrogram extraction process, please refer to the relevant technical details of the spectrogram, which will not be repeated here.

[0025] In an implementation scenario, the speech to be recognized can be set according to the actual application scenario. For example, the speech to be recognized can be collected at the conference site. In this case, the target language involved in the speech to be recognized can be recognized through the embodiment disclosed in this application, so that the translation model, recognition model, etc. of the target language can be used to translate and recognize the speech to be recognized, so as to obtain the recognition text, translation text or simultaneous interpretation of the speech to be recognized, thereby greatly reducing the communication barriers of people of different languages ​​at the conference site; or, the speech to be recognized can also be collected from an online conversation. In this case, the target language involved in the speech to be recognized can be recognized through the implementation disclosed in this application, so that the translation model and recognition model of the target language can be used to translate and recognize the speech to be recognized, so as to obtain the recognition text, translation text or simultaneous interpretation of the speech to be recognized, thereby greatly reducing the communication barriers of people of different languages ​​participating in the online conversation. Other application scenarios can be deduced by analogy, and no more examples are given here.

[0026] Step S12: extracting and re-identifying the spectrum segment from the spectrogram based on the first recognition result of the language recognition performed by the first sliding window on the spectrogram window by window.

[0027] In one implementation scenario, the sliding window size may include a window length and a window shift, and the size of the first sliding window may be set according to actual application needs. For example, the window length of the first sliding window may be set to 5 seconds, and the window shift of the first sliding window may be set to 0.1 seconds, which are not limited here. It should be noted that the window length represents the overall length of the window function, and the window shift represents the step size of each sliding of the window function. Taking a window length of 5 seconds and a window shift of 0.1 seconds as an example, each sliding can extract a 5-second long spectral segment from the spectrogram, and the next sliding needs to be 0.1 seconds apart. If the starting time of the spectral segment extracted by this sliding in the spectrogram is the tth second, then the next sliding needs to extract a 5-second spectral segment from the t+0.1th second of the spectrogram. When the first sliding window is other cases, it can be deduced by analogy, and examples are not given one by one here.

[0028] In one implementation scenario, language recognition can be performed window by window in the spectrogram based on the first sliding window to obtain first recognition results of several first spectrogram segments, and the first recognition results include first correlation scores of the first spectrogram segments with several preset languages, respectively. On this basis, for each first spectrogram segment, a first score difference between the highest first correlation score and the second highest first correlation score can be obtained, and based on the first score difference, the spectrogram segment is extracted and re-recognized from the spectrogram. For ease of description, the first spectrogram segment can be denoted as X spec_windowIn the above method, the re-recognized spectrum segment is extracted from the spectrogram by using the difference between the highest correlation score and the second highest correlation score in the first recognition result of each first spectrum segment obtained by performing language recognition on the spectrogram window by window through the first sliding window. Therefore, the re-recognized spectrum segment can be extracted according to the language recognition result of each first spectrum segment, thereby reducing the length of the spectrum that needs to be recognized twice as much as possible, and further reducing the impact on the recognition efficiency as much as possible on the basis of improving the recognition accuracy.

[0029] In a specific implementation scenario, several preset languages ​​can be set according to actual application needs. Taking the aforementioned conference site as an example, for a European national conference, the languages ​​of European countries can be used as several preset languages ​​(such as English, French, German, Spanish, Portuguese, etc.), and for a Middle Eastern national conference, the languages ​​of Middle Eastern countries can be used as several preset languages ​​(such as Arabic, Persian, etc.), and other situations can be deduced by analogy, and no examples are given here one by one.

[0030] In a specific implementation scenario, in order to improve the efficiency of language recognition, a language recognition model can be pre-trained, so that the language recognition model can be used to perform language recognition on the first spectrum segment to obtain a first recognition result of the first spectrum segment. Specifically, the language recognition model can include but is not limited to: DNN (Deep Neural Network), TDNN (Time Delay Neural Network), LSTM (Long Short Term Memory), etc., which are not limited here. During the training process, a sample spectrum of a sample speech can be obtained first, and the sample speech can be annotated with the sample language involved. On this basis, the sample spectrum can be input into the language recognition model for prediction to obtain the predicted relevant scores of several preset languages ​​of the sample speech, and a loss function such as cross entropy is used to calculate the loss based on the predicted relevant scores of the sample language and several preset languages ​​to obtain the loss value of the language recognition model, and an optimization method such as gradient descent is used to adjust the network parameters of the language recognition model based on the loss value. For the specific process of loss calculation, please refer to the technical details of loss functions such as cross entropy. For the specific process of parameter adjustment, please refer to the technical details of optimization methods such as gradient descent, which will not be repeated here.

[0031] In a specific implementation scenario, for each first spectrum segment X spec_window, the first correlation scores of the first recognition results and the preset languages ​​can be sorted in descending order, so as to obtain the highest first correlation score and the second highest first correlation score in the first recognition results, and calculate the first score difference between the two. In the above manner, each first spectrum segment X can be obtained. spec_window Corresponding first score difference. It should be noted that the highest first correlation score is the first correlation score ranked first, and the second highest first correlation score is the first correlation score ranked second.

[0032] In a specific implementation scenario, at least one group of extraction time periods can be determined based on at least one group of first spectrum segments whose first score difference is lower than the first threshold and is continuous, and each group of extraction time periods includes a start time and an end time. On this basis, the re-identified spectrum segments can be extracted from the spectrogram based on each group of extraction time periods. Still taking the example that the window length of the first sliding window is 5 seconds and the window shift is 0.1 seconds, the first extracted first spectrum segment starts from 0 seconds of the spectrogram and ends at 5 seconds of the spectrogram. For the convenience of description, the first spectrum segment can be recorded as X spec_window (0,5), and so on, the first spectral fragment of the i-th can be recorded as X spec_window((i-1)*0.1,(i-1)*0.1+5). In the case that the first difference scores of the 1st to 3rd first spectrogram segments are not lower than the first threshold, if the first score differences of a subsequent group of first spectrogram segments (the 4th spectrogram segment and the 5th spectrogram segment) are all lower than the first threshold, then a group of extraction time periods can be obtained based on the start time (i.e. 0.3 seconds) of the first spectrogram segment (i.e. the 4th first spectrogram segment) and the end time (i.e. 5.4 seconds) of the last spectrogram segment (i.e. the 5th first spectrogram segment) in the group of first spectrogram segments, and a re-recognized spectrogram segment is extracted from 0.3 seconds to 5.4 seconds of the spectrogram; further, in the 6th In the case where the first difference scores of the first spectrogram segments from the 1st to the 10th spectrogram segments are not lower than the first threshold, if the first score differences of a subsequent group of first spectrogram segments (the 11th to the 13th first spectrogram segments) are all lower than the first threshold, then the first group of extraction time periods can be obtained based on the start time (i.e. 1 second) of the first spectrogram segment (i.e. the 11th first spectrogram segment) in the group of first spectrogram segments and the end time (i.e. 6.2 seconds) of the last first spectrogram segment (i.e. the 13th spectrogram segment), and a re-recognized spectrogram segment is extracted from 1 second to 6.2 seconds in the spectrogram. Other situations can be deduced by analogy, and examples are not given one by one here. That is to say, a group of extraction time periods can be obtained based on the start time of the first first spectrogram segment and the end time of the last spectrogram segment in each group of first spectrogram segments, and a re-recognized spectrogram segment can be cut out from the start time in the extraction time period to the end time in the extraction time period in the spectrogram. The above method determines at least one group of extraction time periods based on at least one group of first spectrum segments whose first score difference is lower than the first threshold and is continuous, and each group of extraction time periods includes a start time and an end time. On this basis, re-identified spectrum segments are extracted from the spectrogram based on each group of extraction time periods. Therefore, re-identification can be performed only based on the first spectrum segments whose languages ​​are difficult to distinguish during the first language recognition process, which is beneficial to minimize the impact on recognition efficiency.

[0033] It should be noted that in order to improve the accuracy of language re-identification, redundancy can be reserved when determining the extraction period. Specifically, at least one first spectrum segment whose first score difference is lower than the first threshold and is continuous and the next first spectrum segment whose first score difference is not lower than the first threshold can be taken as a group of first spectrum segments, and a group of extraction periods can be determined according to the start time of the first first spectrum segment and the end time of the last first spectrum segment in the group of first spectrum segments. Still taking the example that the window length of the first sliding window is 5 seconds and the window shift is 0.1 seconds, the i-th first spectrum segment can be recorded as X spec_window((i-1)*0.1,(i-1)*0.1+5), if the first score difference between the 4th spectrogram segment and the 5th spectrogram segment is lower than the first threshold, and the first score difference of the 6th first spectrogram segment is not lower than the first threshold, then the 4th to 6th first spectrogram segments can be taken as a group of first spectrogram segments. On this basis, a group of extraction time periods can be obtained based on the start time (i.e. 0.3 seconds) of the first spectrogram segment (i.e. the 4th first spectrogram segment) and the end time (i.e. 5.5 seconds) of the last spectrogram segment (i.e. the 6th first spectrogram segment) in the group of first spectrogram segments, and a re-recognition spectrogram segment is extracted from 0.3 seconds to 5.5 seconds of the spectrogram. It can be seen that compared with the aforementioned method, the re-recognition spectrogram segment extracted by this method is slightly increased, which is conducive to appropriately improving redundancy, thereby improving the accuracy of re-recognition while ensuring recognition efficiency.

[0034] In a specific implementation scenario, based on the first score difference of the first speech segment not being lower than the first threshold, the preset language corresponding to the highest first correlation score can be used as the target language involved in the first speech segment. Still taking the first sliding window length of 5 seconds and the window shift of 0.1 seconds as an example, as mentioned above, the i-th first speech segment can be recorded as X spec_window ((i-1)*0.1,(i-1)*0.1+5), the first correlation scores of the first spectrum segment and several preset languages ​​(English, French, German, Spanish, Portuguese, etc.) are: 0.96, 0.01, 0.01, 0.01.01, then the first score difference between the highest first correlation score and the second highest first correlation score is 0.95. When the first threshold is set to 0.8, the first score difference is not lower than the first threshold, then the preset language (i.e. English) corresponding to the highest first correlation score can be used as the target language involved in the first first spectrum segment. In other words, the start time and end time of the target language at the speech to be recognized can be located based on the start time and end time of the first spectrum segment. For example, when the first difference scores of the first to third first spectrogram segments are not lower than the first threshold, the preset languages ​​corresponding to the highest first correlation scores of the above three first spectrogram segments are all English, then English can be located at the start time (0 seconds) and the end time (5.2 seconds) of the speech to be recognized. Other situations can be deduced by analogy, and no examples are given here. The above method, based on the fact that the first score difference of the first spectrogram segment is not lower than the first threshold, uses the preset language corresponding to the highest first correlation score as the target language involved in the first spectrogram segment, which can directly determine the target language involved in the first spectrogram segment, which is conducive to further improving the efficiency of language recognition on the basis of ensuring the accuracy of language recognition.

[0035] It should be noted that, since the greater the first score difference between the highest first correlation score and the second highest first correlation score, the more reliable the language identification of the first spectrum segment is, and conversely, the smaller the first score difference between the highest first correlation score and the second highest first correlation score is, the less reliable the language identification of the first spectrum segment is. Therefore, the above-mentioned first threshold can be set according to the actual application requirements. For example, in the case of high requirements for the accuracy of language identification, the first threshold can be set larger, and conversely, in the case of relatively loose requirements for the accuracy of language identification, the first threshold can be set slightly smaller, and the specific value is not limited here.

[0036] In one implementation scenario, language recognition can also be performed window by window in the spectrogram based on the first sliding window to obtain a first recognition result of the first spectrogram segment corresponding to the current sliding window, and for the first spectrogram segment corresponding to the current sliding window, a first score difference between the highest first correlation score and the second highest first correlation score can be obtained. If the first score difference is not lower than a first threshold, the preset language corresponding to the highest first correlation score can be directly used as the target language involved in the first spectrogram segment corresponding to the current sliding window. Conversely, if the first score difference is lower than the first threshold, the start time of the current sliding window can be recorded as T start , and continue to perform the above-mentioned step of performing language identification window by window in the spectrogram based on the first sliding window, so as to continue sliding the window in the time domain until the first score difference of the first spectrogram segment corresponding to this sliding window is higher than the first threshold value, and record the end time T of this sliding window end , and extract a segment of T from the spectrogram start ~T end Re-identify the spectral fragment.

[0037] Step S13: Based on the second recognition results of the language recognition performed window by window on the re-recognized spectral segment by the plurality of second sliding windows, the target language involved in the re-recognized spectral segment is analyzed.

[0038] In the disclosed embodiment, the sizes of the plurality of second sliding windows are different from each other, and the size of the second sliding window is smaller than the size of the first sliding window.

[0039] In one implementation scenario, as described above, the sliding window size may include a window length and a window shift, and the window lengths and / or window shifts of the plurality of second sliding windows may be different. That is, the window lengths of the plurality of second sliding windows may be different, or the window shifts of the plurality of second sliding windows may be different; or both the window lengths and window shifts of the plurality of second sliding windows may be different; or, as long as any one of the window lengths and window shifts of the plurality of second sliding windows is different, in short, the plurality of second sliding windows are not completely the same, and this is not limited here. Exemplarily, six second sliding windows may be set, the window length of second sliding window No. 1 is w=2 seconds and the window shift is s=0.1 seconds, the window length of second sliding window No. 2 is w=2 seconds and the window shift is s=0.05 seconds, the window length of second sliding window No. 3 is w=2 seconds and the window shift is s=0.01 seconds, the window length of second sliding window No. 4 is w=1 second and the window shift is s=0.1 seconds, the window length of second sliding window No. 5 is w=1 second and the window shift is s=0.05 seconds, and the window length of second sliding window No. 6 is w=1 second and the window shift is s=0.01 seconds. It should be noted that the above example is only a possible solution in actual application, and does not limit the actual number of second sliding windows and the specific values ​​of window length and window shift.

[0040] In one implementation scenario, the sliding window size may include a window length, and the window length of the second sliding window is not greater than the window length of the first sliding window. As mentioned above, the window length of the first sliding window may be set to 5 seconds, and the window lengths of the above six second sliding windows may be set to 2 seconds, 1 second, etc., which are not limited here.

[0041] In one implementation scenario, the sliding window size may include a window shift, and the window shift of the second sliding window is not greater than the window shift of the first sliding window. As mentioned above, the window shift of the first sliding window can be set to 0.1 seconds, and the window shifts of the above six second sliding windows can be set to 0.1 seconds, 0.05 seconds, 0.01 seconds, etc., which are not limited here.

[0042] In one implementation scenario, the second recognition results of each second sliding window performing language recognition on the re-recognized spectral segment window by window can be obtained first, and the second recognition results include the second correlation scores of the re-recognized spectral segment and several preset languages. On this basis, the second recognition results corresponding to each second sliding window can be directly fused to obtain the final recognition result, and the target language involved in the re-recognized spectral segment can be determined based on the final recognition result. The above method can greatly reduce the complexity of language recognition for the re-recognized spectral segment by obtaining the second recognition results of each second sliding window performing language recognition on the re-recognized spectral segment window by window, and directly fusion The second recognition results corresponding to each second sliding window are obtained to obtain the final recognition result, and the target language involved in the re-recognized spectral segment is determined based on the final recognition result. It should be noted that the specific process of obtaining the second recognition result can refer to the following related public embodiments, which will not be repeated here.

[0043] In a specific implementation scenario, for each preset language, the second relevant score of the preset language in the second recognition results corresponding to each second sliding window can be obtained, and these second relevant scores can be averaged, mediand, and other fusion operations performed to obtain the final relevant score of the preset language. On this basis, the final recognition result can include the final relevant scores of several preset languages.

[0044] In a specific implementation scenario, as mentioned above, the final recognition result may include final relevance scores of several preset languages. On this basis, the preset language corresponding to the highest final relevance score may be used as the target language involved in the re-recognition of the spectral segment.

[0045] In one implementation scenario, different from the above-mentioned method, after obtaining the second recognition results of each second sliding window for language recognition in the re-recognized speech spectrum segment, for each second sliding window, the second recognition results corresponding to the second sliding window can be regularized using the statistical recognition results corresponding to the second sliding window to obtain the regularized recognition results corresponding to the second sliding window, and the statistical recognition results are statistically obtained based on the sample recognition results of the second sliding window for language recognition in several sample spectra, and on this basis, the target language involved in the re-recognized speech spectrum segment is determined based on the final recognition result obtained by fusing the regularized recognition results corresponding to several second sliding windows. In the above-mentioned method, the second recognition results corresponding to the second sliding window are first regularized using the statistical recognition results corresponding to the second sliding window, so the inherent differences between the second recognition results obtained by language recognition based on the second sliding windows of different sizes can be eliminated as much as possible, which is conducive to improving the accuracy of the final recognition result, and further can be conducive to improving the accuracy of the target language involved in the re-recognized speech spectrum segment. In addition, the specific process of obtaining the second recognition result, regularizing the recognition result, and fusing the recognition result can refer to the following related public embodiments, which will not be repeated here.

[0046] In one implementation scenario, after determining the target language involved in the re-recognized speech segment, the start time and end time of the target language in the speech to be recognized can be located. For details, please refer to the description of the target language involved in the first speech segment mentioned above, which will not be repeated here.

[0047] In an implementation scenario, please refer to Figure 2 , Figure 2 1 is a process diagram of an embodiment of the speech recognition method of the present application. Figure 2As shown, a first sliding window (as shown in the bold rectangular box in the figure) can be used to perform language recognition on a window-by-window basis in the spectrogram to obtain a first recognition result for the time period corresponding to the window, and the first result includes first correlation scores with several preset languages. On this basis, a first score difference between the highest first correlation score and the second highest first correlation score can be obtained. If the first score difference is not lower than a first threshold, the preset language corresponding to the highest first correlation score can be directly used as the target language involved in the time period corresponding to the window. Otherwise, a multi-scale fusion decision can be made based on several second sliding windows. The specific process can refer to the above-mentioned related description, which will not be repeated here.

[0048] The above scheme obtains the spectrogram of the speech to be recognized, and obtains the first recognition result of language recognition based on the first sliding window in the spectrogram window by window, extracts the re-recognized spectrogram segment from the spectrogram, and on this basis, performs the second recognition result of language recognition based on a plurality of second sliding windows in the re-recognized spectrogram segment window by window, analyzes and obtains the target language involved in the re-recognized spectrogram segment, and the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window. Therefore, on the one hand, in the whole recognition process, at most two language recognitions can be performed, namely, the first language recognition based on the first sliding window and the second language recognition based on the second sliding window. Therefore, the double recognition can be beneficial to improve the recognition accuracy. On the other hand, in the second recognition process, a plurality of second sliding windows of different sizes are used, which can fuse the multi-scale recognition results, which is beneficial to improve the recognition accuracy again. On the other hand, the size of the second sliding window used in the second recognition process is smaller than the size of the first sliding window, that is, the second recognition has a finer scale than the first recognition, which is beneficial to further improve the recognition accuracy. Therefore, the accuracy of the language involved in the recognized speech can be improved.

[0049] See also Figure 3 , Figure 3 yes Figure 1 The flowchart of step S13 in the embodiment of FIG. 1 includes the following steps:

[0050] Step S31: obtaining second recognition results of each second sliding window performing language recognition window by window on the re-recognized speech spectrum segment.

[0051] In the disclosed embodiment, the second recognition result includes the second correlation scores between the re-recognized speech segment and several preset languages. Taking the six second sliding windows used in the disclosed embodiment as an example, after the second sliding window No. 1 performs language recognition window by window, a second recognition result can be obtained, which indicates the second correlation scores between the re-recognized speech segment and several preset languages ​​after the second sliding window No. 1 performs language recognition. The other second sliding windows can be deduced by analogy, and no examples are given one by one here. Specifically, each second sliding window can be used as the current sliding window, and based on the current sliding window, language recognition is performed window by window on the re-recognized speech segment to obtain the third recognition results of several second speech segments, and the third recognition result includes the third correlation scores between the second speech segment and several preset languages. On this basis, based on the third correlation scores between each second speech segment and the preset language, the second correlation scores between the re-recognized speech segment and the corresponding preset language can be obtained, so that the second correlation scores between the re-recognized speech segment and each preset language can be combined to obtain the second recognition result corresponding to the current sliding window. The above method, by taking each second sliding window as the current sliding window, and performing language recognition window by window in the re-identified spectrum segment based on the current sliding window, obtains the third recognition results of several second spectrum segments, and the third recognition results include the third correlation scores of the second spectrum segments and several preset languages, thereby obtaining the second correlation score of the re-identified spectrum segment and the corresponding preset language based on the third correlation score of each second spectrum segment and the preset language, and then quickly obtaining the second recognition results corresponding to each second sliding window, which is beneficial to improving the overall recognition efficiency.

[0052] In one implementation scenario, taking the six second sliding windows used in the aforementioned disclosed embodiment as an example, the second sliding window No. 1, the second sliding window No. 2, ..., the second sliding window No. 6 can be respectively used as the current sliding window, and language recognition is performed window by window in the re-identified speech spectrum segment based on the current sliding window. Other situations can be deduced by analogy, and no examples are given here one by one.

[0053] In an implementation scenario, the relevant meaning of the second speech segment can refer to the relevant description of the first speech segment in the aforementioned public embodiment, which will not be repeated here. Taking the third recognition result of n1 second speech segments obtained by taking the second sliding window No. 1 as the current sliding window and performing language recognition window by window as an example, for the convenience of calculation, the third recognition result of each second speech segment can be represented as a score vector. For example, the i-th third recognition result of the third recognition result of n1 second speech segments obtained by performing language recognition on the second sliding window No. 1 can be represented as the score vector S 1 i , where i = 1, 2, ..., n1. In addition, the score vector S 1 i contains N elements, each element represents the third correlation score of a preset language.

[0054] In one implementation scenario, the third correlation scores of each second spectral segment and the preset language can be averaged (or median) to serve as the second correlation score of the re-identified spectral segment and the preset language. As mentioned above, the third recognition result of each second spectral segment can be represented as a score vector, and the elements at each position in the score vector of each second spectral segment can be averaged (or median) to obtain the second recognition result. Taking the average value as an example, the second recognition result corresponding to the second sliding window No. 1 can be expressed as:

[0055]

[0056] In the above formula (1), S 1 represents the second recognition result corresponding to the second sliding window No. 1, It represents the ith third recognition result among the third recognition results of n1 second speech spectrum segments obtained by performing language recognition using the second sliding window No. 1, and n1 represents the total number of second speech spectrum segments obtained by performing language recognition window by window using the second sliding window No. 1. It should be noted that formula (1) is a calculation formula for the corresponding second recognition result when the second sliding window No. 1 is used as an example, and the same can be applied to other second sliding windows, which are not given one by one here.

[0057] Step S32: for each second sliding window, regularize the second recognition result corresponding to the second sliding window by using the statistical recognition result corresponding to the second sliding window to obtain the regularized recognition result corresponding to the second sliding window.

[0058] In the embodiment of the present disclosure, the statistical recognition result is obtained by statistically analyzing the sample recognition results of the second sliding window performing language recognition on a plurality of sample spectra window by window.

[0059] In an implementation scenario, for each second sliding window, each sample spectrum can be used as the current spectrum, and the second sliding window can be used to perform language recognition in the current spectrum window by window to obtain sample correlation scores of several sample spectrum segments and several preset languages. On this basis, when the second sliding window is used, for each sample spectrum, the sample correlation scores of several sample spectrum segments and several preset languages ​​can be obtained, and the standard deviation of these sample correlation scores can be counted to obtain the standard deviation when the second sliding window is used, and for each preset language, the average value of these sample correlation scores can be counted to obtain the average correlation score of each preset language when the second sliding window is used. That is to say, the sample recognition result includes the sample correlation scores of several sample spectrum segments extracted by the second sliding window in the sample spectrum and several preset languages, and the statistical recognition result is the standard deviation counted based on the sample correlation score and the average correlation score of each preset language.

[0060] In one implementation scenario, as mentioned above, for the convenience of description, the average relevant scores of each preset language included in the statistical recognition result can also be represented by a score vector. For example, the average relevant scores of each preset language included in the statistical recognition result corresponding to the j-th second sliding window can be recorded as The standard deviation of the statistical recognition result corresponding to the jth second sliding window is recorded as σ j .

[0061] In one implementation scenario, the regularized recognition result includes the regularized correlation scores of the re-recognized speech segment and several preset languages. On this basis, for each second sliding window, each preset language can be used as the current language, and the second score difference between the second correlation score corresponding to the current language and the average correlation score is obtained, and the ratio of the second score difference to the standard deviation is used as the regularized correlation score of the re-recognized speech segment and the current language. Still taking the jth second sliding window as an example, for the convenience of description, the regularized recognition result can also be expressed in the form of a score vector, such as Then regularize the recognition results It can be expressed as:

[0062]

[0063] In the above formula (2), S j represents the second recognition result corresponding to the j-th second sliding window, The vector form representing the average correlation scores of each preset language contained in the statistical recognition result corresponding to the j-th second sliding window, σ j Represents the standard deviation of the statistical recognition result corresponding to the j-th second sliding window.

[0064] Step S33: Determine the target language involved in the re-recognized spectral segment based on the final recognition result obtained by fusing the regularized recognition results corresponding to the second sliding windows.

[0065] Specifically, as mentioned above, the regularized recognition result includes the regularization correlation scores of the re-recognized spectral segments and several preset languages. For the convenience of calculation, the regularized recognition result corresponding to the j-th second sliding window can be expressed in the form of a score vector, which can be expressed as Correspondingly, the final recognition result may include the final correlation scores of the re-recognized speech segments and several preset languages. For the convenience of calculation, it can also be expressed in the form of a score vector, such as S. On this basis, the final recognition result S can be expressed as:

[0066]

[0067] In the above formula (3), M represents the total number of the second sliding windows (e.g., 6). In addition, as mentioned above, Represents the regularized recognition result expressed in the form of a score vector. It should be noted that, in the process of vector summation, the elements at corresponding positions can be summed. That is to say, each preset language can be taken as the current language, and the regularized correlation scores of the re-recognized spectral fragments and the current language in the regularized recognition results corresponding to each second sliding window can be summed to obtain the final correlation scores of the re-recognized spectral fragments and the current language, and finally the final correlation scores of the re-recognized spectral fragments and each preset language can be combined to obtain the final recognition result. On this basis, the preset language corresponding to the highest final correlation score can be used as the target language involved in the re-recognized spectral fragments. The above method, by taking each preset language as the current language respectively, and summing up the regular correlation scores of the re-recognized spectral fragments and the current language in the regular recognition results corresponding to each second sliding window, obtains the final correlation score of the re-recognized spectral fragments and the current language, and takes the preset language corresponding to the highest final correlation score as the target language involved in the re-recognized spectral fragments. Therefore, the final correlation score can be obtained through simple mathematical operations, which is beneficial to improving recognition efficiency while ensuring the accuracy of language recognition.

[0068] The above scheme first uses the statistical recognition result corresponding to the second sliding window to regularize the second recognition result corresponding to the second sliding window, so it can eliminate the inherent differences between the second recognition results obtained by performing language recognition window by window based on second sliding windows of different sizes as much as possible, which is beneficial to improving the accuracy of the final recognition result, and further can be beneficial to improving the accuracy of the target language involved in the re-identification of the spectral fragment.

[0069] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of language recognition. Specifically, it may include the following steps:

[0070] Step S41: extracting features from the spectrogram to be recognized to obtain a first feature graph and a language feature representation.

[0071] In an implementation scenario, please refer to Figure 5 , Figure 5It is a framework diagram of an embodiment of a language recognition model. As described in the aforementioned disclosed embodiment, in order to improve the efficiency of language recognition, a language recognition model can be pre-trained, and in the disclosed embodiment, the language recognition model can include a feature extraction network. The feature extraction network can include but is not limited to: CNN (Convolutional Neural Network, CNN), TDNN, LSTM, etc., which are not limited here. On this basis, the feature extraction network can be used to perform feature extraction operations. Specifically, the feature map finally output by the feature extraction network can be used as the first feature map, and the hidden layer representation of the feature extraction network can be used as the language feature representation. It should be noted that in order to improve the feature extraction effect, the feature extraction network can specifically adopt TDNN.

[0072] Step S42: Process the residual of the first feature map to obtain a second feature map.

[0073] In one implementation scenario, the first feature map can be upsampled to obtain an upsampled feature map, and the upsampled feature map and the third feature map extracted from the upsampled feature map are fused to obtain a fused feature map, and on this basis, the fused feature map is downsampled to obtain a second feature map. The above method completes residual processing through upsampling, feature fusion, and downsampling processing processes, which can help improve the accuracy of language recognition on the basis of deepening the feature level.

[0074] In a specific implementation scenario, upsampling is used to convert the three-dimensional first feature map into four-dimensional, and downsampling is used to restore the four-dimensional fused feature map to three-dimensional.

[0075] In a specific implementation scenario, the upsampled feature map and the third feature map can be fused by a point addition operation. For ease of description, the spectrogram to be recognized can be denoted as X spec , and the upsampled feature map obtained by upsampling is recorded as F sepc , by upsampling feature map F sepc The extracted third feature map is denoted as R spec , then the second feature map Y spec It can be expressed as:

[0076] Y spec =F sepc +R spec ……(4)

[0077] In an implementation scenario, as mentioned above, in order to improve the efficiency of language recognition, a language recognition model can be pre-trained, and please refer to Figure 5In the disclosed embodiment, the language recognition model may include a residual processing network, and the residual processing network is used to perform residual processing operations. Specifically, the residual processing network may include an upsampling layer for performing an upsampling operation, the residual processing network may include an extraction layer for performing an operation of extracting a third feature map of the upsampled features, the residual processing network may include a first fusion layer for performing an operation of fusing the upsampled feature map with the third feature map, and the residual processing network may include a downsampling layer for performing a downsampling operation. It should be noted that the three-dimensional first feature map can be converted into a four-dimensional upsampled feature map by upsampling to match the network size of the subsequent extraction layer, and the four-dimensional fused feature map can be downsampled and restored to a three-dimensional second feature map, which can match the network size of the subsequent feature extraction network.

[0078] In a specific implementation scenario, the extraction layer can be composed of several residual blocks, such as two residual blocks, to deepen the network depth, which is beneficial to avoid problems such as gradient vanishing and network degradation in the language recognition model during the training process, thereby improving the model performance of the language recognition model.

[0079] Step S43: extracting features from the second feature map to obtain a new first feature map and a new language feature representation.

[0080] Specifically, the specific method of feature extraction can refer to the aforementioned description of feature extraction of the spectrogram to be recognized, which will not be repeated here.

[0081] Step S44: Perform prediction processing based on the language feature representations extracted previously to obtain the language recognition result of the spectrogram to be recognized.

[0082] In one implementation scenario, the language feature representations extracted each time can be pooled separately to obtain the language embedding vectors (Language Embedding) corresponding to each feature extraction. On this basis, all language embedding vectors can be accumulated to obtain the overall language embedding vector, which is used to describe the language information represented by different speech data. After that, the overall language embedding vector can be linearly transformed to predict the correlation scores between the spectrogram to be identified and several preset languages. It should be noted that the correlation scores of several preset languages ​​constitute the language recognition results of the spectrogram to be identified.

[0083] In an implementation scenario, as mentioned above, in order to improve the efficiency of language recognition, a language recognition model can be pre-trained, and please refer to Figure 5In the disclosed embodiment, the language recognition model may include a category prediction network for performing a prediction processing operation. Specifically, the category prediction network may include a pooling layer for performing a pooling process on the language feature representation to obtain a language embedding vector, the category prediction network may include a second fusion layer for accumulating all language embedding vectors to obtain an overall language embedding vector, and the category prediction network may include a fully connected layer for performing a linear transformation on the overall language embedding vector to obtain a language recognition result.

[0084] It should be noted that the language recognition result includes the language-related scores of the spectrogram to be recognized and several preset languages. In the case of language recognition based on the first sliding window, the spectrogram to be recognized is the first spectrogram segment extracted by the first sliding window from the spectrogram window by window. In the case of language recognition based on the second sliding window, the spectrogram to be recognized is the second spectrogram segment extracted by the second sliding window from the re-recognized spectrogram segment window by window. In other words, in the process of language recognition based on the first sliding window window by window and in the process of language recognition based on the second sliding window window by window, the same language recognition steps can be adopted, which can help to further reduce the complexity of overall recognition.

[0085] In addition, in order to further improve the accuracy of language recognition, before performing the prediction process, it is also possible to detect whether the number of executions of feature extraction is not higher than the second threshold. If the number of executions is not higher than the second threshold, the residual processing of the first feature map can be re-executed to obtain the step of the second feature map and the subsequent steps, that is, the above step S42 and the subsequent steps can be re-executed. On the contrary, if the number of executions is higher than the second threshold, the prediction process based on the language feature representation extracted in the past can be performed to obtain the language recognition result of the speech spectrum to be identified, that is, the above step S44 can be directly executed. The above method, by first detecting whether the number of executions of feature extraction is not higher than the second threshold before the prediction process, and re-executing the residual processing of the first feature map to obtain the step of the second feature map and the subsequent steps when the number of executions is not higher than the second threshold, can help improve the accuracy of language recognition. In addition, the second threshold can be set according to actual needs, such as 2, 3, 4, etc., which is not limited here.

[0086] It should be noted that, as mentioned above, in order to improve the efficiency of language recognition, a language recognition model can be pre-trained, and please refer to Figure 5 The language recognition model may include a second threshold number of feature extraction networks, and a residual processing network is arranged between adjacent feature extraction networks, so that the accuracy of language recognition can be improved through a multi-layer network.

[0087] The above scheme performs feature extraction on the spectrogram to be identified to obtain a first feature graph and a language feature representation, performs residual processing on the first feature graph to obtain a second feature graph, and performs feature extraction on the second feature graph to obtain a new first feature graph and a new language feature representation. On this basis, prediction processing is performed based on the language feature representations extracted in the past to obtain the language recognition result of the spectrogram to be identified. Therefore, in the process of language recognition, language prediction can be performed by combining the language feature representations extracted multiple times, which is beneficial to improving the accuracy of language recognition.

[0088] See also Figure 6 , Figure 6 1 is a flow chart of an embodiment of training a language recognition model. It should be noted that in the embodiment of the present disclosure, the language recognition model may include: a feature extraction network, a residual processing network and a category prediction network. Specifically, the following steps may be included:

[0089] Step S61: extract features from the sample spectrogram using a feature extraction network to obtain a first sample feature graph and a sample language feature representation.

[0090] In the embodiment of the present disclosure, the sample spectrogram is extracted based on the sample speech, and the sample speech is annotated with the sample language, which indicates the language actually used by the sample speech. Figure 5 As well as the related descriptions in the aforementioned disclosed embodiments, they will not be repeated here.

[0091] Step S62: Perform residual processing on the first sample feature map using a residual processing network to obtain a second sample feature map.

[0092] The specific process of residual processing can be found in Figure 5 As well as the related descriptions in the aforementioned disclosed embodiments, they will not be repeated here.

[0093] Step S63: Use the feature extraction network to extract features from the second sample feature map to obtain a new first sample feature map and a new sample language feature representation.

[0094] The specific process of feature extraction can be found in Figure 5 As well as the related descriptions in the aforementioned disclosed embodiments, they will not be repeated here.

[0095] Step S64: using the category prediction network to perform prediction processing based on the sample language feature representations extracted previously, to obtain the sample language recognition result of the sample spectrogram.

[0096] In the embodiment of the present disclosure, the sample language recognition result includes the predicted correlation scores of the sample spectrogram and several preset languages. The specific process of the prediction processing can be referred to Figure 5As well as the related descriptions in the aforementioned disclosed embodiments, they will not be repeated here.

[0097] In addition, as described in the aforementioned disclosed embodiment, before performing the prediction process, it is also possible to detect whether the current feature extraction network is the last feature extraction network of the language recognition model. If not, step S62 and subsequent steps can be continued to be executed, otherwise step S64 can be directly executed.

[0098] Step S65: adjusting the network parameters of the language recognition model based on the sample language recognition result and the difference between the sample languages.

[0099] Specifically, a loss function such as cross entropy can be used to measure the above differences to obtain a loss value of the language recognition model, and based on the loss value, an error back propagation (BP) algorithm is used to adjust the network parameters of the language recognition model.

[0100] In one implementation scenario, the above training steps may be repeated for multiple rounds (eg, 5 rounds, 6 rounds, etc.) to stabilize the model performance of the language recognition model.

[0101] In an implementation scenario, in order to further improve the model performance, after the language recognition model becomes stable, the language recognition model can be further optimized for several rounds. Specifically, in the first round of optimization, the language recognition model can be used to perform language recognition on the sample spectrogram of each sample speech to obtain a sample recognition score for each sample speech, and based on the sample recognition score of each sample speech (i.e., the highest value among the sample-related scores of several preset languages) and the decision threshold corresponding to this round of optimization (e.g., the decision threshold corresponding to the first round of optimization can be set It can be set to 0.9) for comparison. If the judgment threshold is exceeded, the sample spectrogram of the sample speech can be retained, otherwise the sample spectrogram of the sample speech can be discarded, so that the first round of re-optimized training subset S1 can be constructed based on the sample spectrograms of the N retained sample speech, and the training subset can be constructed based on the sample spectrograms of the M discarded sample speech. On this basis, the training subset S1 can be used to train the language recognition model for several rounds (e.g., 5 rounds, 6 rounds, etc.) according to the aforementioned training steps to re-optimize the language recognition model, thereby completing the first round of re-optimization. Similarly, in the second round of re-optimization, the language recognition model can be used to optimize the training subset constructed in the first round. Perform language recognition to obtain a training subset The sample recognition score of each sample speech in the , and based on the sample recognition score of each sample speech and the decision threshold corresponding to the current round of optimization (e.g., the decision threshold corresponding to the second round of optimization It can be set to 0.85) for comparison. If it exceeds the decision threshold, the sample spectrogram of the sample speech can be retained, otherwise the sample spectrogram of the sample speech can be discarded, so that the second round of re-optimized training subset S2 can be constructed based on the sample spectrogram of the retained sample speech, and the training subset can be constructed based on the sample spectrogram of the discarded sample speech On this basis, the language recognition model can be trained for several rounds (e.g., 5 rounds, 6 rounds, etc.) using the training subset S2 according to the aforementioned training steps to re-optimize the language recognition model, thereby completing the second round of re-optimization. The third round of re-optimization, the fourth round of re-optimization, and other subsequent rounds of re-optimization can be deduced in the same way, and examples are not given here one by one. In addition, it should be noted that in the last round of re-optimization (e.g., the Tth round), when using the training subset S N After training the language recognition model for several rounds (e.g., 5 rounds, 6 rounds, etc.) according to the above training steps, you can further use the training subset According to the aforementioned training steps, the language recognition model is trained for several rounds (e.g., 5 rounds, 6 rounds, etc.), and then the final language recognition model can be obtained. The above method can continuously improve the model performance of the language recognition model as the number of re-optimization rounds increases, and continuously mine difficult-to-recognize samples with increasing recognition difficulty, so as to continuously re-optimize the language recognition model, which is conducive to continuously improving the model performance of the language recognition model.

[0102] In the above scheme, the language recognition model includes a feature extraction network, a residual processing network and a category prediction network. On this basis, the feature extraction network is used to extract features from the sample spectrogram to obtain a first sample feature graph and a sample language feature representation, and the residual processing network is used to perform residual processing on the first sample feature graph to obtain a second sample feature graph, and the feature extraction network is used to extract features from the second sample feature graph to obtain a new first sample feature graph and a new sample language feature representation, so that the category prediction network is used to perform prediction processing based on the sample language feature representations extracted in the past to obtain a sample language recognition result of the sample spectrogram, and then based on the difference between the sample language recognition result and the sample language, the network parameters of the language recognition model are adjusted. Therefore, during the training process of the language recognition model, the sample language feature representations extracted multiple times can be combined to perform language prediction, which is beneficial to improving the model performance of the language recognition model.

[0103] See also Figure 7 , Figure 7It is a schematic diagram of the framework of an embodiment of a speech recognition device 70 of the present application. The speech recognition device 70 includes: a spectrum acquisition module 71, a first recognition module 72 and a second recognition module 73. The spectrum acquisition module 71 is used to acquire a spectrum graph of the speech to be recognized; the first recognition module 72 is used to extract a re-recognized spectrum segment from the spectrum graph based on a first recognition result of language recognition performed window by window on the spectrum graph by a first sliding window; the second recognition module 73 is used to analyze and obtain the target language involved in the re-recognized spectrum segment based on a second recognition result of language recognition performed window by window on the re-recognized spectrum segment by a plurality of second sliding windows; wherein the sizes of the plurality of second sliding windows are different, and the size of the second sliding window is smaller than the size of the first sliding window.

[0104] The above scheme, on the one hand, can perform language recognition twice at most during the entire recognition process, namely, the first language recognition based on the first sliding window and the second language recognition based on the second sliding window, so the double recognition can help improve the recognition accuracy, on the other hand, the use of several second sliding windows of different sizes in the second recognition process can fuse the multi-scale recognition results, which is conducive to further improving the recognition accuracy, and on the other hand, the size of the second sliding window used in the second recognition process is smaller than the size of the first sliding window, that is, the second recognition has a finer scale than the first recognition, which is conducive to further improving the recognition accuracy. Therefore, the accuracy of the language involved in the recognition of speech can be improved.

[0105] In some disclosed embodiments, the first recognition module 72 includes a first recognition submodule, which is used to perform language recognition window by window in the spectrogram based on the first sliding window to obtain first recognition results of several first spectrogram segments; wherein the first recognition results include first correlation scores of the first spectrogram segments with several preset languages ​​respectively; the first recognition module 72 includes a first difference submodule, which is used to obtain a first score difference between the highest first correlation score and the second highest first correlation score for each first spectrogram segment; the first recognition module 72 includes a spectrogram extraction submodule, which is used to extract and re-recognize the spectrogram segment from the spectrogram based on the first score difference.

[0106] Therefore, the re-recognized spectrogram segment is extracted from the spectrogram by using the difference between the highest correlation score and the second highest correlation score in the first recognition results of each first spectrogram segment obtained by performing language recognition window by window in the spectrogram through the first sliding window. Therefore, the re-recognized spectrogram segment can be extracted according to the language recognition results of each first spectrogram segment, thereby reducing the length of the spectrogram requiring secondary recognition as much as possible, and further reducing the impact on the recognition efficiency as much as possible on the basis of improving the recognition accuracy.

[0107] In some disclosed embodiments, the spectrogram extraction submodule includes a time period determination unit, which is used to determine at least one group of extraction time periods based on at least one group of first spectrogram segments whose first score difference is lower than the first threshold and is continuous; wherein each group of extraction time periods includes a start time and an end time; the spectrogram extraction submodule includes a segment extraction unit, which is used to extract re-identified spectrogram segments from the spectrogram based on each group of extraction time periods.

[0108] Therefore, based on at least one group of first spectrogram segments whose first score difference is lower than the first threshold and is continuous, at least one group of extraction time periods is determined, and each group of extraction time periods includes a start time and an end time. On this basis, re-identified spectrogram segments are extracted from the spectrogram based on each group of extraction time periods. Therefore, re-identification can be performed based only on the first spectrogram segments whose languages ​​are difficult to distinguish during the first language recognition process, which is beneficial to minimizing the impact on recognition efficiency.

[0109] In some disclosed embodiments, the first recognition module 72 includes a language determination submodule for determining a preset language corresponding to the highest first correlation score as the target language involved in the first spectrum segment based on the first score difference of the first spectrum segment being not less than a first threshold.

[0110] Therefore, based on the fact that the first score difference of the first spectrogram segment is not lower than the first threshold, the preset language corresponding to the highest first correlation score is used as the target language involved in the first spectrogram segment. This can directly determine the target language involved in the first spectrogram segment, which is conducive to further improving the efficiency of language recognition on the basis of ensuring the accuracy of language recognition.

[0111] In some disclosed embodiments, the second recognition module 73 includes a second recognition submodule, which is used to obtain the second recognition results of each second sliding window performing language recognition window by window in the re-recognized spectrum segment; the second recognition results include second correlation scores of the re-recognized spectrum segment and several preset languages ​​respectively; the second recognition module 73 includes a result regularization submodule, which is used to regularize the second recognition results corresponding to the second sliding window for each second sliding window using the statistical recognition results corresponding to the second sliding window to obtain the regularized recognition results corresponding to the second sliding window; wherein the statistical recognition results are obtained by statistics based on the sample recognition results of the second sliding window performing language recognition window by window in several sample spectra; the second recognition module 73 includes a result fusion submodule, which is used to determine the target language involved in the re-recognized spectrum segment based on the final recognition result obtained by fusing the regularized recognition results corresponding to several second sliding windows.

[0112] Therefore, the second recognition result corresponding to the second sliding window is first regularized using the statistical recognition result corresponding to the second sliding window, so as to eliminate the inherent differences between the second recognition results obtained by performing language recognition window by window based on second sliding windows of different sizes as much as possible, which is beneficial to improving the accuracy of the final recognition result, and further can be beneficial to improving the accuracy of the target language involved in the re-identification of the spectral fragment.

[0113] In some disclosed embodiments, the second recognition submodule includes a sliding window selection unit, which is used to use each second sliding window as a current sliding window; the second recognition submodule includes a segment recognition unit, which is used to perform language recognition window by window in the re-recognized spectrum segment based on the current sliding window, and obtain a third recognition result of several second spectrum segments; wherein the third recognition result includes third correlation scores between the second spectrum segments and several preset languages ​​respectively; the second recognition submodule includes a score acquisition unit, which is used to obtain a second correlation score between the re-recognized spectrum segment and the corresponding preset language based on the third correlation score between each second spectrum segment and the preset language respectively.

[0114] Therefore, by taking each second sliding window as the current sliding window, and performing language recognition window by window in the re-identified spectrum segment based on the current sliding window, a third recognition result of several second spectrum segments is obtained, and the third recognition result includes the third correlation scores of the second spectrum segments and several preset languages, respectively. Therefore, based on the third correlation scores of each second spectrum segment and the preset language, the second correlation score of the re-identified spectrum segment and the corresponding preset language is obtained, and then the second recognition result corresponding to each second sliding window can be quickly obtained, which is beneficial to improving the overall recognition efficiency.

[0115] In some disclosed embodiments, the sample recognition result includes sample correlation scores of several sample spectral segments extracted by the second sliding window in the sample spectral and several preset languages ​​respectively, the statistical recognition result is based on the standard deviation of the sample correlation scores and the average correlation score of each preset language, and the regularized recognition result includes the regularized correlation scores of the re-recognized spectral segments and several preset languages ​​respectively; the result regularization submodule includes a first selection unit, which is used to take each preset language as the current language respectively; the result regularization submodule includes a score regularization unit, which is used to obtain a second score difference between the second correlation score corresponding to the current language and the average correlation score, and use the ratio of the second score difference to the standard deviation as the regularized correlation score of the re-recognized spectral segment and the current language.

[0116] Therefore, by taking each preset language as the current language respectively and summing up the regular correlation scores of the re-recognized spectral fragments and the current language in the regular recognition results corresponding to each second sliding window, the final correlation score of the re-recognized spectral fragments and the current language is obtained, and the preset language corresponding to the highest final correlation score is taken as the target language involved in the re-recognized spectral fragments. Therefore, the final correlation score can be obtained through simple mathematical operations, which is beneficial to improving recognition efficiency while ensuring the accuracy of language recognition.

[0117] In some disclosed embodiments, the regularized recognition results include regularized correlation scores of the re-recognized spectral segments with several preset languages, and the final recognition results include final correlation scores of the re-recognized spectral segments with several preset languages; the result fusion submodule includes a second selection unit, which is used to take each preset language as the current language respectively; the result fusion submodule includes a score summing unit, which is used to sum the regularized correlation scores of the re-recognized spectral segments with the current language in the regularized recognition results corresponding to each second sliding window to obtain the final correlation score of the re-recognized spectral segments with the current language; the result fusion submodule includes a language determination unit, which is used to take the preset language corresponding to the highest final correlation score as the target language involved in the re-recognized spectral segment.

[0118] Therefore, by taking each preset language as the current language respectively and summing up the regular correlation scores of the re-recognized spectral fragments and the current language in the regular recognition results corresponding to each second sliding window, the final correlation score of the re-recognized spectral fragments and the current language is obtained, and the preset language corresponding to the highest final correlation score is taken as the target language involved in the re-recognized spectral fragments. Therefore, the final correlation score can be obtained through simple mathematical operations, which is beneficial to improving recognition efficiency while ensuring the accuracy of language recognition.

[0119] In some disclosed embodiments, the speech recognition device 70 includes a first extraction module for performing feature extraction on a spectrogram to be identified to obtain a first feature graph and a language feature representation; the speech recognition device 70 includes a residual processing module for performing residual processing on the first feature graph to obtain a second feature graph; the speech recognition device 70 includes a second extraction module for performing feature extraction on the second feature graph to obtain a new first feature graph and a new language feature representation; the speech recognition device 70 includes a category prediction module for performing prediction processing based on the language feature representations extracted previously to obtain a language recognition result of the spectrogram to be identified; wherein the language recognition result includes language correlation scores of the spectrogram to be identified and several preset languages, respectively. In the case of language recognition based on the first sliding window, the spectrogram to be identified is the first spectrogram segment extracted window by window from the spectrogram by the first sliding window, and in the case of language recognition based on the second sliding window, the spectrogram to be identified is the second spectrogram segment extracted window by window from the re-identified spectrogram segment by the second sliding window.

[0120] Therefore, feature extraction is performed on the spectrogram to be identified to obtain a first feature graph and a language feature representation, and the residual of the first feature graph is processed to obtain a second feature graph, and feature extraction is performed on the second feature graph to obtain a new first feature graph and a new language feature representation. On this basis, prediction processing is performed based on the language feature representations extracted in the past to obtain the language recognition result of the spectrogram to be identified. Therefore, in the process of language recognition, language prediction can be performed by combining the language feature representations extracted multiple times, which is conducive to improving the accuracy of language recognition.

[0121] In some disclosed embodiments, the speech recognition device 70 includes a frequency detection module for detecting whether the number of executions of feature extraction is not higher than a second threshold; the speech recognition device 70 includes a loop execution module for, when the number of executions is not higher than the second threshold, combining the residual processing module and the second extraction module to re-execute the residual processing of the first feature map to obtain the step of obtaining the second feature map and subsequent steps.

[0122] Therefore, by first detecting whether the number of executions of feature extraction is not higher than the second threshold before the prediction processing, and re-executing the residual processing of the first feature map when the number of executions is not higher than the second threshold, the step of obtaining the second feature map and subsequent steps can help improve the accuracy of language recognition.

[0123] In some disclosed embodiments, the residual processing module includes an upsampling submodule for upsampling the first feature map to obtain an upsampled feature map; the residual processing module includes a feature fusion submodule for fusing the upsampled feature map and a third feature map extracted from the upsampled feature map to obtain a fused feature map; the residual processing module includes a downsampling module for downsampling the fused feature map to obtain a second feature map.

[0124] Therefore, residual processing is completed through upsampling, feature fusion, and downsampling, which can help improve the accuracy of language recognition on the basis of deepening the feature level.

[0125] In some disclosed embodiments, language recognition is performed by a language recognition model, and the language recognition model includes a feature extraction network, a residual processing network, and a category prediction network, the feature extraction network is used to perform feature extraction, the residual processing network is used to perform residual processing, and the category prediction network is used to perform prediction processing.

[0126] Therefore, language recognition is performed using a language recognition model, and feature extraction, residual processing, and category prediction are performed using a feature extraction network, a residual processing network, and a category prediction network, respectively, which can help improve language recognition efficiency.

[0127] In some disclosed embodiments, the size includes window length and window shift, and the window length and / or window shift of several second sliding windows are different; and / or, the size includes window length, and the window length of the second sliding window is not greater than the window length of the first sliding window; and / or, the size includes window shift, and the window shift of the second sliding window is not greater than the window shift of the first sliding window.

[0128] Therefore, setting the window lengths and / or window shifts of several second sliding windows to be different can improve the diversity of the second sliding windows, which is conducive to integrating multiple scales for language discrimination and improving the accuracy of language recognition; and setting the window length of the second sliding window to be no greater than the window length of the first sliding window, or setting the window shift of the second sliding window to be no greater than the window shift of the first sliding window, can more finely recognize the spectrum fragments in the re-identification through the second sliding window, which is conducive to improving the accuracy of language recognition.

[0129] See also Figure 8 , Figure 8 1 is a schematic diagram of a framework of an embodiment of an electronic device 80 of the present application. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, the memory 81 stores program instructions, and the processor 82 is used to execute the program instructions to implement the steps in any of the above-mentioned speech recognition method embodiments. Specifically, the electronic device 80 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0130] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps in any of the above-mentioned speech recognition method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0131] The above scheme, on the one hand, can perform language recognition twice at most during the entire recognition process, namely, the first language recognition based on the first sliding window and the second language recognition based on the second sliding window, so the double recognition can help improve the recognition accuracy, on the other hand, the use of several second sliding windows of different sizes in the second recognition process can fuse the multi-scale recognition results, which is conducive to further improving the recognition accuracy, and on the other hand, the size of the second sliding window used in the second recognition process is smaller than the size of the first sliding window, that is, the second recognition has a finer scale than the first recognition, which is conducive to further improving the recognition accuracy. Therefore, the accuracy of the language involved in the recognition of speech can be improved.

[0132] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 90 of the present application. The computer-readable storage medium 90 stores program instructions 91 that can be executed by a processor, and the program instructions 91 are used to implement the steps in any of the above-mentioned speech recognition method embodiments.

[0133] The above scheme, on the one hand, can perform language recognition twice at most during the entire recognition process, namely, the first language recognition based on the first sliding window and the second language recognition based on the second sliding window, so the double recognition can help improve the recognition accuracy, on the other hand, the use of several second sliding windows of different sizes in the second recognition process can fuse the multi-scale recognition results, which is conducive to further improving the recognition accuracy, and on the other hand, the size of the second sliding window used in the second recognition process is smaller than the size of the first sliding window, that is, the second recognition has a finer scale than the first recognition, which is conducive to further improving the recognition accuracy. Therefore, the accuracy of the language involved in the recognition of speech can be improved.

[0134] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0135] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A speech recognition method, characterized in that: include: Obtaining a spectrogram of the speech to be recognized; Extracting a re-recognized spectrum segment from the spectrogram based on a first recognition result of language recognition performed by the first sliding window on the spectrogram window by window; Based on the second recognition results of the language recognition performed by the plurality of second sliding windows on the re-recognized spectral segment window by window, the target language involved in the re-recognized spectral segment is analyzed and obtained; The sizes of the plurality of second sliding windows are different from each other, and the size of the second sliding window is smaller than the size of the first sliding window.

2. The method according to claim 1, characterized in that The first recognition result of performing language recognition window by window on the spectrogram based on the first sliding window, extracting the re-recognized spectrogram segment from the spectrogram, comprises: Based on the first sliding window, language recognition is performed window by window on the spectrogram to obtain first recognition results of a plurality of first spectrogram segments; wherein the first recognition results include first correlation scores of the first spectrogram segments and a plurality of preset languages ​​respectively; For each of the first spectrum segments, obtaining a first score difference between the highest first correlation score and the second highest first correlation score; The re-recognized spectrogram segment is extracted from the spectrogram based on the first score difference.

3. The method according to claim 2, characterized in that The step of extracting the re-recognized spectrum segment from the spectrogram based on the first score difference includes: Determine at least one group of extraction time periods based on at least one group of the first spectrum segments whose first score difference is lower than a first threshold and are continuous; wherein each group of the extraction time periods includes a start time and an end time; The re-recognized spectrogram segments are extracted from the spectrogram based on each group of the extraction time periods.

4. The method according to claim 2, characterized in that: The method further comprises: Based on the fact that the first score difference of the first spectrum segment is not less than a first threshold, a preset language corresponding to the highest first correlation score is used as a target language involved in the first spectrum segment.

5. The method according to claim 1, characterized in that The second recognition results based on the plurality of second sliding windows performing language recognition window by window on the re-recognized speech segment, analyzing and obtaining the target language involved in the re-recognized speech segment, include: Obtaining second recognition results of language recognition performed by each of the second sliding windows on the re-recognized speech spectrum segment window by window; the second recognition results include second correlation scores between the re-recognized speech spectrum segment and a plurality of preset languages; For each of the second sliding windows, the second recognition result corresponding to the second sliding window is regularized using the statistical recognition result corresponding to the second sliding window to obtain the regularized recognition result corresponding to the second sliding window; wherein the statistical recognition result is obtained by statistics based on the sample recognition results of the second sliding window performing language recognition window by window in a plurality of sample spectra; Based on a final recognition result obtained by fusing the regular recognition results corresponding to the plurality of second sliding windows, a target language involved in the re-recognized spectral segment is determined.

6. The method according to claim 5, characterized in that The step of obtaining the second recognition results of the second sliding windows performing language recognition window by window on the re-recognized speech spectrum segment comprises: Using each of the second sliding windows as a current sliding window; Based on the current sliding window, language recognition is performed window by window in the re-recognized speech spectrum segment to obtain a third recognition result of a plurality of second speech spectrum segments; wherein the third recognition result includes a third correlation score between the second speech spectrum segment and a plurality of preset languages ​​respectively; Based on the third correlation scores between each of the second speech spectrum segments and the preset language, the second correlation scores between the re-recognized speech spectrum segments and the corresponding preset language are obtained.

7. The method according to claim 5, characterized in that The sample recognition result includes sample correlation scores of the plurality of sample spectral segments extracted by the second sliding window from the sample spectral and the plurality of preset languages, the statistical recognition result is a standard deviation calculated based on the sample correlation scores and an average correlation score of each of the preset languages, and the regular recognition result includes the regular correlation scores of the re-recognized spectral segments and the plurality of preset languages; The step of regularizing the second recognition result corresponding to the second sliding window by using the statistical recognition result corresponding to the second sliding window to obtain the regularized recognition result corresponding to the second sliding window includes: respectively taking each of the preset languages ​​as the current language; A second score difference between the second correlation score corresponding to the current language and the average correlation score is obtained, and a ratio of the second score difference to the standard deviation is used as a regularized correlation score between the re-recognized speech spectrum segment and the current language.

8. The method according to claim 5, characterized in that The regularized recognition result includes regularized correlation scores of the re-recognized speech segments and the plurality of preset languages, and the final recognition result includes final correlation scores of the re-recognized speech segments and the plurality of preset languages; The determining of the target language involved in the re-recognized speech segment based on the final recognition result obtained by fusing the regularized recognition results corresponding to the plurality of second sliding windows comprises: respectively taking each of the preset languages ​​as the current language; Sum the regularized correlation scores of the re-recognized speech segment and the current language in the regularized recognition results corresponding to each of the second sliding windows to obtain a final correlation score of the re-recognized speech segment and the current language; The preset language corresponding to the highest final correlation score is used as the target language involved in the re-identified spectrum segment.

9. The method according to claim 1, characterized in that: The language identification step comprises: Perform feature extraction on the spectrogram to be recognized to obtain a first feature graph and a language feature representation; Processing the residual of the first feature map to obtain a second feature map; Performing feature extraction on the second feature graph to obtain a new first feature graph and a new language feature representation; Performing prediction processing based on the language feature representations extracted previously to obtain a language recognition result of the to-be-recognized spectrogram; Among them, the language recognition result includes the language-related scores of the to-be-recognized spectrogram and several preset languages ​​respectively. When language recognition is performed based on the first sliding window, the to-be-recognized spectrogram is the first spectrogram segment extracted window by window from the spectrogram by the first sliding window. When language recognition is performed based on the second sliding window, the to-be-recognized spectrogram is the second spectrogram segment extracted window by window from the re-recognized spectrogram segment by the second sliding window.

10. The method according to claim 9, characterized in that Before performing prediction processing based on the language feature representations extracted in succession to obtain the language recognition result of the to-be-recognized spectrogram, the method further includes: Detecting whether the number of executions of the feature extraction is not higher than a second threshold; When the number of executions is not higher than the second threshold, the step of processing the residual of the first feature map to obtain the second feature map and subsequent steps are re-executed.

11. The method according to claim 9, characterized in that The residual processing of the first feature map to obtain a second feature map includes: Upsampling the first feature map to obtain an upsampled feature map; Fusing the upsampled feature map with a third feature map extracted from the upsampled feature map to obtain a fused feature map; Down-sample the fused feature map to obtain the second feature map.

12. The method according to claim 9, characterized in that The language recognition is performed by a language recognition model, and the language recognition model includes a feature extraction network, a residual processing network and a category prediction network, the feature extraction network is used to perform the feature extraction, the residual processing network is used to perform the residual processing, and the category prediction network is used to perform the prediction processing.

13. The method according to claim 1, characterized in that The size includes window length and window displacement, and the window lengths and / or window displacements of the plurality of second sliding windows are different; And / or, the size includes a window length, and the window length of the second sliding window is not greater than the window length of the first sliding window; And / or, the size includes a window shift, and the window shift of the second sliding window is not greater than the window shift of the first sliding window.

14. A speech recognition device, characterized in that: include: A spectrogram acquisition module, used to acquire the spectrogram of the speech to be recognized; A first recognition module is used to extract and re-recognize a spectral segment from the spectrogram based on a first recognition result of language recognition performed by a first sliding window on the spectrogram window by window; A second recognition module is used for analyzing the second recognition results of language recognition on the re-recognized spectral segment window by window based on a plurality of second sliding windows, and obtaining the target language involved in the re-recognized spectral segment; The sizes of the plurality of second sliding windows are different from each other, and the size of the second sliding window is smaller than the size of the first sliding window.

15. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the speech recognition method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech recognition method according to any one of claims 1 to 13.

Images