Language recognition method and device, storage medium and electronic equipment

Abstract

The invention discloses a language recognition method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring to-be-recognized audio needing language recognition; inputting the to-be-recognized audio into an acoustic model of the speech recognition model, and extracting high-dimensional acoustic features of the to-be-recognized audio through the acoustic model; inputting the high-dimensional acoustic features into a phoneme conversion layer of a language recognition model, and converting the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer; inputting the phoneme sequence into a language feature extraction layer of a language recognition model, and extracting a first language feature of the audio to be recognized through the language feature extraction layer; and inputting the first language feature into a language recognition layer of a language recognition model, and performing language recognition through the language recognition layer to obtain a language recognition result of the to-be-recognized audio. According to the language recognition method, the to-be-recognized audio of multiple languages is recognized by using one language recognition model, and the occupation of computing resources can be reduced.

Description

Technical Field

[0001] The present application relates to the field of speech technology, and in particular to a language recognition method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of speech recognition technology, the current speech recognition technology has been widely used in various scenarios, such as voice input, voice search, voice translation, smart home, etc. At the same time, there are more and more users using speech recognition technology. These users may come from different regions and use different languages. Therefore, for the audio to be recognized input by the user, it is necessary to identify the language category of the audio to be recognized. However, in the related technology, if it is necessary to recognize audio to be recognized in multiple languages, it is necessary to switch to use multiple language recognition models, and the computing resources occupied are relatively large. Summary of the invention

[0003] Embodiments of the present application provide a language recognition method, device, storage medium and electronic device. The language recognition method can use the same language recognition model to recognize audio to be recognized in multiple languages ​​and can reduce the occupation of computing resources.

[0004] In a first aspect, an embodiment of the present application provides a language recognition method, the method comprising:

[0005] Obtain the audio to be recognized that needs to be recognized in the language;

[0006] Inputting the audio to be recognized into the acoustic model of the speech recognition model, and extracting high-dimensional acoustic features of the audio to be recognized through the acoustic model;

[0007] Inputting the high-dimensional acoustic features into a phoneme conversion layer of a language recognition model, and converting the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer;

[0008] Inputting the phoneme sequence into the language feature extraction layer of the language recognition model, and extracting the first language feature of the audio to be recognized through the language feature extraction layer;

[0009] The first language feature is input into the language recognition layer of the language recognition model, and language recognition is performed through the language recognition layer to obtain a language recognition result of the audio to be recognized.

[0010] In a second aspect, an embodiment of the present application provides a language recognition device, the device comprising:

[0011] An acquisition module is used to acquire the audio to be recognized that needs to be recognized in the language;

[0012] A first extraction module, used for inputting the audio to be recognized into an acoustic model of a speech recognition model, and extracting high-dimensional acoustic features of the audio to be recognized through the acoustic model;

[0013] A conversion module, used for inputting the high-dimensional acoustic features into a phoneme conversion layer of a language recognition model, and converting the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer;

[0014] A second extraction module, configured to input the phoneme sequence into a language feature extraction layer of the language recognition model, and extract a first language feature of the audio to be recognized through the language feature extraction layer;

[0015] The recognition module is used to input the first language feature into the language recognition layer of the language recognition model, perform language recognition through the language recognition layer, and obtain the language recognition result of the audio to be recognized.

[0016] The storage medium provided in the present application stores a computer program thereon, and when the computer program is loaded by a processor, the steps in the pronunciation skill detection method provided in the present application are executed.

[0017] The electronic device provided in the present application includes a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the steps in the pronunciation skill detection method provided in the present application by loading the computer program.

[0018] In the embodiment of the present application, the language recognition method uses a language recognition model to perform language recognition on the audio to be recognized. The language recognition model can reuse the acoustic model in the speech recognition model, and use the high-dimensional acoustic features extracted by the acoustic model from the audio to be recognized as the input of the language recognition model. There is no need to separately set up a layer with similar functions to the acoustic model of the speech recognition model to extract high-dimensional acoustic features, nor is there any need to train the layer, thereby reducing the occupation of computing resources and avoiding cost waste. More computing resources can be reserved for training other layers in the language recognition model to improve the performance of the language recognition model and improve the accuracy of language recognition. Moreover, in the embodiment of the present application, the high-dimensional acoustic features are converted into a phoneme sequence through the phoneme conversion layer for subsequent language recognition, instead of using text information for subsequent language recognition. It can be understood that no matter which language, even if it is a small language to be recognized audio, the high-dimensional acoustic features can be extracted through the acoustic model and converted into a phoneme sequence through the phoneme conversion layer. Therefore, even if some small languages ​​have fewer training sample resources, the phoneme conversion layer can be trained by obtaining training samples of large languages ​​such as Chinese and English, so that the trained phoneme conversion layer can also be applied to small languages. In addition, the language feature extraction layer and the language recognition layer of the language recognition model can also be first trained with training samples of large languages ​​such as Chinese and English to train the language feature extraction layer and the language recognition layer, so that the language feature extraction layer and the language recognition layer can be used for large languages. When it is necessary to perform language recognition on the audio to be recognized of the small language, the training samples of the small language can be input into the language feature extraction layer and the language recognition layer after the large language training for training with a small amount of data, and a language recognition effect close to that of the large language can be obtained. Therefore, the embodiment of the present application does not need to train a language recognition model for each language separately. One language recognition model can be used to recognize multiple languages. When recognizing different languages, there is no need to switch the corresponding language recognition model. In particular, for small languages, there is no need to use training samples of the small language to train a language recognition model separately. With fewer training sample resources, a higher language recognition accuracy can still be obtained through the language recognition model of the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation modes of the present application in conjunction with the accompanying drawings.

[0020] Figure 1 A first flow chart of a language recognition method provided in an embodiment of the present application.

[0021] Figure 2 This is a first structural block diagram of the language recognition model provided in the embodiment of the present application.

[0022] Figure 3This is a second flow chart of the language recognition method provided in the embodiment of the present application.

[0023] Figure 4 This is a first structural block diagram of the language recognition model provided in the embodiment of the present application.

[0024] Figure 5 A schematic diagram of the training process of the language recognition model provided in the embodiment of the present application.

[0025] Figure 6 A schematic diagram of the training of the phoneme conversion layer of the language recognition model provided in an embodiment of the present application.

[0026] Figure 7 A training diagram of the language feature extraction layer, feature pooling layer and language recognition layer of the language recognition model provided in an embodiment of the present application.

[0027] Figure 8 A training diagram of the phoneme conversion layer, language feature extraction layer, feature pooling layer and language recognition layer of the language recognition model provided in an embodiment of the present application.

[0028] Fig. 9 A first structural diagram of a language recognition device provided in an embodiment of the present application.

[0029] Fig.10 A second structural diagram of the language recognition device provided in an embodiment of the present application.

[0030] Fig.11 A first structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0031] Fig.12 A second structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] It should be noted that the principles of the present application are illustrated by implementing in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be construed as limiting other specific embodiments not described in detail herein. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0033] The relational terms such as first and second involved in the following embodiments of the present application are only used to distinguish one object or operation from another object or operation, and are not used to limit the actual sequential relationship between these objects or operations. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0034] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in computer vision, natural language processing and other fields.

[0036] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding training data sets, network models that implement different functions can be trained. For example, a deep learning network for gender classification can be trained based on one training data set, and a deep learning network for image optimization can be trained based on another training data set.

[0037] The embodiment of the present application introduces deep learning into the language recognition of multi-language audio to be recognized, and accordingly provides a language recognition method, a language recognition device, a storage medium and an electronic device. The language recognition method can be executed by an electronic device, wherein the electronic device can be any device equipped with a processor and having processing capabilities, such as a mobile electronic device with a processor such as a smart phone, a tablet computer, a PDA, a laptop computer, or a fixed electronic device with a processor such as a desktop computer, a television, a server, etc. Figure 1 , Figure 1 A first flow chart of a language recognition method provided in an embodiment of the present application.

[0038] In 101, audio to be recognized that needs to be language recognized is obtained.

[0039] It is understandable that users may come from different regions of the world and use different languages. Therefore, the audio to be recognized obtained by the embodiment of the present application may be a piece of audio to be recognized in a certain language, for example, it may be a piece of Chinese speech or a piece of foreign language speech, such as English, French, Japanese, etc.; it may also be a piece of Mandarin speech or a piece of dialect speech, such as Sichuanese, Minnan dialect, Northeastern dialect, Cantonese, etc. The audio to be recognized may be collected by a microphone or other equipment, and the present application does not specifically limit the method of obtaining the audio to be recognized.

[0040] In 102, the audio to be recognized is input into the acoustic model of the speech recognition model, and the high-dimensional acoustic features of the audio to be recognized are extracted through the acoustic model.

[0041] Please refer to Figure 2 , Figure 2 The first structural block diagram of the language recognition model provided in the embodiment of the present application. The language recognition method in the embodiment of the present application performs language recognition on the audio to be recognized based on the language recognition model, and the language recognition model is built on the basis of the acoustic model in the speech recognition model, that is, the embodiment of the present application can reuse the acoustic model in the speech recognition model, and the speech recognition model can be, for example, a recurrent neural network converter (Recurrent Neural Network Transducer, RNN-T), a coding structure based on an attention mechanism, etc.

[0042] The acoustic model can extract features of one-dimensional audio to be identified to obtain high-dimensional acoustic features, such as two-dimensional high-dimensional acoustic features, wherein the one-dimensional audio to be identified is, for example, audio to be identified in the time dimension, and the two-dimensional high-dimensional acoustic features include, for example, the time dimension and the frequency dimension, wherein the high-dimensional acoustic features can be, for example, short-time Fourier transform features, thereby obtaining the frequency components of the audio to be identified in different time frames, and obtaining the high-dimensional acoustic features of the audio to be identified in the time-frequency (T*F) two-dimensional plane.

[0043] In the related art, when a speech recognition model is used for speech recognition, if the audio to be recognized input by the user may be in multiple languages, then the speech recognition model will usually also include a language recognition model, which serves as a sub-module to perform language recognition on the audio to be recognized, wherein the language recognition is independent of the construction of the speech recognition model, that is, the speech recognition model has an acoustic model, and the language recognition model may also need to separately set up a layer with similar functions to the acoustic model of the speech recognition model, and perform corresponding training, thus resulting in a waste of computing resources and costs. The embodiment of the present application extracts high-dimensional acoustic features of the audio to be recognized by reusing the acoustic model in the speech recognition model. The extracted high-dimensional acoustic features can be used for both the speech recognition model and the language recognition model. That is, the high-dimensional acoustic features extracted by the acoustic model in the speech recognition model can be used as the input of the language recognition model for subsequent language recognition. There is no need to separately set up a layer with similar functions to the acoustic model of the speech recognition model to extract high-dimensional acoustic features, nor is there any need to train the layer, thereby reducing the occupation of computing resources and avoiding cost waste. More resources can be reserved for training other layers in the language recognition model to improve the performance of the language recognition model and the accuracy of language recognition.

[0044] In 103, the high-dimensional acoustic features are input into the phoneme conversion layer of the language recognition model, and the high-dimensional acoustic features are converted into a phoneme sequence of the audio to be recognized through the phoneme conversion layer.

[0045] In the embodiment of the present application, high-dimensional acoustic features are input into the phoneme conversion layer of the language recognition model, and the high-dimensional acoustic features are converted into a phoneme sequence of the audio to be recognized through the phoneme conversion layer. It can be understood that phonemes are the smallest speech units divided according to the natural properties of speech. From the perspective of acoustic properties, phonemes are the smallest speech units divided from the perspective of sound quality. A pronunciation action forms a phoneme. Phonemes can be, for example, international phonetic symbols, Roman phonetic symbols, pinyin, etc. The phoneme sequence can be understood as the arrangement of phonemes in the time sequence of the audio to be recognized. The phoneme sequence is based on phonemes. For example, taking "hello" as an example, its corresponding phoneme sequence is "ni", "hao". It can be understood that the embodiment of the present application converts the audio to be recognized "hello" into a phoneme sequence "ni", "hao", instead of the words "you" and "good" by setting a phoneme conversion layer. In the related art, the acoustic model extracts features from the audio to be recognized, and the obtained high-dimensional acoustic features can be used to convert the actual content information of the audio to be recognized, such as the audio to be recognized "hello" can be converted into "you" and "good", that is, the high-dimensional acoustic features extracted by the acoustic model only retain the common content information, and delete some other features, such as accents, tones and other features. It can be understood that for users in different regions, even if the content information in the audio to be recognized is the same, there will be different accents, tones and other features. In the embodiment of the present application, for example, continue to take the audio to be recognized "hello" as an example, if it is Mandarin, "hello" is converted into the phoneme sequence "ni", "hao", if it is Sichuan dialect, due to different pronunciation habits, the phoneme sequence converted from "hello" can be: "li", "hao", so different phoneme sequences can be used to distinguish the accent information of different languages. Therefore, the embodiment of the present application outputs a more fine-grained phoneme sequence through the phoneme conversion layer, which is more conducive to language recognition.

[0046] In 104, the phoneme sequence is input into the language feature extraction layer of the language recognition model, and the first language feature of the audio to be recognized is extracted through the language feature extraction layer.

[0047] Further, in the embodiment of the present application, the phoneme sequence is input into the language feature extraction layer, and the language feature extraction layer extracts the first language feature of the audio to be recognized. It can be understood that the phonemes can be universal for different languages, that is, the phonemes can be used to mark the audio to be recognized in different languages, and the first language feature can be understood as the correlation feature between the phoneme sequence and the language. For example, taking the phoneme as pinyin as an example, for example, inputting "ni" into the language feature extraction layer, the language feature extraction layer extracts that "n" and "i" in Mandarin are often combined together to form "ni", and for another example, inputting "li" into the language feature extraction layer, the language feature extraction layer can extract that "l" and "i" in Sichuan dialect are often combined together to form "li", and it can also be understood that Mandarin and Sichuan dialect may have different phoneme combinations for the same word due to different accents. The first language feature obtained by the language feature extraction layer can extract the first language features related to the accent of different languages, that is, the first language feature obtained by the language feature extraction layer can reflect the pronunciation habits of different languages ​​to a certain extent.

[0048] For another example, let's continue to use phonemes as pinyin. For example, if the input audio to be recognized is "hello", then the phoneme sequence obtained by the phoneme conversion layer is, for example, "halou". "Halou" is input into the language feature extraction layer. The language feature extraction layer extracts that "h", "a", "l", "o", and "u" in English are often combined together to form "halou". It can also be understood that the difference in phoneme sequences between Chinese and English is due to the different linguistics. Different languages ​​correspond to some specific or idiomatic phoneme sequences, or a certain phoneme sequence has a higher probability of appearing in a certain language. Therefore, through the association relationship obtained by the language feature extraction layer, the first language features related to different languages ​​and linguistics such as semantics can be extracted. Regardless of whether the phoneme sequences are different due to pronunciation habits or semantics, different languages ​​correspond to some idiomatic or specific phoneme sequences. Therefore, the extracted first language features can be used to characterize the use probability and occurrence probability of phoneme sequences in different languages.

[0049] In 105, the first language feature is input into the language recognition layer of the language recognition model, and language recognition is performed through the language recognition layer to obtain a language recognition result of the audio to be recognized.

[0050] In the embodiment of the present application, the first language feature is input into the language recognition layer of the language recognition model for language recognition. Since the first language feature can be used to characterize the probability of using the phoneme sequence of the audio to be recognized in different languages, that is, it can identify the probability that the audio to be recognized belongs to different languages. For example, in the above example, the phoneme sequence "ni" may be more likely to be Mandarin, the phoneme sequence "li" may be more likely to be Sichuan dialect, and the phoneme sequence "halou" may be more likely to be English, thereby obtaining the language recognition result of the audio to be recognized.

[0051] It should be noted that in the embodiment of the present application, the high-dimensional acoustic features are converted into phoneme sequences through a phoneme conversion layer for subsequent language recognition, rather than directly using words, that is, text information is not used for subsequent language recognition. It can be understood that different languages, such as Chinese and English, are not interchangeable. Therefore, the overall modeling granularity of language recognition through text is coarse and not easy to identify the language. For example, if the language is not converted into a phoneme sequence through a phoneme conversion layer, but text information is used for language recognition, then for each small language, there must be enough training samples corresponding to the small language before the text information can be used for subsequent recognition of the small language. The collection and annotation of small language training samples are relatively difficult, so it takes up a lot of computing resources and also causes higher costs.

[0052] However, no matter which language, even for minority languages, such as Cambodian, Vietnamese, Korean, Burmese, Urdu, Lao, Arabic, Persian and Hungarian, phonemes are universal, that is, no matter which language the audio to be recognized is, it can be converted into a phoneme sequence through the phoneme conversion layer after extracting high-dimensional acoustic features through the acoustic model. Therefore, even if there are fewer training sample resources for some minority languages, the phoneme conversion layer can be trained by obtaining training samples of large languages, such as Chinese and English training samples, so that the trained phoneme conversion layer can also be applied to minority languages. For example, taking phonemes as pinyin as an example, the phoneme conversion layer can be trained by Chinese training samples. When the phoneme conversion layer receives audio to be recognized in other languages, it can also convert the audio to be recognized in other languages ​​into a combination of pinyin, that is, into a phoneme sequence, so as to realize the sharing of phoneme conversion layers between different languages. In addition, the language feature extraction layer and the language recognition layer can also first use the training samples of the major language, such as the training samples of Chinese and English, to train the language feature extraction layer, so that the language feature extraction layer and the language recognition layer can be used for the major language. When it is necessary to perform language recognition on the audio to be recognized in the minor language, the training samples of the minor language can be input into the language feature extraction layer and the language recognition layer after the training of the major language for training with a small amount of data, so as to obtain the same language recognition effect as the major language. For example, in the embodiment of the present application, the language feature extraction layer and the language recognition layer trained with the major language are equivalent to having trained about 90% of the parameters in the language feature extraction layer and the language recognition layer. For the minor language, it is only necessary to train the language feature extraction layer and the language recognition layer for the last 10% or so, and adjust the relevant parameters of the language feature extraction layer and the language recognition layer accordingly, without the need to independently train each minor language with a large amount of data.

[0053] Therefore, the embodiments of the present application do not need to train a language recognition model for each language separately. One language recognition model can be used to recognize multiple languages. When recognizing different languages, there is no need to switch the corresponding language recognition model. In particular, for small languages, there is no need to use training samples of the small language to train a language recognition model separately. Fewer training sample resources can be used, and a higher language recognition accuracy can still be obtained through the language recognition model of the embodiments of the present application. It can be understood that the language recognition results obtained can be used as language / ethnicity / regional information for downstream natural language processing tasks to assist in making responses that are more suitable for the user group.

[0054] See also Figure 3 , Figure 3 This is a second flow chart of the language recognition method provided in the embodiment of the present application.

[0055] In 201, audio to be recognized that needs to be language recognized is obtained.

[0056] It is understandable that users may come from different regions of the world and use different languages. Therefore, the audio to be recognized obtained by the embodiment of the present application may be a piece of audio to be recognized in a certain language, for example, it may be a piece of Chinese speech or a piece of foreign language speech, such as English, French, Japanese, etc.; it may also be a piece of Mandarin speech or a piece of dialect speech, such as Sichuanese, Minnan dialect, Northeastern dialect, Cantonese, etc. The audio to be recognized may be collected by a microphone or other equipment, and the present application does not specifically limit the method of obtaining the audio to be recognized.

[0057] In 202, the audio to be recognized is input into the acoustic model of the speech recognition model, and the high-dimensional acoustic features of the audio to be recognized are extracted through the acoustic model.

[0058] Please refer to Figure 4 , Figure 4 A second structural block diagram of a language recognition model provided in an embodiment of the present application. The embodiment of the present application can obtain the acoustic model of the speech recognition model, and it can also be understood that the embodiment of the present application can reuse the acoustic model in the speech recognition model, that is, the language recognition method in the embodiment of the present application performs language recognition on the audio to be recognized based on the language recognition model, and the language recognition model is built on the basis of the acoustic model in the speech recognition model. The speech recognition model can be, for example, a recurrent neural network converter (RNN-T).

[0059] The acoustic model can extract features of one-dimensional audio to be identified to obtain high-dimensional acoustic features, for example, it can be a two-dimensional high-dimensional acoustic feature, wherein the one-dimensional audio to be identified is, for example, audio to be identified in the time dimension, and the two-dimensional high-dimensional acoustic feature includes, for example, the time dimension and the frequency dimension, wherein the high-dimensional acoustic feature can be, for example, a short-time Fourier transform feature, thereby obtaining the frequency components of the audio to be identified in different time frames, and obtaining the high-dimensional acoustic features of the audio to be identified in the time-frequency (T*F) two-dimensional plane.

[0060] In the related art, when a speech recognition model is used for speech recognition, if the audio to be recognized input by the user may be in multiple languages, then the speech recognition model will usually also include a language recognition model, which serves as a sub-module to perform language recognition on the audio to be recognized, wherein the language recognition is independent of the construction of the speech recognition model, that is, the speech recognition model has an acoustic model, and the language recognition model may also need to separately set up a layer with similar functions to the acoustic model of the speech recognition model, and perform corresponding training, thus resulting in a waste of computing resources and costs. The embodiment of the present application extracts high-dimensional acoustic features of the audio to be recognized by reusing the acoustic model in the speech recognition model. The extracted high-dimensional acoustic features can be used for both the speech recognition model and the language recognition model. That is, the high-dimensional acoustic features extracted by the acoustic model in the speech recognition model can be used as the input of the language recognition model for subsequent language recognition. There is no need to separately set up a layer with similar functions to the acoustic model of the speech recognition model to extract high-dimensional acoustic features, nor is there any need to train the layer, thereby reducing the occupation of computing resources and avoiding cost waste. More resources can be reserved for training other layers in the language recognition model to improve the performance of the language recognition model and the accuracy of language recognition.

[0061] In 203, the high-dimensional acoustic features are input into the phoneme conversion layer of the language recognition model, and the high-dimensional acoustic features are converted into a phoneme sequence of the audio to be recognized through the phoneme conversion layer.

[0062] In the embodiment of the present application, high-dimensional acoustic features are input into the phoneme conversion layer of the language recognition model, and the high-dimensional acoustic features are converted into a phoneme sequence of the audio to be recognized through the phoneme conversion layer. It can be understood that phonemes are the smallest speech units divided according to the natural properties of speech. From the perspective of acoustic properties, phonemes are the smallest speech units divided from the perspective of sound quality. A pronunciation action forms a phoneme. Phonemes can be, for example, international phonetic symbols, Roman phonetic symbols, pinyin, etc. The phoneme sequence can be understood as the arrangement of phonemes in the time sequence of the audio to be recognized. The phoneme sequence is based on phonemes. For example, taking "hello" as an example, its corresponding phoneme sequence is "ni", "hao". It can be understood that the embodiment of the present application converts the audio to be recognized "hello" into a phoneme sequence "ni", "hao", instead of the words "you" and "good" by setting a phoneme conversion layer.

[0063] In the related art, the acoustic model extracts features from the audio to be recognized, and the obtained high-dimensional acoustic features can be used to convert the actual content information of the audio to be recognized, such as the audio to be recognized "hello" can be converted into "you" and "good", that is, the high-dimensional acoustic features extracted by the acoustic model only retain the common content information, and delete some other features, such as accents, tones and other features. It can be understood that for users in different regions, even if the content information in the audio to be recognized is the same, there will be different accents, tones and other features. In the embodiment of the present application, for example, continue to take the audio to be recognized "hello" as an example, if it is Mandarin, "hello" is converted into the phoneme sequence "ni", "hao", if it is Sichuan dialect, due to different pronunciation habits, the phoneme sequence converted from "hello" can be: "li", "hao", so different phoneme sequences can be used to distinguish the accent information of different languages. Therefore, the embodiment of the present application outputs a more fine-grained phoneme sequence through the phoneme conversion layer, which is more conducive to language recognition.

[0064] In 204, the phoneme sequence is input into the language feature extraction layer of the language recognition model, and the first language feature of the audio to be recognized is extracted through the language feature extraction layer.

[0065] Further, in the embodiment of the present application, the phoneme sequence is input into the language feature extraction layer, and the language feature extraction layer extracts the first language feature of the audio to be recognized. It can be understood that the phonemes can be universal for different languages, that is, the phonemes can be used to mark the audio to be recognized in different languages, and the first language feature can be understood as the correlation feature between the phoneme sequence and the language. For example, taking the phoneme as pinyin as an example, for example, inputting "ni" into the language feature extraction layer, the language feature extraction layer extracts that "n" and "i" in Mandarin are often combined together to form "ni", and for another example, inputting "li" into the language feature extraction layer, the language feature extraction layer can extract that "l" and "i" in Sichuan dialect are often combined together to form "li", and it can also be understood that Mandarin and Sichuan dialect may have different phoneme combinations for the same word due to different accents. The first language feature obtained by the language feature extraction layer can extract the first language features related to the accent of different languages, that is, the first language feature obtained by the language feature extraction layer can reflect the pronunciation habits of different languages ​​to a certain extent.

[0066] For another example, let's continue to use phonemes as pinyin. For example, if the input audio to be recognized is "hello", then the phoneme sequence obtained by the phoneme conversion layer is, for example, "halou". "Halou" is input into the language feature extraction layer. The language feature extraction layer extracts that "h", "a", "l", "o", and "u" in English are often combined together to form "halou". It can also be understood that the difference in phoneme sequences between Chinese and English is due to the different linguistics. Different languages ​​correspond to some specific or idiomatic phoneme sequences, or a certain phoneme sequence has a higher probability of appearing in a certain language. Therefore, through the association relationship obtained by the language feature extraction layer, the first language features related to different languages ​​and linguistics such as semantics can be extracted. Regardless of whether the phoneme sequences are different due to pronunciation habits or semantics, different languages ​​correspond to some idiomatic or specific phoneme sequences. Therefore, the extracted first language features can be used to characterize the use probability and occurrence probability of phoneme sequences in different languages.

[0067] In 205 , the first language feature is input into the feature pooling layer of the language recognition model to perform pooling on the first language feature in the time domain and reduce the dimension of the first language feature to obtain a one-dimensional second language feature.

[0068] In the embodiment of the present application, the acoustic model is used to extract the features of the one-dimensional audio to be identified, and a high-dimensional acoustic feature is obtained, such as a two-dimensional high-dimensional acoustic feature. For the phoneme conversion layer, the input is a two-dimensional high-dimensional acoustic feature, and the output is also a two-dimensional phoneme sequence; for the language feature extraction layer, the input is a two-dimensional phoneme sequence, and the output is also a two-dimensional first language feature. In the embodiment of the present application, the two-dimensional first language feature is input into the feature pooling layer to pool the first language feature in the time domain, reduce the dimension of the first language feature, and obtain a one-dimensional second language feature. It can be understood that the first language feature (T*F) of different durations and frequencies can be converted into a one-dimensional second language feature (1*F) through the feature pooling layer, so that no matter how long the duration of the audio to be identified is, such as 10 seconds, 20 seconds, 40 seconds, the second language feature as the input parameter for judging language recognition has nothing to do with the duration, but only the second language feature of the fixed duration is related, so as to facilitate the subsequent unified language recognition work of the audio to be identified of different durations in the actual scene. Among them, the pooling layer may include at least one of a global average pooling layer, a maximum pooling layer and a minimum pooling layer.

[0069] At 206, the second language feature is input into the language recognition layer, and language recognition is performed through the language recognition layer. The language recognition layer outputs the language distribution probability corresponding to the audio to be recognized, and generates a language recognition result based on the language distribution probability.

[0070] The first language feature is extracted from the phoneme sequence through the language feature extraction layer, and the first language feature is reduced in dimension through the feature pooling layer to obtain a one-dimensional second language feature. The second language feature can be used to reflect the probability that the phoneme sequence belongs to different languages. Moreover, the second language feature, as the input of the language recognition layer, has nothing to do with different durations, which facilitates the subsequent unified language recognition work of the language recognition layer for the audio to be recognized of different durations in actual scenarios. This can avoid the problem of reduced language recognition accuracy due to the non-fixed duration of the input parameters of the language recognition layer, thereby improving the reliability and accuracy of language recognition.

[0071] The one-dimensional second language feature is input into the language recognition layer, and the language recognition layer outputs the language distribution probability corresponding to the audio to be recognized, and generates a language recognition result based on the language distribution probability. Exemplarily, no matter how many second language features of the audio to be recognized are, the language recognition layer can output feature values ​​corresponding to the number of languages ​​according to the number of languages ​​that can be recognized by the language recognition model, and the feature values ​​can be language distribution probabilities. The current language recognition method can be used to recognize three languages: Chinese, English, and French. Then the language recognition layer can output three feature values, and the three feature value distributions indicate that the audio to be recognized is respectively the distribution probability of these three languages. The larger the probability value of the language, the greater the possibility that the audio to be recognized is the language, and the smaller the probability value of the language, the smaller the possibility that the audio to be recognized is the language, so that the language recognition result can be determined based on the language distribution probability. It should be noted that the feature value can also be other values, and other values ​​can also be used to characterize the language distribution probability.

[0072] The language recognition method of the embodiment of the present application performs language recognition on the audio to be recognized based on a language recognition model. The language recognition model includes a phoneme conversion layer, a language feature extraction layer, a feature pooling layer and a language recognition layer. In order for the language recognition model to accurately perform language recognition, the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer in the language recognition model can be obtained through deep learning training based on the basic phoneme conversion layer, the basic language feature extraction layer, the basic feature pooling layer and the basic language recognition layer. Please refer to Figure 5 , Figure 5 A schematic diagram of the training process of the language recognition model provided in the embodiment of the present application.

[0073] In 301, a first training sample is obtained, where the first training sample includes a first sample audio to be recognized and a first sample phoneme sequence corresponding to the first sample audio to be recognized;

[0074] In 302, a basic phoneme conversion layer is trained according to a first sample audio to be recognized and a first sample phoneme sequence to obtain a phoneme conversion layer.

[0075] The acoustic model of the embodiment of the present application can reuse the acoustic model in the speech recognition model. When training the language recognition model, the embodiment of the present application does not need to train the acoustic model, but can directly use the acoustic model of the speech recognition model, which will not affect the performance of the speech recognition model. The language recognition model of the embodiment of the present application includes a phoneme conversion layer, which is used to convert the high-dimensional acoustic features extracted by the acoustic model into a phoneme sequence of the audio to be recognized. In order to enable the phoneme conversion layer to perform accurate conversion, the phoneme conversion layer of the language recognition model can be obtained through training. Please refer to Figure 6 , Figure 6 A training diagram of the phoneme conversion layer of the language recognition model provided in the embodiment of the present application. Specifically, a first training sample is obtained, the first training sample includes a first sample audio to be recognized and a first sample phoneme sequence corresponding to the first sample audio to be recognized, the first sample audio to be recognized is input into the acoustic model, and the acoustic model outputs a first high-dimensional acoustic feature that matches the first sample audio to be recognized; the first high-dimensional acoustic feature is input into the basic phoneme conversion layer, and the first high-dimensional acoustic feature is converted into a first phoneme sequence of the first sample audio to be recognized through the basic phoneme conversion layer. Among them, the basic phoneme conversion layer can, for example, adopt a single-layer time series model structure, such as a recurrent neural network RNN, a self-attention mechanism layer, etc.

[0076] Based on the difference information between the first phoneme sequence and the first sample phoneme sequence, a first loss value is obtained, and based on the first loss value, the basic phoneme conversion layer is trained to obtain the phoneme conversion layer. The first phoneme sequence and the first sample phoneme sequence can be input into the first loss function, the first loss value is determined by the first loss function, and the gradient is returned to the basic phoneme conversion layer to adjust the parameters of the basic phoneme conversion layer to obtain the trained phoneme conversion layer. The first loss function can be, for example, a connectionist temporal classification loss function (CTC).

[0077] The embodiment of the present application can convert high-dimensional acoustic features into phoneme sequences through a trained phoneme conversion layer for subsequent language recognition. No matter which language, even the audio to be recognized in a minority language, can be accurately converted into a phoneme sequence through a trained phoneme conversion layer after extracting high-dimensional acoustic features through an acoustic model. Some minority languages ​​have fewer training sample resources, and the phoneme conversion layer can also be trained by obtaining training samples of large languages, such as Chinese and English training samples, so that the trained phoneme conversion layer can also be applied to minority languages, and the sharing of phoneme conversion layers between different languages ​​can be achieved.

[0078] In 303, a second training sample is obtained, where the second training sample includes the first sample high-dimensional acoustic features of the second sample audio to be recognized and the first sample language recognition result;

[0079] In 304, the phoneme conversion layer is frozen, and the basic language feature extraction layer, the basic feature pooling layer, and the basic language recognition layer are trained according to the high-dimensional acoustic features of the first sample and the language recognition result of the first sample to obtain the language feature extraction layer, the feature pooling layer, and the language recognition layer.

[0080] The language recognition model includes four neural network layers, namely: phoneme conversion layer, language feature extraction layer, feature pooling layer and language recognition layer. The trained phoneme conversion layer can output a relatively accurate phoneme sequence based on the high-dimensional acoustic features output by the acoustic model. Therefore, the phoneme conversion layer can be frozen, that is, the parameters of the phoneme conversion layer are not trained. In order for the language recognition model to accurately identify the language recognition results, the language feature extraction layer, feature pooling layer and language recognition layer can also be obtained through training. Please refer to Figure 7 , Figure 7 A training diagram of the language feature extraction layer, feature pooling layer and language recognition layer of the language recognition model provided in an embodiment of the present application.

[0081] Specifically, a second training sample is obtained, and the second training sample includes the first sample high-dimensional acoustic features of the second sample audio to be recognized and the first sample language recognition result. Exemplarily, the second sample audio to be recognized can be obtained first; the second sample audio to be recognized is input into a reused acoustic model, and the acoustic model outputs the first sample high-dimensional acoustic features of the second sample audio to be recognized, and then the first sample high-dimensional acoustic features of the second sample audio to be recognized and the first sample language recognition result are used as the second training sample. It should be clear that the acoustic model does not need to be trained. The embodiment of the present application can use the first sample high-dimensional acoustic features output by the acoustic model to train the language recognition model.

[0082] After obtaining the second training sample, the high-dimensional acoustic features of the first sample are input into the phoneme conversion layer, and the high-dimensional acoustic features of the first sample are converted into a second phoneme sequence of the second sample audio to be recognized through the phoneme conversion layer; the second phoneme sequence is input into the basic language feature extraction layer, and the third language features of the second sample audio to be recognized are extracted through the basic language feature extraction layer; the third language features are input into the basic feature pooling layer, and the basic feature pooling layer reduces the dimension of the third language features to obtain a one-dimensional fourth language feature; the fourth language features are input into the basic language recognition layer, and the language recognition is performed through the basic language recognition layer to obtain a first predicted language recognition result of the second sample audio to be recognized; based on the difference information between the first predicted language recognition result and the first sample language recognition result, a second loss value is obtained, and the basic language feature extraction layer, the basic feature pooling layer and the basic language recognition layer are trained based on the second loss value to obtain a language feature extraction layer, a feature pooling layer and a language recognition layer. The first predicted language recognition result and the first sample language recognition result can be input into the second loss function, and the second loss value is determined by the second loss function, and the gradient is transmitted back to the basic language feature extraction layer, the basic feature pooling layer and the basic language recognition layer to adjust the parameters of the basic language feature extraction layer, the basic feature pooling layer and the basic language recognition layer to obtain the trained language feature extraction layer, the feature pooling layer and the language recognition layer. The second loss function can be, for example, a cross entropy (CE) loss function. The basic language feature extraction layer and the basic language recognition layer can, for example, select a time series model structure, such as a recurrent neural network RNN, a self-attention mechanism layer, etc.

[0083] In 305, a third training sample is obtained, where the third training sample includes the second sample high-dimensional acoustic features of the third sample audio to be recognized and the second sample language recognition result;

[0084] In 306, the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer are trained according to the second sample high-dimensional acoustic features and the second sample language recognition results to obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer.

[0085] Furthermore, after the phoneme conversion layer, language feature extraction layer, feature pooling layer and language recognition layer are trained separately, the embodiment of the present application can also continue to train the trained phoneme conversion layer, language feature extraction layer, feature pooling layer and language recognition layer as a whole model again, please refer to Figure 8 , Figure 8A training diagram of the phoneme conversion layer, language feature extraction layer, feature pooling layer, and language recognition layer of the language recognition model provided for the embodiment of the present application. Specifically, a third training sample is obtained, and the third training sample includes the second sample high-dimensional acoustic features of the third sample audio to be recognized and the second sample language recognition result. Exemplarily, the third sample audio to be recognized can be obtained first; the third sample audio to be recognized is input into the acoustic model, and the acoustic model outputs the second sample high-dimensional acoustic features of the third sample audio to be recognized, and then the second sample high-dimensional acoustic features of the third sample audio to be recognized and the second sample language recognition result are used as the third training sample.

[0086] After obtaining the third training sample, the high-dimensional acoustic features of the second sample are input into the phoneme conversion layer, and the high-dimensional acoustic features of the second sample are converted into a third phoneme sequence of the third sample audio to be recognized through the phoneme conversion layer; the third phoneme sequence is input into the language feature extraction layer, and the fifth language features of the third sample audio to be recognized are extracted through the language feature extraction layer; the fifth language features are input into the feature pooling layer, and the feature pooling layer reduces the dimension of the fifth language features to obtain a one-dimensional sixth language feature; the sixth language features are input into the language recognition layer, and the language recognition layer is used to perform language recognition to obtain a second predicted language recognition result of the third sample audio to be recognized; based on the difference information between the second predicted language recognition result and the second sample language recognition result, a third loss value is obtained, and the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer are trained based on the third loss value to obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer. Among them, the second predicted language recognition result and the second sample language recognition result can be input into the third loss function, and the third loss value is determined by the third loss function, and the gradient is returned to the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer, so as to adjust the parameters of the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer as a whole, and obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer after training. Among them, the third loss function can be, for example, a cross entropy (CE) loss function, and the first loss function, the second loss function, and the third loss function can be selected as needed, and can be the same or different, and the embodiment of the present application is not limited to this.

[0087] It should be noted that the embodiment of the present application does not limit the number of separate training and overall training of each level of the language recognition model. The embodiment of the present application can perform iterative training multiple times until the preset stop condition is met to obtain the language recognition model of the present application. It can be understood that after the separate training and overall training of each level of the language recognition model, the new phoneme conversion layer can replace the phoneme conversion layer, the new language feature extraction layer can replace the language feature extraction layer, the new feature pooling layer can replace the feature pooling layer, and the new language recognition layer can replace the language recognition layer. Moreover, in the embodiment of the present application, only the above-mentioned separate training or overall training can be performed on each level of the language recognition model, and the embodiment of the present application does not limit this.

[0088] It should be noted that the language recognition model of the embodiment of the present application can reuse the acoustic model in the speech recognition model and take the high-dimensional acoustic features output by the acoustic model as input. There is no need to set up a separate layer with similar functions to the acoustic model of the speech recognition model to extract high-dimensional acoustic features, nor is there any need to train the layer. The feature pooling layer of the language recognition model is used to reduce the dimension of the first language features output by the language feature extraction layer. It may have no computing parameters or only very few computing parameters. The language recognition model mainly includes the training of three neural network layers during the training process. Therefore, the training process of the language recognition model occupies fewer computing resources.

[0089] In the embodiment of the present application, even if there are fewer training sample resources for some small languages, the phoneme conversion layer can be trained by obtaining training samples of large languages, such as Chinese and English training samples, so that the trained phoneme conversion layer can also be applied to small languages. For example, taking phonemes as pinyin as an example, the phoneme conversion layer can be trained by Chinese training samples. When the phoneme conversion layer receives the audio to be recognized in other languages, it can also convert the audio to be recognized in other languages ​​into a combination of pinyin, that is, into a phoneme sequence, so as to realize the sharing of phoneme conversion layers between different languages. In addition, the language feature extraction layer and the language recognition layer can also first use the training samples of large languages, such as Chinese and English training samples to train the language feature extraction layer, so that the language feature extraction layer and the language recognition layer can be used for large languages. When it is necessary to perform language recognition on the audio to be recognized in a small language, the training samples of the small language can be input into the language feature extraction layer and the language recognition layer after the large language training for training with a small amount of data, so as to obtain the same language recognition effect as the large language. Exemplarily, the language feature extraction layer and language recognition layer that have been trained for a large language are equivalent to having trained about 90% of the parameters in the language feature extraction layer and the language recognition layer. For small languages, only the last 10% or so of the language feature extraction layer and the language recognition layer need to be trained, and the relevant parameters of the language feature extraction layer and the language recognition layer need to be adjusted accordingly, without the need to independently train a large amount of data for each small language. Therefore, the embodiment of the present application does not need to train a language recognition model for each language, and a language recognition model can be used to recognize multiple languages. When different languages ​​are recognized, there is no need to switch the corresponding language recognition model. In particular, for small languages, there is no need to train a language recognition model separately, and it is possible to use fewer training sample resources. A higher language recognition accuracy can still be obtained through a shared language recognition model, which reduces computing resources and reduces the cost of the language recognition model. It can be understood that the language recognition results obtained can be used as language / ethnicity / regional information for downstream natural language processing tasks to assist in making responses that are more suitable for user groups.

[0090] In one embodiment, the language recognition model in the embodiment of the present application includes four neural network layers, namely, a phoneme extraction layer, a language feature extraction layer, a feature pooling layer, and a language recognition layer. As needed, not only four neural network layers can be set, for example, multiple phoneme extraction layers, and / or multiple language feature extraction layers, and / or multiple feature pooling layers, and / or multiple language recognition layers can be set to achieve better language recognition effects. In addition, the language recognition model can also include other levels, such as non-linear activation functions and normalization layers, etc., which are not limited in the embodiment of the present application.

[0091] See also Fig. 9 and Fig.10 , Fig. 9 A first structural diagram of a language recognition device provided in an embodiment of the present application is shown in FIG. Fig.10 A second structural diagram of a language identification device provided in an embodiment of the present application: The language identification device 400 may include an acquisition module 401 , a first extraction module 402 , a conversion module 403 , a second extraction module 404 , and a recognition module 405 .

[0092] The acquisition module 401 is used to: acquire the audio to be recognized that needs to be recognized in the language;

[0093] The first extraction module 402 is used to: input the audio to be recognized into the acoustic model of the speech recognition model, and extract the high-dimensional acoustic features of the audio to be recognized through the acoustic model;

[0094] The conversion module 403 is used to: input the high-dimensional acoustic features into the phoneme conversion layer of the language recognition model, and convert the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer;

[0095] The second extraction module 404 is used to: input the phoneme sequence into the language feature extraction layer of the language recognition model, and extract the first language feature of the audio to be recognized through the language feature extraction layer;

[0096] The recognition module 405 is used to: input the first language feature into the language recognition layer of the language recognition model, perform language recognition through the language recognition layer, and obtain a language recognition result of the audio to be recognized.

[0097] In one embodiment, the language recognition device also includes a pooling module 406, which can be used to: input the first language feature into the feature pooling layer to pool the first language feature in the time domain, and reduce the dimension of the first language feature to obtain a one-dimensional second language feature; the recognition module 405 can also be used to: input the second language feature into the language recognition layer of the language recognition model, perform language recognition through the language recognition layer, and obtain a language recognition result of the audio to be recognized.

[0098] In one embodiment, the recognition module 405 can also be used to: input the second language feature into the language recognition layer, perform language recognition through the language recognition layer, the language recognition layer outputs the language distribution probability corresponding to the audio to be recognized, and generates a language recognition result based on the language distribution probability.

[0099] In one embodiment, the language recognition device 400 also includes a training module 407, which can be used to: obtain a first training sample, the first training sample including a first sample audio to be recognized and a first sample phoneme sequence corresponding to the first sample audio to be recognized; train the basic phoneme conversion layer according to the first sample audio to be recognized and the first sample phoneme sequence to obtain a phoneme conversion layer.

[0100] In one embodiment, the training module 407 can be used to: obtain a second training sample, where the second training sample includes the first sample high-dimensional acoustic features of the second sample audio to be recognized and the first sample language recognition result; freeze the phoneme conversion layer, and train the basic language feature extraction layer, the basic feature pooling layer, and the basic language recognition layer according to the first sample high-dimensional acoustic features and the first sample language recognition result to obtain the language feature extraction layer, the feature pooling layer, and the language recognition layer.

[0101] In one embodiment, the training module 407 can be used to: obtain a third training sample, the third training sample includes the second sample high-dimensional acoustic features of the third sample audio to be recognized and the second sample language recognition result; unfreeze the phoneme conversion layer, and train the phoneme conversion layer, language feature extraction layer, feature pooling layer and language recognition layer according to the second sample high-dimensional acoustic features and the second sample language recognition result to obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer.

[0102] In one implementation, the training module 407 may be used to: obtain a third sample audio to be recognized; input the third sample audio to be recognized into the acoustic model, and the acoustic model outputs a second sample high-dimensional acoustic feature of the third sample audio to be recognized.

[0103] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the process in the language recognition method provided in the embodiment.

[0104] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the process in the language recognition method provided in the present embodiment by calling a computer program stored in the memory.

[0105] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.11 , Fig.11 A first structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0106] The electronic device 500 may include components such as a memory 502 and a processor 501. Those skilled in the art will appreciate that Fig.11 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0107] The memory 502 can be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications can be composed of various functional modules. The processor 501 executes various functional applications and data processing by running the applications stored in the memory 502. The processor 501 is electrically connected to the memory 502.

[0108] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.

[0109] In this embodiment, the processor 501 in the electronic device loads the executable code corresponding to the process of one or more applications into the memory 502 according to the following instructions, and the processor 501 runs the application stored in the memory 502 to execute: obtaining the audio to be recognized that needs to be recognized in language; inputting the audio to be recognized into the acoustic model of the speech recognition model, and extracting the high-dimensional acoustic features of the audio to be recognized through the acoustic model; inputting the high-dimensional acoustic features into the phoneme conversion layer of the language recognition model, and converting the high-dimensional acoustic features into the phoneme sequence of the audio to be recognized through the phoneme conversion layer; inputting the phoneme sequence into the language feature extraction layer of the language recognition model, and extracting the first language feature of the audio to be recognized through the language feature extraction layer; inputting the first language feature into the language recognition layer of the language recognition model, and performing language recognition through the language recognition layer to obtain the language recognition result of the audio to be recognized.

[0110] See also Fig.12 , Fig.12 A second structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 500 further includes components such as a display screen 503, a radio frequency circuit 504, a control circuit 505, an input unit 506, an audio circuit 507, a sensor 508, and a power supply 509. The processor 501 is electrically connected to the display screen 503, the radio frequency circuit 504, the control circuit 505, the input unit 506, the audio circuit 507, the sensor 508, and the power supply 509, respectively.

[0111] The display screen 503 may be used to display information input by a user or information provided to a user and various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of images, texts, icons, videos and any combination thereof.

[0112] The radio frequency circuit 504 is used to send and receive radio frequency signals to communicate with network devices or other electronic devices through wireless communication.

[0113] The control circuit 505 is electrically connected to the display screen 503 and is used to control the display screen 501 to display information.

[0114] The input unit 506 can be used to receive input numbers, character information or user feature information (such as fingerprints), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. The input unit 506 can include a fingerprint recognition module.

[0115] The audio circuit 507 can provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 507 includes a microphone. The microphone is electrically connected to the processor 501. The microphone is used to receive voice information input by the user.

[0116] The sensor 508 is used to collect external environment information. The sensor 508 may include one or more sensors such as an environment brightness sensor, an acceleration sensor, and a gyroscope.

[0117] The power supply 509 is used to supply power to various components of the electronic device 500. In some embodiments, the power supply 509 may be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, power consumption, and other functions.

[0118] Although not shown in the figure, the electronic device 500 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0119] In this embodiment, the processor 501 in the electronic device 500 will load the executable code corresponding to the process of one or more applications into the memory 502 according to the following instructions, and the processor 501 will run the application stored in the memory 502 to execute: obtaining the audio to be recognized that needs to be recognized in language; inputting the audio to be recognized into the acoustic model of the speech recognition model, and extracting the high-dimensional acoustic features of the audio to be recognized through the acoustic model; inputting the high-dimensional acoustic features into the phoneme conversion layer of the language recognition model, and converting the high-dimensional acoustic features into the phoneme sequence of the audio to be recognized through the phoneme conversion layer; inputting the phoneme sequence into the language feature extraction layer of the language recognition model, and extracting the first language feature of the audio to be recognized through the language feature extraction layer; inputting the first language feature into the language recognition layer of the language recognition model, and performing language recognition through the language recognition layer to obtain the language recognition result of the audio to be recognized.

[0120] In one embodiment, when inputting the first language feature into the language recognition layer of the language recognition model, performing language recognition through the language recognition layer, and obtaining the language recognition result of the audio to be recognized, the processor 501 executes: inputting the first language feature into the feature pooling layer to pool the first language feature in the time domain, and reducing the dimension of the first language feature to obtain a one-dimensional second language feature; inputting the second language feature into the language recognition layer of the language recognition model, performing language recognition through the language recognition layer, and obtaining the language recognition result of the audio to be recognized.

[0121] In one embodiment, when inputting the second language feature into the language recognition layer of the language recognition model, performing language recognition through the language recognition layer, and obtaining the language recognition result of the audio to be recognized, the processor 501 executes: inputting the second language feature into the language recognition layer, performing language recognition through the language recognition layer, the language recognition layer outputs the language distribution probability corresponding to the audio to be recognized, and generates the language recognition result based on the language distribution probability.

[0122] In one embodiment, before inputting high-dimensional acoustic features into the phoneme conversion layer of the language recognition model, the processor 501 executes: obtaining a first training sample, the first training sample including a first sample audio to be recognized and a first sample phoneme sequence corresponding to the first sample audio to be recognized; training the basic phoneme conversion layer according to the first sample audio to be recognized and the first sample phoneme sequence to obtain the phoneme conversion layer.

[0123] In one embodiment, after obtaining the phoneme conversion layer, the processor 501 executes: obtaining a second training sample, the second training sample including a first sample high-dimensional acoustic feature of a second sample audio to be recognized and a first sample language recognition result; freezing the phoneme conversion layer, and training a basic language feature extraction layer, a basic feature pooling layer, and a basic language recognition layer according to the first sample high-dimensional acoustic feature and the first sample language recognition result to obtain a language feature extraction layer, a feature pooling layer, and a language recognition layer.

[0124] In one embodiment, after obtaining the language feature extraction layer, the feature pooling layer and the language recognition layer, the processor 501 executes: obtaining a third training sample, the third training sample including the second sample high-dimensional acoustic features of the third sample audio to be recognized and the second sample language recognition result; unfreezing the phoneme conversion layer, and training the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer according to the second sample high-dimensional acoustic features and the second sample language recognition result to obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer.

[0125] In one embodiment, before obtaining the third training sample, the processor 501 executes: obtaining the third sample audio to be recognized; inputting the third sample audio to be recognized into the acoustic model, and the acoustic model outputs the second sample high-dimensional acoustic features of the third sample audio to be recognized.

[0126] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the language recognition method above, which will not be repeated here. The language recognition device provided in the embodiment of the present application belongs to the same concept as the language recognition method in the above embodiment. Any method provided in the language recognition method embodiment can be run on the language recognition device. The specific implementation process is detailed in the language recognition method embodiment, which will not be repeated here.

[0127] It should be noted that, for the language identification method of the embodiment of the present application, a person of ordinary skill in the art can understand that all or part of the process of implementing the language identification method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the language identification method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0128] For the language recognition device of the embodiment of the present application, its various functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0129] The language recognition method, language recognition device, storage medium and electronic device provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A language recognition method, characterized in that: The method comprises: Obtain the audio to be recognized that needs to be recognized in the language; Inputting the audio to be recognized into the acoustic model of the speech recognition model, and extracting high-dimensional acoustic features of the audio to be recognized through the acoustic model; Inputting the high-dimensional acoustic features into a phoneme conversion layer of a language recognition model, and converting the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer; Inputting the phoneme sequence into the language feature extraction layer of the language recognition model, and extracting the first language feature of the audio to be recognized through the language feature extraction layer; The first language feature is input into the language recognition layer of the language recognition model, and language recognition is performed through the language recognition layer to obtain a language recognition result of the audio to be recognized.

2. The language identification method according to claim 1, characterized in that: The step of inputting the first language feature into the language recognition layer of the language recognition model, performing language recognition through the language recognition layer, and obtaining the language recognition result of the audio to be recognized includes: Inputting the first language feature into the feature pooling layer of the language recognition model to perform time domain pooling on the first language feature and reduce the dimension of the first language feature to obtain a one-dimensional second language feature; The second language feature is input into the language recognition layer of the language recognition model, and language recognition is performed through the language recognition layer to obtain a language recognition result of the audio to be recognized.

3. The language identification method according to claim 2, characterized in that: The step of inputting the second language feature into the language recognition layer of the language recognition model, performing language recognition through the language recognition layer, and obtaining the language recognition result of the audio to be recognized includes: The second language feature is input into the language recognition layer, and language recognition is performed through the language recognition layer. The language recognition layer outputs the language distribution probability corresponding to the audio to be recognized, and generates a language recognition result based on the language distribution probability.

4. The language identification method according to claim 3, characterized in that: Before inputting the high-dimensional acoustic features into the phoneme conversion layer of the language recognition model, the method further includes: Acquire a first training sample, where the first training sample includes a first sample audio to be recognized and a first sample phoneme sequence corresponding to the first sample audio to be recognized; The basic phoneme conversion layer is trained according to the first sample audio to be recognized and the first sample phoneme sequence to obtain the phoneme conversion layer.

5. The language identification method according to claim 4, characterized in that: After obtaining the phoneme conversion layer, the method further includes: Obtaining a second training sample, where the second training sample includes the first sample high-dimensional acoustic features of the second sample audio to be recognized and the first sample language recognition result; The phoneme conversion layer is frozen, and a basic language feature extraction layer, a basic feature pooling layer, and a basic language recognition layer are trained according to the high-dimensional acoustic features of the first sample and the language recognition result of the first sample to obtain the language feature extraction layer, the feature pooling layer, and the language recognition layer.

6. The language identification method according to claim 5, characterized in that: After obtaining the language feature extraction layer, the feature pooling layer and the language recognition layer, the method further includes: Acquire a third training sample, wherein the third training sample includes a second sample high-dimensional acoustic feature of the third sample audio to be recognized and a second sample language recognition result; The phoneme conversion layer is unfrozen, and the phoneme conversion layer, the language feature extraction layer, the feature pooling layer and the language recognition layer are trained according to the high-dimensional acoustic features of the second sample and the language recognition result of the second sample to obtain a new phoneme conversion layer, a new language feature extraction layer, a new feature pooling layer and a new language recognition layer.

7. The language identification method according to claim 6, characterized in that: Before obtaining the third training sample, the method further includes: Obtaining the third sample audio to be recognized; The third sample audio to be recognized is input into the acoustic model, and the acoustic model outputs the second sample high-dimensional acoustic features of the third sample audio to be recognized.

8. A language recognition device, characterized in that: The device comprises: An acquisition module is used to acquire the audio to be recognized that needs to be recognized in the language; A first extraction module, used for inputting the audio to be recognized into an acoustic model of a speech recognition model, and extracting high-dimensional acoustic features of the audio to be recognized through the acoustic model; A conversion module, used for inputting the high-dimensional acoustic features into a phoneme conversion layer of a language recognition model, and converting the high-dimensional acoustic features into a phoneme sequence of the audio to be recognized through the phoneme conversion layer; A second extraction module, configured to input the phoneme sequence into a language feature extraction layer of the language recognition model, and extract a first language feature of the audio to be recognized through the language feature extraction layer; The recognition module is used to input the first language feature into the language recognition layer of the language recognition model, perform language recognition through the language recognition layer, and obtain the language recognition result of the audio to be recognized.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. An electronic device, comprising a memory and a processor, characterized in that: The processor is configured to execute the method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

Images