A language recognition method, device and electronic device based on neural network
Through the neural network-based language recognition method, combined with feature extraction and switching judgment model, the problem of language switching recognition errors in the intelligent recruitment system is solved, the accuracy of language recognition is improved, and the reliability of interview results is enhanced.
Patent Information
- Application Number
- CN202110842017.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-07-23
AI Technical Summary
The existing intelligent recruitment system cannot accurately identify voice recognition errors caused by switching languages in the middle of speaking during the interview, resulting in biased interview results.
Using a neural network-based language recognition method, feature extraction is performed by receiving audio data, sending it to the language recognition model and switching judgment model, calculating the probability value and confusion score of the candidate language, combining the neural network score weighted summing, determine the language, and Improve the accuracy of language recognition.
It improves the accuracy of language recognition, reduces errors in the speech and text conversion process, and enhances the perfection of the intelligent recruitment system.
Smart Images

Figure CN113380227B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data information processing, and more specifically relates to a language recognition method, device and electronic equipment based on neural network. Background Art
[0002] In traditional recruitment interviews, the communication with candidates is usually done offline face-to-face. With the development of Internet technology, more and more companies are using online intelligent recruitment systems to interact with candidates for recruitment interviews. In the process of interviewing with intelligent recruitment systems, there will be many questions that require candidates to answer by voice.
[0003] When the interviewee answers via voice, he or she may switch languages to answer, such as a common mixture of Chinese and English. The existing technology usually determines the language and recognizes it based on voice features. However, many people's non-native spoken language is very poor, so the pronunciation of non-native languages is often close to that of their native language. In this case, there is a high probability that the wrong language will be determined, causing problems in the subsequent voice and text conversion process, resulting in a large deviation in the interview results. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] The present invention aims to solve the problem that in the existing intelligent recruitment system, when performing speech recognition, the speech in which the language is switched in the middle of speaking cannot be accurately recognized.
[0006] (II) Technical solution
[0007] In order to solve the above technical problems, the first aspect of the present invention provides a language recognition method based on a neural network, comprising:
[0008] Receiving audio data input by a user, and performing feature extraction on the audio data;
[0009] Sending the feature-extracted data to a language recognition model to obtain a first probability value of a candidate language, and selecting the top n candidate languages, where n is a natural number;
[0010] The audio data and the top n candidate languages are sent to a language determination model to obtain second probability values of the n candidate languages, and the language used by the audio data is determined according to the second probability values of the n candidate languages.
[0011] According to a preferred embodiment of the present invention, the feature extraction of the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
[0012] According to a preferred embodiment of the present invention, before sending the feature-extracted data to the language recognition model, the feature-extracted data is also sent to a switching judgment model.
[0013] According to a preferred embodiment of the present invention, the switching judgment model judges whether the language has been switched in units of frames, and obtains the number of language switching times of the audio information, the type of language switching and the corresponding duration.
[0014] According to a preferred embodiment of the present invention, the audio data and the top n candidate languages are sent to a language determination model, and the second probability values of the n candidate languages are obtained as follows:
[0015] Determining confusion scores of n candidate languages based on the audio data and the top n candidate languages;
[0016] Determining a neural network score according to first probability values of the n candidate languages;
[0017] The confusion scores and neural network scores of the n candidate languages are weighted and summed to obtain second probability values of the n candidate languages.
[0018] According to a preferred embodiment of the present invention, the confusion scores of n candidate languages are determined as follows:
[0019] The audio data and the top n candidate languages are sent to a recognition engine, texts of the n candidate languages are transcribed, and confusion scores of the n candidate languages are calculated based on the texts of the n languages.
[0020] According to a preferred embodiment of the present invention, the neural network score is a posterior probability value of the n candidate languages.
[0021] A second aspect of the present invention provides a language recognition device based on a neural network, comprising:
[0022] A feature extraction module, used to receive audio data input by a user and perform feature extraction on the audio data;
[0023] A language recognition module, used to send the feature-extracted data to a language recognition model, obtain a first probability value of a candidate language, and select the top n candidate languages, where n is a natural number;
[0024] The language determination module is used to send the audio data and the top n candidate languages to the language determination model, obtain second probability values of the n candidate languages, and determine the language used by the audio data according to the second probability values of the n candidate languages.
[0025] According to a preferred embodiment of the present invention, the feature extraction of the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
[0026] According to a preferred embodiment of the present invention, before sending the feature-extracted data to the language recognition model, the feature-extracted data is also sent to a switching judgment model.
[0027] According to a preferred embodiment of the present invention, the switching judgment model judges whether the language has been switched in units of frames, and obtains the number of language switching times of the audio information, the type of language switching and the corresponding duration.
[0028] According to a preferred embodiment of the present invention, the audio data and the top n candidate languages are sent to a language determination model, and the second probability values of the n candidate languages are obtained as follows:
[0029] Determining confusion scores of n candidate languages based on the audio data and the top n candidate languages;
[0030] Determining a neural network score according to first probability values of the n candidate languages;
[0031] The confusion scores and neural network scores of the n candidate languages are weighted and summed to obtain second probability values of the n candidate languages.
[0032] According to a preferred embodiment of the present invention, the confusion scores of n candidate languages are determined as follows:
[0033] The audio data and the top n candidate languages are sent to a recognition engine, texts of the n candidate languages are transcribed, and confusion scores of the n candidate languages are calculated based on the texts of the n languages.
[0034] According to a preferred embodiment of the present invention, the neural network score is a posterior probability value of the n candidate languages.
[0035] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the described method.
[0036] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method described is implemented.
[0037] (III) Beneficial effects
[0038] The present invention finally determines the language information used in the speech data by calculating the confusion score and the neural network score of the text corresponding to the speech data, thereby improving the accuracy of language recognition and making the intelligent recruitment system more perfect. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a flow chart of a language recognition method based on a neural network according to an embodiment of the present invention;
[0040] Figure 2 It is a schematic diagram of a flow chart of a first embodiment of the present invention;
[0041] Figure 3 It is a schematic diagram of the structure of a language recognition device based on a neural network according to an embodiment of the present invention;
[0042] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
[0043] Figure 5 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.
[0045] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.
[0046] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.
[0047] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.
[0048] In order to solve the above technical problems, the present invention proposes a language recognition method based on neural network, the method flow chart is as follows Figure 1 As shown, including:
[0049] S101: Receive audio data input by a user, and perform feature extraction on the audio data.
[0050] On the basis of the above technical solution, further, extracting features from the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
[0051] During the interview process, many test questions require users to answer or explain through voice. At this time, the user's audio data is received, and the test sound is feature extracted, and the Mel frequency cepstral coefficient feature or Mel filter bank feature is extracted. In this embodiment, the pitch feature is also added to enhance its distinguishability. Because the pronunciation frequency, voiced sound, and whether the sound is aspirated are different in different languages, the mean variance of the frequency, the zero crossing rate, the chromaticity feature, and the rhythm feature are also added.
[0052] S102: Send the feature-extracted data to a language recognition model to obtain a first probability value of a candidate language, and select n candidate languages with the highest ranking, where n is a natural number.
[0053] In this embodiment, the feature-extracted data is sent to the language recognition model. The language recognition model is an RNN network model based on deep learning. It is obtained by training with a large amount of language data. After inputting the feature-extracted data, a group of candidate language probabilities are output and arranged in order from large to small. N high-scoring candidate languages are selected as alternative languages.
[0054] On the basis of the above technical solution, further, before sending the feature-extracted data to the language recognition model, the feature-extracted data is also sent to the switching judgment model.
[0055] In this embodiment, since there may be a problem of language switching during speaking, a switching judgment model is set to judge the language switching.
[0056] On the basis of the above technical solution, further, the switching judgment model judges whether the language has been switched in units of frames, and obtains the number of language switching times of the audio information, the type of language switching and the corresponding duration.
[0057] In this embodiment, the switching judgment model determines whether the language has switched in units of frames, and a frame is usually 25 milliseconds. Once the switching judgment model detects the occurrence of a switch, it will send a signal to the language recognition model, and the language recognition model will output the results from the beginning of the audio to the current moment and the duration of the current audio, and reset the previous history to avoid interference with subsequent results. Until the audio ends.
[0058] The following is an example to illustrate.
[0059] In the 60-second audio, the speaker speaks in Chinese from 1 to 20 seconds, then switches to English from 20 to 40 seconds, and then switches back to Chinese from 40 to 60 seconds. The following is the situation when this audio enters the language recognition model:
[0060] First, the language recognition model requires two parts of information. One part is the features of the current audio frame, and the second part is the historical features left by the previous audio frame.
[0061] The language recognition model will continue to calculate, and the 1-20s switching judgment model will always give a signal of no switching. Then, when the language changes at 20s, the switching judgment model will give a switching signal. At this time, the switching judgment model will reset the historical features and only retain the features of the current frame. At the same time, it will give a timestamp and result based on the current time point: "1~20s: Language A". Since the language has not been finalized, it is only recorded in language A at this time.
[0062] Then, the signal is not switched from 20 to 40 seconds. Until 40 seconds, the situation is the same as above, and the timestamp and result are output: "20-40s: Language B", and the timestamp and result are output from 40 to 60 seconds: "40-60s: Language A".
[0063] After the switching judgment model is used, the language switching information of the audio includes: "1-20s: Language A; 20-40s: Language B; 40-60s: Language A", the number of switching times is 3, and the language switching types are 2.
[0064] S103: Send the audio data and the top n candidate languages to a language determination model to obtain second probability values of the n candidate languages, and determine the language used by the audio data according to the second probability values of the n candidate languages.
[0065] On the basis of the above technical solution, further, the audio data and the top n candidate languages are sent to the language determination model, and the second probability values of the n candidate languages are obtained as follows:
[0066] Determining confusion scores of n candidate languages based on the audio data and the top n candidate languages;
[0067] Determining a neural network score according to first probability values of the n candidate languages;
[0068] The confusion scores and neural network scores of the n candidate languages are weighted and summed to obtain second probability values of the n candidate languages.
[0069] In this embodiment, the final probability value of the candidate language, i.e., the second probability value, is obtained by calculating the weighted sum of the confusion score and the neural network score. The language determined in this way is more accurate, and the possibility of errors in subsequent operations such as speech recognition is greatly reduced.
[0070] On the basis of the above technical solution, the confusion scores of the n candidate languages are further determined as follows:
[0071] The audio data and the top n candidate languages are sent to a recognition engine, texts of the n candidate languages are transcribed, and confusion scores of the n candidate languages are calculated based on the texts of the n languages.
[0072] In this embodiment, the audio is converted into text in n candidate languages according to the speech conversion of the audio, and the text is converted into text in n candidate languages, and the confusion degree of these texts is calculated. The higher the confusion degree, the less the content of the text is like a sentence. The confusion degree can be calculated in the following way:
[0073] First, calculate the probability of the sentence. The probability of the sentence is P(S), and the calculation formula is as follows:
[0074] P(S)=P(W1,W2,…,W k )=P(W1)P(W2|W1)…P(W k |W1, W2, ... W k-1 )
[0075] Where W is a word or phrase, W1 to W k Sentence S is composed, k is a positive integer. For example, "Today's breakfast is fried dough sticks", k = 4, W1 is today, W2 is breakfast, W3 is eating, and W4 is fried dough sticks. P(Wk |W1, W2, ... W k-1 ) is the first word W1 to k-1 words W k-1 The kth word W appears under the premise of k The confusion is related to the probability of the sentences in the test set. The language model that gives the sentences in the test set a higher probability value is better. When the language model is consistent, the more the sentences in the test set are like a sentence, the higher its probability and the lower the confusion. At this time, the calculation formula of the confusion PPL(S) is as follows:
[0076]
[0077] On the basis of the above technical solution, further, the neural network score is a posterior probability value of n candidate languages.
[0078] In this embodiment, the confusion score and the neural network score are finally calculated by weighted summation to obtain a second probability value, which is sorted in descending order. The one or more languages used in the audio are determined in combination with the language switching information of the audio. For example, if there are two types of language switching in the example, the top two languages are the languages used in the audio.
[0079] The specific process of the language recognition method based on neural network is described below through Example 1. Figure 2 shown.
[0080] S1011, receiving audio data input by a user;
[0081] S1012, extracting features from the audio data;
[0082] S2011, sending the feature-extracted data to a language recognition model to obtain a first probability value of a candidate language;
[0083] S2012, sending the feature-extracted data to a switching judgment model, and the switching judgment model sending the language switching information of the audio to the language recognition model;
[0084] S2013. Select the top n candidate languages, where n is a natural number;
[0085] S3011, sending the audio data and the top n candidate languages to a recognition engine, and transcribing texts of the n candidate languages;
[0086] S3012, calculating confusion scores of n candidate languages according to the texts in the n languages;
[0087] S3013, determining a neural network score according to the first probability values of the n candidate languages;
[0088] S3014, performing weighted summation of the confusion scores and neural network scores of the n candidate languages to obtain a second probability value of the n candidate languages;
[0089] S3015: Determine the language used by the audio data according to the second probability values of the n candidate languages.
[0090] Figure 3 A language recognition device 300 based on a neural network according to an embodiment of the present invention comprises:
[0091] The feature extraction module 301 is used to receive audio data input by a user and perform feature extraction on the audio data.
[0092] On the basis of the above technical solution, further, extracting features from the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
[0093] During the interview process, many test questions require users to answer or explain through voice. At this time, the user's audio data is received, and the test sound is feature extracted, and the Mel frequency cepstral coefficient feature or Mel filter bank feature is extracted. In this embodiment, the pitch feature is also added to enhance its distinguishability. Because the pronunciation frequency, voiced sound, and whether the sound is aspirated are different in different languages, the mean variance of the frequency, the zero crossing rate, the chromaticity feature, and the rhythm feature are also added.
[0094] The language identification module 302 is used to send the feature-extracted data to the language identification model, obtain the first probability value of the candidate language, and select the top n candidate languages, where n is a natural number.
[0095] In this embodiment, the feature-extracted data is sent to the language recognition model. The language recognition model is an RNN network model based on deep learning. It is obtained by training with a large amount of language data. After inputting the feature-extracted data, a group of candidate language probabilities are output and arranged in order from large to small. N high-scoring candidate languages are selected as alternative languages.
[0096] On the basis of the above technical solution, further, before sending the feature-extracted data to the language recognition model, the feature-extracted data is also sent to the switching judgment model.
[0097] In this embodiment, since there may be a problem of language switching during speaking, a switching judgment model is set to judge the language switching.
[0098] On the basis of the above technical solution, further, the switching judgment model judges whether the language has been switched in units of frames, and obtains the number of language switching times of the audio information, the type of language switching and the corresponding duration.
[0099] In this embodiment, the switching judgment model determines whether the language has switched in units of frames, and a frame is usually 25 milliseconds. Once the switching judgment model detects the occurrence of a switch, it will send a signal to the language recognition model, and the language recognition model will output the results from the beginning of the audio to the current moment and the duration of the current audio, and reset the previous history to avoid interference with subsequent results. Until the audio ends.
[0100] The following is an example to illustrate.
[0101] In the 60-second audio, the speaker speaks in Chinese from 1 to 20 seconds, then switches to English from 20 to 40 seconds, and then switches back to Chinese from 40 to 60 seconds. The following is the situation when this audio enters the language recognition model:
[0102] First, the language recognition model requires two parts of information. One part is the features of the current audio frame, and the second part is the historical features left by the previous audio frame.
[0103] The language recognition model will continue to calculate, and the 1-20s switching judgment model will always give a signal of no switching. Then, when the language changes at 20s, the switching judgment model will give a switching signal. At this time, the switching judgment model will reset the historical features and only retain the features of the current frame. At the same time, it will give a timestamp and result based on the current time point: "1~20s: Language A". Since the language has not been finalized, it is only recorded in language A at this time.
[0104] Then, the signal is not switched from 20 to 40 seconds. Until 40 seconds, the situation is the same as above, and the timestamp and result are output: "20-40s: Language B", and the timestamp and result are output from 40 to 60 seconds: "40-60s: Language A".
[0105] After the switching judgment model is used, the language switching information of the audio includes: "1-20s: Language A; 20-40s: Language B; 40-60s: Language A", the number of switching times is 3, and the language switching types are 2.
[0106] The language determination module 303 is used to send the audio data and the top n candidate languages to the language determination model to obtain second probability values of the n candidate languages, and determine the language used by the audio data according to the second probability values of the n candidate languages.
[0107] On the basis of the above technical solution, further, the audio data and the top n candidate languages are sent to the language determination model, and the second probability values of the n candidate languages are obtained as follows:
[0108] Determining confusion scores of n candidate languages based on the audio data and the top n candidate languages;
[0109] Determining a neural network score according to first probability values of the n candidate languages;
[0110] The confusion scores and neural network scores of the n candidate languages are weighted and summed to obtain second probability values of the n candidate languages.
[0111] In this embodiment, the final probability value of the candidate language, i.e., the second probability value, is obtained by calculating the weighted sum of the confusion score and the neural network score. The language determined in this way is more accurate, and the possibility of errors in subsequent operations such as speech recognition is greatly reduced.
[0112] On the basis of the above technical solution, the confusion scores of the n candidate languages are further determined as follows:
[0113] The audio data and the top n candidate languages are sent to a recognition engine, texts of the n candidate languages are transcribed, and confusion scores of the n candidate languages are calculated based on the texts of the n languages.
[0114] In this embodiment, the audio is converted into text in n candidate languages according to the speech conversion of the audio, and the text is converted into text in n candidate languages, and the confusion degree of these texts is calculated. The higher the confusion degree, the less the content of the text is like a sentence. The confusion degree can be calculated in the following way:
[0115] First, calculate the probability of the sentence. The probability of the sentence is P(S), and the calculation formula is as follows:
[0116] P(S)=P(W1,W2,…,W k )=P(W1)P(W2|W1)…P(W k |W1, W2, ... W k-1 )
[0117] Where W is a word or phrase, W1 to W k Sentence S is composed, k is a positive integer. For example, "Today's breakfast is fried dough sticks", k = 4, W1 is today, W2 is breakfast, W3 is eating, and W4 is fried dough sticks. P(W k |W1, W2, ... W k-1 ) is the first word W1 to k-1 words W k-1 The kth word W appears under the premise of kThe confusion is related to the probability of the sentences in the test set. The language model that gives the sentences in the test set a higher probability value is better. When the language model is consistent, the more the sentences in the test set are like a sentence, the higher its probability and the lower the confusion. At this time, the calculation formula of the confusion PPL(S) is as follows:
[0118]
[0119] On the basis of the above technical solution, further, the neural network score is a posterior probability value of n candidate languages.
[0120] In this embodiment, the confusion score and the neural network score are finally calculated by weighted summation to obtain a second probability value, which is sorted in descending order. The one or more languages used in the audio are determined in combination with the language switching information of the audio. For example, if there are two types of language switching in the example, the top two languages are the languages used in the audio.
[0121] Figure 4 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a vehicle intelligent power-assisted propulsion method based on rotation angle monitoring.
[0122] like Figure 4 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0123] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.
[0124] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0125] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.
[0126] It should be understood that Figure 4 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0127] Figure 5 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 5 As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the vehicle intelligent power-assisted pushing method based on rotation angle monitoring of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate or transmit a program for use by or in combination with an instruction execution system, device or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0128] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0129] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.
[0130] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A language recognition method based on neural network, characterized in that: Methods include: Receive audio data input by a user, perform feature extraction on the audio data and enhance resolution by adding frequency mean variance, zero crossing rate, chromaticity feature, and rhythmic feature tonal features; First, the feature-extracted data is sent to the switching judgment model. When the switching is detected, a switching signal is sent to the language recognition model. The language recognition model outputs the result from the beginning of the audio to the current moment and the current audio duration, and at the same time resets the previous history until the end of the audio. Sending the feature-extracted data to a language recognition model to obtain a set of first probability values of candidate languages, arranging the first probabilities in descending order, and selecting the top n candidate languages, where n is a natural number; Sending the audio data and the top n candidate languages to a language determination model, determining confusion scores of the n candidate languages according to the audio data and the top n candidate languages, determining a neural network score according to the first probability values of the n candidate languages, performing weighted summation of the confusion scores and the neural network scores of the n candidate languages to obtain a second probability value of the n candidate languages, and determining the language used by the audio data according to the second probability values of the n candidate languages; Among them, determining the confusion scores of n candidate languages includes calculating the probability of sentences, and the confusion is related to the sentence probability on the test set. The calculation specifically includes: sending the audio data and the top n candidate languages to the recognition engine, transcribing the texts of the n candidate languages, and calculating the confusion scores of the n candidate languages based on the texts of the n candidate languages.
2. The neural network-based language recognition method according to claim 1, characterized in that: The feature extraction of the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
3. The language recognition method based on neural network as claimed in claim 1, characterized in that: Also includes: The language recognition model requires the features of the current audio frame and the historical features left by the previous audio frames and performs continuous calculations; The switching judgment model will always give a non-switching signal and give a switching signal when the language changes, reset the historical features and only retain the features of the current frame, and give a timestamp and result based on the current time point.
4. The neural network-based language recognition method according to claim 3, characterized in that: The switching judgment model specifically includes: judging whether a language switching occurs in units of frames, and obtaining the number of language switching times, the type of language switching, and the corresponding duration of the audio information.
5. The neural network-based language recognition method according to claim 3, characterized in that: The neural network scores are the posterior probability values of the n candidate languages.
6. A language recognition device based on a neural network, characterized in that: The device includes: A feature extraction module, for receiving audio data input by a user, performing feature extraction on the audio data and adding pitch features including frequency mean variance, zero crossing rate, chromaticity features, and rhythmic features to enhance resolution; A language recognition module is used to first send the feature-extracted data to the switching judgment model, and when the occurrence of switching is detected, send a switching signal to the language recognition model, and the language recognition model outputs the result from the beginning of the audio to the current moment and the current audio duration and resets the previous history until the end of the audio; send the feature-extracted data to the language recognition model to obtain a set of first probability values of candidate languages, arrange the first probabilities in descending order, and select the top n candidate languages, where n is a natural number; a language determination module, configured to send the audio data and the top n candidate languages to a language determination model, determine confusion scores of the n candidate languages according to the audio data and the top n candidate languages, determine a neural network score according to the first probability values of the n candidate languages, perform a weighted summation of the confusion scores and the neural network scores of the n candidate languages to obtain a second probability value of the n candidate languages, and determine the language used by the audio data according to the second probability values of the n candidate languages; Among them, determining the confusion scores of n candidate languages includes calculating the probability of sentences, and the confusion is related to the sentence probability on the test set. The calculation specifically includes: sending the audio data and the top n candidate languages to the recognition engine, transcribing the texts of the n candidate languages, and calculating the confusion scores of the n candidate languages based on the texts of the n candidate languages.
7. The neural network-based language recognition device according to claim 6, characterized in that: The feature extraction of the audio data specifically includes: extracting Mel-frequency cepstral coefficient features or Mel-filter bank features from the audio data.
8. The language recognition device based on neural network as claimed in claim 6, characterized in that: Also includes: The language recognition model requires the features of the current audio frame and the historical features left by the previous audio frames and performs continuous calculations; The switching judgment model will always give a non-switching signal and give a switching signal when the language changes, reset the historical features and only retain the features of the current frame, and give a timestamp and result based on the current time point.
9. The neural network-based language recognition device according to claim 8, characterized in that: The switching judgment model specifically includes: judging whether a language switching occurs in units of frames, obtaining the number of language switching times of the audio information, the type of language switching and the corresponding duration.
10. The language recognition device based on neural network as claimed in claim 8, characterized in that: The neural network scores are the posterior probability values of the n candidate languages.
11. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 5.
12. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Bilingual hybrid speech recognition method, device and equipment and storage medium
CN110634487A
Multilingual speech recognition method and device and electronic equipment
CN112185348A
Language detection method and device based on artificial intelligence and electronic equipment
CN112989108A
Systems and Methods for Voice Identification
US20140350934A1
Method and Apparatus for Multi-Lingual End-to-End Speech Recognition
US20190189111A1
