Voice text recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice text recognition method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the following steps: carrying out sound spectrum feature extraction on target voice data to obtain target sound spectrum features; performing feature vector embedding on the target sound spectrum feature through an input layer of the target acoustic language model to obtain a target sound spectrum embedded vector; performing deep feature mapping on the target sound spectrum embedded vector through a hidden layer of the target acoustic language model to obtain a target sound spectrum deep vector; performing context feature extraction on the target sound spectrum deep layer vector through a recurrent neural network layer of the target acoustic language model to obtain a target sound spectrum context vector; performing pronunciation feature mapping on the target sound spectrum context vector through an output layer of the target acoustic language model to obtain a target pronunciation feature vector; and performing text decoding on the target pronunciation feature vector to obtain a target text. According to the embodiment of the invention, the voice text recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech-to-text recognition method and device, an electronic device, and a storage medium. Background Art

[0002] Speech-to-text recognition is a technology that analyzes and identifies the speaker's text from speech data. In the field of financial technology, it is often necessary to identify text from the user's language data, thereby identifying the user's intention and providing relevant services to the user. For example, in the insurance industry, the user's intention to purchase insurance products can be determined based on the text recognized from the user's speech data, so as to recommend relevant insurance products to the user.

[0003] Currently, pre-trained neural network models are usually used to directly perform text recognition on voice data. However, the components of voice data are relatively complex, such as noise and background sound. The structure of conventional neural network models is relatively simple, and it is difficult to extract high-accuracy features from voice data, resulting in a large room for improvement in recognition accuracy.

[0004] Therefore, how to provide a speech-to-text recognition method that can improve the accuracy of speech-to-text recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a speech-to-text recognition method and device, an electronic device, and a storage medium, which can improve the accuracy of speech-to-text recognition.

[0006] To achieve the above object, a first aspect of an embodiment of the present application proposes a method for speech-to-text recognition, the method comprising:

[0007] Obtain target voice data;

[0008] Extracting the spectral features of the target speech data to obtain the target spectral features;

[0009] Embedding the target sound spectrum feature with a feature vector through an input layer of a pre-trained target acoustic language model to obtain a target sound spectrum embedding vector;

[0010] Performing deep feature mapping on the target sound spectrum embedding vector through the hidden layer of the target acoustic language model to obtain a target sound spectrum deep vector;

[0011] Extracting context features of the target spectrum deep vector through the recurrent neural network layer of the target acoustic language model to obtain a target spectrum context vector;

[0012] Performing pronunciation feature mapping on the target sound spectrum context vector through the output layer of the target acoustic language model to obtain a target pronunciation feature vector;

[0013] The target pronunciation feature vector is subjected to text decoding to obtain a target text.

[0014] In some embodiments, before embedding the target sound spectrum feature through the input layer of the pre-trained target acoustic language model to obtain the target sound spectrum embedding vector, the method further includes:

[0015] Training the target acoustic language model specifically includes:

[0016] Obtain sample voice data;

[0017] Extracting spectral features of the sample speech data to obtain spectral features of the sample speech;

[0018] Inputting the sample sound spectrum feature into a preset initial acoustic language model to extract pronunciation features, and obtaining a sample pronunciation feature vector; wherein the initial acoustic language model includes an input layer, a hidden layer, a recurrent neural network layer, a recurrent connection layer and an output layer connected in sequence;

[0019] Perform loss calculation based on the sample pronunciation feature vector to obtain target loss data;

[0020] Adjusting parameters of the initial acoustic language model according to the target loss data to obtain a pseudo acoustic language model;

[0021] The recurrent connection layer in the pseudo acoustic language model is deleted to obtain the target acoustic language model.

[0022] In some embodiments, the step of inputting the sample sound spectrum feature into a preset initial acoustic language model to extract pronunciation features to obtain a sample pronunciation feature vector includes:

[0023] Performing feature vector embedding on the sample sound spectrum feature through the input layer to obtain a sample sound spectrum embedding vector;

[0024] Performing deep feature mapping on the sample sound spectrum embedding vector through the hidden layer to obtain a sample sound spectrum deep vector;

[0025] Performing a first context feature extraction on the sample sound spectrum deep vector through the recurrent neural network layer to obtain an initial sample sound spectrum context vector;

[0026] Performing a second context feature extraction on the initial sample spectrum context vector through the recurrent connection layer to obtain a target sample spectrum context vector;

[0027] The target sample sound spectrum context vector is pronunciation mapped through the output layer to obtain the sample pronunciation feature vector.

[0028] In some embodiments, the recurrent neural network layer includes a long short-term memory neural network sublayer and an activation function sublayer;

[0029] The step of extracting a first context feature from the sample sound spectrum deep vector by the recurrent neural network layer to obtain an initial sample sound spectrum context vector includes:

[0030] Performing linear dependency feature extraction on the sample sound spectrum deep vector through the long short-term memory neural network sublayer to obtain a sample linear dependency feature vector;

[0031] The sample linear dependency feature vector is nonlinearly mapped through the activation function sublayer to obtain the initial sample spectrum context vector.

[0032] In some embodiments, the target loss data includes text recognition loss data; and the loss calculation based on the sample pronunciation feature vector to obtain the target loss data includes:

[0033] Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature vector;

[0034] Determining a sample recognition text according to the text corresponding to the sample reference pronunciation feature vector;

[0035] The text recognition loss data is obtained by performing text error calculation based on the sample recognition text and the preset sample label text; wherein the sample label text is used to characterize the text corresponding to the sample speech data.

[0036] In some embodiments, the target loss data includes pronunciation recognition loss data; and the loss calculation is performed according to the sample pronunciation feature vector to obtain the target loss data, including:

[0037] Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature;

[0038] The pronunciation feature error is calculated based on the sample reference pronunciation feature and the preset sample label pronunciation feature to obtain the pronunciation recognition loss data; wherein the sample label pronunciation feature vector is used to characterize the pronunciation feature corresponding to the sample speech data.

[0039] In some embodiments, the step of performing loss calculation according to the sample pronunciation feature vector to obtain target loss data includes:

[0040] Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature vector;

[0041] Determining a sample recognition text according to the text corresponding to the sample reference pronunciation feature vector;

[0042] Calculating text error based on the sample recognition text and the preset sample label text to obtain text recognition loss data; wherein the sample label text is used to characterize the text corresponding to the sample speech data;

[0043] Pronunciation feature error calculation is performed based on the sample reference pronunciation feature and the preset sample label pronunciation feature to obtain pronunciation recognition loss data; wherein the sample label pronunciation feature vector is used to characterize the pronunciation feature corresponding to the sample speech data;

[0044] A weighted sum is performed on the text recognition loss data and the pronunciation recognition loss data to obtain target loss data.

[0045] In some embodiments, the step of performing text decoding on the target pronunciation feature vector to obtain the target text includes:

[0046] Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the target pronunciation feature vector to obtain a target reference pronunciation feature vector;

[0047] The text corresponding to the target reference pronunciation feature vector is determined as the target text.

[0048] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech text recognition device, the device comprising:

[0049] A language data acquisition module, used to acquire target speech data;

[0050] A spectrum feature extraction module is used to extract spectrum features from the target speech data to obtain target spectrum features;

[0051] A feature vector embedding module, used for embedding the target sound spectrum feature into a feature vector through the input layer of the pre-trained target acoustic language model to obtain a target sound spectrum embedding vector;

[0052] A deep feature mapping module, used for performing deep feature mapping on the target sound spectrum embedding vector through the hidden layer of the target acoustic language model to obtain a target sound spectrum deep vector;

[0053] A context feature extraction module, used for extracting context features of the target spectrum deep vector through the recurrent neural network layer of the target acoustic language model to obtain a target spectrum context vector;

[0054] A pronunciation feature mapping module, used for performing pronunciation feature mapping on the target sound spectrum context vector through the output layer of the target acoustic language model to obtain a target pronunciation feature vector;

[0055] The text decoding module is used to perform text decoding on the target pronunciation feature vector to obtain the target text.

[0056] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the speech-to-text recognition method described in the first aspect when executing the computer program.

[0057] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the speech-to-text recognition method described in the first aspect.

[0058] The speech-text recognition method, speech-text recognition device, electronic device and computer-readable storage medium proposed in the present application introduce a new model, namely a target acoustic language model, when it is necessary to recognize the target text from the target speech data. The target acoustic language model not only includes an input layer, a hidden layer and an output layer, but also has a recurrent neural network layer between the hidden layer and the output layer. By processing the target spectral features extracted from the target speech data through the target acoustic language model, a target pronunciation feature vector containing rich pronunciation information can be obtained, thereby improving the accuracy of the target text.

[0059] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a speech-to-text recognition method provided by an embodiment of the present application;

[0061] Figure 2 is a flow chart of a speech-to-text recognition method provided by another embodiment of the present application;

[0062] Figure 3 yes Figure 2 Flowchart of step 203 in

[0063] Figure 4 yes Figure 3Flow chart of step 303 in;

[0064] Figure 5 yes Figure 2 A flowchart of step 204 in FIG.

[0065] Figure 6 yes Figure 2 Another flow chart of step 204 in;

[0066] Figure 7 yes Figure 1 Flow chart of step 107 in;

[0067] Figure 8 It is a module structure block diagram of the speech text recognition device provided in the embodiment of the present application;

[0068] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] First, some nouns involved in this application are analyzed:

[0073] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0075] Speech-to-text recognition technology: is a technology that analyzes and recognizes the speaker's text / speech content from speech data. It can be used to improve the naturalness and friendliness of human-computer interaction, as well as to provide more text information and services. The main task of speech-to-text recognition is to convert speech data into corresponding text.

[0076] In the field of financial technology, speech and text recognition technology has many application scenarios. For example, users actively call in or robots make intelligent outbound calls. In the process of talking between the robot and the user, the user's call voice can be accurately converted into corresponding text through speech and text recognition technology, so that the system can more accurately identify the user's intention, reduce the error rate of filtering invalid lists or low-quality lists, and improve the contact efficiency of the agent list. However, the components of voice data are relatively complex, such as noise and background sound. The structure of conventional neural network models is relatively simple, and it is difficult to extract high-accuracy features from voice data, resulting in a large room for improvement in recognition accuracy.

[0077] Based on this, the embodiments of the present application propose a speech-to-text recognition method and a speech-to-text recognition device, an electronic device, and a computer-readable storage medium, which can improve the accuracy of emotion recognition and improve the recognition efficiency.

[0078] The speech text recognition method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech text recognition method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] The embodiments of the present application provide a speech-text recognition method and a speech-text recognition device, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the speech-text recognition method in the embodiments of the present application is described.

[0081] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] Reference Figure 1, Figure 1 It is an optional flowchart of the speech-to-text recognition method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 107.

[0083] Step 101, obtaining target voice data;

[0084] Step 102, extracting the spectral features of the target speech data to obtain the target spectral features;

[0085] Step 103, embedding the target sound spectrum features into feature vectors through the input layer of the pre-trained target acoustic language model to obtain a target sound spectrum embedding vector;

[0086] Step 104, performing deep feature mapping on the target spectrum embedding vector through the hidden layer of the target acoustic language model to obtain a target spectrum deep vector;

[0087] Step 105, extracting context features from the target spectrum deep vector through the recurrent neural network layer of the target acoustic language model to obtain a target spectrum context vector;

[0088] Step 106, performing pronunciation feature mapping on the target sound spectrum context vector through the output layer of the target acoustic language model to obtain a target pronunciation feature vector;

[0089] Step 107, text decoding is performed on the target pronunciation feature vector to obtain the target text.

[0090] Steps 101 to 107 shown in the embodiment of the present application introduce a new model, namely a target acoustic language model, when it is necessary to identify the target text from the target speech data. The target acoustic language model not only includes an input layer, a hidden layer, and an output layer, but also has a recurrent neural network layer between the hidden layer and the output layer. By processing the target spectral features extracted from the target speech data through the target acoustic language model, a target pronunciation feature vector containing rich pronunciation information can be obtained, thereby improving the accuracy of the target text.

[0091] In step 101 of some embodiments, target voice data is obtained. The target voice data is used to represent the voice uttered by a certain speaker. Specifically, when the speech-to-text recognition method is applied to a terminal, the target voice data can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the target voice data is obtained by recording, the terminal is correspondingly configured with a microphone, and audio is collected by the microphone to achieve recording of the target voice data. When the speech-to-text recognition method is applied to a server, the target voice data can be uploaded to the server by the terminal, or downloaded by the server from other servers or databases.

[0092] In step 102 of some embodiments, spectral features are extracted from the target speech data to obtain target spectral features. Spectral features refer to the distribution characteristics of sound in the frequency domain, which are usually displayed through a spectrogram. The spectrogram reflects the frequency components and intensity of the sound, which can help identify and analyze the characteristics of the sound, such as pitch, timbre, volume, and frequency components. For example, the spectral feature is Mel Frequency Cepstral Coefficents (MFCC), which can reflect the human perception characteristics of sound.

[0093] The MFCC features are calculated by the following steps: first, the target speech signal is pre-emphasized and framed; then, each frame is windowed with a Hamming window and fast Fourier transformed; then, the spectrum is filtered by a Mel filter bank and logarithmically operated; finally, the filter bank output is discrete cosine transformed to obtain the MFCC coefficients. In addition, other acoustic features such as the fundamental frequency (F0) and energy (E) of the speech signal are extracted and spliced ​​together with the MFCC features to form a richer feature vector. In this embodiment, the sampling rate can be set to 16000 Hz, the frame length to 25 ms, the frame shift to 10 ms, the number of Mel filter banks to 40, and the number of MFCC coefficients to 13.

[0094] In step 103 of some embodiments, the target spectrum feature is embedded with a feature vector through the input layer of the pre-trained target acoustic language model to obtain a target spectrum embedding vector. The target acoustic language model is a neural network model, including multiple processing layers connected in sequence. Each processing layer outputs the processed result to the next processing layer as input. In this embodiment, the target acoustic language model includes an input layer, a hidden layer, a recurrent neural network layer, and an output layer that are connected in sequence / sequentially. The input layer is the first processing layer of the target acoustic language model, which is used to receive the target speech data and map the target speech data into a target spectrum feature embedding vector. The input layer, the hidden layer, the recurrent neural network layer, and the output layer are obtained through joint training.

[0095] In step 104 of some embodiments, deep feature mapping is performed on the target sound spectrum embedding vector through the hidden layer of the target acoustic language model to obtain the target sound spectrum deep vector. The hidden layer is usually a processing layer between the first processing layer and the last processing layer of the target acoustic language model. In this embodiment, the hidden layer is specifically a processing layer after the first processing layer and before the recurrent neural network layer. There are usually more than one hidden layer, and its main function is to perform deep feature mapping.

[0096] In step 105 of some embodiments, context features are extracted from the target spectrum deep vector by the recurrent neural network layer of the target acoustic language model to obtain the target spectrum context vector. The recurrent neural network layer is very effective in processing data with sequence characteristics, and it can mine the time sequence information and semantic information in the data. The recurrent neural network layer can be a long short term memory (LSTM) network layer, also known as an LSTM layer.

[0097] In one embodiment, the recurrent neural network layer includes a long short-term memory neural network sublayer and an activation function sublayer, and step 105 may include:

[0098] The linear dependency feature of the target sound spectrum deep vector is extracted through the long short-term memory neural network sublayer to obtain the target linear dependency feature vector;

[0099] The target linear dependency feature vector is nonlinearly mapped through the activation function sublayer to obtain the target spectrogram context vector.

[0100] The benefit of the above embodiment is that the accuracy of the target sound spectrum context vector is improved by using a linear processing plus a nonlinear processing method, thereby helping to improve the accuracy of speech text recognition.

[0101] In step 106 of some embodiments, the target spectrum context vector is subjected to pronunciation feature mapping through the output layer of the target acoustic language model to obtain a target pronunciation feature vector. The output layer is the last processing layer of the target acoustic language model and is used to output the final processing result, i.e., the target pronunciation feature vector.

[0102] In step 107 of some embodiments, text decoding is performed on the target pronunciation feature vector to obtain the target text. A pre-trained text decoder may be used to perform text decoding on the target pronunciation feature vector to obtain the target text.

[0103] Text decoding is a prediction problem. The prediction problem mainly discusses the known model parameters and observation sequence, and finds the maximum state sequence under the given observation sequence. For solving the prediction problem, the Viterbi algorithm is mainly used. The idea of ​​dynamic programming is adopted, that is, how to make the nodes passed by the path the optimal path at a certain moment. If it is optimal, this path is the best path under this condition. When solving the path with the maximum probability, the corresponding state sequence must be the probability of the optimal path. In the speech text recognition task, the state sequence corresponds to the phoneme state of each sentence. Among them, the observation sequence is randomly generated by each state. Each position of the state sequence in the speech text recognition task represents the moment of a certain speech segment. And the observation sequence corresponds to a certain feature extracted from speech. The hidden Markov model is mainly represented by the initial probability, state transition probability and observation probability.

[0104] In one embodiment, referring to Figure 2 Before step 102, the speech text recognition method provided in this embodiment further includes: training the target acoustic language model, specifically including:

[0105] Step 201, obtaining sample voice data;

[0106] Step 202, extracting the spectral features of the sample speech data to obtain the spectral features of the sample;

[0107] Step 203, inputting the sample sound spectrum feature into a preset initial acoustic language model to extract pronunciation features, and obtaining a sample pronunciation feature vector; wherein the initial acoustic language model includes an input layer, a hidden layer, a recurrent neural network layer, a recurrent connection layer, and an output layer connected in sequence;

[0108] Step 204, performing loss calculation based on the sample pronunciation feature vector to obtain target loss data;

[0109] Step 205, adjusting parameters of the initial acoustic language model according to the target loss data to obtain a pseudo acoustic language model;

[0110] Step 206: Delete the recurrent connection layer in the pseudo acoustic language model to obtain the target acoustic language model.

[0111] In step 201, sample voice data is used to represent the voice uttered by a certain speaker. Specifically, when the speech-to-text recognition method is applied to a terminal, the target voice data can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the speech-to-text recognition method is applied to a server, the sample voice data can be uploaded to the server by the terminal, or downloaded by the server from another server or database.

[0112] In step 202, the explanation of the sample sound spectrum feature is basically the same as the explanation of the target sound spectrum feature in the above text, and will not be repeated here.

[0113] In step 203, the initial acoustic language model can be initialized based on the monophone and triphone HMM-GMM acoustic models. The acoustic model can obtain the state posterior through the state clustering of the triphone. The acoustic model can be composed of a hidden Markov model and a mixed Gaussian model, and its phoneme state corresponds to each mixed Gaussian model, and the pronunciation of a certain phoneme state corresponds to the parameters of k Gaussian models. The phoneme state characterizes the likelihood calculation of the input feature vector at a given state. For this probability density function based on continuous space in the acoustic model, it is necessary to use the parameters of the mixed Gaussian model to represent it. The parameters of the Gaussian distribution are composed of a mean of and a variance of, which is mainly characterized by a weighted mixed multivariate Gaussian distribution f, as shown below:

[0114]

[0115] Among them, C t Represents the mixing parameter. The solution of the mixed Gaussian model is based on the mixing parameter and the mean u k , variance ε. K refers to the Kth, and M refers to the total number of data points. T refers to transposition. Since we don’t know which Gaussian distribution the data point comes from, we can use the posterior probability method to represent it, that is, P(G|X), which means the probability that the current data point X belongs to Gaussian G, or the probability that the current data point belongs to one of the hidden variables. The acoustic model can be used to find out which hidden Markov model state the training data corresponds to, and the obtained training data is marked as align-raw, and the Viterbi algorithm is used on the training data to force the alignment state.

[0116] In one embodiment, referring to Figure 3 , step 203 may include:

[0117] Step 301, embedding the sample sound spectrum features into feature vectors through the input layer to obtain the sample sound spectrum embedding vector;

[0118] Step 302, performing deep feature mapping on the sample sound spectrum embedding vector through a hidden layer to obtain a deep vector of the sample sound spectrum;

[0119] Step 303, extracting a first context feature from the sample spectrum deep vector through a recurrent neural network layer to obtain an initial sample spectrum context vector;

[0120] Step 304, extracting a second context feature from the initial sample spectrum context vector through a recurrent connection layer to obtain a target sample spectrum context vector;

[0121] Step 305: Perform pronunciation mapping on the target sample spectrum context vector through the output layer to obtain a sample pronunciation feature vector.

[0122] Steps 301 to 302 are basically the same as the above explanations of steps 103 to 104 and will not be repeated here.

[0123] In one embodiment, the recurrent neural network layer includes a long short-term memory neural network sublayer and an activation function sublayer. Figure 4 , step 303 may include:

[0124] Step 401, extracting linear dependency features from the deep vector of the sample spectrum through the long short-term memory neural network sublayer to obtain a sample linear dependency feature vector;

[0125] Step 402: Perform nonlinear mapping on the sample linear dependency feature vector through an activation function sublayer to obtain an initial sample spectrum context vector.

[0126] Specifically, the long short-term memory network sublayer is composed of LSTM, and the activation function sublayer is composed of activation functions, such as the softmax function.

[0127] The benefit of the embodiment of the above steps 401 to 402 is that the accuracy of the initial sample spectrum context vector is improved by using a linear processing plus a nonlinear processing method, thereby helping to improve the model training effect.

[0128] In step 304, a recurrent connection layer is added between the recurrent neural network layer and the output layer, and the recurrent connection layer extracts features related to time and space from the initial sample spectrum context vector to obtain the target sample spectrum context vector. The target sample spectrum context vector can carry feature information about time and space. The recurrent connection layer can use a convolution kernel to extract the second context feature.

[0129] Step 305 is basically the same as the above explanation of step 106 and will not be repeated here.

[0130] The benefit of the above-mentioned embodiment of step 301 to step 305 is that, on the basis of introducing a recurrent neural network layer to improve the accuracy of pronunciation feature extraction, introducing a recurrent neural network layer in the training process can reduce the difficulty of model training and improve the model training effect.

[0131] In step 204, a loss calculation is performed based on the sample pronunciation feature vector to obtain target loss data. The target loss data is used to characterize the error degree of the initial acoustic language model in extracting pronunciation features. The larger the target loss data, the larger the error of the initial acoustic language model. The target loss data may include at least one of the following: text recognition loss data and pronunciation recognition loss data.

[0132] In one embodiment, referring to Figure 5 , step 204 may include:

[0133] Step 501, performing pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature vector;

[0134] Step 502, determining the sample recognition text according to the text corresponding to the sample reference pronunciation feature vector;

[0135] Step 503, performing text error calculation based on the sample recognition text and the preset sample label text to obtain text recognition loss data; wherein the sample label text is used to characterize the text corresponding to the sample speech data.

[0136] In step 501, the reference pronunciation feature vector refers to a preset pronunciation feature vector. For example, a reference text is input into a speech model for speech recognition to obtain a reference pronunciation feature vector; a pronunciation dictionary is constructed based on at least one reference pronunciation feature vector. Specifically, step 501 may include: reading reference pronunciation feature vectors from a pronunciation dictionary in sequence; performing feature similarity calculation based on a sample pronunciation feature vector and a reference pronunciation feature vector to obtain a sample pronunciation similarity; and filtering a sample reference pronunciation feature vector from the reference pronunciation feature vector based on the sample pronunciation similarity. The sample reference pronunciation feature vector is a reference pronunciation feature vector with the maximum sample pronunciation similarity or a sample pronunciation similarity greater than a sample similarity threshold.

[0137] In step 502, the text corresponding to the sample reference pronunciation feature vector may be obtained from the database to obtain the sample recognition text. The sample reference pronunciation feature vector may also be text decoded to obtain the sample recognition text.

[0138] In step 503, a common function such as a mean square error function or a similarity function may be selected to calculate the text error to obtain text recognition loss data.

[0139] The benefit of the above-mentioned embodiment of step 501 to step 503 is that the loss data is calculated based on the text corresponding to the preset reference pronunciation feature vector, and the initial acoustic language model can be trained based on the text recognition loss data.

[0140] In one embodiment, referring to Figure 6 , step 204 may include:

[0141] Step 601, performing pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature;

[0142] Step 602, calculate the pronunciation feature error based on the sample reference pronunciation feature and the preset sample label pronunciation feature to obtain pronunciation recognition loss data; wherein the sample label pronunciation feature vector is used to characterize the pronunciation feature corresponding to the sample speech data.

[0143] Step 601 is basically the same as the explanation of step 501 above, and will not be repeated here. A common function such as a mean square error function or a similarity function can be selected to calculate the pronunciation error to obtain the pronunciation recognition loss data.

[0144] The benefit of the above-mentioned embodiment of step 601 to step 602 is that it provides a method for calculating loss data and increases the diversity of model training.

[0145] In one embodiment, step 204 may include:

[0146] Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature vector;

[0147] Determining a sample recognition text according to the text corresponding to the sample reference pronunciation feature vector;

[0148] Calculate text error based on sample recognition text and preset sample label text to obtain text recognition loss data; wherein the sample label text is used to characterize the text corresponding to the sample speech data;

[0149] Pronunciation feature error is calculated based on the sample reference pronunciation feature and the preset sample label pronunciation feature to obtain pronunciation recognition loss data; wherein the sample label pronunciation feature vector is used to characterize the pronunciation feature corresponding to the sample speech data;

[0150] The target loss data is obtained by taking a weighted sum of the text recognition loss data and the pronunciation recognition loss data.

[0151] The benefit of the above embodiment is that it can balance the text recognition error and the pronunciation recognition error, and further improve the training effect of the initial acoustic language model.

[0152] In one embodiment, referring to Figure 7 , step 107 may include:

[0153] Step 701, performing pronunciation feature search on at least one preset reference pronunciation feature vector according to the target pronunciation feature vector to obtain a target reference pronunciation feature vector;

[0154] Step 702: determine the text corresponding to the target reference pronunciation feature vector as the target text.

[0155] Specifically, step 701 may include: reading out reference pronunciation feature vectors from a pronunciation dictionary in sequence; calculating feature similarity based on the target pronunciation feature vector and the reference pronunciation feature vector to obtain the target pronunciation similarity; and filtering out a target reference pronunciation feature vector from the reference pronunciation feature vector based on the target pronunciation similarity. The target reference pronunciation feature vector is a reference pronunciation feature vector with the maximum target pronunciation similarity or a target pronunciation similarity greater than a target similarity threshold. The text corresponding to the target reference pronunciation feature vector may be obtained from a database to obtain a target text. The target reference pronunciation feature vector may also be text-decoded by a decoder to obtain the target text.

[0156] The benefit of the embodiment of the above steps 701 to 702 is that the target text can be determined with the help of the reference pronunciation feature vector, thereby improving the accuracy of speech text recognition.

[0157] In summary, the present application has at least the following beneficial effects: during a conversation between a robot / client and a user, the speech-to-text recognition method provided by the present application can accurately convert the user's conversation speech into corresponding text, thereby more accurately identifying the user's intention, reducing the error rate of filtering invalid lists or low-quality lists, solving the time-consuming, labor-intensive and inefficient problems of the current seat contact list, and improving the contact efficiency of the seat list.

[0158] See also Figure 8 The embodiment of the present application also provides a speech text recognition device, which can implement the above-mentioned speech text recognition method. Figure 8This is a module structure block diagram of the speech-to-text recognition device provided in an embodiment of the present application, and the device includes: a language data acquisition module 801, a spectrum feature extraction module 802, a feature vector embedding module 803, a deep feature mapping module 804, a context feature extraction module 805, a pronunciation feature mapping module 806 and a text decoding module 807. Among them, the language data acquisition module 801 is used to acquire the target speech data; the spectral feature extraction module 802 is used to extract the spectral features of the target speech data to obtain the target spectral features; the feature vector embedding module 803 is used to embed the target spectral features into feature vectors through the input layer of the pre-trained target acoustic language model to obtain the target spectral embedding vector; the deep feature mapping module 804 is used to perform deep feature mapping on the target spectral embedding vector through the hidden layer of the target acoustic language model to obtain the target spectral deep vector; the context feature extraction module 805 is used to perform context feature extraction on the target spectral deep vector through the recurrent neural network layer of the target acoustic language model to obtain the target spectral context vector; the pronunciation feature mapping module 806 is used to perform pronunciation feature mapping on the target spectral context vector through the output layer of the target acoustic language model to obtain the target pronunciation feature vector; the text decoding module 807 is used to perform text decoding on the target pronunciation feature vector to obtain the target text.

[0159] In one embodiment, before embedding the target spectral features into feature vectors through the input layer of the pre-trained target acoustic language model to obtain the target spectral embedding vector, the speech-to-text recognition device also includes: a model training module for training the target acoustic language model, specifically including: obtaining sample speech data; extracting spectral features from the sample speech data to obtain sample spectral features; inputting the sample spectral features into a preset initial acoustic language model to extract pronunciation features to obtain a sample pronunciation feature vector; wherein the initial acoustic language model includes an input layer, a hidden layer, a recurrent neural network layer, a recurrent connection layer and an output layer connected in sequence; performing loss calculation based on the sample pronunciation feature vector to obtain target loss data; adjusting the parameters of the initial acoustic language model based on the target loss data to obtain a pseudo acoustic language model; deleting the recurrent connection layer in the pseudo acoustic language model to obtain the target acoustic language model.

[0160] It should be noted that the specific implementation of the speech-to-text recognition device is substantially the same as the specific implementation of the above-mentioned speech-to-text recognition method, and will not be described in detail here.

[0161] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned speech text recognition method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech text recognition method of the embodiment of this application;

[0165] Input / output interface 903, used to implement information input and output;

[0166] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0168] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0169] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned speech text recognition method.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The speech-text recognition method, speech-text recognition device, electronic device and computer-readable storage medium proposed in the present application introduce a new model, namely a target acoustic language model, when it is necessary to recognize the target text from the target speech data. The target acoustic language model not only includes an input layer, a hidden layer and an output layer, but also has a recurrent neural network layer between the hidden layer and the output layer. By processing the target spectral features extracted from the target speech data through the target acoustic language model, a target pronunciation feature vector containing rich pronunciation information can be obtained, thereby improving the accuracy of the target text.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for speech-to-text recognition, characterized in that: The method comprises: Obtain target voice data; Extracting the spectral features of the target speech data to obtain the target spectral features; Embedding the target sound spectrum feature with a feature vector through an input layer of a pre-trained target acoustic language model to obtain a target sound spectrum embedding vector; Performing deep feature mapping on the target sound spectrum embedding vector through the hidden layer of the target acoustic language model to obtain a target sound spectrum deep vector; Extracting context features of the target spectrum deep vector through the recurrent neural network layer of the target acoustic language model to obtain a target spectrum context vector; Performing pronunciation feature mapping on the target sound spectrum context vector through the output layer of the target acoustic language model to obtain a target pronunciation feature vector; The target pronunciation feature vector is subjected to text decoding to obtain a target text.

2. The method according to claim 1, characterized in that Before embedding the target sound spectrum feature into a feature vector through the input layer of the pre-trained target acoustic language model to obtain the target sound spectrum embedding vector, the method further includes: Training the target acoustic language model specifically includes: Obtain sample voice data; Extracting spectral features of the sample speech data to obtain spectral features of the sample speech; Inputting the sample sound spectrum feature into a preset initial acoustic language model to extract pronunciation features, and obtaining a sample pronunciation feature vector; wherein the initial acoustic language model includes an input layer, a hidden layer, a recurrent neural network layer, a recurrent connection layer and an output layer connected in sequence; Perform loss calculation based on the sample pronunciation feature vector to obtain target loss data; Adjusting parameters of the initial acoustic language model according to the target loss data to obtain a pseudo acoustic language model; The recurrent connection layer in the pseudo acoustic language model is deleted to obtain the target acoustic language model.

3. The method according to claim 2, characterized in that The step of inputting the sample sound spectrum feature into a preset initial acoustic language model to extract pronunciation features to obtain a sample pronunciation feature vector includes: Performing feature vector embedding on the sample sound spectrum feature through the input layer to obtain a sample sound spectrum embedding vector; Performing deep feature mapping on the sample sound spectrum embedding vector through the hidden layer to obtain a sample sound spectrum deep vector; Performing a first context feature extraction on the sample sound spectrum deep vector through the recurrent neural network layer to obtain an initial sample sound spectrum context vector; Performing a second context feature extraction on the initial sample spectrum context vector through the recurrent connection layer to obtain a target sample spectrum context vector; The target sample sound spectrum context vector is pronunciation mapped through the output layer to obtain the sample pronunciation feature vector.

4. The method according to claim 3, characterized in that The recurrent neural network layer includes a long short-term memory neural network sublayer and an activation function sublayer; The step of extracting a first context feature from the sample sound spectrum deep vector by the recurrent neural network layer to obtain an initial sample sound spectrum context vector includes: Performing linear dependency feature extraction on the sample sound spectrum deep vector through the long short-term memory neural network sublayer to obtain a sample linear dependency feature vector; The sample linear dependency feature vector is nonlinearly mapped through the activation function sublayer to obtain the initial sample spectrum context vector.

5. The method according to claim 2, characterized in that: The target loss data includes text recognition loss data; The loss calculation is performed according to the sample pronunciation feature vector to obtain target loss data, including: Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature vector; Determining a sample recognition text according to the text corresponding to the sample reference pronunciation feature vector; The text recognition loss data is obtained by performing text error calculation based on the sample recognition text and the preset sample label text; wherein the sample label text is used to characterize the text corresponding to the sample speech data.

6. The method according to claim 2 or 5, characterized in that: The target loss data includes pronunciation recognition loss data; The loss calculation is performed according to the sample pronunciation feature vector to obtain target loss data, including: Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the sample pronunciation feature vector to obtain a sample reference pronunciation feature; The pronunciation feature error is calculated based on the sample reference pronunciation feature and the preset sample label pronunciation feature to obtain the pronunciation recognition loss data; wherein the sample label pronunciation feature vector is used to characterize the pronunciation feature corresponding to the sample speech data.

7. The method according to any one of claims 1 to 5, characterized in that: The step of performing text decoding on the target pronunciation feature vector to obtain the target text includes: Performing a pronunciation feature search on at least one preset reference pronunciation feature vector according to the target pronunciation feature vector to obtain a target reference pronunciation feature vector; The text corresponding to the target reference pronunciation feature vector is determined as the target text.

8. A speech text recognition device, characterized in that: The device comprises: A language data acquisition module, used to acquire target speech data; A spectrum feature extraction module is used to extract spectrum features from the target speech data to obtain target spectrum features; A feature vector embedding module, used for embedding the target sound spectrum feature into a feature vector through the input layer of the pre-trained target acoustic language model to obtain a target sound spectrum embedding vector; A deep feature mapping module, used for performing deep feature mapping on the target sound spectrum embedding vector through the hidden layer of the target acoustic language model to obtain a target sound spectrum deep vector; A context feature extraction module, used for extracting context features of the target spectrum deep vector through the recurrent neural network layer of the target acoustic language model to obtain a target spectrum context vector; A pronunciation feature mapping module, used for performing pronunciation feature mapping on the target sound spectrum context vector through the output layer of the target acoustic language model to obtain a target pronunciation feature vector; The text decoding module is used to perform text decoding on the target pronunciation feature vector to obtain the target text.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images