A speech recognition method, system, storage medium and terminal device

Abstract

The embodiment of the present invention discloses a speech recognition method, system, storage medium and terminal device, which are applied to the field of information processing technology based on artificial intelligence. The speech recognition system will recognize the speech data to be recognized according to the pre-trained speech recognition model to obtain the corresponding text information, wherein, when pre-training the speech recognition model, the feature extraction module and the attention encoding module can be first trained with the second speech sample without annotations, the feature extraction module and the attention encoding module can be initialized, and then the speech recognition model is obtained by training with the first speech sample with annotations, including: a feature extraction module for secondary training, an attention encoding module for secondary training, an attention decoding module after training and a speech classification module. Since the second speech sample is not labeled, it is relatively easy to obtain, so that a large number of second speech samples can be fully utilized, thereby making the accuracy of the obtained speech recognition model relatively high.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology based on artificial intelligence, and in particular to a speech recognition method, system, storage medium and terminal device. Background Art

[0002] Nowadays, speech recognition technology has been applied to many business scenarios, such as voice search, voice interaction process or voice translation. As users' demand for voice technology-based business continues to grow, more accurate speech recognition technology is needed to meet user needs.

[0003] In the existing speech recognition technology, speech recognition can be performed through a speech recognition model based on artificial intelligence, and then the results of speech recognition can be applied to various business scenarios. Among them, the speech recognition model is trained by a certain training method. Specifically, a large number of labeled training samples are required to train the speech recognition model. However, the labeled training samples are limited, so the accuracy of the trained speech recognition model is also limited to a certain range. Summary of the invention

[0004] The embodiments of the present invention provide a speech recognition method, system, storage medium and terminal device, which improve the accuracy of speech recognition.

[0005] An embodiment of the present invention provides a speech recognition method, comprising:

[0006] Obtaining voice data to be recognized;

[0007] Calling a pre-trained speech recognition model to recognize the speech data to be recognized, and obtaining corresponding text information;

[0008] Wherein, during pre-training, the speech recognition model:

[0009] Determine a training sample, wherein the training sample includes: a first speech sample with an annotation and a second speech sample without an annotation;

[0010] Determine an initial model for speech recognition, wherein the initial model for speech recognition includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module, and a speech classification module;

[0011] The feature extraction module is used to extract the speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain the encoded features, the attention decoding module is used to decode according to the encoded features to obtain the decoded features, and the speech classification module is used to perform speech classification on any speech sample according to the decoded features to obtain the text information corresponding to any speech sample;

[0012] According to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain a first-trained feature extraction module and a first-trained attention encoding module;

[0013] The speech recognition model is trained according to the first speech sample, the feature extraction module trained for the first time, and the attention encoding module, attention decoding module and speech classification module trained for the first time. The speech recognition model includes: a feature extraction module trained for the second time, an attention encoding module trained for the second time, an attention decoding module and a speech classification module after training.

[0014] An embodiment of the present invention provides a speech recognition system, including:

[0015] A data acquisition unit, used to acquire speech data to be recognized;

[0016] A recognition unit, used to call a pre-trained speech recognition model to recognize the speech data to be recognized, and obtain corresponding text information;

[0017] Wherein, the speech recognition model is pre-trained by a training unit:

[0018] The training unit is used to determine a training sample, wherein the training sample includes: a first speech sample with annotations and a second speech sample without annotations; determine an initial speech recognition model, wherein the initial speech recognition model includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module and a speech classification module; wherein the feature extraction module is used to extract speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain an encoded feature, the attention decoding module is used to decode according to the encoded feature to obtain a decoded feature, and the speech classification module is used to decode according to the encoded feature to obtain a decoded feature. After decoding, the features are used to perform speech classification on any of the speech samples to obtain text information corresponding to any of the speech samples; according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain the feature extraction module trained for the first time and the attention encoding module trained for the first time; according to the first speech sample, the feature extraction module trained for the first time and the attention encoding module trained for the first time, the attention decoding module and the speech classification module, the speech recognition model is trained to obtain the speech recognition model, which includes: the feature extraction module trained for the second time, the attention encoding module trained for the second time, the attention decoding module after training and the speech classification module.

[0019] Another aspect of the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the speech recognition method as described in one aspect of the embodiment of the present invention.

[0020] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0021] The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the speech recognition method as described in one aspect of an embodiment of the present invention; the processor is used to implement each computer program in the multiple computer programs.

[0022] It can be seen that in the method of this embodiment, the speech recognition system will recognize the speech data to be recognized according to the pre-trained speech recognition model to obtain the corresponding text information, wherein, when pre-training the speech recognition model, the feature extraction module and the attention encoding module can be first trained with the second speech sample without annotations, the feature extraction module and the attention encoding module can be initialized, and then the speech recognition model is obtained by training with the first speech sample with annotations, including: a feature extraction module for secondary training, an attention encoding module for secondary training, an attention decoding module after training, and a speech classification module. Since the second speech sample is not labeled, it is relatively easy to obtain, so that a large number of second speech samples can be fully utilized in training the speech recognition model, thereby making the accuracy of the obtained speech recognition model relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 is a schematic diagram to which a speech recognition method provided by an embodiment of the present invention is applied;

[0025] Figure 2 is a flow chart of a speech recognition method provided by one embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of the logical structure of an initial model of speech recognition determined in one embodiment of the present invention;

[0027] Figure 4 is a flow chart of a method for first training a feature extraction module and an attention encoding module in one embodiment of the present invention;

[0028] Figure 5 is a flow chart of a method for training a speech recognition model in one embodiment of the present invention;

[0029] Figure 6 It is a schematic diagram of the logical structure of the initial model of speech recognition determined in an application embodiment of the present invention;

[0030] Figure 7 is a schematic diagram of self-supervising training of a feature extraction module and an attention encoding module in an application embodiment of the present invention;

[0031] Figure 8 is a schematic diagram of performing speech recognition in an application embodiment of the present invention;

[0032] Fig. 9 is a schematic diagram of a distributed system to which a speech recognition method in another application embodiment of the present invention is applied;

[0033] Fig.10 is a schematic diagram of a block structure in another application embodiment of the present invention;

[0034] Fig.11 is a schematic diagram of the logical structure of a speech recognition system provided by an embodiment of the present invention;

[0035] Fig.12 It is a schematic diagram of the logical structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] The embodiment of the present invention provides a speech recognition method, which is mainly a method performed by a speech recognition system, such as Figure 1 As shown, the speech recognition system can implement speech recognition in the following steps:

[0039] Acquire speech data to be recognized; call a pre-trained speech recognition model to recognize the speech data to be recognized, and obtain corresponding text information;

[0040] Wherein, during pre-training, the speech recognition model:

[0041] Determine a training sample, wherein the training sample includes: a first speech sample with annotations and a second speech sample without annotations; determine an initial speech recognition model, wherein the initial speech recognition model includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module and a speech classification module; wherein the feature extraction module is used to extract speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain an encoded feature, the attention decoding module is used to decode according to the encoded feature to obtain a decoded feature, and the speech classification module is used to perform speech classification on any speech sample according to the decoded feature to obtain text information corresponding to any speech sample;

[0042] According to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain a first-trained feature extraction module and a first-trained attention encoding module;

[0043] The speech recognition model is trained according to the first speech sample, the feature extraction module trained for the first time, and the attention encoding module, attention decoding module and speech classification module trained for the first time. The speech recognition model includes: a feature extraction module trained for the second time, an attention encoding module trained for the second time, an attention decoding module and a speech classification module after training.

[0044] In practical applications, the speech recognition system of the embodiment of the present invention can be applied to any application system. In one case, the application system includes an application terminal, and the above-mentioned speech recognition system is set in the application terminal; in another case, the application system includes an application terminal and an application background. The user can initiate the speech recognition process through the application terminal, and the above-mentioned speech recognition system is set in the application background, and the speech recognition is implemented by the application background.

[0045] Among them, application terminals may include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc., and the applications involved in the application system may include but are not limited to the following applications: music, radio, songs, videos, audio, multimedia, refueling, charging, parking, speed, energy consumption, itinerary sharing, sharing itinerary, advertising, parking spaces, audiobooks and travel applications.

[0046] The above-mentioned speech recognition model is a machine learning model based on artificial intelligence. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0047] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0049] It can be seen that since the second speech samples are not labeled, they are relatively easy to obtain. In this way, a large number of second speech samples can be fully utilized in training the speech recognition model, thereby making the obtained speech recognition model more accurate.

[0050] The embodiment of the present invention provides a speech recognition method, which is mainly a method performed by the above-mentioned speech recognition system. The flow chart is as follows: Figure 2 As shown, including:

[0051] Step 101: Acquire speech data to be recognized.

[0052] It can be understood that the speech recognition system can provide an interactive interface with the user, so that the user can input speech data through the interactive interface, and the speech recognition system will use the speech data input by the user as the speech data to be recognized and initiate the speech recognition process of the embodiment of the present invention.

[0053] Step 102: Call the pre-trained speech recognition model to recognize the speech data to be recognized, and obtain the corresponding text information.

[0054] The speech recognition model is a machine learning model. The speech recognition system can be trained in advance by a certain training method, and the relevant information of the trained speech recognition model, such as the operation logic, is preset in the speech recognition system. When the speech recognition system initiates the process of this embodiment, the speech recognition model preset in the system can be directly called.

[0055] Specifically, the speech recognition system can implement the following steps when pre-training the speech recognition model:

[0056] Step 201: determine a training sample, where the training sample includes: a first speech sample with annotations and a second speech sample without annotations.

[0057] Among them, any first voice sample may include a segment of voice data and annotation information of the segment of voice data, and the annotation information is mainly text information corresponding to the segment of voice data, while any second voice sample only includes a segment of voice data.

[0058] Furthermore, after the training samples are determined, each speech sample can be preprocessed, and the following steps can be performed on the preprocessed training samples, wherein the preprocessing of each speech sample may include but is not limited to the following processing: noise reduction, sound source location, echo elimination, etc. of the speech samples, so as to avoid the influence of factors such as noise in the speech samples on the training speech recognition model.

[0059] Step 202, determine the initial model of speech recognition, such as Figure 3 As shown, the initial model of speech recognition includes: a feature extraction module 10, a feature discretization module 11, an attention encoding module 12, an attention decoding module 13 and a speech classification module 14.

[0060] Specifically, the speech recognition system will determine the multi-layer structure included in the speech recognition initial model and the initial values ​​of the parameters in each layer of the structure. Among them, the parameters of the speech recognition initial model refer to the fixed parameters used in the calculation process of each layer of the speech recognition initial model, which do not need to be assigned at any time, such as parameter scale, number of network layers, user vector length and other parameters.

[0061] Among them, the feature extraction module 10 in the initial speech recognition model determined by the speech recognition system is used to extract the speech feature information of any speech sample, the feature discretization module 11 is used to discretize the speech feature information extracted by the feature extraction module 10, the attention encoding module 12 is used to encode the context information of the speech feature information extracted by the feature extraction module 10 to obtain the encoded features, the attention decoding module 13 is used to decode the encoded features obtained by the attention encoding module 12 to obtain the decoded features, and the speech classification module 14 is used to perform speech classification on any speech sample according to the decoded features obtained by the attention decoding module 13 to obtain the text information corresponding to the speech sample.

[0062] In a specific implementation, the initial speech recognition model determined by the speech recognition system may be a connectionist temporal classification (CTC) / attention model, etc.

[0063] Step 203, according to the second speech sample, the feature extraction module 10, the feature discretization module 11 and the attention encoding module 12 in the initial speech recognition model, train the feature extraction module 10 and the attention encoding module 12 to obtain the first trained feature extraction module and the first trained attention encoding module.

[0064] Step 204, based on the first speech sample, the feature extraction module trained for the first time and the attention encoding module trained for the first time, the attention decoding module 13 and the speech classification module 14, a speech recognition model is trained, and the speech recognition model includes: a feature extraction module trained for the second time, an attention encoding module trained for the second time, a trained attention decoding module and a speech classification module.

[0065] It can be seen that in the method of this embodiment, the speech recognition system will recognize the speech data to be recognized according to the pre-trained speech recognition model to obtain the corresponding text information, wherein, when pre-training the speech recognition model, the feature extraction module and the attention encoding module can be first trained with the second speech sample without annotations, the feature extraction module and the attention encoding module can be initialized, and then the speech recognition model is obtained by training with the first speech sample with annotations, including: a feature extraction module for secondary training, an attention encoding module for secondary training, an attention decoding module after training, and a speech classification module. Since the second speech sample is not labeled, it is relatively easy to obtain, so that a large number of second speech samples can be fully utilized in training the speech recognition model, thereby making the accuracy of the obtained speech recognition model relatively high.

[0066] In a specific embodiment, Figure 4As shown, when the speech recognition system executes the above step 203, the training of the feature extraction module 10 and the attention encoding module 12 can be realized through the following steps, which is mainly a self-supervised training process:

[0067] Step 2031, determining the encoded features of the second speech sample through the feature extraction module 10 and the attention encoding module 12, and determining the discretized features of the second speech sample through the feature extraction module 10 and the feature discretization module 11.

[0068] Step 2032, calculate the first loss function related to the feature extraction module 10 and the attention encoding module 12 according to the encoded features and discretized features of the second speech sample.

[0069] Specifically, since each second speech sample includes speech data of multiple continuous moments, the discretized features of the second speech sample obtained by the feature discretization module 11 may also include discretized features of multiple moments involved in the second speech sample, and the encoded features of the second speech sample obtained by the attention encoding module 12 include the encoded features corresponding to these multiple moments. Then the first loss function calculated by the speech recognition system may include: the correlation between the discretized features corresponding to any moment in the multiple moments and the encoded features.

[0070] Among them, the discretized features obtained by the feature discretization module 11 are mainly used to supervise the encoded features obtained by the feature extraction module 10 and the attention encoding module 12, so as to train the feature extraction module 10 and the attention encoding module 12.

[0071] Specifically, the first loss function can be expressed by the following formula 1:

[0072]

[0073] Among them, L m is the first loss function, c t is the encoded feature at time t in any second speech sample, q t is the discretization feature at time t, is the encoded feature of all moments in any second speech sample, K is a hyperparameter, Q t is the number of time instants involved in any second speech sample; and function sim(a,b)=a T b / ||a||||b|| is used to represent the correlation between two vectors a and b.

[0074] Step 2033, adjust the parameter values ​​of the parameters in the feature extraction module 10 and the attention encoding module 12 according to the first loss function to obtain the first trained feature extraction module and the first trained attention encoding module.

[0075] Here, the process of training the feature extraction module 10 and the attention encoding module 12 is to minimize the value of the above-mentioned first loss function. The training process continuously optimizes and adjusts the initial values ​​of the parameters in the feature extraction module 10 and the attention encoding module 12 determined in the above-mentioned step 202 through a series of mathematical optimization methods such as back propagation derivation and gradient descent, and reduces the calculated value of the above-mentioned first loss function to the minimum.

[0076] It should be noted that the above steps 2031 to 2033 are an adjustment of the parameter values ​​in the feature extraction module 10 and the attention encoding module 12 by respectively obtaining the encoded features and discretized features through the attention encoding module 12 and the feature discretization module 11. In actual applications, it is necessary to continuously loop through the above steps 2031 to 2033 until the adjustment of the parameter values ​​meets certain stop conditions.

[0077] Therefore, after executing steps 2031 to 2033 of the above embodiment, the speech recognition system also needs to determine whether the current adjustment of the parameter value meets the preset stop condition. If it does, the parameter value adjusted in the above step 2033 is used as the parameter value in the first trained feature extraction module and the first trained attention coding module, that is, the feature extraction module and the attention coding module are initialized; if it does not meet the condition, the above steps 2031 to 2033 are returned to be executed after the parameter value is adjusted. The preset stop condition includes but is not limited to any one of the following conditions: the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold value, that is, the adjusted parameter value reaches convergence; and the number of adjustments to the parameter values ​​in the feature extraction module 10 and the attention coding module 12 is equal to the preset number of times, etc.

[0078] In another specific embodiment, Figure 5 As shown, when the speech recognition system executes the above step 204, the speech recognition model can be trained through the following steps:

[0079] Step 2041, determining the text information corresponding to the first voice sample through the first trained feature extraction module, the first trained attention encoding module, the attention decoding module 13 and the voice classification module 14.

[0080] Step 2042, calculating a second loss function related to the speech recognition model according to the text information corresponding to the first speech sample determined in the above step 2041 and the annotation information corresponding to the first speech sample in the training sample.

[0081] Specifically, the speech recognition system will first calculate the second loss function related to the speech recognition model based on the text information corresponding to the first speech sample determined in the above step 2041 and the annotation information of the corresponding first speech sample in the training sample. The second loss function is used to indicate the error between the text of the first speech sample determined by the feature extraction module trained for the first time, the attention encoding module trained for the first time, the attention decoding module 13 and the speech classification module 14, and the actual text information of each first speech sample (obtained according to the annotation information of the first speech sample), such as the cross entropy loss function.

[0082] Step 2043, adjusting the parameter values ​​of the parameters in the attention decoding module 13 and the speech classification module 14 according to the second loss function to obtain a speech recognition model.

[0083] Specifically, the training process of the speech recognition model is to minimize the value of the above-mentioned second loss function. The training process continuously optimizes the parameter values ​​of the parameters in the attention decoding module 13 and the speech classification module 14 determined in the above-mentioned step 202 through a series of mathematical optimization methods such as back propagation derivation and gradient descent, and reduces the calculated value of the above-mentioned second loss function to the minimum.

[0084] It should be noted that the above steps 2041 to 2043 are an adjustment of the parameter values ​​in the attention decoding module 13 and the speech classification module 14 based on the text information of the first speech sample determined by the feature extraction module trained for the first time, the attention encoding module trained for the first time, the attention decoding module 13 and the speech classification module 14. In actual applications, it is necessary to continuously loop through the above steps 2041 to 2043 until the adjustment of the parameter values ​​meets certain stop conditions.

[0085] Therefore, after executing steps 2041 to 2043 of the above embodiment, the speech recognition system needs to determine whether the current adjustment of the parameter value meets the preset stop condition. If it meets the condition, the parameter value adjusted in step 2043 is used as the parameter value in the speech recognition model finally trained; if it does not meet the condition, the above steps 2041 to 2043 are returned to be executed after the parameter value is adjusted. The preset stop condition includes but is not limited to any one of the following conditions: the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold value, that is, the adjusted parameter value reaches convergence; and the number of parameter value adjustments is equal to the preset number of times, etc.

[0086] It should also be noted that in all loops of adjusting the parameter values ​​of the parameters in the attention decoding module 13 and the speech classification module 14, in the first n (n is a natural number greater than or equal to 1, and its value can be preset in the system) loops, the parameter values ​​in the above-mentioned first trained feature extraction module and the first trained attention encoding module can be kept unchanged, and only the parameter values ​​in the attention decoding module 13 and the speech classification module 14 are updated; and in the subsequent multiple loops, the parameter values ​​in the first trained feature extraction module, the first trained attention encoding module, the attention decoding module and the speech classification model are adjusted.

[0087] The following is a specific application example to illustrate the speech recognition method of the present invention. The method of the embodiment of the present invention mainly includes the following two parts:

[0088] (I) Training of speech recognition model

[0089] In the embodiment of the present invention, the speech recognition model can be trained as described in the above embodiment, which will not be described in detail here. The difference is that in this embodiment:

[0090] (1) When the speech recognition system determines the training sample, the first annotated speech sample used can be the open source Chinese speech recognition dataset Aishell-1, which includes a training set, a validation set, and a test set. The second unannotated speech sample is 960 hours of librispeech data, as shown in Table 1 below:

[0091] Second voice sample Number of items Aishell-1 training set 120098 Aishell-1 validation set 14326 Aishell-1 test set 7176

[0092] Table 1

[0093] The speech recognition initial model of (2) can specifically adopt the CTC / attention initial model, such as Figure 6 As shown, the feature extraction module 10 can be specifically composed of multiple layers (Convolutional Neural Network, CNN). For example, a 7-layer CNN network can be used, each CNN network layer has 512 channels, the corresponding step sizes are (5, 2, 2, 2, 2, 2, 2), and the corresponding convolution kernel sizes are (10, 3, 3, 3, 3, 2, 2). The feature extraction module 10 can be a spectrum after short-time Fourier transformation, and can also further enhance the Mel frequency cepstral coefficient (MFCC) features, etc.

[0094] The attention encoding module 12 determined by the speech recognition system can be specifically composed of M transformer sublayers, each sublayer includes a self-attention structure and a feed forward structure. For example, the attention encoding module 12 can use a 12-layer transformer, the model dimension is 768, the internal fully connected layer dimension is 3072, and the self attention can use an 8-head multi-head self-attention structure. The attention decoding module 13 determined by the speech recognition system can be specifically composed of N (for example, N is 6) sublayers, each sublayer includes a selfattention structure, a cross attention structure and a feed forward structure.

[0095] Furthermore, in each sublayer of the attention encoding module 12 and the attention decoding module 13, a module can be added after the self-attention structure, the cross-attention structure and the feedforward structure, that is, Add or Norm can mainly include normalization processing.

[0096] The speech classification module 14 determined by the speech recognition system can specifically use a linear layer as a linear classifier, and the output of the linear layer is subjected to softmax to obtain a probability prediction distribution.

[0097] (3) When the speech recognition system performs self-supervisory training based on the unlabeled second speech sample, such as Figure 7 As shown, the unlabeled second speech sample X outputs speech feature information Z after passing through the feature extraction module 10. The speech feature information Z respectively passes through the feature discretization module 11 and the attention encoding module 12, and outputs the discretized feature Q and the encoded feature C respectively. Then, the first loss function L is calculated, and the parameter values ​​in the feature extraction module 10 and the attention encoding module 12 are adjusted based on the first loss function L.

[0098] In self-supervised training, 400k iterations can be trained. The first 8% of iterations use the warming up learning rate, the maximum learning rate is 0.005, and the learning rate of subsequent iterations will decrease linearly. After training, the feature extraction module of the first training and the attention encoding module of the first training can be obtained.

[0099] (4) When the speech recognition system performs supervised training based on the first annotated speech sample, the first annotated speech sample passes through the first trained feature extraction module, the first trained attention encoding module, the attention decoding module 13 and the speech classification module 14, and then obtains the text information corresponding to the first speech sample, and then calculates the second loss function, and adjusts the parameter values ​​in the attention decoding module 13 and the speech classification module 14 based on the second loss function.

[0100] In this training process, the parameter values ​​in the feature extraction module and the attention encoding module of the first training are kept unchanged in the first 10,000 iterations, and only the parameter values ​​in the attention decoding module 13 and the speech classification module 14 are adjusted; starting from the 10,000th iteration, the parameter values ​​of all networks in the speech recognition initial module will be adjusted, and the initial learning rate is 0.00002.

[0101] The speech recognition model trained by the above method may specifically include: a feature extraction module of secondary training, an attention encoding module of secondary training, a trained attention decoding module and a speech classification module, and then the information of the trained speech recognition model is preset into the speech recognition system.

[0102] (2) Recognize the voice data.

[0103] In practical applications, the above-mentioned voice recognition system can be applied to any application scenario. For example, the above-mentioned voice recognition system can be applied to scenarios such as vehicle-connected network voice interaction. Specifically, the voice recognition system is set in the vehicle-connected terminal, which can realize functions such as waking up the engine cloud, chatting, vehicle control, and weather knowledge query through the voice data input by the user.

[0104] In this way, Figure 8 As shown in the figure, the car connection terminal will collect the user's voice data through the microphone array. The collected voice data can be processed by the acoustic front-end algorithm, such as noise / reverberation suppression, echo cancellation and sound source localization, and the processed voice data is used as the voice data to be recognized, and the preset voice recognition model in the system is called to recognize the voice data to be recognized through the voice recognition model to obtain the corresponding text information. Then, other operations can be continued based on the recognized text information, such as related searches based on the words in the text information.

[0105] Further, when the speech recognition model 1 is obtained by training according to the annotated speech sample in the prior art, and the speech recognition model 2 is obtained by training according to the first speech sample and the second speech sample in the embodiment of the present invention, the speech recognition model 1 and the speech recognition model 2 are used to perform speech recognition on the speech data in the test set respectively, and the model measurement parameters, such as the word error rate (%), are calculated, which can be shown in the following Table 2:

[0106] Dataset Prior art methods Method of the embodiment of the present invention AIshell-1 validation set 5.9 5.4 AIshell-1 test set 6.5 5.6

[0107] Table 2

[0108] It can be seen that since the speech recognition model is obtained by initializing the feature extraction module and the attention encoding module using the unlabeled second speech sample in the embodiment of the present invention, the relative decrease rate of the word error rate of the speech recognition model reaches 13.8%. This shows that the accuracy of the speech recognition model obtained by training using the method in the embodiment of the present invention is improved, achieving better results, and effectively utilizing a large amount of unlabeled speech data.

[0109] The following is another specific application example to illustrate the speech recognition method in the present invention. The speech recognition system in the embodiment of the present invention is mainly a distributed system 100, which may include a client 300 and multiple nodes 200 (any form of computing device in the access network, such as a server, a user terminal), and the client 300 and the node 200 are connected in the form of network communication.

[0110] Take the distributed system as the blockchain system as an example, see Fig. 9 This is an optional structural diagram of the distributed system 100 provided by the embodiment of the present invention applied to the blockchain system, which is formed by multiple nodes 200 (any form of computing devices in the access network, such as servers and user terminals) and clients 300, and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0111] See also Fig. 9 The functions of each node in the blockchain system shown include:

[0112] 1) Routing: a basic function of a node, used to support communication between nodes.

[0113] In addition to the routing function, the node can also have the following functions:

[0114] 2) Applications, which are deployed in the blockchain to implement specific businesses according to actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to the temporary block when they successfully verify the source and integrity of the record data.

[0115] For example, the business implemented by the application includes the code for implementing the speech recognition function, which mainly includes:

[0116] Acquire speech data to be recognized; call a pre-trained speech recognition model to recognize the speech data to be recognized, and obtain corresponding text information;

[0117] Among them, during pre-training, the speech recognition model: determines training samples, which include: a first speech sample with annotations and a second speech sample without annotations; determines an initial speech recognition model, which includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module and a speech classification module; wherein the feature extraction module is used to extract speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain encoded features, the attention decoding module is used to decode according to the encoded features to obtain decoded features, and the speech classification module is used to According to the decoded features, speech classification is performed on any of the speech samples to obtain text information corresponding to the any of the speech samples; according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain the first trained feature extraction module and the first trained attention encoding module; according to the first speech sample, the first trained feature extraction module and the first trained attention encoding module, the attention decoding module and the speech classification module, the speech recognition model is trained to obtain the speech recognition model, which includes: a second trained feature extraction module, a second trained attention encoding module, a trained attention decoding module and a speech classification module.

[0118] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0119] See also Fig.10An optional schematic diagram of the block structure provided by the embodiment of the present invention, each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block, and each block is connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains relevant information for verifying the validity of its information (anti-counterfeiting) and generating the next block.

[0120] The embodiment of the present invention also provides a speech recognition system, the structural diagram of which is shown in FIG. Fig.11 As shown, it may specifically include:

[0121] A data acquisition unit 20, used to acquire speech data to be recognized;

[0122] The recognition unit 21 is used to call the speech recognition model pre-trained by the training unit 22 to recognize the speech data to be recognized acquired by the data acquisition unit 20 to obtain corresponding text information;

[0123] The speech recognition model is pre-trained by the training unit 22:

[0124] The training unit 22 is used to determine the training samples, which include: a first speech sample with annotations and a second speech sample without annotations; determine the initial speech recognition model, which includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module and a speech classification module; wherein the feature extraction module is used to extract the speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain the encoded features, the attention decoding module is used to decode according to the encoded features to obtain the decoded features, and the speech classification module is used to decode according to the encoded features to obtain the decoded features. The decoded features are used to perform speech classification on any of the speech samples to obtain text information corresponding to any of the speech samples; the feature extraction module and the attention encoding module are trained according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model to obtain the feature extraction module trained for the first time and the attention encoding module trained for the first time; the speech recognition model is trained according to the first speech sample, the feature extraction module trained for the first time and the attention encoding module trained for the first time, the attention decoding module and the speech classification module, and the speech recognition model includes: the feature extraction module trained for the second time, the attention encoding module trained for the second time, the attention decoding module after training and the speech classification module.

[0125] On the one hand, when the training unit 22 trains the feature extraction module and the attention encoding module according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, it is specifically used to determine the encoded features of the second speech sample through the feature extraction module and the attention encoding module, and determine the discretized features of the second speech sample through the feature extraction module and the feature discretization module; calculate the first loss function related to the feature extraction module and the attention encoding module according to the encoded features and discretized features of the second speech sample; adjust the parameter values ​​of the parameters in the feature extraction module and the attention encoding module according to the first loss function to obtain the feature extraction module trained for the first time and the attention encoding module trained for the first time.

[0126] Among them, the discretized features of the second speech sample include discretized features at multiple moments, and the encoded features include encoded features corresponding to the multiple moments; then the training unit 22 calculates the first loss function including: the correlation between the discretized features corresponding to any moment in the multiple moments and the encoded features.

[0127] Furthermore, when the training unit 22 trains the feature extraction module and the attention encoding module for the first time, when the number of adjustments to the parameter values ​​in the feature extraction module and the attention encoding module is equal to a preset number, or if the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, the adjustment of the fixed parameter value is stopped.

[0128] On the other hand, when the training unit 22 trains the speech recognition model based on the first speech sample, the feature extraction module trained for the first time, and the attention encoding module, attention decoding module and speech classification module trained for the first time, it is specifically used to determine the text information corresponding to the first speech sample through the feature extraction module trained for the first time, the attention encoding module trained for the first time, the attention decoding module and the speech classification module; calculate the second loss function related to the speech recognition model based on the text information corresponding to the determined first speech sample and the annotation information corresponding to the first speech sample in the training sample; adjust the parameter values ​​of the parameters in the attention decoding module and the speech classification model according to the second loss function to obtain the speech recognition model.

[0129] Furthermore, when the training unit 22 is training the speech recognition model, when the number of adjustments to the parameter values ​​in the attention decoding module and the speech classification model is equal to a preset number, or if the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, the adjustment of the fixed parameter value is stopped.

[0130] Among them, in all loops of adjusting the parameter values ​​in the attention decoding module and the speech classification model, the training unit 22 keeps the parameter values ​​in the feature extraction module and the attention encoding module of the first training unchanged in the first n loops, and only adjusts the parameter values ​​in the attention decoding module and the speech classification model; n is a natural number greater than or equal to 1; in multiple loops after the first n loops, the parameter values ​​in the feature extraction module, the attention encoding module, the attention decoding module and the speech classification model of the first training are adjusted.

[0131] In the system of this embodiment, the recognition unit 21 will recognize the speech data to be recognized according to the pre-trained speech recognition model to obtain the corresponding text information, wherein the training unit 22 can first use the unlabeled second speech sample to perform the first training on the feature extraction module and the attention encoding module when pre-training the speech recognition model, initialize the feature extraction module and the attention encoding module, and then use the labeled first speech sample to train to obtain the speech recognition model, including: a second-trained feature extraction module, a second-trained attention encoding module, a trained attention decoding module and a speech classification module. Since the second speech sample is unlabeled, it is relatively easy to obtain, so that a large number of second speech samples can be fully utilized, thereby making the accuracy of the obtained speech recognition model relatively high.

[0132] The embodiment of the present invention further provides a terminal device, the structural diagram of which is shown in FIG. Fig.12 As shown, the terminal device may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 30 (for example, one or more processors) and memory 31, and one or more storage media 32 (for example, one or more mass storage devices) storing application programs 321 or data 322. Among them, the memory 31 and the storage medium 32 can be short-term storage or permanent storage. The program stored in the storage medium 32 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the terminal device. Furthermore, the central processing unit 30 can be configured to communicate with the storage medium 32 to execute a series of instruction operations in the storage medium 32 on the terminal device.

[0133] Specifically, the application 321 stored in the storage medium 32 includes a speech recognition application, and the program may include the data acquisition unit 20, the recognition unit 21 and the training unit 22 in the above-mentioned speech recognition system, which will not be described in detail here. Furthermore, the central processing unit 30 can be configured to communicate with the storage medium 32 and execute a series of operations corresponding to the speech recognition application stored in the storage medium 32 on the terminal device.

[0134] The terminal device may also include one or more power supplies 33, one or more wired or wireless network interfaces 34, one or more input and output interfaces 35, and / or one or more operating systems 323, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0135] The steps performed by the speech recognition system in the above method embodiment can be based on the Fig.12The structure of the terminal device shown.

[0136] Furthermore, another aspect of the embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the speech recognition method executed by the above-mentioned speech recognition system.

[0137] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0138] The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the speech recognition method executed by the above-mentioned speech recognition system; the processor is used to implement each computer program in the multiple computer programs.

[0139] In addition, according to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the speech recognition method provided in the above various optional implementations.

[0140] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0141] The above is a detailed introduction to a speech recognition method, system, storage medium and terminal device provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A speech recognition method, characterized in that: include: Obtaining voice data to be recognized; Calling a pre-trained speech recognition model to recognize the speech data to be recognized, and obtaining corresponding text information; Wherein, during pre-training, the speech recognition model: Determine a training sample, wherein the training sample includes: a first speech sample with an annotation and a second speech sample without an annotation; Determine an initial model for speech recognition, wherein the initial model for speech recognition includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module, and a speech classification module; The feature extraction module is used to extract the speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain the encoded features, the attention decoding module is used to decode according to the encoded features to obtain the decoded features, and the speech classification module is used to perform speech classification on any speech sample according to the decoded features to obtain the text information corresponding to any speech sample; According to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain a first-trained feature extraction module and a first-trained attention encoding module; The speech recognition model is trained according to the first speech sample, the feature extraction module trained for the first time, and the attention encoding module, attention decoding module and speech classification module trained for the first time. The speech recognition model includes: a feature extraction module trained for the second time, an attention encoding module trained for the second time, an attention decoding module and a speech classification module after training.

2. The method according to claim 1, characterized in that The step of training the feature extraction module and the attention encoding module according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model specifically includes: Determine the encoded features of the second speech sample through the feature extraction module and the attention encoding module, and determine the discretized features of the second speech sample through the feature extraction module and the feature discretization module; Calculating a first loss function associated with the feature extraction module and the attention encoding module according to the encoded features and discretized features of the second speech sample; Adjust parameter values ​​of parameters in the feature extraction module and the attention encoding module according to the first loss function to obtain a first trained feature extraction module and a first trained attention encoding module.

3. The method according to claim 2, characterized in that The discretized features of the second speech sample include discretized features at multiple moments, and the encoded features include encoded features corresponding to the multiple moments; Then, based on the encoded features and discretized features of the second speech sample, the first loss function related to the feature extraction module and the attention encoding module is calculated, specifically including: calculating the first loss function includes: the correlation between the discretized features corresponding to any of the multiple moments and the encoded features.

4. The method according to claim 2, characterized in that When the number of adjustments to the parameter values ​​in the feature extraction module and the attention encoding module is equal to a preset number, or if the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, the adjustment of the fixed parameter value is stopped.

5. The method according to any one of claims 1 to 4, characterized in that: The training of the speech recognition model according to the first speech sample, the first trained feature extraction module, and the first trained attention encoding module, attention decoding module, and speech classification module specifically includes: Determine text information corresponding to the first voice sample through the first trained feature extraction module, the first trained attention encoding module, the attention decoding module and the voice classification module; Calculating a second loss function associated with the speech recognition model according to the determined text information corresponding to the first speech sample and the annotation information corresponding to the first speech sample in the training sample; The parameter values ​​of the parameters in the attention decoding module and the speech classification model are adjusted according to the second loss function to obtain the speech recognition model.

6. The method according to claim 5, characterized in that When the number of adjustments to the parameter values ​​in the attention decoding module and the speech classification model is equal to a preset number, or if the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, the adjustment of the fixed parameter value is stopped.

7. The method according to claim 6, characterized in that In all cycles of adjusting the parameter values ​​in the attention decoding module and the speech classification model, in the first n cycles, the parameter values ​​in the first trained feature extraction module and the first trained attention encoding module are kept unchanged, and only the parameter values ​​in the attention decoding module and the speech classification model are adjusted; wherein n is a natural number greater than or equal to 1; In multiple cycles after the first n cycles, the parameter values ​​in the first trained feature extraction module, the first trained attention encoding module, the attention decoding module and the speech classification model are adjusted.

8. A speech recognition system, characterized in that: include: A data acquisition unit, used to acquire speech data to be recognized; A recognition unit, used to call a pre-trained speech recognition model to recognize the speech data to be recognized, and obtain corresponding text information; Wherein, the speech recognition model is pre-trained by a training unit: The training unit is used to determine a training sample, wherein the training sample includes: a first speech sample with annotations and a second speech sample without annotations; determine an initial speech recognition model, wherein the initial speech recognition model includes: a feature extraction module, a feature discretization module, an attention encoding module, an attention decoding module and a speech classification module; wherein the feature extraction module is used to extract speech feature information of any speech sample, the feature discretization module is used to discretize the speech feature information, the attention encoding module is used to encode the context information of the speech feature information to obtain an encoded feature, the attention decoding module is used to decode according to the encoded feature to obtain a decoded feature, and the speech classification module is used to decode according to the encoded feature to obtain a decoded feature. After decoding, the features are used to perform speech classification on any of the speech samples to obtain text information corresponding to any of the speech samples; according to the second speech sample, the feature extraction module, the feature discretization module and the attention encoding module in the initial speech recognition model, the feature extraction module and the attention encoding module are trained to obtain the feature extraction module trained for the first time and the attention encoding module trained for the first time; according to the first speech sample, the feature extraction module trained for the first time and the attention encoding module trained for the first time, the attention decoding module and the speech classification module, the speech recognition model is trained to obtain the speech recognition model, which includes: the feature extraction module trained for the second time, the attention encoding module trained for the second time, the attention decoding module after training and the speech classification module.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor and executing the speech recognition method according to any one of claims 1 to 7.

10. A terminal device, characterized in that: including a processor and a memory; The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the speech recognition method as described in any one of claims 1 to 7; the processor is used to implement each computer program in the multiple computer programs.

Images