Voice-to-text conversion method and device, equipment and storage medium

Abstract

The invention provides a voice-to-text method, device and equipment and a storage medium, and relates to the technical field of voice recognition, and the method comprises the steps: obtaining voice data and hot word list information; acquiring acoustic feature information according to the voice data; according to the voice data and the hot word list information, hot word index information and attention feature information are obtained based on attention mechanism processing; obtaining hot word feature information according to the hot word index information; and generating text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information. In combination with hot word list information and an attention mechanism, the hot word list information is utilized to make up the vocabulary basis in a low-resource language or terminology scene, and attention is utilized to adjust the priority of hot words, so that the transliteration key vocabularies are recognized more accurately in the speech recognition process, and the speech recognition efficiency is improved. The text conversion result information is more accurate in a low-resource language or terminology scene, and the accuracy and reliability of voice-to-text conversion are improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech-to-text method, device, equipment and storage medium. Background Art

[0002] Automatic speech recognition technology is widely used in various real-time interactive services and automated services. The implementation of automatic speech recognition requires a large amount of voice and text data for training to learn the language features and language patterns in specific scenarios.

[0003] However, in low-resource language scenarios, that is, scenarios without sufficient voice and text data support or dialect language scenarios with less data, the accuracy of automatic speech recognition will drop significantly. At the same time, in the existing automatic speech recognition technology, since the voice and text data used for training are usually frequently used voice and text data, they cannot cover rare words or professional terminology, and the recognition accuracy of rare words or professional terminology is low. Summary of the invention

[0004] The present invention provides a speech-to-text method, device, equipment and storage medium, which are used to solve the defect of low recognition accuracy of speech recognition in low-resource language or professional terminology scenarios in the prior art.

[0005] The present invention provides a speech-to-text method, comprising: Get voice data and hot word list information; Input the voice data and the hot word list information into a processing model to obtain text conversion result information; The processing performed by the processing model includes: Acquiring acoustic feature information according to the speech data; According to the voice data and the hot word list information, hot word index information and attention feature information are obtained based on the attention mechanism processing, wherein the hot word index information represents the correlation between the hot word and the voice, and the attention feature information represents the correlation between the voice in the context and the hot word; Acquire hot word feature information according to the hot word index information; The text conversion result information is generated according to the acoustic feature information, the attention feature information and the hot word feature information.

[0006] According to a speech-to-text method provided by the present invention, the processing model includes a preprocessing module, a speech encoding module and an acoustic decoding module; the acquiring of acoustic feature information according to the speech data includes: Inputting the voice data into the preprocessing module for preprocessing to obtain voice feature information; Inputting the speech feature information into the speech encoding module for encoding processing to obtain an acoustic feature sequence; The acoustic feature sequence is input into the acoustic decoding module for decoding processing to obtain the acoustic feature information.

[0007] According to a speech-to-text method provided by the present invention, the processing model includes a hot word encoding module and an attention decoding module; the hot word index information and attention feature information are obtained based on the speech data and the hot word list information and the attention mechanism processing, including: Get historical marker sequence information; Inputting the hot word list information into the hot word encoding module for encoding processing to obtain a hot word feature sequence; The acoustic feature sequence, the hot word feature sequence and the historical mark sequence information are input into the attention decoding module, and the hot word index information and the attention feature information are obtained based on cross-attention processing.

[0008] According to a speech-to-text method provided by the present invention, the acoustic feature sequence, the hot word feature sequence and the historical mark sequence information are input into the attention decoding module, and the hot word index information and the attention feature information are obtained based on cross attention processing, including: Converting the historical mark sequence information to obtain a historical mark feature sequence; Performing a first cross-attention processing according to the historical mark feature sequence and the acoustic feature sequence to obtain a first cross-attention feature sequence; Perform a second cross-attention processing according to the first cross-attention feature sequence and the hot word feature sequence to obtain a second cross-attention feature sequence; Generating the hot word index information according to the second cross-attention feature sequence; Perform linear transformation processing on the second cross-attention feature sequence to obtain the attention feature information.

[0009] According to a speech-to-text method provided by the present invention, the hot word list information is input into the hot word encoding module for encoding processing to obtain a hot word feature sequence, including: Encode the hot word list information to obtain the hot word embedding vector corresponding to each hot word; According to the hot word embedding vector, the dependency relationship of the hot word is captured based on the conversion process to obtain the hot word hidden feature information; According to the hot word hidden feature information, a hot word feature sequence is extracted.

[0010] According to a speech-to-text method provided by the present invention, the processing model includes a hot word decoding module; the hot word feature information is obtained according to the hot word index information, including: The hot word index information is input into the hot word decoding module to determine the probability distribution of the hot words, the probability score of each hot word is calculated according to the hot word probability distribution, and the hot word index is selected according to the probability score to form the hot word feature information.

[0011] According to a speech-to-text method provided by the present invention, the processing model includes a beam search decoding module; the text conversion result information is generated according to the acoustic feature information, the attention feature information and the hot word feature information, including: The acoustic feature information, the attention feature information and the hot word feature information are input into the beam search decoding module to determine the tag sequence with the highest probability and obtain the text conversion result information.

[0012] The present invention also provides a speech-to-text device, comprising: A preprocessing module is used to preprocess the speech data and obtain speech feature information; A speech encoding module, used for encoding the speech feature information to obtain an acoustic feature sequence; An acoustic decoding module, used to decode the acoustic feature sequence to obtain acoustic feature information; A hot word encoding module is used to encode the hot word list information and obtain the hot word feature sequence; An attention decoding module, used to obtain hot word index information and attention feature information according to the acoustic feature sequence and the hot word feature sequence; A hot word decoding module, used to obtain hot word feature information according to the hot word index information; A beam search decoding module, used to generate the text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information; The preprocessing module, the speech encoding module, the acoustic decoding module, the attention decoding module, the hot word encoding module, the hot word decoding module and the beam search decoding module cooperate with each other to implement the above-mentioned speech-to-text method.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a speech-to-text method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, a speech-to-text conversion method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech-to-text conversion method as described above is implemented.

[0016] The present invention provides a speech-to-text method, device, equipment and storage medium, which have at least the following beneficial effects: in speech recognition, in addition to processing based on speech data, processing is also performed in combination with hot word list information, and the hot word list information may include low-frequency words, dialect words, and professional words, etc. The processing model is based on the input speech data and hot word list information, and obtains acoustic feature information according to the speech data; based on the speech data and the hot word list information, the attention mechanism is used to process to obtain hot word index information and attention feature information, the hot word index information reflects the hot word information related to the speech, and the attention feature information reflects the information combined with the speech context and the correlation between the speech and the hot word; the hot word feature information is further obtained according to the hot word index information; finally, according to the acoustic feature information, the attention feature information and the hot word feature information, the speech-to-text tag sequence is obtained, and the text conversion result information is generated to realize the speech recognition function. In this way, the hot word list information and the attention mechanism are combined, and the hot word list information is used to make up for the vocabulary base in low-resource languages ​​or professional terminology scenarios. At the same time, attention is used to adjust the priority of hot words, so that key words can be more accurately identified and transcribed during the speech recognition process, and the text conversion result information can be more accurate in low-resource languages ​​or professional terminology scenarios, which is conducive to improving the accuracy and reliability of speech-to-text. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a speech-to-text method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the architecture of the processing model provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the architecture of the speech coding module provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the architecture of the hot word encoding module provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the architecture of the attention decoding module provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the architecture of the hot word decoding module provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the processing process of the beam search decoding module provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1 A speech-to-text method of the present invention is described, comprising: Get voice data and hot word list information; Input the voice data and the hot word list information into a processing model to obtain text conversion result information; The processing performed by the processing model includes: S100: Acquiring acoustic feature information according to the speech data; S200: According to the voice data and the hot word list information, hot word index information and attention feature information are obtained based on the attention mechanism processing, wherein the hot word index information represents the correlation between the hot word and the voice, and the attention feature information represents the correlation between the voice in the context and the hot word; S300: Acquire hot word feature information according to the hot word index information; S400: Generate the text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information.

[0028] In speech recognition, in addition to processing based on speech data, it is also processed in combination with hot word list information, which may include low-frequency words, dialect words, and professional words, etc. The processing model is based on the input speech data and hot word list information, and obtains acoustic feature information according to the speech data; based on the speech data and hot word list information, it processes based on the attention mechanism to obtain hot word index information and attention feature information, and the hot word index information reflects the hot word information related to speech, and the attention feature information reflects the information combined with the speech context and the correlation between speech and hot words; hot word feature information is further obtained based on the hot word index information; finally, based on the acoustic feature information, attention feature information, and hot word feature information, the speech-to-text tag sequence is obtained, and the text conversion result information is generated to realize the speech recognition function.

[0029] In this way, the hot word list information and the attention mechanism are combined, and the hot word list information is used to make up for the vocabulary base in low-resource languages ​​or professional terminology scenarios. At the same time, attention is used to adjust the priority of hot words, so that key words can be more accurately identified and transcribed during the speech recognition process, and the text conversion result information can be more accurate in low-resource languages ​​or professional terminology scenarios, which is conducive to improving the accuracy and reliability of speech-to-text.

[0030] refer to Figure 2 In some embodiments of the present invention, the architecture of the processing model mainly includes a preprocessing module, a speech encoding module, an acoustic decoding module, an attention decoding module, a hot word encoding module, a hot word decoding module and a beam search decoding module.

[0031] refer to Figure 2 In some embodiments of a speech-to-text method of the present invention, the processing model includes a preprocessing module, a speech encoding module, and an acoustic decoding module; S100 includes: Inputting the voice data into the preprocessing module for preprocessing to obtain voice feature information; Inputting the speech feature information into the speech encoding module for encoding processing to obtain an acoustic feature sequence; The acoustic feature sequence is input into the acoustic decoding module for decoding processing to obtain the acoustic feature information.

[0032] The speech data is input into the preprocessing module for preprocessing such as framing, window function processing, pre-emphasis, standardization, and feature extraction to form speech feature information. The original speech data is not convenient for direct processing. The speech feature information formed through preprocessing is convenient for subsequent analysis and processing. The speech feature information is input into the speech encoding module, which converts the speech feature information into an acoustic feature sequence in the hidden layer state for further processing and recognition. The acoustic decoding module decodes the input acoustic feature sequence to obtain acoustic feature information, which represents the tag sequence that may be obtained based on acoustic recognition of speech.

[0033] In some embodiments of the present invention, the preprocessing module performs feature extraction, which may be MFCCs (Mel Frequency Cepstral Coefficients) or Log-Mel (Logarithmic Mel Frequency Spectrum Features), etc. The speech feature information may be an audio feature sequence.

[0034] In some embodiments of the present invention, the output of the preprocessing module can be expressed by the following expression:

[0035] in, is the feature vector at time t; MelSpectrogram() is the Mel spectrum conversion function; is the speech frame of the speech data at time t; X is the speech feature information.

[0036] refer to Figure 3 In some embodiments of the present invention, the speech coding module includes a convolutional layer, a first linear layer, a Conformer layer, and a second linear layer, and there may be multiple Conformer layers. After the speech feature information is processed and transformed by the convolutional layer and the first linear layer, it is subjected to deep feature extraction by the Conformer layer, and then converted and processed by the second linear layer to form an acoustic feature sequence of the hidden layer state. The acoustic feature sequence may include vectors of multiple dimensions, represented in the form of a matrix.

[0037] In some embodiments of the present invention, the output of the speech encoding module can be expressed by the following expression:

[0038] Where X is the speech feature information; H is the acoustic feature sequence of the hidden layer state after encoding; AudioEncoder( ) is the encoding process. H can be a T×d matrix, where T represents the vector length and d represents the dimension.

[0039] In some embodiments of the present invention, the acoustic decoding module may be a CTC decoding module, which performs CTC decoding processing on the acoustic feature sequence, calculates all possible output paths according to the acoustic feature sequence, that is, calculates the tag sequence and corresponding probability corresponding to all possible character sequences, and can align the input and output through the "blank" tag. In other words, the CTC decoding module outputs all possible tag sequences and probability distributions to form acoustic feature information. The output of the CTC decoding module can be expressed by the following expression:

[0040] in, is the probability distribution corresponding to the speech feature information X, that is, the acoustic feature information; are all possible ways to align the sequence y to length T; Π is a path of length T, which contains the token sequence y and the blank token; Given the hidden layer state at time t The mark on the output path Π The probability of , characterizing the probability distribution of possible label sequences, that is, acoustic feature information.

[0041] refer to Figure 2 In some embodiments of a speech-to-text method of the present invention, the processing model includes a hot word encoding module and an attention decoding module; S200 includes: Get historical marker sequence information; Inputting the hot word list information into the hot word encoding module for encoding processing to obtain a hot word feature sequence; The acoustic feature sequence, the hot word feature sequence and the historical mark sequence information are input into the attention decoding module, and the hot word index information and the attention feature information are obtained based on cross-attention processing.

[0042] The processing model determines the token sequence with the highest probability based on the acoustic feature information, attention feature information, and hot word feature information, and converts the token sequence into a corresponding text sequence to form the final output text conversion result information. The historical token sequence information includes multiple token sequences obtained in the previous speech-to-text processing cycle.

[0043] The hot word list information is input into the hot word encoding module for encoding processing to obtain a hot word feature sequence that is easy to analyze and process. The hot word feature sequence may include vectors of multiple dimensions, which may be specifically represented in the form of a matrix.

[0044] Based on the cross-attention mechanism, the acoustic feature sequence, hot word feature sequence and historical mark sequence information are processed, and the cross-attention can extract the relationship between the two sequences. Therefore, based on the acoustic feature sequence and the hot word feature sequence, the hot words that may be associated with the speech can be extracted and the hot word index information can be generated; according to the historical mark sequence information, combined with the acoustic feature sequence and the hot word feature sequence, the degree of association between the speech and the hot words can be extracted based on the association of the context, and then the attention weight can be determined to generate the attention feature information.

[0045] In this way, the cross-attention mechanism is utilized to combine the acoustic feature sequence with the hot word feature sequence, that is, to combine the speech and the hot words, extract the relationship between the two, generate hot word index information and attention feature information, and realize the adjustment of the attention weight and priority of the hot words, so that the final generated tag sequence is the result of the comprehensive hot words and speech, and the text conversion result information obtained based on the tag sequence focuses on the low-frequency words, professional terminology words, etc. included in the hot word list information, so as to improve the accuracy and reliability of speech-to-text in low-resource language or professional terminology scenarios.

[0046] Understandably, reference Figure 2 , the acoustic feature sequence is output by the speech encoding module and input into the acoustic decoding module and the attention decoding module respectively.

[0047] refer to Figure 2 In some embodiments of the speech-to-text method of the present invention, the step of inputting the hot word list information into the hot word encoding module for encoding processing to obtain a hot word feature sequence includes: Encode the hot word list information to obtain the hot word embedding vector corresponding to each hot word; According to the hot word embedding vector, the dependency relationship of the hot word is captured based on the conversion process to obtain the hot word hidden feature information; According to the hot word hidden feature information, a hot word feature sequence is extracted.

[0048] By encoding the hot word list information to obtain the hot word embedding vector that is easy to process and analyze, the hot word embedding vector is analyzed to capture the dependencies within and between hot word phrases, generate hot word hidden feature information, and then convert the hot word hidden feature information to form a hot word feature sequence. In this way, since there is a correlation between the words used in speech, especially in the scene of professional terminology, by considering the correlation between hot words and reflecting the correlation between hot words, it is convenient to consider the impact of the correlation between hot words and adjust the attention in the subsequent processing based on the attention mechanism, which is conducive to making the final text conversion result more accurate.

[0049] In some embodiments of the present invention, reference Figure 4The hot word encoding module includes a word embedding layer, a Transformer layer, and a linear layer. The hot word list information is processed by the word embedding layer to form a sequence that is convenient for word segmentation processing, and then processed by the Transformer layer to capture the dependency relationship between hot words, and then converted by the linear processing layer into a hot word feature sequence in the hidden layer state. The hot word feature sequence can include multi-dimensional vectors, which are specifically represented in the form of a matrix.

[0050] In some embodiments of the present invention, the output of the hot word encoding module can be expressed by the following expression:

[0051] Among them, V is the hot word feature sequence; Liner() is the linear transformation processing of the linear layer; Transformer() is the processing function of the Transformer layer; Embedding() is the processing function of the word embedding layer; B is the hot word list information.

[0052] refer to Figure 2 and Figure 5 In some embodiments of a speech-to-text method of the present invention, the acoustic feature sequence, the hot word feature sequence, and the historical tag sequence information are input into the attention decoding module, and the hot word index information and the attention feature information are obtained based on cross-attention processing, including: Converting the historical mark sequence information to obtain a historical mark feature sequence; Performing a first cross-attention processing according to the historical mark feature sequence and the acoustic feature sequence to obtain a first cross-attention feature sequence; Perform a second cross-attention processing according to the first cross-attention feature sequence and the hot word feature sequence to obtain a second cross-attention feature sequence; Generating the hot word index information according to the second cross-attention feature sequence; Perform linear transformation processing on the second cross-attention feature sequence to obtain the attention feature information.

[0053] The historical mark sequence information is converted to form a historical mark feature sequence that is easy to process and analyze. The historical mark feature sequence and the acoustic feature sequence are subjected to a first cross-attention process, and the first cross-attention feature sequence obtained is then subjected to a second cross-attention process with the hot word feature sequence to obtain a second cross-attention feature sequence, which can reflect the correlation between the speech and the hot words based on the context association, and the hot words associated with the speech are determined to generate hot word index information, and the second cross-attention feature sequence is further subjected to a linear conversion process to obtain attention feature information.

[0054] In some embodiments of the present invention, reference Figure 5 The attention decoding module may include a word embedding layer, a relative position encoding layer, a self-attention layer, an acoustic cross-attention layer, a hot word cross-attention layer, and a linear layer. The historical tag sequence information, such as the tag sequence (token) finally obtained in the previous processing cycle, is input into the word embedding layer to be converted into a historical tag feature sequence that is easy to analyze and process. The historical tag feature sequence is processed by the relative position encoding layer to identify the relative position between elements in the sequence, considering the order influence between elements, and then processed by the self-attention layer to assign different attention weights to each element, and then input into the acoustic cross-attention layer and the acoustic feature sequence for the first cross-attention processing to adjust the attention weight according to the current speech combined with the context to form a first cross-attention feature sequence, and then input into the hot word cross-attention layer and the hot word feature sequence for the second cross-attention processing to adjust the attention weight according to the relevance of the hot words to form a second cross-attention feature sequence. The second cross-attention feature sequence is formed after linear transformation through the linear formation to form attention feature information.

[0055] It should be emphasized that the present invention can effectively combine the acoustic feature sequence output by the speech encoder and the hot word feature sequence output by the hot word encoder by adding additional cross-attention layers, namely, the acoustic cross-attention layer and the hot word cross-attention layer, so as to optimize the speech recognition process, enhance the ability to make adjustments using low-frequency words, professional words, etc. contained in the hot word list information, and improve the adaptability of the processing model to new words and professional field terms outside the training data after completing the model training process.

[0056] In some embodiments of the present invention, the processing of data by the attention decoding module may include: According to the acoustic feature sequence and the hot word feature sequence, based on cross attention processing, the hot word index information is obtained; Acquire attention weight information according to the acoustic feature sequence and the hot word feature sequence in combination with historical attention states; Acquire context feature information according to the attention weight information, the acoustic feature sequence, and the hot word feature sequence; The attention feature information is obtained based on the context feature information and the historical mark sequence information in combination with the historical attention state.

[0057] In some embodiments of the present invention, the output of the attention decoding module can be expressed by the following expression:

[0058] in, is the set of attention weights for t processing cycles; is the historical attention state of the previous cycle of the t-1 processing cycle, which can be understood as the state parameter of the attention decoding module in the previous cycle; H is the acoustic feature sequence; V is the hot word feature sequence; Attention () is the function for calculating the attention weight; is the context feature vector, i.e., the context feature information; is the tag sequence output by t processing cycle; is the tag sequence output before t processing cycles, that is, the historical tag sequence information; Decoder() is the decoding function; softmax() is the softmax function; It is the probability distribution output by the attention decoding module, that is, the attention feature information.

[0059] refer to Figure 2 In some embodiments of a speech-to-text method of the present invention, the processing model includes a hot word decoding module; S300 includes: The hot word index information is input into the hot word decoding module to determine the probability distribution of the hot words, the probability score of each hot word is calculated according to the hot word probability distribution, and the hot word index is selected according to the probability score to form the hot word feature information.

[0060] The hot word index information is obtained by the attention decoding module based on the attention processing by combining the acoustic feature sequence and the hot word feature sequence, which can reflect the probability distribution of hot words associated with the speech. The hot word index information is input into the hot word decoding module to determine the probability distribution of hot words, that is, the degree of association between different hot words and speech, and then combine the context association, the association between hot words and other factors to calculate the probability score of each hot word, and then select the hot word index corresponding to the hot word according to the probability score to form the hot word feature information. In this way, the hot word set with the highest degree of relevance to the currently processed speech is determined by combining the influence of multi-dimensional factors such as the acoustic characteristics of the speech itself, the context association and the association between hot words, and the hot word feature information is formed, which is conducive to making the final text conversion result more accurate.

[0061] In some embodiments of the present invention, reference Figure 6 The hot word decoding module can include a linear layer, a Tranformer layer and an Argmax layer. After the hot word index information is linearly transformed by the linear layer, it is input into the Transformer layer to capture the correlation between the hot word and the current decoding state, that is, the correlation between the hot word and the context and the correlation between the hot words, etc., and then the hot word set with the highest probability is determined by the argmax function, and the hot word feature information is output.

[0062] In some embodiments of the present invention, the output of the hot word decoding module can be expressed by the following expression:

[0063] Among them, U is the output of the cross attention calculation in the attention decoding module, that is, the hot word index information; Represents the probability distribution of hot words; Liner() is the linear transformation function of the linear layer; Transformer() is the processing function of the Transformer layer; softmax() is the softmax function; is the hot word set with the highest probability score, i.e., the hot word feature information; is the probability score of the i-th hot word; argmax() is the argmax function.

[0064] It can be understood that the hot word set with the highest probability score refers to the hot word set formed by hot words selected in descending order of probability scores. The number of hot words included in the hot word set can be pre-set or determined through the model training process.

[0065] refer to Figure 2 In some embodiments of a speech-to-text method of the present invention, the processing model includes a beam search decoding module; S400 includes: The acoustic feature information, the attention feature information and the hot word feature information are input into the beam search decoding module to determine the tag sequence with the highest probability and obtain the text conversion result information.

[0066] Through the beam search decoding module, the strategy of beam search decoding is adopted, combining acoustic feature information, attention feature information and hot word feature information, outputting the most likely tag sequence, and then generating the corresponding text sequence according to the tag sequence to form text conversion result information. In this way, the beam search decoding method is adopted to retain multiple candidate paths when generating the sequence, and then the optimal path is selected to output the most likely tag sequence. At the same time, by controlling the beam width, the performance and computational efficiency can be balanced to flexibly meet the needs of different scenarios.

[0067] In some embodiments of the present invention, the acoustic feature information includes the probability distribution of possible tag sequences obtained by CTC decoding, the attention feature information includes the probability distribution output by the attention decoding module, and the hot word feature information includes the probability distribution of hot words. The three probability distributions are input into the beam search decoding module, and a tag sequence is output through the beam search decoding process. The tag sequence is determined based on the condition with the maximum probability.

[0068] It should be noted that the processing model is obtained through a pre-model training process. During the model training process, an adaptive loss function combination may be set to perform iterative optimization of the model to improve the accuracy of the model's performance. In some embodiments of the present invention, the loss function combination used by the processing model during the model training process may be:

[0069] in, , , To adjust the weight parameters of the loss function; is the hot word loss function; N is the total number of hot words; X is the speech feature information; is the i-th hot word; Under the condition that the speech feature information X is probability; is the attention loss function; is the tag sequence output by t processing cycle; is the token sequence output before t processing cycles; is the speech feature information X and the output tag sequence before t cycles is Under the condition of probability; Acoustic loss function for CTC decoding.

[0070] A speech-to-text device provided by the present invention is described below. The speech-to-text device described below and the speech-to-text method described above can be referenced to each other.

[0071] refer to Figure 2 The present invention also provides a speech-to-text device, comprising: A preprocessing module is used to preprocess the speech data and obtain speech feature information; A speech encoding module, used for encoding the speech feature information to obtain an acoustic feature sequence; An acoustic decoding module, used to decode the acoustic feature sequence to obtain acoustic feature information; A hot word encoding module is used to encode the hot word list information and obtain the hot word feature sequence; An attention decoding module, used to obtain hot word index information and attention feature information according to the acoustic feature sequence and the hot word feature sequence; A hot word decoding module, used to obtain hot word feature information according to the hot word index information; A beam search decoding module, used to generate the text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information; The preprocessing module, the speech encoding module, the acoustic decoding module, the attention decoding module, the hot word encoding module, the hot word decoding module and the beam search decoding module cooperate with each other to implement the above-mentioned speech-to-text method.

[0072] During speech recognition, in addition to processing based on speech data, it is also processed in combination with hot word list information, which may include low-frequency words, dialect words, and professional words.

[0073] The preprocessing module, speech encoding module and acoustic decoding module process the speech data to obtain the acoustic feature sequence and acoustic feature information. The hot word encoding module processes the hot word list information to obtain the hot word feature sequence. The attention decoding module processes the acoustic feature sequence and the hot word feature sequence based on the attention mechanism to obtain the hot word index information and attention feature information. The hot word index information reflects the hot word information related to the speech, and the attention feature information reflects the information combined with the speech context and the correlation between the speech and the hot words. The hot word decoding module further processes the hot word index information to obtain the hot word feature information. The beam search decoding module obtains the speech-to-text tag sequence based on the acoustic feature information, attention feature information and hot word feature information, generates text conversion result information, and realizes the speech recognition function.

[0074] In this way, the hot word list information and the attention mechanism are combined, and the hot word list information is used to make up for the vocabulary base in low-resource languages ​​or professional terminology scenarios. At the same time, attention is used to adjust the priority of hot words, so that key words can be more accurately identified and transcribed during the speech recognition process, and the text conversion result information can be more accurate in low-resource languages ​​or professional terminology scenarios, which is conducive to improving the accuracy and reliability of speech-to-text.

[0075] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the above-mentioned speech-to-text method.

[0076] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a speech-to-text method provided by the above methods.

[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it is implemented to perform the above-mentioned methods to provide a speech-to-text method.

[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0081] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0082] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0083] All actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech-to-text method, characterized in that: include: Get voice data and hot word list information; Input the voice data and the hot word list information into a processing model to obtain text conversion result information; The processing performed by the processing model includes: Acquiring acoustic feature information according to the speech data; According to the voice data and the hot word list information, hot word index information and attention feature information are obtained based on the attention mechanism processing, wherein the hot word index information represents the correlation between the hot word and the voice, and the attention feature information represents the correlation between the voice in the context and the hot word; Acquire hot word feature information according to the hot word index information; The text conversion result information is generated according to the acoustic feature information, the attention feature information and the hot word feature information.

2. A speech-to-text method according to claim 1, characterized in that: The processing model includes a preprocessing module, a speech encoding module and an acoustic decoding module; The acquiring of acoustic feature information according to the speech data includes: Inputting the voice data into the preprocessing module for preprocessing to obtain voice feature information; Inputting the speech feature information into the speech encoding module for encoding processing to obtain an acoustic feature sequence; The acoustic feature sequence is input into the acoustic decoding module for decoding processing to obtain the acoustic feature information.

3. A speech-to-text method according to claim 2, characterized in that: The processing model includes a hot word encoding module and an attention decoding module; The method of obtaining hot word index information and attention feature information based on the speech data and the hot word list information and the attention mechanism includes: Get historical marker sequence information; Inputting the hot word list information into the hot word encoding module for encoding processing to obtain a hot word feature sequence; The acoustic feature sequence, the hot word feature sequence and the historical mark sequence information are input into the attention decoding module, and the hot word index information and the attention feature information are obtained based on cross-attention processing.

4. A speech-to-text method according to claim 3, characterized in that: The step of inputting the acoustic feature sequence, the hot word feature sequence, and the historical tag sequence information into the attention decoding module, and obtaining the hot word index information and the attention feature information based on cross attention processing includes: Converting the historical mark sequence information to obtain a historical mark feature sequence; Performing a first cross-attention processing according to the historical mark feature sequence and the acoustic feature sequence to obtain a first cross-attention feature sequence; Perform a second cross-attention processing according to the first cross-attention feature sequence and the hot word feature sequence to obtain a second cross-attention feature sequence; Generating the hot word index information according to the second cross-attention feature sequence; Perform linear transformation processing on the second cross-attention feature sequence to obtain the attention feature information.

5. A speech-to-text method according to claim 3, characterized in that: The step of inputting the hot word list information into the hot word encoding module for encoding processing to obtain a hot word feature sequence includes: Encode the hot word list information to obtain the hot word embedding vector corresponding to each hot word; According to the hot word embedding vector, the dependency relationship of the hot word is captured based on the conversion process to obtain the hot word hidden feature information; According to the hot word hidden feature information, a hot word feature sequence is extracted.

6. A speech-to-text method according to claim 1, characterized in that: The processing model includes a hot word decoding module; the hot word feature information is obtained according to the hot word index information, including: The hot word index information is input into the hot word decoding module to determine the probability distribution of the hot words, the probability score of each hot word is calculated according to the hot word probability distribution, and the hot word index is selected according to the probability score to form the hot word feature information.

7. A speech-to-text method according to claim 1, characterized in that: The processing model includes a beam search decoding module; the generating the text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information includes: The acoustic feature information, the attention feature information and the hot word feature information are input into the beam search decoding module to determine the tag sequence with the highest probability and obtain the text conversion result information.

8. A speech-to-text device, characterized in that: include: A preprocessing module is used to preprocess the speech data and obtain speech feature information; A speech encoding module, used for encoding the speech feature information to obtain an acoustic feature sequence; An acoustic decoding module, used to decode the acoustic feature sequence to obtain acoustic feature information; A hot word encoding module is used to encode the hot word list information and obtain the hot word feature sequence; An attention decoding module, used to obtain hot word index information and attention feature information according to the acoustic feature sequence and the hot word feature sequence; A hot word decoding module, used to obtain hot word feature information according to the hot word index information; A beam search decoding module, used to generate the text conversion result information according to the acoustic feature information, the attention feature information and the hot word feature information; The preprocessing module, the speech encoding module, the acoustic decoding module, the attention decoding module, the hot word encoding module, the hot word decoding module and the beam search decoding module cooperate with each other to implement a speech-to-text method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the speech-to-text method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a speech-to-text method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a speech-to-text method as described in any one of claims 1 to 7 is implemented.

Images