Speech recognition method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of information processing, and relates to a voice recognition method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a voice signal; performing feature extraction on the voice signal according to the voice signal; establishing a mapping relation between voice features and pronunciation units according to the voice signal features obtained after feature extraction of the voice signals; according to the mapping relation between the voice features and the pronunciation units, language modeling is carried out, statistical laws in languages are captured through language modeling, and the probability of occurrence of a next word or phrase in a given context is predicted; and obtaining a text sequence through a search algorithm according to the output of the acoustic model and the language model. And the accuracy, robustness and real-time performance of speech recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a speech recognition method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Speech recognition technology, as an important branch of artificial intelligence, aims to convert human speech into text or instructions, enabling natural interaction between humans and machines.

[0003] Existing speech recognition methods typically suffer from low accuracy due to issues with speech signal processing and recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a speech recognition method, apparatus, computer device, and storage medium to solve the problem of low accuracy in existing speech recognition methods, which are often due to the processing and recognition of speech signals.

[0005] To address the aforementioned technical problems, this invention provides a speech recognition method, employing the technical solution described below, including:

[0006] Acquire audio signals;

[0007] Based on the speech signal, feature extraction is performed on the speech signal;

[0008] Based on the speech signal features after feature extraction, a mapping relationship between speech features and articulatory units is established;

[0009] Based on the mapping relationship between speech features and articulatory units, language modeling is performed. Through language modeling, statistical patterns in language are captured, and the probability of the next word or phrase appearing in a given context is predicted.

[0010] Based on the outputs of the acoustic model and the language model, a search algorithm is used to obtain the text sequence.

[0011] Preferably, the step of acquiring the voice signal specifically includes:

[0012] Choose a microphone that meets the requirements for sensitivity, noise, and bandwidth, depending on the specific application scenario.

[0013] Set up a microphone array to collect voice signals;

[0014] Store the voice signal.

[0015] Preferably, the step of extracting features from the speech signal specifically includes:

[0016] The speech signal is preprocessed;

[0017] Feature parameters useful for speech recognition are extracted from the preprocessed speech signal.

[0018] Preferably, the step of establishing the mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction from the speech signal specifically includes:

[0019] Establish a database of labeled speech data, with the labeling content including the start and end times of each pronunciation unit and the corresponding text or phoneme;

[0020] Based on the extracted feature parameters and labeled data, one or more classifiers or regressors are trained to learn the mapping relationship between speech features and articulatory units.

[0021] For the speech signal to be recognized, repeat the feature extraction steps to obtain the corresponding feature sequence;

[0022] The extracted feature sequences are input into the trained model, which then predicts the corresponding vocal units for each time frame based on the learned mapping relationships.

[0023] Preferably, the step of performing language modeling based on the mapping relationship between speech features and articulatory units, capturing statistical patterns in language through language modeling, and predicting the probability of the next word or phrase appearing in a given context specifically includes:

[0024] A statistical language model is used, based on the statistical frequency method, to predict the next word by calculating the probability of N consecutive words or morphemes co-occurring, where N is a natural number greater than 1;

[0025] By using neural networks to learn long-range dependencies and contextual information in language, probability distributions can be generated.

[0026] A language model is constructed by combining historical dialogues, geographical locations, and external information about time.

[0027] Preferably, the step of obtaining the text sequence through a search algorithm based on the outputs of the acoustic model and the language model specifically includes:

[0028] The Viterbi algorithm is used to find the optimal state sequence given the state and transition probabilities through dynamic programming.

[0029] The beam search approach is adopted in neural network decoding by maintaining a fixed-size candidate list or candidate beam, gradually expanding and retaining the path with the highest score;

[0030] The connectionist temporal classification method is adopted to directly model the sequence, without the need to align the input and output sequences in the training data, thus simplifying the training and decoding process.

[0031] Preferably, after the step of obtaining the text sequence through a search algorithm based on the outputs of the acoustic model and the language model, the method further includes:

[0032] By analyzing the emotional information in the speech signal, the speaker's emotional state can be determined.

[0033] To address the aforementioned technical problems, the present invention also provides a voice recognition device, which employs the following technical solution, including:

[0034] The acquisition module is used to acquire voice signals;

[0035] The feature extraction module is used to extract features from the speech signal based on the speech signal.

[0036] The mapping module is used to establish a mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction of the speech signal;

[0037] The modeling module is used to perform language modeling based on the mapping relationship between speech features and articulatory units. Through language modeling, it captures statistical patterns in language and predicts the probability of the next word or phrase appearing in a given context.

[0038] The text module is used to obtain text sequences based on the outputs of the acoustic model and the language model using a search algorithm.

[0039] To address the aforementioned technical problems, the present invention also provides a computer device that employs the technical solution described below, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the aforementioned speech recognition method.

[0040] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium, which employs the technical solution described below. The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the aforementioned speech recognition method.

[0041] Compared with existing technologies, the present invention has the following advantages: First, it acquires a speech signal; then, it extracts features from the speech signal; next, it establishes a mapping relationship between speech features and articulatory units based on the extracted speech signal features; then, it performs language modeling based on the mapping relationship between speech features and articulatory units, and captures statistical patterns in language through language modeling to predict the probability of the next word or phrase appearing in a given context; finally, it obtains a text sequence through a search algorithm based on the output of the acoustic model and the language model; thus improving the accuracy, robustness, and real-time performance of speech recognition. Attached Figure Description

[0042] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is an exemplary system architecture diagram in which the present invention can be applied;

[0044] Figure 2 This is a flowchart of an embodiment of the speech recognition method of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of one embodiment of the speech recognition device of the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device of the present invention. Detailed Implementation

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects and not to describe a particular order.

[0048] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0050] like Figure 1As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0051] Users can use the first terminal device 101, the second terminal device 102, or the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0052] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0053] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0054] It should be noted that the speech recognition method provided in the embodiments of the present invention is generally executed by a server / terminal device, and correspondingly, the speech recognition device is generally set in the server / terminal device.

[0055] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0056] Example 1

[0057] Continue to refer to Figure 2 A flowchart of an embodiment of the speech recognition method of the present invention is shown. The speech recognition method includes the following steps:

[0058] Step S1: Acquire the voice signal.

[0059] In this embodiment, the speech recognition method operates on an electronic device (e.g., Figure 1 The server / terminal device shown can receive voice recognition requests via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, Wi-Fi connections, Bluetooth connections, Wi-Fi Max connections, Zigbee connections, UWB (ultra-wireless) connections, and other currently known or future wireless connection methods.

[0060] The voice signal acquisition step is the starting point of speech recognition. It involves capturing the user's voice signal through hardware devices such as microphones and converting it into a digital signal for computer processing.

[0061] In some optional implementations of this embodiment, step S1, acquiring the voice signal, may further include the following steps:

[0062] S11 allows you to select a microphone that meets the requirements for sensitivity, noise, and bandwidth, depending on the application scenario.

[0063] For example, choose a high-sensitivity, low-noise microphone to ensure clear voice signal capture. Depending on the application scenario, select a microphone with high sensitivity, low noise, wide bandwidth, or specific directionality. For instance, in noisy environments, a microphone with noise suppression capabilities can be used.

[0064] S12, Set up a microphone array to collect voice signals.

[0065] For situations requiring the capture of sound from multiple directions or the enhancement of sound from a specific direction, a microphone array can be used. The array's arrangement (e.g., linear, circular, spherical) and algorithms (e.g., beamforming, sound source localization) directly affect the sound acquisition results.

[0066] Sound acquisition is the starting point for obtaining speech signals, and its quality directly affects the effectiveness of subsequent processing. Choosing appropriate microphones and their placement can minimize environmental noise interference and improve the signal-to-noise ratio of the speech signal.

[0067] S13, stores the voice signal.

[0068] After the voice signal is acquired, it is stored. During storage, keywords can be set according to actual needs, such as acquisition time, acquisition location, and acquisition requirements, to create a voice signal database.

[0069] Step S2: Extract features from the speech signal based on the speech signal.

[0070] In this embodiment, step S2, extracting features from the speech signal based on the speech signal, may further include the following steps:

[0071] S21, preprocess the speech signal.

[0072] The acquired digital speech signals undergo preliminary processing, including endpoint detection, pre-emphasis, and frame windowing, in preparation for subsequent feature extraction.

[0073] Endpoint detection refers to identifying the start and end points of a speech signal, removing silence, and reducing computational load. Endpoint detection can be performed using methods such as energy thresholding and zero-crossing rate. Short-time energy reflects the energy level of the signal within a frame, while the short-time average zero-crossing rate reflects the frequency at which the signal crosses zero points; both are commonly used for endpoint detection in speech signals.

[0074] Since the high-frequency part of the voice signal has low energy and is easily affected by noise and transmission loss, pre-emphasis technology can be used to enhance the energy of the high-frequency part, making the signal spectrum flatter and facilitating subsequent processing.

[0075] Since speech signals are time-varying, to perform stable analysis, continuous speech signals can be divided into multiple shorter frames (typically 10-30ms), and each frame can be processed independently. Frames usually overlap to ensure signal continuity.

[0076] To reduce discontinuities between frames, windowing is typically applied to each frame after framing. For example, rectangular window functions, Hamming window functions, and Hanning window functions can be used, which can smooth frame edges and reduce spectral leakage to some extent.

[0077] Digital signal processing techniques (such as spectral subtraction and Wiener filtering) can also be used to reduce the impact of background noise on speech signals.

[0078] Signal preprocessing, through sub-steps such as pre-emphasis, framing, and windowing, performs preliminary processing on the raw speech signal, making it more suitable for subsequent feature extraction and pattern recognition. These processes enhance useful information in the signal, suppress useless information, and improve the robustness and accuracy of the system.

[0079] S22, extract feature parameters useful for speech recognition from the preprocessed speech signal.

[0080] Useful feature parameters for speech recognition, such as Mel-frequency cepstral coefficients (MFCC) and linear predictive cepstral coefficients (LPCC), are extracted from the preprocessed speech frames.

[0081] The most commonly used feature is the Mel frequency cepstral coefficient (MFCC), which simulates the nonlinear perception of frequency by the human ear and extracts feature vectors from each frame of speech signal through a series of mathematical transformations.

[0082] MFCC extraction: First, the speech signal is passed through a Mel filter bank. Then, logarithmic transform and discrete cosine transform (DCT) are performed on the filtered signal to obtain MFCC features. MFCC can effectively simulate the human ear's perception of sound.

[0083] Other features: In addition to MFCC, features such as PLP (Perceptual Linear Prediction) and GFCC (Gamma Frequency Cepstral Coefficients) can also be used to improve the performance of speech recognition systems.

[0084] Step S3: Based on the speech signal features after feature extraction, establish the mapping relationship between speech features and articulatory units.

[0085] In this embodiment, step S3, establishing the mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction, may further include the following steps:

[0086] S31. Establish a database of labeled speech data, with the labeling content including the start and end times of each pronunciation unit and the corresponding text or phoneme.

[0087] Collecting high-quality speech data and its corresponding accurate annotations is crucial. Annotated data should record the start and end times of each speech segment, as well as the corresponding articulatory unit. First, a database containing a large amount of annotated speech data is needed. This data is typically labeled by experts or automatic annotation tools, and the annotations include the start and end times of each articulatory unit, the corresponding text, or phoneme.

[0088] The purpose of data annotation is to provide the model with the supervision information needed for learning, namely the correspondence between speech signals and articulatory units.

[0089] S32, based on the extracted feature parameters and labeled data, train one or more classifiers or regressors to learn the mapping relationship between speech features and articulatory units.

[0090] Based on the extracted feature parameters and labeled data, various classifiers or regressors can be selected for training. For phoneme recognition (such as phoneme recognition), classifiers used include Support Vector Machines (SVM), Hidden Markov Models (HMM), and deep learning models (such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) and their variants LSTM and GRU). These models each have their advantages; for example, SVM performs well on small datasets, while deep learning models demonstrate powerful learning capabilities in large-scale complex tasks.

[0091] During training, it is necessary to appropriately divide the training, validation, and test sets, and employ strategies such as cross-validation to prevent overfitting. Simultaneously, model performance can be optimized by adjusting hyperparameters such as the learning rate, batch size, and regularization term. For deep learning models, the design of the network structure, such as the number of layers, neurons, and activation functions, must also be considered.

[0092] The purpose of step S32 is to optimize the model parameters so that the model can accurately map the input speech features to the corresponding articulatory units.

[0093] S33, For the speech signal to be recognized, repeat the feature extraction step to obtain the corresponding feature sequence.

[0094] Repeating the feature extraction step to obtain the feature sequence corresponding to the speech signal to be recognized is a core and meticulous process that directly affects the efficiency of subsequent processing steps and the recognition accuracy. This process typically involves multiple stages such as preprocessing, feature extraction, and postprocessing, aiming to extract the most critical information representation for the recognition task from the original speech signal.

[0095] Preprocessing the input speech signal to be recognized is an indispensable step. This includes sampling rate adjustment to ensure signal consistency and compatibility, noise suppression to remove background noise interference with recognition performance, and pre-emphasis processing to enhance the energy of high-frequency components, as the high-frequency components of speech signals often carry important consonant information but are easily attenuated during propagation. Additionally, endpoint detection may be included, i.e., determining the start and end points of the speech signal to remove silence segments and reduce the computational load of subsequent processing.

[0096] Feature extraction is the most crucial step in speech recognition, aiming to convert the preprocessed speech signal into a sequence of parameters that reflects the essential characteristics of speech. Commonly used feature extraction methods include Short Time Fourier Transform (STFT) and its derivative, Mel Frequency Cepstral Coefficients (MFCC). MFCC is a feature parameter based on the characteristics of human hearing. By simulating the differences in human ear sensitivity to different frequencies of sound, it converts the speech signal into a set of coefficients from low to high frequencies, which can effectively characterize the spectral envelope of speech.

[0097] In the MFCC feature extraction process, the speech signal is first segmented into multiple short-time frames using a window function (such as a Hamming window), and a Fast Fourier Transform (FFT) is performed on each frame to obtain spectral information. Subsequently, a Mel filter bank is used to smooth the spectrum, simulating the auditory filtering characteristics of the human ear. The logarithm of the filtered energy is then taken, and finally, the data is further compressed using a Discrete Cosine Transform (DCT) to obtain the MFCC coefficients. This process is typically repeated across multiple frames to generate the feature sequence of the entire speech signal.

[0098] While features such as MFCC can already represent speech signals well, in practice, it is still necessary to combine them with some post-processing techniques to optimize the feature sequence. These include feature normalization to eliminate differences in feature distribution among different speakers or in different environments, and feature dimensionality reduction to reduce computational complexity while maintaining recognition performance. Furthermore, with the rise of deep learning technology, deep learning models such as autoencoders and convolutional neural networks (CNNs) are widely used in feature learning. They can automatically learn more complex and effective feature representations from the original speech signal, further improving the performance of the speech recognition system.

[0099] The purpose of step S33 is to convert the input speech signal into a feature representation that the model can understand.

[0100] S34. The extracted feature sequence is input into the trained model, and the model predicts the corresponding pronunciation unit for each time frame based on the learned mapping relationship.

[0101] Training the model involves using a large amount of labeled data, i.e., audio sequences of known speech units for each time frame. Through backpropagation, the model gradually adjusts its internal parameters to minimize the difference between predicted and true values ​​(e.g., cross-entropy loss). Regularization, dropout, and other techniques may also be used during training to prevent overfitting and ensure the model has good generalization ability.

[0102] After model training is complete, new, unlabeled audio signals can be processed using the same feature extraction process, and the extracted feature sequences are then input into the model for prediction. The model outputs the probability distribution of the corresponding articulators for each time frame. To transform these probability distributions into the final articulator sequence, decoding is required. The decoding process typically involves path search algorithms, such as the Viterbi algorithm or beam search, which can find the globally optimal or near-optimal articulator sequence while ensuring computational efficiency.

[0103] Since there may be multiple candidate paths (i.e. multiple possible sequences of articulatory units), decoding is usually required to find the optimal sequence path.

[0104] The purpose of step S34 is to use the model to map the speech signal to the vocal unit and to determine the final recognition result through decoding.

[0105] Step S4: Based on the mapping relationship between speech features and articulatory units, language modeling is performed. Through language modeling, statistical patterns in language are captured, and the probability of the next word or phrase appearing in a given context is predicted.

[0106] In this embodiment, step S4, which involves performing language modeling based on the mapping relationship between speech features and articulatory units, and capturing statistical patterns in language through language modeling to predict the probability of the next word or phrase appearing in a given context, may further include the following steps:

[0107] S41 uses a statistical language model based on statistical frequency to predict the next word by calculating the probability of N consecutive words or morphemes appearing together, where N is a natural number greater than 1.

[0108] The basic assumption of statistical language models is that the probability of a word sequence can be approximated by the product of the probabilities of its constituent words, that is:

[0109] $$P(w_1,w_2,...,w_n)approxP(w_cdotP(w_2|w_cdotP(w_3|w_1,w_cdot...cdotP(w_n|w_1,...,w_{n-1})$$.

[0110] However, directly calculating the conditional probability of long sequences is very difficult and computationally intensive in practice. Therefore, a simplified n-gram model is often used, where n represents the number of context words considered by the model.

[0111] In an n-gram model, the probability of a word appearing depends only on the n-1 words preceding it. For example, in a binary model (bigram, n=2), the probability of the next word is determined solely by the current word.

[0112] $$P(w_i|w_1,...,w_{i-1})approxP(w_i|w_{i-1})$$

[0113] Similarly, the ternary model (trigram, n=3) considers the first two terms:

[0114] $$P(w_i|w_1,...,w_{i-1})approxP(w_i|w_{i-2},w_{i-1})$$.

[0115] A large amount of text data was collected as training corpus. Subsequently, the text was preprocessed, including word segmentation, stop word removal, and word form restoration, in order to construct a unified vocabulary.

[0116] Construct a word frequency table, traverse the training corpus, count the frequency of each n-gram, and construct an n-gram word frequency table.

[0117] Probability calculations are performed using a term frequency table to calculate the probability of each n-gram. Common smoothing techniques (such as Laplace smoothing, Good-Turing smoothing, etc.) are used to handle zero-probability problems, i.e., cases where some n-grams do not appear in the training data.

[0118] Model evaluation and application: The model's performance is evaluated using methods such as cross-validation, after which it can be used in prediction tasks. Given the first N-1 words, the model outputs the next word with the highest probability as the prediction result.

[0119] S42 utilizes neural networks to learn long-distance dependencies and contextual information in language to generate probability distributions.

[0120] The first step for a neural network model is to transform text data into a low-dimensional vector representation that the model can process, known as word embedding. This process maps words into a continuous vector space, enabling the model to capture semantic relationships between words. Subsequently, the model processes these vectors through a multi-layered network structure (including input, hidden, and output layers) to learn dependencies and contextual information between words.

[0121] During training, the neural network model continuously adjusts its parameters by optimizing the objective function (such as the cross-entropy loss function) to make the probability distribution predicted by the model as close as possible to the true distribution. Specifically, the model considers the constraints between the current word and several preceding words, and predicts the probability of the next word by calculating conditional probabilities. This process involves complex nonlinear transformations and parameter optimization, ultimately yielding an optimal set of parameters used to generate the probability distribution.

[0122] S43, combining historical dialogues, geographical location, and external information about time to construct a language model.

[0123] Building a deep learning model capable of fusing multiple input types is crucial. Multimodal learning frameworks, such as variants of the Transformer architecture, can be employed. These models can process sequential data (like text) while also integrating information from other modalities through attention mechanisms. Dedicated subnetworks or modules can be set up within the model to handle different types of input and perform information fusion at a higher level.

[0124] During the training phase, by designing a reasonable loss function and optimization algorithm, the model is guided to learn how to effectively integrate external information such as historical dialogues, geographical location, and time to generate accurate and coherent responses or predictions. Hyperparameter tuning, regularization, and early stopping are also necessary during training to prevent overfitting and improve the model's generalization ability.

[0125] The model can then be evaluated using a test set containing multi-dimensional information, focusing not only on the fluency and accuracy of text generation but also on its performance in incorporating external information such as geographic location and time. Through user feedback and iterative optimization, the model's practicality and intelligence can be continuously improved.

[0126] Language models learn word collocations, grammatical rules, and semantic information in a language by analyzing large amounts of text data, providing context-sensitive word selection suggestions for the decoder.

[0127] Step S43 involves context fusion, combining external information such as user history, geolocation, and time to construct a more personalized and context-aware language model.

[0128] Step S5: Based on the outputs of the acoustic model and the language model, obtain the text sequence through a search algorithm.

[0129] In this embodiment, step S5, obtaining the text sequence through a search algorithm based on the outputs of the acoustic model and the language model, may further include the following steps:

[0130] S51 uses the Viterbi algorithm to find the optimal state sequence through dynamic programming, given the state and transition probabilities.

[0131] The Viterbi algorithm is an efficient dynamic programming algorithm used to find the sequence of hidden states most likely to produce the observation sequence in a given Hidden Markov Model (HMM). The core idea of ​​the Viterbi algorithm is to recursively calculate the optimal path probability for all possible states at each time step and then backtrack to obtain the globally optimal sequence of hidden states. The specific implementation process is as follows:

[0132] For initialization, for the first observation, calculate and record the probability of reaching that observation from each initial state. This typically involves the initial state probability and the probability of generating the first observation from that state.

[0133] Recursively, for each subsequent observation, consider all previous states and calculate the cumulative probability of transitioning from those states to the current state and generating the current observation. Select the path with the highest probability at each step as the path to continue forward and save the information of this state. This step is implemented using a dynamic programming table (DP table), where each cell records the maximum probability of reaching that state and its predecessor state.

[0134] Path backtracking involves processing all observations and then starting from the last cell of the DP table (i.e., the optimal state at the last time point), tracing back along the previously recorded highest-probability path until returning to the initial state. During backtracking, the hidden state path with the highest probability in the entire sequence can be constructed based on the saved predecessor state information.

[0135] The advantage of the Viterbi algorithm lies in its ability to efficiently process long sequences of data, significantly reducing computational complexity by avoiding redundant calculations. Furthermore, the algorithm can be extended to find multiple n-best paths, which is very useful in some practical applications.

[0136] S52 employs a beam search approach, which, during neural network decoding, maintains a fixed-size candidate list or candidate bundle and progressively expands and retains the path with the highest score.

[0137] In neural network decoding, beam search is an efficient and widely used heuristic search algorithm, especially suitable for situations with a large solution space and limited computational resources. Its core idea is to maintain a fixed-size candidate list (or candidate beam) at each expansion step, gradually expanding and retaining the path with the highest score.

[0138] In its implementation, beam search begins with an initial node (such as an empty sequence) and then iteratively expands this node, generating all possible subsequent nodes (i.e., adding a new word or symbol). In each expansion step, the algorithm calculates a score for each candidate node based on a predefined scoring function (such as probability-based log-likelihood) and retains the top k nodes with the highest scores (k being the beam width). These k nodes are then used in the next round of expansion, while other nodes are discarded, thus effectively controlling the size of the search space.

[0139] As the search process progresses, the beam search continuously expands and updates the candidate beams until a stopping condition is met (such as reaching the maximum length, the candidate beam containing the target sequence, or resource exhaustion). Finally, the beam search returns the candidate sequence with the highest score as the decoding result.

[0140] S53 employs a connectionist temporal classification method, directly modeling sequences without requiring alignment of input and output sequences in the training data, thus simplifying the training and decoding process.

[0141] Connectionist Temporal Classification (CTC) is an efficient algorithm specifically designed for sequence labeling problems, particularly in fields such as speech recognition and handwriting recognition. It significantly simplifies the training and decoding process, eliminating the need for strict alignment of input and output sequences in the training data.

[0142] The core of the CTC algorithm lies in its unique loss function design, which can handle problems such as inconsistent lengths and ambiguous alignment between the input and output sequences. In CTC, the output label set is expanded by introducing a blank label to represent mismatches or silent segments between the input and output. This design allows CTC to handle many-to-one alignment relationships, meaning that one or more input elements can correspond to a single output element, but the reverse is not true.

[0143] During the training phase, CTC estimates the conditional probability of the output sequence given an input sequence by summing the probabilities of all possible alignment paths. These alignment paths, after merging duplicate characters and removing null tokens, can be mapped to the target output sequence. CTC uses dynamic programming to efficiently compute the probabilities of these paths, thus avoiding the computational complexity explosion problem caused by directly enumerating all possible paths.

[0144] During the decoding process, the CTC algorithm uses a trained model to predict the input sequence and outputs the most likely output sequence. Since CTC directly outputs the probability distribution of the sequence prediction, no external post-processing steps are required, further simplifying the decoding process.

[0145] In some optional implementations of this embodiment, after obtaining the text sequence through a search algorithm based on the outputs of the acoustic model and the language model in step S5, the following step is further included:

[0146] S6, by analyzing the emotional information in the speech signal, the speaker's emotional state is judged.

[0147] Step S6 provides new application scenarios for fields such as intelligent customer service and mental health.

[0148] Speech emotion recognition typically combines acoustic and linguistic features for analysis. First, acoustic features such as pitch, speech rate, and volume are extracted from the speech signal. Simultaneously, natural language processing techniques are used to perform sentiment analysis on the text content. Then, these two types of features are input into an emotion recognition model. The model learns feature patterns under different emotional states through training, ultimately achieving accurate recognition of speech emotions.

[0149] In some optional implementations of this embodiment, voice fingerprint recognition can also be used, which extracts and matches feature information in the voice to identify an individual. This method has broad application prospects in fields such as voice payment and voice unlocking.

[0150] Voice fingerprint recognition first requires preprocessing the voice signal to extract representative features, such as Mel-frequency cepstral coefficients (MFCCs). Then, these features are used to construct a voice fingerprint template library. During the recognition phase, the input voice signal is matched against fingerprints in the template library, and identity is determined by calculating the similarity. To improve recognition accuracy, complex matching algorithms and optimized feature extraction methods are typically employed.

[0151] In some optional implementations of this embodiment, speech recognition can also be combined with other interaction methods (such as images, gestures, etc.) to achieve a more natural and intelligent human-computer interaction. This multimodal interaction method can make full use of complementary information between different modalities, improving the accuracy and efficiency of the interaction.

[0152] Multimodal interaction fusion requires designing model architectures capable of simultaneously processing multiple input signals. For example, in smart home scenarios, speech recognition and image recognition technologies can be combined to control home appliances via voice commands and gestures. To achieve this, it is necessary to develop sensors and algorithms capable of simultaneously receiving and processing multiple input signals, and to design efficient fusion strategies to integrate information from different modalities.

[0153] The beneficial effects of implementing this embodiment are as follows: First, a speech signal is acquired; then, features are extracted from the speech signal; next, a mapping relationship between speech features and articulatory units is established based on the extracted speech signal features; then, language modeling is performed based on the mapping relationship between speech features and articulatory units, and statistical patterns in language are captured through language modeling to predict the probability of the next word or phrase appearing in a given context; finally, a text sequence is obtained through a search algorithm based on the output of the acoustic model and the language model; thus improving the accuracy, robustness, and real-time performance of speech recognition.

[0154] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0156] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0157] Example 2

[0158] Further reference Figure 3 As a response to the above Figure 2 The present invention provides an embodiment of a speech recognition device, which is similar to the method shown. Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0159] like Figure 3 As shown, the speech recognition device 60 in this embodiment includes: an acquisition module 61, a feature extraction module 62, a mapping module 63, a modeling module 64, and a text module 65. Wherein:

[0160] Acquisition module 61 is used to acquire voice signals;

[0161] The feature extraction module 62 is used to extract features from the speech signal based on the speech signal.

[0162] Mapping module 63 is used to establish a mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction of the speech signal;

[0163] Modeling module 64 is used to perform language modeling based on the mapping relationship between speech features and articulatory units. Through language modeling, statistical patterns in language are captured, and the probability of the next word or phrase appearing in a given context is predicted.

[0164] Text module 65 is used to obtain text sequences through a search algorithm based on the outputs of the acoustic model and the language model.

[0165] The beneficial effects of implementing this embodiment are as follows: First, a speech signal is acquired; then, features are extracted from the speech signal; next, a mapping relationship between speech features and articulatory units is established based on the extracted speech signal features; then, language modeling is performed based on the mapping relationship between speech features and articulatory units, and statistical patterns in language are captured through language modeling to predict the probability of the next word or phrase appearing in a given context; finally, a text sequence is obtained through a search algorithm based on the output of the acoustic model and the language model; thus improving the accuracy, robustness, and real-time performance of speech recognition.

[0166] Example 3

[0167] To address the aforementioned technical problems, embodiments of the present invention also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The aforementioned computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that only the computer device 7 with components 71, 72, and 73 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital digital processors (DSPs), embedded devices, etc.

[0169] The aforementioned computer devices can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. These devices can facilitate human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0170] The aforementioned memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the aforementioned memory 71 may be an internal storage unit of the aforementioned computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the aforementioned memory 71 may also be an external storage device of the aforementioned computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Of course, the aforementioned memory 71 may also include both the internal storage unit and its external storage device of the aforementioned computer device 7. In this embodiment, the aforementioned memory 71 is typically used to store the operating system and various application software installed on the aforementioned computer device 7, such as computer-readable instructions for speech recognition methods. In addition, the aforementioned memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0171] In some embodiments, the processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to execute computer-readable instructions stored in the memory 71 or to process data, such as executing computer-readable instructions for the speech recognition method.

[0172] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 7 and other electronic devices.

[0173] The beneficial effects of implementing this embodiment are as follows: First, a speech signal is acquired; then, features are extracted from the speech signal; next, a mapping relationship between speech features and articulatory units is established based on the extracted speech signal features; then, language modeling is performed based on the mapping relationship between speech features and articulatory units, and statistical patterns in language are captured through language modeling to predict the probability of the next word or phrase appearing in a given context; finally, a text sequence is obtained through a search algorithm based on the output of the acoustic model and the language model; thus improving the accuracy, robustness, and real-time performance of speech recognition.

[0174] Example 4

[0175] The present invention also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the speech recognition method described above.

[0176] The beneficial effects of implementing this embodiment are as follows: First, a speech signal is acquired; then, features are extracted from the speech signal; next, a mapping relationship between speech features and articulatory units is established based on the extracted speech signal features; then, language modeling is performed based on the mapping relationship between speech features and articulatory units, and statistical patterns in language are captured through language modeling to predict the probability of the next word or phrase appearing in a given context; finally, a text sequence is obtained through a search algorithm based on the output of the acoustic model and the language model; thus improving the accuracy, robustness, and real-time performance of speech recognition.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0178] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A speech recognition method, characterized in that, Includes the following steps: Acquire audio signals; Based on the speech signal, feature extraction is performed on the speech signal; Based on the speech signal features after feature extraction, a mapping relationship between speech features and articulatory units is established; Based on the mapping relationship between speech features and articulatory units, language modeling is performed. Through language modeling, statistical patterns in language are captured, and the probability of the next word or phrase appearing in a given context is predicted. Based on the outputs of the acoustic model and the language model, a search algorithm is used to obtain the text sequence.

2. The speech recognition method according to claim 1, characterized in that, The steps for acquiring the speech signal specifically include: Choose a microphone that meets the requirements for sensitivity, noise, and bandwidth, depending on the specific application scenario. Set up a microphone array to collect voice signals; Store the voice signal.

3. The speech recognition method according to claim 1, characterized in that, The step of extracting features from the speech signal specifically includes: The speech signal is preprocessed; Feature parameters useful for speech recognition are extracted from the preprocessed speech signal.

4. The speech recognition method according to claim 1, characterized in that, The step of establishing a mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction from the speech signal specifically includes: Establish a database of labeled speech data, with the labeling content including the start and end times of each pronunciation unit and the corresponding text or phoneme; Based on the extracted feature parameters and labeled data, train one or more classifiers or regressors to learn the mapping relationship between speech features and articulatory units; For the speech signal to be recognized, repeat the feature extraction steps to obtain the corresponding feature sequence; The extracted feature sequences are input into the trained model, which then predicts the corresponding vocal units for each time frame based on the learned mapping relationships.

5. The speech recognition method according to claim 1, characterized in that, The steps of performing language modeling based on the mapping relationship between speech features and articulatory units, capturing statistical patterns in language through language modeling, and predicting the probability of the next word or phrase appearing in a given context specifically include: A statistical language model is used, based on the statistical frequency method, to predict the next word by calculating the probability of N consecutive words or morphemes co-occurring, where N is a natural number greater than 1; By using neural networks to learn long-range dependencies and contextual information in language, probability distributions can be generated. A language model is constructed by combining historical dialogues, geographical locations, and external information about time.

6. The speech recognition method according to claim 1, characterized in that, The steps of obtaining the text sequence through a search algorithm based on the outputs of the acoustic model and the language model specifically include: The Viterbi algorithm is used to find the optimal state sequence given the state and transition probabilities through dynamic programming. The beam search approach is adopted in neural network decoding by maintaining a fixed-size candidate list or candidate beam, gradually expanding and retaining the path with the highest score; The connectionist temporal classification method is adopted to directly model the sequence, without the need to align the input and output sequences in the training data, thus simplifying the training and decoding process.

7. The speech recognition method according to any one of claims 1 to 6, characterized in that, Following the step of obtaining the text sequence using a search algorithm based on the outputs of the acoustic model and the language model, the method further includes: By analyzing the emotional information in the speech signal, the speaker's emotional state can be determined.

8. A voice recognition device, characterized in that, include: The acquisition module is used to acquire voice signals; The feature extraction module is used to extract features from the speech signal based on the speech signal. The mapping module is used to establish a mapping relationship between speech features and articulatory units based on the speech signal features after feature extraction of the speech signal; The modeling module is used to perform language modeling based on the mapping relationship between speech features and articulatory units. Through language modeling, it captures statistical patterns in language and predicts the probability of the next word or phrase appearing in a given context. The text module is used to obtain text sequences based on the outputs of the acoustic model and the language model using a search algorithm.

9. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech recognition method as described in any one of claims 1 to 7.

Images