Speech recognition method and device based on scene

Abstract

The present invention discloses a scene-based speech recognition method and device. The method includes: obtaining speech coding features of a speech signal to be recognized; obtaining scene keywords corresponding to the application scene of the speech signal to be recognized, and generating a scene feature vector containing context dependency based on the scene keywords; using a multi-layer bidirectional neural network, using context features of different depths layer by layer to extract deep hidden features related to the scene keywords from the scene feature vector; based on a dynamic attention mechanism, fusing the deep hidden features with the speech coding features to generate fused features containing the scene keywords; based on the fused features, calculating the probability distribution of the speech units of the speech signal to be recognized through a classifier, and classifying the speech units based on the probability distribution; performing speech recognition on the speech signal to be recognized according to the classification results of the speech units. The present invention solves the technical problem of inaccurate speech recognition in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a scene-based speech recognition method and device. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, speech recognition technology has been increasingly used in smart devices, in-vehicle systems, virtual assistants and other fields. However, existing speech recognition systems often face the problem of low recognition accuracy when processing speech signals in different application scenarios, especially in complex or changing environments, where the system's ability to understand and process speech signals is limited. This is mainly because existing technologies lack effective modeling of the context and sufficient mining of semantic associations during speech recognition, which makes it difficult for the system to accurately capture the deep information in the speech signal and cannot dynamically adjust the recognition strategy for specific scenarios.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a scene-based speech recognition method and device to at least solve the technical problem of inaccurate speech recognition in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a scene-based speech recognition method is provided, comprising: obtaining a speech signal to be recognized, and preprocessing the speech signal to be recognized to generate speech coding features; obtaining scene keywords corresponding to the application scenario of the speech signal to be recognized, and generating a scene feature vector containing context dependencies based on the scene keywords; using a multi-layer bidirectional neural network, using context features of different depths layer by layer to extract deep hidden features related to the scene keywords from the scene feature vector; based on a dynamic attention mechanism, fusing the deep hidden features with the speech coding features to generate fused features containing the scene keywords; based on the fused features, calculating the probability distribution of the speech units of the speech signal to be recognized through a classifier, and classifying the speech units based on the probability distribution; performing speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result.

[0006] According to another aspect of an embodiment of the present invention, a scene-based speech recognition device is also provided, including: an acquisition module, configured to acquire a speech signal to be recognized, and preprocess the speech signal to be recognized to generate speech coding features; a generation module, configured to acquire scene keywords corresponding to the application scenario of the speech signal to be recognized, and generate a scene feature vector containing context dependencies based on the scene keywords; an extraction module, configured to use context features of different depths layer by layer through a multi-layer bidirectional neural network to extract deep hidden features related to the scene keywords from the scene feature vector; a fusion module, configured to fuse the deep hidden features with the speech coding features based on a dynamic attention mechanism to generate fused features containing the scene keywords; a classification module, configured to calculate the probability distribution of the speech units of the speech signal to be recognized through a classifier based on the fused features, and classify the speech units based on the probability distribution; a recognition module, configured to perform speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result.

[0007] In an embodiment of the present invention, a speech signal to be recognized is obtained, and the speech signal to be recognized is preprocessed to generate speech coding features; a scene keyword corresponding to the application scene of the speech signal to be recognized is obtained, and a scene feature vector containing context dependency is generated based on the scene keyword; through a multi-layer bidirectional neural network, context features of different depths are used layer by layer to extract deep hidden features related to the scene keywords from the scene feature vector; based on a dynamic attention mechanism, the deep hidden features are fused with the speech coding features to generate fused features containing the scene keywords; based on the fused features, the probability distribution of the speech units of the speech signal to be recognized is calculated by a classifier, and the speech units are classified based on the probability distribution; according to the classification result of the speech unit, speech recognition is performed on the speech signal to be recognized to obtain a recognized speech result. The above scheme solves the technical problem of inaccurate speech recognition in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0009] Figure 1 is an architecture diagram of a scene-based speech recognition system according to an embodiment of the present invention;

[0010] Figure 2 is a flow chart of a scene-based speech recognition method according to an embodiment of the present invention;

[0011] Figure 3 is a flow chart of another scene-based speech recognition method according to an embodiment of the present invention;

[0012] Figure 4 is a flow chart of a method for generating a corresponding scene feature vector according to an embodiment of the present invention;

[0013] Figure 5 is an architecture diagram of a neural network model based on an attention mechanism according to an embodiment of the present invention;

[0014] Figure 6 4 is a schematic diagram of the structure of a scene-based speech recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] According to an embodiment of the present invention, a method embodiment of a scene-based speech recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0018] Artificial intelligence (AI) refers to technologies and systems that simulate human intelligence through computers and algorithms. AI enables machines to perform tasks that usually require human intelligence, including learning, reasoning, problem solving, perception, natural language understanding, image recognition, decision-making, etc. The core purpose of AI is to develop systems that can imitate, enhance or replace human thinking and behavior, and improve the ability to automate and handle complex tasks.

[0019] Natural language processing is a branch of artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language. Its core goal is to enable computers to process and analyze information in natural language to achieve effective communication between humans and machines.

[0020] Machine learning is a branch of artificial intelligence that refers to the ability of computer systems to automatically learn and improve from data through algorithms and statistical models without explicit programming instructions. In other words, machine learning enables systems to analyze and predict based on input data, so as to make decisions or actions when encountering similar data in the future.

[0021] With the continuous development and innovation of artificial intelligence technology, AI technology has been widely studied and applied in many fields. For example, application scenarios such as smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, and smart transportation can be seen everywhere. In the future, as the technology matures further, artificial intelligence will show greater potential in more industries and fields and play an increasingly important role.

[0022] The speech recognition method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 responds to the speech recognition request for the speech to be recognized and sends the speech recognition request to the server. The server 104 receives the speech recognition request, parses the speech recognition request to obtain the speech signal to be recognized, and pre-processes the speech signal to be recognized to generate speech coding features; obtains the scene keyword corresponding to the application scene of the speech signal to be recognized, and generates a scene feature vector containing context dependency based on the scene keyword; uses a multi-layer bidirectional neural network, layer by layer, using context features of different depths to extract deep hidden features related to the scene keyword from the scene feature vector; based on the dynamic attention mechanism, fuses the deep hidden features with the speech coding features to generate fused features containing the scene keywords; based on the fused features, calculates the probability distribution of the speech unit of the speech signal to be recognized through a classifier, and classifies the speech unit based on the probability distribution; performs speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain the recognized speech result. The terminal 102 receives the recognition result and displays it. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0023] Figure 2 is a scene-based speech recognition method according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0024] Step S202, obtaining a speech signal to be recognized, and preprocessing the speech signal to be recognized to generate speech coding features;

[0025] The system obtains the original speech signal to be recognized through a microphone or other voice acquisition device. Since speech signals usually contain interference factors such as background noise and irregular volume changes, the speech signal needs to be preprocessed before formal processing to improve the quality and recognizability of the speech signal. This application can remove background noise and improve signal quality by preprocessing the speech signal.

[0026] After the noise reduction process is completed, the speech signal may still have uneven amplitude and energy distribution. Therefore, the present application further ensures the consistency and stability of the signal through normalization processing. Normalization processing includes peak normalization, energy normalization and zero mean normalization. The above processing can balance the volume differences between different speech segments and eliminate the influence of low-frequency components caused by DC offset or other factors, thereby generating a smoother and more uniform speech signal.

[0027] After noise reduction and normalization, the preprocessed speech signal is subjected to coding feature extraction. First, the short-time Fourier transform (STFT) is used to convert the time domain signal into a frequency domain signal to extract the time-frequency features of the speech. Subsequently, speech coding features such as Mel-frequency cepstral coefficients (MFCC) are extracted from the generated spectrogram.

[0028] Step S204, obtaining a scene keyword corresponding to the application scene of the speech signal to be recognized, and generating a scene feature vector containing context dependency based on the scene keyword;

[0029] According to the application scenario, the scenario keywords corresponding to the application scenario are extracted from the keyword library, and the scenario keywords are mapped to vector representations through word embedding technology to obtain word vectors related to the application scenario; the word vectors are processed using a context modeling method to generate the scenario feature vector containing context dependency. The embodiment of the present application can effectively capture scene-specific semantic information by extracting keywords related to the application scenario from the keyword library and converting them into vector representations using word embedding technology. In addition, further processing these word vectors in combination with the context modeling method to generate a scene feature vector containing context dependency can help enhance the model's understanding of the scene context, thereby improving the accuracy and adaptability of speech recognition.

[0030] Step S206, using a multi-layer bidirectional neural network, layer by layer, using context features of different depths to extract deep hidden features related to the scene keywords from the scene feature vector;

[0031] First, through the leading layer of the multi-layer bidirectional neural network, based on forward and backward information flow processing, the local dependency relationship and micro-semantic pattern between adjacent words in the scene feature vector are captured to identify the inter-word association relationship features in the local context. Then, through the middle layer of the multi-layer bidirectional neural network, through recursive calculation and inter-layer feature transfer, the long-distance dependency across multiple words in the sentence is extracted from the scene feature vector to capture the cross-context association features in the global semantics; then, through the highest layer of the multi-layer bidirectional neural network, the inter-word association relationship and the cross-context association features are integrated, and the overall background information of the application scene is captured using the global attention mechanism to obtain the wide-area semantic structure features; finally, the hidden layer state of the highest layer is nonlinearly mapped through the activation function, and the inter-word association relationship features, the cross-context association features, and the wide-area semantic structure features are fused and weighted to generate the deep hidden layer features.

[0032] This application uses hierarchical processing of multi-layer bidirectional neural networks to gradually extract rich semantic features from local to global. The pre-order layer captures the local dependencies and micro-semantic patterns between adjacent words, the middle layer obtains the long-distance dependencies across multiple words in the sentence through recursive calculation, and the top layer integrates local and global information through the global attention mechanism to generate deep hidden features with wide-area semantics. This hierarchical processing method enhances the semantic understanding of speech signals and effectively improves the accuracy of speech recognition in complex scenarios.

[0033] Step S208, based on a dynamic attention mechanism, fusing the deep hidden layer features with the speech coding features to generate fused features containing the scene keywords;

[0034] For example, the correlation between the deep hidden features and the speech coding features is calculated, and weights are assigned to the deep hidden features based on the dynamic attention mechanism; the deep hidden features whose weights are greater than a preset threshold are fused with the speech coding features to generate fused features containing the scene keywords.

[0035] This application can effectively improve the recognition accuracy of key features by calculating the correlation between deep hidden features and speech coding features and assigning weights to deep hidden features based on a dynamic attention mechanism. At the same time, fusing deep hidden features with weights greater than a preset threshold with speech coding features helps generate more representative fusion features, thereby improving the recognition effect of scene keywords.

[0036] Step S210, based on the fusion features, a probability distribution of the speech units of the speech signal to be recognized is calculated by a classifier, and the speech units are classified based on the probability distribution;

[0037] For example, the fused features are nonlinearly mapped layer by layer through the multi-layer neurons of the classifier, the fused features are mapped to the corresponding categories of the speech units, and the probability value of the category of the speech units is calculated through the activation function to generate the probability distribution of the speech units; based on the probability distribution, the speech units are classified by maximizing the likelihood. The present application performs nonlinear mapping of the fused features layer by layer through the multi-layer neurons of the classifier, and accurately maps the fused features to the categories of the corresponding speech units, thereby improving the accuracy of classification. The probability value of the speech unit category is calculated by the activation function to generate a probability distribution, making the classification process more robust and accurate. Based on this probability distribution, the maximum likelihood method is used for classification, which not only improves the generalization ability of the model, but also enhances the reliability and stability of speech unit classification.

[0038] Step S212: performing speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result.

[0039] The speech units are reorganized according to the classification results of the speech units to hierarchically reconstruct the sentence structure in the speech signal to be recognized; the post-processing technology of grammatical rules is used to correct the reconstructed sentence structure in the speech signal to be recognized, so as to obtain the recognized speech result. The present application performs reorganization according to the classification results of the speech units, thereby realizing the hierarchical reconstruction of the sentence structure in the speech signal to be recognized, thereby effectively improving the structural accuracy of speech recognition. In addition, through the post-processing technology of grammatical rules, the reconstructed sentence structure is further corrected, and the grammatical correctness and language fluency of the speech recognition results are optimized. The present application not only improves the overall accuracy of speech recognition, but also enhances the processing capability of complex speech signals, making the recognition results more in line with the expression of natural language.

[0040] Figure 3 FIG. 1 is a flow chart of a scene-based speech recognition method according to another embodiment of the present invention. The method comprises the following steps:

[0041] Step S302: extracting speech coding features from the speech signal to be recognized.

[0042] Speech signals are often accompanied by background noise and other interference factors. In order to eliminate these interferences, this application uses filtering technology for noise reduction processing. Specifically, high-frequency noise is removed by a low-pass filter, and low-frequency noise is filtered out by a high-pass filter to ensure that the main frequency components of the speech signal are retained. In addition, the system can also dynamically adjust the parameters of the filter according to the changes in environmental noise through adaptive filtering technology, thereby improving the noise reduction effect and ensuring a clear speech signal.

[0043] After removing the noise, the speech signal to be recognized is framed and windowed. The speech signal is non-stationary. The purpose of framing is to divide it into multiple short-term stable frame segments, usually 20-40 milliseconds per frame. During the framing process, there will be some overlap between frames (usually 50%) to ensure the continuity of the signal. Subsequently, each frame signal is multiplied by a window function (such as a Hamming window or a Hanning window) to reduce the spectrum leakage problem caused by discontinuities at the boundaries through windowing.

[0044] Next, frequency domain conversion is performed, that is, the time domain signal of each frame is converted into a frequency domain signal through short-time Fourier transform (STFT). STFT can analyze the spectrum information of the speech signal and reveal the distribution of frequency components in each frame. The spectrum obtained by STFT contains the amplitude and phase information of the speech signal.

[0045] Feature extraction is then performed. First, the power spectrum is obtained by calculating the square of the spectrum to describe the energy distribution of each frequency component. Then, the power spectrum is mapped to the Mel frequency scale that conforms to human auditory perception using a Mel filter bank. The result is logarithmic and subjected to discrete cosine transform (DCT) to extract MFCC features. In addition, the linear prediction cepstral coefficient (LPCC) reflects the characteristics of the speech channel through linear prediction analysis. LPCC and MFCC are the most common feature expressions in speech signals.

[0046] In the feature combination and encoding stage, all extracted features are combined into feature vectors. In order to optimize the computational efficiency and the performance of the results, methods such as principal component analysis (PCA) or linear discriminant analysis (LDA) are usually used to reduce the dimension of the features, remove redundant features, and retain the most informative parts.

[0047] Step S304: Obtain scene keywords related to the usage scene of the voice signal to be recognized, and generate a corresponding scene feature vector.

[0048] like Figure 4 As shown, the method for generating the corresponding scene feature vector includes the following steps:

[0049] Step S3042: extract scene keywords from the keyword library.

[0050] First, according to the current application scenario, a set of keywords matching the scenario is extracted from the pre-built keyword library. The keyword library can be defined and constructed in advance according to different application scenarios (such as smart guidance, smart home, autonomous driving, etc.), and contains specific terms, expressions and common words related to various scenarios. When the application scenario is identified or input, the system retrieves keywords related to the scenario from the corresponding scenario entry. For example, in the smart home scenario, keywords may include "lighting control", "temperature adjustment", "voice command", etc., while in the autonomous driving scenario, keywords may include "route planning", "obstacle detection", "vehicle speed adjustment", etc. In the hospital's intelligent guidance scenario, keywords may be "navigation", "floor guidance", "clinic location", etc.

[0051] Step S3044, mapping the keywords into vector representations through word embedding technology.

[0052] After the scene keywords are extracted, these keywords are converted into vector representations using word embedding technology. Word embedding technology maps discrete words into low-dimensional continuous vector space, so that the semantic similarity and contextual relationship between words can be effectively captured. In the present invention, Word2Vec, GloVe, or BERT can be used to convert the extracted scene keywords into vector representations. The vector representation corresponding to each keyword contains the semantic information of the word, and the distribution of the vector space can reflect the correlation between different keywords.

[0053] Through word embedding, the vector representation of keywords not only retains the semantics of the words themselves, but also reveals the potential connection between scene keywords through the similarity of word vectors. For example, the keywords "temperature adjustment" and "air conditioning control" may be close in the word embedding space, indicating that they are semantically related.

[0054] Step S3046, use the context modeling method to process the word vector.

[0055] After obtaining the word vector corresponding to the scene keyword, the word vector is further processed through the context modeling method to generate a scene feature vector containing context dependencies. Context modeling is to place the vocabulary in a specific scene for semantic interpretation, so that the system can understand the changes in the meaning of the vocabulary in different situations.

[0056] Specifically, context modeling can be based on sequence models (such as LSTM, GRU and other recurrent neural networks) or on Figure 5The model can capture the dependencies between words in different contexts from the word vector sequence and dynamically adjust the weights between words. For example, in the "smart home" scenario, the keywords "lighting control" and "voice command" may be more closely related, so context modeling can assign a higher weight to the keyword "voice command", making the system more inclined to capture the semantic relationship between these keywords.

[0057] Step S3048, generating a scene feature vector including context dependency.

[0058] Through the word vectors processed by context modeling, a set of scene feature vectors with context dependencies are generated. These scene feature vectors not only contain the semantic information of scene keywords, but also embed the dependency relationship and context information between words, which can more comprehensively reflect the semantic structure in the current application scenario. The scene feature vectors can be used in subsequent semantic analysis, speech recognition or natural language processing tasks. For example, in intelligent voice assistants, these scene feature vectors can help the system understand the user's voice commands in specific scenarios more accurately and make corresponding intelligent responses.

[0059] Through the above steps, the present invention can efficiently extract keywords related to application scenarios, and generate scenario feature vectors through word embedding and context modeling, which significantly improves the system's ability to understand semantics and provides strong support for scenario-based intelligent applications.

[0060] Step S306: extracting deep hidden features related to the scene keywords.

[0061] First, we need to build a multi-layer bidirectional neural network structure. A multi-layer bidirectional neural network contains multiple layers of bidirectional recurrent neural networks (RNNs), such as bidirectional long short-term memory networks (BiLSTM) or bidirectional gated recurrent units (BiGRU). These layers process the information in the input sequence through forward and backward information flow processing to ensure that important features in the context can be captured.

[0062] The pre-order layer is used as the primary layer of the multi-layer bidirectional neural network to convert the scene feature vector into a high-dimensional feature representation. The pre-order layer includes an embedding layer for converting each word into a dense vector representation. Subsequently, the bidirectional RNN structure is used to process the word sequence through the forward information flow (from the beginning to the end of the sequence of the scene feature vector) and the backward information flow (from the end to the beginning of the sequence of the scene feature vector). The bidirectional processing method of the present application can take into account the contextual information of the words at the same time, thereby improving the ability to capture local dependencies and micro-semantic patterns. Specifically, each layer of the network captures the local dependencies between words through a recursive structure, that is, the relationship between adjacent words. The bidirectional structure enables the network to pay attention to the contextual information of the context, thereby capturing micro-semantic patterns more comprehensively. On the whole, through the processing of the pre-order layer, the network can generate the contextual representation of each word, that is, the inter-word association relationship feature, which reflects the local contextual features between words and can help the network accurately identify the association between words in the local context.

[0063] The middle layer of the multi-layer bidirectional neural network uses recursive calculations to deeply mine the contextual information of each word in the sentence, so that the time series dependency between words can be effectively captured. Specifically, the hidden state is continuously updated and transmitted through the recursive structure of the network, and a representation of the long-distance dependency is formed in the middle layer. In addition, the feature information of the middle layer is transmitted between different levels of the network. For example, jump connections or residual connections can be used to achieve efficient transmission of information in the middle layer. Through recursive calculations of the middle layer and inter-layer feature transmission, the present application can extract long-distance dependencies across multiple words in a sentence from the scene feature vector, which helps to capture the global semantic information in the sentence and its cross-context correlation features. Through multi-level feature extraction and synthesis, the network can identify the complex relationships between each word in the sentence, and can apply these relationships to specific tasks, such as semantic understanding, text classification, etc.

[0064] The top layer of the multi-layer bidirectional neural network receives inter-lexical association features and cross-context association features. The first few layers of the bidirectional neural network have captured the semantic information of words and sentences in different contexts through forward and backward propagation, allowing the top layer to synthesize these associations and further extract useful semantic features. Specifically, the top layer performs global context association calculations on the input words through the global attention mechanism. The role of the global attention mechanism is to include all words in the attention range and assign weights based on the correlation between each word and other words, thereby capturing the overall background information of the application scenario. Through this mechanism, the top layer can identify deep semantic associations between words, not only limited to the local context, but also generate wide-area semantic structural features that represent the overall background information of the application scenario.

[0065] After capturing the inter-lexical association features, cross-context association features, and wide-area semantic structure features, the top layer integrates these features. The hidden state of the top layer is nonlinearly mapped by the activation function, so that the entire network can handle complex nonlinear relationships. The activation function (such as ReLU, Tanh, or Sigmoid, etc.) maps the linear combination features in the hidden layer to nonlinear results, thereby ensuring that the network has stronger expressive power and can capture more complex semantic patterns. Through the nonlinear mapping process, the top layer fuses the extracted inter-lexical association features, cross-context association features, and wide-area semantic structure features, and adjusts the weights according to the importance of the input features. Finally, after being processed by the activation function, deep hidden features are generated. The deep hidden features not only contain local and global semantic information, but also can reflect the wide-area semantic structure of the application scenario in a nonlinear, fusion-optimized manner.

[0066] Step S308: Generate fusion features containing the scene keywords.

[0067] The deep hidden features include the inter-lexical association features, cross-context association features and wide-area semantic structure features of the application scenario. The speech coding features come from the processing of speech signals, for example, the frequency, timing and other speech information extracted by the encoder (such as a convolutional neural network or a self-attention network). During the fusion process, the dynamic attention mechanism can dynamically adjust the weight allocation of different features according to the changes in the deep hidden features. Specifically, the dynamic attention mechanism assigns a weight value to each deep hidden feature based on the correlation between the input deep hidden features and the speech coding features to determine the importance of different features. Unlike traditional static attention, dynamic attention can be adaptively adjusted as the input features change, ensuring that more accurate keywords can be captured in different scenarios. In some embodiments, in addition to assigning weights to deep hidden features, weights can also be assigned to speech coding features.

[0068] Through the dynamic attention mechanism, the network can automatically determine which deep hidden features and speech coding features are more important for keyword recognition in the current scene. Then, the attention mechanism will give these important features higher weights and reduce the weights of relatively unimportant features, so as to more effectively highlight key information during fusion. After the feature weight adjustment is completed, the deep hidden features and speech coding features will be weighted fused. Through linear combination, the features optimized by the attention mechanism are integrated so that the fused features contain both the semantic information in the text and the key information in the speech. Finally, the features fused by the dynamic attention mechanism will generate fused features containing scene keywords. These fused features can efficiently express the core semantic information in the scene, not only can they capture keywords through text analysis, but also can combine the audio features in the speech signal to improve the accuracy and robustness of keyword recognition.

[0069] Step S310: classify the speech units based on probability distribution.

[0070] 1) Layer-by-layer nonlinear mapping.

[0071] The classifier consists of a multi-layer neural network, and each layer of neurons receives the output features of the previous layer. First, the input fused features are linearly transformed in each layer to generate intermediate results. The linear transformation uses a weight matrix and a bias vector to map the fused features to a higher dimension or a different feature space. Then, the activation function is used to perform nonlinear processing on the results of the linear transformation. The role of the activation function is to introduce nonlinear characteristics, so that the classifier can handle complex patterns and relationships and enhance the expressiveness of the features. As the number of network layers increases, the layer-by-layer nonlinear mapping can extract deeper semantic information from the input features, making the correlation between the fused features and the speech unit categories more obvious.

[0072] For example, the result after linear transformation can be generated based on the index of the current network layer, the number of different scales (used for multi-scale feature mapping, each scale captures different levels of information), the weight matrix of the current network layer and the current scale (representing the linear transformation of the input and hidden layer features), the output features of the previous layer (i.e., the input of the current layer), and the bias vector of the current scale. Then, the result after linear transformation is nonlinearly mapped using the activation function to obtain the result of nonlinear mapping. The present application utilizes multi-scale feature mapping and layer-by-layer linear transformation and nonlinear activation to extract rich feature information from different scales and levels. By capturing features of different levels at multiple scales, the pattern of input data can be better understood locally and globally. In addition, the linear transformation of the weight matrix and the bias vector enables the network to flexibly adjust the input features, and introduces nonlinearity through the activation function to enhance the model's ability to learn complex patterns and relationships. Finally, the nonlinear mapping result enables the network to more deeply explore the potential representation of features and improve the overall expression ability and classification effect.

[0073] ·2) Calculation of category probability values.

[0074] In the last layer of the neural network, the features after multi-layer processing will be mapped to the output space, which corresponds to different speech unit categories. Each category generates a score to indicate the relevance of the input feature to the category. These scores are not the final classification results, but the scores are converted into probability values ​​through the Softmax activation function. For example, the category score can be generated based on the weight matrix of the last layer of the network, the output features of the previous layer (i.e., the input of the last layer), and the bias vector of the last layer. Specifically, the weight matrix of the last layer maps the output features of the previous layer (usually representing the high-level semantic information extracted by the multi-layer neural network) to the category space, thereby generating the score of each category. The bias vector of the last layer is used to adjust the category score result to make it more suitable for specific classification tasks. The present application uses the weight matrix and bias vector of the last layer of the network to accurately map the high-level semantic information extracted by the previous layer to the category space, thereby generating scores of different categories, effectively converting complex feature representations into category-related linear outputs, so that the score of each category can reflect the relevance of the input features to the category. At the same time, by introducing the bias vector, the category score results can be adjusted more flexibly to adapt to the needs of specific classification tasks and improve the accuracy and generalization ability of classification.

[0075] The Softmax function maps multiple score values ​​to probability values ​​between 0 and 1, and ensures that the sum of the probabilities of all categories is 1. The Softmax function normalizes them into an interpretable probability distribution by calculating the exponential value of each category score. The probability value of each category represents the confidence of the classifier that the input feature belongs to that category. In this way, the classifier generates a probability distribution for each speech unit category and determines the possibility of the input fusion feature in different categories. For example, the category probability value can be calculated based on the score of each category and the correlation weight matrix between categories, combined with the temperature parameter. Specifically, first, for each category, the scores of other categories are weighted and summed by the correlation weights between categories to capture the dependencies between categories. Then, this weighted sum is divided by the temperature parameter for adjustment to control the smoothness of the output distribution. The score is converted to a positive number by an exponential function, and then normalized using the Softmax function so that the sum of the probabilities of all categories is 1, and finally the probability distribution of each category is obtained. By introducing the correlation weights between categories, this application can effectively capture the dependencies between different categories and enhance the model's perception of complex category structures. At the same time, the introduction of the temperature parameter provides an adjustment for the smoothness of the output distribution, making the model more robust during the training process. In the early training stage, a higher temperature helps to make the probability distribution smoother, thereby avoiding premature focus on a single category and encouraging the model to explore a variety of possible classification decisions. In the inference stage, a lower temperature can make the Softmax output sharper and improve the decision accuracy of the model. In short, in this way, a more reasonable and balanced probability distribution can be generated, ensuring more accurate classification decisions in multi-category tasks.

[0076] 3) Maximizing likelihood classification based on probability distribution.

[0077] After generating the probability distribution, the system classifies the speech units by maximizing the likelihood method. Select the category that makes the fusion feature appear with the highest probability, in other words, select the category with the maximum value in the probability distribution as the final classification result. For example, the system scans the probability distribution output by Softmax and selects the speech unit category with the highest probability value as the final category of the input. This method ensures that the classifier can make the best classification decision among multiple possible categories. By maximizing the likelihood method, the classifier can effectively classify the speech units, that is, identify which speech category the fusion feature is most likely to belong to. For example, based on the probability distribution output by the Softmax function, the classification can be maximized by selecting the category with the maximum probability value. Specifically, the probability value of each category is scanned, and the category with the highest probability is selected as the final classification result. Based on the principle of probability maximization, the present application ensures that the category to which the input feature is most likely to belong is selected as the final predicted category, so that the input data can be effectively classified and the model can be ensured to make the best decision between different categories.

[0078] Through the above process, the classifier processes the fusion features layer by layer through a multi-layer neural network, generates a probability distribution containing the probability value of each category, and achieves accurate classification of speech units through the maximum likelihood method. In this way, it ensures that the classifier can extract effective information from local and global semantics and make the best classification decision when facing complex fusion features.

[0079] Step S312: performing speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result.

[0080] After preliminary processing, speech signals are decomposed into several speech units (such as phonemes, words, syllables, etc.). These speech units are classified by the classifier to generate category information for each speech unit. The classifier maps speech units to specific language unit categories through the previously trained model and determines the category labels of these phonemes or words, such as syllables, words, or phrases. The classification results provide the basis for subsequent reorganization and reconstruction.

[0081] Based on the classification results, the system will reorganize these speech units and arrange them in chronological order or contextual logic. The reorganization process not only simply connects these speech units in sequence, but also constructs a hierarchical sentence structure based on the semantic and syntactic relationships between speech units.

[0082] The reorganization of speech units adopts a hierarchical approach to reconstruct sentence structure. The system will first identify smaller language units, such as phonemes and phrases, and then combine these small units into larger language structures, such as phrases and clauses, based on language models or syntactic rules, and finally construct complete sentences. This step-by-step construction method can better preserve the syntactic relationship between speech units, ensuring that the reorganized sentence structure is reasonable and conforms to natural language expression habits.

[0083] During the reorganization process, the system not only uses the classification results of individual speech units, but also uses contextual information to ensure the coherence of the reorganized sentence structure. For example, if some words or phonemes are ambiguous in the context, the system will use the language model to infer the most likely combination. By combining context and hierarchical modeling, the system is able to generate sentence structures that match the actual speech signal.

[0084] After the initial reorganization of speech units, the resulting sentence structure may have some grammatical inconsistencies or errors. To this end, the system will apply post-processing technology based on grammatical rules to correct and optimize the reconstructed sentence structure. The system has a built-in grammatical rule library that covers the grammatical structure, syntactic rules, vocabulary collocation, etc. of the target language. For example, the grammatical rule library can handle common grammatical problems such as subject-verb agreement, tense selection, word collocation errors, and syntactic errors in sentences. According to these rules, the system will check the reconstructed sentences one by one to find possible grammatical errors or unreasonable structures. The system automatically corrects each grammatical error or inconsistency by traversing the reconstructed sentences. For example, the system finds that the number between the subject and the predicate is inconsistent, or the verb tense is used improperly. Through grammatical rule correction, these problems can be adjusted and the grammatical correctness of the sentence can be ensured. At the same time, the correction process will also take into account the contextual relationship of the vocabulary to avoid semantic confusion caused by correcting a single grammatical error.

[0085] By reorganizing speech units and correcting grammatical rules, the system generates sentences that are well-structured and grammatical. The final result is the text output of speech recognition, which can usually be further processed or applied to actual scenarios, such as displaying text, executing commands, or storing as records.

[0086] In the final output stage, the system may also perform some optimization processing to ensure the semantic coherence and continuity of the generated sentences. For multiple sentences, the system will check punctuation and the logical connection between sentences to ensure that the entire speech recognition result is a complete and fluent expression.

[0087] After classifying the speech units through the classifier, this application performs hierarchical reconstruction based on the classification results of the speech units, and gradually builds a complete sentence structure. Then, the post-processing technology of grammatical rules is used to correct grammatical errors and structural problems to generate the final speech recognition results. The whole process combines the classification of speech units, contextual information, syntactic reconstruction and grammatical correction to ensure that the generated speech recognition results are not only accurate and coherent, but also conform to the grammatical rules of the language.

[0088] The present application also provides a scene-based speech recognition device, such as Figure 6 As shown, it includes: an acquisition module 62, which is configured to acquire a speech signal to be recognized and pre-process the speech signal to be recognized to generate speech coding features; a generation module 64, which is configured to acquire scene keywords corresponding to the application scenario of the speech signal to be recognized, and generate a scene feature vector containing context dependencies based on the scene keywords; an extraction module 66, which is configured to use context features of different depths layer by layer through a multi-layer bidirectional neural network to extract deep hidden features related to the scene keywords from the scene feature vector; a fusion module 68, which is configured to fuse the deep hidden features with the speech coding features based on a dynamic attention mechanism to generate fusion features containing the scene keywords; a classification module 70, which is configured to calculate the probability distribution of the speech units of the speech signal to be recognized through a classifier based on the fusion features, and classify the speech units based on the probability distribution; a recognition module 72, which is configured to perform speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result.

[0089] It should be noted that the speech recognition device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the speech recognition device provided in the above embodiment and the speech recognition method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0090] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A scene-based speech recognition method, characterized in that: include: Acquire a speech signal to be recognized, and preprocess the speech signal to be recognized to generate speech coding features; Acquire a scene keyword corresponding to an application scene of the speech signal to be recognized, and generate a scene feature vector containing context dependency based on the scene keyword; By using a multi-layer bidirectional neural network, context features of different depths are used layer by layer to extract deep hidden features related to the scene keywords from the scene feature vector; Based on a dynamic attention mechanism, the deep hidden layer features are fused with the speech coding features to generate fused features containing the scene keywords; Based on the fusion features, a probability distribution of the speech units of the speech signal to be recognized is calculated by a classifier, and the speech units are classified based on the probability distribution; According to the classification result of the speech unit, performing speech recognition on the speech signal to be recognized to obtain a recognized speech result; Among them, through a multi-layer bidirectional neural network, context features of different depths are used layer by layer to extract deep hidden features related to the scene keywords from the scene feature vector, including: through the leading layer of the multi-layer bidirectional neural network, based on forward and backward information flow processing, the local dependency relationship and micro-semantic pattern between adjacent words in the scene feature vector are captured to identify the inter-word association relationship characteristics in the local context; through the middle layer of the multi-layer bidirectional neural network, through recursive calculation and inter-layer feature transfer, the long-distance dependency relationship across multiple words in the sentence is extracted from the scene feature vector to capture the cross-context association characteristics in the global semantics; through the highest layer of the multi-layer bidirectional neural network, the inter-word association relationship and the cross-context association features are integrated, and the global attention mechanism is used to capture the overall background information of the application scene to obtain the wide-area semantic structure features; the hidden state of the highest layer is nonlinearly mapped through the activation function, and the inter-word association relationship features, the cross-context association features, and the wide-area semantic structure features are fused and weighted to generate the deep hidden features.

2. The method according to claim 1, characterized in that Based on the fusion features, a probability distribution of the speech units of the speech signal to be recognized is calculated by a classifier, and the speech units are classified based on the probability distribution, including: The fused features are nonlinearly mapped layer by layer through the multi-layer neurons of the classifier, the fused features are mapped to the corresponding categories of the speech units, and the probability values ​​of the categories of the speech units are calculated through an activation function to generate a probability distribution of the speech units; Based on the probability distribution, the speech units are classified using a maximum likelihood method.

3. The method according to claim 1, characterized in that Acquiring a scene keyword corresponding to an application scene of the speech signal to be recognized, and generating a scene feature vector containing context dependency based on the scene keyword, including: According to the application scenario, extract the scenario keywords corresponding to the application scenario from the keyword library, and map the scenario keywords into vector representations through word embedding technology to obtain word vectors related to the application scenario; The word vector is processed using a context modeling method to generate the scene feature vector containing context dependency.

4. The method according to claim 1, characterized in that Based on the dynamic attention mechanism, the deep hidden layer features are fused with the speech coding features to generate fused features containing the scene keywords, including: Calculating the correlation between the deep hidden layer features and the speech coding features, and assigning weights to the deep hidden layer features based on the dynamic attention mechanism; The deep hidden layer features whose weights are greater than a preset threshold are fused with the speech coding features to generate the fused features containing the scene keywords.

5. The method according to claim 1, characterized in that According to the classification result of the speech unit, speech recognition is performed on the speech signal to be recognized to obtain a recognized speech result, including: Reorganizing the speech units according to the classification results of the speech units to hierarchically reconstruct the sentence structure in the speech signal to be recognized; The sentence structure in the reconstructed speech signal to be recognized is corrected by using post-processing technology of grammatical rules to obtain the recognized speech result.

6. A scene-based speech recognition device, characterized in that: include: An acquisition module is configured to acquire a speech signal to be recognized and preprocess the speech signal to be recognized to generate speech coding features; A generating module configured to obtain a scene keyword corresponding to an application scene of the speech signal to be recognized, and generate a scene feature vector containing context dependency based on the scene keyword; An extraction module is configured to extract deep hidden features related to the scene keywords from the scene feature vector by using context features of different depths layer by layer through a multi-layer bidirectional neural network; A fusion module is configured to fuse the deep hidden layer features with the speech coding features based on a dynamic attention mechanism to generate a fusion feature containing the scene keywords; A classification module, configured to calculate the probability distribution of the speech units of the speech signal to be recognized through a classifier based on the fusion feature, and classify the speech units based on the probability distribution; A recognition module, configured to perform speech recognition on the speech signal to be recognized according to the classification result of the speech unit to obtain a recognized speech result; Among them, the extraction module is also configured to capture the local dependencies and micro-semantic patterns between adjacent words in the scene feature vector through the leading layer of the multi-layer bidirectional neural network, based on forward and backward information flow processing, so as to identify the inter-word association relationship features in the local context; through the middle layer of the multi-layer bidirectional neural network, through recursive calculation and inter-layer feature transfer, extract the long-distance dependency across multiple words in the sentence from the scene feature vector to capture the cross-context association features in the global semantics; through the highest layer of the multi-layer bidirectional neural network, integrate the inter-word association relationship and the cross-context association features, and use the global attention mechanism to capture the overall background information of the application scenario to obtain the wide-area semantic structure features; nonlinearly map the hidden layer state of the highest layer through the activation function, fuse and weight the inter-word association relationship features, the cross-context association features, and the wide-area semantic structure features to generate the deep hidden layer features.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

8. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images