Adaptive traceability analysis method for speech recognition errors, storage medium and device

Abstract

The application discloses a speech recognition error adaptive trace analysis method, a storage medium and equipment. Error patterns are automatically discovered through comparison clustering, so that the error classification accuracy is improved. A dynamic trace graph constructed by a graph neural network reveals hidden error correlations, displays the conduction process of errors from acoustic features to final output, and calculates and recommends an intervention priority list through causal intervention, so that the repair efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to an adaptive source analysis method, storage medium, and device for speech recognition errors. Background Technology

[0002] A speech recognition system mainly consists of four core modules: an audio input layer, an acoustic model, a language model, and an output layer.

[0003] 1. Audio Input Layer (Feature Extraction) The audio input layer is responsible for receiving raw speech signals, performing preprocessing (such as framing and windowing), and extracting audio features (such as Mel-spectral features), converting continuous sound signals into machine-understandable digital feature vectors. It is the starting point of the entire recognition process, providing the basic data input for downstream acoustic models.

[0004] 2. Acoustic model (sound-based character recognition) It is used to receive audio features and map sound signals to basic speech units (such as phonemes, pinyin, or single characters). It mainly calculates the acoustic matching probability between sounds and words, solving the problem of "what sound was heard" (e.g., distinguishing the nasal features of "qing" and "qing"). The audio input layer serves as a bridge, transforming the "physical sound" of the audio input layer into preliminary "text probabilities" and passing the results to the output layer and the language model.

[0005] 3. Language Model (Semantic Understanding) It is responsible for evaluating the rationality of word combinations. Utilizing grammatical rules and contextual relationships, it calculates the linguistic probability of phrases or sentences appearing, addressing the question of "what is the context?", thereby effectively correcting homophone or acoustic confusion errors (e.g., judging from context that "Tsinghua University" is more reasonable than "Frog University"). The language model and acoustic model work in parallel or collaboratively, providing error correction and prediction support from a "semantic" dimension, compensating for the shortcomings of relying solely on sound recognition.

[0006] 4. Output Layer (Comprehensive Decision Making) As the final decision-making center of the system, the output layer integrates the "pronunciation probability" provided by the acoustic model and the "context probability" provided by the language model, and uses a decoding search algorithm to obtain the most reasonable and fluent final recognized text. The output layer is the endpoint of the entire process, summarizing the information from all upstream modules and outputting the final result.

[0007] These four modules constitute a complete processing chain from "physical signal" to "semantic text". The audio input layer provides features, while the acoustic model and language model provide probabilistic support from the two dimensions of "sound" and "semantics" respectively, complementing each other; the output layer then integrates the results of both to make the final decision.

[0008] Current error source analysis in speech recognition systems mainly relies on predefined rules and static decision trees, which suffer from insufficient flexibility and poor adaptability. Traditional methods require manual pre-setting of error categories and analysis paths, making it difficult to cope with emerging error types and compound errors (such as simultaneous errors in the acoustic and language models). They also lack the ability to dynamically track error propagation paths and perform cross-case correlation analysis, resulting in inaccurate error localization and a lack of systematic optimization suggestions. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention aims to provide an adaptive source analysis method, storage medium, and device for speech recognition errors.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive source analysis method for speech recognition errors includes the following steps: S1. When a speech recognition error occurs, the audio spectrum features, text embedding features, and system log features generated during the speech recognition process are jointly encoded using a multimodal Transformer model to generate a comparable error feature vector. The system log is used to record audio and environmental indicators, system and hardware status, and model operating parameters during the operation of the speech recognition system; The text embedding features include identifying text embedding features and reference text embedding features. The identifying text embedding features are derived from the identified text output by the speech recognition system from the audio, and the reference text embedding features are derived from standard reference text. S2. Automatically discover error patterns in the erroneous feature vectors obtained in step S1 through comparative cluster analysis; S3. Use graph neural networks to construct a dynamic source map of the speech recognition system in this recognition process, reveal the propagation path of the error in the speech recognition system, and quantify the influence weight of each link through influence degree calculation to locate which module of the speech recognition system the error in this recognition mainly originates from, and identify it as the main error module. S4. Based on the main error modules located in step S3, generate system-level optimization suggestions, recommend the optimal repair strategy, and output an intervention priority list.

[0011] Furthermore, in step S1, the audio and environmental indicators include signal-to-noise ratio, sampling rate, ambient noise level, and network latency or packet loss rate; the system and hardware status includes device model, CPU / memory usage, decoding processing time, and system or APP version number; and the model running parameters include the confidence score output by the decoder, the parameters of the search algorithm, and abnormal error codes or interruption records.

[0012] Furthermore, the joint encoding process in step S1 is as follows: Audio spectral characteristics X audio ∈ R (T×F) Where T is the number of time frames and F is the frequency dimension; identify text embedding features X text1 ∈R Lrec×D Reference text embedding feature X text2 ∈R Lref×D Where Lrec and Lref are the sequence lengths of the identified text embedding feature and the reference text embedding feature, respectively, and D is the embedding dimension; system log feature X log ∈ R M , where M is the dimension of system parameters; The comparable erroneous feature vector is obtained by calculating the correlation between features through a cross-modal attention mechanism, as shown in the formula: ; Where Q, K, and V represent the query, key, and value matrices, respectively, obtained by projecting from the original features through a linear transformation layer. Q comes from audio spectral features, K comes from the recognized text embedding features and the reference text embedding features, V comes from system log features, and d is a scaling factor; softmax() is a normalized exponential function that constrains the attention weights to the [0,1] interval; QK T This represents the query-key similarity matrix, reflecting the strength of the association between features.

[0013] Furthermore, the specific process of step S2 is as follows: S2.1 Based on the historically accumulated error case library, error feature vectors of the same type of error are combined into positive sample pairs according to feature similarity, and error feature vectors of different types of error are combined into negative sample pairs. The constructed positive and negative sample pairs are used to optimize the contrastive learning model. S2.2 Input the new error feature vector obtained in step S1 into the optimized contrastive learning model. The optimized contrastive learning model maps the new error feature vector obtained in step S1 to an optimized feature space, so that error features of the same type move closer to each other. S2.3 Using a clustering algorithm, the clustered error feature vectors in the optimized feature space are automatically divided into the same error pattern, thereby determining the error pattern to which the current speech recognition error belongs.

[0014] Furthermore, in step S3, the process of constructing the dynamic source map is as follows: Define a graph neural network G=(V,E), where nodes V include the audio input layer, acoustic model, language model, and output layer of the speech recognition system; edges E are the influence weights calculated by gradient backpropagation, and the calculation formula is: I_{i→j} = ∑| y_j / x_i|; y_j is the output of node j, and x_i is the input from node i. I_{i→j} represents the influence index of node i on node j, and its value directly reflects the strength of error propagation from upstream node i to downstream node j; y_j / x_i is the partial derivative operator, which calculates the sensitivity of the output of node j to the input from node i; the main error module is determined based on the influence index I_{i→j}.

[0015] Further, in step S4, the causal intervention suggestion generation process includes: S4.1 Based on the main error modules identified in step S3, determine the corresponding interventionable variable X, and calculate the intervention effect using the following formula: P(Y|do(X=x)) = ∑_z P(Y|X=x,Z=z)P(Z=z); Wherein, do(X=x) represents the intervention operation on the modifiable variable X; Y represents the performance index of the speech recognition system, which is the target variable to be optimized; Z represents the set of confounding variables, including all variables that simultaneously affect X and Y; P(Y|do(X=x)) represents the causal effect after implementing the intervention X=x, that is, the expected impact of the intervention X=x on the target Y. When calculating, the distribution of all confounding variables Z needs to be considered, and an unbiased estimate is obtained by adjusting the confounding factors; ∑_z represents the summation operation over all possible values ​​of the confounding variable Z; P(Y|X=x,Z=z) represents the probability of the performance Y of the speech recognition system under the specific intervention X=x and confounding factor Z=z. S4.2. Based on the effects of each intervention calculated in step S4.1, output an intervention priority list. The higher the P(Y|do(X=x)), the higher the priority of the intervention.

[0016] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0017] The present invention also provides a computer device, characterized in that it includes a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program to implement the above-described method.

[0018] The beneficial effects of this invention are as follows: 1. This invention automatically discovers error patterns through comparative clustering, which can improve the accuracy of misclassification; 2. The dynamic source graph constructed by the present invention through graph neural network can reveal hidden error associations (such as specific pronunciation errors appearing only in specific grammatical structures), and demonstrate the transmission process of errors from acoustic features to the final output; 3. This invention improves repair efficiency by calculating and outputting an intervention priority list through causal intervention.

[0019] In summary, this invention can yield macro-level classification results (error patterns) and micro-level source tracing results (locating the main error modules) for speech recognition errors, and can also produce decision optimization results (intervention priority list). Detailed Implementation

[0020] The present invention will be further described below. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0021] This embodiment provides an adaptive source analysis method for speech recognition errors, including the following steps: S1. When a speech recognition error occurs, the audio spectrum features, text embedding features, and system log features generated during the speech recognition process are jointly encoded using a multimodal Transformer model to generate a comparable error feature vector. The system log records audio and environmental metrics, system and hardware status, and model operating parameters during the operation of the speech recognition system. This log is used to troubleshoot errors not caused by the model algorithm itself. Audio and environmental metrics include signal-to-noise ratio, sampling rate, ambient noise level, network latency, or packet loss rate. System and hardware status includes device model, CPU / memory usage, decoding processing time, and system or app version number. The model operating parameters include the confidence score output by the decoder, parameters of the search algorithm (such as beam width), error codes, or interrupt records.

[0022] The text embedding features include identifying text embedding features and reference text embedding features. The identifying text embedding features are derived from the identified text output by the speech recognition system after recognizing the audio, and the reference text embedding features are derived from standard reference text (i.e., correct manually annotated text / Ground Truth). By jointly encoding the identifying text embedding features and the reference text embedding features, features of error patterns such as misspellings, omissions, and extra characters can be extracted and compared.

[0023] S2. Automatically discover error patterns in the erroneous feature vectors obtained in step S1 through comparative cluster analysis; S3. Use graph neural networks to construct a dynamic source map of the speech recognition system in this recognition process, reveal the propagation path of the error in the speech recognition system, and quantify the influence weight of each link through influence degree calculation to locate which module of the speech recognition system the error in this recognition mainly originates from, and identify it as the main error module. S4. Based on the main error modules located in step S3, generate system-level optimization suggestions, recommend the optimal repair strategy, and output an intervention priority list.

[0024] In this embodiment, the joint encoding process in step S1 is as follows: Audio spectral characteristics X audio ∈ R (T×F) Where T is the number of time frames and F is the frequency dimension; identify text embedding features X text1 ∈R Lrec×D Reference text embedding feature X text2 ∈R Lref×D Where Lrec and Lref are the sequence lengths of the identified text embedding feature and the reference text embedding feature, respectively, and D is the embedding dimension; system log feature X log ∈ R M , where M is the system parameter dimension (e.g., M=256 (including 50-dimensional system indicators such as GPU utilization and memory usage + 206-dimensional decoding status)).

[0025] The identifying text embedding features and the reference text embedding features are high-dimensional vector representations obtained by mapping through the same text embedding model (such as a pre-trained language model), and therefore have the exact same embedding dimension D. Although the actual number of words in the identifying text and the reference text may differ, their sequence lengths can be aligned to a uniform set length L by padding or truncation operations before joint encoding in the input multimodal Transformer model.

[0026] Specifically, R represents the set of all real numbers, including positive numbers, negative numbers, and zero. In feature representation in machine learning / deep learning, R... 维度 This indicates that the feature takes values ​​in the real number vector space. For a 1-second 16kHz audio file, T = 160 (frames), F = 80 (number of Mel filter banks).

[0027] The comparable erroneous feature vector is obtained by calculating the correlation between features through a cross-modal attention mechanism, as shown in the formula: ; Where Q, K, and V represent the query, key, and value matrices, respectively, projected from the original features through a linear transformation layer. Q comes from the audio spectral features, K comes from the recognized text embedding features and the reference text embedding features, V comes from the system log features, and d is the scaling factor (usually the dimension of the key vector). softmax() is a normalized exponential function that constrains the attention weights to the [0,1] interval. T This represents the query-key similarity matrix, reflecting the strength of the association between features.

[0028] It's important to note that K (the key) primarily comes from the identified text embedding features and the reference text embedding features, serving as a "query index." Because text has a clear sequence structure and semantic attributes, it's well-suited as an index, allowing the audio spectral features (Q) to calculate similarity, achieving precise alignment between sound signals and text semantics. V (the value) mainly comes from system log features, providing the extracted "actual content." After calculating the attention weights through K (i.e., determining the correspondence between audio and text), these weights are used to extract information from V. At this point, system log features (such as decoding status, confidence scores, and other specific numerical indicators) are extracted as the actual state content and integrated into the final feature vector.

[0029] In this embodiment, multimodal feature joint encoding is achieved by introducing a cross-modal attention mechanism as the core algorithm for feature fusion, which solves the problem of correlation modeling of multi-source heterogeneous data. Specifically, the dimensional space of each feature is clearly defined by mathematical symbols to provide dimensional constraints for subsequent calculations.

[0030] In this embodiment, the specific process of step S2 is as follows: S2.1 Based on the historically accumulated error case library, error feature vectors of the same type of error are combined into positive sample pairs according to feature similarity, and error feature vectors of different types of error are combined into negative sample pairs. The constructed positive and negative sample pairs are used to optimize the contrastive learning model. S2.2 Input the new error feature vector obtained in step S1 into the optimized contrastive learning model. The optimized contrastive learning model maps the new error feature vector obtained in step S1 to an optimized feature space, so that error features of the same type move closer to each other. S2.3 Using a clustering algorithm, the clustered error feature vectors in the optimized feature space are automatically divided into the same error pattern, thereby determining the error pattern to which the current speech recognition error belongs.

[0031] The core of step S2 is to optimize the feature space through contrastive learning (bringing similar errors closer together and dissimilar errors further apart), and then use cluster analysis to automatically discover and classify error patterns, rather than using traditional fixed-category supervised recognition.

[0032] In this embodiment, the process of constructing the dynamic source map in step S3 is as follows: Define a graph neural network G=(V,E), where nodes V include the audio input layer, acoustic model, language model, and output layer of the speech recognition system; edges E are the influence weights calculated by gradient backpropagation, and the calculation formula is: I_{i→j} = ∑| y_j / x_i|; y_j is the output of node j, and x_i is the input from node i. I_{i→j} represents the influence index of node i on node j, and its value directly reflects the strength of error propagation from upstream node i to downstream node j.

[0033] In this embodiment, the upstream node i with the largest influence index I_{i→j} is taken as the main error module. For example, in dialect recognition errors, if the I value from the acoustic model node to the output layer node reaches 0.8, while the I value from the language model node to the output layer node is only 0.2, it indicates that the acoustic model should be optimized first. y_j / x_i is the partial derivative operator, which calculates the sensitivity of the output of node j to the input from node i.

[0034] For example, when the speech recognition system mistakenly identifies "Tsinghua University" as "Frog University", the dynamic source map will show that the I value from the audio input layer node to the acoustic model node is 0.6 (reflecting the acoustic confusion between "qing" and "qing"); the I value from the language model node to the output layer node is 0.3 (reflecting the weak contextual association of "university"); and the gradient feedback of the final output layer node indicates that the acoustic model needs to add nasal sound discrimination training data.

[0035] It should be noted that in the error source analysis of the embodiment, the gradient influence weights (i.e. error propagation strength) between each node of the speech recognition model are calculated to accurately locate whether the error is caused by fuzzy audio features, acoustic model confusion, or weak contextual correlation of the language model.

[0036] In this embodiment, step S4, the causal intervention suggestion generation process includes: S4.1 Based on the main error modules identified in step S3, determine the corresponding interventionable variable X, and calculate the intervention effect using the following formula: P(Y|do(X=x)) = ∑_z P(Y|X=x,Z=z)P(Z=z); Here, `do(X=x)` represents the intervention operation on the modifiable variable X (such as model parameters, training data, etc.), which is the core operator of causal inference. For example, `do(acoustic model learning rate=0.001)` represents the intervention action of fixing the acoustic model learning rate to 0.001. `Y` represents the performance index of the speech recognition system (such as recognition accuracy, word error rate, etc.), which is the target variable to be optimized. In the speech recognition scenario, `Y` can be the percentage value of the overall sentence recognition accuracy. `Z` represents the set of confounding variables, containing all variables that simultaneously affect both X and Y. For example, when analyzing the performance of a language model, `Z` may include confounding factors such as acoustic model performance and audio quality. `P(Y|do(X=x))` represents the causal effect after implementing the intervention X=x, that is, the expected impact of the intervention X=x on the target Y. The calculation needs to consider the distribution of all confounding variables Z, and an unbiased estimate is obtained by adjusting for confounding factors. ∑_z represents the summation operation over all possible values ​​of the confounding variable Z. For example, when Z contains discrete variables (such as dialect types), it is necessary to iterate through all dialect categories for calculation. P(Y|X=x,Z=z) represents the probability of the speech recognition system performance Y under the condition of a specific intervention X=x and confounding factor Z=z. For example, when the acoustic model learning rate is set to 0.001 (X=x) and the audio signal-to-noise ratio is >20dB (Z=z), the probability that the speech recognition system achieves a recognition accuracy of 85% is given.

[0037] S4.2 Based on the effects of each intervention calculated in step S4.1, output an intervention priority list, such as model parameter adjustment, training data supplementation, etc. The higher the P(Y|do(X=x)), the higher the priority of the intervention.

[0038] For those skilled in the art, various corresponding changes and modifications can be made based on the above technical solutions and concepts, and all such changes and modifications should be included within the protection scope of the claims of this invention.

Claims

1. An adaptive source analysis method for speech recognition errors, characterized in that, Includes the following steps: S1. When a speech recognition error occurs, the audio spectrum features, text embedding features, and system log features generated during the speech recognition process are jointly encoded using a multimodal Transformer model to generate a comparable error feature vector. The system log is used to record audio and environmental indicators, system and hardware status, and model operating parameters during the operation of the speech recognition system; The text embedding features include identifying text embedding features and reference text embedding features. The identifying text embedding features are derived from the identified text output by the speech recognition system from the audio, and the reference text embedding features are derived from standard reference text. S2. Automatically discover error patterns in the erroneous feature vectors obtained in step S1 through comparative cluster analysis; S3. Use graph neural networks to construct a dynamic source map of the speech recognition system in this recognition process, reveal the propagation path of the error in the speech recognition system, and quantify the influence weight of each link through influence degree calculation to locate which module of the speech recognition system the error in this recognition mainly originates from, and identify it as the main error module. S4. Based on the main error modules located in step S3, generate system-level optimization suggestions, recommend the optimal repair strategy, and output an intervention priority list.

2. The method according to claim 1, characterized in that, In step S1, the audio and environmental metrics include signal-to-noise ratio, sampling rate, ambient noise level, and network latency or packet loss rate. The system and hardware status includes device model, CPU / memory usage, decoding processing time, and system or APP version number. The model running parameters include the confidence score output by the decoder, the parameters of the search algorithm, and abnormal error codes or interruption records.

3. The method according to claim 1, characterized in that, The joint encoding process in step S1 is as follows: Audio spectral characteristics X audio ∈ R (T×F) Where T is the number of time frames and F is the frequency dimension; identify text embedding features X text1 ∈R Lrec×D Reference text embedding feature X text2 ∈R Lref×D Where Lrec and Lref are the sequence lengths of the identified text embedding feature and the reference text embedding feature, respectively, and D is the embedding dimension; system log feature X log ∈ R M , where M is the dimension of system parameters; The comparable erroneous feature vector is obtained by calculating the correlation between features through a cross-modal attention mechanism, as shown in the formula: ； Where Q, K, and V represent the query, key, and value matrices, respectively, obtained by projecting from the original features through a linear transformation layer. Q comes from audio spectral features, K comes from the recognized text embedding features and the reference text embedding features, V comes from system log features, and d is a scaling factor; softmax() is a normalized exponential function that constrains the attention weights to the [0,1] interval; QK T This represents the query-key similarity matrix, reflecting the strength of the association between features.

4. The method according to claim 1, characterized in that, The specific process of step S2 is as follows: S2.1 Based on the historically accumulated error case library, error feature vectors of the same type of error are combined into positive sample pairs according to feature similarity, and error feature vectors of different types of error are combined into negative sample pairs. The constructed positive and negative sample pairs are used to optimize the contrastive learning model. S2.2 Input the new error feature vector obtained in step S1 into the optimized contrastive learning model. The optimized contrastive learning model maps the new error feature vector obtained in step S1 to an optimized feature space, so that error features of the same type move closer to each other. S2.3 Using a clustering algorithm, the clustered error feature vectors in the optimized feature space are automatically divided into the same error pattern, thereby determining the error pattern to which the current speech recognition error belongs.

5. The method according to claim 1, characterized in that, In step S3, the process of constructing the dynamic source map is as follows: Define a graph neural network G=(V,E), where nodes V include the audio input layer, acoustic model, language model, and output layer of the speech recognition system; edges E are the influence weights calculated by gradient backpropagation, and the calculation formula is: I_{i→j} = ∑| y_j / x_i|; y_j is the output of node j, and x_i is the input from node i; I_{i→j} represents the influence index of node i on node j, and its value directly reflects the error propagation strength of upstream node i to downstream node j; y_j / x_i is the partial derivative operator, which calculates the sensitivity of the output of node j to the input from node i; the main error module is determined based on the influence index I_{i→j}.

6. The method according to claim 1, characterized in that, In step S4, the process of generating the causal intervention suggestion includes: S4.1 Based on the main error modules identified in step S3, determine the corresponding interventionable variable X, and calculate the intervention effect using the following formula: P(Y|do(X=x)) = ∑_z P(Y|X=x,Z=z)P(Z=z); Wherein, do(X=x) represents the intervention operation on the modifiable variable X; Y represents the performance index of the speech recognition system, which is the target variable to be optimized; Z represents the set of confounding variables, including all variables that simultaneously affect X and Y; P(Y|do(X=x)) represents the causal effect after implementing the intervention X=x, that is, the expected impact of the intervention X=x on the target Y. When calculating, the distribution of all confounding variables Z needs to be considered, and an unbiased estimate is obtained by adjusting the confounding factors; ∑_z represents the summation operation over all possible values ​​of the confounding variable Z; P(Y|X=x,Z=z) represents the probability of the performance Y of the speech recognition system under the specific intervention X=x and confounding factor Z=z. S4.

2. Based on the effects of each intervention calculated in step S4.1, output an intervention priority list. The higher the P(Y|do(X=x)), the higher the priority of the intervention.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

8. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method of any one of claims 1-6.

Images