Adaptive speech recognition method and system based on domain feature fusion

Abstract

The application relates to the technical field of speech recognition, and specifically discloses a self-adaptive speech recognition method and system based on field feature fusion, which comprises the following steps: in response to an interaction request, obtaining the hardware parameters of a speech interaction device, determining the current computing power mode of the speech interaction device based on the hardware parameters; based on the determined computing power mode, activating the corresponding speech recognition model structure; and performing end-side preprocessing on the input original speech signal to generate a preprocessed speech signal. Through hardware computing power sensing and dynamic model scheduling, the application realizes the optimal balance between recognition accuracy and computing efficiency under different resource constraints; through a field prediction and hierarchical feature fusion mechanism, the adaptability and recognition robustness to multi-field speech are enhanced; and further fusion of non-standard speech calibration, context memory management and end-cloud collaborative updating capability significantly improves the understanding accuracy of personalized pronunciation, continuous dialogue and dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to an adaptive speech recognition method and system based on domain feature fusion. Background Technology

[0002] Currently, the core challenge facing edge speech recognition systems lies in the fact that limited computing resources cannot support the deployment of high-precision, high-generalization speech recognition models. Existing technologies typically employ fixed, simplified models to adapt to device computing power, but this one-size-fits-all approach leads to a significant drop in recognition accuracy in complex acoustic environments, multi-domain speech content, and personalized pronunciation scenarios. In particular, it fails to achieve effective domain adaptation and dynamic feature fusion on low-computing-power devices.

[0003] Existing solutions generally lack the ability to perceive and dynamically schedule the real-time computing power status of devices when achieving cross-domain adaptation, resulting in an inability to flexibly balance computational overhead and recognition accuracy based on current resources. At the same time, in non-standard speech processing and contextual memory utilization, existing systems often ignore device differences and user-specific characteristics, leading to rigid recognition results, poor scenario adaptability, and insufficient continuous learning ability, forming a vicious cycle of resource constraints, model solidification, and recognition degradation. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, the objective of this invention is to propose an adaptive speech recognition method and system based on domain feature fusion, in order to improve speech recognition accuracy and reduce resource consumption.

[0005] To achieve the above objectives, a first aspect of the present invention proposes an adaptive speech recognition method based on domain feature fusion, comprising the following steps:

[0006] In response to an interaction request, the hardware parameters of the voice interaction device are obtained, and the current computing power mode of the voice interaction device is determined based on the hardware parameters.

[0007] Based on the determined computing power mode, the corresponding speech recognition model structure is activated;

[0008] The input raw speech signal is preprocessed on the device to generate a preprocessed speech signal, and the preprocessed speech signal is used for domain prediction to determine the main domain label to which the preprocessed speech signal belongs.

[0009] Using the aforementioned speech recognition model structure, hierarchical feature extraction is performed on the preprocessed speech signal based on the main domain label to obtain the current domain feature vector;

[0010] Calculate the domain similarity between the current domain feature vector and the preset known domain feature vector, and dynamically adjust the fusion weights based on the domain similarity to generate the final acoustic features;

[0011] The final acoustic features are used for decoding to output the recognition result.

[0012] To achieve the above objectives, a second aspect of the present invention proposes an adaptive speech recognition system based on domain feature fusion, comprising:

[0013] The hardware computing power detection module is configured to acquire device hardware parameters, determine the computing power mode based on a preset computing power classification table, and activate the corresponding model structure.

[0014] The speech preprocessing module is configured to perform noise reduction, endpoint detection, and determine the main domain label on the input speech;

[0015] The hierarchical feature extraction module is configured to perform hierarchical extraction of cross-domain acoustic common features and domain-specific features based on the computing power mode and the main domain label;

[0016] The lightweight feature fusion module is configured to calculate the domain similarity between the current feature and known features, and dynamically adjust the fusion weights based on the domain similarity to generate the final acoustic features;

[0017] The offline adaptation decoding module is configured to use the final acoustic features for decoding and output the recognition result.

[0018] To achieve the above objectives, a third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, it implements the above-described adaptive speech recognition method based on domain feature fusion.

[0019] The adaptive speech recognition method and system based on domain feature fusion in this invention achieves an optimal balance between recognition accuracy and computational efficiency under different resource constraints through hardware computing power awareness and dynamic model scheduling; it enhances adaptability and recognition robustness to multi-domain speech through domain prediction and hierarchical feature fusion mechanisms; and it further integrates non-standard speech calibration, context memory management, and edge-cloud collaborative update capabilities, significantly improving the accuracy of understanding personalized pronunciation, continuous dialogue, and dynamic scenes, ultimately achieving a highly available, highly adaptive, and continuously optimized speech recognition experience under limited edge resources. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the adaptive speech recognition method based on domain feature fusion provided by the present invention.

[0021] Figure 2 This is a schematic diagram showing the comparison curves of recognition accuracy and computation delay under different computing power modes in the adaptive speech recognition method based on domain feature fusion provided by the present invention.

[0022] Figure 3 This is a comparative diagram of the speech recognition model structure under different computing power modes in the adaptive speech recognition method based on domain feature fusion provided by the present invention;

[0023] Figure 4 This is a schematic diagram of dialect pronunciation feature space mapping in the adaptive speech recognition method based on domain feature fusion provided by the present invention;

[0024] Figure 5 This is a schematic diagram of the personalized model distillation training process in the adaptive speech recognition method based on domain feature fusion provided by the present invention;

[0025] Figure 6 This is a schematic diagram of weighted time decay similarity calculation in the adaptive speech recognition method based on domain feature fusion provided by the present invention;

[0026] Figure 7 This is a graph showing the relationship between KL divergence and dynamic fusion coefficients in the adaptive speech recognition method based on domain feature fusion provided by this invention.

[0027] Figure 8 This is a schematic diagram illustrating the implementation of the adaptive speech recognition system based on domain feature fusion provided by the present invention;

[0028] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0030] The following description, with reference to the accompanying drawings, outlines an adaptive speech recognition method, system, and electronic device based on domain feature fusion according to embodiments of the present invention.

[0031] Example 1:

[0032] Figure 1This is a flowchart illustrating an embodiment of the adaptive speech recognition method based on domain feature fusion according to the present invention. This embodiment details a technical solution that enables high-precision speech recognition in resource-constrained environments such as edge computing devices by dynamically sensing hardware status and adaptively adjusting the model structure and feature fusion strategy. Specifically, it includes the following steps:

[0033] Step S1: In response to the interaction request, obtain the hardware parameters of the voice interaction device, and determine the current computing power mode of the voice interaction device based on the hardware parameters.

[0034] For example, when a user triggers a voice interaction request through a wake word such as "Hello, Xiaozhi" or a physical button, the system first initializes a hardware status awareness process. This process reads the hardware parameters of the voice interaction device in real time by calling the application programming interface (API) of the underlying operating system, such as the SystemHAL interface in Android or the / proc file system in Linux.

[0035] The hardware parameters are comprehensive indicators reflecting the current computing power and physical state of the device, including but not limited to: the real-time clock speed of the central processing unit (CPU), the percentage of idle computing power of the neural processing unit (NPU), the remaining available space of random access memory (RAM), the remaining battery power (mAh), and the current core chip temperature (degrees Celsius).

[0036] To achieve accurate pattern matching, the system has a pre-built computing power grading table. This table defines the mapping relationship between different hardware parameter ranges and computing power modes. The system inputs the read hardware parameters into the computing power grading table for weighted scoring. For example, the scoring formula can be set as: computing power score equals CPU clock speed multiplied by weight coefficient one plus NPU idle rate multiplied by weight coefficient two minus temperature value multiplied by weight coefficient three; then, based on the calculated computing power score, the system determines the current computing power mode of the voice interaction device.

[0037] The computing power modes include at least a first computing power mode, a second computing power mode, and a third computing power mode. For ease of understanding, the first computing power mode can be defined as a low-power energy-saving mode (Level L), corresponding to scenarios where the device's battery level is below 20% or the temperature is above 45 degrees Celsius; the second computing power mode can be defined as a standard balanced mode (Level M), corresponding to scenarios where the device is in normal operating condition; and the third computing power mode can be defined as a high-performance ultra-fast mode (Level H), corresponding to scenarios where the device is connected to power and has good heat dissipation. The computing power of the first computing power mode is lower than that of the second computing power mode, and the computing power of the second computing power mode is lower than that of the third computing power mode. This hierarchical mechanism ensures that the system will not force the running of large models when the battery level is low, thus preventing shutdown, nor will it waste hardware computing power during high-performance periods.

[0038] like Figure 2 The dual-axis line graph comparison demonstrates the performance differences of speech recognition models under different computing power modes in terms of recognition accuracy and computation latency, vividly reflecting the system advantages of flexibly selecting recognition models based on equipment resources.

[0039] Figure 2 The blue curve represents speech recognition accuracy, which gradually increases from 85.2 to 94.8 as the computing power level increases. This indicates that with higher hardware resource support, the system can load more complex deep models such as Transformer or multi-module parallel frameworks, thereby achieving higher recognition accuracy. The red curve corresponds to the average computation latency, which rapidly decreases from 120 milliseconds in low computing power mode to 40 milliseconds in high computing power mode. This shows that high-performance devices can not only support more complex network structures, but also reduce response time through parallel processing and acceleration mechanisms.

[0040] Figure 2 The two indicators show a highly inverse relationship, highlighting the key balancing challenge this invention faces during model scheduling: the trade-off between accuracy and latency. The system needs to intelligently select the optimal mode based on the current CPU, NPU, and RAM status to ensure continuous user experience and device security. The difference between the curves also reflects the cost-effectiveness of switching models. For example, the medium-power computing mode can achieve a recognition rate of over 91% with computational latency controlled within 75 milliseconds, indicating that it has optimal balance performance in most scenarios.

[0041] Step S2: Based on the determined computing power mode, activate the corresponding speech recognition model structure.

[0042] To adapt to different computing power modes, the system pre-stores various speech recognition model structures with different architectures in local memory or generates them through dynamic graph compilation technology. After determining the computing power mode, the system performs model loading and activation operations.

[0043] If the system is set to the first computing power mode, it calls a model structure that includes a depthwise separable convolutional layer (DSC) as the lower-level network and a gated recurrent unit (GRU) as the higher-level network, configured for low-precision integer operations, such as INT8 quantized inference. In this mode, the depthwise separable convolutional layer significantly reduces the amount of multiplication and addition operations by decomposing standard convolution into depthwise convolution and pointwise convolution; while the GRU reduces the number of parameters by eliminating one gate structure compared to LSTM. Furthermore, the low-precision integer operation mode maps 32-bit floating-point numbers to 8-bit integers, compressing memory usage to one-quarter of the original, greatly reducing the pressure on the device in the first computing power mode.

[0044] If the second computing power mode is selected, the system calls a model structure that includes depthwise separable convolutional layers as the lower-level network and a bidirectional long short-term memory network (BiLSTM) as the higher-level network, and configures it for half-precision floating-point arithmetic mode, such as FP16. Bidirectional LSTM can utilize both past and future contextual information simultaneously, and has stronger sequence modeling capabilities compared to GRU, making it suitable for providing better recognition accuracy under standard computing power.

[0045] If the third computing power mode is determined, the system calls for more depth-separable convolutional layers and bidirectional long short-term memory networks than the second computing power mode, such as stacking more layers of BiLSTM or introducing Transformer encoder layers, and enables a hash retrieval acceleration mechanism. The hash retrieval acceleration mechanism refers to using Locality Sensitive Hashing (LSH) to quickly filter the subset of the vocabulary with the highest probability during the decoding stage, thereby improving the response speed when processing large-scale vocabularies and fully unleashing the potential of high-performance hardware.

[0046] like Figure 3 In the diagram, the horizontal axis represents the typical structural components of a speech recognition model, which include, in order, the input layer, depthwise separable convolutional layer, gated recurrent unit, bidirectional long short-term memory network, Transformer encoder layer, hash retrieval module, and output layer; the vertical axis represents three computing power modes, corresponding to low, medium, and high-performance operating environments of devices.

[0047] Figure 3 The blue squares represent the activated structure in this mode, while gray indicates it is not enabled. The diagram clearly shows that the first computing power mode activates a lightweight combination, namely depthwise separable convolutional layers paired with gated recurrent units. This results in a simple overall structure with low resource consumption, suitable for scenarios with limitations such as low power consumption and high temperature. The second computing power mode, by removing the GRU, enables a bidirectional long short-term memory network to improve sequence modeling capabilities. In the third computing power mode, the model structure is the most complex, activating the Transformer encoder and hash retrieval module, combining front-end convolutions and bidirectional LSTMs to achieve multi-layer heterogeneous modeling capabilities, and enabling a high-performance acceleration mechanism.

[0048] This bottom-up, layer-by-layer structural complexity fully demonstrates the advantages of dynamic model scheduling based on computing power. Figure 3 The blue area gradually deepens in color from top to bottom.

[0049] Step S3: Perform edge preprocessing on the input raw speech signal to generate a preprocessed speech signal, and determine the main domain label of the preprocessed speech signal.

[0050] Optionally, the on-device preprocessing of the input raw speech signal is environment-aware. The system first converts the acquired analog speech signal into a digital signal and performs frame-by-frame windowing. Then, it uses Fast Fourier Transform (FFT) to decompose the raw speech signal into multiple frequency sub-bands and calculates the signal-to-noise ratio (SNR) of each frequency sub-band.

[0051] The system identifies damaged subbands with a signal-to-noise ratio lower than a preset decibel threshold (e.g., set to 5dB) and simultaneously obtains the device type identifier of the voice interaction device through the system configuration file.

[0052] If the device type is identified as an in-vehicle device, considering that the main noise sources in the in-vehicle environment are low-frequency tire noise and engine roar, the system automatically applies a first noise reduction intensity threshold to the damaged subband with a frequency lower than the preset boundary frequency (such as 500Hz), for example, by using strong spectrum subtraction or Wiener filtering.

[0053] If the device type is identified as a wearable device, including a smartwatch or headphones, and considering that its main noise source is high-frequency wind noise or friction noise, the system applies a second noise reduction intensity threshold to damaged sub-bands with frequencies higher than a preset threshold frequency. This targeted preprocessing based on the physical characteristics of the device preserves more effective speech components than general full-band noise reduction.

[0054] After generating the preprocessed speech signal, the system extracts its acoustic features (such as Fbank or MFCC) and inputs them into a lightweight domain classifier for domain prediction, outputting the main domain label to which the current speech belongs. The main domain label is used to indicate the semantic category of the speech content, such as navigation, music, medical, or casual conversation.

[0055] Step S4: Using the speech recognition model structure, perform hierarchical feature extraction on the preprocessed speech signal based on the main domain label to obtain the current domain feature vector.

[0056] To balance versatility and specialization, this embodiment employs a hierarchical feature extraction strategy. The system first utilizes the convolutional neural network at the model's bottom layer to extract cross-domain acoustic common features as low-level features. These low-level features typically correspond to acoustic representations at the phoneme or syllable level, exhibiting universality across different domains.

[0057] At this point, to further conserve computing power, the system introduces a dynamic triggering mechanism. The system calculates the similarity difference between the currently extracted low-level features and the preset domain feature template (which corresponds to the main domain determined in step S3).

[0058] The similarity difference is compared with a preset domain feature fluctuation threshold. When the similarity difference is greater than the threshold, it means that the current speech features deviate significantly from the standard template, and low-level features alone may not be able to accurately identify the speech. Therefore, a high-level feature extraction step targeting domain-specific features is triggered, such as activating a high-level RNN or Attention module for deep semantic extraction. Conversely, if the difference is small, high-level extraction can be skipped, and low-level features can be used directly for decoding, thus achieving early stopping to save power.

[0059] The domain feature fluctuation threshold is set according to the computing power mode: it is set to a first value (e.g., 0.8) in the first computing power mode (low power), and a second value (e.g., 0.5) in the second and third computing power modes, with the first value being greater than the second value. This means that in low power mode, the system has a higher tolerance for feature differences and tends not to trigger high-level calculations, using an extreme strategy to save battery life; while when the power is sufficient, the system lowers the threshold and triggers high-level extraction more frequently to ensure accuracy.

[0060] Step S5: Calculate the domain similarity between the current domain feature vector and the preset known domain feature vector, and dynamically adjust the fusion weights based on the domain similarity to generate the final acoustic features.

[0061] For example, in order to integrate knowledge from different domains, the system needs to calculate the distance between the current feature and the features in the knowledge base. The steps for calculating the domain similarity between the current domain feature vector and the pre-set known domain feature vectors specifically include: using the vector operation unit to calculate the Euclidean distance between the current domain feature vector and the known domain feature vectors.

[0062] Domain similarity is calculated using Euclidean distance, which is defined as a value negatively correlated with Euclidean distance; that is, the smaller the Euclidean distance, the greater the domain similarity. Specifically, the distance can be mapped to similarity using a Gaussian kernel function.

[0063] After obtaining the similarity, the fusion weight is dynamically adjusted based on the domain similarity. The specific process is as follows: The system selects the N reference domain templates with the highest domain similarity to the current domain feature vector from the offline domain library on the edge side. N is a positive integer.

[0064] Subsequently, the attention weights corresponding to each reference domain template are calculated using a normalized exponential function (a variant of Softmax). The formula for calculating the attention weights is as follows:

[0065] ;

[0066] In the formula, This represents the attention weight of the i-th reference domain template; This represents the domain similarity of the i-th reference domain template; i represents the index variable of the reference domain template, with a value ranging from 1 to... ;j represents the loop index variable in the summation operation, taking values ​​from 1 to 1. ; This is a temperature coefficient used to adjust the smoothness of the Softmax function; The larger the size, the smoother the distribution. The smaller the size, the sharper the distribution; This represents the total number of templates selected from the reference domain. The normalized exponential function ensures that the sum of all weights is 1, and templates with higher similarity receive greater weights.

[0067] Finally, the calculated attention weights are used to sum the feature vectors from different domains to generate the final acoustic features. This step achieves the technical effect of allowing more relevant domain knowledge to dominate the current recognition process.

[0068] Step S6: Decode using the final acoustic features and output the recognition result.

[0069] Finally, the decoder receives the final acoustic features and uses a beam search algorithm in conjunction with the language model to generate the optimal text sequence.

[0070] In summary, this embodiment constructs a complete adaptive speech recognition system, effectively solving the pain point of difficulty in achieving both computing power and accuracy in existing technologies.

[0071] Example 2:

[0072] This embodiment focuses on the advanced processing mechanism for non-standard speech scenarios in this invention, particularly how it solves the recognition challenges faced by edge devices when encountering dialect accents, environmental noise interference, and fluctuations in the user's physiological state through environmental awareness, knowledge transfer, and a dual-path arbitration backoff strategy. Specifically, it includes the following:

[0073] The adaptive speech recognition method in this embodiment also includes a non-standard speech calibration step, which aims to address the problem of decreased recognition rate of the general model in specific long-tail scenarios.

[0074] In actual voice interaction, users' pronunciation habits are often influenced by both the external environment and their internal physiological state, causing the input speech to deviate from the acoustic distribution of standard Mandarin. To accurately capture this deviation, the system first uses an environmental perception module to generate environmental noise type labels in parallel during the generation of preprocessed speech signals. These environmental noise type labels are not simply signal-to-noise ratio values, but rather qualitative classification labels for the current acoustic background environment. These categories include, but are not limited to, vehicle wind noise, street noise, human voice noise, echo reverberation noise, and silent environments. The system then uses these environmental noise type labels to determine whether non-standard speech scenarios exist.

[0075] Optionally, the logic for determining whether a non-standard speech scenario exists is based on the Lombard effect in acoustic psychology.

[0076] When the ambient noise type indicator suggests a high-decibel environment of human voice noise or vehicle wind noise, the system infers that the user will subconsciously increase volume, change fundamental frequency, or lengthen vowels to make the device hear them clearly. This shouting-like pronunciation differs significantly from standard pronunciation, thus constituting a non-standard speech scenario. Furthermore, the system will combine prosodic features from acoustic characteristics, such as the dynamic range of pitch and the rate of change of speech rate, to help determine whether the user is using a dialect or has a heavy accent.

[0077] If there are non-standard speech scenarios, the system will select different calibration strategies based on the specific situation: call the preset non-standard speech sub-library to perform real-time calibration, or call the local personalized pronunciation model to perform personalized adaptation.

[0078] For example, for dialect groups with common patterns, such as dialect speakers in a specific region, the system adopts a strategy of calling a pre-set non-standard speech sub-library to perform real-time calibration.

[0079] The non-standard speech sub-library contains dialect pronunciation feature templates. These templates are not simple audio clips, but rather acoustic statistical models refined through clustering analysis, such as the parameter set of a Gaussian Mixture Model (GMM) or the activation patterns of specific layers in a deep neural network. The dialect pronunciation feature templates cover the initial and final vowel mapping relationships of mainstream dialects such as Sichuanese, Cantonese, and Northeastern Mandarin. For example, addressing the indistinguishability of "n" and "l" in Sichuanese, the non-standard speech sub-library stores a transformation matrix that maps the feature vector of the nasal consonant "n" to the feature space of the lateral consonant "l". When the system detects that a user's pronunciation matches a certain type of dialect feature, it uses this transformation matrix to perform linear or non-linear transformations on the input acoustic features, making them approximate the feature distribution of standard pronunciation, thereby improving the recognition rate without retraining the model.

[0080] For example, for users with highly personalized characteristics, such as specific device owners, the system employs a strategy of calling a local personalized pronunciation model to perform personalized adaptation.

[0081] The local personalized pronunciation model is built upon user-recorded domain-specific phrases, extracting unique pronunciation features. During device initialization or user registration, the system guides the user to read aloud several domain-specific phrases. From these recordings, the system extracts unique pronunciation features that characterize the user's unique pronunciation habits, such as fundamental frequency trajectory, formant position, and vocal tract length normalization factor.

[0082] like Figure 4It shows the distribution of Mandarin and Sichuan dialect in the acoustic feature space, reflecting the theoretical basis and practical effectiveness of dialect calibration strategies.

[0083] Figure 4 The blue dots represent standard Mandarin pronunciation samples, clustered around the coordinate point (2,2) in the feature space, exhibiting a stable and concentrated distribution. The red triangles represent uncalibrated Sichuan dialect pronunciations, whose distribution deviates from the blue area and whose center of gravity shifts to the upper left, indicating pronunciation deviations in Sichuan dialect within the acoustic vector space, such as confusion between "n" and "l" and tonal fluctuations. These deviations significantly reduce the accuracy of the original model. The green squares represent the corrected state of Sichuan dialect samples in the feature space after calling a local non-standard speech sub-library. It is clearly visible that they have clustered towards the standard Mandarin region, with some points even completely overlapping the blue cluster, indicating that the transformation matrix effectively aligns the feature differences between the original dialect pronunciations and the standard model.

[0084] The color changes shown in the figure represent three different pronunciation states, the shape changes are used to distinguish the source, and the spatial distribution changes of the point cloud vividly illustrate the key technical approach of feature alignment without retraining the model.

[0085] To achieve high-precision personalization with limited computing resources on edge devices, the local personalized pronunciation model is not trained from scratch, but is generated by transferring from a teacher model with a large number of parameters to a student model with a small number of parameters through knowledge distillation technology.

[0086] Optionally, the specific implementation process of knowledge distillation technology is as follows:

[0087] Teacher models are typically deployed on cloud servers or as high-precision local benchmark models, possessing hundreds of millions or even billions of parameters and exhibiting extremely strong generalization and feature extraction capabilities. Student models, on the other hand, are lightweight models deployed on edge devices, with only a fraction of the number of parameters compared to the teacher model. When generating a local personalized pronunciation model, the system simultaneously inputs user-recorded speech data into both the teacher and student models. The teacher model outputs soft targets containing rich probabilistic information, which are the Logits values ​​preceding the Softmax function in the output layer.

[0088] The system uses a distillation loss function to guide the training of the student model, ensuring that the output distribution of the student model approximates the output distribution of the teacher model as closely as possible. Distillation loss function The calculation formula is as follows:

[0089] ;

[0090] In the formula, This represents the total loss from knowledge distillation; is a balance coefficient used to adjust the weight ratio between distillation loss and cross-entropy loss, and its value ranges from [0,1]. The distillation temperature parameter is used for the softening probability distribution; This represents the Körbek-Leibler divergence (KL divergence). Indicates the distillation temperature The soft probability distribution output by the teacher model; Indicates the distillation temperature The soft probability distribution output by the student model; Represents the cross-entropy loss function; Represents the actual text label; This represents the original output probability distribution of the student model at a standard temperature (i.e., temperature 1). In this way, the student model (i.e., the local personalized pronunciation model) not only learns the text information corresponding to the speech, but also inherits the teacher model's ability to understand the user's special pronunciation, thus achieving high-precision adaptation of the small model to a specific user.

[0091] like Figure 5 This study demonstrates the trend of the difference in output probability distribution between the teacher model and the student model as the number of training rounds changes during personalized model distillation training, which is used to characterize the effectiveness and fusion speed of knowledge transfer.

[0092] Figure 5 The horizontal axis represents the number of training epochs, and the vertical axis represents the KL divergence, an indicator that measures the difference in output between the two models. A lower value indicates that the student model has more successfully learned the behavior of the teacher model. The three curves correspond to training strategies at different distillation temperatures. The red curve represents the distillation process at standard temperature, where the initial decrease is rapid but tends to be unstable, reflecting the risk of overfitting local probabilities in the student model under strong constraints on the original output. The blue curve corresponds to the medium temperature, where the overall decrease is stable and enters the smooth convergence region after fifteen epochs, indicating that the model at this temperature can both preserve the semantic distribution of the teacher model and adapt to the lightweight characteristics of edge devices. The green curve represents the high-temperature distillation strategy, where the decrease is slow but the most stable, indicating that under extremely smooth label guidance, the student model converges more slowly but has stronger generalization ability. Figure 5 The KL divergence value fluctuates between approximately 0.5 and 0.05, reflecting the complete evolution of the personalized model from initialization to a process that highly matches the teacher's model output.

[0093] However, in practical applications, a user's physiological state is not static. When a user has a cold, nasal congestion, sore throat, or when wearing a mask or face shield causes physical changes to the acoustic characteristics of their vocal tract, their pronunciation characteristics will deviate drastically from their normally recorded personalized pronunciation characteristics. In this case, forcibly using a local personalized pronunciation model trained based on normal conditions will result in negative optimization and lead to recognition errors. To resolve this technical conflict, this embodiment introduces a rigorous arbitration mechanism.

[0094] For example, the process of calling the local personalized pronunciation model to perform personalized adaptation also includes arbitration and rollback steps for personalized failures.

[0095] The core of this step lies in establishing a real-time monitoring mechanism to monitor whether the personalized model's performance is abnormal. The system first initiates a dual-path decoding mechanism, meaning that after receiving a voice input segment, the system runs two decoding processes in parallel in the background. The first process uses the local personalized pronunciation model to decode the voice and generate the first recognized text sequence; the second process simultaneously uses the voice recognition model structure (i.e., the pre-installed, unmodified general basic model in the device) to decode the same voice segment and generate the second recognized text sequence.

[0096] Optionally, in order to determine whether there is a substantial difference between the two results, the system calculates the Levenstein edit distance between the first identified text sequence and the second identified text sequence.

[0097] The Levenshtein distance refers to the minimum number of editing operations required to transform one string into another, including insertion, deletion, and replacement. If the Levenshtein distance is zero, it means that the recognition results of the two models are completely consistent. In this case, no arbitration is needed, and either result can be output directly. If the Levenshtein distance is greater than zero, it indicates that there is a discrepancy between the personalized model and the general model. In this case, the system cannot directly determine which result is correct, so objective evidence from a third party needs to be introduced, i.e., triggering the voiceprint consistency verification process.

[0098] For example, in the voiceprint consistency verification process, the system aims to quantify the deviation between the current speaker's voice state and their health state at the time of registration.

[0099] The system utilizes a specialized voiceprint extraction network, such as a neural network based on ResNet or TDNN architecture, to extract the voiceprint feature vector of the current speech signal. This vector is typically a fixed-dimensional embedding vector (e.g., d-vector or x-vector), capable of highly abstractly representing the speaker's identity and physiological acoustic characteristics. Simultaneously, the system retrieves a pre-stored baseline voiceprint feature vector from memory, bound to a local personalized pronunciation model. This baseline voiceprint feature vector is obtained by extracting features and calculating the centroid from multiple recordings during user personalized registration, representing the user's voiceprint baseline under standard physiological conditions.

[0100] Subsequently, the system uses the vector operation unit to calculate the cosine similarity between the current voiceprint feature vector and the pre-stored baseline voiceprint feature vector. Cosine similarity measures the directional consistency between two vectors in the vector space, and its calculation formula is as follows:

[0101] ;

[0102] In the formula, Cosine similarity is used to measure the directional consistency between the current voiceprint feature and the baseline voiceprint feature. Index variables representing the dimensions of a vector, with values ​​ranging from... arrive ; This represents the total number of dimensions of the voiceprint feature vector, such as 512 or 1024. This represents the speaker feature vector of the current speech signal at the th... Numerical components in each dimension; This indicates that the pre-stored baseline voiceprint feature vector is at the 1st... Numerical components in each dimension.

[0103] This formula calculates the dot product of two vectors and divides it by the product of their magnitudes, yielding a value between -1 and 1. The closer the value is to 1, the closer the current pronunciation state is to the baseline state; the lower the value, the more serious the deviation.

[0104] Optionally, the system compares the calculated cosine similarity with a preset physiological state drift threshold.

[0105] The physiological state drift threshold is an empirical value used to define the boundary between normal fluctuations and abnormal mutations. If the cosine similarity is lower than the preset physiological state drift threshold, such as 0.75, the system determines that the current speech is in a non-steady-state feature range. This means that the user's voice has undergone significant physical changes (such as severe nasalization). At this point, the acoustic prior knowledge relied upon by the personalized model has become invalid, and continued use will lead to overfitting errors.

[0106] Therefore, the system decisively performs a model rollback operation, that is, in this interaction, it masks the first recognized text sequence and directly outputs the second recognized text sequence as the final recognition result. This is because the general-purpose basic model is trained on a massive amount of diverse samples, and has better robustness and generalization ability to abnormal acoustic environments such as illness and wearing masks.

[0107] In addition, to prevent erroneous samples from contaminating the personalized model, the system also implements a data isolation strategy, which prohibits the use of the current voice data to update the local personalized pronunciation model.

[0108] If the system mistakenly identifies non-steady-state speech with severe nasality as the user's normal accent and uses it to update the model, it will cause irreversible shifts in the model parameters, affecting the user's recognition experience after recovery. Through the aforementioned policy of prohibiting updates, the system effectively protects the purity of the model, ensuring the stability and reliability of personalized services throughout their entire lifecycle.

[0109] In summary, this embodiment flexibly switches between preset sub-libraries and personalized models in non-standard voice scenarios and utilizes a dual-arbitration mechanism to build a security defense, ensuring that the system can still provide robust and reliable voice recognition services even in extreme situations such as abnormal user physiological states, demonstrating extremely high technical practicality.

[0110] Example 3:

[0111] In smart home or in-vehicle voice interaction scenarios, users often use abbreviated references, such as "turn it on," or engage in continuous multi-turn dialogues. However, limited by the storage space and offline computing power of edge computing devices, traditional full-history recording schemes are difficult to implement, while simple keyword matching is prone to logical errors. Therefore, this embodiment proposes a semantic compression storage based on time decay and a logic disambiguation scheme based on entity state graphs, achieving long-range context understanding and highly reliable instruction execution with low resource consumption. Specifically, it includes the following:

[0112] The adaptive speech recognition method in this embodiment also includes an edge-side context memory management step, which aims to resolve the contradiction between the limited memory of edge devices and the memory requirements of multi-turn dialogues.

[0113] During voice interaction, the dialogue history is a key clue for understanding the user's current intent. To efficiently utilize storage resources, the system first employs a Least Recently Used (LRU) caching strategy to store key information from the most recent Q rounds of dialogue. The most recent Q rounds typically refer to the interaction data from the 3 to 5 rounds prior to the current moment, and the specific value of Q can be dynamically adjusted based on the device's RAM size. Key information refers to the pre-parsed text or intent tags, rather than the raw audio data. The LRU strategy maintains an ordered queue; whenever a new dialogue is generated, the system inserts it at the head of the queue; when the queue length exceeds Q, the data at the tail of the queue, i.e., the least recently accessed dialogue, is removed from the cache.

[0114] For example, for conversations that have exceeded the cache rounds, the system does not discard them directly, but instead initiates a semantic compression mechanism to transfer them to long-term memory.

[0115] The system utilizes a lightweight entity extraction model on the edge to extract triples containing entity, intent, and slot value. The lightweight entity extraction model on the edge is a sequence labeling model that has undergone pruning and quantization, such as a model based on the BiLSTM-CRF (Bidirectional Long Short-Term Memory Network combined with Conditional Random Field) architecture. Its size is usually controlled within a few megabytes and can reside in the edge's memory.

[0116] When old data is removed from the LRU queue, it is fed into the lightweight entity extraction model on the edge. The model performs semantic parsing on the text and outputs standardized triple information. For example, for the historical instruction "set the living room air conditioner temperature to 26 degrees", the model extracts the following triple information: entity is "living room air conditioner", intent is "adjust temperature", and slot value is "26 degrees". This structured triple information greatly reduces storage space compared to raw natural language text, while preserving the core semantics.

[0117] Optionally, in order to support efficient fuzzy retrieval, the system maps triple information to a low-dimensional index vector.

[0118] The system employs a pre-trained word embedding layer or a lightweight semantic hashing algorithm to convert text-based triples into fixed-dimensional numerical vectors. The low-dimensional index vectors are typically set to 64 or 128 dimensions to balance retrieval accuracy and computational cost.

[0119] For example, considering the time-sensitivity of memory, this embodiment introduces a time decay mechanism.

[0120] Human memory exhibits a forgetting curve, meaning that information from older times generally has a weaker impact on current conversations. To simulate this characteristic, the system calculates a time decay factor based on the timestamps of the corresponding conversations. The time decay factor is a value between 0 and 1 used to quantify the freshness of historical information.

[0121] The system stores the low-dimensional index vector and time decay factor in a local vector database. The local vector database is a lightweight indexed file system deployed on flash memory, supporting Approximate Nearest Neighbor (ANN) search based on vector distance. During storage, each index record contains not only a semantic vector but also an absolute timestamp of when the record was generated.

[0122] Optionally, during offline recognition, the system performs memory-based implicit completion.

[0123] When a user inputs a new voice command, such as "turn it on" or "turn it up," the system first extracts the feature vector of the current speech. Then, the system iterates through or indexes the local vector database, calculating the weighted similarity between the current speech features and the indexed vectors in the local vector database.

[0124] Here, the weighted similarity is not simply the distance in the vector space, but a comprehensive score that incorporates the time dimension. Its calculation formula is designed as follows:

[0125] ;

[0126] In the formula, This indicates a weighted similarity score that incorporates the time dimension, used to determine whether a particular historical memory is triggered. The feature vector representing the current input speech; This represents a historical record index vector stored in a local vector database; "Indicates the vector dot product operation;" " represents the Euclidean norm of a vector, i.e., its magnitude; is the base of the natural logarithm; This is a preset memory forgetting rate constant; The absolute timestamp corresponding to the current system time; This is the absolute timestamp recorded when the historical record was generated.

[0127] The first part of the formula calculates cosine similarity, representing semantic relevance; the second part is an exponential decay function, representing temporal relevance. Using this calculation method, the system tends to match historical records that are semantically related and occurred relatively recently.

[0128] like Figure 6 This study demonstrates how the similarity between the current voice input and the historical memory vector gradually decreases over time during the on-device speech recognition process, and how this has a quantitative impact on the final memory recall result.

[0129] Figure 6The blue dashed line represents semantic similarity, which is set to a constant value, indicating that the historical record is highly relevant to the current input in the semantic space. However, without a time adjustment mechanism, even semantics from long ago may be mistakenly recalled. The red dashed line represents the time decay factor, which gradually decreases from 0 to 60 minutes with the historical time interval, reflecting the design idea of ​​simulating the human forgetting curve. The green solid line is the product of semantic and time factors, representing the final similarity score after combining semantic matching degree and time freshness.

[0130] from Figure 6 As can be seen, although the original semantics are consistent, due to the effect of time decay, the final similarity drops from the initial 0.9 to less than 0.1. Only historical semantics close to the current moment can trigger memory recall, thus avoiding the risk of misunderstanding caused by using early outdated content.

[0131] If the weighted similarity is higher than the memory recall threshold (e.g., 0.85), the system determines that the currently input fuzzy command is strongly associated with a certain historical record. At this point, the system performs a recovery operation, retrieving the corresponding historical entity information from the database for decoding and completion. For example, "open it" is completed as "turn on the living room air conditioner".

[0132] However, relying solely on historical memory for completion can lead to logical fallacies, especially when the physical environment has changed. To avoid hallucinatory control errors, this embodiment introduces a more rigorous verification process.

[0133] For example, after restoring the corresponding historical entity information for decoding completion, the process also includes entity state logic verification and active disambiguation steps.

[0134] This step establishes a security firewall before the command is executed. The system first parses the controlled entity and operation intent contained in the completed identification result. Continuing with the above example, the completed result is "Turn on the living room air conditioner," where the controlled entity is "living room air conditioner" and the operation intent is "turn on the power."

[0135] Optionally, the system needs to obtain the real state of the physical world.

[0136] The system accesses a real-time entity state graph stored locally on the voice interaction device. This real-time entity state graph is a dynamically updated data structure, such as a JSON object or graph database node. It maintains a heartbeat connection with IoT devices in the home via a local area network (LAN), such as Wi-Fi, Zigbee, or Bluetooth Mesh, synchronizing the status of each device in real time. The system retrieves the current physical state parameters of the controlled entity objects from this graph. To meet basic logical verification requirements, the current physical state parameters must include at least an online / offline status and an on / off status.

[0137] For example, the system performs logical conflict detection based on the acquired status parameters.

[0138] Logical conflicts are categorized into multiple levels. This embodiment focuses on high-level logical conflicts that may lead to operation failure or invalidity. The specific determination logic is as follows:

[0139] First, check the device's connectivity. If the controlled entity is offline, it means the device is powered off or experiencing a network failure. In this case, any commands sent to the device will be ignored. Therefore, the system determines that a high-level logical conflict exists.

[0140] Secondly, the necessity of the operation intention is checked. If the target state indicated by the operation intention is consistent with the switch state, for example, the user's intention is to "turn on the air conditioner," but the physical status shows that the air conditioner is already "on," sending the command is redundant and may confuse the user, making them think the device is not responding. Therefore, the system also determines that there is a high-level logical conflict.

[0141] In response to the existence of high-level logical conflicts, the system adopts a strategy of interception and clarification.

[0142] The system intercepts the direct output of the recognition results, meaning it does not send control commands to the underlying hardware interface to prevent invalid signaling from consuming network resources or triggering erroneous responses from the device. Simultaneously, to provide feedback to the user, the system triggers the text-to-speech (TTS) module to generate a clarification request as a system response.

[0143] Optionally, the generation of the clarification request is a natural language response dynamically constructed based on the conflict type.

[0144] If the conflict is caused by a device being offline, the system will generate the following response: "Sorry, I found that the living room air conditioner does not seem to be connected to power. Please check it."

[0145] If the conflict is caused by the same state, the system will generate the following response: "The living room air conditioner is already on. Would you like to adjust the temperature or switch modes?"

[0146] Through this entity state logic verification and proactive disambiguation process, the system is no longer a machine blindly executing instructions, but an intelligent steward with environmental awareness and logical judgment capabilities. It effectively solves the problem of operational disconnect from reality caused by blindly supplementing historical memory, greatly improving the user's interactive experience in complex physical environments and the system's overall intelligence level.

[0147] In summary, this embodiment achieves accurate memorization and recall of long-range contexts through a vector retrieval mechanism that incorporates time decay factors; simultaneously, it constructs a closed-loop logical verification system before instruction execution by combining real-time entity state graphs. This entire solution, with limited resources at the edge, ensures both the smoothness of multi-turn dialogues and the rigor of physical control, fully demonstrating its practical value.

[0148] Example 4:

[0149] In the era of the Internet of Things (IoT), voice interaction devices often face a hybrid scenario where offline operation and online updates coexist. How to leverage the powerful computing capabilities of the cloud to improve edge models without compromising the personalized experience accumulated over time at the edge is a major technical challenge. This embodiment constructs a secure and efficient model self-evolution closed loop by introducing a dynamic time warping algorithm, bottleneck adapter layer fine-tuning technology, and a dynamic fusion strategy based on KL divergence. Specifically, it includes the following:

[0150] The adaptive speech recognition method in this embodiment includes an edge-cloud collaborative error correction step, which aims to use the high-precision capabilities of the cloud to correct the recognition defects of the edge model.

[0151] Most of the time, voice interaction devices, such as smart speakers and in-vehicle infotainment systems, may be in a weak network or offline state, relying on local models for inference. However, with the emergence of trending internet terms or the release of new function commands, local models may encounter situations where they do not understand new words.

[0152] When a voice interaction device switches to online mode, such as when an in-vehicle device connects to home Wi-Fi or a smartwatch detects a stable 5G signal, the system triggers a model update service in the background. At this time, the device does not directly download the entire large new model file; instead, it performs lightweight sample interaction. The system obtains model correction samples from the cloud.

[0153] Optionally, the generation mechanism for model correction samples is as follows:

[0154] The cloud server collects recently uploaded speech segments with low recognition confidence from the device (or a group of similar devices). The cloud then uses a massive speech model (Foundation Model) with hundreds of billions of parameters to transcribe these segments with high precision, generating accurate text labels—the cloud-recognized text. Subsequently, the cloud packages these text labels with the original audio features uploaded by the device, such as EL spectrograms or MFCC features, to form paired training data—model correction samples. This method avoids transmitting the original audio waveform, protecting user privacy and saving bandwidth.

[0155] After acquiring the samples, the first challenge faced by the edge is time alignment:

[0156] Due to network transmission delays or packet loss, the text sequence delivered from the cloud and the audio feature sequence stored locally may be misaligned on the timeline. To ensure training accuracy, the system uses the Dynamic Time Warping (DTW) algorithm to align the text recognized in the cloud with the original audio features.

[0157] The described Dynamic Time Warping (DTW) algorithm constructs a distance matrix to find the shortest path from the start to the end of the sequence, achieving optimal temporal matching between the phoneme sequence in the text and the frame sequence in the audio. Through DTW alignment, the system can accurately determine which frame in the audio corresponds to which word in the text, thus providing high-quality frame-level labels for subsequent supervised learning.

[0158] The second challenge faced by the edge is the computing power bottleneck:

[0159] Because edge devices have extremely limited computing resources, they are usually unable to support backpropagation training of the entire speech recognition model. To address this issue, this embodiment employs a parameter efficient fine-tuning (PEFT) strategy.

[0160] The system freezes the backbone network parameters of the speech recognition model. Freezing, in this context, means setting the weight parameters of the backbone network, such as deep convolutional neural networks or Transformer encoders, to non-trainable during training; that is, gradients are not calculated and values ​​are not updated. These parameters account for over 95% of the total model capacity.

[0161] Meanwhile, the system only unlocks the parameters of the preset bottleneck adapter layer in the speech recognition model structure. The bottleneck adapter layer is a special neural network structure, usually designed as a sandwich structure of dimensionality reduction-nonlinear activation-dimensionality increase, inserted between the key layers of the backbone network. Because its intermediate layers have extremely low dimensions, such as only 64 or 32, its number of parameters is very small, usually only about 1% of the original model.

[0162] Based on the model-corrected samples, the system performs forward propagation on the input model, calculating the difference between the output and the true label, i.e., calculating the loss function, typically using Connection-Time Classification (CTC Loss) or Cross-Entropy Loss. Subsequently, the gradient is calculated using the backpropagation algorithm, and gradient update training is performed only on the parameters of the bottleneck adapter layer. This training method makes it possible to iterate the model on automotive chips or mobile NPUs.

[0163] However, simply importing cloud data for updates can easily lead to catastrophic forgetting, where the model learns new words but forgets long-standing user habits. Therefore, this embodiment incorporates a robust protection mechanism.

[0164] For example, the above process of performing gradient update training only on the parameters of the bottleneck adapter layer also includes a local feature protection step.

[0165] The core idea behind this step is to examine whether new knowledge will disrupt existing core experiences before absorbing new knowledge from the cloud.

[0166] Specifically, before performing gradient update training, the system first builds a temporary shadow model in memory containing the parameters to be updated. The temporary shadow model is a lightweight copy of the current model, or simply a temporary copy of the bottleneck adapter layer parameters to be updated. The system will then tentatively apply the update gradients delivered from the cloud to this shadow model.

[0167] Simultaneously, the system accesses the local anchor point sample set stored in the voice interaction device. This local anchor point sample set consists of high-value samples automatically filtered and retained by the device during long-term operation. It typically contains the user's most frequently used high-frequency commands (such as opening the car window or navigating home) as well as typical ambient background sounds. These samples represent the core usage habits and acoustic environment characteristics of the local user.

[0168] Next, the system performs a comparison between the old and new models: the system inputs the local anchor sample set into the current version of the speech recognition model (i.e., the old model that has not been updated) and the temporary shadow model (i.e., the new model after trial updates). For each input sample, both models output a probability distribution vector for the vocabulary.

[0169] The system acquires the output posterior probability distribution for the same sample and calculates the average KL divergence between the two posterior probability distributions. KL divergence is an asymmetric measure of the difference between two probability distributions. Its calculation formula is as follows:

[0170] ;

[0171] In the formula, This represents the average KL divergence of the local anchor sample set before and after the model update, used to quantify the overall shift in the model output distribution. This represents the total number of samples in the local anchor point sample set; The index variable represents the sample, and its value ranges from... arrive ; Represents the first local anchor point sample set. One input voice sample; represents the total number of output categories of the speech recognition model (i.e., vocabulary size); f represents the index variable of the output categories, with values ​​ranging from... arrive ; : Indicates the current version (old model) for the input sample The posterior probability value for predicting class f; : Indicates a temporary shadow model (new model) for the input sample The posterior probability value for the predicted class f.

[0172] The system calculates the KL divergence for all samples in the local anchor sample set and averages them to obtain the average KL divergence. The physical meaning of this average KL divergence is quite clear: it quantifies how much the model's understanding of the old data has changed before and after the update. A large KL divergence indicates a drastic change in the new model's predictions for the old data. For example, instructions that were previously easily recognized may now be met with hesitation, implying a significant risk of forgetting. Conversely, a small KL divergence indicates good compatibility with the new model.

[0173] Based on this quantitative indicator, the system determines the intensity of the update. The system calculates the dynamic fusion coefficient based on the average KL divergence. The dynamic fusion coefficient is negatively correlated with the average KL divergence. That is, the larger the KL divergence (the greater the risk), the smaller the fusion coefficient (the more conservative the update).

[0174] Optionally, dynamic fusion coefficient The calculation formula can be expressed as:

[0175] ;

[0176] In the formula, This represents the calculated dynamic fusion coefficient, used to control the magnitude of model parameter updates; This indicates the preset maximum allowed update step size, which is the ideal update rate when there are no local feature conflicts; This represents the sensitivity adjustment factor, used to adjust the degree of response of the fusion coefficient to changes in KL divergence; This represents the average KL divergence calculated above.

[0177] Finally, the system performs a controlled update. The parameters of the bottleneck adapter layer are updated using a weighted average with dynamic fusion coefficients. The specific update formula is as follows:

[0178] ;

[0179] In the formula, This indicates the final updated parameter values ​​that are written to the model; This represents the original parameter values ​​in the current local model (old model); This represents the dynamic fusion coefficient obtained from the aforementioned calculation; This represents the parameter values ​​in the temporary shadow model after the trial gradient update.

[0180] By introducing dynamic fusion coefficients The system implements a flexible update strategy: when there is little conflict between cloud updates and local characteristics (low KL divergence). near The system fully absorbs new knowledge; when conflicts are intense (high KL divergence). The system drastically reduces the size of the local system and refuses to make significant modifications to the parameters, thus prioritizing the preservation of the local experience.

[0181] This process is described as suppressing the forgetting of local acoustic characteristics while introducing cloud optimizations. It acts like a smart filter, allowing only cloud knowledge that is harmless or beneficial to the local experience to flow into the edge model, thereby achieving a steady increase in model performance and avoiding performance fluctuations caused by patching up problems elsewhere.

[0182] like Figure 7 The response curves of the fusion strategy upon which the local feature protection mechanism is based as a function of KL divergence during the model distillation process are shown.

[0183] Figure 7 The horizontal axis represents the average KL divergence, used to quantify the difference in the model's output distribution before and after the update. A higher value indicates a greater deviation between the new model parameters and the original model's output under the same input, usually signifying a conflict between new knowledge and locally learned content. The vertical axis represents the dynamic fusion coefficient, indicating the proportion of the original model retained during parameter updates, with its value decreasing from close to 1 to close to 0.

[0184] exist Figure 7 As can be seen, when the KL divergence is close to 0, that is, the two models are highly consistent, the system gives the maximum value of 1 for the fusion coefficient, indicating that the new knowledge is fully accepted; while when the KL divergence rises above 1, the fusion coefficient drops rapidly to below 0.1, indicating that the system almost completely shields the new knowledge and only retains the local parameters to protect the important local behaviors that have been learned.

[0185] Figure 7The green curve in the image reflects the fusion mechanism that adjusts the update magnitude based on the degree of difference, demonstrating a deep protection of original personalized features during adaptive learning. This mechanism not only avoids model shift and mislearning when cloud samples are introduced, but also achieves customized adaptation capabilities for different users and environments through an adjustable curve slope (controlled by a sensitivity adjustment factor).

[0186] In summary, this embodiment successfully solves the problems of difficult and risky edge model updates by using a fine-tuned architecture of edge-cloud collaboration and a local protection mechanism based on KL divergence, providing a safe and feasible technical path for the intelligent upgrade of massive IoT devices.

[0187] Example 5:

[0188] like Figure 8 As shown, corresponding to the above method embodiments, the present invention also proposes an adaptive speech recognition system based on domain feature fusion, comprising:

[0189] The hardware computing power detection module is configured to acquire device hardware parameters, determine the computing power mode based on a preset computing power classification table, and activate the corresponding model structure.

[0190] The speech preprocessing module is configured to perform noise reduction, endpoint detection, and determine the main domain label on the input speech;

[0191] The hierarchical feature extraction module is configured to perform hierarchical extraction of cross-domain acoustic common features and domain-specific features based on computing power mode and main domain labels;

[0192] The lightweight feature fusion module is configured to calculate the domain similarity between the current feature and known features, and dynamically adjust the fusion weights based on the domain similarity to generate the final acoustic features;

[0193] The offline adaptation decoding module is configured to use the final acoustic features for decoding and output the recognition result.

[0194] The system operates in a coordinated manner, forming the following workflow:

[0195] First, when a user initiates a voice interaction request, the hardware computing power detection module is activated. By reading hardware parameters such as the device's CPU clock speed, NPU idle rate, memory and power consumption, it determines the current computing power mode of the device based on a preset computing power grading table, such as low power, standard or high performance mode, and activates the corresponding lightweight or high performance voice recognition model structure accordingly.

[0196] Next, the speech preprocessing module receives the raw speech signal, performs targeted noise reduction and endpoint detection based on the device type (such as in-vehicle or wearable), generates a high-quality preprocessed speech signal, and uses a lightweight classifier to predict the main domain label to which the speech belongs, such as navigation, music, etc.

[0197] Subsequently, the hierarchical feature extraction module drives the activated model structure to perform hierarchical feature extraction on the preprocessed signal based on the determined computing power mode and main domain label: first, it extracts cross-domain acoustic common features as low-level features, and then determines whether to trigger the extraction of higher-level domain individual features with higher computational cost based on the dynamic threshold set by the computing power mode, and finally obtains the domain feature vector of the current speech.

[0198] Then, the lightweight feature fusion module calculates the similarity between the current domain feature vector and the pre-set known domain feature vector in the local knowledge base. Based on the similarity, it dynamically calculates the attention weight and performs weighted fusion of features from different domains to generate more robust final acoustic features.

[0199] Finally, the offline adaptive decoding module utilizes this final acoustic feature, combined with the language model, to decode and output the final recognized text result. The entire process is completed adaptively on the edge, achieving a dynamic balance between computational overhead and recognition accuracy based on real-time computing power, and improving robustness to diverse speech recognition through domain feature fusion.

[0200] Example 6:

[0201] Corresponding to the above embodiments, the present invention also proposes an electronic device.

[0202] like Figure 9 The diagram shows a structural schematic of an electronic device according to the present invention. The electronic device 100 includes a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, for example, via a bus 102. Optionally, the electronic device 100 may further include a transceiver 104. It should be noted that in practical applications, the transceiver 104 is not limited to one unit, and the structure of this electronic device 100 does not constitute a limitation on the embodiments of the present invention.

[0203] Processor 101 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0204] Bus 102 may include a pathway for transmitting information between the aforementioned components. Bus 102 may be a PCI bus or an EISA bus, etc. Bus 102 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0205] The memory 103 stores a computer program corresponding to the adaptive speech recognition method based on domain feature fusion according to the above embodiments of the present invention. This computer program is executed under the control of the processor 101. The processor 101 executes the computer program stored in the memory 103 to implement the content shown in the aforementioned method embodiments.

[0206] Among them, electronic devices 100 include, but are not limited to: mobile terminals such as laptops and PADs (tablet computers) and fixed terminals such as desktop computers. Figure 9 The electronic device 100 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0207] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An adaptive speech recognition method based on domain feature fusion, characterized in that, Includes the following steps: In response to an interaction request, the hardware parameters of the voice interaction device are obtained, and the current computing power mode of the voice interaction device is determined based on the hardware parameters; the computing power mode includes at least a first computing power mode, a second computing power mode and a third computing power mode, wherein the computing power of the first computing power mode is lower than that of the second computing power mode, and the computing power of the second computing power mode is lower than that of the third computing power mode. Based on the determined computing power mode, the corresponding speech recognition model structure is activated; specifically, if the first computing power mode is determined, a model structure containing depthwise separable convolutional layers as low-level networks and gated recurrent units as high-level networks is invoked and configured as a low-precision integer operation mode. If the second computing power mode is determined, the model structure containing depthwise separable convolutional layers as the low-level network and bidirectional long short-term memory network as the high-level network is called and configured as half-precision floating-point operation mode. If the third computing power mode is determined, the depth of the separable convolutional layer and the bidirectional long short-term memory network with more layers than the second computing power mode is called, and the hash retrieval acceleration mechanism is enabled. The input raw speech signal is preprocessed on the device to generate a preprocessed speech signal, and the preprocessed speech signal is used for domain prediction to determine the main domain label to which the preprocessed speech signal belongs. Using the aforementioned speech recognition model structure, hierarchical feature extraction is performed on the preprocessed speech signal based on the main domain label to obtain the current domain feature vector; specifically, this includes: extracting cross-domain acoustic common features as low-level features, and calculating the similarity difference between the currently extracted low-level features and a preset domain feature template; comparing the similarity difference with a preset domain feature fluctuation threshold, and triggering a high-level feature extraction step for domain-specific features when the similarity difference is greater than the domain feature fluctuation threshold; The domain feature fluctuation threshold is set according to the computing power mode: it is set to a first value in the first computing power mode, and to a second value in the second and third computing power modes, and the first value is greater than the second value. The process involves calculating the domain similarity between the current domain feature vector and a preset known domain feature vector, and dynamically adjusting the fusion weights based on the domain similarity to generate the final acoustic features. Specifically, this dynamic adjustment of the fusion weights based on the domain similarity includes: selecting N reference domain templates with the highest domain similarity to the current domain feature vector (N is a positive integer); calculating the attention weight corresponding to each reference domain template using a normalized exponential function, where the attention weight is equal to the exponential function value of the domain similarity of the current reference domain template divided by the sum of the exponential function values ​​of the domain similarity of all N reference domain templates; and using the calculated attention weights to perform a weighted summation of feature vectors from different domains to generate the final acoustic features. The final acoustic features are used for decoding to output the recognition result.

2. The method according to claim 1, characterized in that, The on-device preprocessing of the input raw speech signal includes: The original speech signal is decomposed into multiple frequency sub-bands, and the signal-to-noise ratio of each frequency sub-band is calculated. Identify the damaged sub-bands with a signal-to-noise ratio lower than a preset decibel threshold, and obtain the device type identifier of the voice interaction device; If the device type identifier indicates that it is a vehicle-mounted device, apply a first noise reduction intensity threshold to the damaged sub-band with a frequency lower than the preset boundary frequency. If the device type identifier indicates a wearable device, a second noise reduction intensity threshold is applied to the damaged subband with a frequency higher than the preset threshold frequency.

3. The method according to claim 1, characterized in that, The calculation of the domain similarity between the current domain feature vector and the preset known domain feature vector includes: Calculate the Euclidean distance between the current domain feature vector and the known domain feature vector; The domain similarity is calculated based on the Euclidean distance. The domain similarity is defined as a value that is negatively correlated with the Euclidean distance, that is, the smaller the Euclidean distance, the greater the domain similarity.

4. The method according to claim 1, characterized in that, The method also includes a non-standard speech calibration step: During the process of generating preprocessed speech signals, environmental noise type markers are also generated. The presence of non-standard speech scenarios is determined by combining the environmental noise type markers. If present, a preset non-standard speech sub-library is invoked to perform real-time calibration, wherein the non-standard speech sub-library contains dialect pronunciation feature templates; Alternatively, a local personalized pronunciation model can be invoked to perform personalized adaptation. This local personalized pronunciation model is generated by extracting exclusive pronunciation features from user-recorded domain-specific terms and transferring them from a large-parameter teacher model to a small-parameter student model through knowledge distillation technology.

5. The method according to claim 1, characterized in that, The method also includes an edge-side context memory management step: Use a least recently used caching strategy to store key information from the most recent Q-round conversations; For dialogues that exceed the cache rounds, a lightweight entity extraction model on the client side is used to extract triple information containing entity, intent, and slot value; The triplet information is mapped to a low-dimensional index vector, and a time decay factor is calculated by combining the timestamp of the corresponding dialogue. The low-dimensional index vector and the time decay factor are then stored in a local vector database. During offline recognition, the weighted similarity between the current speech features and the index vectors in the local vector database is calculated. If the weighted similarity is higher than the memory wake-up threshold, the corresponding historical entity information is restored for decoding completion.

6. The method according to claim 1, characterized in that, The method also includes an edge-cloud collaborative error correction step: When the voice interaction device switches to online status, it obtains model correction samples sent from the cloud. The model correction samples include the original audio features of the cloud-recognized text and the local corresponding text. The dynamic time warping algorithm is used to align the cloud-recognized text with the original audio features; Freeze the backbone network parameters of the speech recognition model structure, and only unlock the parameters of the preset bottleneck adapter layer in the speech recognition model structure; The loss function is calculated based on the model, and gradient update training is performed only on the parameters of the bottleneck adapter layer.

7. The method according to claim 4, characterized in that, The process of calling the local personalized pronunciation model to perform personalized adaptation also includes arbitration and rollback steps for personalized failures, specifically including: A dual-path decoding mechanism is activated, using the local personalized pronunciation model to generate a first recognized text sequence, and simultaneously using the speech recognition model structure to generate a second recognized text sequence; Calculate the Lewinstein edit distance between the first recognized text sequence and the second recognized text sequence. If the Lewinstein edit distance is greater than zero, trigger the voiceprint consistency verification process. In the voiceprint consistency verification process, the voiceprint feature vector of the current speech signal is extracted, and the cosine similarity between it and the pre-stored reference voiceprint feature vector is calculated. If the cosine similarity is lower than the preset physiological state drift threshold, the current speech is determined to be in a non-steady-state feature range, a model rollback operation is performed, the second recognized text sequence is directly output as the final recognition result, and the local personalized pronunciation model is prohibited from being updated using the current speech data.

8. The method according to claim 5, characterized in that, After restoring the corresponding historical entity information for decoding and completion, the process also includes entity state logic verification and active disambiguation steps, specifically including: The controlled entity objects and operational intentions contained in the completed recognition results are analyzed; Access the real-time entity state map stored locally on the voice interaction device to obtain the current physical state parameters of the controlled entity object, wherein the current physical state parameters include at least online status and on / off status; If the controlled entity is offline, or if the target state indicated by the operation intention is consistent with the switch state, then a high-level logical conflict is determined to exist. In response to the existence of a high-level logical conflict, the direct output of the recognition result is intercepted, and the text-to-speech module is triggered to generate a clarification question as a system response.

9. The method according to claim 6, characterized in that, The process of performing gradient update training only on the parameters of the bottleneck adapter layer also includes a local feature protection step, specifically including: Before performing gradient update training, a temporary shadow model containing the parameters to be updated is constructed, and the local anchor point sample set stored in the voice interaction device is invoked; The local anchor sample set is input into the current version of the speech recognition model and the temporary shadow model respectively to obtain the output posterior probability distribution for the same sample, and the average KL divergence between the two sets of posterior probability distributions is calculated. The dynamic fusion coefficient is calculated based on the average KL divergence, and the dynamic fusion coefficient is negatively correlated with the average KL divergence. The parameters of the bottleneck adapter layer are weighted and updated using the dynamic fusion coefficients to suppress the forgetting of local acoustic characteristics while introducing cloud optimization.

10. An adaptive speech recognition system based on domain feature fusion, characterized in that, include: The hardware computing power detection module is configured to acquire device hardware parameters, determine the computing power mode based on a preset computing power classification table, and activate the corresponding model structure. The computing power modes include at least a first computing power mode, a second computing power mode, and a third computing power mode, wherein the computing power of the first computing power mode is lower than that of the second computing power mode, and the computing power of the second computing power mode is lower than that of the third computing power mode. Activating the corresponding model structure specifically includes: if the first computing power mode is determined, calling a model structure containing depthwise separable convolutional layers as low-level networks and gated recurrent units as high-level networks, and configuring it for low-precision integer operations; if the second computing power mode is determined, calling a model structure containing depthwise separable convolutional layers as low-level networks and bidirectional long short-term memory networks as high-level networks, and configuring it for half-precision floating-point operations; if the third computing power mode is determined, calling depthwise separable convolutional layers and bidirectional long short-term memory networks with more layers than the second computing power mode, and enabling a hash retrieval acceleration mechanism. The speech preprocessing module is configured to perform noise reduction, endpoint detection, and determine the main domain label on the input speech; The hierarchical feature extraction module is configured to perform hierarchical feature extraction based on the computing power mode and the main domain label to obtain the current domain feature vector. Specifically, it is configured to: extract cross-domain acoustic common features as low-level features, and calculate the similarity difference between the currently extracted low-level features and the preset domain feature template; compare the similarity difference with a preset domain feature fluctuation threshold, and when the similarity difference is greater than the domain feature fluctuation threshold, trigger the high-level feature extraction step for domain-specific features. The domain feature fluctuation threshold is set according to the computing power mode: it is set to a first value in the first computing power mode, and to a second value in the second and third computing power modes, and the first value is greater than the second value. A lightweight feature fusion module is configured to calculate the domain similarity between the current feature and known features, and dynamically adjust the fusion weights based on the domain similarity to generate the final acoustic features. Specifically, the module is configured to: select N reference domain templates with the highest domain similarity to the current domain feature vector, where N is a positive integer; calculate the attention weight corresponding to each reference domain template using a normalized exponential function, where the attention weight is equal to the exponential function value of the domain similarity of the current reference domain template divided by the sum of the exponential function values ​​of the domain similarity of all N reference domain templates; and use the calculated attention weights to perform a weighted summation of feature vectors from different domains to generate the final acoustic features. The offline adaptation decoding module is configured to use the final acoustic features for decoding and output the recognition result.

Images