Immersive Human-Computer Interaction Method and System Based on Speech Recognition

Abstract

This application relates to the field of speech recognition technology, specifically to an immersive human-computer interaction method and system based on speech recognition. The method includes: speech data acquisition and fundamental frequency extraction; quantitative assessment of dialect interference; analysis of speech tone imbalance trends; and adaptive learning rate adjustment and model optimization. This application aims to dynamically adjust the fixed learning rate in an online learning model to update the speech recognition model for the user's region in real time according to dialect characteristics, thereby facilitating immersive human-computer interaction for the user.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, specifically to an immersive human-computer interaction method and system based on speech recognition. Background Technology

[0002] Thanks to significant breakthroughs in deep learning, speech recognition has evolved from initially focusing on isolated words and sentences to supporting continuous speech, multilingual speech, and speech with emotional semantics. Combined with applications in VR / AR, IoT, and wearable devices, this has collectively built a multimodal interactive ecosystem. From initial command-based human-computer dialogue, it has progressed to contextualized natural communication, further enhanced by environmental awareness and contextual understanding, making this communication more seamless. By transforming manual human operations into more intimate and natural speech, it provides a superior human-computer interaction experience and contributes to the development of accessibility technology.

[0003] Most mainstream general-purpose speech recognition models are trained on standard Mandarin corpora. However, the dialects of the Kanto region are part of the Mandarin dialect group, and their unique intonation, vocabulary, and speech flow changes differ significantly from the training data distribution. This leads to biases in feature extraction and decoding, causing frequent errors and interruptions in interaction, greatly affecting user trust and the smoothness of immersive interactions. Currently, methods such as collecting Kanto dialect speech data for targeted model fine-tuning, introducing adversarial training, and accent adaptation are used to improve the model's generalization ability. However, due to the numerous dialect variations in the Kanto region and the significant differences in speech features between the model and standard Mandarin training data, speech recognition models face insufficient robustness in immersive human-computer interaction scenarios. The inadequacy of the fixed learning rate strategy is a key factor preventing the model from perfectly adapting to dialect variations. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an immersive human-computer interaction method and system based on speech recognition, and the specific technical solution adopted is as follows:

[0005] In a first aspect, one embodiment of this application provides an immersive human-computer interaction method based on speech recognition, the method comprising the following steps:

[0006] Real-time acquisition of the raw speech signal and its baseband data for each segment of speech in each user session;

[0007] The fundamental frequency data of the original speech signal of the current speech segment is compared with the preset fundamental frequency dynamic set to calculate the fundamental frequency offset. The speech clarity score of the current speech segment is obtained by using the STOI algorithm. Then, the dialect interaction interference degree of the current speech segment in each user session is calculated and a real-time dialect interaction interference sequence is constructed in chronological order.

[0008] The sequence is smoothed by a sliding window and fitted with a trend to construct an adaptive adjustment factor for dynamically adjusting the fixed learning rate in the online learning model during each user session.

[0009] By using a dynamically adjusted real-time learning rate as the step size for updating the parameters of the online learning model, the updated model parameters of the speech recognition model for the user's region are output, which are then used to update the speech recognition model for the user's region, thereby achieving real-time adaptation to dialect interference.

[0010] Preferably, the fundamental frequency offset is the average absolute deviation between the fundamental frequency data of the original speech signal of the current speech segment and the fundamental frequency dynamic set.

[0011] Preferably, the preset dynamic baseband set consists of the baseband values ​​of all identified voice segments in the current user session.

[0012] Preferably, the formula for calculating the dialect interaction interference degree is:

[0013]

[0014] Where A represents the dialect interference level of the current segment of speech in each user's conversation. This indicates the fundamental frequency offset of the current segment of speech. This represents the maximum fundamental frequency offset of all recognized speech segments in the current user session. This indicates the speech clarity score for the current segment of speech.

[0015] Preferably, the speech clarity score of the current speech segment is calculated by taking the original speech signal of the current speech segment and a standard Mandarin speech template as input, using the STOI algorithm to compare the original speech signal of the current speech segment with the standard Mandarin speech template; wherein, the standard Mandarin speech template is obtained by training on data from a preset standard Mandarin corpus.

[0016] Preferably, the method for performing sliding window smoothing and trend fitting on the sequence is as follows:

[0017] The mean dialect interaction interference degree of each sliding window is output using the sliding window averaging method for the sequence.

[0018] The trend slope of each sliding window is obtained by fitting a linear relationship to the sequence using the sliding window least squares method.

[0019] Both the sliding window averaging method and the sliding window least squares method use the same sliding window size and sliding step size.

[0020] Preferably, the formula for calculating the adaptive adjustment factor is:

[0021]

[0022] Where B represents the tendency of speech imbalance in each user's conversation; Let be the mean of dialect interaction interference within the i-th sliding window; This represents the maximum value of the average dialect interaction interference across all sliding windows. Let be the trend slope of dialect interaction interference within the i-th sliding window, and I be the total number of sliding windows;

[0023] The tendency of speech imbalance in each user session is used as an adaptive adjustment factor to dynamically adjust the online learning rate for each user session.

[0024] Preferably, the method for dynamically adjusting the learning rate is as follows:

[0025]

[0026] The real-time learning rate is dynamically adjusted for each user session. B represents the preset fixed learning rate, and B represents the tendency of speech dysfunction in each user's conversation. It is the hyperbolic tangent function.

[0027] Preferably, the speech recognition model is trained using data from a pre-set standard Mandarin corpus.

[0028] Secondly, another embodiment of this application also provides an immersive human-computer interaction system based on speech recognition, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the immersive human-computer interaction method based on speech recognition described above.

[0029] This application has at least the following beneficial effects:

[0030] 1. In order to improve the accuracy of dialect recognition in the Kanto region, this application adopts a method based on fundamental frequency deviation and STOI to comprehensively reflect the dialect interference features of intonation abnormality and semantic ambiguity, while ensuring a certain level of output fluency, so as to solve the problem of large recognition deviation in existing models.

[0031] 2. Due to the large fluctuations and instability of dialect interference time series, traditional length estimation methods based on fixed window size cannot adapt to its characteristics. This application uses sliding window average value and trend fitting to obtain the magnitude and trend changes of dialect interference intensity persistence, avoiding the adverse factors brought by instantaneous noise and random disturbances to the system's adaptive adjustment, and ensuring the robustness and foresight of the results.

[0032] 3. This application utilizes the dynamic change of the learning rate of the online learning model based on the tendency imbalance trend B, upgrading the original fixed learning rate to an adaptive learning rate driven by B. This enables the speech recognition model to adapt to the actual situation of human speech into dialect features in real time, smoothly and accurately, capturing the changes in the dialect environment without distortion. At the same time, it can improve the fluency of interaction and the user's sense of security, and also avoid overfitting and performance fluctuations.

[0033] This application dynamically adjusts the fixed learning rate in the online learning model to update the speech recognition model of the user's region in real time according to the characteristics of the dialect, so as to facilitate the user to achieve immersive human-computer interaction. Attached Figure Description

[0034] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart of an immersive human-computer interaction method based on speech recognition provided in one embodiment of this application. Detailed Implementation

[0036] Example 1

[0037] This application provides an embodiment of an immersive human-computer interaction method based on speech recognition, which can be found in the following documentation. Figure 1 The method includes the following steps:

[0038] Step 1: Voice data acquisition and fundamental frequency extraction.

[0039] Using publicly available standard corpora such as "Tsinghua THCHS-30" or "AISHELL", we selected pure and clear Mandarin acoustic signals from multiple speakers of different ages and genders collected on a large scale from the standard corpora to construct a reference speech set covering all phonemes and typical phoneme combinations of Mandarin, which serves as a standard Mandarin speech template.

[0040] For each user session, a high-sensitivity microphone array is integrated into the front end of the VR / AR headset or smart interactive device used by the user to collect the raw speech signal of each speech segment for real-time capture of user voice input; secondly, a feature extraction device is deployed in the system's speech processing pipeline to collect the fundamental frequency data of the raw speech signal, specifically by deriving the fundamental frequency data sequence from the raw speech signal in real time through pre-emphasis, framing, and fundamental frequency tracking algorithms.

[0041] At this point, the standard Mandarin speech template, the original speech signal of each segment of speech in each user's conversation, and the base frequency data of the original speech signal have been obtained.

[0042] Step 2: Quantitative assessment of dialect interference.

[0043] Because the pronunciation of the dialect in the Kanto region differs systematically from standard Mandarin in terms of intonation, vocabulary, and phonological changes, the speech recognition model trained using a standard Mandarin corpus may cause numerous recognition errors or even interruptions in the interaction when used by users in the region, greatly affecting the complete semantic understanding and interaction process.

[0044] Therefore, for the current segment of speech in each user session, the fundamental frequency data of the original speech signals of all segments of speech collected in real time in each user session are used as input. The fundamental frequency dynamic set is set as the set of fundamental frequency values ​​of all identified speech segments in the current user session. Based on this, the average absolute deviation between the fundamental frequency value of the original speech signal of the current segment of speech and the fundamental frequency dynamic set is calculated and denoted as the fundamental frequency offset of the current segment of speech. This allows us to measure the impact of dialect on intonation. All identified speech segments are those preceding the current speech segment in each user's conversation.

[0045] First, a dynamic set of base frequencies is established for the current session; then, through... It represents the current speech intonation level relative to the user's intonation level. That is, the output... This indicates the deviation of speech intonation from the user's own fundamental frequency characteristics, thus reflecting the degree of abnormal changes in speech intonation. Such changes are often caused by the intonation patterns unique to dialects and directly affect the accurate parsing of speech modalities in multimodal semantic analysis.

[0046] However, fundamental frequency deviation alone cannot fully reflect the multidimensional interference of dialects on semantic understanding. Lexical and speech flow sound changes can all lead to uncertainty in the decoding results.

[0047] Accordingly, this application uses the original speech signal of the current speech segment and a standard Mandarin speech template as input, and employs the STOI (Short-Time Objective Intelligibility) algorithm to compare the original speech signal of the current speech segment with the standard Mandarin speech template to calculate the speech clarity score of the current speech segment. The higher This indicates that the clearer the current segment of speech, the lower the recognition uncertainty, and the less speech distortion and semantic ambiguity caused by dialect. The STOI algorithm is a well-known technique and will not be elaborated further.

[0048] Based on the above analysis, the dialect interaction interference degree A of the current segment of speech in each user's session is constructed, and the specific calculation formula is as follows:

[0049]

[0050] in, This represents the maximum value of the fundamental frequency offset for all recognized speech segments in the current user session.

[0051] In existing technologies, dynamic fundamental frequency distribution can quantify the degree of abnormality in speech intonation. The larger the value, the greater the difference between the current speech and the speaker's own intonation characteristics, making it more susceptible to interference from dialect-specific intonation patterns and thus more difficult to analyze. In existing technologies, the STOI algorithm is used to objectively evaluate the clarity of two speech signals, and its calculation... The smaller, The larger the value, the greater the deviation between the two speech signals, meaning the less clear the current speech segment is.

[0052] Guided by prior knowledge, the dialect interaction interference of the current segment of speech in each user's conversation is calculated by using the fundamental frequency offset and speech intelligibility score. This can better take into account the anomalies and uncertainties brought by dialects, thereby achieving the purpose of comprehensive evaluation.

[0053] To capture the dynamic changes in dialect influence over time, for each user session, the dialect interaction interference degree of all speech segments preceding the current speech segment in each session can be calculated. The dialect interaction interference degree of all speech segments preceding the current speech segment in each session is sorted according to the input time of the speech segments to obtain the real-time dialect interaction interference sequence of each user session. The dialect interaction interference sequence integrates discrete interference measures into continuous time series data, enabling the system to analyze the instantaneous fluctuations and long-term trends of dialect interference.

[0054] Step 3: Analysis of the trend of speech tone imbalance.

[0055] Since dialect interaction interference is a quantitative indicator that measures the magnitude of interference caused by dialect speech information during dialect speech recognition and understanding, its changes over time also have significant instantaneous fluctuations and randomness due to the instability of the speech signal itself, the instability of user pronunciation, and the influence of external environmental noise. If the dialect interaction interference parameter is used for system adaptive adjustment without considering other factors, it will cause the model parameters to oscillate at a large frequency, resulting in discontinuous or even interrupted interactive responses, thereby destroying the smooth experience of multimodal semantic analysis models in VR / AR and other application scenarios.

[0056] Based on the statistical characteristics of users' conversational language usage behavior, and using the real-time dialect interaction interference sequence of each user's conversation as input, a sliding window averaging method is used. The sliding window is set with a length of 5 and a step size of 1, that is, 5 consecutive speech segments are selected to achieve a trade-off between response speed and stability. That is, if W is too small, it is easily affected by noise; but if W is too large, it will cause response lag. The average dialect interaction interference degree of each sliding window is output, which represents the state of dialect interference degree changing over time, thereby reducing short-term jitter and more robustly representing the influence of dialect in the user's speech.

[0057] However, because the smoothed mean cannot reflect dynamic changes, it may cause the system to lag in adaptive adjustment. When the user's accent increases or decreases, it cannot make corresponding adjustments, resulting in cumulative errors during the interaction process.

[0058] Therefore, taking the real-time dialect interaction interference sequence of each user session as input, the sliding window size and sliding step size are set to be the same as above. The trend slope S of each sliding window is obtained by fitting the linear relationship using the sliding window least squares method. This output indicates the direction and rate of dialect interference changes, determines whether the influence of dialect is strengthening or weakening, and provides a forward-looking adjustment basis for the multimodal semantic analysis model, preventing interaction lag caused by ignoring the changing trend.

[0059] Based on the above analysis, the tendency imbalance trend B for each user's conversation is constructed, and the specific calculation formula is as follows:

[0060]

[0061] Let be the mean of dialect interaction interference within the i-th sliding window; This represents the maximum value of the average dialect interaction interference across all sliding windows. Let be the trend slope of dialect interaction interference degree within the i-th sliding window, used to reflect the rate of change of dialect interaction interference degree per unit time, I is the total number of sliding windows, and i is the summation index, with a value ranging from 1 to the total number of sliding windows I.

[0062] In existing technologies, the sliding window averaging method is used to smooth instantaneous fluctuations in time series, and a larger output mean indicates a higher level of persistence of the time series. In existing technologies, sliding window least squares linear fitting is used to extract local trends in a series, and a larger absolute value of the output slope indicates a stronger trend. Therefore, in this scheme... The explanation of the influence of dialect in the sliding window represents the degree to which a user's speech contains dialectal characteristics; while the trend slope This indicates the dynamic trend of changes in the degree of influence of dialects within the window.

[0063] Since the normalized mean reflects the stability benchmark and the slope represents the degree of change, the normalized mean can be multiplied by the trend slope and multiplied by the multi-window information to obtain the result of the dialectal disorder trend. This result can not only reflect the intensity of dialectal disturbances but also reflect the state of change, and thus complete the fusion between the two interference identification mechanisms.

[0064] Step 4: Adaptive learning rate adjustment and model optimization.

[0065] The speech recognition model is greatly affected by factors such as the intonation, vocabulary, and sound changes of the Kanto dialect. When using general corpora to train the speech recognition model, the speech features will be biased when recognizing multimodal interaction scenarios (VR / AR / smart home). This can easily lead to problems such as increased recognition error rate, intermittent interaction, and damage to the user's immersive experience, thereby greatly reducing users' trust in natural speech interaction.

[0066] Therefore, it is necessary to design an online learning control strategy based on the tendency imbalance trend B, so that the system can understand the severity of dialect interference in the current interaction and dynamically adjust the learning intensity.

[0067] This application takes the real-time calculated speech dissonance trend B, the user's multimodal interaction feedback data, and the parameters of the current speech recognition model as input to the online learning model, and outputs the updated model parameters of the speech recognition model for the user's region. This is used to update the speech recognition model for the user's region, realize real-time adaptation to dialect interference, and facilitate the improvement of the immersive human-computer interaction experience for users in the corresponding region.

[0068] The specific process of the online learning model is as follows: Based on the intonation imbalance trend B, the learning rate is dynamically adjusted using a formula. The adjusted real-time learning rate is used as the step size for updating the parameters of the online learning model, allowing the learning rate to adapt to the intensity of dialect interference. Then, a multimodal semantic analysis model is used to intelligently filter and weight the interaction data, prioritizing user error correction samples and interaction failure cases under high B values. Finally, an incremental learning algorithm is used to update the parameters of the speech recognition model online, combining weighted training data and the dynamic learning rate. A closed-loop optimization mechanism is formed by continuously monitoring the changes in interaction success rate and intonation imbalance trend B. This enables the speech recognition model to adapt smoothly to the dynamic changes in dialect features in real time, improving recognition accuracy and interaction fluency, avoiding overfitting and performance fluctuations, thereby enhancing user experience and system robustness in immersive human-computer interaction scenarios.

[0069] Specifically, multimodal interaction feedback data is captured in real time through system logs, including implicit signals such as direct corrections by users to incorrect recognition (e.g., canceling operations, repeating instructions) and task completion success rates; the parameters of the current speech recognition model are directly read from the memory of the speech recognition engine that is providing the service, including but not limited to embedding layer parameters and learning rate, which together constitute the input samples for online learning.

[0070] However, due to the differences in the speech characteristics of Kanto dialect speakers and the distribution of training data, coupled with the varying intensity of dialect usage at different times and in different situations, online learning strategies with fixed learning rates cannot adapt to this situation. When dialect features are very obvious, a fixed learning rate that is too small will result in slow parameter updates, failing to keep up with changes in user pronunciation. Conversely, when dialect features are very weak, a fixed learning rate that is too large may cause the model to misinterpret certain words based on its own language knowledge, losing its original understanding of the meaning of the words.

[0071] Therefore, it is necessary to use the tendency imbalance trend B, which reflects the strength and trend of dialect influence in real time, as an adaptive adjustment factor to adaptively improve the learning rate, and change the fixed learning rate. The learning rate is transformed into a dynamically adjusted real-time rate, and the specific improved formula is as follows:

[0072]

[0073] The real-time learning rate is dynamically adjusted for each user session. The preset fixed learning rate is a hyperparameter in the optimization of the online learning model, used to control the step size of parameter updates. In this embodiment, the value is 0.01. B is the tone imbalance trend of each user session. As an adaptive adjustment factor of the learning rate, it is a composite index that combines the smoothing mean and the trend of change. The hyperbolic tangent function maps the adaptive adjustment factor B to the range (-1, 1), ensuring a smooth and controllable change in the dynamic learning rate. Since B can take on large positive or negative values, directly defining the learning rate based on the value of B would cause drastic oscillations, easily leading to model divergence and instability. The hyperbolic tangent function ensures that when B is very large, it tends towards 1. near This accelerates the model's adaptability; however, when the B value is very small, it tends towards -1. To avoid overfitting the model, the learning rate is kept close to 0. To maintain a stable match between learning intensity and dialect interference intensity.

[0074] In existing technologies, online learning models use a fixed learning rate to determine the step size of parameter updates, ensuring smooth convergence during training. The fixed learning rate represents the learning intensity of the model in a standard Mandarin environment, maintaining the generality and stability of the learning. In contrast, the adaptive adjustment factor is a physically quantified measure of the danger level of dialect interference, comprehensively assessing the intensity and variation of dialect interference. Effective online learning requires adjusting the learning intensity to match the degree of interference based on the actual situation, while a fixed learning rate only considers general dialect situations based on standard Mandarin and cannot cope with dynamic changes in interference.

[0075] Based on the above principles, a fixed learning rate and an adjustment factor will be used. Multiplication allows the learning rate to increase with the severity of dialect interference, thus achieving a match between learning intensity and interference intensity: when the B value is large, increasing the learning rate accelerates the model's learning rate against interference; when the B value is small, decreasing the learning rate slows down the model's learning rate against interference, ensuring stability while also enabling a rapid response to interference.

[0076] Thus, the invention of an immersive human-computer interaction method and system based on speech recognition has been completed.

[0077] Example 2

[0078] Another embodiment of this application provides an immersive human-computer interaction system based on speech recognition, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the immersive human-computer interaction method based on speech recognition described above.

[0079] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not invented in this application.

[0080] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method of immersive human-machine interaction based on speech recognition, characterized in that, The method comprises the following steps: Real-time acquisition of the original speech signal and its fundamental frequency data of each speech segment in each session of the user; Comparing the fundamental frequency data of the original speech signal of the current speech segment with the preset dynamic set of fundamental frequencies, calculating the fundamental frequency offset, and using the STOI algorithm to obtain the speech intelligibility score of the current speech segment, and then calculating the dialect interaction interference degree of the current speech segment in each session of the user, and forming a real-time dialect interaction interference sequence in time sequence; the calculation formula of the dialect interaction interference degree is: wherein A represents the dialect interaction interference degree of the current segment voice in each session of the user, represents the base frequency offset of the current segment voice, represents the maximum value of the base frequency offset of all recognized voice segments in the current user session, represents the voice articulation score of the current segment voice; Sliding window smoothing and trend fitting are performed on the sequence to build an adaptive adjustment factor for dynamically adjusting the fixed learning rate in the online learning model during each session of the user; the calculation formula of the adaptive adjustment factor is: B is the tendency of the user's tone of speech in each session; is the average of the dialectal interference degree in the i-th sliding window; is the maximum of the average of the dialectal interference degree in all sliding windows; is the tendency slope of the dialectal interference degree in the i-th sliding window, and I is the total number of sliding windows. The speech manner disorder trend of each session of the user is used as an adaptive adjustment factor for dynamically adjusting the online learning rate during each session of the user; The dynamically adjusted real-time learning rate is used as the step size of the parameter update of the online learning model, and the updated model parameters of the speech recognition model in the region where the user is located are output, which are used to update the speech recognition model in the region where the user is located, and realize real-time adaptation to dialect interference; the dynamic adjustment method of the learning rate is: a real-time learning rate dynamically adjusted for each session of the user, a preset fixed learning rate, B is the tendency of the user's each session of the user's speech manner disorder, is a hyperbolic tangent function.

2. The voice recognition based immersive human computer interaction method as claimed in claim 1, wherein, The fundamental frequency offset is the average absolute deviation between the fundamental frequency data of the original speech signal of the current speech segment and the dynamic set of fundamental frequencies.

3. The voice recognition based immersive human computer interaction method as claimed in claim 2, wherein, The preset dynamic set of fundamental frequencies is composed of the fundamental frequency values of all recognized speech segments in the current user session.

4. The voice recognition based immersive human computer interaction method as claimed in claim 1, wherein, The speech intelligibility score of the current speech segment is calculated by using the STOI algorithm to compare the original speech signal of the current speech segment with the standard Mandarin speech template, taking the original speech signal of the current speech segment and the standard Mandarin speech template as input; wherein the standard Mandarin speech template is obtained by training data in a preset standard corpus of Mandarin.

5. The voice recognition based immersive human computer interaction method as claimed in claim 1, wherein, The method for sliding window smoothing and trend fitting of the sequence is: The sliding window average method is used to output the mean value of the dialect interaction interference degree of each sliding window for the sequence; The least square method of sliding window is used to fit the linear relationship to obtain the trend slope of each sliding window; Wherein, the sliding window average method and the least square method of sliding window use the same sliding window size and sliding step.

6. The voice recognition based immersive human computer interaction method as claimed in claim 1, wherein, The speech recognition model is obtained by training data in a preset standard corpus of Mandarin.

7. An immersive human-machine interaction system based on speech recognition, comprising a memory, a processor and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to realize the speech recognition-based immersive human-computer interaction method according to any one of claims 1-6.

Images