Language learning system and method of man-machine interaction equipment

Abstract

The invention discloses a language learning system of man-machine interaction equipment, which comprises a language learning system of the man-machine interaction equipment. The language learning system of the man-machine interaction equipment comprises a voice acquisition module, a preprocessing module, a voice analysis module, a voice matching module, a semantic recognition module, a scene interaction module, a large database module and a system auxiliary module. The invention discloses a language learning method of human-computer interaction equipment. The language learning method comprises the following steps: S1, a user selects a scene mode through a scene interaction module; s2, user voice communication; s3, a preprocessing module preprocesses the voice information; s4, the voice analysis module processes the audio signal; s5, matching the audio signals; s6, performing semantic recognition on the text information; and S7, answering according to the selected mode scene. The system has the advantages that the audio signals are preprocessed, the accuracy of the audio signals is improved, semantic analysis and matching are facilitated, and man-machine communication is performed according to voice.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and more specifically, particularly relates to a language learning system of a human-computer interaction device. At the same time, the present invention also relates to a language learning method of a human-computer interaction device. Background Art

[0002] Human-computer interaction is a discipline that studies the interactive relationship between a system and its users. A system can be a machine, or a computer system and software. Through the interaction, communication, and information exchange between humans and computers, a series of inputs and outputs are generated, and then a task is completed or a goal is achieved.

[0003] Usability is an important part of the human-computer interaction system. It is related to whether the human-computer interaction can achieve the user's expected goals, as well as the efficiency and convenience of achieving this goal.

[0004] Multi-channel interaction allows users to communicate with computer systems using natural interaction methods such as voice, gestures, eye contact, and expressions.

[0005] The ultimate goal of intelligent user interfaces is to make human-computer interaction as natural and convenient as human-human interaction. Context awareness, eye tracking, gesture recognition, 3D input, speech recognition, expression recognition, handwriting recognition, and natural language understanding are all important issues that need to be addressed in cognitive and intelligent user interfaces.

[0006] Voice interaction belongs to the category of human-computer interaction and is one of the most cutting-edge interaction methods currently developed in human-computer interaction: it is the process of giving instructions to machines using human natural language to achieve one's own goals.

[0007] The essence of voice interaction is human-computer interaction. Human-computer interaction is a discipline that studies the interactive relationship between a system and a user. A system can be a machine, or a computer system and software. Through the interaction, communication, and information exchange between humans and computers, a series of inputs and outputs are generated, and then a task is completed or a goal is achieved. Voice interaction is to use voice as an information carrier to interact with a machine.

[0008] The basis of language learning is communication. Only a large amount of language communication can improve the efficiency of language learning. However, in real life, it is impossible to effectively increase the communication time of language learners. In order to increase the communication time of language learners, language learning communication processing is carried out through human-computer interaction.

[0009] However, there are some problems with the existing technology: the existing human-computer interaction is prone to speech recognition errors due to problems such as voice collection, resulting in inaccurate responses to human-computer interaction. Especially when the human-computer performs audio signal processing, it is necessary to accurately extract the audio signal to facilitate matching and analysis of the audio signal. Therefore, we propose a language learning system and method for human-computer interaction devices. Summary of the invention

[0010] In view of the problems existing in the prior art, the purpose of the present invention is to provide a language learning system and method for a human-computer interaction device, which improves the accuracy of the audio signal by preprocessing the audio signal and determining and re-analyzing the analysis content, thereby facilitating improving the accuracy of subsequent analysis and processing of the audio signal.

[0011] To achieve the above object, the present invention provides the following technical solutions: a language learning system for a human-computer interaction device, comprising a language learning system for a human-computer interaction device, wherein the language learning system for the human-computer interaction device comprises a speech acquisition module, a preprocessing module, a speech analysis module, a speech matching module, a semantic recognition module, a scenario interaction module, a large database module and a system auxiliary module;

[0012] The voice collection module uses a microphone to collect the voice of the user when speaking interactively;

[0013] The preprocessing module is used to preprocess the audio signal collected by the voice collection module, remove noise from the collected audio signal, enhance the voice, improve the quality of the audio signal, and reduce distortion; and perform frame processing on the audio signal, dividing the audio signal into small frames to facilitate feature extraction;

[0014] The speech analysis module is used to extract feature information from the audio signal divided into small frames in the preprocessing module by using Mel frequency cepstral coefficients that simulate the human ear's perception of different frequencies;

[0015] The speech matching module is used to match the feature information extracted in the speech analysis module with the vocabulary library, and convert the collected audio signal into text information;

[0016] The semantic recognition module is used to perform spelling correction and semantic analysis on the text information converted and generated by the voice matching module. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the voice analysis module and the voice matching module re-analyze and match, and combine the results of the two analyses and matches to regenerate the text information. The semantic analysis is used to analyze the meaning of the user's voice.

[0017] The scenario interaction module is used to set the interaction scenario and make corresponding responses according to the speech meaning analyzed by the semantic recognition module to achieve human-computer interaction;

[0018] The large database module is used for storing vocabulary and is convenient for storing new vocabulary generated by autonomous learning;

[0019] The system auxiliary module is used to improve the convenience of use of the human-computer interaction device, and the system auxiliary module includes a memory, a buzzer, a microphone and operation buttons.

[0020] Preferably, the algorithm used in the preprocessing module to remove noise from the audio signal is an adaptive mean square error algorithm, and the adaptive mean square error algorithm formula is as follows:

[0021] y(t)=w T (t)(x(t)-f(a)), where y(t) is the audio signal output after filtering, x(t) is the audio signal input to be filtered, and w T (t) is the adaptive filter coefficient vector, f(a) is the estimated noise spectrum, which is used to eliminate environmental noise;

[0022] And, w T The adaptive update formula of (t) is as follows:

[0023] w(t+1)=w(t)+αe(t)(x(t)-f(a)), where w(t+1) is the filter coefficient at time t+1, w(t) is the filter coefficient at time t, α is the step factor that controls the update rate, e(t) is the error signal, x(t) is the input audio signal to be filtered, and f(a) is the estimated noise spectrum.

[0024] Preferably, the error signal e(t) is calculated as the difference between the actual desired signal d(t) and the filter output y(t), and the calculation formula is as follows:

[0025] e(t)=d(t)-y(t);

[0026] The estimated noise spectrum f(a) is calculated by collecting 25ms of environmental noise before the language learning system of the human-computer interaction device collects the user's voice information, and estimating the power spectrum of the noise, and the calculation formula is as follows:

[0027] f(a)=|X(t,a) 2 , where X(t,a) is the environmental noise collected during the time period;

[0028] The calculation formula of the step size factor α is as follows:

[0029] Among them, P n (t-1) is the noise power estimate at the previous moment, P s (t) is the signal power at the current moment, α(t) is the adaptive α value at time t, and α min and α max are the minimum and maximum values ​​of α, It is used to eliminate the large gap between the maximum and minimum values, reduce the efficiency of α update, and keep the noise spectrum estimate updated slowly.

[0030] Preferably, the frame processing process in the preprocessing module is as follows:

[0031] Assume that the input audio signal is x[n], where the length of the audio signal is N, the frame length is L, and the frame overlap is M;

[0032] The calculation formula for each frame signal after framing is as follows:

[0033] xf[n]=x[m·(LM)+n], where m is the index of the frame, n is the index of the sample point in the frame, and the starting position of each frame is n=m·(LM), m=0, 1, 2, ...;

[0034] Each frame of audio signal is processed by the window function w[n];

[0035] xf[n]·w[n], so that the audio signal is stationary within each frame.

[0036] Preferably, the Mel frequency cepstral coefficient is used to filter the audio signal after the frame processing, and the calculation formula of the Mel frequency is:

[0037] Where, f is the linear frequency;

[0038] Perform discrete cosine transform on the logarithmic Mel spectrum to obtain the final MFCC features. The DCT formula is:

[0039]

[0040] Where C[k] is the MFCC coefficient and M[n] is the logarithmic Mel spectrum.

[0041] Preferably, the speech matching module is used to match the collected audio signal features with the features of the words in the vocabulary library, and calculate the sum of the absolute differences between the audio signal features and the features of the words in the vocabulary library in various dimensions by Manhattan distance, and the calculation formula is as follows:

[0042]

[0043] Among them, x i is the audio signal feature, y i The features of the words in the vocabulary library, and when d(x,y)≤1, the audio signal features and the features of the words in the vocabulary library are determined to be the same features, and the words in the vocabulary library are written into the text.

[0044] Preferably, the calculation formula for spelling correction in the semantic recognition module is as follows:

[0045]

[0046] Where Y is the error rate or abnormality rate of the calculation, and when Y reaches more than 30%, re-analysis and matching are performed, and the results of the two analyses and matching are combined to regenerate the text information, x n is the total amount of audio signals after all frame processing, T is the sum of all error rates or abnormal audio signals after frame processing, and n is the total number of audio signals after all frame processing, and i is the position of the audio signal after the error rate or abnormal frame processing.

[0047] Preferably, the scenario interaction module includes several scenario environments, which is convenient for users to select a scenario environment for dialogue, and the scenario environment includes at least work scenarios, social scenarios, school scenarios and shopping scenarios, and is set with multiple language information to facilitate learners to learn languages.

[0048] A language learning method for a human-computer interaction device comprises the following steps:

[0049] S1. The user selects a scenario mode through the scenario interaction module: before starting to use, the user selects a mode through the operation buttons in the system auxiliary module, that is, the user selects the required learning scenario mode in the scenario interaction module according to his own needs, and selects it according to the language information;

[0050] S2. User voice communication: After the human-computer interaction device issues a communication notification, the user speaks, and the microphone in the system auxiliary module collects the voice during the speech;

[0051] S3. The preprocessing module preprocesses the voice information: The preprocessing module first removes the noise in the collected audio signal through an adaptive mean square error algorithm, and enhances the voice to improve the quality of the audio signal and reduce distortion; and the enhancement formula of the audio signal is as follows:

[0052] y(t)=x(t)-β·x(t-1),

[0053] Where β is the pre-emphasis coefficient, x(t) is the original audio signal, y(t) is the pre-emphasized audio signal, and x(t-1) is the audio signal of the previous time period;

[0054] After the audio signal is enhanced, it is framed and divided into small frames to facilitate feature extraction.

[0055] S4, the speech analysis module processes the audio signal: after filtering, enhancing and framing the audio signal, the speech analysis module extracts feature information from the framing audio signal, and extracts feature information through Mel frequency cepstrum coefficients. Because the audio signal has been enhanced and windowed in the preprocessing module, feature extraction is performed directly.

[0056] S5, matching the audio signal: matching the audio feature information processed by the analysis module with the feature information in the vocabulary library, and converting the collected audio signal to generate text information after feature matching;

[0057] S6. Semantic recognition of text information: The semantic recognition module performs spelling correction and semantic analysis on the text information. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the audio signal is re-analyzed and matched, and the results of the two analyses and matches are combined to regenerate the text information, and then the meaning of the user's voice is analyzed through semantic analysis;

[0058] S7. Reply according to the selected mode scenario: The human-computer interaction device responds and communicates according to the selected mode scenario and language information, so that the user can communicate and learn the language.

[0059] Preferably, the steps of reanalyzing and matching in S6 are as follows:

[0060] S601, re-calling the audio signal cached in the memory of the system auxiliary module and then pre-processing it;

[0061] S602, extracting feature information from the audio signal after frame processing by a speech analysis module;

[0062] S603, matching the audio feature information with the feature information in the vocabulary library, and combining the feature information twice for combined matching processing, and finally converting the collected audio signal to generate text information;

[0063] S604, perform spelling correction and semantic analysis on the text information again, and after analysis and processing, communicate and answer through the scenario interaction module.

[0064] Technical effects and advantages of the present invention:

[0065] When the present invention is used, the collected audio signal is processed through the preprocessing module, the noise in the collected audio signal is removed, and the voice is enhanced, the quality of the audio signal is improved, and the distortion is reduced; and the audio signal is framed and processed, and the audio signal is divided into small frames, which is convenient for feature extraction, and when the audio signal is framed, each frame of the audio signal is processed by a window function, so that each frame of the audio signal can maintain a stable boundary, and the enhancement and window function processing of the audio signal are convenient for reducing the subsequent steps of feature extraction and improving the efficiency of feature extraction;

[0066] And when denoising the audio signal, the human-computer interaction device first obtains the environmental noise 25ms before the communication, estimates the power spectrum of the noise, and subtracts the environmental noise when calculating the denoising process to improve the accuracy of the audio signal, and realizes adaptive adjustment of the setting parameters of the filtering process through the step factor, and the step factor is calculated according to the maximum and minimum values, and the intermediate number of the calculation is brought into the calculation to eliminate the huge gap between the maximum and minimum values, reduce the α update efficiency, maintain the slow update of the noise spectrum estimation, improve the smoothness and stability of the audio signal processing, and maintain the stability of the audio signal;

[0067] The speech analysis module extracts features through Mel frequency cepstral coefficients to obtain feature information of the audio signal, and the speech matching module matches the features of the audio signal with the vocabulary library to convert and generate text information;

[0068] The semantic recognition module is used to perform spelling correction and semantic analysis on the converted text information. Spelling correction is mainly used to analyze the error rate or abnormality rate in the text information. When the error rate or abnormality rate in the text information reaches more than 30%, the audio signal is re-analyzed and matched, and the results of the two analyses and matches are combined to regenerate the text information. Semantic analysis is used to analyze the user's voice meaning, facilitate communication and answering based on the voice meaning, so that language learners can communicate a lot and improve learning efficiency.

[0069] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0071] Figure 1 It is a schematic diagram of the system structure provided by the present invention;

[0072] Figure 2 It is a schematic flow chart of the method steps provided by the present invention;

[0073] Figure 3 It is a schematic flow chart of the steps of re-analysis and matching provided by the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] like Figure 1 As shown, a language learning system for a human-computer interaction device provided by an embodiment of the present invention includes a language learning system for a human-computer interaction device, wherein the language learning system for the human-computer interaction device includes a speech acquisition module, a preprocessing module, a speech analysis module, a speech matching module, a semantic recognition module, a scenario interaction module, a large database module and a system auxiliary module;

[0076] The voice collection module uses a microphone to collect the voice of the user when speaking interactively;

[0077] The preprocessing module is used to preprocess the audio signal collected by the voice collection module, remove noise from the collected audio signal, enhance the voice, improve the quality of the audio signal, and reduce distortion; and perform frame processing on the audio signal, dividing the audio signal into small frames to facilitate feature extraction;

[0078] The speech analysis module is used to extract feature information from the audio signal divided into small frames in the preprocessing module by using Mel frequency cepstral coefficients that simulate the human ear's perception of different frequencies;

[0079] The speech matching module is used to match the feature information extracted in the speech analysis module with the vocabulary library, and convert the collected audio signal into text information;

[0080] The semantic recognition module is used to perform spelling correction and semantic analysis on the text information converted and generated by the voice matching module. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the voice analysis module and the voice matching module re-analyze and match, and combine the results of the two analyses and matches to regenerate the text information. The semantic analysis is used to analyze the meaning of the user's voice.

[0081] The scenario interaction module is used to set the interaction scenario and make corresponding responses according to the speech meaning analyzed by the semantic recognition module to achieve human-computer interaction;

[0082] The large database module is used for storing vocabulary and is convenient for storing new vocabulary generated by autonomous learning;

[0083] The system auxiliary module is used to improve the convenience of use of the human-computer interaction device, and the system auxiliary module includes a memory, a buzzer, a microphone and operation buttons.

[0084] In this embodiment, preferably, the algorithm used in the preprocessing module to remove noise from the audio signal is an adaptive mean square error algorithm, and the adaptive mean square error algorithm formula is as follows:

[0085] y(t)=w T (t)(x(t)-f(a)), where y(t) is the audio signal output after filtering, x(t) is the audio signal input to be filtered, and w T (t) is the adaptive filter coefficient vector, f(a) is the estimated noise spectrum, which is used to eliminate environmental noise;

[0086] And, w T The adaptive update formula of (t) is as follows:

[0087] w(t+1)=w(t)+αe(t)(x(t)-f(a)), where w(t+1) is the filter coefficient at time t+1, w(t) is the filter coefficient at time t, α is the step size factor that controls the update rate, e(t) is the error signal, x(t) is the input audio signal to be filtered, and f(a) is the estimated noise spectrum;

[0088] It should be noted that when performing denoising, removing environmental noise can effectively improve the accuracy of the audio signal, prevent environmental noise from affecting the audio signal, and affect the subsequent feature analysis of the audio signal.

[0089] In this embodiment, preferably, the calculation of the error signal e(t) is the difference between the actual desired signal d(t) and the filter output y(t), and the calculation formula is as follows:

[0090] e(t)=d(t)-y(t);

[0091] The estimated noise spectrum f(a) is calculated by collecting 25ms of environmental noise before the language learning system of the human-computer interaction device collects the user's voice information, and estimating the power spectrum of the noise, and the calculation formula is as follows:

[0092] f(a)=|X(t,a) 2 , where X(t,a) is the environmental noise collected during the time period;

[0093] The calculation formula of the step size factor α is as follows:

[0094] Among them, P n (t-1) is the noise power estimate at the previous moment, P s (t) is the signal power at the current moment, α(t) is the adaptive α value at time t, and α min and α max are the minimum and maximum values ​​of α, Used to eliminate the large gap between the maximum and minimum values, reduce the efficiency of α update, and keep the noise spectrum estimate updated slowly;

[0095] It should be noted that for the difference between the expected audio signal and the output audio signal, adaptive filtering of the audio signal is implemented to improve the filtering of the audio signal, and a smaller value of the step factor is selected, which can reduce the update efficiency and effectively improve the smoothness of the audio signal processing.

[0096] In this embodiment, preferably, the frame processing process in the preprocessing module is as follows:

[0097] Assume that the input audio signal is x[n], where the length of the audio signal is N, the frame length is L, and the frame overlap is M;

[0098] The calculation formula for each frame signal after framing is as follows:

[0099] xf[n]=x[m·(LM)+n], where m is the index of the frame, n is the index of the sample point in the frame, and the starting position of each frame is n=m·(LM), m=0, 1, 2, ...;

[0100] Each frame of audio signal is processed by the window function w[n];

[0101] xf[n]·w[n], so that the audio signal is stable in each frame;

[0102] It should be noted that the input audio signal is framed to improve the processing efficiency of the audio signal, and rapid feature extraction of each frame of the audio signal is achieved. In addition, boundary processing is performed on the audio signal after frame processing, which can improve the smoothness of each frame of the audio signal and facilitate subsequent feature extraction of each frame of the audio signal.

[0103] In this embodiment, preferably, the Mel frequency cepstral coefficient is used to filter the audio signal after the frame processing, and the calculation formula of the Mel frequency is:

[0104] Where, f is the linear frequency;

[0105] Perform discrete cosine transform on the logarithmic Mel spectrum to obtain the final MFCC features. The DCT formula is:

[0106]

[0107] Where C[k] is the MFCC coefficient, M[n] is the logarithmic Mel spectrum;

[0108] It should be noted that the Mel-frequency cepstral coefficient simulates the human ear's perception of different frequencies and can improve the accuracy of audio signal feature extraction.

[0109] In this embodiment, preferably, the speech matching module is used to match the collected audio signal features with the features of the words in the vocabulary library, and the sum of the absolute differences between the audio signal features and the features of the words in the vocabulary library in various dimensions is calculated by Manhattan distance, and the calculation formula is as follows:

[0110]

[0111] Among them, x i is the audio signal feature, y i The features of the words in the vocabulary library, and when d(x,y)≤1, determine that the audio signal features and the features of the words in the vocabulary library are the same features, and write the words in the vocabulary library into the text;

[0112] It should be noted that feature matching is performed on the collected audio signal, that is, matching is performed with the vocabulary in the large database module, and a text format is generated for the matched vocabulary to facilitate subsequent semantic recognition analysis.

[0113] In this embodiment, preferably, the calculation formula for spelling correction in the semantic recognition module is as follows:

[0114]

[0115] Where Y is the error rate or abnormality rate of the calculation, and when Y reaches more than 30%, re-analysis and matching are performed, and the results of the two analyses and matching are combined to regenerate the text information, x n is the total amount of audio signals after all frame processing, T is the sum of all error rates or abnormal audio signals after frame processing, and n is the total number of audio signals after all frame processing, and i is the position of the error rate or abnormal audio signal after frame processing;

[0116] It should be noted that in order to improve the matching degree of vocabulary, the information in the text data is detected to determine the error rate or anomaly rate in the text data, and when the error rate or anomaly rate reaches more than 30%, re-analysis and matching are performed, which can improve the analysis accuracy of the data information and facilitate the artificial intelligence device to answer and process.

[0117] In this embodiment, preferably, the scenario interaction module includes several scenario environments, which is convenient for users to select a scenario environment for dialogue, and the scenario environment at least includes work scenario, social scenario, school scenario and shopping scenario, and is set with multiple language information to facilitate learners to learn languages;

[0118] It should be noted that in order to create corresponding scenarios, it is convenient to simulate the most realistic voice dialogue and improve the convenience of language learning for users.

[0119] refer to Figure 2 and Figure 3 , a language learning method of a human-computer interaction device, comprising the following steps:

[0120] S1. The user selects a scenario mode through the scenario interaction module: before starting to use, the user selects a mode through the operation buttons in the system auxiliary module, that is, the user selects the required learning scenario mode in the scenario interaction module according to his own needs, and selects it according to the language information;

[0121] S2. User voice communication: After the human-computer interaction device issues a communication notification, the user speaks, and the microphone in the system auxiliary module collects the voice during the speech;

[0122] S3. The preprocessing module preprocesses the voice information: The preprocessing module first removes the noise in the collected audio signal through an adaptive mean square error algorithm, and enhances the voice to improve the quality of the audio signal and reduce distortion; and the enhancement formula of the audio signal is as follows:

[0123] y(t)=x(t)-β·x(t-1),

[0124] Where β is the pre-emphasis coefficient, x(t) is the original audio signal, y(t) is the pre-emphasized audio signal, and x(t-1) is the audio signal of the previous time period;

[0125] After the audio signal is enhanced, it is framed and divided into small frames to facilitate feature extraction.

[0126] S4, the speech analysis module processes the audio signal: after filtering, enhancing and framing the audio signal, the speech analysis module extracts feature information from the framing audio signal, and extracts feature information through Mel frequency cepstrum coefficients. Because the audio signal has been enhanced and windowed in the preprocessing module, feature extraction is performed directly.

[0127] S5, matching the audio signal: matching the audio feature information processed by the analysis module with the feature information in the vocabulary library, and converting the collected audio signal to generate text information after feature matching;

[0128] S6. Semantic recognition of text information: The semantic recognition module performs spelling correction and semantic analysis on the text information. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the audio signal is re-analyzed and matched, and the results of the two analyses and matches are combined to regenerate the text information, and then the meaning of the user's voice is analyzed through semantic analysis;

[0129] S7. Reply according to the selected mode scenario: The human-computer interaction device responds and communicates according to the selected mode scenario and language information, so that the user can communicate and learn the language.

[0130] In this embodiment, preferably, the steps of re-analyzing and matching in S6 are as follows:

[0131] S601, re-calling the audio signal cached in the memory of the system auxiliary module and then pre-processing it;

[0132] S602, extracting feature information from the audio signal after frame processing by a speech analysis module;

[0133] S603, matching the audio feature information with the feature information in the vocabulary library, and combining the feature information twice for combined matching processing, and finally converting the collected audio signal to generate text information;

[0134] S604, performing spelling correction and semantic analysis on the text information again, and after analysis and processing, communicating and answering through the scenario interaction module;

[0135] It should be noted that by re-analyzing and matching audio features, audio data errors caused by noise can be reduced, resulting in the inability to accurately understand the user's meaning and the inability to communicate and answer correctly. The audio signal can be cached to facilitate multiple analysis and matching, thereby improving the analysis accuracy of the audio signal.

[0136] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A language learning system for a human-computer interaction device, characterized in that: A language learning system comprising a human-computer interaction device, wherein the language learning system comprises a speech acquisition module, a preprocessing module, a speech analysis module, a speech matching module, a semantic recognition module, a scenario interaction module, a large database module and a system auxiliary module; The voice collection module uses a microphone to collect the voice of the user when speaking interactively; The preprocessing module is used to preprocess the audio signal collected by the voice collection module, remove noise from the collected audio signal, enhance the voice, improve the quality of the audio signal, and reduce distortion; and perform frame processing on the audio signal, dividing the audio signal into small frames to facilitate feature extraction; The speech analysis module is used to extract feature information from the audio signal divided into small frames in the preprocessing module by using Mel frequency cepstral coefficients that simulate the human ear's perception of different frequencies; The speech matching module is used to match the feature information extracted in the speech analysis module with the vocabulary library, and convert the collected audio signal into text information; The semantic recognition module is used to perform spelling correction and semantic analysis on the text information converted and generated by the voice matching module. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the voice analysis module and the voice matching module re-analyze and match, and combine the results of the two analyses and matches to regenerate the text information. The semantic analysis is used to analyze the meaning of the user's voice. The scenario interaction module is used to set the interaction scenario and make corresponding responses according to the speech meaning analyzed by the semantic recognition module to achieve human-computer interaction; The large database module is used for storing vocabulary and is convenient for storing new vocabulary generated by autonomous learning; The system auxiliary module is used to improve the convenience of use of the human-computer interaction device, and the system auxiliary module includes a memory, a buzzer, a microphone and operation buttons.

2. A language learning system for a human-computer interaction device according to claim 1, characterized in that: The algorithm used in the preprocessing module to remove noise from the audio signal is an adaptive mean square error algorithm, and the formula of the adaptive mean square error algorithm is as follows: y(t)=w T (t)(x(t)-f(a)), where y(t) is the audio signal output after filtering, x(t) is the audio signal input to be filtered, and w T (t) is the adaptive filter coefficient vector, f(a) is the estimated noise spectrum, which is used to eliminate environmental noise; And, w T The adaptive update formula of (t) is as follows: w(t+1)=w(t)+αe(t)(x(t)-f(a)), where w(t+1) is the filter coefficient at time t+1, w(t) is the filter coefficient at time t, α is the step factor that controls the update rate, e(t) is the error signal, x(t) is the input audio signal to be filtered, and f(a) is the estimated noise spectrum.

3. A language learning system for a human-computer interaction device according to claim 2, characterized in that: The error signal e(t) is calculated as the difference between the actual desired signal d(t) and the filter output y(t), and is calculated as follows: e(t)=d(t)-y(t); The estimated noise spectrum f(a) is calculated by collecting 25ms of environmental noise before the language learning system of the human-computer interaction device collects the user's voice information, and estimating the power spectrum of the noise, and the calculation formula is as follows: f(a)=|X(t,a) 2 , where X(t,a) is the environmental noise collected during the time period; The calculation formula of the step size factor α is as follows: Among them, P n (t-1) is the noise power estimate at the previous moment, P s (t) is the signal power at the current moment, α(t) is the adaptive α value at time t, and α min and α max are the minimum and maximum values ​​of α, It is used to eliminate the large gap between the maximum and minimum values, reduce the efficiency of α update, and keep the noise spectrum estimate slowly updated.

4. The language learning system of a human-computer interaction device according to claim 1, characterized in that: The frame processing process in the pre-processing module is as follows: Assume that the input audio signal is x[n], where the length of the audio signal is N, the frame length is L, and the frame overlap is M; The calculation formula for each frame signal after framing is as follows: xf[n]=x[m·(LM)+n], where m is the index of the frame, n is the index of the sample point in the frame, and the starting position of each frame is n=m·(LM), m=0, 1, 2, ...; Each frame of audio signal is processed by the window function w[n]; xf[n]·w[n], so that the audio signal is stationary within each frame.

5. The language learning system of a human-computer interaction device according to claim 1, characterized in that: The Mel frequency cepstral coefficient is used to filter the audio signal after the frame processing. The calculation formula of the Mel frequency is: Where, f is the linear frequency; Perform discrete cosine transform on the logarithmic Mel spectrum to obtain the final MFCC features. The DCT formula is: Where C[k] is the MFCC coefficient and M[n] is the logarithmic Mel spectrum.

6. The language learning system of a human-computer interaction device according to claim 1, characterized in that: The speech matching module is used to match the collected audio signal features with the features of the words in the vocabulary library, and calculates the sum of the absolute differences between the audio signal features and the features of the words in the vocabulary library in various dimensions through the Manhattan distance, and the calculation formula is as follows: Among them, x i is the audio signal feature, y i The features of the words in the vocabulary library, and when d(x,y)≤1, the audio signal features and the features of the words in the vocabulary library are determined to be the same features, and the words in the vocabulary library are written into the text.

7. The language learning system of a human-computer interaction device according to claim 1, characterized in that: The calculation formula for spelling correction in the semantic recognition module is as follows: Where Y is the error rate or abnormality rate of the calculation, and when Y reaches more than 30%, re-analysis and matching are performed, and the results of the two analyses and matching are combined to regenerate the text information, x n is the total amount of audio signals after all frame processing, T is the sum of all error rates or abnormal audio signals after frame processing, and n is the total number of audio signals after all frame processing, and i is the position of the audio signal after the error rate or abnormal frame processing.

8. The language learning system of a human-computer interaction device according to claim 1, characterized in that: The scenario interaction module includes several scenario environments, which is convenient for users to select a scenario environment for dialogue, and the scenario environment at least includes work scenarios, social scenarios, school scenarios and shopping scenarios, and is set with multiple language information to facilitate learners to learn languages.

9. A language learning method for a human-computer interaction device, characterized in that: The following steps are included: S1. The user selects a scenario mode through the scenario interaction module: before starting to use, the user selects a mode through the operation buttons in the system auxiliary module, that is, the user selects the required learning scenario mode in the scenario interaction module according to his own needs, and selects it according to the language information; S2. User voice communication: After the human-computer interaction device issues a communication notification, the user speaks, and the microphone in the system auxiliary module collects the voice during the speech; S3. The preprocessing module preprocesses the voice information: The preprocessing module first removes the noise in the collected audio signal through an adaptive mean square error algorithm, and enhances the voice to improve the quality of the audio signal and reduce distortion; and the enhancement formula of the audio signal is as follows: y(t)=x(t)-β·x(t-1), Where β is the pre-emphasis coefficient, x(t) is the original audio signal, y(t) is the pre-emphasized audio signal, and x(t-1) is the audio signal of the previous time period; After the audio signal is enhanced, it is framed and divided into small frames to facilitate feature extraction. S4, the speech analysis module processes the audio signal: after filtering, enhancing and framing the audio signal, the speech analysis module extracts feature information from the framing audio signal, and extracts feature information through Mel frequency cepstrum coefficients. Because the audio signal has been enhanced and windowed in the preprocessing module, feature extraction is performed directly. S5, matching the audio signal: matching the audio feature information processed by the analysis module with the feature information in the vocabulary library, and converting the collected audio signal to generate text information after feature matching; S6. Semantic recognition of text information: The semantic recognition module performs spelling correction and semantic analysis on the text information. When performing spelling correction, the error rate or abnormality rate in the text information is analyzed. When the error rate or abnormality rate in the text information reaches more than 30%, the audio signal is re-analyzed and matched, and the results of the two analyses and matches are combined to regenerate the text information, and then the meaning of the user's voice is analyzed through semantic analysis; S7. Reply according to the selected mode scenario: The human-computer interaction device responds and communicates according to the selected mode scenario and language information, so that the user can communicate and learn the language.

10. The language learning method of a human-computer interaction device according to claim 9, characterized in that: The steps of re-analysis and matching in S6 are as follows: S601, re-retrieve the audio signal cached in the memory of the system auxiliary module, and then perform pre-processing; S602, extracting feature information from the audio signal after frame processing by a speech analysis module; S603, matching the audio feature information with the feature information in the vocabulary library, and combining the feature information twice for combined matching processing, and finally converting the collected audio signal to generate text information; S604, perform spelling correction and semantic analysis on the text information again, and after analysis and processing, communicate and answer through the scenario interaction module.

Images