Spoken English emotion intelligent interactive teaching system and method based on deep learning

Abstract

The invention discloses an intelligent interactive teaching system and method for oral English emotion based on deep learning, and relates to the field of speech evaluation. The system comprises an environment detection and evaluation module, a threshold comparison module, an audio and video collection and processing module, an optimization module and a suggestion output module. Threshold value range comparison and image reference threshold value comparison are carried out on the spoken English intelligent interaction environment evaluation coefficient obtained in the environment detection and evaluation module, and threshold value comparison is carried out on the explosion loss judgment evaluation coefficient of the spoken English intelligent interaction audio obtained in the audio and video collection and processing module. According to the method, the intelligent oral English interaction audio data input by the user are evaluated to obtain the explosion judgment evaluation coefficient of the intelligent oral English interaction audio, and then the text, the audio and the visual oral English error correction suggestion are output according to the explosion judgment evaluation coefficient of the intelligent oral English interaction audio, so that the problem that in the prior art, the user experience is poor is solved. And the accuracy of judging the explosion in the context collaborative pronunciation change is insufficient.

Description

Technical Field

[0001] The present invention relates to the field of speech evaluation technology, and in particular to a system and method for interactive teaching of spoken English with emotional intelligence based on deep learning. Background Art

[0002] With the in-depth development of information technology and artificial intelligence, deep learning, as a powerful machine learning technology, has made breakthroughs in speech recognition, natural language processing, sentiment analysis, etc. The development of these technologies has provided a solid foundation for the construction of an intelligent interactive teaching system for spoken English, enabling the system to understand and evaluate learners' spoken expressions, including voice accuracy, grammatical correctness, and emotional expression.

[0003] In the prior art, the intelligent interactive teaching system for spoken English evaluates whether English pronunciation has lost plosive sounds by reflecting the frequency and amplitude of sound through acoustic spectrum characteristics.

[0004] For example, the invention patent with announcement number: CN111916106B discloses a method for improving the pronunciation quality in English teaching, including: obtaining voice information input by a user; recognizing the voice information, obtaining characteristic parameters of the voice information, and transmitting the characteristic parameters to a voice evaluation model; the voice evaluation model is used to evaluate the voice information according to the characteristic parameters and obtain a voice evaluation result; when the voice evaluation result is standard, reminding the user of the pronunciation standard through an output device; when the voice evaluation result is non-standard, obtaining the voice content corresponding to the voice information, and transmitting the voice content to a standard voice model; the standard voice model is used to obtain standard voice information according to the voice content, and output the standard voice information through an output device; comparing the voice information with the standard voice information to obtain corresponding voice guidance information; and transmitting the voice guidance information to the user through the output device to assist the user in pronunciation training.

[0005] For example, the invention patent with announcement number: CN113077822B discloses a method, device, equipment and storage medium for evaluating plosive sounds, including: obtaining English speech to be evaluated; sending the English speech into a decoding graph for forced alignment for recognition, wherein the decoding graph includes a first pronunciation path with plosive sounds and a second pronunciation path without plosive sounds; if the recognition process adopts the second pronunciation path in the decoding graph, then evaluating the pronunciation of the English speech for losing plosive sounds.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] In the prior art, since there are a large number of contextual co-pronunciation changes in long sentences input by user voice during the intelligent interaction process of spoken English, such as connected reading, weak reading, and mispronunciation, there is a problem of insufficient accuracy in judging mispronunciation in contextual co-pronunciation changes. Summary of the invention

[0008] The embodiments of the present application solve the problem of insufficient accuracy in judging misfires in contextual collaborative pronunciation changes in the prior art by providing an English oral emotion intelligent interactive teaching system and method based on deep learning, thereby achieving the effect of improving the accuracy of judging misfires in contextual collaborative pronunciation changes.

[0009] The embodiment of the present application provides an English oral emotion intelligent interactive teaching system and method based on deep learning, including: an environment detection and evaluation module, a threshold comparison module, an audio and video collection and processing module, an optimization module and a suggestion output module; the environment detection and evaluation module is used to collect and evaluate the English oral intelligent interaction environment data, obtain the English oral intelligent interaction environment evaluation coefficient, and transmit the English oral intelligent interaction environment evaluation coefficient to the threshold comparison module; the threshold comparison module is used to compare the English oral intelligent interaction environment evaluation coefficient obtained in the environment detection and evaluation module with the threshold range and the reference threshold of the image, and compare the misfire judgment evaluation coefficient of the English oral intelligent interaction audio obtained in the audio and video collection and processing module with the threshold; the audio and video collection and processing module is used to compare the English oral intelligent interaction audio data input by the user The data are collected and evaluated to obtain the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio, and the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio is transmitted to the suggestion output module; the optimization module is used to optimize the spoken English intelligent interactive environment according to the threshold range comparison result of the spoken English intelligent interactive environment evaluation coefficient. If the optimized spoken English intelligent interactive environment evaluation coefficient still does not pass the threshold range comparison, a command is sent to the audio and video collection and processing module to synchronously collect the spoken English intelligent interactive lip shape feature data when collecting the spoken English intelligent interactive audio data, and optimize the spoken English intelligent interactive lip shape feature according to the threshold comparison result of the spoken English intelligent interactive environment evaluation coefficient; the suggestion output module is used to output text, audio and visual spoken English error correction suggestions according to the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio.

[0010] Furthermore, the threshold comparison module includes a threshold range comparison unit and an image reference threshold comparison unit; the threshold range comparison unit is used to obtain the threshold range of the English speaking intelligent interaction environment evaluation coefficient from the database, compare the English speaking intelligent interaction environment evaluation coefficient with the threshold range of the English speaking intelligent interaction environment evaluation coefficient, if the English speaking intelligent interaction environment evaluation coefficient is within the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the audio and video collection and processing module, collect and evaluate the English speaking intelligent interaction audio data input by the user, and obtain the explosion failure judgment evaluation coefficient of the English speaking intelligent interaction audio, if the English speaking intelligent interaction environment evaluation coefficient is greater than or equal to the maximum value of the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the optimization module, perform dynamic weight allocation for spatial positioning of the English speaking intelligent interaction environment, if the English speaking intelligent interaction environment evaluation coefficient is less than the minimum value of the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the optimization module, perform frequency domain signal processing on the English speaking intelligent interaction environment. Signal processing, after dynamic weight allocation of spatial positioning or frequency domain signal processing, send instructions to the environment detection and evaluation module, re-collect and evaluate the English speaking intelligent interaction environment data, obtain the English speaking intelligent interaction environment evaluation coefficient, and transmit the English speaking intelligent interaction environment evaluation coefficient to the threshold comparison module. If the English speaking intelligent interaction environment evaluation coefficient is still not within the threshold range of the English speaking intelligent interaction environment evaluation coefficient at this time, send instructions to the audio and video collection and processing module, collect the English speaking intelligent interaction lip shape feature data synchronously when collecting the English speaking intelligent interaction audio data, combine the English speaking intelligent interaction audio data with the optimized English speaking intelligent interaction lip shape feature data for analysis, and obtain the explosion misfire judgment evaluation coefficient of the English speaking intelligent interaction audio. If the English speaking intelligent interaction environment evaluation coefficient is within the threshold range of the English speaking intelligent interaction environment evaluation coefficient at this time, send instructions to the audio and video collection and processing module, collect and evaluate the English speaking intelligent interaction audio data input by the user, and obtain the explosion misfire judgment evaluation coefficient of the English speaking intelligent interaction audio;The reference threshold comparison unit of the image is used to obtain the sharpness reference threshold of the image in the intelligent interaction of spoken English from the database, obtain the sharpness of the image in the intelligent interaction of spoken English through the sensor, compare the sharpness of the image in the intelligent interaction of spoken English with the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, if the sharpness of the image in the intelligent interaction of spoken English is less than or equal to the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the optimization module, optimize the lip shape feature data of the intelligent interaction of spoken English, combine and analyze the audio data of the intelligent interaction of spoken English with the optimized lip shape feature data of the intelligent interaction of spoken English, and obtain the evaluation coefficient of the misfire judgment of the audio of the intelligent interaction of spoken English, if the sharpness of the image in the intelligent interaction of spoken English is greater than the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the audio and video collection and processing module, combine and analyze the audio data of the intelligent interaction of spoken English with the lip shape feature data of the intelligent interaction of spoken English, and obtain the evaluation coefficient of the misfire judgment of the audio of the intelligent interaction of spoken English. ;

[0011] Furthermore, the specific process of collecting the spoken English intelligent interaction environment data is as follows: collecting the spoken English intelligent interaction environment data through sensors, preprocessing the spoken English intelligent interaction environment data to obtain preprocessed spoken English intelligent interaction environment data, and performing preliminary denoising on the preprocessed spoken English intelligent interaction environment data; the spoken English intelligent interaction environment data includes the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment.

[0012] Furthermore, the specific process of collecting the English spoken intelligent interactive audio data input by the user is: collecting the English spoken intelligent interactive audio data through sensors, preprocessing the English spoken intelligent interactive audio data, and obtaining the preprocessed English spoken intelligent interactive audio data; the English spoken intelligent interactive audio data includes the phoneme combination coverage of the audio and the instantaneous energy peak of the audio.

[0013] Furthermore, a specific method for obtaining the evaluation coefficient of the spoken English intelligent interaction environment is as follows: obtaining the weight factor of the noise sound pressure level of the spoken English intelligent interaction environment, the weight factor of the light intensity of the spoken English intelligent interaction environment, and the weight factor of the color temperature of the spoken English intelligent interaction environment from the database, arranging the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment in a time series, assigning weights to the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment, comparing the standard values ​​of the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment at each time series point with the real-time values ​​of the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment, and obtaining the evaluation coefficient of the spoken English intelligent interaction environment after averaging processing.

[0014] Furthermore, a specific method for obtaining the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is as follows: if it is necessary to combine the lip feature data, obtain the weight factor of the lip opening and closing distance, the weight factor of the audio phoneme combination coverage rate and the weight factor of the audio instantaneous energy peak value from the database, assign weights to the deviation value of the lip opening and closing distance from the lip opening and closing distance reference value, the audio phoneme combination coverage rate and the audio instantaneous energy peak value, sort the lip opening and closing distance, the audio phoneme combination coverage rate and the audio instantaneous energy peak value according to the number of phonemes, and calculate the weight factor of the lip opening and closing distance, the phoneme combination coverage rate and the audio instantaneous energy peak value. The phoneme combination coverage rate of the frequency and the instantaneous energy peak value of the audio are combined and processed to obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio; if it is not necessary to combine the lip features, the weight factor of the phoneme combination coverage rate of the audio and the weight factor of the instantaneous energy peak value of the audio are obtained from the database, weights are assigned to the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio, the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio are sorted according to the number of phonemes, the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio are combined and processed to obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio.

[0015] Furthermore, the specific optimization steps for optimizing the spoken English intelligent interaction environment according to the comparison results of the threshold range of the spoken English intelligent interaction environment evaluation coefficient are as follows: if the spoken English intelligent interaction environment evaluation coefficient is greater than or equal to the maximum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, send an instruction to the optimization module to perform dynamic weight allocation for spatial positioning of the spoken English intelligent interaction environment; if the spoken English intelligent interaction environment evaluation coefficient is less than the minimum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, send an instruction to the optimization module to perform frequency domain signal processing on the spoken English intelligent interaction environment.

[0016] Furthermore, the specific steps of synchronously collecting the lip shape feature data of spoken English intelligent interaction while collecting the audio data of spoken English intelligent interaction are as follows: acquiring the lip shape feature data of spoken English intelligent interaction through a depth sensor, extracting the lip shape feature data of spoken English intelligent interaction through a deep learning model, and synchronizing the audio and video using hardware triggering or software triggering; the lip shape feature data of spoken English intelligent interaction includes the distance between the opening and closing of the lips; the hardware triggering includes using a synchronization signal generator to send a pulse signal to simultaneously trigger the audio and video devices; the software triggering includes achieving timestamp alignment of the audio and video streams through the NDI protocol.

[0017] Furthermore, a specific output method for outputting text, audio and visual English speaking error correction suggestions based on the misfire judgment evaluation coefficient of the English speaking intelligent interactive audio is as follows: predefined text, audio and visual English speaking error correction suggestions are matched from a database based on the misfire judgment evaluation coefficient of the English speaking intelligent interactive audio, and the text, audio and visual English speaking error correction suggestions include displaying a spectrogram of the user's pronunciation and the standard pronunciation, playing back the user's English speaking recording and highlighting the error parts and pointing out the pronunciation error types and providing correct text pronunciation examples.

[0018] Furthermore, the deep learning-based emotional intelligent interactive teaching method for spoken English is as follows: collect and evaluate the spoken English intelligent interactive environment data to obtain the spoken English intelligent interactive environment evaluation coefficient, and transmit the spoken English intelligent interactive environment evaluation coefficient to the threshold comparison module; compare the spoken English intelligent interactive environment evaluation coefficient obtained in the environment detection and evaluation module with the threshold range and the reference threshold of the image, and compare the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio obtained in the audio and video collection and processing module with the threshold; collect and evaluate the spoken English intelligent interactive audio data input by the user to obtain the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio, and transmit the spoken English intelligent interactive environment evaluation coefficient to the threshold comparison module. The misfire judgment evaluation coefficient of the spoken English intelligent interaction audio is transmitted to the suggestion output module; the spoken English intelligent interaction environment is optimized according to the threshold range comparison result of the evaluation coefficient of the spoken English intelligent interaction environment; if the optimized spoken English intelligent interaction environment evaluation coefficient still fails to pass the threshold range comparison, a command is sent to the audio and video collection and processing module to synchronously collect the spoken English intelligent interaction lip shape feature data when collecting the spoken English intelligent interaction audio data, and optimize the spoken English intelligent interaction lip shape features according to the threshold comparison result of the spoken English intelligent interaction environment evaluation coefficient; text, audio and visual spoken English error correction suggestions are output according to the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio.

[0019] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] 1. By evaluating the English spoken intelligent interactive audio data input by the user, the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is obtained, and then text, audio and visual English spoken error correction suggestions are output according to the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio, thereby achieving the effect of improving the accuracy of judging misfires in contextual collaborative pronunciation changes, and effectively solving the problem of insufficient accuracy in judging misfires in contextual collaborative pronunciation changes in the prior art.

[0021] 2. By synchronously collecting English spoken intelligent interactive lip shape feature data while collecting English spoken intelligent interactive audio data, the English spoken intelligent interactive lip shape features are optimized according to the threshold comparison results of the English spoken intelligent interactive environment evaluation coefficient, thereby enhancing the multimodal learning experience of the English spoken emotional intelligent interactive teaching system based on deep learning, and effectively solving the problem of insufficient multimodal learning experience of the English spoken emotional intelligent interactive teaching system based on deep learning in the prior art.

[0022] 3. By comparing the threshold range of the English speaking intelligent interaction environment evaluation coefficient obtained in the environment detection and evaluation module, the English speaking intelligent interaction environment is optimized according to the threshold range comparison results of the English speaking intelligent interaction environment evaluation coefficient, thereby improving the environmental applicability of the English speaking emotional intelligent interactive teaching system based on deep learning, and effectively solving the problem of insufficient environmental applicability of the English speaking emotional intelligent interactive teaching system based on deep learning in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the structure of the English spoken emotion intelligent interactive teaching system based on deep learning provided in the embodiment of the present application;

[0024] Figure 2 A schematic diagram of the structure of a threshold comparison module in the deep learning-based English spoken emotion intelligent interactive teaching system provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of an evaluation coefficient of an intelligent interactive environment for spoken English in an emotional intelligent interactive teaching system for spoken English based on deep learning provided in an embodiment of the present application;

[0026] Figure 4 Schematic diagram of the deep learning-based emotional intelligent interactive teaching method for spoken English provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application solve the problem of insufficient accuracy in judging misfires in contextual collaborative pronunciation changes in the prior art by providing an English speaking emotional intelligent interactive teaching system and method based on deep learning. By evaluating the English speaking intelligent interactive audio data input by the user, an evaluation coefficient for misfire judgment of the English speaking intelligent interactive audio is obtained, and then text, audio and visual English speaking error correction suggestions are output according to the misfire judgment evaluation coefficient of the English speaking intelligent interactive audio, thereby achieving the effect of improving the accuracy of judging misfires in contextual collaborative pronunciation changes.

[0028] The technical solution in the embodiment of the present application is to solve the problem of insufficient accuracy in judging the change of contextual co-pronunciation. The overall idea is as follows:

[0029] First, the environment detection and evaluation module is responsible for collecting and evaluating the interactive environment data, generating the environment evaluation coefficient, and transmitting it to the threshold comparison module for threshold range comparison. At the same time, the audio and video collection and processing module collects the user's spoken audio data, evaluates its misfire phenomenon, obtains the misfire judgment evaluation coefficient, and also performs threshold comparison. If the environment evaluation coefficient does not pass the threshold range comparison, the system will optimize the environment or instruct the audio and video collection and processing module to synchronously collect lip shape feature data to optimize the lip shape feature. Finally, based on the misfire judgment evaluation coefficient of the audio, the suggestion output module provides text, audio and visual error correction suggestions, thereby forming a comprehensive, dynamic and interactive oral learning feedback mechanism, which achieves the effect of improving the accuracy of judging misfires in contextual collaborative pronunciation changes.

[0030] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] like Figure 1 As shown, it is a structural schematic diagram of the English speaking emotional intelligent interactive teaching system based on deep learning provided in the embodiment of the present application. The English speaking emotional intelligent interactive teaching system based on deep learning provided in the embodiment of the present application includes: an environment detection and evaluation module, a threshold comparison module, an audio and video collection and processing module, an optimization module and a suggestion output module; the environment detection and evaluation module is used to collect and evaluate the English speaking intelligent interactive environment data, obtain the English speaking intelligent interactive environment evaluation coefficient, and transmit the English speaking intelligent interactive environment evaluation coefficient to the threshold comparison module; the threshold comparison module is used to compare the English speaking intelligent interactive environment evaluation coefficient obtained in the environment detection and evaluation module with the threshold range and the reference threshold of the image, and compare the misfire judgment evaluation coefficient of the English speaking intelligent interactive audio obtained in the audio and video collection and processing module with the threshold; the audio and video collection and processing module is used The invention is used to collect and evaluate the spoken English intelligent interaction audio data input by the user, obtain the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio, and transmit the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio to the suggestion output module; the optimization module is used to optimize the spoken English intelligent interaction environment according to the threshold range comparison result of the spoken English intelligent interaction environment evaluation coefficient, if the optimized spoken English intelligent interaction environment evaluation coefficient still does not pass the threshold range comparison, send an instruction to the audio and video collection and processing module, when collecting the spoken English intelligent interaction audio data, synchronously collect the spoken English intelligent interaction lip shape feature data, and optimize the spoken English intelligent interaction lip shape feature according to the threshold comparison result of the spoken English intelligent interaction environment evaluation coefficient; the suggestion output module is used to output text, audio and visual spoken English error correction suggestions according to the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio.

[0032] Furthermore, the threshold comparison module includes a threshold range comparison unit and an image reference threshold comparison unit; the threshold range comparison unit is used to obtain the threshold range of the English speaking intelligent interaction environment evaluation coefficient from the database, compare the English speaking intelligent interaction environment evaluation coefficient with the threshold range of the English speaking intelligent interaction environment evaluation coefficient, if the English speaking intelligent interaction environment evaluation coefficient is within the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the audio and video collection and processing module, collect and evaluate the English speaking intelligent interaction audio data input by the user, and obtain the explosion failure judgment evaluation coefficient of the English speaking intelligent interaction audio, if the English speaking intelligent interaction environment evaluation coefficient is greater than or equal to the maximum value of the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the optimization module, perform dynamic weight allocation for spatial positioning of the English speaking intelligent interaction environment, if the English speaking intelligent interaction environment evaluation coefficient is less than the minimum value of the threshold range of the English speaking intelligent interaction environment evaluation coefficient, send an instruction to the optimization module, perform frequency domain signal processing on the English speaking intelligent interaction environment. Signal processing, after dynamic weight allocation of spatial positioning or frequency domain signal processing, send instructions to the environment detection and evaluation module, re-collect and evaluate the English speaking intelligent interaction environment data, obtain the English speaking intelligent interaction environment evaluation coefficient, and transmit the English speaking intelligent interaction environment evaluation coefficient to the threshold comparison module. If the English speaking intelligent interaction environment evaluation coefficient is still not within the threshold range of the English speaking intelligent interaction environment evaluation coefficient at this time, send instructions to the audio and video collection and processing module, collect the English speaking intelligent interaction lip shape feature data synchronously when collecting the English speaking intelligent interaction audio data, combine the English speaking intelligent interaction audio data with the optimized English speaking intelligent interaction lip shape feature data for analysis, and obtain the explosion misfire judgment evaluation coefficient of the English speaking intelligent interaction audio. If the English speaking intelligent interaction environment evaluation coefficient is within the threshold range of the English speaking intelligent interaction environment evaluation coefficient at this time, send instructions to the audio and video collection and processing module, collect and evaluate the English speaking intelligent interaction audio data input by the user, and obtain the explosion misfire judgment evaluation coefficient of the English speaking intelligent interaction audio;The reference threshold comparison unit of the image is used to obtain the sharpness reference threshold of the image in the intelligent interaction of spoken English from the database, obtain the sharpness of the image in the intelligent interaction of spoken English through the sensor, compare the sharpness of the image in the intelligent interaction of spoken English with the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, if the sharpness of the image in the intelligent interaction of spoken English is less than or equal to the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the optimization module, optimize the lip shape feature data of the intelligent interaction of spoken English, combine and analyze the audio data of the intelligent interaction of spoken English with the optimized lip shape feature data of the intelligent interaction of spoken English, and obtain the evaluation coefficient of the misfire judgment of the audio of the intelligent interaction of spoken English, if the sharpness of the image in the intelligent interaction of spoken English is greater than the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the audio and video collection and processing module, combine and analyze the audio data of the intelligent interaction of spoken English with the lip shape feature data of the intelligent interaction of spoken English, and obtain the evaluation coefficient of the misfire judgment of the audio of the intelligent interaction of spoken English. ;

[0033] like Figure 2 As shown, it is a structural diagram of the threshold comparison module in the deep learning-based English oral emotion intelligent interactive teaching system provided in an embodiment of the present application.

[0034] Furthermore, the specific process of collecting the spoken English intelligent interaction environment data is as follows: collecting the spoken English intelligent interaction environment data through sensors, preprocessing the spoken English intelligent interaction environment data to obtain preprocessed spoken English intelligent interaction environment data, and performing preliminary denoising on the preprocessed spoken English intelligent interaction environment data; the spoken English intelligent interaction environment data includes the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment.

[0035] In this embodiment, the sensor measures that the signal-to-noise ratio of the user's pronunciation is 20dB, indicating that the sound quality is good. Through fast Fourier transform (FFT), the system analyzes that the noise in the environment is mainly concentrated in the low frequency band (100-300Hz), which may be generated by equipment such as air conditioners or fans. The photometric sensor detects that the current light intensity is 500lux, which is suitable for video shooting and lip recognition. At the same time, the sensor does not detect obvious stroboscopic phenomenon. The noise sound pressure level of the English speaking intelligent interaction environment, the light intensity of the English speaking intelligent interaction environment, and the color temperature of the English speaking intelligent interaction environment are normalized and converted into digital signals. The pre-processed data is analyzed, and the evaluation coefficient of the English speaking intelligent interaction environment is 0.95.

[0036] The noise sound pressure level in the English oral intelligent interaction environment refers to the decibel (dB) measurement of sound pressure relative to a reference value, which can be obtained by performing spectral analysis on the collected audio signal.

[0037] The light intensity of the English speaking intelligent interactive environment refers to the brightness level of the light in the environment, which can be measured using an optical sensor.

[0038] The color temperature of the English spoken intelligent interactive environment is used to describe the color of light and can be captured directly by the camera.

[0039] Furthermore, the specific process of collecting the English spoken intelligent interactive audio data input by the user is: collecting the English spoken intelligent interactive audio data through sensors, preprocessing the English spoken intelligent interactive audio data, and obtaining the preprocessed English spoken intelligent interactive audio data; the English spoken intelligent interactive audio data includes the phoneme combination coverage of the audio and the instantaneous energy peak of the audio.

[0040] In this embodiment, the system analysis finds that the user's pronunciation contains 20 different phonemes, accounting for 50% coverage of the 40 phonemes in the English pronunciation phoneme library. When the user pronounces the word "peak", the system detects an instantaneous energy peak of 0.85, indicating that the pronunciation intensity is high. The phoneme combination coverage of the audio and the instantaneous energy peak of the audio are normalized, and the pre-processed data is analyzed to obtain an evaluation coefficient of 0.70 for the misfire judgment of the English spoken intelligent interactive audio.

[0041] The phoneme combination coverage of audio refers to the proportion of factor combinations actually covered to all possible phoneme combinations within a preset time period, which can be directly obtained through calculation.

[0042] The instantaneous energy peak of the audio refers to the maximum energy value of the audio signal within a preset time period, which can be obtained by performing short-time energy analysis on the audio signal.

[0043] Furthermore, a specific method for obtaining the evaluation coefficient of the spoken English intelligent interaction environment is as follows: obtaining the weight factor of the noise sound pressure level of the spoken English intelligent interaction environment, the weight factor of the light intensity of the spoken English intelligent interaction environment, and the weight factor of the color temperature of the spoken English intelligent interaction environment from the database, arranging the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment in a time series, assigning weights to the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment, comparing the standard values ​​of the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment at each time series point with the real-time values ​​of the noise sound pressure level of the spoken English intelligent interaction environment, the light intensity of the spoken English intelligent interaction environment, and the color temperature of the spoken English intelligent interaction environment, and obtaining the evaluation coefficient of the spoken English intelligent interaction environment after averaging processing.

[0044] In this embodiment, the specific method for obtaining the evaluation coefficient of the English oral intelligent interaction environment is:

[0045] ;

[0046] ;

[0047] In the formula, It represents the evaluation coefficient of the English oral intelligent interaction environment, which is used to evaluate the impact of the environment on the system in the English oral intelligent interaction. Several time monitoring points are set. a, Represents the total number of time monitoring points, Indicates The noise sound pressure level of the English oral intelligent interaction environment at the monitoring point in time, The weight factor representing the noise sound pressure level in the English spoken intelligent interaction environment, It represents the noise sound pressure level of the standard English oral intelligent interaction environment. Indicates The light intensity of the English oral intelligent interactive environment at each time monitoring point, The weight factor representing the light intensity of the English spoken intelligent interaction environment, represents the light intensity of the standard English spoken intelligent interaction environment, Indicates The color temperature of the English oral intelligent interactive environment at each time monitoring point, The weight factor representing the color temperature of the English spoken intelligent interaction environment, Represents the color temperature of a standard English spoken intelligent interaction environment.

[0048] If the lighting conditions are poor, the video capture device may not be able to clearly capture the lip shape features, which will indirectly affect the synchronous analysis of the voice signal. The color temperature may cause the video screen to flicker, affecting the collection of visual information. However, the color temperature may interfere with the user's visual experience, thereby indirectly affecting the user's pronunciation.

[0049] When the system is running, the mapping table of weight factors is obtained through the database, and the corresponding weight factors are quickly extracted according to the current noise sound pressure level of the English speaking intelligent interaction environment, the light intensity of the English speaking intelligent interaction environment, and the color temperature of the English speaking intelligent interaction environment, such as the weight factor of the noise sound pressure level of the English speaking intelligent interaction environment, the weight factor of the light intensity of the English speaking intelligent interaction environment, and the weight factor of the color temperature of the English speaking intelligent interaction environment. This mapping table defines a clear set of association rules, which converts the specific values ​​of the noise sound pressure level of the English speaking intelligent interaction environment, the light intensity of the English speaking intelligent interaction environment, and the color temperature of the English speaking intelligent interaction environment into their corresponding weight factors. Under this mechanism, whether it is to achieve one-to-one precise matching or a many-to-one relationship in which multiple parameters converge into a single weight, the dynamic acquisition of weight factors can be effectively realized.

[0050] In a specific embodiment, an example of evaluation coefficient data of the English spoken intelligent interaction environment is shown in the following table.

[0051] Table 1 Example table of evaluation coefficients of oral English intelligent interactive environment

[0052]

[0053] When the weighting factor of the noise sound pressure level of the English speaking intelligent interaction environment, the weighting factor of the light intensity of the English speaking intelligent interaction environment, and the weighting factor of the color temperature of the English speaking intelligent interaction environment are 0.2, 0.4, and 0.4 respectively, the standard values ​​of the noise sound pressure level of the English speaking intelligent interaction environment, the light intensity of the English speaking intelligent interaction environment, and the color temperature of the English speaking intelligent interaction environment are 35dB(A), 400lx, and 6000K respectively. Figure 3 It can be seen from the data in Table 1 that when the light intensity and color temperature of the English speaking intelligent interaction environment are fixed, the noise sound pressure level of the English speaking intelligent interaction environment and the English speaking intelligent interaction environment evaluation coefficient are greater.

[0054] Furthermore, a specific method for obtaining the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is as follows: if it is necessary to combine the lip feature data, obtain the weight factor of the lip opening and closing distance, the weight factor of the audio phoneme combination coverage rate and the weight factor of the audio instantaneous energy peak value from the database, assign weights to the deviation value of the lip opening and closing distance from the lip opening and closing distance reference value, the audio phoneme combination coverage rate and the audio instantaneous energy peak value, sort the lip opening and closing distance, the audio phoneme combination coverage rate and the audio instantaneous energy peak value according to the number of phonemes, and calculate the weight factor of the lip opening and closing distance, the phoneme combination coverage rate and the audio instantaneous energy peak value. The phoneme combination coverage rate of the frequency and the instantaneous energy peak value of the audio are combined and processed to obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio; if it is not necessary to combine the lip features, the weight factor of the phoneme combination coverage rate of the audio and the weight factor of the instantaneous energy peak value of the audio are obtained from the database, weights are assigned to the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio, the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio are sorted according to the number of phonemes, the phoneme combination coverage rate of the audio and the instantaneous energy peak value of the audio are combined and processed to obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio.

[0055] In this embodiment, the specific method for obtaining the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is:

[0056] When there is no need to combine lip feature data,

[0057] ;

[0058] ;

[0059] When it is necessary to combine lip feature data,

[0060] ;

[0061] ;

[0062] In the formula, It represents the evaluation coefficient of the detonation judgment of the intelligent interactive audio of spoken English, which is used to evaluate the load of the software during rendering and set several sentence monitoring points. b, Indicates the total number of statement monitoring points, Indicates The lip opening and closing distance under each sentence monitoring point, Indicates the standard value of the lip opening and closing distance. The weight factor representing the lip opening and closing distance, Indicates The phoneme combination coverage of the audio under the sentence monitoring point, The weight factor representing the phoneme combination coverage of the audio, Indicates The instantaneous energy peak of the audio under the monitoring point of each sentence, A weighting factor representing the instantaneous energy peak of the audio.

[0063] If the instantaneous energy peak in the audio is too high, it may cause sound distortion, reduce the clarity and intelligibility of the phonemes, and thus affect the recognition of the coverage of the phoneme combination. The position of the instantaneous energy peak in the dynamic range of the audio (that is, the difference between the quietest and loudest sounds) affects the perception of the phoneme combination. If the dynamic range is too large, some phonemes may become invisible next to strong sounds.

[0064] When the system is running, the mapping table of weight factors is obtained through the database, and the weight factors of the lip opening and closing distance, the phoneme combination coverage of the audio, and the instantaneous energy peak of the audio are quickly calculated based on the current lip opening and closing distance, the phoneme combination coverage of the audio, and the weight factor of the instantaneous energy peak of the audio. This mapping table defines a clear set of association rules, which converts the specific values ​​of the lip opening and closing distance, the phoneme combination coverage of the audio, and the instantaneous energy peak of the audio into their corresponding weight factors. Under this mechanism, whether it is to achieve a one-to-one precise match or a many-to-one relationship in which multiple parameters are aggregated into a single weight, the dynamic acquisition of weight factors can be effectively achieved.

[0065] Furthermore, the specific optimization steps for optimizing the spoken English intelligent interaction environment according to the comparison results of the threshold range of the spoken English intelligent interaction environment evaluation coefficient are as follows: if the spoken English intelligent interaction environment evaluation coefficient is greater than or equal to the maximum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, send an instruction to the optimization module to perform dynamic weight allocation for spatial positioning of the spoken English intelligent interaction environment; if the spoken English intelligent interaction environment evaluation coefficient is less than the minimum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, send an instruction to the optimization module to perform frequency domain signal processing on the spoken English intelligent interaction environment.

[0066] In this embodiment, the specific steps of dynamic weight allocation for spatial positioning are: using a microphone array to monitor the location of the sound source in the environment, and formulating a weight allocation strategy through a beamforming algorithm according to the location of the sound source. For example, if the sound source is close to a microphone, the weight of the microphone can be set higher, and different weights are applied to the signals received by each microphone, and the weighted signals are synthesized to obtain an optimized audio signal. If the location of the sound source changes, the system will re-localize the sound source and re-allocate the weight to maintain the best audio quality.

[0067] The specific steps of frequency domain signal processing are: collect audio signals in the environment, perform fast Fourier transform on the collected audio signals, convert time domain signals into frequency domain signals, analyze frequency domain signals, identify the frequency components of noise and user voice, apply interference algorithms such as notch filters and adaptive filters to suppress or eliminate noise frequency components, perform inverse Fourier transform on the processed frequency domain signals, convert them back to time domain signals, and output audio signals after frequency domain interference processing, which have a lower noise level.

[0068] Furthermore, the specific steps of synchronously collecting the lip shape feature data of spoken English intelligent interaction while collecting the audio data of spoken English intelligent interaction are as follows: obtaining the lip shape feature data of spoken English intelligent interaction through a depth sensor, extracting the lip shape feature data of spoken English intelligent interaction through a deep learning model, and synchronizing the audio and video using hardware triggering or software triggering; the lip shape feature data of spoken English intelligent interaction refers to the distance between the lips; hardware triggering includes using a synchronization signal generator to send a pulse signal to simultaneously trigger the audio and video devices; software triggering includes achieving timestamp alignment of audio and video streams through the NDI protocol.

[0069] In this embodiment, the depth sensor is a device that can measure the distance between an object and its surface. It is usually used to capture three-dimensional spatial information. In this system, it is used to obtain the user's lip shape data.

[0070] When the user is speaking in English, the depth sensor captures the user's lip movement in real time, records the 3D position data of the lips, and preprocesses the collected lip shape data, including denoising, alignment, and normalization. The long short-term memory network is used to process the time series data. VGG16 is selected as the deep learning model for feature extraction. The fully connected layer of the model is removed, and the convolution layer and pooling layer are retained. The preprocessed lip image frame is input into the VGG16 model. By analyzing the feature map output by the convolution layer, the upper and lower boundaries of the lips can be located. The pixel distance between the upper and lower boundaries is extracted as the lip opening and closing distance.

[0071] VGG16 refers to a convolutional neural network model that contains 16 weight layers, 13 of which are convolutional layers, 5 are maximum pooling layers, 3 are fully connected layers, and the last one is a softmax layer for classification.

[0072] The photos of different lip opening and closing distances in the historical data are divided into training set and verification set according to a preset ratio. The training set is used to train VGG16, and then the verification set is used to verify VGG16. After training and verification, the lip photos of different opening and closing distances collected by the audio and video collection and processing module are input, and the lip opening and closing distance data is output.

[0073] Hardware triggering refers to using a sync signal generator to send a pulse signal to ensure that the audio and video devices start recording at the same time. Software triggering refers to aligning the timestamps of the audio and video streams through the NDI (Network Device Interface) protocol or other audio and video synchronization protocols.

[0074] Furthermore, a specific output method of outputting text, audio and visual English spoken error correction suggestions according to the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is as follows:

[0075] According to the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English, predefined text, audio and visual spoken English error correction suggestions are matched from the database. The text, audio and visual spoken English error correction suggestions include displaying the spectrogram of the user's pronunciation and the standard pronunciation, playing back the user's spoken English recording and highlighting the error parts, pointing out the type of pronunciation error and providing correct text pronunciation examples.

[0076] In this embodiment, for example, when the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English is less than 0.5, the text, audio, and visualized spoken English error correction suggestions that fail to match are matched; when the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English is greater than or equal to 0.5 and less than 1.0, the text, audio, and visualized spoken English error correction suggestions that are well matched are matched; when the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English is greater than or equal to 1.0, the text, audio, and visualized spoken English error correction suggestions that are excellent are matched.

[0077] Furthermore, the deep learning-based emotional intelligent interactive teaching method for spoken English is as follows: collect and evaluate the spoken English intelligent interactive environment data to obtain the spoken English intelligent interactive environment evaluation coefficient, and transmit the spoken English intelligent interactive environment evaluation coefficient to the threshold comparison module; compare the spoken English intelligent interactive environment evaluation coefficient obtained in the environment detection and evaluation module with the threshold range and the reference threshold of the image, and compare the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio obtained in the audio and video collection and processing module with the threshold; collect and evaluate the spoken English intelligent interactive audio data input by the user to obtain the misfire judgment evaluation coefficient of the spoken English intelligent interactive audio, and transmit the spoken English intelligent interactive environment evaluation coefficient to the threshold comparison module. The misfire judgment evaluation coefficient of the spoken English intelligent interaction audio is transmitted to the suggestion output module; the spoken English intelligent interaction environment is optimized according to the threshold range comparison result of the evaluation coefficient of the spoken English intelligent interaction environment; if the optimized spoken English intelligent interaction environment evaluation coefficient still fails to pass the threshold range comparison, a command is sent to the audio and video collection and processing module to synchronously collect the spoken English intelligent interaction lip shape feature data when collecting the spoken English intelligent interaction audio data, and optimize the spoken English intelligent interaction lip shape features according to the threshold comparison result of the spoken English intelligent interaction environment evaluation coefficient; text, audio and visual spoken English error correction suggestions are output according to the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio.

[0078] like Figure 4 As shown, it is a schematic diagram of the deep learning-based emotional intelligent interactive teaching method for spoken English provided in an embodiment of the present application.

[0079] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0084] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. The English spoken emotion intelligent interactive teaching system based on deep learning is characterized by: It includes environment detection and assessment module, threshold comparison module, audio and video collection and processing module, optimization module and suggestion output module; The environment detection and evaluation module is used to collect and evaluate the spoken English intelligent interaction environment data, obtain the spoken English intelligent interaction environment evaluation coefficient, and transmit the spoken English intelligent interaction environment evaluation coefficient to the threshold comparison module; The threshold comparison module is used to compare the English spoken intelligent interaction environment evaluation coefficient obtained in the environment detection and evaluation module with the threshold range and the reference threshold of the image, and to compare the misfire judgment evaluation coefficient of the English spoken intelligent interaction audio obtained in the audio and video collection and processing module with the threshold; The audio and video collection and processing module is used to collect and evaluate the English spoken intelligent interactive audio data input by the user, obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio, and transmit the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio to the suggestion output module; The optimization module is used to optimize the spoken English intelligent interaction environment according to the comparison result of the threshold range of the spoken English intelligent interaction environment evaluation coefficient. If the optimized spoken English intelligent interaction environment evaluation coefficient still fails to pass the threshold range comparison, send a command to the audio and video collection and processing module to collect the spoken English intelligent interaction lip shape feature data synchronously when collecting the spoken English intelligent interaction audio data, and optimize the spoken English intelligent interaction lip shape feature according to the comparison result of the threshold range of the spoken English intelligent interaction environment evaluation coefficient; The suggestion output module is used to output text, audio and visual English spoken error correction suggestions according to the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio.

2. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The threshold comparison module includes a threshold range comparison unit and an image reference threshold comparison unit; The threshold range comparison unit is used to obtain the threshold range of the oral English intelligent interaction environment evaluation coefficient from the database, compare the oral English intelligent interaction environment evaluation coefficient with the threshold range of the oral English intelligent interaction environment evaluation coefficient, if the oral English intelligent interaction environment evaluation coefficient is within the threshold range of the oral English intelligent interaction environment evaluation coefficient, send a command to the audio and video collection and processing module, collect and evaluate the oral English intelligent interaction audio data input by the user, and obtain the misfire judgment evaluation coefficient of the oral English intelligent interaction audio, if the oral English intelligent interaction environment evaluation coefficient is greater than or equal to the maximum value of the threshold range of the oral English intelligent interaction environment evaluation coefficient, send a command to the optimization module, perform dynamic weight allocation for spatial positioning of the oral English intelligent interaction environment, if the oral English intelligent interaction environment evaluation coefficient is less than the minimum value of the threshold range of the oral English intelligent interaction environment evaluation coefficient, send a command to the optimization module, perform frequency domain signal processing on the oral English intelligent interaction environment, in the dynamic weight allocation or spatial positioning of the optimization module. After the frequency domain signal is processed, a command is sent to the environment detection and evaluation module to collect and evaluate the spoken English intelligent interaction environment data again to obtain the spoken English intelligent interaction environment evaluation coefficient, and the spoken English intelligent interaction environment evaluation coefficient is transmitted to the threshold comparison module. If the spoken English intelligent interaction environment evaluation coefficient is still not within the threshold range of the spoken English intelligent interaction environment evaluation coefficient at this time, a command is sent to the audio and video collection and processing module to synchronously collect the spoken English intelligent interaction lip shape feature data when collecting the spoken English intelligent interaction audio data, and the spoken English intelligent interaction audio data is combined with the optimized spoken English intelligent interaction lip shape feature data for analysis to obtain the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio. If the spoken English intelligent interaction environment evaluation coefficient is within the threshold range of the spoken English intelligent interaction environment evaluation coefficient at this time, a command is sent to the audio and video collection and processing module to collect and evaluate the spoken English intelligent interaction audio data input by the user to obtain the misfire judgment evaluation coefficient of the spoken English intelligent interaction audio. The reference threshold comparison unit of the image is used to obtain the sharpness reference threshold of the image in the intelligent interaction of spoken English from the database, obtain the sharpness of the image in the intelligent interaction of spoken English through the sensor, compare the sharpness of the image in the intelligent interaction of spoken English with the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, if the sharpness of the image in the intelligent interaction of spoken English is less than or equal to the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the optimization module, optimize the lip shape feature data of the intelligent interaction of spoken English, combine and analyze the audio data of the intelligent interaction of spoken English with the optimized lip shape feature data of the intelligent interaction of spoken English, and obtain the misfire judgment evaluation coefficient of the audio of the intelligent interaction of spoken English; if the sharpness of the image in the intelligent interaction of spoken English is greater than the reference threshold of the sharpness of the image in the intelligent interaction of spoken English, send an instruction to the audio and video collection and processing module, combine and analyze the audio data of the intelligent interaction of spoken English with the lip shape feature data of the intelligent interaction of spoken English, and obtain the misfire judgment evaluation coefficient of the audio of the intelligent interaction of spoken English.

3. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The specific process of collecting the English spoken intelligent interactive environment data is as follows: Collecting spoken English intelligent interaction environment data through sensors, preprocessing the spoken English intelligent interaction environment data to obtain preprocessed spoken English intelligent interaction environment data, and performing preliminary noise removal on the preprocessed spoken English intelligent interaction environment data; The English speaking intelligent interaction environment data includes the noise sound pressure level of the English speaking intelligent interaction environment, the light intensity of the English speaking intelligent interaction environment, and the color temperature of the English speaking intelligent interaction environment.

4. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The specific process of collecting the English spoken intelligent interactive audio data input by the user is as follows: The English spoken intelligent interactive audio data is collected by a sensor, and the English spoken intelligent interactive audio data is preprocessed to obtain the preprocessed English spoken intelligent interactive audio data; The English spoken intelligent interactive audio data includes the phoneme combination coverage of the audio and the instantaneous energy peak of the audio.

5. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The specific method for obtaining the evaluation coefficient of the English oral intelligent interactive environment is: The weight factors of the noise sound pressure level, the light intensity and the color temperature of the spoken English intelligent interaction environment are obtained from the database, the noise sound pressure level, the light intensity and the color temperature of the spoken English intelligent interaction environment are arranged in time series, weights are assigned to the noise sound pressure level, the light intensity and the color temperature of the spoken English intelligent interaction environment, the standard values ​​of the noise sound pressure level, the light intensity and the color temperature of the spoken English intelligent interaction environment at each time series point are compared with the real-time values ​​of the noise sound pressure level, the light intensity and the color temperature of the spoken English intelligent interaction environment, and the evaluation coefficient of the spoken English intelligent interaction environment is obtained after averaging.

6. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The specific method for obtaining the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio is: If it is necessary to combine the lip feature data, obtain the weight factor of the lip opening and closing distance, the weight factor of the phoneme combination coverage rate of the audio, and the weight factor of the instantaneous energy peak of the audio from the database, assign weights to the deviation value of the lip opening and closing distance from the lip opening and closing distance reference value, the phoneme combination coverage rate of the audio, and the instantaneous energy peak value of the audio, sort the lip opening and closing distance, the phoneme combination coverage rate of the audio, and the instantaneous energy peak value of the audio according to the number of phonemes, combine and process the deviation value of the lip opening and closing distance from the lip opening and closing distance reference value, the phoneme combination coverage rate of the audio, and the instantaneous energy peak value of the audio, and obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio; If it is not necessary to combine lip features, obtain the weight factor of the phoneme combination coverage rate of the audio and the weight factor of the instantaneous energy peak of the audio from the database, assign weights to the phoneme combination coverage rate of the audio and the instantaneous energy peak of the audio, sort the phoneme combination coverage rate of the audio and the instantaneous energy peak of the audio according to the number of phonemes, combine and process the phoneme combination coverage rate of the audio and the instantaneous energy peak of the audio to obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio.

7. The deep learning-based English oral emotional intelligent interactive teaching system according to claim 1, characterized in that: The specific optimization steps for optimizing the spoken English intelligent interaction environment according to the comparison result of the threshold range of the spoken English intelligent interaction environment evaluation coefficient are as follows: If the evaluation coefficient of the spoken English intelligent interaction environment is greater than or equal to the maximum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, an instruction is sent to the optimization module to perform dynamic weight allocation for spatial positioning of the spoken English intelligent interaction environment; if the evaluation coefficient of the spoken English intelligent interaction environment is less than the minimum value of the threshold range of the spoken English intelligent interaction environment evaluation coefficient, an instruction is sent to the optimization module to perform frequency domain signal processing on the spoken English intelligent interaction environment.

8. The deep learning-based English oral emotional intelligent interactive teaching system as claimed in claim 1, characterized in that: The specific steps of synchronously collecting the English spoken intelligent interactive lip shape feature data when collecting the English spoken intelligent interactive audio data are as follows: Acquire English spoken intelligent interactive lip shape data through depth sensors, extract English spoken intelligent interactive lip shape feature data through deep learning models, and use hardware triggers or software triggers to synchronize audio and video; The English spoken intelligent interactive lip shape feature data includes the lip opening and closing distance; Hardware triggering involves using a sync signal generator to send a pulse signal to trigger both audio and video devices simultaneously; Software triggering includes aligning audio and video stream timestamps through the NDI protocol.

9. The English spoken language emotional intelligence interactive teaching system based on deep learning as claimed in claim 1, characterized in that: The specific output method of outputting text, audio and visual English spoken error correction suggestions according to the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio: According to the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English, predefined text, audio and visual spoken English error correction suggestions are matched from the database. The text, audio and visual spoken English error correction suggestions include displaying the spectrogram of the user's pronunciation and the standard pronunciation, playing back the user's spoken English recording and highlighting the error parts, pointing out the type of pronunciation error and providing correct text pronunciation examples.

10. The method applied to the deep learning-based English spoken emotional intelligence interactive teaching system according to any one of claims 1 to 9, characterized in that: Collect and evaluate the spoken English intelligent interaction environment data, obtain the spoken English intelligent interaction environment evaluation coefficient, and transmit the spoken English intelligent interaction environment evaluation coefficient to the threshold comparison module; The English spoken intelligent interactive environment evaluation coefficient obtained in the environment detection and evaluation module is compared with the threshold range and the reference threshold of the image, and the English spoken intelligent interactive audio misfire judgment evaluation coefficient obtained in the audio and video collection and processing module is compared with the threshold; Collect and evaluate the English spoken intelligent interactive audio data input by the user, obtain the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio, and transmit the misfire judgment evaluation coefficient of the English spoken intelligent interactive audio to the suggestion output module; The English speaking intelligent interaction environment is optimized according to the comparison result of the threshold range of the evaluation coefficient of the English speaking intelligent interaction environment. If the optimized evaluation coefficient of the English speaking intelligent interaction environment still fails to pass the comparison of the threshold range, a command is sent to the audio and video collection and processing module to collect the English speaking intelligent interaction lip shape feature data synchronously when collecting the English speaking intelligent interaction audio data, and the English speaking intelligent interaction lip shape feature is optimized according to the comparison result of the threshold range of the English speaking intelligent interaction environment evaluation coefficient; According to the misfire judgment evaluation coefficient of the intelligent interactive audio of spoken English, text, audio and visual English speaking error correction suggestions are output.

Images