A method and system for training eloquence based on biosensing and multimodal feedback

Abstract

The present invention provides an eloquence training method and system based on biosensing and multimodal feedback, the method comprising: acquiring user physiological data, user voice data and user video data; analyzing the user voice data by a preset integrated processor to obtain a first evaluation result; inputting the user voice data into a preset multi-layer network and a multi-layer perceptron fusion to obtain a second evaluation result; inputting the user physiological data, user voice data and user video data into a preset neural network and a multimodal deep fusion network to obtain a third evaluation result; combining the first evaluation result, the second evaluation result and the third evaluation result to generate personalized eloquence suggestions; according to the personalized eloquence suggestions, using virtual reality technology to guide users to perform eloquence training; improving the eloquence training effect by providing users with scientific, comprehensive and accurate eloquence evaluation results and a rich and realistic eloquence training environment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an eloquence training method and system based on biosensing and multi-modal feedback. Background Art

[0002] Oratory ability is an important manifestation of personal ability, and providing users with scientific, accurate and convenient eloquence training has gradually become an important direction.

[0003] Existing eloquence training technologies lack the support of deep learning algorithms and personalized algorithms, and are unable to provide accurate analysis and personalized feedback for users' eloquence performance, resulting in unsatisfactory eloquence training results. At the same time, existing technologies are often limited to traditional eloquence training environments, such as paper-based materials or simple software applications, and are unable to provide a rich and realistic training experience, limiting the improvement of eloquence training results. Summary of the invention

[0004] The present invention provides an eloquence training method and system based on biosensing and multi-modal feedback, which provide users with scientific, comprehensive and accurate eloquence evaluation results and a rich and realistic eloquence training environment, thereby improving the eloquence training effect.

[0005] In order to solve the above technical problems, the present invention provides an eloquence training method based on biosensing and multimodal feedback, comprising:

[0006] Obtain user physiological data, user voice data and user video data;

[0007] Analyzing the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result, and a sound feature evaluation result;

[0008] Inputting user voice data into a preset multi-layer network, outputting sound spectrum features, time series features and semantic relationship features, and using a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features to obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network;

[0009] Inputting the user's physiological data, the user's voice data and the user's video data into a preset neural network, outputting voice features, visual features and text features, and using a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network;

[0010] Combining the first evaluation result, the second evaluation result and the third evaluation result, generating personalized eloquence suggestions;

[0011] According to the personalized eloquence suggestions, virtual reality technology is used to guide users to perform eloquence training.

[0012] The present invention collects user physiological data, user voice data and user video data in real time, providing scientific and comprehensive dynamic data support for eloquence training; uses a preset integrated processor to perform preliminary analysis on the data to quickly screen and extract key voice information; uses a preset multi-layer network to perform more complex and in-depth data analysis and feature extraction on the data to obtain evaluation results for multiple speech dimensions; uses a preset neural network to extract features of the data on eloquence speech vision, speech hearing and speech content, and fuses multiple types of features to obtain deeper evaluation results; generates personalized eloquence suggestions based on multiple evaluation results, thereby providing users with scientific, comprehensive and accurate eloquence evaluation results and personalized feedback suggestions, thereby improving the eloquence training effect; provides users with a training environment based on virtual reality technology, thereby enriching the eloquence training environment and improving the eloquence training effect.

[0013] Furthermore, the obtaining of user physiological data, user voice data and user video data includes:

[0014] Collecting user physiological data through biosensors; wherein the user physiological data includes user visual information, user facial expression information, user skin resistance information and user heart rate information;

[0015] Use array microphone technology to collect user voice data;

[0016] Use video acquisition technology to collect user video data.

[0017] Furthermore, collecting the user's physiological data through the biosensor includes:

[0018] Collect user visual information through eye tracking sensors;

[0019] Collect user's facial expression information through facial electromyography sensor;

[0020] Collect user's skin resistance information through skin conductivity meter;

[0021] The user's heart rate information is collected through a heart rate detector.

[0022] The present invention collects user physiological data through a variety of biosensors and combines user voice data and user video data to provide scientific and comprehensive data support for eloquence training, thereby improving the accuracy of eloquence training and facilitating personalized training feedback analysis.

[0023] Further, the analyzing the user's physiological data and the user's voice data by a preset integrated processor to obtain a first evaluation result includes:

[0024] Using wireless transmission to transmit the user's physiological data and the user's voice data to a preset integrated processor;

[0025] Analyze the user's physiological data and the user's voice data through a preset integrated processor to obtain fluency evaluation results, logic evaluation results, emotional diversity evaluation results, and voice feature evaluation results;

[0026] A first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results.

[0027] Furthermore, the user's voice data is analyzed by a preset integrated processor to obtain a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result and a sound feature evaluation result, including:

[0028] The duration and number of words of the user's voice data are analyzed by a preset integrated processor to obtain a fluency evaluation result; wherein the fluency evaluation result is specifically as follows:

[0029]

[0030] In the formula, F L Indicates the fluency assessment result; T s represents the duration of user voice data, W i represents the number of words in the i-th sentence in the user's voice data, T i represents the duration of the i-th sentence in the user's voice data, and N represents the number of speech paragraphs in the user's voice data;

[0031] The logical coherence and logical relevance of the user voice data are analyzed by a preset integrated processor to obtain a logical evaluation result; wherein the logical evaluation result is specifically as follows:

[0032]

[0033] Where, L c Indicates the result of logical evaluation; N s represents the number of sentences in the user's voice data, P j Represents the probability of logical relevance between the jth sentence and the context in the user's voice data;

[0034] The preset integrated processor analyzes the emotional state type and the emotional state occurrence probability of the user's voice data to obtain an emotional diversity evaluation result; wherein the emotional diversity evaluation result is specifically as follows:

[0035]

[0036] In the formula, E D represents the result of emotion diversity evaluation; M represents the number of emotion states, q k represents the probability of the kth emotional state occurring;

[0037] The sound amplitude and sound frequency of the user's voice data are analyzed by a preset integrated processor to obtain a sound feature evaluation result; wherein the sound feature evaluation result is specifically as follows:

[0038] V F =α·RMS(A t )+β·Var(F t )

[0039] Where V F Indicates the sound characteristic evaluation result; RMS (A t ) represents the root mean square value of the sound amplitude in the time series t, and α represents RMS(A t ) weight coefficient, Var(F t ) represents the variance of the sound frequency in the time series t, and β represents Var(F t ) is the weight coefficient of .

[0040] Further, the first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results, including:

[0041] A first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results; wherein the first evaluation result is specifically as follows:

[0042] P F =ω1·F L +ω2·L C +ω3·E D +ω4·V F

[0043] Where P F Indicates the first evaluation result; F L represents the fluency evaluation result, ω1 represents the weight coefficient of the fluency evaluation result; L c represents the logical evaluation result, ω2 represents the weight coefficient of the logical evaluation result; ED represents the evaluation result of sentiment diversity, ω3 represents the weight coefficient of the evaluation result of sentiment diversity; V F represents the sound feature evaluation result, and ω4 represents the weight coefficient of the sound feature evaluation result.

[0044] The present invention utilizes an embedded system to integrate a variety of processors, and uses the integrated processors to perform preliminary analysis on data to quickly screen and extract key voice information, thereby completing an evaluation of the user's speech expressiveness.

[0045] Furthermore, the user voice data is input into a preset multi-layer network, sound spectrum features, time series features and semantic relationship features are output, and the sound spectrum features, time series features and semantic relationship features are fused using a multi-layer perceptron to obtain a second evaluation result, including:

[0046] Input user voice data into an advanced convolutional neural network and output sound spectrum features;

[0047] Performing time series processing on the user's voice data to obtain time series data of the voice, and inputting the time series data of the voice into a bidirectional long short-term memory network to output time series features;

[0048] Converting user voice data into graph structure data of voice, and inputting the graph structure data of voice into a graph convolutional neural network to output semantic relationship features;

[0049] The sound spectrum features, time series features and semantic relationship features are input into the multi-layer perceptron for feature fusion and feature classification, and the comprehensive feature set and the second evaluation result are output.

[0050] Furthermore, the user voice data is input into the advanced convolutional neural network to output the sound spectrum features, including:

[0051] The user voice data is input into the advanced convolutional neural network, and the sound spectrum features are output; wherein the advanced convolutional neural network is specifically as follows:

[0052]

[0053] In the formula, Represents the output feature map of the cth layer of the advanced convolutional neural network at position (i, j); represents the convolution kernel weight of the cth layer of the advanced convolutional neural network, S (c―1) Represents the input spectrum feature map of the c-1 layer of the advanced convolutional neural network, B (c) Represents the bias term of the c-th layer of the high-level convolutional neural network.

[0054] Furthermore, the user voice data is subjected to time series processing to obtain time series data of the voice, and the time series data of the voice is input into a bidirectional long short-term memory network to output time series features, including:

[0055] The user's voice data is processed in time series to obtain the time series data of the voice, and the time series data of the voice is input into a bidirectional long short-term memory network to output time series features; wherein the bidirectional long short-term memory network is specifically as follows:

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] In the formula, represents the forget gate of the bidirectional long short-term memory network, represents the input gate of the bidirectional long short-term memory network, represents the output gate of the bidirectional long short-term memory network, Indicates the cell state, Indicates hidden state; and represents the weight matrix; and Represents the bias term.

[0063] Furthermore, the method of converting the user voice data into graph structure data of the voice, and inputting the graph structure data of the voice into a graph convolutional neural network to output semantic relationship features includes:

[0064] The user voice data is converted into graph structure data of the voice, and the graph structure data of the voice is input into the graph convolutional neural network to output the semantic relationship features; wherein the graph convolutional neural network is specifically as follows:

[0065]

[0066] In the formula, R (n) Represents the output graph convolution of the nth layer of the graph convolutional neural network; represents the angle matrix, represents the adjacency matrix A plus self-connection, X (n―1)represents the input features of the n-1th layer of the graph convolutional neural network, W (n) Represents the weight matrix of the nth layer of the graph convolutional neural network.

[0067] Furthermore, the sound spectrum features, time series features and semantic relationship features are input into a multi-layer perceptron for feature fusion and feature classification, and a comprehensive feature set and a second evaluation result are output, including:

[0068] The sound spectrum features, time series features and semantic relationship features are input into a multi-layer perceptron for feature fusion and feature classification, and a comprehensive feature set and a second evaluation result are output; wherein the multi-layer perceptron is specifically as follows:

[0069]

[0070] In the formula, represents the kth output node of the qth layer of the multilayer perceptron; represents the weight of the qth layer of the multilayer perceptron, represents the rth fusion feature of the qth layer of the multilayer perceptron, Represents the bias term of the qth layer of the multilayer perceptron.

[0071] The present invention uses a preset multi-layer network to process user voice data to perform deep feature extraction to obtain sound spectrum features, time series features and semantic relationship features. By fusing multiple dimensional features of sound spectrum, time series and semantic relationship, a more comprehensive and accurate eloquence evaluation is provided to the user. Based on the deep feature relationship obtained by learning a large amount of data, a multi-layer perceptron is used to fuse multiple dimensional features to improve the accuracy and scientificity of eloquence evaluation.

[0072] Furthermore, the user physiological data, user voice data and user video data are input into a preset neural network, voice features, visual features and text features are output, and the voice features, visual features and text features are fused using a multimodal deep fusion network to obtain a third evaluation result, including:

[0073] Input the user's physiological data and user's video data into the improved convolutional neural network for feature extraction to obtain visual features;

[0074] Input the user's voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features;

[0075] A multimodal deep fusion network is used to fuse speech features, visual features and text features to obtain a third evaluation result; wherein the multimodal deep fusion network includes a multimodal feature conversion layer, an attention adjustment layer, a multimodal deep fusion layer and an eloquence dimension analysis layer.

[0076] Furthermore, the user physiological data and the user video data are input into the improved convolutional neural network for feature extraction to obtain visual features, including:

[0077] Input the user's physiological data and the user's video data into the improved convolutional neural network for feature extraction to obtain visual features; wherein the improved convolutional neural network is specifically as follows:

[0078]

[0079] In the formula, Represents the output feature map of the mth convolutional layer of the improved convolutional neural network; LeakyReLU() represents the activation function, represents the convolution kernel weight of the improved convolutional neural network, Q (m―1) Denotes the input feature map of the m-1th layer of the improved convolutional neural network, D (m) Represents the bias term of the improved convolutional neural network.

[0080] Furthermore, the user voice data is input into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features, including:

[0081] The user voice data is input into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; wherein the multi-layer long short-term memory network is specifically as follows:

[0082]

[0083]

[0084] In the formula, represents the input gate of a multi-layer long short-term memory network, Represents the forget gate of a multi-layer long short-term memory network; and represents the weight matrix, represents the hidden state at the previous moment, Represents the input at the current moment, and Represents the bias term.

[0085] Furthermore, the multimodal deep fusion network is used to fuse speech features, visual features and text features to obtain a third evaluation result, including:

[0086] Using a multimodal feature conversion layer to perform feature conversion on speech features, visual features, and text features to obtain a first feature;

[0087] Based on the dynamic attention mechanism, an attention adjustment layer is used to calculate the attention weights of speech features, visual features, and text features, and weighted feature fusion is performed on the first features according to the attention weights to obtain the second features;

[0088] Using a multimodal deep fusion layer to deeply fuse the second feature to obtain a third feature;

[0089] The eloquence dimension analysis layer is used to perform feature analysis on the third feature to obtain a third evaluation result.

[0090] Furthermore, the method of using a multimodal feature conversion layer to perform feature conversion on speech features, visual features, and text features to obtain a first feature includes:

[0091] The multimodal feature conversion layer is used to perform feature conversion on the speech feature, the visual feature and the text feature to obtain a first feature; wherein the first feature is specifically as follows:

[0092] F trans =LeakyReLU(W trans concat(F v ,F a ,F t )+b trans )

[0093] In the formula, F trans represents the first feature after feature conversion; W trans represents the weight of the multimodal feature conversion layer, b trans represents the bias term of the multimodal feature conversion layer, F v represents the visual feature, F a represents the speech feature, F t Represents text features.

[0094] Furthermore, the dynamic attention mechanism is based on which the attention weights of speech features, visual features and text features are calculated using the attention adjustment layer, and the first features are weightedly fused according to the attention weights to obtain the second features, including:

[0095] Based on the dynamic attention mechanism, an attention adjustment layer is used to calculate the attention weights of speech features, visual features, and text features, and weighted feature fusion is performed on the first features according to the attention weights to obtain the second features;

[0096] The attention weight is as follows:

[0097] β=tanh(W att ·F trans +b att )

[0098] In the formula, β represents the attention weight; F trans represents the first feature, W att represents the weight of the attention adjustment layer, b att Represents the bias term of the attention adjustment layer;

[0099] The second feature is as follows:

[0100] F att =β⊙F trans

[0101] In the formula, F att represents the second feature after weighted feature fusion; β represents the attention weight, F trans Indicates the first feature.

[0102] Further, the using a multimodal deep fusion layer to deeply fuse the second feature to obtain a third feature includes:

[0103] The second feature is deeply fused using a multimodal deep fusion layer to obtain a third feature; wherein the third feature is specifically as follows:

[0104] F deep =Sigmoid(W deep ·F att +b deep )

[0105] In the formula, F deep represents the third feature after deep fusion; Sigmoid() represents the activation function, W deep represents the weight of the multimodal deep fusion layer, F att represents the second feature, b deep Represents the bias term of the multimodal deep fusion layer.

[0106] Further, the using the eloquence dimension analysis layer to perform feature analysis on the third feature to obtain a third evaluation result includes:

[0107] The third feature is analyzed using the eloquence dimension analysis layer to obtain a third evaluation result; wherein the third evaluation result is specifically as follows:

[0108] R dim =ELU(W dim ·F deep +b dim )

[0109] In the formula, R dim represents the third evaluation result output by the eloquence dimension analysis layer; ELU() represents the activation function, W dim represents the weight of the eloquence dimension analysis layer, Fdeep represents the third feature, b dim Represents the bias term of the analysis layer of the eloquence dimension.

[0110] The present invention uses a preset neural network to perform in-depth analysis of user physiological data, user voice data and user video data to obtain voice features, visual features and text features, and uses a multimodal deep fusion network to perform cross-modal fusion of multiple features, which can extract deeper correlations between different types of features and avoid the evaluation result bias caused by a single data source, thereby improving the accuracy of eloquence evaluation. At the same time, it can also adjust the weights and bias items in real time according to the real-time nature of the data to achieve dynamic adjustment of the eloquence training evaluation method, thereby ensuring the accuracy, scientificity and comprehensiveness of the eloquence evaluation.

[0111] Furthermore, the method of using virtual reality technology to guide the user to perform eloquence training according to the personalized eloquence suggestion includes:

[0112] Use virtual reality technology to build a user's eloquence training environment;

[0113] Using augmented reality technology to visualize the personalized eloquence suggestions;

[0114] Based on the first evaluation result, the second evaluation result and the third evaluation result, the personalized eloquence suggestion is updated using a machine learning algorithm

[0115] The present invention uses virtual reality technology and augmented reality technology to superimpose personalized eloquence suggestions in the user's actual field of vision, thereby providing the user with a rich and realistic eloquence training environment and improving the eloquence training effect.

[0116] Based on the above method embodiment, the present invention provides a corresponding system embodiment, providing an eloquence training system based on biosensing and multimodal feedback, including: a data acquisition module, a first evaluation module, a second evaluation module, a third evaluation module, a comprehensive evaluation module and a training guidance module;

[0117] The data acquisition module is used to acquire user physiological data, user voice data and user video data;

[0118] The first evaluation module is used to analyze the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result and a sound feature evaluation result;

[0119] The second evaluation module is used to input the user voice data into a preset multi-layer network, output sound spectrum features, time series features and semantic relationship features, and use a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features to obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network;

[0120] The third evaluation module is used to input the user's physiological data, the user's voice data and the user's video data into a preset neural network, output voice features, visual features and text features, and use a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network;

[0121] The comprehensive evaluation module is used to generate personalized eloquence suggestions by combining the first evaluation result, the second evaluation result and the third evaluation result;

[0122] The training guidance module is used to guide the user to perform eloquence training using virtual reality technology based on the personalized eloquence suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Figure 1 : A flow chart of an embodiment of an eloquence training method based on biosensing and multimodal feedback provided by the present invention;

[0124] Figure 2 : A structural diagram of an embodiment of an eloquence training system based on biosensing and multimodal feedback provided by the present invention. DETAILED DESCRIPTION

[0125] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0126] In the description of the present application, it should be understood that the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first", "second" and "third" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "several" means two or more.

[0127] The present invention provides an eloquence training method and system based on biosensing and multi-modal feedback, which provide users with scientific, comprehensive and accurate eloquence evaluation results and a rich and realistic eloquence training environment, thereby improving the eloquence training effect.

[0128] Example 1

[0129] Based on the above requirements, an embodiment of the present invention provides an eloquence training method based on biosensing and multi-modal feedback, the method flow is as follows Figure 1 As shown, the method includes steps S1 to S6, and each step is specifically as follows:

[0130] S1. Obtain user physiological data, user voice data and user video data. This step includes steps S1.1 to S1.3, and each step is as follows:

[0131] S1.1, collecting user physiological data through biosensors; wherein the user physiological data includes user visual information, user facial expression information, user skin resistance information and user heart rate information. This step includes steps S1.1.1 to S1.1.4, each step is as follows:

[0132] S1.1.1. Collect user visual information through eye tracking sensors.

[0133] This embodiment collects user visual information through an eye tracking sensor. Specifically, the eye tracking sensor can be embedded in a glasses frame or a head-mounted device to capture information such as eye movement trajectory and pupil changes in real time.

[0134] S1.1.2. Collect user facial expression information through facial electromyography sensor.

[0135] This embodiment collects user facial expression information through facial electromyography sensors. The facial electromyography sensors can be arranged at key positions that fit the facial muscles, and non-verbal communication behaviors are analyzed by detecting electrical signals generated by facial expression activities.

[0136] S1.1.3. Collect the user's skin resistance information through a skin conductivity meter.

[0137] This embodiment collects the user's skin resistance information through a skin conductivity meter, and monitors the user's skin resistance change through the skin conductivity meter, wherein the change in skin resistance is related to emotional excitement or stress level.

[0138] S1.1.4. Collect the user's heart rate information through a heart rate detector.

[0139] This embodiment collects the user's heart rate information through a heart rate detector, and utilizes a wearable biosensor such as a heart rate monitor to collect the user's heart rate data to assess their tension or excitement level.

[0140] S1.2. Use array microphone technology to collect user voice data.

[0141] This embodiment uses array microphone technology to collect user voice data, which can not only be used to collect user voice information, but also realize functions such as sound source positioning and noise reduction to ensure high-quality voice signal input.

[0142] S1.3. Collect user video data using video acquisition technology.

[0143] This embodiment uses video acquisition technology to collect user video data, such as using a camera to capture the user's facial expression dynamics and gesture action data for inferring emotional state.

[0144] After collecting the user's physiological data, user's voice data and user's video data, this embodiment first uses a signal preprocessing circuit to perform preliminary amplification, filtering and digital processing on the collected raw data to improve the data quality. The process of processing data by the signal preprocessing circuit is as follows:

[0145] (1) Preliminary amplification and filtering:

[0146] After obtaining the original user physiological data, user voice data and user video data, the signal preprocessing circuit inside each sensor starts to work, amplifying the weak and noisy original data to increase its amplitude for subsequent analysis. At the same time, the integrated filter removes unnecessary high-frequency noise and other interference components to ensure a purer signal quality and obtain an analog signal.

[0147] (2) Analog-to-digital conversion process:

[0148] After amplification and filtering, the analog signal needs to be further digitized to meet the input requirements of digital signal processing. Therefore, an analog-to-digital converter is built into the signal processing unit to convert the continuously changing analog biological signal into a discrete digital signal, which facilitates the computer system to perform efficient and accurate data processing and analysis.

[0149] (3) Data integration and transmission preparation:

[0150] After the initial amplification, filtering and digitization, the high-quality digital signal is integrated into a data packet suitable for transmission. During this process, some data compression or encoding operations may be required to reduce the transmission bandwidth requirements and ensure the integrity and consistency of the data.

[0151] In this embodiment, after using the signal preprocessing circuit to process the collected user physiological data, user voice data and user video data, wireless communication is used to transmit the processed high-quality data. The specific process is as follows:

[0152] (1) Data packaging and encryption:

[0153] After the signal processing unit completes the initial amplification, filtering and digitization of the raw data, the pre-processed data is effectively organized and formatted to form a data packet suitable for wireless transmission. In order to ensure user privacy and data security, this embodiment uses an advanced encryption algorithm to encrypt the data to ensure that sensitive information is not stolen or tampered with during transmission.

[0154] ②Select the communication protocol and establish a connection:

[0155] Data is transmitted using a wireless communication module that supports a variety of low-power wireless communication protocols such as Bluetooth LE (Low Energy Bluetooth) and Zigbee. Based on actual needs and device compatibility, the system will select the most appropriate communication protocol and automatically establish a secure and stable wireless connection with the target receiving device.

[0156] ③Real-time transmission of physiological data:

[0157] Once an effective wireless connection is established, the wireless communication module starts to send pre-processed and encrypted physiological data to the target device in real time. Due to the use of low-power technology, this process can ensure continuous data transmission without affecting the endurance of the sensor.

[0158] ④Data integrity check and confirmation:

[0159] During data transmission, the system may also include a data integrity check mechanism to ensure that the receiving end can correctly receive and decrypt all data packets. If packet loss or transmission errors occur, the communication module can implement a retransmission mechanism to ensure data integrity and system reliability.

[0160] S2. Analyze the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result, and a sound feature evaluation result. This step includes steps S2.1 to S2.3, and each step is as follows:

[0161] S2.1. Use wireless transmission to transmit the user's physiological data and user's voice data to a preset integrated processor.

[0162] This embodiment uses wireless transmission to transmit user physiological data and user voice data to a preset integrated processor with the support of multiple low-power wireless communication protocols; wherein the preset integrated processor is obtained by integrating a high-performance microprocessor (such as the ARM Cortex series) and a digital signal processor (DSP) using an embedded system, and can quickly execute preprocessing algorithms such as Kalman filter denoising, sliding window average smoothing, spectrum analysis, etc., and can also deploy deep learning models specifically optimized for eloquence training, such as convolutional neural networks (CNN) for pattern recognition and recurrent neural networks (RNN) for time series analysis.

[0163] S2.2, analyzing the user's physiological data and the user's voice data through a preset integrated processor to obtain a fluency evaluation result, a logic evaluation result, an emotional diversity evaluation result, and a sound feature evaluation result. This step includes steps S2.2.1 to S2.2.4, and each step is as follows:

[0164] S2.2.1. Analyze the duration and number of words of the user's voice data by a preset integrated processor to obtain a fluency evaluation result; wherein the fluency evaluation result is specifically as follows:

[0165]

[0166] In the formula, F L Indicates the fluency assessment result; T s represents the duration of user voice data, W i represents the number of words in the i-th sentence in the user's voice data, T i represents the duration of the i-th sentence in the user's voice data, and N represents the number of utterances in the user's voice data.

[0167] This embodiment uses a preset integrated processor to analyze the user's voice data and records the duration T of each sentence in the user's voice data. i and the corresponding number of words W i Then, for each sentence in the speech data, calculate its speaking speed (speaking speed = number of words / duration of each sentence), and finally average the speaking speed of all sentences to obtain the overall fluency score F L , and the calculated fluency score F L As a result of the fluency assessment.

[0168] S2.2.2. Analyze the logical coherence and logical relevance of the user's voice data through a preset integrated processor to obtain a logic evaluation result; wherein the logic evaluation result is specifically as follows:

[0169]

[0170] Where, L c Indicates the result of logical evaluation; N s represents the number of sentences in the user's voice data, P j Represents the probability of logical relevance between the jth sentence in the user's voice data and the context.

[0171] This embodiment is based on a preset integrated processor, and first uses a natural language processing algorithm to analyze the content of each sentence in the user's voice data to determine the logical correlation probability P of each sentence and its context. j Then, the logarithmic operation is performed on the logical relevance probability of all sentences and the average value is taken to obtain the logical coherence score L c , and the logical coherence score L c As a logical evaluation result.

[0172] In this embodiment, a natural language processing model based on deep learning (such as Bidirectional Encoder Representation Transformer (BERT) or other similar sequence models) is used to determine the logical correlation probability P j , the specific process is as follows:

[0173] (1) Data preprocessing:

[0174] ① Sentence segmentation: extract text content from user voice data, and then segment the text content into separate sentences.

[0175] ②Tokenization: Tokenize each sentence into words or sub-word units.

[0176] ③Encoding: Convert tokenized words into digital codes that the model can understand.

[0177] (2) Context encoding:

[0178] ① Sequence model input: The tokenized and encoded sentence and its context (previous and next sentences) are input into the sequence model.

[0179] ②Context understanding: The sequence model understands the relationship between each sentence and its context through its internal self-attention mechanism.

[0180] (3) Logical correlation modeling:

[0181] ① Feature extraction: The model extracts features from the input sentence and its context to capture semantic and logical relationships.

[0182] ② Logical relevance evaluation: Based on the extracted features, the model calculates the logical relevance of the current sentence and its context.

[0183] (4) Probability calculation

[0184] ① Output layer design: In the output layer of the model, a probability output unit (such as the Softmax layer) is designed to output the probability P of the logical relevance between the current sentence and its context. j .

[0185] ②Probability output: The model outputs a probability value between 0 and 1, indicating the strength of the logical relevance between the current sentence and its context.

[0186] (5) Training and Optimization

[0187] ① Training data preparation: Prepare labeled training data, which contains logical relevance labels of sentences and their contexts.

[0188] ②Model training: Use training data to train the sequence model and optimize its performance on the logical relevance evaluation task.

[0189] ③Parameter tuning: Adjust model parameters through cross-validation and other methods to achieve the best logical correlation evaluation effect.

[0190] S2.2.3. Analyze the emotional state type and emotional state occurrence probability of the user's voice data through a preset integrated processor to obtain an emotional diversity evaluation result; wherein the emotional diversity evaluation result is as follows:

[0191]

[0192] In the formula, E D represents the result of emotion diversity evaluation; M represents the number of emotion states, q k represents the probability of the kth emotional state occurring.

[0193] This embodiment is based on a preset integrated processor, firstly uses speech emotion recognition technology to analyze the various emotional states contained in the user's speech data and the probability of occurrence of various emotional states q k , and then use the information entropy calculation formula to calculate the sentiment diversity index E D , to quantify the diversity of emotional expressions in user eloquence training, and to use the emotional diversity index E D As the result of sentiment diversity assessment.

[0194] Among them, this embodiment obtains the probability q of the occurrence of various emotional states k The specific process is as follows:

[0195] (1) Data collection and preprocessing:

[0196] ① Collect data: First, you need to collect a large amount of text or voice data containing various emotional states from big data. These data can come from public data sets, social media, speech recordings, etc.

[0197] ② Preprocessing: Preprocess the collected data, including denoising, standardization, feature extraction, etc., to facilitate subsequent analysis.

[0198] (2) Emotion recognition model training:

[0199] ① Model selection: Select a suitable emotion recognition model, such as a deep learning model for sentiment analysis (including models based on convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), etc.).

[0200] ②Model training: Use the preprocessed data to train the emotion recognition model. During the training process, the model will learn the characteristics of different emotional states and be able to perform emotion classification on new input data.

[0201] (3) Emotional state classification and probability calculation:

[0202] ① Emotion classification: Input the user voice data to be analyzed into the trained emotion recognition model, and the model will output the probability distribution of each emotional state.

[0203] ②Probability extraction: Extract the probability q of the kth emotional state from the emotional state probability distribution output by the model k .

[0204] (4) Parameter optimization and verification:

[0205] ① Parameter tuning: According to the performance of the model on the validation set, adjust the model parameters to optimize the performance.

[0206] ②Result verification: Use an independent test set to verify the accuracy and robustness of the model to ensure q k accuracy.

[0207] (5) Application and feedback

[0208] ① Practical application: Apply the trained model to the actual eloquence training system to analyze the user’s emotional state in real time.

[0209] ② Feedback adjustment: Based on user feedback and system performance, the emotion recognition model is continuously optimized to improve q k accuracy and overall system performance.

[0210] S2.2.4. Analyze the sound amplitude and sound frequency of the user's voice data through a preset integrated processor to obtain a sound feature evaluation result; wherein the sound feature evaluation result is specifically as follows:

[0211] V F =α·RMS(A t )+β·Var(Ft )

[0212] Where V F Indicates the sound characteristic evaluation result; RMS (A t ) represents the root mean square value of the sound amplitude in the time series t, and α represents RMS(A t ) weight coefficient, Var(F t ) represents the variance of the sound frequency in the time series t, and β represents Var(F t ) is the weight coefficient of .

[0213] In this embodiment, the root mean square (RMS) of the sound amplitude of the user's voice data in its time series is first calculated. t ) and the sound frequency variance Var(F t ), and then calculate the sound feature analysis score V by combining the amplitude root mean square and frequency variance F , and the sound feature analysis score V F As a result of the voice characteristic evaluation, the voice quality of the user can be reflected.

[0214] S2.3. Obtain a first evaluation result based on the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results.

[0215] This embodiment combines multiple evaluation results and the weights of the multiple evaluation results to calculate a comprehensive personalized feedback score P F , output P F As the first evaluation result of the comprehensive evaluation of the user's eloquence, specifically:

[0216] A first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results; wherein the first evaluation result is specifically as follows:

[0217] P F =ω1·F L +ω2·L C +ω3·E D +ω4·V F

[0218] Where P F Indicates the first evaluation result; F L represents the fluency evaluation result, ω1 represents the weight coefficient of the fluency evaluation result; L c represents the logical evaluation result, ω2 represents the weight coefficient of the logical evaluation result; E D represents the evaluation result of sentiment diversity, ω3 represents the weight coefficient of the evaluation result of sentiment diversity; V Frepresents the sound feature evaluation result, and ω4 represents the weight coefficient of the sound feature evaluation result.

[0219] Through step S2, this embodiment can process the acquired user data in real time, reduce data transmission time and improve response speed, and also perform basic data preprocessing and preliminary feature analysis on the user data, which can quickly screen and extract key information, reduce the burden for subsequent in-depth analysis, and thus improve the efficiency of eloquence evaluation.

[0220] S3, input the user voice data into a preset multi-layer network, output the sound spectrum features, time series features and semantic relationship features, use a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features, and obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network. This step includes steps S3.1 to S3.4, and each step is as follows:

[0221] S3.1. Input the user voice data into the advanced convolutional neural network and output the sound spectrum features.

[0222] In this embodiment, user voice data is input into an advanced convolutional neural network to output sound spectrum features; wherein the advanced convolutional neural network is specifically as follows:

[0223]

[0224] In the formula, Represents the output feature map of the cth layer of the advanced convolutional neural network at position (i, j); represents the convolution kernel weight of the cth layer of the advanced convolutional neural network, S (c―1) Represents the input spectrum feature map of the c-1 layer of the advanced convolutional neural network, B (c) Represents the bias term of the c-th layer of the high-level convolutional neural network.

[0225] This embodiment uses an advanced convolutional neural network (CNN) to process user voice data. As the convolution layer of the convolution kernel, each convolution layer inputs the spectral feature map S (c―1) After feature extraction, convolution operation is performed to output feature maps. The output feature maps contain the sound spectrum characteristics of the user's voice data, such as pitch, rhythm, volume change and other information, reflecting the sound performance dimension in eloquence evaluation.

[0226] S3.2. Perform time series processing on the user's voice data to obtain time series data of the voice, and input the time series data of the voice into a bidirectional long short-term memory network to output time series features.

[0227] The user's voice data is processed in time series to obtain the time series data of the voice, and the time series data of the voice is input into a bidirectional long short-term memory network to output time series features; wherein the bidirectional long short-term memory network is specifically as follows:

[0228]

[0229]

[0230]

[0231]

[0232]

[0233]

[0234] In the formula, represents the forget gate of the bidirectional long short-term memory network, represents the input gate of the bidirectional long short-term memory network, represents the output gate of the bidirectional long short-term memory network, Indicates the cell state, Indicates hidden state; and represents the weight matrix; and Represents the bias term.

[0235] First, the user's voice data is processed in time series to obtain the time series data of the voice (such as the time series features of the voice). Then, this embodiment uses the structure of the bidirectional LSTM to process the time dependency, based on the time series information before and after the voice data, through the forget gate and output gate Update cell status Output Gate Controlling the hidden state To provide memory information for the next moment, a bidirectional long short-term memory network (Bi-LSTM) is used to process the time series data of speech to obtain the time-dependent features of eloquence performance, such as language fluency, pauses, speaking speed and other information.

[0236] S3.3. Convert the user voice data into graph structure data of the voice, input the graph structure data of the voice into the graph convolutional neural network, and output semantic relationship features.

[0237] The user voice data is converted into graph structure data of the voice, and the graph structure data of the voice is input into the graph convolutional neural network to output the semantic relationship features; wherein the graph convolutional neural network is specifically as follows:

[0238]

[0239] In the formula, R (n) Represents the output graph convolution of the nth layer of the graph convolutional neural network; represents the angle matrix, represents the adjacency matrix A plus self-connection, X (n―1) represents the input features of the n-1th layer of the graph convolutional neural network, W (n) Represents the weight matrix of the nth layer of the graph convolutional neural network.

[0240] First, this embodiment represents the semantic relationship in the user's voice data as a graph structure to obtain the graph structure data of the voice; wherein the nodes in the graph structure represent words, and the edges in the graph structure represent voice relationships. Then, this embodiment uses a graph convolutional neural network (GCN) to process the graph structure data of the voice, using a graph convolution method. and To aggregate the features of neighboring nodes, a new feature representation of each node is obtained. The convolutional neural network can capture the deep relationship at the semantic level and provide richer semantic information for eloquence evaluation, such as the logical relationship between discourses, thematic coherence, etc., which reflects the content understanding and expression ability in eloquence performance.

[0241] S3.4. Input the sound spectrum features, time series features and semantic relationship features into the multi-layer perceptron for feature fusion and feature classification, and output the comprehensive feature set and the second evaluation result.

[0242] The sound spectrum features, time series features and semantic relationship features are input into a multi-layer perceptron for feature fusion and feature classification, and a comprehensive feature set and a second evaluation result are output; wherein the multi-layer perceptron is specifically as follows:

[0243]

[0244] In the formula, represents the kth output node of the qth layer of the multilayer perceptron; represents the weight of the qth layer of the multilayer perceptron, represents the rth fusion feature of the qth layer of the multilayer perceptron, Represents the bias term of the qth layer of the multilayer perceptron.

[0245] First, this embodiment combines the sound spectrum features, time series features and semantic relationship features to form a comprehensive feature set Then, a multi-layer perceptron is used to integrate the feature set Classify by weight and bias Score each feature dimension and output the scoring results of multiple dimensions including sound spectrum features, time series features and semantic relationship features. Finally, combine the scoring results of multiple dimensions to output the second evaluation result of the comprehensive evaluation of the user's eloquence. Through step S3.4, this embodiment can integrate the various extracted features (sound spectrum features, time series features and semantic relationship features) to obtain a comprehensive feature set that comprehensively reflects the eloquence performance, and then use a multi-layer perceptron to conduct in-depth analysis and classification of this comprehensive feature set, evaluate each feature dimension through the learned weights and biases, and obtain the final second evaluation result.

[0246] Through step S3, this embodiment can integrate sound spectrum features, time series features and semantic relationship features to improve the comprehensiveness of eloquence evaluation; through the multi-layer perceptron based on the learned weights and biases, deep feature extraction is performed on each feature to improve the accuracy of eloquence evaluation.

[0247] S4. Input the user's physiological data, user's voice data and user's video data into a preset neural network, output voice features, visual features and text features, and use a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network.

[0248] In this embodiment, it is necessary to pre-process the user's physiological data, user's voice data, and user's video data, and convert them into high-dimensional feature vectors suitable for the neural network model through a specific feature extraction algorithm, as follows:

[0249] (1) Data standardization:

[0250] ① Normalization scale: Normalize the user physiological data, user voice data, and user video data from different sensors to a unified scale to reduce the differences between data sources.

[0251] ② Normalization: Normalize all data so that they are distributed within a reasonable range (such as between 0 and 1) to facilitate neural network processing.

[0252] (2) Data denoising and data cleaning:

[0253] ① Noise filtering: Remove the noise components in the collected data through a variety of denoising algorithms (such as low-pass filter, median filter, etc.).

[0254] ②Outlier processing: Identify and process outliers in the data to ensure the accuracy and reliability of the data.

[0255] (3) Data serialization:

[0256] ① Time series processing: Perform time series processing on time series data (such as voice data, physiological data) to maintain the timing characteristics of the data.

[0257] ②Structured conversion: Convert unstructured data (such as video data, image data, text data) into a structured format to meet the needs of feature extraction algorithms.

[0258] (4) Feature extraction:

[0259] ① Algorithm selection: Select a feature extraction algorithm suitable for various data types, such as using Mel-frequency cepstral coefficients (MFCC) for speech data and convolutional neural networks (CNN) for image data.

[0260] ② Generation of high-dimensional feature vectors: The processed data is converted into high-dimensional feature vectors through feature extraction algorithms. These vectors cover all the key elements required to evaluate eloquence.

[0261] (5) Data integration and formatting:

[0262] ① Feature fusion: feature vectors from different data sources are fused to form a comprehensive feature representation.

[0263] ②Format adjustment: Adjust the format of the feature vector to ensure that it matches the input requirements of the subsequent neural network.

[0264] (6) Data transmission:

[0265] ① Data transmission: The data after preprocessing and feature extraction is transmitted to the corresponding neural network.

[0266] ② Maintain data quality: Ensure that data quality is not lost during data transmission and maintain its integrity and accuracy.

[0267] Step S4 includes steps S4.1 to S4.3, and each step is specifically as follows:

[0268] S4.1. Input the user physiological data and user video data into the improved convolutional neural network for feature extraction to obtain visual features.

[0269] Input the user's physiological data and the user's video data into the improved convolutional neural network for feature extraction to obtain visual features; wherein the improved convolutional neural network is specifically as follows:

[0270]

[0271] In the formula, Represents the output feature map of the mth convolutional layer of the improved convolutional neural network; LeakyReLU() represents the activation function, represents the convolution kernel weight of the improved convolutional neural network, Q (m―1) Denotes the input feature map of the m-1th layer of the improved convolutional neural network, D (m) Represents the bias term of the improved convolutional neural network.

[0272] This embodiment uses an improved convolutional neural network to process user physiological data and user video data, using the convolution layer The key visual features of the user’s physiological data and facial expression information in the user’s video data are extracted, and nonlinearity is added through activation functions (such as LeakyReLU) to capture complex visual patterns.

[0273] S4.2. Input the user voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features.

[0274] The user voice data is input into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; wherein the multi-layer long short-term memory network is specifically as follows:

[0275]

[0276]

[0277] In the formula, represents the input gate of a multi-layer long short-term memory network, Represents the forget gate of a multi-layer long short-term memory network; and represents the weight matrix, represents the hidden state at the previous moment, Represents the input at the current moment, and Represents the bias term.

[0278] This example uses a multi-layer long short-term memory network (LSTM) with a LSTM layer and Process user voice data to capture long-term dependencies in voice data, extract features such as speaking speed and intonation, and extract text features about the content of eloquent speeches from user voice data.

[0279] This embodiment also customizes a self-network to process a specific eloquence dimension: first, a specific sub-network structure is designed for different eloquence dimensions (such as logical expression and sentiment analysis); then, a customized network layer S is used to process the data of the specific eloquence dimension through a specific activation function (such as ELU); finally, information related to the specific eloquence dimension is extracted from the speech and visual features.

[0280] S4.3, using a multimodal deep fusion network to fuse speech features, visual features and text features to obtain a third evaluation result; wherein the multimodal deep fusion network includes a multimodal feature conversion layer, an attention adjustment layer, a multimodal deep fusion layer and an eloquence dimension analysis layer. This step includes steps S4.3.1 to S4.3.4, and each step is as follows:

[0281] S4.3.1. Use a multimodal feature conversion layer to perform feature conversion on speech features, visual features, and text features to obtain a first feature.

[0282] The multimodal feature conversion layer is used to perform feature conversion on the speech feature, the visual feature and the text feature to obtain a first feature; wherein the first feature is specifically as follows:

[0283] F trans =LeakyReLU(W trans ·concat(F v ,F a ,F t )+b trans )

[0284] In the formula, F trans represents the first feature after feature conversion; W trans represents the weight of the multimodal feature conversion layer, b trans represents the bias term of the multimodal feature conversion layer, F v represents the visual feature, F a represents the speech feature, F t Represents text features.

[0285] S4.3.2. Based on the dynamic attention mechanism, use the attention adjustment layer to calculate the attention weights of the speech features, visual features and text features, and perform weighted feature fusion on the first features according to the attention weights to obtain the second features.

[0286] This embodiment first calculates the attention weights of multiple features, and then performs weighted fusion on each feature based on the attention weights to obtain a second feature that represents the comprehensive feature.

[0287] Based on the dynamic attention mechanism, an attention adjustment layer is used to calculate the attention weights of speech features, visual features, and text features, and weighted feature fusion is performed on the first features according to the attention weights to obtain the second features;

[0288] The attention weight is as follows:

[0289] β=tanh(W att ·F trans +b att )

[0290] In the formula, β represents the attention weight; F trans represents the first feature, W att represents the weight of the attention adjustment layer, b att Represents the bias term of the attention adjustment layer;

[0291] The second feature is as follows:

[0292] F att =β⊙F trans

[0293] In the formula, F att represents the second feature after weighted feature fusion; β represents the attention weight, F trans Indicates the first feature.

[0294] S4.3.3. Use a multimodal deep fusion layer to deeply fuse the second feature to obtain a third feature.

[0295] This embodiment uses a multimodal deep fusion layer to perform a deeper feature fusion on the second feature representing the comprehensive feature, integrating key information from different modalities.

[0296] The second feature is deeply fused using a multimodal deep fusion layer to obtain a third feature; wherein the third feature is specifically as follows:

[0297] F deep =Sigmoid(W deep ·F att +b deep )

[0298] In the formula, F deep represents the third feature after deep fusion; Sigmoid() represents the activation function, W deep represents the weight of the multimodal deep fusion layer, F att represents the second feature, b deep Represents the bias term of the multimodal deep fusion layer.

[0299] Through step S4.3.1 to step S4.3.3, this embodiment integrates the features of different modalities, adopts advanced attention mechanism or gating mechanism to balance the importance of each modal feature, and forms a comprehensive feature representing multi-modality to comprehensively reflect the user's eloquence performance characteristics.

[0300] S4.3.4. Use the eloquence dimension analysis layer to perform feature analysis on the third feature to obtain a third evaluation result.

[0301] The third feature is analyzed using the eloquence dimension analysis layer to obtain a third evaluation result; wherein the third evaluation result is specifically as follows:

[0302] R dim =ELU(W dim ·F deep +b dim )

[0303] In the formula, R dim represents the third evaluation result output by the eloquence dimension analysis layer; ELU() represents the activation function, W dim represents the weight of the eloquence dimension analysis layer, F deep represents the third feature, b dim Represents the bias term of the analysis layer of the eloquence dimension.

[0304] Through step S4, this embodiment uses a preset neural network to conduct an in-depth analysis of the user's physiological data, user voice data, and user video data in step S1 to improve the accuracy and reliability of the eloquence evaluation results; a multimodal deep fusion network is used to fuse multiple features to ensure the comprehensiveness of the eloquence evaluation and avoid the evaluation bias caused by a single data source. In addition, this embodiment can also dynamically adjust the model parameters according to changes in the user's actual performance, continuously optimize the accuracy of the eloquence evaluation, and adapt to individual differences, providing feedback suggestions that are more in line with user needs. At the same time, it also takes into account the characteristics of data edge computing, reduces the time consumption of data transmission, ensures low-latency real-time response, and gives users immediate feedback, which improves the user's training experience and training effect.

[0305] This embodiment can also receive and process data in real time, realize dynamic adjustment of evaluation results, and improve the reliability of eloquence evaluation results. The specific flow of data is as follows:

[0306] (1) Data preprocessing and standardization:

[0307] ① Data formatting: Convert all user physiological data, user voice data and user video data into a unified format for subsequent processing.

[0308] ② Noise filtering and data cleaning: Perform noise filtering and cleaning on the formatted data to remove irrelevant or interfering information and improve data quality.

[0309] ③Data synchronization: Ensure that data from different sensors are synchronized in time to facilitate accurate comprehensive analysis.

[0310] (2) Feature extraction and encapsulation:

[0311] ① Extract key features: Extract features that are crucial to eloquence evaluation from the original user physiological data, user voice data, and user video data, such as the rhythm and intonation of the voice, subtle changes in facial expressions, etc.

[0312] ②Data encapsulation: Encapsulate the extracted features into a format suitable for neural network processing, for example, using feature vectors or feature matrices as network input.

[0313] (3) Data flows to each neural network:

[0314] ① Data transmission: The preprocessed and packaged data is sent to the corresponding neural network according to its feature type, such as improved convolutional neural network, multi-layer long short-term memory network, etc.

[0315] ②Interface management: Ensure the security and integrity of data during transmission to avoid data loss or tampering.

[0316] (4) Real-time data update and storage:

[0317] ① Real-time update: During system operation, sensor data is continuously received and updated to support real-time analysis.

[0318] ②Data storage: Store key data and analysis results for subsequent analysis and historical comparison.

[0319] S5. Generate personalized eloquence suggestions based on the first evaluation result, the second evaluation result and the third evaluation result.

[0320] First, this embodiment generates a quantitative score for multiple eloquence performance dimensions in eloquence training (such as fluency, logic, emotional expression, etc.), scores the multiple eloquence performance dimensions based on the first evaluation result, the second evaluation result, and the third evaluation result, and generates personalized eloquence suggestions based on the scores of multiple eloquence performance dimensions, including a comprehensive report of the scores of each dimension and related analysis, so as to provide a comprehensive understanding of the user's eloquence performance.

[0321] Then, the scores of multiple eloquence performance dimensions and personalized eloquence suggestions are visualized in the form of charts, graphs, etc., so that users can understand their performance and room for improvement more intuitively.

[0322] Finally, the scores of multiple eloquence performance dimensions and personalized eloquence suggestions are stored in the database for historical comparison or data tracing. The eloquence training feedback suggestions can also be adjusted in combination with the current personalized eloquence suggestions and historical personalized eloquence suggestions to form a continuous improvement cycle.

[0323] S6. Based on the personalized eloquence suggestion, use virtual reality technology to guide the user to perform eloquence training. This step includes steps S6.1 to S6.2, and each step is as follows:

[0324] S6.1. Use virtual reality technology to build an eloquence training environment for users.

[0325] This embodiment uses virtual reality (VR) technology to build a user's eloquence training environment, including: using game engines such as Unity or Unreal Engine to create a highly simulated three-dimensional speech environment, such as a simulated conference room, lecture hall, etc.; using dynamic rendering technology to simulate the reactions and situational feedback of real audiences, such as adjusting the expressions and body language of virtual audiences according to the content and style of the user's speech to enhance the realism of the speech experience; adjusting the constructed virtual training scene according to the real-time eloquence training evaluation results output by the algorithm to strengthen the feedback results. Based on the construction of the above virtual scenarios, users can enter a highly simulated eloquence training environment by wearing VR equipment and obtain a good eloquence training experience.

[0326] S6.2. Use augmented reality technology to visualize the personalized eloquence suggestions.

[0327] This embodiment uses augmented reality (AR) technology to visualize the personalized eloquence suggestions, including: using development frameworks such as ARKit or ARCore to overlay personalized eloquence suggestions (such as real-time scoring, improvement tips or demonstration videos, etc.) in the user's actual field of vision, so that the user can intuitively understand their own strengths and weaknesses in eloquence performance, and obtain real-time feedback and timely adjustments. For example, using AR markers to display scoring charts, or projecting floating prompts to guide users on how to improve pronunciation, body language and other issues.

[0328] S6.3. Based on the first evaluation result, the second evaluation result and the third evaluation result, use a machine learning algorithm to update the personalized eloquence suggestion.

[0329] Based on the first evaluation results, the second evaluation results and the third evaluation results, this embodiment uses machine learning algorithms (including collaborative filtering, cluster analysis, reinforcement learning algorithms, etc.) to dynamically generate personalized training plans and personalized training suggestions that adapt to the characteristics of users based on the historical evaluation results of their eloquence training, continuously track the user's progress and feedback, and constantly adjust and optimize the training plan to meet the needs of users at different stages.

[0330] According to the above-mentioned eloquence training method based on biosensing and multimodal feedback, this embodiment provides a personalized eloquence training platform for executing the method, which is as follows:

[0331] 1. Biosensor acquisition module and data preprocessing module: The biosensor acquisition module integrates an advanced biometric sensor array, including but not limited to heart rate sensors, voice recognition and emotion analysis microphones, facial expression capture cameras, etc., to capture the user's physiological reactions, language expression characteristics, and facial micro-expression changes during the speech in real time. The data preprocessing module is responsible for cleaning, standardizing, and initially converting the original biological signals into measurable indicators to provide accurate data input for subsequent analysis.

[0332] (1) Integration and application of biometric sensor arrays

[0333] ① Heart rate variability sensor: monitors the user's heart rate variability in real time to assess their anxiety or tension levels.

[0334] ② Voice recognition and emotion analysis microphone: captures the user's voice signal and performs emotion analysis to identify the emotional color in their voice.

[0335] ③Facial expression capture camera: records the user’s facial micro-expression changes during the speech to analyze emotional reactions and expression effects.

[0336] (2) Collection of original data:

[0337] ① Physiological signal acquisition: collect physiological data from devices such as heart rate sensors, such as heart rate frequency, heart rate variability, etc.

[0338] ② Voice data collection: record and analyze the user’s voice data, and extract parameters such as pitch, volume, and speaking speed.

[0339] ③Facial expression data collection: Through facial recognition technology, capture and analyze the subtle changes and expression dynamics of the user's face.

[0340] (3) Workflow of the data preprocessing subsystem

[0341] ① Data cleaning: remove noise and irrelevant information from the raw data, such as background sounds, occasional sensor misreadings, etc.

[0342] ② Data standardization: Convert all types of data into a unified format and scale for subsequent processing. For example, normalize all sensor data to the same range.

[0343] ③ Preliminary feature conversion: perform preliminary feature extraction on the original signal, such as extracting rhythm patterns from speech signals and extracting stress indicators from heart rate variability.

[0344] (4) Provide data input for subsequent analysis

[0345] ① Data integration: Integrate data from different sensors into a unified dataset for use by deep learning models and other analytical tools.

[0346] ②Data transmission: The preprocessed data is securely transmitted to the next analysis module of the platform, such as the neural network inference engine or the performance evaluation module.

[0347] 2. Edge intelligent analysis engine module: This module realizes efficient localized analysis of collected data based on edge computing technology. The module is equipped with multiple intelligent algorithm models designed for eloquence, such as emotion recognition model, speech rhythm analysis model, sound quality evaluation model, etc. The module is built with a distributed computing architecture to ensure fast response and low-latency real-time feedback.

[0348] (1) Application of edge computing technology

[0349] ① Localized data processing: Using edge computing technology, data is processed on local devices rather than remote servers, reducing data transmission time and improving processing speed.

[0350] ② Real-time analysis capability: ensuring that data analysis can be performed instantly, providing quick response and real-time feedback, which is particularly critical for eloquence training.

[0351] (2) Integration of intelligent algorithm models

[0352] ① Emotion recognition model: Analyze the user’s voice and facial expression data to identify their emotional state, such as happiness, tension, anxiety, etc.

[0353] ② Speech speed and rhythm analysis model: evaluates the user's speech speed and language rhythm to determine whether it is suitable for the purpose and occasion of the speech.

[0354] ③ Sound quality evaluation model: analyzes the user's voice quality, including factors such as pitch and clarity.

[0355] (3) Distributed computing architecture

[0356] ① Modular design: The edge intelligent analysis engine is constructed into multiple modules, each of which is responsible for different analysis tasks.

[0357] ② Distributed processing: Each module can work in parallel on multiple edge computing devices, improving the efficiency and speed of the overall analysis.

[0358] (4) Data input and processing

[0359] ① Receive data: Receive preprocessed data from the biosensor acquisition and data preprocessing module.

[0360] ②Analysis and calculation: Each intelligent algorithm model independently analyzes and calculates the received data to extract key eloquence dimension information.

[0361] (5) Synthesis and feedback of results

[0362] ① Result synthesis: The analysis results from different intelligent algorithm models are synthesized to form a comprehensive eloquence skill assessment report.

[0363] ② Generate feedback: Generate personalized training feedback and suggestions based on the comprehensive results.

[0364] (6) Continuous optimization and iteration

[0365] ① Performance monitoring: Continuously monitor the performance of the edge intelligent analysis engine to ensure the accuracy and response speed of the analysis.

[0366] ②Algorithm iteration: Based on user feedback and system analysis results, continuously optimize and iterate the intelligent algorithm model to improve the accuracy and efficiency of analysis.

[0367] 3. Personalized training content generation and optimization module: Based on the multi-dimensional oral ability assessment results provided by the first two modules, this module uses machine learning and reinforcement learning algorithms to dynamically generate personalized training courses, situational simulation exercises, and challenge tasks tailored to individual user characteristics. This module also includes a content library management system that integrates various high-quality educational resources and automatically recommends learning materials that best suit the user's current level and needs through algorithms.

[0368] (1) Analysis and evaluation results

[0369] ① Receive evaluation data: Receive the user's oral ability evaluation results from the edge intelligent analysis engine module, including multi-dimensional analysis data such as emotion recognition, speech speed rhythm, and sound quality evaluation.

[0370] ② Identify training needs: Analyze the evaluation results and determine the areas in which users need to improve their eloquence, such as improving emotional expression, optimizing language organization, and enhancing voice control.

[0371] (2) Personalized training course generation

[0372] ① Application of machine learning algorithms: Use machine learning algorithms to generate personalized training courses based on user evaluation results and historical learning data.

[0373] ② Situational simulation exercise design: Design situational simulation exercises related to user needs, such as simulated speeches, interview scenarios, etc., to enhance practical application capabilities.

[0374] ③Challenge task setting: Set challenge tasks of appropriate difficulty based on the user’s progress and ability level to motivate users to make continuous progress.

[0375] (3) Content library management system integration

[0376] ① Integration of educational resources: Collect and integrate various high-quality educational resources, including video tutorials, articles, speech transcripts, etc.

[0377] ②Algorithm recommendation: Use algorithms to automatically recommend learning materials that best match the user’s current level and needs, ensuring the relevance and effectiveness of the learning content.

[0378] (4) Reinforcement learning algorithm optimization

[0379] ① Dynamically adjust the learning plan: Based on the user's performance and feedback during the training process, use reinforcement learning algorithms to continuously adjust and optimize the training courses and exercise content.

[0380] ②Personalized learning path planning: Develop and update users’ personalized learning paths to maximize learning efficiency and effectiveness.

[0381] (5) User feedback and iterative improvement

[0382] ① Collect user feedback: Regularly collect user feedback on training courses and materials.

[0383] ② Course content iteration: Based on user feedback and learning outcomes, continuously iterate and improve course content and teaching methods.

[0384] (6) Learning outcomes tracking

[0385] ① Monitor learning progress: Track the user's learning progress and achievements to ensure that they are progressing smoothly according to the personalized plan.

[0386] ②Effectiveness evaluation: Regularly evaluate users’ learning outcomes to ensure the effectiveness of the training plan.

[0387] 4. Immersive interactive feedback experience module: This module uses virtual reality (VR) or augmented reality (AR) technology to build realistic simulation scenes, allowing users to practice their eloquence skills in a close-to-real environment. Users can receive intuitive feedback from virtual audiences through the immersive interactive feedback experience module. These feedbacks are dynamically generated based on the results obtained by the intelligent analysis engine, covering improvement tips in multiple dimensions such as voice quality, emotional control, and logical expression.

[0388] (1) Building an immersive simulation scene

[0389] ① Virtual environment design: Use VR or AR technology to design and build realistic simulation scenes, such as lecture halls, conference rooms or public lecture venues.

[0390] ② Scene interaction setting: Set interactive elements in the virtual environment, such as virtual audiences, interactive prompt boards, etc., to enhance the user's immersive experience.

[0391] (2) Practice your eloquence skills

[0392] ① Simulated speech practice: Users perform speeches or eloquence exercises in a virtual environment, simulating real speech scenarios.

[0393] ②Skill application: Users practice various eloquence skills, such as language expression, emotion control, body language, etc.

[0394] (3) Receive virtual feedback

[0395] ① Application of intelligent analysis results: Generate feedback from virtual audiences based on the intelligent analysis engine’s evaluation of the user’s eloquence performance.

[0396] ② Dynamic feedback display: Display intuitive feedback to users in a virtual environment, such as changes in virtual audience expressions, interactive prompts, etc.

[0397] (4) Multi-dimensional improvement tips

[0398] ① Comprehensive feedback generation: providing improvement tips in multiple dimensions including voice quality, emotion control, logical expression, etc.

[0399] ② Real-time adjustment suggestions: Provide instant adjustment suggestions and guidance based on the user’s real-time performance.

[0400] (5) Interactive learning experience

[0401] ① User interactive response: Users can make instant adjustments and responses based on feedback in the virtual environment, such as changing the rhythm of their speech, adjusting their emotional expression, etc.

[0402] ② Real-time practice adjustment: The system adjusts the training content and difficulty in real time based on the user’s interaction and performance.

[0403] (6) Learning Outcome Assessment

[0404] ① Practice effect tracking: record the user’s performance and progress in the virtual environment and evaluate the effectiveness of the practice.

[0405] ②Feedback loop: Feedback the user's practice results to other modules, such as personalized training content generation and optimization module, to further adjust the training plan.

[0406] The implementation of the above embodiments of the present invention has the following beneficial effects:

[0407] The present invention provides an eloquence training method based on biosensing and multimodal feedback: real-time collection of user physiological data, user voice data and user video data to provide scientific and comprehensive dynamic data support for eloquence training; using a preset integrated processor to perform preliminary analysis on the data to quickly screen and extract key voice information; using a preset multi-layer network to perform more complex and in-depth data analysis and feature extraction on the data to obtain evaluation results for multiple speech dimensions; using a preset neural network to extract features of the data on eloquence speech vision, speech hearing and speech content, and fusing multiple types of features to obtain deeper evaluation results; generating personalized eloquence suggestions based on multiple evaluation results, thereby providing users with scientific, comprehensive and accurate eloquence evaluation results and personalized feedback suggestions, and improving the eloquence training effect; providing users with a training environment based on virtual reality technology, thereby enriching the eloquence training environment and improving the eloquence training effect.

[0408] Example 2

[0409] Based on the contents of the above embodiments, an embodiment of the present invention provides an eloquence training system based on biosensing and multimodal feedback, including: a data acquisition module 101, a first evaluation module 102, a second evaluation module 103, a third evaluation module 104, a comprehensive evaluation module 105 and a training guidance module 106. The system structure is as follows Figure 2 shown.

[0410] The data acquisition module 101 is used to acquire user physiological data, user voice data and user video data;

[0411] The first evaluation module 102 is used to analyze the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result and a sound feature evaluation result;

[0412] The second evaluation module 103 is used to input the user voice data into a preset multi-layer network, output the sound spectrum features, time series features and semantic relationship features, and use a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features to obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network;

[0413] The third evaluation module 104 is used to input the user's physiological data, the user's voice data and the user's video data into a preset neural network, output voice features, visual features and text features, and use a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network;

[0414] The comprehensive evaluation module 105 is used to generate personalized eloquence suggestions by combining the first evaluation result, the second evaluation result and the third evaluation result;

[0415] The training guidance module 106 is used to guide the user to perform eloquence training using virtual reality technology according to the personalized eloquence suggestion.

[0416] In a possible implementation, the data acquisition module 101 includes: a physiological data unit 201, a voice data unit 202 and a video data unit 203;

[0417] The physiological data unit 201 is used to collect user physiological data through a biosensor; wherein the user physiological data includes user visual information, user facial expression information, user skin resistance information and user heart rate information;

[0418] The voice data unit 202 is used to collect user voice data using array microphone technology;

[0419] The video data unit 203 is used to collect user video data using video collection technology.

[0420] Further, the physiological data unit 201 includes: a visual information subunit 301, a facial information subunit 302, a skin information subunit 303 and a heart rate information subunit 304;

[0421] The visual information subunit 301 is used to collect user visual information through an eye tracking sensor;

[0422] The facial information subunit 302 is used to collect the user's facial expression information through a facial electromyography sensor;

[0423] The skin information subunit 303 is used to collect the user's skin resistance information through a skin conductivity meter;

[0424] The heart rate information subunit 304 is used to collect the user's heart rate information through a heart rate detector.

[0425] In a possible implementation, the first evaluation module 102 includes: a data transmission unit 204, a first comprehensive evaluation unit 205, and a first evaluation result unit 206;

[0426] The data transmission unit 204 is used to transmit the user's physiological data and the user's voice data to a preset integrated processor by wireless transmission;

[0427] The first comprehensive evaluation unit 205 is used to analyze the user's physiological data and the user's voice data through a preset integrated processor to obtain a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result, and a sound feature evaluation result;

[0428] The first evaluation result unit 206 is used to obtain a first evaluation result according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results.

[0429] Further, the first comprehensive evaluation unit 205 includes: a fluency subunit 305, a logic subunit 306, an emotion diversity subunit 307 and a sound feature subunit 308;

[0430] The fluency subunit 305 is used to analyze the duration and number of words of the user's voice data through a preset integrated processor to obtain a fluency evaluation result; wherein the fluency evaluation result is specifically as follows:

[0431]

[0432] In the formula, F L Indicates the fluency assessment result; T s represents the duration of user voice data, W i represents the number of words in the i-th sentence in the user's voice data, T i represents the duration of the i-th sentence in the user's voice data, and N represents the number of speech paragraphs in the user's voice data;

[0433] The logic subunit 306 is used to analyze the logic coherence and logic relevance of the user voice data through a preset integrated processor to obtain a logic evaluation result; wherein the logic evaluation result is specifically as follows:

[0434]

[0435] Where, L c Indicates the result of logical evaluation; N s represents the number of sentences in the user's voice data, P j Represents the probability of logical relevance between the jth sentence and the context in the user's voice data;

[0436] The emotion diversity subunit 307 is used to analyze the emotion state type and the emotion state occurrence probability of the user voice data through a preset integrated processor to obtain an emotion diversity evaluation result; wherein the emotion diversity evaluation result is specifically as follows:

[0437]

[0438] In the formula, E D represents the result of emotion diversity evaluation; M represents the number of emotion states, q k represents the probability of the kth emotional state occurring;

[0439] The sound feature subunit 308 is used to analyze the sound amplitude and sound frequency of the user's voice data through a preset integrated processor to obtain a sound feature evaluation result; wherein the sound feature evaluation result is specifically as follows:

[0440] V F =α·RMS(A t )+β·Var(F t )

[0441] Where V F Indicates the sound characteristic evaluation result; RMS (A t ) represents the root mean square value of the sound amplitude in the time series t, and α represents RMS(A t ) weight coefficient, Var(F t ) represents the variance of the sound frequency in the time series t, and β represents Var(F t ) is the weight coefficient of .

[0442] Further, the first evaluation result unit 206 is used to obtain a first evaluation result according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results, including:

[0443] The first evaluation result unit 206 is used to obtain a first evaluation result according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results; wherein the first evaluation result is specifically as follows:

[0444] P F =ω1·F L +ω2·L C +ω3·E D +ω4·V F

[0445] Where P F Indicates the first evaluation result; F Lrepresents the fluency evaluation result, ω1 represents the weight coefficient of the fluency evaluation result; L c represents the logical evaluation result, ω2 represents the weight coefficient of the logical evaluation result; E D represents the evaluation result of sentiment diversity, ω3 represents the weight coefficient of the evaluation result of sentiment diversity; V F represents the sound feature evaluation result, and ω4 represents the weight coefficient of the sound feature evaluation result.

[0446] In a possible implementation, the second evaluation module 103 includes: a sound spectrum unit 401, a time series unit 402, a semantic relationship unit 403 and a multi-layer perceptron unit 404;

[0447] The sound spectrum unit 401 is used to input the user voice data into the advanced convolutional neural network and output the sound spectrum features;

[0448] The time series unit 402 is used to perform time series processing on the user's voice data to obtain the time series data of the voice, and input the time series data of the voice into the bidirectional long short-term memory network to output the time series features;

[0449] The semantic relationship unit 403 is used to convert the user voice data into graph structure data of the voice, and input the graph structure data of the voice into the graph convolutional neural network to output semantic relationship features;

[0450] The multi-layer perceptron unit 404 is used to input the sound spectrum features, time series features and semantic relationship features into the multi-layer perceptron for feature fusion and feature classification, and output a comprehensive feature set and a second evaluation result.

[0451] Furthermore, the sound spectrum unit 401 is used to input the user voice data into the advanced convolutional neural network and output the sound spectrum features, including:

[0452] The sound spectrum unit 401 is used to input the user voice data into the advanced convolutional neural network and output the sound spectrum features; wherein the advanced convolutional neural network is specifically as follows:

[0453]

[0454] In the formula, Represents the output feature map of the cth layer of the advanced convolutional neural network at position (i, j); represents the convolution kernel weight of the cth layer of the advanced convolutional neural network, S (c―1) Represents the input spectrum feature map of the c-1 layer of the advanced convolutional neural network, B (c) Represents the bias term of the c-th layer of the high-level convolutional neural network.

[0455] Furthermore, the time series unit 402 is used to perform time series processing on the user voice data to obtain the time series data of the voice, and input the time series data of the voice into the bidirectional long short-term memory network to output the time series features, including:

[0456] The time series unit 402 is used to perform time series processing on the user voice data to obtain the time series data of the voice, and input the time series data of the voice into the bidirectional long short-term memory network to output the time series features; wherein the bidirectional long short-term memory network is specifically as follows:

[0457]

[0458]

[0459]

[0460]

[0461]

[0462]

[0463] In the formula, represents the forget gate of the bidirectional long short-term memory network, represents the input gate of the bidirectional long short-term memory network, represents the output gate of the bidirectional long short-term memory network, Indicates the cell state, Indicates hidden state; and represents the weight matrix; and Represents the bias term.

[0464] Furthermore, the semantic relationship unit 403 is used to convert the user voice data into graph structure data of voice, and input the graph structure data of voice into the graph convolutional neural network to output semantic relationship features, including:

[0465] The semantic relationship unit 403 is used to convert the user voice data into the graph structure data of the voice, and input the graph structure data of the voice into the graph convolutional neural network to output the semantic relationship features; wherein the graph convolutional neural network is specifically as follows:

[0466]

[0467] In the formula, R (n) Represents the output graph convolution of the nth layer of the graph convolutional neural network; represents the angle matrix, represents the adjacency matrix A plus self-connection, X (n―1) represents the input features of the n-1th layer of the graph convolutional neural network, W (n) Represents the weight matrix of the nth layer of the graph convolutional neural network.

[0468] Furthermore, the multi-layer perceptron unit 404 is used to input the sound spectrum features, time series features and semantic relationship features into the multi-layer perceptron for feature fusion and feature classification, and output a comprehensive feature set and a second evaluation result, including:

[0469] The multi-layer perceptron unit 404 is used to input the sound spectrum features, time series features and semantic relationship features into the multi-layer perceptron for feature fusion and feature classification, and output a comprehensive feature set and a second evaluation result; wherein the multi-layer perceptron is specifically as follows:

[0470]

[0471] In the formula, represents the kth output node of the qth layer of the multilayer perceptron; represents the weight of the qth layer of the multilayer perceptron, represents the rth fusion feature of the qth layer of the multilayer perceptron, Represents the bias term of the qth layer of the multilayer perceptron.

[0472] In a possible implementation, the third evaluation module 104 includes: a visual feature unit 501, a speech and text feature unit 502, and a multimodal fusion unit 503;

[0473] The visual feature unit 501 is used to input the user's physiological data and the user's video data into the improved convolutional neural network for feature extraction to obtain visual features;

[0474] The speech-text feature unit 502 is used to input the user speech data into a multi-layer long short-term memory network for feature extraction to obtain speech features and text features;

[0475] The multimodal fusion unit 503 is used to use a multimodal deep fusion network to fuse speech features, visual features and text features to obtain a third evaluation result; wherein the multimodal deep fusion network includes a multimodal feature conversion layer, an attention adjustment layer, a multimodal deep fusion layer and an eloquence dimension analysis layer.

[0476] Furthermore, the visual feature unit 501 is used to input the user physiological data and the user video data into the improved convolutional neural network for feature extraction to obtain visual features, including:

[0477] The visual feature unit 501 is used to input the user's physiological data and the user's video data into the improved convolutional neural network for feature extraction to obtain visual features; wherein the improved convolutional neural network is specifically as follows:

[0478]

[0479] In the formula, Represents the output feature map of the mth convolutional layer of the improved convolutional neural network; LeakyReLU() represents the activation function, represents the convolution kernel weight of the improved convolutional neural network, Q (m―1) Denotes the input feature map of the m-1th layer of the improved convolutional neural network, D (m) Represents the bias term of the improved convolutional neural network.

[0480] Furthermore, the visual feature unit 501 is used to input the user voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features, including:

[0481] The visual feature unit 501 is used to input the user voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; wherein the multi-layer long short-term memory network is specifically as follows:

[0482]

[0483]

[0484] In the formula, represents the input gate of a multi-layer long short-term memory network, Represents the forget gate of a multi-layer long short-term memory network; and represents the weight matrix, represents the hidden state at the previous moment, Represents the input at the current moment, and Represents the bias term.

[0485] Furthermore, the multimodal fusion unit 503 includes: a first feature subunit 601, a second feature subunit 602, a third feature subunit 603 and a feature analysis subunit 604;

[0486] The first feature subunit 601 is used to perform feature conversion on the speech feature, the visual feature and the text feature using a multimodal feature conversion layer to obtain a first feature;

[0487] The second feature subunit 602 is used to calculate the attention weights of the speech feature, the visual feature and the text feature using the attention adjustment layer based on the dynamic attention mechanism, and perform weighted feature fusion on the first feature according to the attention weight to obtain the second feature;

[0488] The third feature subunit 603 is used to perform deep fusion on the second feature using a multimodal deep fusion layer to obtain a third feature;

[0489] The feature analysis subunit 604 is used to perform feature analysis on the third feature using the eloquence dimension analysis layer to obtain a third evaluation result.

[0490] Furthermore, the first feature subunit 601 is used to perform feature conversion on the speech feature, the visual feature and the text feature using a multimodal feature conversion layer to obtain a first feature, including:

[0491] The first feature subunit 601 is used to perform feature conversion on speech features, visual features and text features using a multimodal feature conversion layer to obtain a first feature; wherein the first feature is specifically as follows:

[0492] F trans =LeakyReLU(W trans ·concat(F v ,F a ,F t )+b trans )

[0493] In the formula, F trans represents the first feature after feature conversion; W trans represents the weight of the multimodal feature conversion layer, b trans represents the bias term of the multimodal feature conversion layer, F v represents the visual feature, F a represents the speech feature, F t Represents text features.

[0494] Furthermore, the second feature subunit 602 is used to calculate the attention weights of the speech features, visual features and text features using the attention adjustment layer based on the dynamic attention mechanism, and perform weighted feature fusion on the first features according to the attention weights to obtain the second features, including:

[0495] The second feature subunit 602 is used to calculate the attention weights of the speech feature, the visual feature and the text feature using the attention adjustment layer based on the dynamic attention mechanism, and perform weighted feature fusion on the first feature according to the attention weight to obtain the second feature;

[0496] The attention weight is as follows:

[0497] β=tanh(W att ·F trans +b att )

[0498] In the formula, β represents the attention weight; F trans represents the first feature, W att represents the weight of the attention adjustment layer, b att Represents the bias term of the attention adjustment layer;

[0499] The second feature is as follows:

[0500] F att =β⊙F trans

[0501] In the formula, F att represents the second feature after weighted feature fusion; β represents the attention weight, F trans Indicates the first feature.

[0502] Furthermore, the third feature subunit 603 is used to perform deep fusion on the second feature using a multimodal deep fusion layer to obtain a third feature, including:

[0503] The third feature subunit 603 is used to perform deep fusion on the second feature using a multimodal deep fusion layer to obtain a third feature; wherein the third feature is specifically as follows:

[0504] F deep =Sigmoid(W deep ·F att +b deep )

[0505] In the formula, F deep represents the third feature after deep fusion; Sigmoid() represents the activation function, W deep represents the weight of the multimodal deep fusion layer, F att represents the second feature, b deep Represents the bias term of the multimodal deep fusion layer.

[0506] Furthermore, the feature analysis subunit 604 is used to perform feature analysis on the third feature using the eloquence dimension analysis layer to obtain a third evaluation result, including:

[0507] The third feature is analyzed using the eloquence dimension analysis layer to obtain a third evaluation result; wherein the third evaluation result is specifically as follows:

[0508] R dim=ELU(W dim ·F deep +b dim )

[0509] In the formula, R dim represents the third evaluation result output by the eloquence dimension analysis layer; ELU() represents the activation function, W dim represents the weight of the eloquence dimension analysis layer, F deep represents the third feature, b dim Represents the bias term of the analysis layer of the eloquence dimension.

[0510] In a possible implementation, the training guidance module 106 includes: a virtual reality unit 701, an augmented reality unit 702, and a feedback update unit 703;

[0511] The virtual reality unit 701 is used to construct a user's eloquence training environment using virtual reality technology;

[0512] The augmented reality unit 702 is used to visualize the personalized eloquence suggestion using augmented reality technology;

[0513] The feedback updating unit 703 is used to update the personalized eloquence suggestion using a machine learning algorithm according to the first evaluation result, the second evaluation result and the third evaluation result.

[0514] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0515] The implementation of the above embodiments of the present invention has the following beneficial effects:

[0516] The present invention provides an eloquence training system based on biosensing and multi-modal feedback, comprising: a data acquisition module, a first evaluation module, a second evaluation module, a third evaluation module, a comprehensive evaluation module and a training guidance module. The present invention acquires multiple types of user eloquence training data in real time through the data module; performs multi-dimensional, in-depth and comprehensive evaluation of users through the first evaluation module, the second evaluation module and the third evaluation module; provides scientific, comprehensive and accurate eloquence evaluation results for users through the comprehensive evaluation module and the training guidance module, provides personalized training suggestions and a rich and real eloquence training environment, thereby improving the eloquence training effect.

[0517] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training eloquence based on biosensing and multimodal feedback, characterized in that: include: Obtain user physiological data, user voice data and user video data; Analyzing the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result, and a sound feature evaluation result; Inputting user voice data into a preset multi-layer network, outputting sound spectrum features, time series features and semantic relationship features, and using a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features to obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network; Inputting the user's physiological data, the user's voice data and the user's video data into a preset neural network, outputting voice features, visual features and text features, and using a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network; Combining the first evaluation result, the second evaluation result and the third evaluation result, generating personalized eloquence suggestions; According to the personalized eloquence suggestions, using virtual reality technology to guide the user to perform eloquence training; The method includes inputting the user physiological data, the user voice data and the user video data into a preset neural network, outputting voice features, visual features and text features, and using a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result, including: Input the user's physiological data and user's video data into the improved convolutional neural network for feature extraction to obtain visual features; Input the user's voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; A multimodal deep fusion network is used to fuse speech features, visual features and text features to obtain a third evaluation result; wherein the multimodal deep fusion network includes a multimodal feature conversion layer, an attention adjustment layer, a multimodal deep fusion layer and an eloquence dimension analysis layer.

2. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 1, characterized in that: The obtaining of user physiological data, user voice data and user video data includes: Collecting user physiological data through biosensors; wherein the user physiological data includes user visual information, user facial expression information, user skin resistance information and user heart rate information; Use array microphone technology to collect user voice data; Use video acquisition technology to collect user video data.

3. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 2, characterized in that: The collecting of user physiological data through a biosensor includes: Collect user visual information through eye tracking sensors; Collect user's facial expression information through facial electromyography sensor; Collect user's skin resistance information through skin conductivity meter; The user's heart rate information is collected through a heart rate detector.

4. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 1, characterized in that: The analyzing the user voice data by a preset integrated processor to obtain a first evaluation result includes: Using wireless transmission to transmit user voice data to a preset integrated processor; Analyze the user's voice data through a preset integrated processor to obtain fluency evaluation results, logic evaluation results, emotional diversity evaluation results and sound feature evaluation results; A first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results.

5. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 4, characterized in that: The user's voice data is analyzed by a preset integrated processor to obtain a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result, and a sound feature evaluation result, including: The duration and number of words of the user's voice data are analyzed by a preset integrated processor to obtain a fluency evaluation result; wherein the fluency evaluation result is specifically as follows: In the formula, Indicates the results of the fluency assessment; Indicates the duration of the user's voice data. represents the number of words in the i-th sentence in the user's voice data, represents the duration of the i-th sentence in the user's voice data, Indicates the number of utterance segments of the user's voice data; The logical coherence and logical relevance of the user voice data are analyzed by a preset integrated processor to obtain a logical evaluation result; wherein the logical evaluation result is specifically as follows: In the formula, Indicates the result of logical evaluation; Indicates the number of sentences in the user's voice data, Represents the probability of logical relevance between the jth sentence and the context in the user's voice data; The preset integrated processor analyzes the emotional state type and the emotional state occurrence probability of the user's voice data to obtain an emotional diversity evaluation result; wherein the emotional diversity evaluation result is specifically as follows: In the formula, represents the result of emotion diversity assessment; M represents the number of emotion states, represents the probability of the kth emotional state occurring; The sound amplitude and sound frequency of the user's voice data are analyzed by a preset integrated processor to obtain a sound feature evaluation result; wherein the sound feature evaluation result is specifically as follows: In the formula, Indicates the sound characteristic evaluation result; represents the root mean square value of the sound amplitude in the time series t, express The weight coefficient of represents the variance of the sound frequency in the time series t, express The weight coefficient of .

6. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 4, characterized in that: The first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results, including: A first evaluation result is obtained according to the fluency evaluation result, the logic evaluation result, the emotion diversity evaluation result, the sound feature evaluation result and the weights of the multiple evaluation results; wherein the first evaluation result is specifically as follows: In the formula, Indicates the first evaluation result; represents the fluency assessment result, represents the weight coefficient of the fluency assessment result; Indicates the result of logical evaluation. The weight coefficient representing the result of logic evaluation; represents the result of sentiment diversity evaluation, Represents the weight coefficient of the sentiment diversity evaluation result; Indicates the sound characteristic evaluation result, Indicates the weight coefficient of the sound feature evaluation result.

7. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 1, characterized in that: The user voice data is input into a preset multi-layer network, sound spectrum features, time series features and semantic relationship features are output, and the sound spectrum features, time series features and semantic relationship features are fused using a multi-layer perceptron to obtain a second evaluation result, including: Input user voice data into an advanced convolutional neural network and output sound spectrum features; Performing time series processing on the user's voice data to obtain time series data of the voice, and inputting the time series data of the voice into a bidirectional long short-term memory network to output time series features; Converting user voice data into graph structure data of voice, and inputting the graph structure data of voice into a graph convolutional neural network to output semantic relationship features; The sound spectrum features, time series features and semantic relationship features are input into the multi-layer perceptron for feature fusion and feature classification, and the comprehensive feature set and the second evaluation result are output.

8. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 7, characterized in that: The user voice data is input into the advanced convolutional neural network, and the sound spectrum features are output, including: The user voice data is input into the advanced convolutional neural network, and the sound spectrum features are output; wherein the advanced convolutional neural network is specifically as follows: In the formula, Represents the output feature map of the cth layer of the advanced convolutional neural network at position (i, j); represents the convolution kernel weight of the cth layer of the advanced convolutional neural network, Represents the input spectrum feature map of the c-1 layer of the advanced convolutional neural network, Represents the bias term of the c-th layer of the high-level convolutional neural network.

9. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 7, characterized in that: The time series processing of the user voice data is performed to obtain the time series data of the voice, and the time series data of the voice is input into the bidirectional long short-term memory network to output the time series features, including: The user's voice data is processed in time series to obtain the time series data of the voice, and the time series data of the voice is input into a bidirectional long short-term memory network to output time series features; wherein the bidirectional long short-term memory network is specifically as follows: In the formula, represents the forget gate of the bidirectional long short-term memory network, represents the input gate of the bidirectional long short-term memory network, represents the output gate of the bidirectional long short-term memory network, Indicates the cell state, Indicates hidden state; , , and represents the weight matrix; , , and Represents the bias term.

10. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 7, characterized in that: The method of converting user voice data into graph structure data of voice, inputting the graph structure data of voice into a graph convolutional neural network, and outputting semantic relationship features includes: The user voice data is converted into graph structure data of the voice, and the graph structure data of the voice is input into the graph convolutional neural network to output the semantic relationship features; wherein the graph convolutional neural network is specifically as follows: In the formula, Represents the output graph convolution of the nth layer of the graph convolutional neural network; represents the angle matrix, represents the adjacency matrix A plus self-connection, represents the input features of the n-1th layer of the graph convolutional neural network, Represents the weight matrix of the nth layer of the graph convolutional neural network.

11. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 7, characterized in that: The sound spectrum features, time series features and semantic relationship features are input into a multi-layer perceptron for feature fusion and feature classification, and a comprehensive feature set and a second evaluation result are output, including: The sound spectrum features, time series features and semantic relationship features are input into a multi-layer perceptron for feature fusion and feature classification, and a comprehensive feature set and a second evaluation result are output; wherein the multi-layer perceptron is specifically as follows: In the formula, represents the kth output node of the qth layer of the multilayer perceptron; represents the weight of the qth layer of the multilayer perceptron, represents the rth fusion feature of the qth layer of the multilayer perceptron, Represents the bias term of the qth layer of the multilayer perceptron.

12. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 1, characterized in that: The step of inputting the user physiological data and the user video data into the improved convolutional neural network for feature extraction to obtain visual features includes: Input the user's physiological data and the user's video data into the improved convolutional neural network for feature extraction to obtain visual features; wherein the improved convolutional neural network is specifically as follows: In the formula, Represents the output feature map of the mth convolutional layer of the improved convolutional neural network; LeakyReLU() represents the activation function, represents the convolution kernel weight of the improved convolutional neural network, represents the input feature map of the m-1th layer of the improved convolutional neural network, Represents the bias term of the improved convolutional neural network.

13. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 1, characterized in that: The user voice data is input into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features, including: The user voice data is input into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; wherein the multi-layer long short-term memory network is specifically as follows: In the formula, represents the input gate of a multi-layer long short-term memory network, Represents the forget gate of a multi-layer long short-term memory network; and represents the weight matrix, represents the hidden state at the previous moment, Represents the input at the current moment, and Represents the bias term.

14. The method for training eloquence based on biosensing and multimodal feedback as claimed in claim 1, characterized in that: The method uses a multimodal deep fusion network to fuse speech features, visual features, and text features to obtain a third evaluation result, including: Using a multimodal feature conversion layer to perform feature conversion on speech features, visual features, and text features to obtain a first feature; Based on the dynamic attention mechanism, an attention adjustment layer is used to calculate the attention weights of speech features, visual features, and text features, and weighted feature fusion is performed on the first features according to the attention weights to obtain the second features; Using a multimodal deep fusion layer to deeply fuse the second feature to obtain a third feature; The eloquence dimension analysis layer is used to perform feature analysis on the third feature to obtain a third evaluation result.

15. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 14, characterized in that: The method of using a multimodal feature conversion layer to perform feature conversion on speech features, visual features, and text features to obtain a first feature includes: The multimodal feature conversion layer is used to perform feature conversion on the speech feature, the visual feature and the text feature to obtain a first feature; wherein the first feature is specifically as follows: In the formula, Represents the first feature after feature transformation; represents the weight of the multimodal feature conversion layer, represents the bias term of the multimodal feature conversion layer, Represents visual features, Represents the voice features, Represents text features.

16. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 14, characterized in that: The method uses an attention adjustment layer to calculate the attention weights of speech features, visual features, and text features based on the dynamic attention mechanism, and performs weighted feature fusion on the first features according to the attention weights to obtain the second features, including: Based on the dynamic attention mechanism, an attention adjustment layer is used to calculate the attention weights of speech features, visual features, and text features, and weighted feature fusion is performed on the first features according to the attention weights to obtain the second features; The attention weight is as follows: In the formula, represents the attention weight; Represents the first feature, represents the weight of the attention adjustment layer, Represents the bias term of the attention adjustment layer; The second feature is as follows: In the formula, Represents the second feature after weighted feature fusion; represents the attention weight, Indicates the first feature.

17. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 14, characterized in that: The method of using a multimodal deep fusion layer to deeply fuse the second feature to obtain a third feature includes: The second feature is deeply fused using a multimodal deep fusion layer to obtain a third feature; wherein the third feature is specifically as follows: In the formula, Represents the third feature after deep fusion; Sigmoid() represents the activation function, represents the weight of the multimodal deep fusion layer, Represents the second feature, Represents the bias term of the multimodal deep fusion layer.

18. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 14, characterized in that: The using the eloquence dimension analysis layer to perform feature analysis on the third feature to obtain a third evaluation result includes: The third feature is analyzed using the eloquence dimension analysis layer to obtain a third evaluation result; wherein the third evaluation result is specifically as follows: In the formula, represents the third evaluation result output by the eloquence dimension analysis layer; ELU() represents the activation function, represents the weight of the eloquence dimension analysis layer, Represents the third feature, Represents the bias term of the analysis layer of the eloquence dimension.

19. The method for training eloquence based on biosensing and multi-modal feedback as claimed in claim 1, characterized in that: The method of guiding the user to perform eloquence training using virtual reality technology according to the personalized eloquence suggestion includes: Use virtual reality technology to build a user's eloquence training environment; Using augmented reality technology to visualize the personalized eloquence suggestions; According to the first evaluation result, the second evaluation result and the third evaluation result, the personalized eloquence suggestion is updated using a machine learning algorithm.

20. An eloquence training system based on biosensing and multimodal feedback, characterized in that: include: A data acquisition module, a first evaluation module, a second evaluation module, a third evaluation module, a comprehensive evaluation module and a training guidance module; The data acquisition module is used to acquire user physiological data, user voice data and user video data; The first evaluation module is used to analyze the user's voice data through a preset integrated processor to obtain a first evaluation result; wherein the first evaluation result includes a fluency evaluation result, a logic evaluation result, an emotion diversity evaluation result and a sound feature evaluation result; The second evaluation module is used to input the user voice data into a preset multi-layer network, output sound spectrum features, time series features and semantic relationship features, and use a multi-layer perceptron to fuse the sound spectrum features, time series features and semantic relationship features to obtain a second evaluation result; wherein the preset multi-layer network includes an advanced convolutional neural network, a bidirectional long short-term memory network and a graph convolutional neural network; The third evaluation module is used to input the user's physiological data, the user's voice data and the user's video data into a preset neural network, output voice features, visual features and text features, and use a multimodal deep fusion network to fuse the voice features, visual features and text features to obtain a third evaluation result; wherein the preset neural network includes an improved convolutional neural network and a multi-layer long short-term memory network; The comprehensive evaluation module is used to generate personalized eloquence suggestions by combining the first evaluation result, the second evaluation result and the third evaluation result; The training guidance module is used to guide the user to perform eloquence training using virtual reality technology according to the personalized eloquence suggestions; Wherein, the third evaluation module includes: Input the user's physiological data and user's video data into the improved convolutional neural network for feature extraction to obtain visual features; Input the user's voice data into a multi-layer long short-term memory network for feature extraction to obtain voice features and text features; A multimodal deep fusion network is used to fuse speech features, visual features and text features to obtain a third evaluation result; wherein the multimodal deep fusion network includes a multimodal feature conversion layer, an attention adjustment layer, a multimodal deep fusion layer and an eloquence dimension analysis layer.

Images