A personalized eloquence training method, system, device and medium

Through the deep learning model combined with multi-perception technology, the perceived data of eloquent expressors are analyzed and personalized suggestions are generated, which solves the problem of lack of interactivity and personalization of traditional eloquent training, and achieves a more comprehensive and accurate eloquent training effect.

CN117788235BActive Publication Date: 2024-12-06CHINA NEW LINE EDUCATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311700258.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-11
Publication Date
2024-12-06
Estimated Expiration
2043-12-11

AI Technical Summary

Technical Problem

Traditional eloquence training lacks interactivity and personalization, makes it difficult to guide personal situations, and ignores perceived data such as brain waves of eloquence expressers during communication, resulting in too thin analysis results and unable to provide targeted suggestions.

Method used

Deep learning model is used to combine speech, brain wave, eye movement and facial data acquisition technology to analyze the perceived data of eloquent expressors, generate performance characteristics, and filter out personalized suggestions through the information database to achieve the combination of multi-perception technology and deep learning technology. , improve training efficiency.

Benefits of technology

By comprehensively and accurately analyzing the performance of eloquent expressors, providing personalized feedback, improving the efficiency and effectiveness of eloquent training, and enhancing the interactiveness and pertinence of training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117788235B_ABST
    Figure CN117788235B_ABST
Patent Text Reader

Abstract

The present invention proposes a personalized eloquence training method, system, device and medium, the method comprising: collecting the perception data corresponding to the eloquence speaker and / or the viewer during the eloquence training process, the perception data being at least one of voice data, brain wave data, eye movement data or facial data; performing deep learning and analysis on the perception data based on a pre-established deep learning model to obtain performance characteristics corresponding to different perception data; the performance characteristics include voice characteristics, psychological characteristics, attention characteristics, sight characteristics and emotional characteristics; calling an information library to screen the suggestion information corresponding to different performance characteristics. The present invention can comprehensively and accurately analyze the performance of the eloquence speaker, give the trainee personalized feedback, and improve the efficiency of eloquence training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of eloquence training, and in particular to a personalized eloquence training method, system, equipment and medium. Background Art

[0002] Traditional eloquence training mainly adopts the teaching method of teachers, and the eloquent speakers communicate with the teachers. The teachers observe the performance of the eloquent speakers during the communication process to subjectively judge the effectiveness of the eloquent speakers. However, the teaching method of teachers lacks interactivity and is inefficient. In addition, the traditional eloquence training method is difficult to provide personalized guidance based on the specific situation of individuals, and it is also difficult to evaluate the learning effect of students. At the same time, in the field of eloquence training, generally only the eloquent speakers' expression language is evaluated, ignoring the perceptual data such as the brain wave data of the eloquent speakers during communication, resulting in the lack of analysis in other dimensions, making the eloquence analysis results too thin and unable to provide targeted suggestions and guidance for eloquent speakers. Summary of the invention

[0003] The embodiment of the present invention provides a personalized eloquence training method, system, device and medium to solve the problems existing in the related technologies. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present invention provides a personalized eloquence training method, comprising:

[0005] Collecting perception data corresponding to the eloquent speaker and / or the viewer during the eloquence training of the eloquent speaker, where the perception data is at least one of voice data, brain wave data, eye movement data or facial data;

[0006] Based on the pre-established deep learning model, the perception data is deeply learned and analyzed to obtain the performance characteristics corresponding to different perception data; the deep learning model includes a sentiment analysis model and a behavior prediction model; the performance characteristics include voice characteristics, psychological characteristics, attention characteristics, sight characteristics and emotional characteristics;

[0007] Call the information database to filter the recommended information corresponding to different performance characteristics.

[0008] In one implementation, the perception data is voice data, and further includes:

[0009] Preprocessing the voice data to generate preprocessed voice data;

[0010] Converting the preprocessed voice data into text to generate first display information using display text;

[0011] Perform semantic analysis on the text to obtain semantic analysis results;

[0012] Determine whether there is a semantic error in the semantic analysis result and determine the content of the semantic error;

[0013] According to the semantically erroneous content and the semantic analysis result, second presentation information displaying the semantically erroneous content and the semantic analysis result is generated.

[0014] In one implementation, analyzing speech data based on a deep learning module includes:

[0015] The preprocessed speech data is used as the model input and input into the pre-established speech deep learning model for learning and training, and the speech features are output. The speech features include speech clarity, speaking speed and tone.

[0016] In one embodiment, the brain wave data includes brain wave signals and cortical signals; if the brain wave data is collected, the method for analyzing the brain wave data is:

[0017] Extract and label the signal features of EEG data, including frequency, amplitude and spectral density;

[0018] The brain wave data marked with signal characteristics are used as model input and input into the pre-established brain wave deep learning model for learning and training, and psychological characteristics and attention characteristics are output. The attention characteristics include concentration, memory ability and reaction speed.

[0019] In one embodiment, if eye movement data is collected, the method for analyzing the eye movement data is:

[0020] The eye movement data is used as the model input and analyzed by the pre-established eye movement deep learning model to output the gaze features; the gaze features include the location of the focus point and the gaze time.

[0021] In one embodiment, if facial data is collected, the method for analyzing the facial data is:

[0022] The facial data is used as the model input, and the facial data is analyzed based on the pre-established facial deep learning model to output emotional features.

[0023] In one implementation, after filtering out the suggestion information corresponding to the performance feature, the method further includes:

[0024] The performance characteristics and the suggested information are displayed visually; the visual display includes displaying using at least one of a chart, an animation, and an interactive game.

[0025] In a second aspect, an embodiment of the present invention provides a personalized eloquence training system, which executes the personalized eloquence training method as described above; the system comprises:

[0026] A collection module, used for collecting perception data corresponding to the eloquent speaker and / or the viewer during the eloquence training of the eloquent speaker, wherein the perception data is one of voice data, brain wave data, eye movement data or facial data, or a combination of two or more thereof;

[0027] The model analysis module is used to perform deep learning and analysis on the perception data according to the pre-established deep learning model, and output the performance characteristics corresponding to different perception data; the performance characteristics include voice characteristics, psychological characteristics, attention characteristics, sight characteristics and emotional characteristics;

[0028] The suggestion push model is used to call the information library to filter the suggestion information corresponding to the different performance characteristics and push it.

[0029] In a third aspect, an embodiment of the present invention provides an electronic device, the device comprising: a memory and a processor. The memory and the processor communicate with each other through an internal connection path, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor executes the method in any one of the above-mentioned embodiments.

[0030] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is executed.

[0031] The advantages or beneficial effects of the above technical solution include at least:

[0032] The use of multi-sensory technologies such as brain waves, eye tracking and facial expression recognition to collect the perceptual data of eloquent communicators during the expression and communication process can comprehensively and accurately analyze the performance of eloquent communicators; at the same time, deep learning technology is used to analyze the students' expression and communication data, so as to give students personalized feedback; the use of Internet of Things technology to realize the interaction between students and teachers can improve the efficiency of eloquence training.

[0033] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present invention and should not be regarded as limiting the scope of the present invention.

[0035] Figure 1 It is a schematic diagram of the process of the personalized eloquence training method of the present invention;

[0036] Figure 2 This is a schematic diagram of the modules of the personalized eloquence training system of the present invention;

[0037] Figure 3 It is a schematic block diagram of the module structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0038] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0039] Embodiment 1

[0040] This embodiment provides a personalized eloquence training method. Figure 1 As shown, the specific steps include:

[0041] Step S1: in response to an eloquence training request, activating a perception device of the eloquence speaker and / or the viewer to collect perception data corresponding to the eloquence speaker and / or the viewer during the eloquence training process of the eloquence speaker, wherein the perception data is at least one of voice data, brain wave data, eye movement data or facial data;

[0042] Step S2: performing deep learning and analysis on the perception data based on a pre-established deep learning model, and outputting performance features corresponding to different perception data; the performance features include voice features, psychological features, attention features, sight features, and emotional features;

[0043] Step S3: calling the information library to filter and push the suggestion information corresponding to different performance characteristics.

[0044] Among them, the eloquent speaker is the student who needs to express and communicate, and the viewer is the person who watches the expression and communication; the eloquent speaker can be a teacher who teaches, and the viewer can be a student; in addition, the identities of the student and the teacher can be swapped according to actual needs, that is, the eloquent speaker can also be a student, and the viewer can be a teacher; by observing the perception data of the teacher and / or the student, personalized feedback can be given to the teacher or student who is expressing and communicating.

[0045] When a request for eloquence training is issued, the perception devices of the eloquent speaker and / or the viewer are activated. That is, when the eloquent speaker is expressing and communicating, the perception device of the eloquent speaker can be activated to collect the perception data of the eloquent speaker, and directly and automatically evaluate the performance of the eloquent speaker; or when the eloquent speaker is expressing and communicating, the perception device of the viewer can be activated to indirectly reflect the eloquence of the current eloquent speaker by acquiring the perception data of the viewer, provide the eloquent speaker with feedback from the viewer's perspective, and comprehensively analyze the expression and communication performance of the eloquent speaker.

[0046] Perception devices include voice collection devices, brain wave collection devices, eye tracking devices and cameras; the voice collection device is mainly a microphone, which can be a dynamic microphone, a condenser microphone, a crystal microphone, etc. The microphone is responsible for converting sound into electrical signals; the perception data obtained by the microphone is voice data.

[0047] The brainwave acquisition equipment is mainly a brainwave sensor, such as an electrode-type brainwave sensor, a photoelectric brainwave sensor and a magnetoelectric brainwave sensor. The electrodes are attached to the user's scalp to collect brainwave signals, and the weak electrical signals generated by the brain can be converted into digital signals for analysis and processing. The brainwave acquisition equipment can also be a combined brainwave-cortical signal collector, which can collect brainwave signals and cortical signals at the same time to better understand brain activity. The perception data collected by the brainwave acquisition equipment is brainwave data, which includes brainwave signals and cortical signals.

[0048] Eye tracking devices are mainly eye trackers, which can collect information such as the user's eye position, movement speed and direction. Eye trackers usually use infrared optical technology to collect eye position information, and use visual inertial navigation technology to calculate eye movement speed and direction. The perception data collected by the eye tracking device is the eye movement data.

[0049] The camera is mainly used to capture the user's facial features for facial analysis; therefore, the perception data collected by the camera is facial data.

[0050] It should be noted that the above-mentioned perception devices can be activated and used according to actual conditions. In the best case, the voice acquisition device, brain wave acquisition device, eye tracking device and camera are all turned on and put into use, which can analyze the various characteristics of the eloquent expresser and / or the viewer in the process of expression and communication from different angles in the most comprehensive way, thereby improving the comprehensiveness and accuracy of the analysis. In addition, according to actual needs, one of the voice acquisition devices, brain wave acquisition devices, eye tracking devices or cameras can be used, or a combination of two or more of the voice acquisition devices, brain wave acquisition devices, eye tracking devices or cameras can be used to collect the user's perception during the expression and communication process to obtain the corresponding perception data, so as to generate corresponding eloquence training suggestions after analysis.

[0051] The analysis of the perception data in step S2 of this embodiment is implemented by using a deep learning model including a sentiment analysis model and a behavior prediction model in a specific field of eloquence, as follows:

[0052] 1. Sentiment analysis model.

[0053] The general method of sentiment analysis is to convert the perception data of digital signals into corresponding text data and then map them to the sentiment polarity space, such as positive, negative and neutral. Perception data includes voice data, brain wave data, eye movement data and facial data. The digital signals of each perception data are converted into text data, such as converting voice data into text, or converting the frequency and amplitude signals corresponding to brain wave data into text data recognizable by the model, etc. The model analyzes the emotions of the speaker and obtains the corresponding features, and then gives appropriate suggestions to the speaker based on the features. The model can use the following mathematical formula:

[0054] When it comes to more advanced text representation, some more complex concepts such as attention mechanism, concentration, fatigue, interjections, long-tail words, frequency, etc. can be introduced to achieve more advanced text representation. The following is the updated high-level mathematical operation formula:

[0055] (I) Input representation (Advanced):

[0056] Given a text sequence X, it can be represented as a word embedding matrix E, where each row corresponds to the word embedding of a word. The length of X is T, and the input of each time step t is a word embedding vector x_t, so X = [x_1, x_2, ..., x_T].

[0057] In this more advanced model, a complex attention mechanism is introduced to capture the information in the text in a more refined way. Specifically, consider the following concepts:

[0058] 1. Attention Mechanism, which is expressed as:

[0059] αt=Attention(E,ht-1)

[0060] αt(Attention): This is an attention weight that indicates the degree of attention paid to different word embeddings in the text at time step t. Specifically, it is used to dynamically weight different word embeddings to capture important information in the text. This weight can be calculated based on the previous hidden state h_{t-1} and the word embedding matrix E to ensure that different words are paid attention to at different time steps.

[0061] The Attention() function determines how to calculate the attention weights, and h_{t-1} is the hidden state of the previous time step. The attention weights can then be used to calculate the input representation of the current time step:

[0062]

[0063] xt (input representation): This is the input representation at time step t, which represents the weighted sum of each word embedding in the text. It is dynamically calculated by the above attention weight αt and word embedding E to represent the text information more accurately.

[0064] 2. Focus: Introduce a focus variable γt, which represents the focus level in the text at time step $t$. This focus can be dynamically adjusted based on context and historical information. This process can be expressed as:

[0065] γ t =Focus(X,h t-1 ,context);

[0066] γt(Focus): This is a variable that represents the level of focus in the text at time step t. It is used to dynamically adjust the focus level based on context and historical information. Focus can affect the fluency and coherence of eloquence, because different parts of the text may require different levels of attention at different time steps. Among them, the Focus() function determines how to calculate focus.

[0067] 3. Fatigue: Consider the fatigue of the user or text reader, which can change dynamically over time. Introducing a fatigue variable φt can affect the attention mechanism and concentration. This process can be expressed as:

[0068] φ t =Fatigue(t);

[0069] φt (Fatigue): This is a variable that represents the fatigue of the user or text reader. It can change dynamically over time and is used to affect the attention mechanism and concentration. Fatigue can be calculated based on time t to ensure that the intensity of eloquence training is appropriately adjusted during long-term eloquence training. Among them, the Fatigue() function can calculate fatigue based on time t.

[0070] 4. Exclamations: Consider the exclamations in the text, which usually represent the parts with strong emotions. We introduce the exclamation weight βt, which is dynamically adjusted according to the appearance of exclamations in the text. This process can be expressed as:

[0071] β t =Exclamations(X);

[0072] βt (Exclamations): This is a variable that represents the weight of the exclamations in the text. Exclamations usually represent parts with strong emotions. The weights are dynamically adjusted according to the appearance of exclamations in the text. This ensures that the model pays more attention to the emotionally rich parts and improves the appeal of eloquence. Among them, the Exclamations() function calculates the weights according to the appearance of exclamations in the text.

[0073] 5. Long-Tail Words: Some words appear less frequently in the text, but may have important information. The long-tail word weight ωt is introduced to weight the word embedding of these uncommon words. This process can be expressed as:

[0074] ω t =LongTailWords(X);

[0075] ωt(LongTailWords): This is a variable that represents the weight of long-tail words. Some words appear less frequently in the text, but may contain important information, especially in specific fields or professional texts. The weight of long-tail words can be dynamically adjusted according to the frequency of occurrence of the word. This helps the model better capture the meaning of uncommon words and improve the accuracy of eloquence. Among them, the LongTailWords() function calculates the weight according to the frequency of occurrence of the word.

[0076] Finally, we can combine the above elements to calculate the input representation x_t at the current time step t to more accurately capture the information in the text:

[0077] x t =α t,i γ t ·φ t β t ·ω t ·Ei ;

[0078] The following is a detailed explanation of the definition of each indicator:

[0079] αt (Attention): This is an attention weight that indicates the degree of attention paid to different word embeddings in the text at time step t.

[0080] xt (input representation): This is the input representation at time step t, which represents the weighted sum of each word embedding in the text. It is dynamically calculated by the above attention weight αt and word embedding E to represent the text information more accurately.

[0081] γt(Focus): This is a variable that represents the level of focus in the text at time step t. It is used to dynamically adjust the focus level based on context and historical information. Focus can affect the fluency and coherence of eloquence, because different parts of the text may require different levels of attention at different time steps.

[0082] φt (Fatigue): This is a variable that represents the fatigue of the user or text reader. It can change dynamically over time and is used to influence the attention mechanism and concentration. Fatigue can be calculated based on time $t$ to ensure that the intensity of eloquence training is appropriately adjusted during long-term eloquence training.

[0083] βt (Exclamations): This is a variable that represents the weight of the interjections in the text. Interjections usually represent parts with strong emotions. It dynamically adjusts the weight according to the appearance of interjections in the text. This ensures that the model pays more attention to the emotionally rich parts and improves the appeal of eloquence.

[0084] ωt(LongTailWords): This is a variable representing the weight of long-tail words. Some words appear less frequently in the text, but may contain important information, especially in specific fields or professional texts. The weight of long-tail words can be dynamically adjusted according to the frequency of occurrence of the words. This helps the model better capture the meaning of uncommon words and improve the accuracy of eloquence.

[0085] Finally, these indicators are combined to calculate the input representation x_t at the current time step t to better capture the information in the text and provide more accurate and personalized eloquence training suggestions. The dynamic adjustment and weighting of these indicators can improve the adaptability and performance of the eloquence expression model. This more advanced input representation model fully considers various factors in the text to provide a more accurate and informative text representation. This can help the model better understand the semantic and emotional information of the text.

[0086] The effects that can be obtained based on the above content are:

[0087] 1. Focus: Introducing the focus variable allows the model to assign different levels of attention to different parts of the text at different time steps. This is very important for personalized eloquence training in invention patents, because the text content at certain moments may be more critical than at other moments and require higher attention. In this way, our invention can better adapt to the needs and situations of users and provide more targeted eloquence training suggestions.

[0088] 2. Fatigue: Considering the fatigue of users or text readers helps improve the user experience. During long-term eloquence training, users may feel fatigued, and their concentration and interest may decrease. By introducing the fatigue variable, we can adjust the intensity and content of eloquence training to ensure that users stay focused during training and improve the training effect.

[0089] 3. Exclamations: Considering the exclamations in the text allows the model to better identify the emotionally strong parts. In eloquence training, these emotionally strong parts can be key because they can help improve the appeal and expressiveness of eloquence. By introducing exclamation weights, we can ensure that the model pays more attention to these important emotional markers.

[0090] 4. Long-Tail Words: Some uncommon words may contain key information, especially in specific fields or professional texts. Introducing long-tail word weights allows the model to better capture the meaning of these uncommon words. In eloquence training, this is very important for eloquence covering a wide range of topics and professional terms.

[0091] Taking all these factors into consideration, we can analyze the text more comprehensively and understand the semantic and emotional information more accurately, thus providing more personalized and effective eloquence training suggestions and improving the user's eloquence. This is not only beneficial to the fields of education and communication, but also has broad application prospects in many fields such as business and entertainment.

[0092] (ii) At the sentiment classification layer, a more advanced model is introduced to integrate multiple features of eloquence expression and dynamically weight word embeddings of different words. This more advanced model will include sentiment-behavior dual attention RNN and attention mechanism to provide more refined sentiment classification results. The following is the updated high-level mathematical operation formula:

[0093] In this more advanced model, the output of the Sentiment-Behavior Dual-Attention RNN is introduced as input to incorporate multiple features of eloquence expression:

[0094] 1. Output of emotion-behavior dual attention RNN:

[0095] The hidden state sequence $H$ obtained by the emotion-behavior dual attention RNN will be divided into two parts, one for emotion classification and the other for considering the eloquence expression characteristics. This can be expressed as:

[0096] H Sentiment =[h Sentiment,1 ,h Sentiment,2 , ..., h Sentiment , T]

[0097] H Behavior =[h Behavior,1 ,h Behavior,2 , ..., h Behavior,T ];

[0098] H Sentiment: This is a sequence of hidden states related to sentiment, representing the sentiment information at each time step in the text. It is calculated by the Sentiment-Behavior Dual-Attention RNN. h at each time step Sentiment,t Contains the sentiment information of this time step, which is very important in sentiment analysis tasks.

[0099] H Behavior: This is the behavior-related hidden state sequence, which represents the behavior information at each time step in the text. It is also calculated by the emotion-behavior dual attention RNN. h at each time step Behavior,t Contains the behavior information of this time step, which is very important in the training of eloquence expression. Sentiment Contains the hidden state sequence for sentiment classification, H Behavior Contains the hidden state sequence used to take into account the expressive characteristics of eloquence.

[0100] 2. Sentiment Attention:

[0101] The sentiment classification attention mechanism is introduced to dynamically weight the sentiment hidden state sequence $H_{\text{Sentiment}}$ to improve the sensitivity to sentiment and consider the characteristics of sentiment expression, such as positive and negative sentiment. This process can be expressed as:

[0102] α Sentiment =SentimentAttention(H Sentiment );

[0103] α Sentiment: This is the sentiment-related attention weight, which is used to capture important information in the hidden state sequence H Sentiment. This weight is calculated by the SentimentAttention function, which ensures that more attention is paid to the sentiment-related parts in the sentiment analysis task.

[0104] 3. Expression Feature Attention:

[0105] The eloquence expression characteristic attention mechanism is introduced to dynamically weight the eloquence expression characteristics, such as phonation, loudness, frequency, timbre, coherence, and the integration of voice and emotion. This process can be expressed as:

[0106] α Expression =ExpressionFeatureAttention(X);

[0107] α Expression : This is the attention weight of the eloquence expression feature, which is used to capture important information in the input text sequence X. This weight is calculated by the ExpressionFeatureAttention function, which ensures that more attention is paid to the expression-related parts in eloquence expression training.

[0108] 4. Sentiment classification results:

[0109] Finally, the sentiment classification attention weight and the eloquence expression characteristic attention weight are comprehensively considered to fuse the sentiment classification and eloquence expression characteristics to obtain the final sentiment classification result Y. This process can be expressed as:

[0110] Y=FC(α Sentiment ⊙H Sentiment +α Expression ⊙X);

[0111] Y: This is the final sentiment classification result, which indicates the sentiment polarity of the text. It is calculated by the fully connected layer (FC), using the sentiment-related attention weight α Sentimen and the corresponding sentiment hidden state sequence H Sentiment, and the expression-related attention weight α Expression and the input text sequence X. By combining sentiment and expression information, sentiment classification can be performed more accurately, which is very important for personalized feedback of eloquence training. Where ⊙ represents element-by-element multiplication, and FC represents a multi-layer fully connected neural network.

[0112] This more advanced sentiment classification model not only focuses on emotional information, but also fully considers multiple characteristics of eloquence expression, such as phonation, loudness, frequency, timbre, coherence, voice and emotion fusion, positive and negative emotions, etc., to provide more accurate sentiment classification results. At the same time, the attention mechanism is used to dynamically adjust the weights to better capture important information in the text.

[0113] (III) Sentiment classification layer:

[0114] Finally, the hidden state sequence H is mapped to the sentiment polarity space through the fully connected layer to obtain the sentiment classification result Y. In this advanced model, we can introduce multi-layer fully connected neural networks and batch normalization techniques to increase the depth and expressiveness of the model.

[0115] Y=FC(H)

[0116] Here FC stands for multi-layer fully connected neural network.

[0117] The above has the following effects on the performance and functionality of the Eloquent Expression Model:

[0118] 1. Output of emotion-behavior dual attention RNN: Splitting the output of emotion-behavior dual attention RNN into emotion and eloquence expression helps to divide the model's tasks more clearly, thereby better capturing and understanding the emotion and eloquence expression characteristics in the text. This helps to provide more precise training feedback, allowing the model to better adapt to different tasks.

[0119] 2. Sentiment classification attention and eloquence expression feature attention: The introduction of these two attention mechanisms helps the model better focus on the important parts of sentiment and eloquence expression. Sentiment classification attention improves sentiment sensitivity and helps to classify sentiment more accurately, while eloquence expression feature attention allows the model to dynamically focus on different eloquence features according to different tasks. The introduction of these attention mechanisms helps improve the adaptability and performance of the model.

[0120] 3. Comprehensive sentiment classification results: By taking into account the sentiment classification attention weight and the eloquence expression characteristic attention weight, the model is able to integrate sentiment classification and eloquence expression characteristics to provide richer sentiment analysis results. This is very important for personalized feedback of eloquence training, because it not only tells users the sentiment of the text, but also provides information related to the expression method, such as voice characteristics and intonation.

[0121] 4. Dynamically adjust weights: The introduction of the attention mechanism allows the model to dynamically adjust weights to better capture important information in the text. This helps improve the flexibility and performance of the model because it can adjust the focus according to different texts and tasks.

[0122] These additions enrich the eloquence expression model, enabling it to more comprehensively process text emotions and expressions, and provide more accurate and personalized training feedback. This is of great significance for eloquence training applications, and can improve users' eloquence expression ability and sentiment analysis accuracy. The relevant technical innovation part of the invention patent may include these contents to protect this innovation.

[0123] 2. Behavioral prediction models in specific areas of eloquence.

[0124] Behavioral prediction models for specific areas of eloquence. This can be further complicated to more fully consider multiple dimensions of eloquence. Here is a more complex example model that includes multiple sub-models to capture different eloquence characteristics:

[0125] Feature extraction: Extract multiple features from the speaker’s speech, including text, audio, facial expression, speaking speed, voice pitch, coherence, grammatical correctness, etc. These features can be represented as X, where X = [text, audio, face, speaking speed, pitch, coherence, grammar, etc.].

[0126] Target value: Define a target value Y, which represents the measurement of multiple eloquence-related dimensions, such as speech effectiveness, voice quality, facial expression confidence, grammar score, etc. These are the values ​​we want to predict, which can be expressed as Y = [Y_1, Y_2, ..., Y_m].

[0127] Deep learning model: Use a multimodal deep learning model that includes the following components:

[0128] Text analysis sub-model: Use recurrent neural network (RNN) or convolutional neural network (CNN) to process speech text and predict text-related eloquence dimensions.

[0129] Audio analysis sub-model: Use convolutional neural networks or recurrent neural networks to process audio data and capture audio-related eloquence features.

[0130] Facial expression analysis sub-model: Use convolutional neural networks or facial feature extractors to analyze facial expression data and predict eloquence dimensions related to facial expressions.

[0131] Coherence and grammar analysis sub-model: Use natural language processing (NLP) technology to analyze the coherence and grammatical correctness of speech texts and predict coherence and grammar scores.

[0132] Comprehensive model: The outputs of the above sub-models are fused together to form the final eloquence dimension prediction. This can be achieved through a multi-layer fully connected neural network or an attention mechanism.

[0133] Training model: Use the data of known eloquent speakers and the corresponding measurement values ​​of eloquence-related dimensions to train the model and find the optimal parameters so that the model can best predict the values ​​of eloquence-related dimensions.

[0134] Predictive behavior: Once the model is trained, the speech features of new eloquent speakers can be input into the model and used to predict the values ​​of multiple eloquence-related dimensions, i.e., Y.

[0135] This more complex eloquence behavior prediction model takes into account multiple aspects of eloquence expression, including text, voice, facial expressions, coherence, grammar, etc., to provide more comprehensive and accurate training feedback. The model structure and mathematical formula will depend on the specific task and data, and can be further designed and tuned according to needs. This comprehensive model helps eloquent speakers to comprehensively improve their eloquence ability and better understand all aspects of their performance.

[0136] In one embodiment, after the voice collection device is activated to obtain the voice data of the eloquent speaker, the method further includes:

[0137] After preprocessing the voice data, the voice data is converted into text, and the text converted from the voice is displayed as the first display information, so that the speaker can understand the content of his expression and communication;

[0138] Among them, the pre-processing module can reduce noise, remove noise and enhance the clarity of speech signals through speech enhancement algorithms;

[0139] Performing semantic analysis on the text to obtain a semantic analysis result, determining whether there is a semantic error in the semantic analysis result, and determining the content of the semantic error;

[0140] According to the semantically erroneous content and the semantic analysis result, second display information displaying the semantically erroneous content and the semantic analysis result is generated and displayed, so that the eloquent speaker can understand the semantically erroneous content in his expression and communication.

[0141] Semantic analysis may be to divide the words in the text and then analyze the semantics. If the semantic analysis shows that the semantics are wrong, the wrong text position will be marked for display.

[0142] In one embodiment, the method for analyzing voice data based on the deep learning module is:

[0143] The preprocessed speech data is used as the model input and input into a pre-established speech deep learning model for learning and training to output speech features, which include speech clarity, speaking speed and tone.

[0144] Among them, the pre-built speech deep learning model is trained through a large number of sample packages, taking a large amount of speech data as sample input and taking speech features as sample output; the trained speech deep learning model can analyze the speech clarity, speaking speed and tone in the speech data, among which the tone can be distinguished according to the intonation, mainly including declarative tone, imperative tone, exclamatory tone, interrogative tone, etc.

[0145] Ultimately, the speech characteristics of the eloquent speaker can be displayed through voice playback combined with text display and graphic display, so that the eloquent speaker can understand the problems in his expression and communication based on the voice characteristics and make timely corrections.

[0146] The above-mentioned voice analysis part can mainly record and analyze the voice expressed by the eloquent speaker. If there is no special requirement, there is no need to record and analyze the voice of the viewer.

[0147] In one embodiment, an adaptive algorithm based on deep learning may be provided for eloquence training of speech data, and the specific steps include:

[0148] 1. Data representation and feature extraction:

[0149] Data representation: Let X represent the speech input, where X is a time series of a sound signal with a sampling frequency of fs; use Short-Time Fourier Transform (STFT) to convert the speech signal into a spectral representation to obtain the speech spectrogram; divide the spectrogram into multiple time windows and extract features for each window. These features can include spectral envelope, tone features, etc.

[0150] Among them, the continuous spectrum representation formula in Fourier transform is as follows:

[0151] X(t,f)=fx(τ)·ω(t-τ)·e -j2πfτ dτ;

[0152] Where X(t,f) represents the spectrum value at time t and frequency f;

[0153] x(τ) is the input speech signal, which represents the signal value at time τ. In Fourier transform, we convert this function from the time domain (time domain) to the frequency domain to obtain information about different frequency components;

[0154] w(t-τ) is a window function, which is used to window the signal. The window function is usually used to limit the effective part of the signal within a certain time period to avoid spectrum leakage in spectrum analysis. The window function usually has a specific shape, such as Hanning window or Blackman window, to meet different analysis requirements.

[0155] e^-j2πfτ: This is a complex exponential function used to describe the frequency components of a signal; where f represents frequency and τ represents time delay; this complex exponential function can represent the frequency information of a signal, and its amplitude and phase correspond to the amplitude and phase of the frequency component;

[0156] j represents an imaginary unit. The imaginary unit j is usually used in engineering and physics to represent the imaginary part of a complex number, satisfying j^2=-1.

[0157] The whole formula means that by performing Fourier transform on the input signal x(t), it is converted into the frequency domain representation X(t,f). At different times t and frequencies f, spectrum values ​​can be obtained, which represent the components of the signal at different frequencies. This is very useful in signal processing and spectrum analysis to analyze the frequency characteristics in the signal.

[0158] 2. Deep Learning Model:

[0159] Convolutional Neural Network (CNN) is used as part of the model to extract speech features. CNN can learn local and global features through convolutional layers and pooling layers.

[0160] The operation of the convolution layer is as follows (assuming there are multiple convolution kernels):

[0161] Conv i =σ(W i *X+b i );

[0162] Here Conv_i represents the output of the i-th convolutional layer, σ: This is the activation function, usually the sigmoid function or the more common ReLU (Rectified Linear Unit) function, which maps the output of the convolutional layer nonlinearly to introduce nonlinear properties; Wi and bi are the weights and biases of the convolutional layer, which are used for convolution operations; X: This is the input data, such as images or time series data. In convolutional neural networks, it is usually the pixel value of the image or the feature vector of the sequence data; bi: This is the bias of the convolutional layer, which is used to offset the output of the convolutional layer.

[0163] Add multiple layers of Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) layers on top of the model to capture the temporal dependencies of speech. The operation of LSTM is as follows:

[0164] h t =LSTM(X t ,ht-1 );

[0165] ht: This is the hidden state in the Long Short-Term Memory (LSTM) network, which represents the model state at time step t; LSTM is a Recurrent Neural Network (RNN) variant suitable for sequence data, which is used to process sequence data with long-distance dependencies;

[0166] Xt: This is the input data or features at time step t;

[0167] ht-1: This is the hidden state at time step t-1, which contains information from past time steps; LSTM uses hidden states to maintain and pass information to handle long-term dependencies in sequence data.

[0168] 3. Adaptive training algorithm:

[0169] During training, an adaptive method is used to adjust the model parameters based on the user's speech performance and feedback. If the user's speech performance is poor, the gradient descent method can be used to update the model parameters to reduce the loss function. The loss function can be the mean square error, etc.

[0170] Parameter update:

[0171] Among them, θnew: This is the new value of the model parameters, the result after a parameter update;

[0172] θold: This is the old value of the model parameter, which means the value before the parameter update;

[0173] η: This is the learning rate, a hyperparameter that controls the step size or rate of parameter updates. The choice of learning rate is very important for the model training process.

[0174] This is the gradient of the loss function with respect to the parameter θold, which represents the rate of change of the loss function in the parameter space. By calculating the gradient, we can determine how the loss function responds to changes in the parameters and update the parameters accordingly to minimize the loss function.

[0175] The above formulas are commonly used in deep learning to describe the operation of convolutional neural networks and recurrent neural networks, as well as the rules for parameter updates. They are key components of deep learning model training and optimization.

[0176] The above is just an example to illustrate the possibilities of deep learning models and adaptive algorithms for the specific field of eloquence. The actual model and algorithm design will depend on the specific eloquence training task and data. The complexity and specific mathematical formulas will vary depending on the task.

[0177] In one embodiment, the brain wave data includes brain wave signals and cortical signals. After the brain wave data is acquired, the acquired data may be subjected to noise removal, such as removing interference signals. If the brain wave acquisition device is activated and brain wave data is acquired, the method for analyzing the brain wave data is as follows:

[0178] Extract and label the signal features of EEG data, including frequency, amplitude and spectral density;

[0179] The brain wave data marked with signal characteristics are used as model input and input into a pre-established brain wave deep learning model for learning and training to output psychological characteristics and attention characteristics. The attention characteristics include concentration, memory ability and reaction speed.

[0180] Specifically, the speaker wears an electrode-type brain wave sensor and attaches the electrode to the scalp to collect brain wave signals; brain waves are electrical signals generated by the electrical activity of brain nerve cells, which can be recorded on the scalp through electrodes and transmitted to a computer or other device to analyze the brain wave signals using a deep learning algorithm. Among them, a brain wave deep learning model is established in advance. The model uses a deep learning algorithm to learn and train a large amount of brain wave data in advance, and learns the characteristic patterns of brain wave signals under different psychological states, such as frequency, amplitude, spectral density, distribution in the time domain or frequency domain, etc. These are also used to describe the characteristics of the pattern and changes of brain waves.

[0181] After analyzing the brainwave data through the brainwave deep learning model, the psychological state of the speaker can be determined based on the signal characteristics, where the psychological state includes tension, anxiety or fatigue. For example, during the communication process, if the speaker feels nervous, he or she may show a specific brainwave pattern, which may be different from a calm state or an anxious state; by learning the brainwave signals of these patterns, the user's current psychological state can be identified.

[0182] Since brain wave signals and cortical signals may behave differently under different psychological states, these features are learned from large amounts of data through deep learning algorithms. During the training phase, the algorithm uses labeled data, that is, data on the user's known psychological state, as a benchmark for learning; these labeled data are used to establish a correspondence between brain wave signal patterns and psychological states, allowing the algorithm to accurately identify unknown data.

[0183] However, people's psychological states vary greatly from person to person, so in practical applications, the model may need to take individual differences into account, such as adjusting the model's discrimination threshold based on each user's historical data to achieve more personalized recognition.

[0184] Similarly, brain wave data can also be used to analyze the talented speaker's attention characteristics such as concentration, memory ability, and reaction speed.

[0185] Attention refers to the level of attention a user pays to a specific task or content during communication or other activities. The user's attention can be assessed by analyzing relevant features in the user's brain wave signals, such as the frequency and amplitude ratio of alpha waves and beta waves. When the attention level is high, the amplitude of the alpha wave may be low, while the amplitude of the beta wave may be high.

[0186] Memory refers to the user's ability to remember and store information. The user's memory can be assessed by analyzing the patterns of brain wave signals during communication, such as event-related potentials (ERP). Different memory tasks may cause different brain wave patterns, and through deep learning algorithms, these patterns can be associated with the user's memory ability.

[0187] Reaction speed refers to the speed at which a user responds to external stimuli. The user's reaction speed can be assessed by recording the event-related potential (ERP) of the eloquent speaker during communication or other activities. ERP can produce specific brain wave responses within a few hundred milliseconds after stimulation, and users with fast reaction speed may show more obvious ERP.

[0188] The above process mainly relies on deep learning algorithms and data analysis technology. The EEG deep learning model can use deep learning algorithms to train a large amount of EEG data, learn the relationship between EEG signals and psychological state, concentration, memory, reaction speed, etc., so as to realize the recognition of these states. Data analysis technology is used to extract features from EEG signals, classify and predict them, and obtain corresponding results.

[0189] Deep learning of brain wave data can obtain psychological characteristics and attention characteristics, and provide personalized feedback to the eloquent speaker based on the psychological characteristics and attention characteristics, helping users improve their communication skills and eloquence. For example, the brain wave collector identifies the brain wave data corresponding to the eloquent speaker's nervousness, and analyzes that the eloquent speaker feels nervous when expressing and communicating. At this time, some relaxation methods can be provided to the user based on the eloquent speaker's nervousness, such as deep breathing or meditation.

[0190] When an eloquent speaker feels tired during communication, the brain wave collector can identify his tired emotions. At this time, the brain wave collector can provide him with some refreshing methods based on his tired emotions, such as drinking water or taking a rest.

[0191] When an eloquent speaker feels distracted during communication, the brain wave collector can identify the eloquent speaker's distracted emotions. At this time, the eloquent speaker can be provided with some methods to focus his attention based on the eloquent speaker's distracted emotions, such as prompting the eloquent speaker to review the key points of the communication, or reminding the eloquent speaker to take a deep breath and relax. Through the brain wave collection module, the eloquent speaker can be provided with personalized feedback to help the eloquent speaker improve his communication skills.

[0192] The above-mentioned brain wave analysis part can mainly collect and analyze the brain wave data of the eloquent speaker when expressing and communicating, and there is no need to collect the brain wave data of the viewer without special requirements.

[0193] In one embodiment, if eye movement data is collected, the method for analyzing the eye movement data is:

[0194] The eye movement data is used as model input and analyzed by a pre-established eye movement deep learning model to output gaze features; gaze features include the location of the focus point and the gaze time.

[0195] Eye tracking devices measure the user's concentration by tracking the user's eye movements. When users focus on a task or information, their gaze points will be concentrated in a specific area and their gaze will stay there for a long time. Eye tracking technology can record this information, including the location of the gaze point, the duration of the gaze, and the order of the gaze. By analyzing this data, it is possible to derive the user's concentration level in different tasks or situations.

[0196] Based on the above principles, the eye movement data of the eloquent speaker is collected, and the line of sight characteristics such as the visual search path, gaze position and gaze time of the eloquent speaker are analyzed based on the eye movement deep learning module. According to the line of sight characteristics, the eye situation of the eloquent speaker in the process of expression and communication is understood, and the eloquent speaker's attention is focused. Then, the eloquent speaker is helped to improve eye contact and eye communication, etc., which can guide the eloquent speaker to improve eye contact, enhance interaction with the audience, improve the effect and attractiveness of expression and communication, and thus improve the level of eloquence.

[0197] During the process of communication, the audience will also look at the eloquent speaker or the content displayed by the eloquent speaker, that is, the audience's eyes will also perform a series of scans and gazes. Therefore, during the process of the eloquent speaker's expression and communication, the audience's eye movement data can also be collected, and the audience's eye movement data can be analyzed through the eye movement deep learning model to understand the audience's sight characteristics. According to the audience's gaze point, gaze time and other characteristics, the audience's learning performance and viewing interest in this expression and communication process can be evaluated. According to the learning performance feedback from the audience, the expression and communication effect of the eloquent speaker can be indirectly understood, and then the eloquent speaker can be given corresponding guidance and suggestions.

[0198] For example, an eloquent communicator collects the viewer's eye movement data during the process of expression and communication. If it is found based on the eye movement data that the viewer's focus is not on the eloquent communicator or the content presented by the eloquent communicator, it means that the eloquent communicator's expression and communication are not effective, and the eloquent communicator is advised to enhance the attractiveness of his expression and communication.

[0199] In addition, in order to achieve more natural and intuitive user interaction, eye tracking technology can be combined with virtual reality technology. For example, in a virtual expression and communication environment, the speaker can control the turning of virtual slides through eye movements, or make eye contact with virtual audiences, thereby enhancing the immersion and realism of virtual expression and communication.

[0200] In one implementation, if the camera is activated to collect facial data, the method for analyzing the facial data is:

[0201] The facial data is used as the model input, and the facial data is analyzed based on the pre-established facial deep learning model to output emotional features.

[0202] The facial deep learning module also uses a large number of facial images as model input, and takes the emotional state represented by the user's facial expression in the image as model output. The model is learned and trained to obtain a facial deep learning model. The facial deep learning model can be used to analyze the user's facial expression and identify the user's emotional characteristics, such as happiness, sadness, anger, etc.; thus, the eloquence of the speaker can be evaluated based on the emotional characteristics.

[0203] Similarly, the facial data of the viewer can also be collected during the expression and communication process, and the emotional expression of the viewer when listening to the expression and communication can be analyzed to reflect the eloquence level of the talented communicator, so as to give corresponding guidance and opinions.

[0204] It should be noted that the training process of the above deep learning model has been disclosed in the prior art, and the specific training process of the model will not be described in detail here, as long as the model can achieve the corresponding purpose.

[0205] After different deep learning models analyze the performance characteristics of the expresser and / or the viewer, the performance characteristics are converted into feedback and the feedback is output to the user. For example, the following feedback can be provided to the user:

[0206] The user's attention span;

[0207] The user's gaze movement trajectory;

[0208] The user's facial expression changes;

[0209] The user's expression and communication content......

[0210] The above feedback helps the eloquent communicator understand his or her communication performance and improve it according to the suggested information. The method of pushing corresponding improvement suggestions according to the performance characteristics is: calling the information library to extract the suggested information corresponding to different performance characteristics and push it; the information library stores the suggested information related to different performance characteristics, giving the eloquent communicator personalized feedback.

[0211] Here is a specific example:

[0212] During the communication process, a user is not focused, often looks away from the communication draft, and has a dull facial expression. After analyzing the user's performance characteristics, the following feedback will be provided to the user:

[0213] The user’s attention span is not focused;

[0214] Users often look away from the presentation;

[0215] The user's facial expressions are dull;

[0216] Based on the feedback, users can take the following actions to improve their communication performance:

[0217] Be prepared before communicating, understand the content of the communication, and practice communicating;

[0218] During the communication process, stay focused and look at the audience;

[0219] Use facial expressions and gestures to express emotions during expressive communication;

[0220] Through feedback, users can improve their expression and communication skills.

[0221] The performance characteristics analyzed by the deep learning algorithm and the suggestion information corresponding to the performance characteristics can be displayed in a visual way; the visualization method includes at least one of the following methods: charts, animations and interactive games. Among them, animations and interactive games can be realized through virtual reality technology, that is, a virtual eloquent speaker or viewer is generated, and the virtual person can use animation to show the performance characteristics and suggestion information of the current eloquent speaker; or if the eloquent speaker's voice, concentration, memory, reaction speed, emotion, etc. are analyzed in the process of expression and communication and meet the pre-set good standards, the corresponding points can be accumulated, and the game can be won after reaching the preset score. The efficiency of eloquence training can be improved through animation, games, etc., making the eloquence training process more interesting.

[0222] Embodiment 2

[0223] This embodiment provides a personalized eloquence training system, which executes the personalized eloquence training method as described in the first embodiment; Figure 2 As shown, the personalized eloquence training system includes:

[0224] A collection module, which is used to activate the perception device of the eloquent speaker and / or the viewer after receiving the training request to collect the perception data corresponding to the eloquent speaker and / or the viewer during the eloquence training of the eloquent speaker, wherein the perception data is one of voice data, brain wave data, eye movement data or facial data, or a combination of two or more thereof;

[0225] A model analysis module, used to perform deep learning and analysis on the perception data according to a pre-established deep learning model, and output performance features corresponding to different perception data; the performance features include voice features, psychological features, attention features, sight features, and emotional features;

[0226] The suggestion push model is used to call the information library to filter the suggestion information corresponding to the different performance characteristics and push it.

[0227] The functions of each module in the system of the embodiment of the present invention can be found in the corresponding description of the above method, which will not be repeated here.

[0228] Embodiment 3

[0229] Figure 3 FIG. 2 shows a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device includes: a memory 910 and a processor 920, wherein the memory 910 stores a computer program that can be run on the processor 920. When the processor 920 executes the computer program, the personalized eloquence training method in the above embodiment is implemented. The number of the memory 910 and the processor 920 can be one or more.

[0230] The electronic device also includes:

[0231] The communication interface 930 is used to communicate with external devices and perform data exchange transmission.

[0232] If the memory 910, the processor 920 and the communication interface 930 are implemented independently, the memory 910, the processor 920 and the communication interface 930 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0233] Optionally, in a specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

[0234] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present invention when the program is executed by a processor.

[0235] An embodiment of the present invention further provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present invention.

[0236] An embodiment of the present invention also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided by the embodiment of the invention.

[0237] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0238] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0240] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0241] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0242] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, segment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present invention includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0243] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0244] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0245] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0246] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A personalized eloquence training method, It is characterized in that include: Collecting perception data corresponding to the eloquent speaker and / or the viewer during the eloquence training of the eloquent speaker, wherein the perception data is at least one of voice data, brain wave data, eye movement data or facial data; Based on a pre-established deep learning model, the perception data is subjected to deep learning and analysis to obtain performance features corresponding to different perception data; wherein the deep learning model includes a sentiment analysis model and a behavior prediction model; the performance features include voice features, psychological features, attention features, sight features, and emotional features; Among them, the method of sentiment analysis is to convert the perception data of digital signals into corresponding text data and then map them to the sentiment polarity space, which includes positive, negative and neutral; the perception data includes voice data, brain wave data, eye movement data and facial data, and convert the digital signals of each perception data into text data, including converting voice data into text, or converting the frequency and amplitude signals corresponding to the brain wave data into text data recognizable by the model; and realize text representation through attention mechanism, concentration, fatigue, interjections, long-tail words and frequency, that is, calculate the input representation x of the current time step t t , represents the weighted sum of each word embedding in the text to capture the information in the text: ; α t,i : is an attention weight, which represents the attention mechanism through the attention weight. The degree of attention to different word embeddings i in the text at time step t is calculated based on the previous hidden state and word embedding matrix to ensure that different words are paid attention to at different time steps; γ t : is a variable representing the level of concentration in the text at time step t, calculated by the Focus() function, which is used to dynamically adjust the degree of attention based on context and historical information; : is a variable that represents the fatigue of the user or text reader, calculated by the Fatigue() function; β t : is a variable representing the weight of the interjection in the text. The weight is dynamically adjusted according to the appearance of the interjection in the text. The Exclamations() function calculates the weight according to the appearance of the interjection in the text. ω t : is a variable representing the weight of long-tail words. The weight of long-tail words is dynamically adjusted according to the frequency of occurrence of the words. The weight is calculated according to the frequency of occurrence of the words through the LongTailWords() function; Calling the information database to filter the suggestion information corresponding to the different performance characteristics; Also includes: Using sentiment-related attention weights α Sentimen and the corresponding sentiment hidden state sequence H Sentiment , and the attention weight α Expression The final sentiment classification result Y is calculated through the fully connected layer and the input text sequence X is used to provide more accurate and personalized training feedback based on the final sentiment classification result Y. The final sentiment classification result Y is expressed as: ; in, represents element-by-element multiplication, FC represents a multi-layer fully connected neural network; the emotion hidden state sequence H is calculated by the emotion-behavior dual attention RNN Sentiment , which is used to represent the sentiment information at each time step in the text.

2. The personalized eloquence training method according to claim 1, It is characterized in that The perception data is voice data, and further includes: Preprocessing the voice data to generate preprocessed voice data; Converting the preprocessed voice data into text to generate first display information for displaying the text; Perform semantic analysis on the text to obtain semantic analysis results; Determine whether there is a semantic error in the semantic analysis result and determine the content of the semantic error; According to the semantically erroneous content and the semantic analysis result, second presentation information displaying the semantically erroneous content and the semantic analysis result is generated.

3. The personalized eloquence training method according to claim 1, It is characterized in that The analysis method of the voice data comprises: Based on the preprocessed speech data as model input, it is input into a pre-established speech deep learning model for learning and training and outputs speech features, wherein the speech features include speech clarity, speaking speed and tone.

4. The personalized eloquence training method according to claim 1, It is characterized in that The brain wave data includes brain wave signals and cortical signals; if the brain wave data is collected, the method for analyzing the brain wave data is: Extracting and marking signal features of the brain wave data, wherein the signal features include frequency, amplitude and spectral density; The brain wave data marked with the signal characteristics are used as model input and input into a pre-established brain wave deep learning model for learning and training, and psychological characteristics and attention characteristics are output. The attention characteristics include concentration, memory ability and reaction speed.

5. The personalized eloquence training method according to claim 1, It is characterized in that If the eye movement data is collected, the method for analyzing the eye movement data is: The eye movement data is used as a model input and analyzed by a pre-established eye movement deep learning model to output the sight line features; the sight line features include the focus point position and the gaze time.

6. The personalized eloquence training method according to claim 1, It is characterized in that If the facial data is collected, the method for analyzing the facial data is: The facial data is used as a model input, the facial data is analyzed based on a pre-established facial deep learning model, and the emotional features are output.

7. The personalized eloquence training method according to claim 1, It is characterized in that After filtering out the suggestion information corresponding to the performance characteristics, it also includes: The performance characteristics and the suggestion information are visually displayed; the visual display includes displaying using at least one of a chart, an animation, and an interactive game.

8. A personalized eloquence training system, It is characterized in that Execute the personalized eloquence training method as described in any one of claims 1 to 7; The system comprises: A collection module, used for collecting perception data corresponding to the eloquent speaker and / or the viewer during the eloquence training of the eloquent speaker, wherein the perception data is one of voice data, brain wave data, eye movement data or facial data, or a combination of two or more thereof; A model analysis module, used to perform deep learning and analysis on the perception data according to a pre-established deep learning model, and output performance features corresponding to different perception data; the performance features include voice features, psychological features, attention features, sight features, and emotional features; The suggestion push model is used to call the information library, filter the suggestion information corresponding to the different performance characteristics and push it.

9. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the personalized eloquence training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the personalized eloquence training method according to any one of claims 1 to 7 is implemented.