Feedback method, system, device and storage medium for eloquence training

Abstract

The present application proposes a feedback method, system, device and storage medium for eloquence training, the method comprising: obtaining multimodal data of the trainee during the current eloquence training; performing speech and non-verbal behavior analysis on the multimodal data to obtain the trainee's speech intensity change data, intonation pattern data and synchronization data of non-verbal behavior and speech; performing eloquence quality analysis on the speech intensity change data, intonation pattern data and synchronization data of non-verbal behavior and speech to obtain the trainee's multidimensional eloquence quality index during the current eloquence training process; based on the multidimensional eloquence quality index, generating and outputting targeted eloquence improvement suggestions for the trainee. The present application can provide personalized feedback on the trainee's eloquence training performance in a timely and effective manner, realizes training guidance based on the trainee's specific performance, and can effectively improve the trainee's eloquence training efficiency and eloquence skills.

Description

Technical Field

[0001] The present application relates to the technical field of speech and eloquence industry, and in particular to a feedback method, system, device and storage medium for eloquence training. Background Art

[0002] In the existing field of eloquence training, traditional eloquence training methods usually rely on face-to-face instruction or use basic recording and playback technology to help individuals improve their eloquence skills. However, although these eloquence training methods can basically improve the eloquence skills of individuals, these eloquence training methods usually have the following defects:

[0003] (1) Lack of personalized real-time feedback: Public speaking trainers may not be able to capture all the subtle changes in the trainees’ speech and non-verbal behaviors in real time, resulting in the inability to effectively provide customized public speaking training feedback based on the unique needs of each individual, making the public speaking training feedback for the trainees lack of personalization and real-time nature.

[0004] (2) Inefficient eloquence training: The interaction between eloquence trainers and trainees is usually unstructured and relies on face-to-face instruction. This requires eloquence trainers to spend a lot of time evaluating and providing feedback on trainees’ eloquence training performance and giving targeted suggestions for improvement, which may lead to inefficient eloquence training for trainees. Summary of the invention

[0005] The present application provides a feedback method, system, device and storage medium for eloquence training to solve the problems existing in the related technologies. The technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a feedback method for eloquence training, comprising:

[0007] Acquiring multimodal data of the trainee during current eloquence training, wherein the multimodal data includes audio data and video data;

[0008] Performing speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity change data, intonation pattern data, and synchronization data between non-verbal behavior and speech of the trainee;

[0009] Performing eloquence quality analysis on the voice intensity variation data, the intonation pattern data, and the synchronization data of the non-verbal behavior and voice to obtain a multi-dimensional eloquence quality index of the trainee in the current eloquence training process;

[0010] Based on the multi-dimensional eloquence quality index, generating targeted eloquence improvement suggestions for the trainee;

[0011] Output the targeted eloquence improvement suggestions.

[0012] In one embodiment, performing speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech of the trainee includes:

[0013] Performing speech dynamic range analysis on the multimodal data to obtain the speech intensity change data;

[0014] Performing intonation pattern recognition analysis on the multimodal data to obtain the intonation pattern data;

[0015] Non-verbal behavior synchronization analysis is performed on the multimodal data to obtain synchronization data of the non-verbal behavior and speech.

[0016] In one embodiment, the speech intensity change data, the intonation pattern data, and the synchronization data of the non-verbal behavior and speech are analyzed for eloquence quality to obtain the multi-dimensional eloquence quality index of the trainee in the current eloquence training process, including:

[0017] Performing eloquence quality analysis on the speech intensity change data to obtain a speech coherence index;

[0018] Performing eloquence quality analysis on the intonation pattern data to obtain an emotional expression index;

[0019] Performing eloquence quality analysis on the synchronization data of the non-verbal behavior and speech to obtain a pause pattern index;

[0020] The multi-dimensional eloquence quality index is obtained based on the speech coherence index, the emotion expression index and the pause pattern index.

[0021] In one embodiment, performing an eloquence quality analysis on the speech intensity variation data to obtain a speech coherence index comprises: performing an eloquence quality analysis on the speech intensity variation data to determine speech intensities in each of two adjacent continuous time windows in the speech intensity variation data; obtaining the speech coherence index based on the speech intensities in each of the two adjacent continuous time windows;

[0022] Performing an eloquence quality analysis on the intonation pattern data to obtain an emotional expression index includes: performing an eloquence quality analysis on the intonation pattern data to obtain a change in an intonation pattern recognition value of each frame of audio in the intonation pattern data; and obtaining the emotional expression index based on the change in the intonation pattern recognition value of each frame of audio;

[0023] Performing an eloquence quality analysis on the synchronization data of the non-verbal behavior and speech to obtain a pause pattern index comprises: performing an eloquence quality analysis on the synchronization data of the non-verbal behavior and speech to obtain each pause area of ​​the trainee in the current eloquence training process; and obtaining the pause pattern index based on the ratio of the duration of each pause area to the total eloquence practice time.

[0024] In one embodiment, generating targeted eloquence improvement suggestions for the trainee based on the multi-dimensional eloquence quality index includes:

[0025] If it is determined that the speech coherence index is lower than a first threshold value based on the multi-dimensional eloquence quality indicator, generating eloquence improvement suggestions for improving speech coherence for the trainee;

[0026] If it is determined that the emotional expression index is lower than a second threshold value based on the multi-dimensional eloquence quality index, generating eloquence improvement suggestions for adjusting emotional expression for the trainee;

[0027] If it is determined that the pause pattern index is lower than a third threshold based on the multi-dimensional eloquence quality indicator, an eloquence improvement suggestion for optimizing the pause pattern is generated for the trainee.

[0028] In one implementation, outputting the targeted eloquence improvement suggestion includes:

[0029] Obtain data on trainees’ preferences for various feedback modes;

[0030] Obtaining an effectiveness score of each feedback mode in the multiple feedback modes;

[0031] Determining an optimal feedback mode based on the preference data and the effectiveness score of each feedback mode;

[0032] The targeted eloquence improvement suggestions are output according to the optimal feedback mode.

[0033] In one embodiment, the method further comprises:

[0034] Based on the targeted eloquence improvement suggestions, formulate personalized eloquence training content for the trainee, wherein the eloquence training content includes several training plans carried out in chronological order;

[0035] Dynamically monitor the training progress of the trainee and the performance data of the trainee during the eloquence training process;

[0036] Based on the training progress and the performance data, the subsequent training plan of the eloquence training content is dynamically adjusted.

[0037] In a second aspect, the embodiment of the present application further provides a feedback system for eloquence training, characterized in that it includes:

[0038] An analysis unit is used to obtain multimodal data of a trainee during current eloquence training, wherein the multimodal data includes audio data and video data; perform speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity variation data, intonation pattern data, and synchronization data between non-verbal behavior and speech of the trainee; perform eloquence quality analysis on the speech intensity variation data, the intonation pattern data, and the synchronization data between non-verbal behavior and speech to obtain multi-dimensional eloquence quality indicators of the trainee during the current eloquence training process;

[0039] A feedback unit is used to generate targeted eloquence improvement suggestions for the trainee based on the multi-dimensional eloquence quality index; and output the targeted eloquence improvement suggestions.

[0040] In one embodiment, the feedback unit is further used for:

[0041] Based on the targeted eloquence improvement suggestions, formulate personalized eloquence training content for the trainee, wherein the eloquence training content includes several training plans carried out in chronological order;

[0042] Dynamically monitor the training progress of the trainee and the performance data of the trainee during the eloquence training process;

[0043] Based on the training progress and the performance data, the subsequent training plan of the eloquence training content is dynamically adjusted.

[0044] In a third aspect, an embodiment of the present application further provides a computer device, comprising: a memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned aspects, wherein the memory and the processor communicate with each other through an internal connection path.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is implemented.

[0046] The advantages or beneficial effects of the above technical solution include at least:

[0047] In the present application, by performing speech and non-verbal behavior analysis on the multimodal data of the trainee during the current eloquence training, the trainee's speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech are obtained, and then the speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech are analyzed for eloquence quality, so as to obtain the multidimensional eloquence quality index of the trainee in the current eloquence training process, and finally, based on the multidimensional eloquence quality index, targeted eloquence improvement suggestions are generated for the trainee, and the targeted eloquence improvement suggestions are output. All subtle changes in the trainee's speech and non-verbal behavior during the current eloquence training process can be captured in real time, and the trainee's eloquence quality indicators during the current eloquence training process can be accurately extracted through these changes. Through these eloquence quality indicators, the trainee's eloquence training performance can be evaluated and fed back in a structured manner, and targeted eloquence improvement suggestions can be given, and personalized feedback can be given to the trainee's eloquence training performance in a timely and effective manner, so as to realize training guidance for the trainee's specific performance, thereby effectively improving the trainer's eloquence training efficiency and eloquence skills. At the same time, this application does not require manual intervention by eloquence trainers, thereby reducing the workload of eloquence trainers.

[0048] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0050] Figure 1 An example flow chart of a feedback method for eloquence training provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of another feedback method for eloquence training provided in an embodiment of the present application;

[0052] Figure 3 A structural block diagram of a feedback system for eloquence training provided in an embodiment of the present application;

[0053] Figure 4 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0055] Figure 1 A flow chart of a feedback method for eloquence training according to an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps:

[0056] S110, obtaining multimodal data of the trainee during current eloquence training.

[0057] In one embodiment, the multimodal data may include, but is not limited to, audio data and video data.

[0058] As an example, during the current eloquence training, the trainee can use a microphone to perform eloquence training and use a video camera to shoot a real-time eloquence training video. That is, in this example, the audio data in the multimodal data can be the audio data collected by the microphone, and the video data in the multimodal data can be the video data collected by the video camera.

[0059] As an example, multimodal data can be encrypted in real time, such as using the AES-256 algorithm to ensure the security of multimodal data during transmission. At the same time, a network security transmission channel can be built, such as using protocols such as SSL / TLS to protect the privacy and integrity of multimodal data when it is transmitted on the Internet. When multimodal data needs to be decrypted, it can be synchronously decrypted through distributed edge nodes to ensure the consistency and security of multimodal data, while also reducing network load.

[0060] As an example, after obtaining multimodal data, the acquired multimodal data can be preprocessed in real time through distributed edge nodes, where the preprocessing can include basic operations such as denoising and signal enhancement to improve the quality of audio data and video data, prepare data for more complex analysis, and further provide the accuracy of subsequent feedback.

[0061] S120, performing speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity variation data, intonation pattern data, and synchronization data between non-verbal behavior and speech of the trainee.

[0062] In one implementation, the implementation process of step S120 may include the following steps:

[0063] S121. Perform voice dynamic range (VDR) analysis on the multimodal data to obtain voice intensity change data.

[0064] In a specific implementation, VDR analysis may be performed on the audio data in the multimodal data to obtain the speech intensity change data.

[0065] Exemplarily, the following formula (1) may be used to perform VDR analysis on the audio data to obtain the speech intensity variation data.

[0066]

[0067] Among them, VDR(t1) represents the speech intensity corresponding to time t1 during the eloquence training of the trainee, which can be expressed in decibels (dB); P(t1) represents the instantaneous power of the audio data at time t1.

[0068] That is, in the above example, the maximum instantaneous power and the minimum instantaneous power in the audio data are decoded and measured, and the ratio between the two is used to dynamically reflect the change in the trainee's voice intensity. The maximum instantaneous power and the minimum instantaneous power can be obtained by processing the audio data using existing voice signal processing technology, and this embodiment will not be repeated here.

[0069] In this embodiment, by executing step S121, the trainee's voice intensity change data can be decoded from the audio data to measure the trainee's voice intensity change range during the current eloquence training process, which can truly reflect the dynamics of the trainee's expression, so as to facilitate subsequent effective personalized feedback on the trainee's eloquence training performance.

[0070] S122. Performing Intonation Pattern Recognition (IPR) analysis on the multimodal data to obtain intonation pattern data.

[0071] In a specific implementation, IPR analysis may be performed on the audio data in the multimodal data to obtain the intonation pattern data.

[0072] Exemplarily, the following formula (2) may be used to perform IPR analysis on the audio data to obtain the intonation pattern data.

[0073]

[0074] Wherein, IPR(f) represents the trainee's intonation pattern recognition value at frequency f, wherein the intonation pattern includes but is not limited to question, statement or emphasis; f represents a specific frequency component; s(n) is the voice signal value of the nth sampling point in the audio data; N is the total number of sampling points for sampling the voice signal in the audio data, 1≤n≤N. Wherein, through the correspondence between the preset intonation pattern recognition value and the intonation pattern, the intonation pattern corresponding to the corresponding intonation pattern recognition value can be determined.

[0075] As an example, J in the above formula (2) is a key parameter for understanding and classifying the trainee's intonation pattern, which can take a fixed value based on phonetics, for example, it can take a value of 100 Hz, 200 Hz, etc. based on the fundamental frequency of human speech. In practical applications, the frequency spectrum in the audio data can also be analyzed in real time to dynamically select the frequency component with the highest energy or the most information in the current speech sample of the audio data as J.

[0076] That is, in the above example, the main frequency components in the speech signal of the audio data are decoded, for example, J in the speech signal of the audio data is determined by Fourier analysis decoding to identify the trainer's questions, statements or other intonation patterns.

[0077] In this embodiment, by executing step S122, the trainer's intonation pattern data can be decoded from the audio data to identify and classify the trainer's intonation pattern, so as to facilitate subsequent effective personalized feedback on the trainer's eloquence training performance.

[0078] S123. Perform nonverbal synchronization (NVS) analysis on the multimodal data to obtain synchronization data between nonverbal behavior and speech.

[0079] In a specific implementation, NVS analysis may be performed on the audio data and video data in the multimodal data to obtain synchronization data of the non-verbal behavior and the speech.

[0080] Exemplarily, the following formula (3) may be used to perform NVS analysis on the audio data and the video data to obtain synchronization data of the non-verbal behavior and speech.

[0081]

[0082] Among them, NVS(t1) represents the non-verbal behavior synchronization index corresponding to time t1 during the eloquence training of the trainee, g(t1) is the normalized non-verbal behavior signal of the video data at time t1 (such as normalized gesture action signal, normalized expression signal, etc.), v(t1) represents the normalized speech signal of the audio data at time t1, T1 represents the analysis window duration for analyzing the audio data and video data, 1≤t1≤T1.

[0083] That is, in the above example, the coordination between the trainee's non-verbal behavior and speech is evaluated by decoding and analyzing the time alignment between the speech signal in the audio data and the non-verbal behavior signal in the video data. Among them, the normalized non-verbal behavior signal can be obtained by capturing the non-verbal behavior signal in the video data using existing video image processing technology and then performing normalization processing, which will not be described in detail in this embodiment. The normalized speech signal can be obtained by capturing the speech signal in the audio data using existing speech signal processing technology and then performing normalization processing, which will not be described in detail in this embodiment.

[0084] As an example, when it is necessary to consider the impact of speech and non-verbal behavior on the non-verbal behavior synchronization index, a corresponding parameter model can be set to count the weight parameters of different speech and non-verbal behavior on the non-verbal behavior synchronization index, so that appropriate weight parameters can be selected for speech and non-verbal behavior. For example, the weight parameter of g(t1) is set to α, and the weight parameter of v(t1) is set to β, α+β=1. In this example, g'(t)=g(t1)×α, v′(t)=v(t1)×β, g'(t) replaces g(t1), v'(t) replaces v(t1), and substituting into the above formula (3) can calculate NVS(t1).

[0085] In this embodiment, by executing step S123, the synchronization data of the trainee's non-verbal behavior and voice can be decoded from the audio data and video data to analyze the synchronization of the trainee's non-verbal behavior (such as gestures, expressions, etc.) and voice, and evaluate the coordination of the trainee's non-verbal behavior and voice in the current eloquence training process, so as to facilitate the subsequent effective personalized feedback on the trainee's eloquence training performance.

[0086] It should be understood that the above steps S121 to S123 can be performed synchronously. For example, a graphics processing unit (GPU) or a customized digital signal processor (DSP) can be used to synchronously perform the above steps S121 to S123.

[0087] In the present application, by executing step S120, the voice data and non-verbal behavior data of the trainee during the eloquence training process can be decoded in real time based on multimodal data, so that the eloquence quality indicators of the trainee in the current eloquence training process can be accurately extracted from these data later, so that personalized feedback can be provided effectively and timely on the trainee's eloquence training performance.

[0088] S130, performing eloquence quality analysis on the voice intensity variation data, intonation pattern data, and synchronization data between non-verbal behavior and voice, and obtaining a multi-dimensional eloquence quality index of the trainee in the current eloquence training process.

[0089] In one implementation, the implementation process of step S130 may include the following steps:

[0090] S131. Analyze the speech intensity change data to obtain the speech coherence index. e Inde x .CI).

[0091] In specific implementation, the speech intensity variation data may be subjected to eloquence quality analysis to determine the speech intensity in each of two adjacent continuous time windows in the speech intensity variation data; and then CI may be obtained based on the speech intensity in each of two adjacent continuous time windows.

[0092] Exemplarily, the CI may be obtained according to the following formula (4) based on the speech intensity in each of two adjacent continuous time windows.

[0093]

[0094] in, and It is represented by the speech intensity in two adjacent continuous time windows in the speech intensity change data, T2 is represented by the total number of continuous time windows in the speech intensity change data, and 1≤t2≤T2-1.

[0095] That is, in this example, the speech intensity data can be used to calculate the speech intensity change within a continuous time window to evaluate the coherence of the trainee's speech flow during the current eloquence training process.

[0096] In this embodiment, by executing step S131, the eloquence quality analysis of the trainee's current eloquence training performance can be performed through the speech intensity change data, such as analyzing the coherence and fluency of the trainee's speech flow during the current eloquence training process to obtain the trainee's one-dimensional eloquence quality index, namely CI. The CI can reflect the coherence of the trainee's speech during the current eloquence training process.

[0097] S132. Analyze the eloquence quality of the intonation pattern data to obtain the Emotional Expression Index (EEI).

[0098] In specific implementation, the intonation pattern data can be subjected to eloquence quality analysis to obtain the variation of the intonation pattern recognition value of each frame of audio in the intonation pattern data; and then the EEI can be obtained based on the variation of the intonation pattern recognition value of each frame of audio.

[0099] Exemplarily, the EEI may be obtained according to the following formula (5) based on the change in the intonation pattern recognition value of each frame of audio.

[0100]

[0101] Where, ΔF l It is represented as the change of the intonation pattern recognition value of the lth audio frame in the intonation pattern data, σ is represented as the standard deviation function, L is represented as the total number of audio frames in the intonation pattern data, 1≤l≤L.

[0102] That is, in this example, the emotional fluctuation of the trainee during the current eloquence training process can be evaluated through standard deviation analysis of the intonation pattern data based on changes in the intonation pattern recognition value.

[0103] In this embodiment, by executing step S132, the eloquence quality analysis of the trainee's current eloquence training performance can be performed through the intonation pattern data, such as analyzing the pitch changes of the trainee's voice during the current eloquence training process to obtain the trainee's one-dimensional eloquence quality index, namely EEI. The EEI can reflect the trainee's emotional expression ability during the current eloquence training process.

[0104] S133, analyze the eloquence quality of non-verbal behavior and speech synchronization data to obtain the pause pattern index (Paus e Patter n Inde x , PPI).

[0105] In specific implementation, the synchronization data of non-verbal behavior and speech can be used to analyze the eloquence quality to obtain each pause area of ​​the trainee in the current eloquence training process; and then the PPI is obtained based on the ratio of the duration of each pause area to the total eloquence practice time.

[0106] For example, the PPI can be obtained based on the ratio of the duration of each pause zone to the total time of eloquence practice according to the following formula (6).

[0107]

[0108] Among them, p mIt is expressed as the ratio of the duration of the mth pause zone to the total time of eloquence practice, and M is represented by the total number of pause zones, 1≤m≤M.

[0109] Exemplarily, each pause area during the current eloquence training process can be determined by the following method:

[0110] From the synchronization data of the non-verbal behavior and speech, select the non-verbal behavior synchronization index that is lower than the preset threshold (which can be set according to actual needs), and then determine the time position t1 where the selected non-verbal behavior synchronization index is located. Then, the corresponding speech recognition technology can be used to mark and measure the natural pause filler words (such as um, uh, etc.), repeated words (such as "for example" repeated more than twice in a row) and repeated actions (such as the same waving action repeated more than twice in a row) in the corresponding time period, and finally determine the duration of the pause zone according to the corresponding time positions of these natural pause filler words, repeated words and repeated actions. Among them, the time value range of the corresponding time period can be: t1-T3≤t≤t1+T3, among which T3 can be set according to actual needs, such as a value of 10 seconds, which is not limited in this embodiment.

[0111] For example, the time position t1 corresponding to a non-verbal behavior synchronization index below the preset threshold is: 00:00:13 during the eloquence training period. When the value is 10 seconds, the corresponding time period is: 00:00:03-00:00:23. Then the corresponding speech recognition technology can be used to mark and measure the natural pause filler words, repeated words and repeated actions during this period of 00:00:03-00:00:23. If during this period of 00:00:03-00:00:23, the time position corresponding to the natural pause filler word "uh" is identified as 00:00:08-00:00:14, and there are no other repeated words and repeated actions, then it can be determined that the duration of a pause zone is 00:00:08-00:00:14.

[0112] Alternatively, if during the period of 00:00:03-00:00:23, the time position corresponding to the natural pause filler word "uh" is identified as 00:00:08-00:00:14, the time position corresponding to the repeated waving action is: 00:00:18-00:00:22, and there are no other repeated words, then it can be determined that the duration of the two pause areas are 00:00:08-00:00:14 and 00:00:18-00:00:22 respectively.

[0113] Alternatively, if within the period of 00:00:03-00:00:23, the time position corresponding to the natural pause filler word "uh" is identified as 00:00:08-00:00:14, the time position corresponding to the repeated waving action is: 00:00:12-00:00:20, and the time position corresponding to the repeated word is: 00:00:16-00:00:23, then it can be determined that the duration of a pause zone is 00:00:08-00:00:23.

[0114] That is, in this example, the frequency and duration of pauses of the trainee during the current eloquence training process can be analyzed through the synchronization data of non-verbal behavior and speech, and the pause strategy used by the trainee during the current eloquence training process can be evaluated.

[0115] In this embodiment, by executing step S133, the eloquence quality analysis of the trainee's current eloquence training performance can be performed through the synchronization data of non-verbal behavior and speech, such as analyzing the frequency and duration of pauses of the trainee in the current eloquence training process to quantify the distribution and frequency of pauses of the trainee in the current eloquence training process, and obtaining the trainee's one-dimensional eloquence quality index, namely PPI. The PPI can reflect the appropriateness of the pause strategy used by the trainee in the current eloquence training process.

[0116] It should be understood that the above steps S131 to S133 can be performed simultaneously. For example, a lightweight machine learning model (such as a decision tree and a lightweight neural network) can be used to synchronously perform the above steps S131 to S133 through an AI-based analysis algorithm. This can significantly reduce computing requirements and costs while ensuring the quality of the analysis.

[0117] S134. Based on CI, EEI and PPI, a multidimensional eloquence quality index is obtained.

[0118] It can be understood that the multi-dimensional eloquence quality index may include but is not limited to: CI, EEI and PPI.

[0119] In the present application, by executing step S130, the speech intensity change data, intonation pattern data, and the synchronization data of non-verbal behavior and speech can be analyzed for eloquence quality, and key multi-dimensional eloquence quality indicators such as CI, EEI and PPI can be extracted from these decoded data in real time, so that personalized feedback on the trainee's eloquence training performance can be provided in a timely and effective manner based on these eloquence quality indicators.

[0120] S140. Generate targeted eloquence improvement suggestions for trainees based on multi-dimensional eloquence quality indicators.

[0121] In one embodiment, if it is determined that the CI is lower than a first threshold based on the multi-dimensional eloquence quality index, an eloquence improvement suggestion for improving speech coherence is generated for the trainee.

[0122] Exemplarily, when the CI is lower than the first threshold, it indicates that the trainee's speech continuity in the current eloquence training process is poor. For example, the trainee has frequent pauses and sentence breaks in the current eloquence training process. In this case, eloquence improvement suggestions for improving speech coherence can be generated for the trainee.

[0123] For example, the trainee may be provided with eloquence improvement suggestions for improving speech coherence through one of the following three feedback modes:

[0124] Text feedback mode, such as "Try to transition smoothly between sentences and avoid unnecessary pauses."

[0125] Audio feedback mode, such as playing a sample of fluent speaking training, pointing out the specific areas for improvement.

[0126] Visual chart feedback mode, such as showing a eloquence training rhythm chart, marking areas with excessive pauses, and pointing out ways to improve these areas.

[0127] In one embodiment, if it is determined that the EEI is lower than the second threshold based on the multi-dimensional eloquence quality index, an eloquence improvement suggestion for adjusting emotional expression is generated for the trainee.

[0128] Exemplarily, when the EEI is lower than the second threshold, it indicates that the trainee has insufficient emotional expression in the current eloquence training process. For example, the trainee's emotional expression in the current eloquence training process is too single and lacks the necessary emotional fluctuations to reinforce key information. In this case, eloquence improvement suggestions for adjusting emotional expression can be generated for the trainee.

[0129] For example, the trainee can be provided with speech improvement suggestions for adjusting emotional expression through one of the following three feedback modes:

[0130] Text feedback mode, such as "Try to raise the pitch and volume when discussing important points to attract the audience's attention."

[0131] The voice feedback model, such as providing an eloquence training sample with rich emotional expression, focuses on the part that describes the emotional changes.

[0132] Visual chart feedback mode, such as using an emotion intensity chart to show the emotional ups and downs of the entire eloquence training, pointing out the parts that need to strengthen emotional expression.

[0133] In one embodiment, if it is determined that the PPI is lower than the third threshold based on the multi-dimensional eloquence quality index, an eloquence improvement suggestion for optimizing the pause pattern is generated for the trainee.

[0134] Exemplarily, when the PPI is lower than the third threshold, it means that the trainee has problems with inappropriate or too long pauses in the current eloquence training process. For example, the trainee pauses for too long or the pause position is inappropriate in the current eloquence training process, which affects the natural fluency of the eloquence training. In this case, eloquence improvement suggestions for optimizing the pause pattern can be generated for the trainee.

[0135] For example, the trainee may be provided with speech improvement suggestions for optimizing pause patterns through one of the following three feedback modes:

[0136] Text feedback patterns, such as “Try to pause between paragraphs and avoid pausing within a sentence.”

[0137] Audio feedback modes, such as demonstrating correct pause placement and duration.

[0138] Visual graphical feedback modes, such as displaying a pause distribution graph, highlighting locations of inappropriate pauses.

[0139] It should be understood that the first threshold, the second threshold and the third threshold can be set according to actual needs, and this embodiment does not limit this.

[0140] In the present application, by executing step S140, targeted eloquence improvement suggestions can be provided to the trainee in real time, so that the trainee can instantly obtain specific feedback on his / her eloquence training performance and immediately take corresponding measures to improve it, thereby achieving training guidance based on the trainee's specific performance, thereby effectively improving the eloquence skills of the trainee.

[0141] S150. Output the targeted eloquence improvement suggestions.

[0142] In one implementation, the implementation process of step S150 may include the following steps:

[0143] S151. Obtain trainees' preference data for multiple feedback modes.

[0144] As an example, the preference data may be set by the trainer before the current eloquence training, or may be obtained based on corresponding data of the trainer's historical selection feedback mode, which is not limited in this embodiment.

[0145] In a specific implementation, in the preference data, a preference weight may be assigned to each feedback mode to indicate the trainee's preference degree.

[0146] In a specific implementation, the multiple feedback modes may include but are not limited to: a text feedback mode, a sound feedback mode, and a visual chart feedback mode.

[0147] S152. Obtaining the effectiveness score of each feedback mode among the multiple feedback modes.

[0148] As an example, the historical effect data of each feedback mode can be collected, and the historical effect data includes the improvement degree and feedback satisfaction of different trainees; then, based on the historical effect data of each feedback mode, the effectiveness of each feedback mode is scored to obtain the effectiveness score of each feedback mode, so as to reflect the effectiveness of each feedback mode in actual teaching. That is, the effectiveness score of each feedback mode can be obtained by statistical analysis of past usage data.

[0149] S153. Determine the best feedback mode based on the preference data and the effectiveness score of each feedback mode.

[0150] As an example, based on the preference data and the effectiveness score (also called response score) of each feedback mode, the comprehensive score of each feedback mode can be determined, and then the feedback mode with the highest comprehensive score is selected as the best feedback mode. This process can be implemented by following the following formula (7).

[0151]

[0152] Among them, F opt Represents the optimal feedback mode, R y It is represented as the response score of feedback mode y, u y Expressed as the preference weight of feedback mode y.

[0153] In this example, the comprehensive score of each feedback mode is determined by formula (7), which can reflect the comprehensive utility of each feedback mode, taking into account both the improvement effect of each feedback mode and the trainee's preference for each feedback mode.

[0154] For example, it is assumed that the effectiveness scores and preference weights corresponding to the text feedback mode (text), the voice feedback mode (audio), and the visual chart feedback mode (visual) are as follows:

[0155] text:R text =0.8.u text =0.5

[0156] audio:R audio =0.9,u audio =0.3

[0157] visual:R visual =0.85.Nvisual =0.7

[0158] Calculate the comprehensive score of each feedback mode:

[0159] text:0.8×0.5=0.4

[0160] audio:0.9×0.3=0.27

[0161] visual:0.85×0.7=0.595

[0162] In this case, the visual chart feedback mode has the highest comprehensive score and can be regarded as the best feedback mode.

[0163] S154. Output the targeted eloquence improvement suggestions based on the optimal feedback model.

[0164] In this embodiment, by executing steps S151 to S154, the current feedback mode that best suits the trainee can be intelligently selected in combination with the trainee's personal preferences and the actual teaching effects of various feedback modes. This is not only conducive to improving the trainee's learning efficiency, but also can increase the trainee's satisfaction and acceptance of the targeted eloquence improvement suggestions.

[0165] In the present application, in step S150, the targeted eloquence suggestions can be output through a variety of feedback modes, and the most suitable feedback mode can be provided according to the trainee's preferences and specific situations, which can ensure the trainee's participation in eloquence training and help improve the trainee's learning effect.

[0166] In summary, in the feedback method for eloquence training provided in the present application, by performing speech and non-verbal behavior analysis on the multimodal data of the trainee during the current eloquence training, the trainee's speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech are obtained, and then the speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech are analyzed for eloquence quality to obtain the multidimensional eloquence quality index of the trainee in the current eloquence training process. Finally, based on the multidimensional eloquence quality index, targeted eloquence improvement suggestions are generated for the trainee, and the targeted eloquence improvement suggestions are output. All subtle changes in the trainee's speech and non-verbal behavior during the current eloquence training process can be captured in real time, and the eloquence quality indicators of the trainee in the current eloquence training process can be accurately extracted through these changes. Through these eloquence quality indicators, the trainee's eloquence training performance can be evaluated and fed back in a structured manner, and targeted eloquence improvement suggestions can be given. Personalized feedback on the trainee's eloquence training performance can be given in a timely and effective manner, and training guidance based on the trainee's specific performance is achieved, thereby effectively improving the trainer's eloquence training efficiency and eloquence skills. At the same time, the feedback method for eloquence training provided by the present application does not require manual intervention by eloquence trainers, thereby reducing the workload of eloquence trainers.

[0167] In an applicable scenario, since there is usually a lack of evaluation and tracking of the trainee's training progress in the existing eloquence training field, it is impossible to monitor the trainee's progress. Based on this, the feedback method for eloquence training provided in the embodiment of the present application may also include the following steps:

[0168] S160. Based on targeted eloquence improvement suggestions, develop personalized eloquence training content for trainees.

[0169] In one implementation, the eloquence training content may include, but is not limited to: a number of training plans carried out in chronological order.

[0170] Exemplarily, if the targeted eloquence improvement suggestions include eloquence improvement suggestions for improving speech coherence, then the personalized eloquence training content may include basic speech speed control exercises, such as reading aloud exercises, to emphasize the smooth transition between sentences.

[0171] S170. Dynamically monitor the training progress of the trainee and the trainee's performance data during the eloquence training process.

[0172] For example, the training progress of the trainee can be monitored by monitoring the training plan that the trainee is currently carrying out. At the same time, the performance data of the trainee during the eloquence training process, such as speech speed, sentence segmentation, etc., can also be monitored in real time through audio analysis tools to provide instant feedback.

[0173] S180: Dynamically adjust the subsequent training plan of the eloquence training content based on the training progress and the performance data.

[0174] For example, time series algorithms can be used to track the current progress of the trainee during the eloquence training process, such as the trend of changes in speaking speed and sentence segmentation, and compare the current progress with the expected progress of the ongoing training plan. If the current progress exceeds the expected progress, the difficulty and focus content of the subsequent practice plan can be dynamically adjusted.

[0175] In this embodiment, by executing step S180, the training plan can be continuously optimized according to the trainee's performance and progress in actual eloquence training to ensure the effectiveness and adaptability of the training plan, so that the eloquence training content always matches the trainee's growth trajectory, thereby maximizing the trainee's learning effect and satisfaction.

[0176] In the present application, by executing steps S160 to S180, the training progress of the trainee can be evaluated and tracked, and it can be ensured that the trainee's eloquence training content is always synchronized with his or her development needs. This not only effectively monitors the trainee's progress, but also improves the pertinence, effectiveness and scalability of eloquence training, achieving personalized and efficient eloquence training effects.

[0177] In another applicable scenario, the trainee's relevant data can also be displayed through the interactive interface. For example, the interactive interface can display the trainee's multimodal data and multidimensional eloquence quality indicators and other relevant data in real time, and can also allow the trainee to view the comparison between the current eloquence training performance and the past eloquence training performance, helping the trainee to identify the progress and the relevant content that needs to be improved.

[0178] In addition, the above-mentioned targeted eloquence improvement suggestions and personalized eloquence training content can also be displayed through the interactive interface. The trainer can select the corresponding training module through the interactive interface and perform targeted exercises according to the training plan in the personalized eloquence training content.

[0179] Figure 3 FIG. 2 shows a structural block diagram of a feedback system for eloquence training according to an embodiment of the present application. Figure 3 As shown, the system may include:

[0180] The analysis unit 210 is used to obtain multimodal data of the trainee during the current eloquence training, the multimodal data including audio data and video data; perform speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech of the trainee; perform eloquence quality analysis on the speech intensity change data, intonation pattern data, and synchronization data of non-verbal behavior and speech to obtain multi-dimensional eloquence quality indicators of the trainee during the current eloquence training process;

[0181] The feedback unit 220 is used to generate targeted eloquence improvement suggestions for the trainee based on the multi-dimensional eloquence quality index; and output the targeted eloquence improvement suggestions.

[0182] In one implementation, the feedback unit 220 is further configured to:

[0183] Based on targeted eloquence improvement suggestions, we develop personalized eloquence training content for trainees, which includes several training plans carried out in chronological order;

[0184] Dynamically monitor the trainee's training progress and the trainee's performance data during eloquence training;

[0185] Based on training progress and performance data, dynamically adjust subsequent training plans for eloquence training content.

[0186] It should be understood that the analysis unit 210 and the feedback unit 220 may be integrated in the same device, or distributed in different devices, which is not limited in this embodiment.

[0187] The functions of each unit in the feedback system for eloquence training in the embodiment of the present application can be found in the corresponding description in the above method, and will not be repeated here.

[0188] Figure 4 FIG. 2 shows a block diagram of a computer device according to an embodiment of the present application. Figure 4 As shown, the computer device includes: a memory 310 and a processor 320. The memory 310 stores instructions, and the instructions are loaded and executed by the processor 320 to implement the feedback method for eloquence training in the above embodiment. The number of the memory 310 and the processor 320 can be one or more.

[0189] The computer device also includes:

[0190] The communication interface 330 is used to communicate with external devices and perform data exchange transmission.

[0191] If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0192] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0193] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method provided in the embodiment of the present application is implemented.

[0194] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0195] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0196] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0197] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0199] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0200] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0201] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0202] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0203] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0204] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0205] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A feedback method for eloquence training, characterized in that: include: Acquire multimodal data of the trainee during current eloquence training, wherein the multimodal data includes audio data and video data; Performing speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity change data, intonation pattern data, and synchronization data between non-verbal behavior and speech of the trainee; Performing eloquence quality analysis on the voice intensity variation data, the intonation pattern data, and the synchronization data of the non-verbal behavior and voice to obtain a multi-dimensional eloquence quality index of the trainee in the current eloquence training process; Based on the multi-dimensional eloquence quality index, generating targeted eloquence improvement suggestions for the trainee; Output the targeted eloquence improvement suggestions; Among them, the eloquence quality analysis is performed on the voice intensity change data, the intonation pattern data, and the synchronization data of the non-verbal behavior and voice to obtain the multi-dimensional eloquence quality index of the trainee in the current eloquence training process, including: Performing eloquence quality analysis on the speech intensity variation data to determine the speech intensity in each of two adjacent continuous time windows in the speech intensity variation data; obtaining a speech coherence index based on the speech intensity in each of the two adjacent continuous time windows; Performing eloquence quality analysis on the intonation pattern data to obtain a change in the intonation pattern recognition value of each frame of audio in the intonation pattern data; obtaining an emotional expression index based on the change in the intonation pattern recognition value of each frame of audio; Performing eloquence quality analysis on the synchronization data of the non-verbal behavior and speech to obtain each pause area of ​​the trainee during the current eloquence training process; obtaining a pause pattern index based on the ratio of the duration of each pause area to the total eloquence training time; The multi-dimensional eloquence quality index is obtained based on the speech coherence index, the emotion expression index and the pause pattern index.

2. The method according to claim 1, characterized in that The multimodal data is subjected to speech and non-verbal behavior analysis to obtain speech intensity variation data, intonation pattern data, and synchronization data of non-verbal behavior and speech of the trainee, including: Performing speech dynamic range analysis on the multimodal data to obtain the speech intensity change data; Performing intonation pattern recognition analysis on the multimodal data to obtain the intonation pattern data; Non-verbal behavior synchronization analysis is performed on the multimodal data to obtain synchronization data of the non-verbal behavior and speech.

3. The method according to claim 1, characterized in that Based on the multi-dimensional eloquence quality index, generating targeted eloquence improvement suggestions for the trainee includes: If it is determined that the speech coherence index is lower than a first threshold value based on the multi-dimensional eloquence quality indicator, generating eloquence improvement suggestions for improving speech coherence for the trainee; If it is determined that the emotional expression index is lower than a second threshold value based on the multi-dimensional eloquence quality index, generating eloquence improvement suggestions for adjusting emotional expression for the trainee; If it is determined that the pause pattern index is lower than a third threshold based on the multi-dimensional eloquence quality indicator, an eloquence improvement suggestion for optimizing the pause pattern is generated for the trainee.

4. The method according to claim 1, characterized in that The output of the targeted eloquence improvement suggestions include: Obtain data on trainees’ preferences for various feedback modes; Obtaining an effectiveness score of each feedback mode in the multiple feedback modes; Determining an optimal feedback mode based on the preference data and the effectiveness score of each feedback mode; The targeted eloquence improvement suggestions are output according to the optimal feedback mode.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Based on the targeted eloquence improvement suggestions, formulate personalized eloquence training content for the trainee, wherein the eloquence training content includes several training plans carried out in chronological order; Dynamically monitor the training progress of the trainee and the performance data of the trainee during the eloquence training process; Based on the training progress and the performance data, the subsequent training plan of the eloquence training content is dynamically adjusted.

6. A feedback system for eloquence training, characterized in that: include: An analysis unit, used to obtain multimodal data of the trainee during the current eloquence training, wherein the multimodal data includes audio data and video data; Performing speech and non-verbal behavior analysis on the multimodal data to obtain speech intensity change data, intonation pattern data, and synchronization data between non-verbal behavior and speech of the trainee; Performing eloquence quality analysis on the voice intensity variation data, the intonation pattern data, and the synchronization data of the non-verbal behavior and voice to obtain a multi-dimensional eloquence quality index of the trainee in the current eloquence training process; A feedback unit, used for generating targeted eloquence improvement suggestions for the trainee based on the multi-dimensional eloquence quality index; Output the targeted eloquence improvement suggestions; Wherein, when the analysis unit is used to perform eloquence quality analysis on the voice intensity variation data, the intonation pattern data, and the synchronization data of the non-verbal behavior and voice to obtain the multi-dimensional eloquence quality index of the trainee in the current eloquence training process, it is specifically used to: Performing eloquence quality analysis on the speech intensity variation data to determine the speech intensity in each of two adjacent continuous time windows in the speech intensity variation data; obtaining a speech coherence index based on the speech intensity in each of the two adjacent continuous time windows; Performing eloquence quality analysis on the intonation pattern data to obtain a change in the intonation pattern recognition value of each frame of audio in the intonation pattern data; obtaining an emotional expression index based on the change in the intonation pattern recognition value of each frame of audio; Performing eloquence quality analysis on the synchronization data of the non-verbal behavior and speech to obtain each pause area of ​​the trainee during the current eloquence training process; obtaining a pause pattern index based on the ratio of the duration of each pause area to the total eloquence training time; The multi-dimensional eloquence quality index is obtained based on the speech coherence index, the emotion expression index and the pause pattern index.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the method according to any one of claims 1 to 5 is implemented.

Images