Voice guidance system and method of operating the same

Abstract

A voice guidance system and a method of operating the same, in particular a method of operating a voice guidance system comprising a voice guidance device, the method comprising obtaining audio data representing one or more voices, the audio data comprising first audio data of a first voice; obtaining first voice data based on the first audio data; determining whether the first voice data satisfies a first training criterion; in accordance with a determination that the first voice data satisfies the first training criterion, determining a first training session; outputting, via an interface of the voice guidance device, first training information indicative of the first training session.

Description

Technical Field

[0001] This disclosure relates to speech processing and related tools, methods, and systems, specifically for analyzing, monitoring, guiding, and / or evaluating the speech of one or more speakers (e.g., callers and / or agents of calls and / or one or more participants in a telephone conversation or conference (e.g., an online conference)). Therefore, a system, speech guidance device, and related methods are provided, particularly a method for operating a speech guidance system including a speech guidance device. Background Technology

[0002] Today, interactions between people are increasingly occurring remotely and virtually, such as via telephone and conference calls. However, on each side of a phone call, it can be difficult for the people speaking to understand each other's emotions or feelings simply by listening to a single speech. People's voices and / or verbal abilities are becoming increasingly important.

[0003] For example, almost all support and after-sales service is conducted by telephone between call center agents who initiate and / or answer calls and the potential customers or customers who contact the call center agents for various issues. Call center agents working in call centers, support centers, or contact centers may find the work monotonous and repetitive. This is a negative experience for the agent, but it also leads to poorer voice quality, which in turn reduces customer satisfaction on the other end of the line, resulting in longer call times on average. After many hours on the phone, it may be difficult to remember that someone on the other end was looking for help to resolve a problem. Furthermore, improving the speaker's communication skills can also be challenging. Summary of the Invention

[0004] Therefore, there is a need for a voice guidance system, a voice guidance device, and a method for operating the voice guidance system with improved voice processing capabilities.

[0005] A method for operating a voice guidance system including a voice guidance device and / or a server device is disclosed. The method includes: obtaining audio data representing one or more voices, the audio data including first audio data of a first voice; obtaining first voice data based on the first audio data; determining whether the first voice data meets a first training criterion; determining a first training session based on the determination that the first voice data meets the first training criterion; and outputting first training information indicating the first training session, for example, from the server device to the voice guidance device and / or via an interface of the voice guidance device.

[0006] In addition, a voice guidance system is disclosed, which includes a voice guidance device and an optional server device, wherein the voice guidance device is configured to perform any method according to the present disclosure.

[0007] A voice guidance device is also disclosed, including an interface, a processor, and a memory. The processor can be configured to perform at least a portion of any of the methods disclosed herein. For example, the voice guidance device can perform actions such as determining speech data, speaker measurement data, detecting the termination of a session, determining whether the speech data meets a first training criterion, determining a training session, and / or determining and / or outputting training information. For example, the voice guidance device can perform actions such as determining defect data, training representations, and practicing. The training session and / or training information can be transmitted to a server device, for example, for storage.

[0008] A server device is also disclosed, comprising an interface, one or more processors, and memory, wherein the one or more processors can be configured to perform at least a portion of any of the methods disclosed herein. For example, determining speech data, speaker measurement data, detecting the termination of a session, determining whether the speech data meets a first training criterion, determining a training session, and / or determining and / or outputting training information can be performed at the server device. For example, determining defect data, training representations, and exercises can be performed at the server device. The training session and / or training information can be transmitted to a speech guidance device for output (such as display) via the speech guidance device.

[0009] One advantage of this disclosure is that the voice guidance system can improve the speech and / or voice abilities of users of the system, and in turn, increase customer satisfaction with the customers they are speaking to. This disclosure can also improve the efficiency of conversations and / or calls, for example, resulting in shorter and more effective conversations / calls. This can also reduce costs for employees and / or companies (e.g., call centers), where a significant part of an employee's job is speaking with customers. This disclosure can also provide users with development paths or areas for timely improvement of their skills and / or abilities, for example, helping users achieve and / or improve their key performance indicators (KPIs). The voice guidance system can provide users with continuous and consistent training over time, for example, to keep users of the voice guidance system fit for practice. The voice guidance system can provide users with follow-up plans, for example, training programs. Furthermore, this disclosure can provide personalized / customized training / guidance for one or more users / speakers. The voice guidance system can reduce or eliminate the need for actual coaches or personnel to train users. For some companies, having dedicated personnel train employees may be too costly. Therefore, in some cases, employees may not receive any training at all, which can negatively impact employee performance and leave no prospect of skills development.

[0010] Furthermore, users can receive feedback on improvements to their performance or conversation outcomes. Users can receive feedback after one or more conversations and receive training or guidance based on their performance and / or the feedback. The voice guidance system can identify, demonstrate, or diagnose deficiencies and / or strengths in a user's voice. Additionally, the voice guidance system can link improvements in user voice to improvements in customer satisfaction. The voice guidance system can gamify user performance. For example, the voice guidance system can increase user engagement with their work by visualizing performance improvements. The voice guidance system can, for example, increase user motivation and provide rewards or incentives for better performance. Users can improve their speaking skills, for example, by maintaining a positive / good tone when speaking to a negative speaker, improving daytime consistency, and avoiding poor voice performance such as interruptions, long pauses, monotony, and / or speaking speed.

[0011] The advantage of this disclosure is that the speaker / user can monitor / evaluate sessions, clients, and / or meetings, thereby allowing the speaker / user to adapt or adjust the speaker's voice, such as tone of voice, to improve his / her performance during the session. This can improve the speaker's presentation outcomes and / or the outcome of the session or meeting between the speaker and one or more other speakers. The speaker can further improve their understanding of the session and / or another speaker, for example, better grasping the other speaker's emotions. The speaker / user, for example, can see when a customer / caller suddenly changes during the last session / call or a series of sessions / calls, for example, from a negative speaker state to an affirmative speaker state. Attached Figure Description

[0012] The above and other features and advantages of the present invention will become apparent to those skilled in the art from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings, wherein:

[0013] Figure 1 An exemplary voice guidance system including a server device and a voice guidance device according to the present disclosure is illustrated schematically;

[0014] Figure 2A and Figure 2B This is a flowchart of an exemplary method according to this disclosure;

[0015] Figure 3 An exemplary data structure according to this disclosure is illustrated schematically;

[0016] Figure 4 An exemplary voice guidance device according to this disclosure is schematically illustrated; and

[0017] Figure 5 A flowchart illustrating an exemplary method according to this disclosure is shown schematically. Detailed Implementation

[0018] Various exemplary embodiments and details will be described below with reference to the accompanying drawings. It should be noted that the drawings may be drawn to scale or not, and in all drawings, elements with similar structures or functions are indicated by the same reference numerals. It should also be noted that the drawings are intended only to facilitate the description of embodiments. They are not an exhaustive description of the invention, nor are they intended to limit the scope of the invention. Furthermore, the illustrated embodiments are not required to have all the aspects or advantages shown. Aspects or advantages described in connection with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment, even if not so shown or so explicitly described.

[0019] A method for operating a voice guidance system including a voice guidance device is disclosed. The voice guidance system may optionally include a server device comprising an interface, one or more processors, and memory. The server device may be configured in the cloud, such as a cloud network. The voice guidance system may be a system for training and / or guiding one or more users (e.g., the voice and / or speech of one or more users). The voice guidance system may be a system for monitoring, processing, and / or analyzing audio data representing one or more voices, such as voices comprising one or more audio signals including audio signals of one or more speakers, such as monologues or conversations, for example, a conference conversation, a video / telephone conference conversation, or a call such as a telephone or VoIP call. The system may be a system for monitoring, processing, and / or analyzing one or more audio signals, such as a speaker's monologue or a conversation between two or more people / speakers, such as a conversation in a telephone or conference. The voice guidance system may, for example, include or function as part of a call center system for monitoring, processing, and / or analyzing one or more audio signals, such as a speaker's monologue or a conversation between two or more people, for example, a telephone call between an agent of the call center system and a customer or caller. The system may be configured to use edge processing between one or more voice guidance devices. Different operations and / or steps of the method and / or system may be performed on different devices, such as voice guidance devices and / or server devices.

[0020] The voice guidance device includes an interface, a processor, and memory. The voice guidance device can be an electronic device. For example, the voice guidance device can be or include a mobile phone (e.g., a smartphone), a computer (e.g., a laptop computer or PC), or a tablet computer. The voice guidance device can be a user device, such as a mobile phone or computer, configured to perform a call between a user and one or more people. The voice guidance device can be configured to acquire audio data, for example, including a first audio input, such as a first audio input from a first voice. Audio data can be acquired from a call between a user and another person. For example, the voice guidance device can act as a call agent device, where the user can be an agent, such as an agent for a call center, for example, a support call center, after-sales call center, marketing call center, reception call center, or sales call center, or for companies where a significant part of an employee's work involves conversations with customers. The voice guidance device can be a user device, such as a mobile phone or computer, configured to record a first audio input from a first speaker, for example, recording the first speaker speaking or talking. The voice guidance device can be configured to acquire audio data, for example, including one or more audio signals, such as generating one or more audio signals including first audio data (e.g., including a first audio signal). The first audio signal may be based on a first audio input.

[0021] The method includes obtaining audio data representing one or more voices, also denoted as AD. The audio data includes first audio data of a first voice, also denoted as AD_1, for example, the first voice being the voice of a first speaker, such as a user of a voice guidance system / device.

[0022] Audio data AD may include one or more audio signals, also denoted as AS_i, i = 1, 2, ..., N, where N is the number of speakers / audio signals over time. The one or more audio signals may include a first audio signal of a first speech, also denoted as AS_1. The first speech may be the speech of a first speaker's first voice communication and / or the speech of multiple voice communications of the first speaker. The first audio signal may represent the first speech / voice of a first speaker in a first conversation. The first speaker may be a caller, an agent, or a first participant in a telephone conversation or conference, for example, including the first conversation and / or multiple conversations (e.g., the location where the first conversation and / or multiple conversations take place). Optionally, the one or more audio signals may include a second audio signal, also denoted as AS_2. The second audio signal may represent the second speech / voice of a second speaker in the first conversation and / or multiple conversations. The second speaker may be a caller, an agent, an interviewer, or a second participant in a telephone conversation or conference, for example, including the first conversation (e.g., the location where the first conversation takes place).

[0023] Obtaining audio data can include obtaining one or more audio inputs from one or more dialogues performed by a speaker / user, for example, obtaining a first audio input from one or more dialogues performed by a first speaker / user. In other words, the first audio data can be based on the first audio input. The first audio data can be obtained from a call or dialogue between a user and another person, for example, the first call or the first dialogue and / or multiple dialogues performed by the first speaker / user. The first speaker can be the person who speaks / talks the most during the first call and / or the first dialogue. The first speaker can be the first person to speak during the first call and / or the first dialogue and / or during multiple dialogues. The first speaker can be an interested party, for example, a user who can receive voice guidance and / or training. The first audio data can be obtained from a voice guidance device, for example, recording the first speech, for example, the first speech being the first audio input from the first speaker. The first audio data can include one or more recorded segments from one or more dialogues. The first audio data can indicate the segment being analyzed, for example, a speech segment or utterance. For example, a speech activity detection module can identify one or more speech / voice segments and discard noise. The segment can be, for example, a speech segment of at least 5 seconds or at least 10 seconds. The speech activity detection module can detect pauses longer than, for example, 400 milliseconds, 500 milliseconds, or 1 second. When a pause occurs, a speech segment can be detected when another speaker begins to speak, or when the segment reaches a defined maximum length (e.g., up to 8 seconds), which can indicate the end of the speech segment. For each speech segment of speech data, one or more emotion metrics and / or one or more appearance metrics can be determined. For example, recording a first speaker speaking or talking. Obtaining audio data may include generating first audio data for the first speech. Obtaining audio data may include generating one or more audio signals that include the first audio signal. The first audio signal may be based on a first audio input. The first audio signal may include a first speaker audio signal. The first audio signal may be a proxy audio signal or a caller audio signal, and a second audio signal is another.

[0024] The method includes obtaining speech data based on audio data AD, which is also denoted as VD. The method may include obtaining first speech data based on first audio data AD_1, which is also denoted as VD_1. The first speech data VD_1 may include data indicating, for example, data of a first speaker. Obtaining the first speech data may include generating the first speech data based on the first audio data. The first speech data may be first agent speech data, for example, the first speech data of a first user / agent in a voice guidance system. The first speech data may include speaker measurement data, which is also denoted as SPMD. The speech data (e.g., the first speech data) may be recorded and / or stored in memory. The speech data (e.g., the first speech data) may include timestamps, dates, event types, and / or audio recordings.

[0025] In one or more exemplary methods, obtaining speech data based on audio data includes determining speaker metric data of a speaker based on the audio data, which is also represented as SPMD. In one or more exemplary methods, obtaining first speech data VD_1 based on first audio data AD_1 includes determining first speaker metric data of a first speaker based on the first audio data, which is also represented as SPMD_1. The first speaker metric data SPMD_1 may include first primary speaker metric data SPMD_1_1. The first speaker metric data may include one or more speech parameters / features of the first speech. The first speaker metric data may include one or more of the following: tone scheduling data, speech feature data, and voice feature data of one or more speakers (e.g., the first speaker). In other words, the first speaker metric data may include one or more of the following: tone scheduling, speech feature measurement, and voice feature measurement of one or more speakers (e.g., the first speaker). The first speaker metric data may include a first speaker / speech score. In other words, determining the first speaker metric data may include determining a first speaker / speech score based on the first audio data. The first speaker score may be determined based on one or more of the following: tone scheduling, speech feature measurement, and voice feature measurement. For example, a general first speaker score can be determined based on first speaker metric data. Alternatively or additionally, a first speaker score can be determined for each speaker metric, such as tone dispatch, speech feature metric, and voice feature metric. In other words, the determined first primary speaker metric data SPMD_1_1, first secondary speaker metric data SPMD_1_2, and / or first tertiary speaker metric data SPMD_1_3 can be used to determine (e.g., convert to) a speech score or conversation score. A speech score or conversation score can be determined based on one or more metrics that meet one or more criteria.

[0026] Speaker measurement data SPMD may include one or more speaker metrics that change over time, such as one or more first speaker metrics that change over time. In other words, determining speaker measurement data SPMD may include, for example, determining one or more speaker metrics SM that change over time during a session. Determining first speaker measurement data SPMD_1 may include, for example, determining one or more first speaker metrics SPM_1 that change over time during a first session or a first record of a first speaker.

[0027] Determining the first speaker metric data SPMD_1 may include determining the first primary speaker metric data, also denoted as SPMD_1_1. Determining the first primary speaker metric data SPMD_1_1 may include determining one or more first primary speaker metrics SPM_1_1 that vary over time (e.g., during a first session or a first recording of the first speaker). In other words, the first speaker metric data SPMD_1 may include the first primary speaker metric data SPMD_1_1. Furthermore, the first speaker metric data may include or indicate one or more first primary speaker metric data SPMD_1_1 including one or more first primary speaker metrics, first speaker metric data SPMD_1_2 including one or more first secondary speaker metrics, and first third speaker metric data SPMD_1_3 including one or more first third speaker metrics. In other words, determining whether the first speech data meets the first training criterion may include determining whether the speaker metric data SPMD meets the first training criterion, for example, whether the first speaker metric data SPMD_1 meets the first training criterion. For example, determining whether the first speech data meets the first training criterion may include determining whether the first speaker metric SPM_1 meets the first training criterion.

[0028] In one or more exemplary methods, the first primary speaker measurement data includes sentiment measurement data. Determining the first primary speaker measurement data SPMD_1_1 may include determining sentiment measurement data, which is also denoted as SMD. In one or more exemplary methods, determining the first primary speaker measurement data SPMD_1_1 may include determining one or more first sentiment metrics, which are also denoted as SM_1_i, i = 1, 2, ..., M, where M is the number of first sentiment metrics. In other words, the sentiment measurement data SMD may include, for example, one or more first sentiment metrics that change over time during a first session.

[0029] Affect measure represents a speaker's emotional state. Affect measure data (SMD) can indicate the average affect of a speaker in a conversation. An affect measure can include one or more of affect type identifiers, affect level, and confidence scores. In other words, determining whether first speech data meets a first training criterion can include determining whether the first affect data meets the first training criterion.

[0030] One or more first emotion measures (e.g., SM_1_1 and / or SM_1_2) indicate the first speaker state, i.e., one or more first emotion states of the first speaker, and may be based on a first audio signal and / or a second audio signal. One or more first emotion measures SM_1_i may include one or more parameters indicating the first speaker state.

[0031] One or more first emotion metrics SM_1_i may include a first primary emotion metric, also denoted as SM_1_1, which indicates the primary emotional state of the first speaker. In other words, SM_1_1 may indicate the primary emotion of the first speaker. SM_1_1 may indicate the dominant emotional state and / or primary emotional state of the first speaker over time. The primary emotional state may be disappointment, boredom, fear, sadness, negativity, very negative, frustration, dread, panic, hesitation, dissatisfaction, anger, surprise, worry, doubt, misunderstanding, thinking, neutrality, positivity, very positive, happiness, friendliness, confidence, excitement, joy, satisfaction, high-spiritedness, contentment, relaxation, energy, enthusiasm, service-oriented, helpfulness, interest, and happiness. In one or more exemplary methods / systems, the primary emotional state of the first speaker may be selected from a set of primary emotional states. In one or more exemplary methods, determining whether the first speech data meets a first training criterion may include determining whether one or more first emotion metrics SM_1_i meet the first training criterion, for example, whether the first primary emotion metric SM_1_1 meets the first training criterion. For example, when it has been determined that the first speaker has a negative speaker state in one or more conversations (e.g., the last five calls), a training session can be determined or recommended for the first speaker to perform, for example, a first training session, to improve one or more first sentiment metrics.

[0032] The first sentiment measure SM_1_i may include a first sentiment type identifier, which is also represented as ST_ID_1_i, where i is an index, i = 1, 2, ..., H, and H is the number of first sentiment type identifiers. In other words, determining one or more first sentiment measures SM_1_i may include determining the first sentiment type identifier ST_ID_1_i, for example, the first primary sentiment type identifier ST_ID_1_1 of the first primary sentiment measure SM_1_1. The sentiment type identifier indicates the speaker's emotional state.

[0033] A sentiment type identifier can be, for example, a label, number, or integer value corresponding to a specific sentiment (state), sentiment type, and / or sentiment category. For example, ST_ID_1_i can be selected from the same or different sets of sentiment type identifiers. For example, ST_ID_1_1 can be selected from a primary set of sentiment type identifiers, and / or ST_ID_1_2 can be selected from a secondary set of sentiment type identifiers. The primary set of sentiment type identifiers can be different from or the same as the secondary set of sentiment type identifiers. The primary and secondary sets of sentiment type identifiers can share one or more (e.g., multiple) sentiment type identifiers.

[0034] In one or more exemplary methods, the first primary sentiment metric SM_1_1 includes a first primary sentiment type identifier ST_ID_1_1 selected from a set of primary sentiment type identifiers ST_ID_SET_1, wherein ST_ID_SET_1 includes a plurality of sentiment type identifiers, for example, including at least three, four, five or more sentiment type identifiers.

[0035] In one or more exemplary methods, the primary sentiment type identifier set ST_ID_SET_1 is given by the following formula:

[0036] ST_ID_SET_1 = {1,2,3,4,5}

[0037] In this context, "1" represents, for example, "very negative" emotion, "2" represents, for example, "negative" emotion, "3" represents, for example, "neutral" emotion, "4" (optional) represents, for example, "positive" emotion, and "5" (optional) represents, for example, "very positive" emotion.

[0038] In one or more exemplary methods, the primary sentiment type identifier set ST_ID_SET_1 is given by the following formula:

[0039] ST_ID_SET_1 = {"Very negative", "Negative", "Neutral", "Positive", "Very positive"}

[0040] For example, "very negative" and / or "very positive" are optional.

[0041] The set of emotion type identifiers (e.g., a primary emotion type identifier set and / or a secondary emotion type identifier set) may include at least three or at least four different emotion type identifiers, such as five, six, seven, eight, nine or more emotion type identifiers. In other words, each emotion or emotion type may have a corresponding ST_ID_1_i. For example, the first primary emotion measure SM_1_1 may include a first primary emotion type identifier ST_ID_1_1, which indicates or corresponds to the primary emotion state or the first speaker state as "positive".

[0042] The first sentiment measure SM_1_i may include a sentiment level, also denoted as SL_1_i, i = 1, 2, ..., 0, where 0 is the number of sentiment levels. In other words, determining SM_1_i may include determining SL_1_i; for example, determining SM_1_1 may include determining the first primary sentiment level SL_1_1. The sentiment level SL_1_i may indicate the level of the i-th sentiment type. In other words, SL_1_i may represent the degree of the i-th sentiment type. For example, when ST_ID_1_1 corresponds to the first speaker state "positive," the first primary sentiment level SL_1_1 may indicate or correspond to the degree of the sentiment "positive," for example, in a level from 0 to 1 or from 1 to 10, or selected from "low," "medium," and "high." In other words, the sentiment level of the sentiment measure may be a single level, for example, from 0 to 1 or from 1 to 10. In other words, determining whether the first speech data meets the first training criterion may include determining whether the first emotion level SL_1_i meets the first training criterion, for example, whether the first primary emotion level meets the first training criterion. For example, the first training criterion may include a threshold for the first emotion level SL_1_i of the speaker state that the first speaker needs to be trained on. For example, for the first speech data, the threshold for the first emotion level SL_1_i may have to be higher than 5 to meet the first training criterion.

[0043] The first sentiment metric SM_1_i may include confidence scores, denoted as SCS_1_i, i = 1, 2, ..., P, where P is the number of confidence scores. In other words, determining SM_1_i may include determining the first confidence score SCS_1_i. For example, determining the first primary sentiment metric SM_1_1 may include determining the first primary confidence score SCS_1_1. The confidence score of the sentiment metric may indicate the score or probability that the determined sentiment metric (e.g., sentiment type identifier and / or sentiment level) is correct, for example, the sentiment state or sentiment type (identified by the sentiment type identifier of the sentiment metric) is correct. For example, SCS_1_1 = 0.88 may indicate that the probability that the determined ST_ID_1_1 (e.g., "positive") is correct is 88%. In other words, determining whether the first speech data meets the first training criterion may include determining whether the first confidence score SCS_1_i meets the first training criterion, for example, whether the first primary confidence score meets the first training criterion. For example, the first training criterion may include a threshold for the first confidence score SCS_1_i of the speaker state that the first speaker needs to be trained on. For example, for the first speech data, the threshold for the first confidence score SCS_1_i may have to be higher than 0.6 to meet the first training criterion.

[0044] Determining one or more first emotion measures indicative of a first speaker's state may include extracting one or more speaker features from a first audio signal, for example, wherein the one or more first emotion measures are based on one or more speaker features. The one or more speaker features may include paralinguistic features. The one or more speaker features may, for example, include speaker tone features, speaker intonation features, speaker power or volume features, speaker pitch features, speaker speech quality features, linguistic features, acoustic features, and / or speaker spectral band energy features. Spectral band energy features may include individual bins of a spectrum indicating the signal energy level at a given frequency.

[0045] Linguistic features may include specific sentiment-related words, such as affirmative and / or negative words. Linguistic features can be determined based on a text transcript of the audio signal. The text transcript can be obtained by a human annotator or using an automated speech recognition (speech-to-text) algorithm or service. Linguistic features may include embedded features from deep neural networks (e.g., BERT Transformer Network or other sequence-to-sequence autoencoders).

[0046] In one or more exemplary methods, one or more first sentiment metrics may be determined based on a machine learning (ML) model (e.g., the output of the ML model). The input to the ML model may be speaker features or the audio signal itself. The ML model may include a linear regression model, support vector machine, decision tree classifier (e.g., random forest XGBoost), Gaussian mixture model, hidden Markov model, and / or neural network. The neural network may include, for example, one or more of linear feedforward layers, convolutional layers, recursive layers, and attention layers. The ML model may include a weighted average of one or more speaker features. For example, the ML model may map, for example, speaker tone and / or voice quality to sentiment metrics / types, sentiment levels, and / or sentiment confidence scores. The ML model may include parameters ranging from 100,000 to 1,000,000, for example, 500,000 to 1,000,000 parameters. The ML model may include layers ranging from 5 to 20, for example, 10 to 15 layers.

[0047] Examples of known ML models include: "www.researchgate.net / publication / 222431291_Emotional_speech_recognition_Resources_features_and_methods", "https: / / mediatum.ub.tum.de / doc / 1523509 / 1523509.pdf" and "https: / / www.researchgate.net / publication / 319565810_End-to-end_learning_for_dimensional_emotion_recognition_from_physiological_signals".

[0048] For example, an emotion metric, also denoted as S, can be derived from the speaker speech patterns, which can be the sum of the normalized variances of the fundamental frequency F0 / pitch (e.g., a range from 0 to 1 obtained by multiplying by a factor determined on the audio input, e.g., the inverse of the range of the F0 variance of the training data for the ML model). For example, an emotion metric can be derived from the speaker speech patterns S, which can be the normalized (range 0-1) variance of the signal strength. For example, to determine emotion as “high” and “low”, a threshold of, for example, 1.0 can be applied to S, where high is detected when S is greater than or equal to 1.0, and low is detected when S is less than 1.0. Furthermore, an emotion level can be determined or obtained for “high” or “low” emotion, for example, in the range of 0-1, where S can be divided by 2 (e.g., the number of speaker features as part of the sum). In addition, the sentiment confidence score can be determined or obtained based on the absolute value of the numerical difference between the normalized F0 variance and / or the normalized signal strength variance.

[0049] ML models can be trained on, for example, recordings of calls, where a verifier or supervisor (e.g., a psychologist and / or a human supervisor) has assigned sentiment identifiers / labels to sentiment metrics and / or speaker feature labels to speaker features, for example, based on their own subjective best-effort judgment. Speaker features can be algorithmically determined via signal processing algorithms and / or as the output of another ML model. One or more first sentiment metrics can be inferred by the ML model. The inputs to the ML model can include one or more acoustic features, such as loudness and / or pitch features. Pitch features can be determined by the ML model and can be, for example, negative or positive pitch. Furthermore, the inputs to the ML model can include spectrograms, i.e., latent (hidden layer activation) representations of (deep) neural networks. The inputs to the ML model can include static feature vectors (“fingerprints”), such as mean, variance, slope, peak distance, and modulation spectrum. The input to the ML model can include frame-by-frame (low-level) acoustic features, such as pitch, energy level, spectral parameters (mel frequency cepstral MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectral envelope features (e.g., formants, harmonics, harmonic-to-formant ratio), and / or speech quality metrics, such as harmonic noise ratio (HNR), jitter, and / or luminosity.

[0050] In one or more exemplary methods, one or more first emotion measures SM_1_i include a first secondary emotion measure, also denoted as SM_1_2, which indicates the secondary emotion state of the first speaker.

[0051] Secondary affective states can differ from primary affective states. In other words, SM_1_2 can indicate the primary emotion of the first speaker. SM_1_2 can be an affective state that is less dominant in time than the primary and / or secondary affective states of the first speaker. Secondary affective states can be one of the following: disappointment, boredom, fear, sadness, negativity, dissatisfaction, anger, surprise, worry, doubt, incomprehension, contemplation, neutrality, positivity, very positivity, happiness, friendliness, confidence, excitement, joy, satisfaction, elatedness, contentment, relaxation, energy, enthusiasm, and happiness.

[0052] In one or more exemplary methods / systems, the secondary emotional state of the first speaker can be selected from a set of secondary emotional states.

[0053] The first sentiment measure SM_1_i may include a first sentiment type identifier, also denoted as ST_ID_1_i, where i is an index. In other words, determining one or more first sentiment measures SM_1_i may include determining the first sentiment type identifier ST_ID_1_i, for example, the first sentiment type identifier ST_ID_1_2 for the first desired sentiment measure SM_1_2. The sentiment type identifier indicates the speaker's emotional state.

[0054] The sentiment type identifier can be, for example, a label, number, or integer value corresponding to a specific sentiment (state), sentiment type, and / or sentiment category. For example, ST_ID_1_i can be selected from the same or different sets of sentiment type identifiers. For example, ST_ID_1_2 can be selected from a second set of sentiment type identifiers.

[0055] In one or more exemplary methods, the primary sentiment metric SM_1_2 includes a primary sentiment type identifier ST_ID_1_2 selected from a set of secondary sentiment type identifiers ST_ID_SET_2, wherein ST_ID_SET_2 includes multiple sentiment type identifiers, for example, including at least three, four, five or more sentiment type identifiers.

[0056] In one or more exemplary methods, the set of secondary sentiment type identifiers ST_ID_SET_2 is given by the following formula:

[0057] ST_ID_SET_2 = {1,2,3,4,5}

[0058] In this context, "1" represents, for example, "very negative" emotion, "2" represents, for example, "negative" emotion, "3" represents, for example, "neutral" emotion, "4" (optional) represents, for example, "positive" emotion, and "5" (optional) represents, for example, "very positive" emotion.

[0059] In one or more exemplary methods, the set of secondary sentiment type identifiers ST_ID_SET_2 is given by the following formula:

[0060] ST_ID_SET_2 = {"Very negative", "Negative", "Neutral", "Positive", "Very positive"}

[0061] For example, "very negative" and / or "very positive" are optional.

[0062] The set of emotion type identifiers (e.g., a primary emotion type identifier set and / or a secondary emotion type identifier set) may include at least three or at least four different emotion type identifiers, such as five, six, seven, eight, nine, or more emotion type identifiers. In other words, each emotion or emotion type may have a corresponding ST_ID_1_i. For example, the first desired emotion measure SM_1_2 may include a first desired emotion type identifier ST_ID_1_2 that indicates or corresponds to a secondary emotion state or first speaker state as "positive".

[0063] The first emotion measure SM_1_i can include an emotion level, also denoted as SL_1_i, i = 1, 2, ..., 0, where 0 is the number of emotion levels. In other words, determining SM_1_i can include determining SL_1_i; for example, determining SM_1_2 can include determining the first desired emotion level SL_1_2. The emotion level SL_1_i can indicate the level of the i-th emotion type. In other words, SL_1_i can represent the degree of the i-th emotion type. For example, when ST_ID_1_2 corresponds to the first speaker state "positive," the first desired emotion level SL_1_2 can indicate or correspond to the degree of "positive" emotion, for example, in a level from 0 to 1 or from 1 to 10, or selected from "low," "medium," and "high." In other words, the emotion level of the emotion measure can be in a level from 0 to 1 or from 1 to 10.

[0064] The first sentiment metric SM_1_i may include a confidence score, denoted as SCS_1_i, i = 1, 2, ..., P, where P is the number of confidence scores. In other words, determining SM_1_i may include determining the first confidence score SCS_1_i. For example, determining the first sentiment metric SM_1_2 may include determining the first confidence score SCS_1_2. The confidence score of the sentiment metric may indicate the score or probability that the determined sentiment metric (e.g., sentiment type identifier and / or sentiment level) is correct, such as the sentiment state or sentiment type (identified by the sentiment type identifier of the sentiment metric) being correct. For example, SCS_1_2 = 0.88 may indicate that the determined ST_ID_1_2 (e.g., "positive") has an 88% probability of being correct.

[0065] In one or more exemplary methods, the first speaker measurement data SPMD_1_2 is tone dispatch data, also denoted as TMD. The tone dispatch data may include speech pitch data. Determining the first speaker measurement data SPMD_1_2 may include determining the tone dispatch data. In one or more exemplary methods, determining the first speaker measurement data SPMD_1_2 may include determining one or more first tone dispatches, also denoted as TM_1_i, i = 1, 2, ..., m, where m is the number of first tone dispatches. In other words, the tone dispatch data TMD may include, for example, one or more first tone dispatches that vary over time during the first session.

[0066] A tone quantity represents a speaker's pitch. A tone quantity data TMD can indicate the average pitch of a speaker in a conversation. First speaker metric data can include, for example, one or more first pitch parameters included in the tone quantity data TMD. One or more first pitch parameters can be based on one or more first tone quantities. For example, the tone quantity data TMD can indicate or include a first pitch parameter that is the speaker's average pitch during the conversation and / or the pitch outcome of the conversation (e.g., the speaker's total pitch when the conversation ends). The tone quantity data TMD can indicate the average pitch of a speaker in a conversation. The tone quantity data TMD can include one or more speech metrics. In other words, tone quantity data can include one or more speech metrics, such as interruptions, excessive talking, long pauses, high or low speech rate, high or low pitch, speaker twists or abrupt changes (e.g., shifting a customer from negative to positive or vice versa), and / or responsiveness. The tone quantity data can include counts of events, such as counts of one or more metrics.

[0067] In other words, determining whether the first speech data meets the first training criterion may include determining whether the tone dispatch quantity data (TMD) meets the first training criterion, for example, whether one or more first tone dispatch quantities (TM_1_i) meet the first training criterion. For example, when it has been determined that the first speaker has a negative speaking tone in one or more conversations (e.g., the last five calls), it may be determined or suggested that the first speaker perform a training session, such as a first training session.

[0068] In one or more exemplary methods, the first speaker measurement data includes first and third speaker measurement data as appearance measurement data. Determining the first speaker measurement data SPMD_1 may include determining the first and third speaker measurement data, also denoted as SPMD_1_3. In other words, the first speaker measurement data SPMD_1 may include the first and third speaker measurement data SPMD_1_3. Determining the first and third speaker measurement data SPMD_1_3 may include determining appearance measurement data, also denoted as AMD.

[0069] In one or more exemplary methods, determining the first tertiary speaker metric data SPMD_1_3 may include determining one or more first appearance metrics, also denoted as AM_1_i, i = 1, 2, Q, where Q is the number of first appearance metrics of the first speaker or associated with the first speaker. In other words, the appearance metric data AMD may include, for example, one or more first appearance metrics that change over time during a first session.

[0070] Appearance metrics represent a speaker's appearance. First speaker metric data may include, for example, one or more first appearance parameters included in appearance metric data AMD. One or more first appearance parameters may be based on one or more first appearance metrics. For example, appearance metric data AMD may indicate or include first appearance parameters that are the speaker's average appearance during a session and / or the appearance outcome of the session (e.g., the speaker's overall appearance when the session ends). Appearance metric data AMD may indicate the speaker's average appearance during a session. Appearance metrics may include one or more of appearance identifiers, appearance grades, and confidence scores, and may indicate the first speaker's appearance, for example, based on a first audio signal and / or a second audio signal. Determining one or more first appearance metrics may include, for example, retrieving the first appearance metrics from a database based on information related to the first speaker stored in a database. One or more first appearance metrics AM_1_i may include one or more parameters indicating the first speaker's appearance. One or more first appearance metrics AM_1_i include a first primary appearance metric indicating the first speaker's primary appearance and / or a first secondary appearance metric indicating the first speaker's secondary appearance, the first primary appearance metric also denoted as AM_1_1, and the first secondary appearance metric also denoted as AM_1_2. In one or more exemplary methods, determining whether first speech data meets a first training criterion may include determining whether one or more first appearance metrics AM_1_i meet the first training criterion, such as whether a first primary appearance metric AM_1_1 meets the first training criterion. For example, when it has been determined that a first speaker's first speech sounds like a 10-year-old girl in one or more conversations, but in reality, for example, when compared with actual appearance metrics stored in a database, the first speaker is a 45-year-old man, then it may be determined or recommended that the first speaker perform a training session, such as a first training session, to improve the matching between the first appearance metrics of the first speech and the first speaker's actual appearance.

[0071] In other words, AM_1_1 can indicate the primary physical appearance of the first speaker. AM_1_1 can be selected from gender metrics (e.g., female, male, or genderless), weight metrics, height metrics, age metrics, language metrics, language ability metrics, hearing ability metrics, and comprehension metrics, dialect metrics, health metrics (e.g., breathing condition, speech deficits, and / or speech disorders), personality metrics (e.g., extroverted or introverted), and comprehension metrics (e.g., based on age, health, and / or gender metrics). Comprehension metrics may be relevant when an older person has difficulty listening to a conversation, or when a foreigner is not comfortable with spoken language. Comprehension metrics can provide instructions to the user, for example, that he / she should speak more slowly and clearly.

[0072] The first appearance metric AM_1_i may include a first appearance identifier, also denoted as A_ID_1_i, where i is the index of the i-th first appearance metric, i = 1, 2, ..., I, where I is the number of first appearance identifiers. The first appearance identifier may indicate one of the following: gender metric, weight metric, height metric, age metric, language metric, language ability metric, auditory ability metric, and comprehension metric. In other words, determining AM_1_i may include determining A_ID_1_i, including the first primary appearance identifier A_ID_1_1 of the first primary appearance metric AM_1_1 and / or the first desired appearance identifier A_ID_1_2 of the first desired appearance metric AM_1_2.

[0073] Appearance identifiers can be, for example, labels, numbers, or integer values ​​corresponding to a specific appearance metric, appearance type, and / or appearance category. For instance, A_ID_1_i can be selected from a set of appearance types (e.g., including one or more of gender, height, weight, age, language, language ability, hearing ability, and comprehension ability). Appearance metric identifiers can be labels or numbers mapped to and / or indicating the appearance metric type.

[0074] In one or more exemplary methods, the first primary appearance metric AM_1_1 includes a first primary appearance identifier A_ID_1_1 optionally selected from a set of primary appearance identifiers A_ID_SET_1, wherein A_ID_SET_1 includes multiple appearance identifiers, for example, including at least three, four, five or more sentiment type identifiers. In one or more exemplary methods, the first primary appearance metric AM_1_1 is a gender metric, i.e., the first primary metric identifier A_ID_1_1 indicates gender, for example, A_ID_1_1 = "gender" or A_ID_1_1 = 1, which may be mapped to gender via a table.

[0075] In one or more exemplary methods, the first appearance metric AM_1_2 includes a first appearance identifier A_ID_1_2 optionally selected from a set of secondary appearance identifiers A_ID_SET_2, wherein A_ID_SET_2 includes a plurality of appearance identifiers, for example, including at least three, four, five or more appearance identifiers.

[0076] The set of appearance identifiers may include two or at least three or at least four distinct appearance identifiers, such as five, six, seven, eight, nine or more appearance identifiers. For example, a first primary appearance metric AM_1_1 may include a first primary appearance identifier A_ID_1_1 that indicates or corresponds to the primary appearance of the first speaker (e.g., one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability). For example, a first desired appearance identifier A_ID_1_2 may indicate or correspond to the first desired appearance of the first speaker (e.g., one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability). The first desired appearance identifier may optionally be different from the first primary appearance identifier.

[0077] The first appearance metric AM_1_i may include appearance levels, also denoted as AL_1_1_i, i = 1, 2, ..., R, where R is the number of appearance levels. In other words, determining AM_1_i may include determining AL_1_i; for example, determining AM_1_i may include determining a first primary appearance level AM_1_1. The first appearance level AL_1_1_i may indicate the level, value, range, or label of the appearance metric AM_1_i indicated by the appearance identifier A_ID_1_i. In other words, the first appearance level AL_1_1_i may indicate the level, value, range, or label of the first appearance metric AM_1_i. For example, when A_ID_1_1 corresponds to the first primary appearance of the first speaker as "gender," the first primary appearance level AL_1_1 may indicate or correspond to "male," "female," or optionally "unisex." For example, when the first desired appearance identifier A_ID_1_2 corresponds to a first desired appearance metric of the first speaker being "high," the first desired appearance level AL_1_2 can indicate or correspond to "short," "medium," or "high." For example, when the first desired appearance identifier A_ID_1_2 corresponds to a first desired appearance of the first speaker being "high," the first desired appearance level AL_1_2 can indicate or correspond to "less than 160 cm," "between 160 cm and 185 cm," or "greater than 185 cm." In other words, determining whether the first speech data meets the first training criterion can include determining whether the first appearance level AL_1_i meets the first training criterion, for example, whether the first primary appearance level meets the first training criterion. For example, the first training criterion can include a threshold for the first appearance level AL_1_i of the speaker state that the first speaker needs to be trained on.

[0078] For example, when a first appearance identifier (e.g., a first third appearance identifier A_ID_1_3) corresponds to a first appearance metric of a first speaker, such as the first third appearance metric AM_1_3 being “age”, the first third appearance metric AL_1_3 may indicate or correspond to an age range, such as “less than 20 years old”, “20-40 years old”, “40-60 years old”, or “greater than 60 years old”, or an age label, such as “young”, “middle-aged”, or “old”.

[0079] The first appearance metric AM_1_i may include a confidence score, also denoted as ACS_1_i, i = 1, 2, ..., S, where S is the number of confidence scores. In other words, determining the first appearance metric AM_1_i may include determining the first appearance confidence score ACS_1_i. For example, determining the first primary appearance metric AM_1_1 may include determining the first primary appearance confidence score ACS_1_1. The first appearance confidence score ACS_1_i of the appearance metric AM_1_i may indicate the score or probability that the determined first appearance metric AM_1_i (e.g., the first appearance grade AL_1_1_i) is correct; for example, the appearance metric or appearance grade is correct. For example, ACS_1_1 = 0.95 may indicate that the probability that the determined AL_1_1 is "male" is correct is 95%. In other words, determining whether the first speech data meets the first training criterion may include determining whether the first confidence score ACS_1_i meets the first training criterion, for example, whether the first primary confidence score meets the first training criterion. For example, the first training criterion may include a threshold for the first confidence score ACS_1_i of the speaker state that the first speaker needs to be trained on. For example, for the first speech data, the threshold for the first confidence score ACS_1_i must be higher than 0.6 to meet the first training criterion.

[0080] Determining one or more first appearance metrics that indicate a first speaker may include extracting one or more speaker appearance features from a first audio signal. One or more speaker appearance features may include, for example, speaker tone features, speaker intonation features, speaker power features, speaker pitch features, speaker speech quality features, speaker rate features, linguistic features, acoustic features, and / or speaker spectral band energy features.

[0081] Spectral band energy characteristics can include individual sections of a spectrum that indicate the signal energy level at a given frequency.

[0082] Language features may include specific appearance-related words, such as affirmative and / or negative words. Language features can be determined based on a text transcript of an audio signal. The text transcript can be obtained by a human annotator or using an automatic speech recognition (speech-to-text) algorithm or service. Language features may include embedded features from deep neural networks (e.g., BERT Transformer Network or other sequence-to-sequence autoencoders).

[0083] In one or more exemplary methods, one or more first appearance metrics may be determined based on a machine learning (ML) model (e.g., the output of the ML model). The one or more first appearance metrics may be inferred by the ML model. The ML model may include a linear regression model, a support vector machine, a decision tree classifier (e.g., random forest XGBoost), a Gaussian mixture model, a hidden Markov model, and / or a neural network. The neural network may include, for example, one or more of linear feedforward layers, convolutional layers, recursive layers, and attention layers. The ML model may include a weighted average of one or more speaker features. For example, the ML model may map, for example, speaker tone and / or voice quality to sentiment metrics / types, sentiment levels, and / or sentiment confidence scores. The ML model may include parameters ranging from 100,000 to 1,000,000, for example, 500,000 to 1,000,000 parameters. The ML model may include layers ranging from 5 to 20, for example, 10 to 15 layers.

[0084] ML models can be trained on, for example, recordings of calls, where a validator or supervisor (e.g., a human supervisor) has assigned sentiment identifiers / labels to sentiment metrics and / or speaker feature labels to speaker features. Speaker features can be algorithmically determined using signal processing algorithms. One or more first appearance metrics can be inferred by the ML model. The input to the ML model can include audio data, for example, audio data stored in a database of known audio data matching one or more appearance metrics (e.g., appearance labels). Appearance labels can include labels assigned by people and / or basic facts (e.g., age or height in a passport or social registry). For example, audio data input can include recordings of calls, television programs, and / or film actors, etc.

[0085] The input to an ML model can include one or more acoustic features, such as pitch features. Pitch features can be, for example, negative or positive pitch. Furthermore, the input to an ML model can include a spectrogram, i.e., a latent (hidden layer activation) representation of a (deep) neural network. The input to an ML model can include static feature vectors (“fingerprints”), such as mean, variance, slope, peak distance, and modulation spectrum. The input to an ML model can include frame-by-frame (low-level) acoustic features, such as pitch, energy level, spectral parameters (mel frequency cepstral MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectral envelope features (e.g., formants, harmonics, harmonic-to-formant ratio), and / or speech quality metrics, such as harmonic noise ratio (HNR), jitter, and / or luminosity. For example, acoustic features associated with one or more appearance metrics (e.g., physical appearance) can include the ratio of vowel formants associated with tract length. For example, acoustic features may involve one or more appearance metrics, such as body size; speech quality features, such as age-related HNR, jitter, and / or luminosity (e.g., more breathing and more jitter for older people); and pitch may be gender-related (e.g., males may have pitch below 150 Hz, and females may have pitch above 150 Hz). Furthermore, acoustic features may include, for example, a phoneme list / histogram of language and dialect features, and / or, for example, average spectral envelope features of age and / or gender.

[0086] In one or more exemplary methods, one or more first appearance metrics AM_1_i include a first desired appearance metric that indicates the second appearance of the first speaker, the first desired appearance metric also being denoted as AM_1_2.

[0087] In one or more exemplary methods, the first appearance metric AM_1_2 includes a first appearance identifier A_ID_1_2 optionally selected from a set of secondary appearance identifiers A_ID_SET_1, wherein A_ID_SET_1 includes multiple appearance identifiers, for example, including at least three, four, five or more sentiment type identifiers. In one or more exemplary methods, the first appearance metric AM_1_2 is a gender metric, i.e., the first appearance metric identifier A_ID_1_2 indicates gender, for example, A_ID_1_2 = "gender" or A_ID_1_2 = 1, which can be mapped to gender via a table.

[0088] In one or more exemplary methods, the first appearance metric AM_1_2 includes a first appearance identifier A_ID_1_2 optionally selected from a second set of appearance identifiers A_ID_SET_2, wherein A_ID_SET_2 includes a plurality of appearance identifiers, for example, including at least three, four, five or more appearance identifiers.

[0089] The set of appearance identifiers may include two or at least three or at least four different appearance identifiers, such as five, six, seven, eight, nine or more appearance identifiers. For example, the primary appearance metric AM_1_2 may include a primary appearance identifier A_ID_1_2 that indicates or corresponds to the secondary appearance of the first speaker, such as one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability. For example, the primary appearance identifier A_ID_1_2 may indicate or correspond to the primary appearance of the first speaker, such as one of gender, weight, height, age, language, language ability, hearing ability and comprehension ability, dialect metric, health metric (e.g., breathing condition, speech deficit and / or speech disorder), personality metric (e.g., extroverted or introverted), and comprehension metric (e.g., based on age metric, health metric, and / or gender metric). Comprehension metrics may be relevant when an older person has difficulty listening to a conversation, or when a foreigner is not comfortable with spoken language. Comprehension metrics can provide instructions to the user, such as that he / she should speak more slowly and clearly. The first appearance identifier can be different from the first appearance identifier.

[0090] The first appearance metric AM_1_i can include appearance levels, also denoted as AL_1_1_i, i = 1, 2, ..., R, where R is the number of appearance levels. In other words, determining AM_1_i can include determining AL_1_i; for example, determining AM_1_2 can include determining the first desired appearance level AM_1_2. The first appearance level AL_1_i can indicate the level, value, range, or label of the appearance metric AM_1_i indicated by the appearance identifier A_ID_1_i. In other words, the first appearance level AL_1_i can indicate the level, value, range, or label of the first appearance metric AM_1_i. For example, when A_ID_1_2 corresponds to the first desired appearance of the first speaker being "gender," the first desired appearance level AL_1_2 can indicate or correspond to "male," "female," or optionally "unisex." For example, when the first desired appearance identifier A_ID_1_2 corresponds to the first desired appearance metric of the first speaker being "tall," the first desired appearance level AL_1_2 can indicate or correspond to "short," "medium," or "tall." For example, when the first appearance identifier A_ID_1_2 corresponds to the first appearance of the first speaker being “high”, the first appearance level AL_1_2 can indicate or correspond to “less than 160 cm”, “between 160 cm and 185 cm”, or “greater than 185 cm”.

[0091] For example, when a first appearance identifier (e.g., a first third appearance identifier A_ID_1_3) corresponds to a first appearance metric of a first speaker, such as the first third appearance metric AM_1_3 being “age”, the first third appearance metric AL_1_3 may indicate or correspond to an age range, such as “less than 20 years old”, “20-40 years old”, “40-60 years old”, or “greater than 60 years old”, or an age label, such as “young”, “middle-aged”, or “old”.

[0092] The first appearance metric AM_1_i may include a confidence score, also denoted as ACS_1_i, i = 1, 2, ..., S, where S is the number of confidence scores. In other words, determining the first appearance metric AM_1_i may include determining the first appearance confidence score ACS_1_i. For example, determining the first appearance metric AM_1_2 may include determining the first appearance confidence score ACS_1_2. The first appearance confidence score ACS_1_i of the appearance metric AM_1_i may indicate the score or probability that the determined first appearance metric AM_1_i (e.g., the first appearance grade AL_1_1_i) is correct; for example, the appearance metric or appearance grade is correct. For example, ACS_1_2 = 0.95 may indicate that the probability that AL_1_2 is "male" is 95%.

[0093] One or more first emotion metrics and one or more first appearance metrics may be part of the first speaker metrics data. The first speaker metrics data may also be represented as agent metrics data and / or caller metrics data.

[0094] In one or more exemplary methods, obtaining speech data includes determining defect data that indicates one or more defects in the speech, the defect data also referred to as DD (deficiency data).

[0095] In one or more exemplary methods, obtaining first speech data VD_1 includes determining first defect data, also denoted as DD_1, indicating one or more defects in the first speech. In other words, obtaining the first speech data may include determining whether the first speech data (e.g., the first speech) includes one or more defects. A defect may be defined as speech data, such as a first speaker / user outside a predetermined or defined range, threshold, and / or value, or as a speaker metric and / or speech metric. A defect may also be determined based on an event in which the speaker / user fails to act as required or defined.

[0096] In one or more exemplary methods, determining whether first speech data meets a first training criterion includes determining first defect data indicating one or more defects in the first speech and determining whether the first defect data meets the first training criterion, for example, assuming the first defect data meets one or more thresholds. Obtaining the first defect data may include identifying one or more speaker metrics and / or speech metrics, wherein the performance of the first speaker is below the defined criteria.

[0097] In one or more exemplary methods, defect data includes one or more of speech tone defects, speech feature defects, and voice feature defects. First defect data may include one or more of a first speaker's speech tone defects, speech feature defects, and voice feature defects. Speech tone defects may include, for example, a negative speech tone, an unsatisfied speech tone, a worried tone, an annoyed tone, and / or a fearful tone. Speech feature defects may include, for example, a lack of enthusiasm, a lack of friendliness, a lack of empathy, and / or a lack of professionalism of the first speaker, for example, in the first speech. Voice feature defects may include, for example, a monotonous voice, an unpleasant speech rate, unpleasant intonation variations, and / or unclear pronunciation.

[0098] In one or more exemplary methods, obtaining speech data includes determining intensity data, also referred to as SD, that indicates one or more intensities of the speech.

[0099] In one or more exemplary methods, obtaining first speech data VD_1 includes determining first intensity data, also denoted as SD_1, indicating one or more intensities of the first speech. In other words, obtaining the first speech data may include determining whether the first speech data (e.g., the first speech) includes one or more intensities.

[0100] In one or more exemplary methods, determining whether first speech data meets a first training criterion includes determining first intensity data that indicates one or more intensities of the first speech. Obtaining the first intensity data may include identifying one or more speaker metrics and / or speech metrics, wherein the first speaker's performance exceeds the defined criteria.

[0101] In one or more exemplary methods, intensity data includes one or more of speech pitch intensity, speech feature intensity, and voice feature intensity. First intensity data may include one or more of a first speaker's speech pitch intensity, speech feature intensity, and voice feature intensity. Speech pitch intensity may, for example, include an affirmative speech pitch, a satisfied speech pitch, a service-oriented pitch, an enthusiastic pitch, and / or a helpful pitch. Speech feature intensity may, for example, include qualities such as enthusiasm, friendliness, and / or professionalism of the first speaker in the first speech. Voice feature intensity may, for example, include variations in voice, a pleasant speaking rate, and / or clear pronunciation.

[0102] The first training information can be based on defect data and / or strength data.

[0103] In other words, training information can provide feedback to the user / first speaker about his / her speech parameters / features, such as feedback related to defect data and / or intensity data. The first training information can provide feedback reports to the user / speaker, such as personalized feedback reports. Outputting the first training information can generate a first feedback report. The first training information can provide a summary feedback report, highlighting one or more weaknesses and / or strengths of the first speaker / user. The first training information can include a benchmark of the first speech data, such as one or more other users of the company the first speaker works for and / or other users of the speech guidance system. First training information based on defect data can provide feedback to the first speaker about which speech parameters / features need improvement. First training information based on intensity data can provide positive feedback to the first speaker, such as motivating the first speaker by showing the improvement of the intensity of one or more of the speech parameters / features over time.

[0104] The method includes determining whether speech data VD meets training criteria, also denoted as TC. The method may include determining whether first speech data VD_1 meets a first training criterion, also denoted as TC_1. The method may also include determining whether the first speech data meets or fulfills the first training criterion.

[0105] The first training criterion may indicate whether the first speech data meets a threshold of the first training criterion, such as a first threshold. The first training criterion may indicate whether a user / speaker should be guided or trained for one or more defects in the first speech. In other words, the method includes determining, based on the determined first speech data, whether the first speaker / user should be guided and / or trained, such as through one or more training sessions. For example, when the first speech data meets the training criterion, the user may need to improve one or more defects in his / her speech, such as the first speaker's initial speech. The first training criterion may include one or more parameters to which the first speech data (e.g., first speaker metric data) can be compared. For example, the first training criterion may include a first speech pitch criterion / parameter, a first speech feature criterion / parameter, a first voice feature criterion / parameter, and / or a first time parameter / criterion (e.g., a time limit that the first speaker must perform before the first training session). In other words, the first training criterion may include one or more conditions that the first speech data must meet.

[0106] In one or more exemplary methods, determining whether the first speech data meets the first training criterion includes determining whether the first speech data is higher than, equal to or lower than a first threshold or within a first range.

[0107] Determining that the first speech data meets the first training criterion can be understood as the first speech data being higher than a first threshold, equal to, and / or within a first range. For example, even when the first speech data is equal to the first threshold, it can still be determined that the first speech data meets the first training criterion for precautionary reasons. The first training criterion may be based on a standard speaker score. Determining whether the first speech data meets the first training criterion may include determining whether the first speaker score of the first speech data meets the standard speaker score.

[0108] In one or more exemplary methods, the training criterion (e.g., a first training criterion) is based on speaker metric data, such as first speaker metric data. The first training criterion may be determined based on one or more first speaker metrics. For example, when the first speaker metric data includes a first tone scheduling amount, the first training criterion may include the first tone scheduling amount. The first tone scheduling amount can then be compared with a standard tone scheduling amount and / or a threshold of the tone scheduling amount of the first training criterion.

[0109] In one or more exemplary methods, the training criterion is based on a standard. In one or more exemplary methods, the first training criterion is based on a first standard. The first training criterion can be determined according to the first standard. The first standard may include one or more standard values ​​or model values, such as the gold standard. In other words, for one or more speech metrics, the first standard may include one or more standard values ​​indicating the standard or model performance of the speaker, such as the speaker's voice. For one or more speech metrics, the first standard may include one or more standard values ​​indicating optimal performance. For example, the first standard may include standard pitch regulation, standard speech feature metrics, and / or standard voice feature metrics. The first standard may include one or more parameters, for example, defined by the company, manager, and / or the first speaker / user themselves. In other words, training sessions may be repeated until the first speech data meets / reaches the standard. The first standard may be defined as a direct target of the first speech data, such as a first speaker metric and / or a speech metric. For example, the first speech data may meet a pitch score threshold, a minimum level of pitch variation, a speech rate range, and / or a volume range. Alternatively or additionally, the first criterion can be defined as an indirect objective, for example, in the form of a company KPI, a call center KPI, such as improving the Customer Satisfaction Score (CSAT), increasing sales volume, and / or reducing call processing time. For example, when the first criterion is defined as an indirect objective, the first speaker metric can be inferred from a statistical or machine learning model built upon, for example, audio data obtained / analyzed from multiple users / speakers and / or customers within the company. Such a model can predict the relationship between defined KPIs and voice data obtained for one or more users / speakers. For example, in most cases, speaking within a certain speed range may increase CSAT. In one or more exemplary methods, the first training session can be determined based on a trained model, such as a machine learning model. The trained model can, for example, be optimized for one or more voice metrics and / or speaker metrics.

[0110] This method includes determining a training session, also denoted as TS, based on determining that the speech data meets training criteria. The training session TS can be determined based on audio data AD, speech data VD, and / or speaker metric data SPMD. The training session may include one or more voice training modules. The method may include determining a first training session TS_1 based on determining that first speech data meets a first training criterion. The first training session TS_1 can be determined based on first audio data AD_1, first speech data VD_1, and / or first speaker metric data SPMD_1. The first training session can be determined based on a first speaker profile, for example, a first speaker profile determined based on the first audio data (e.g., the first speech data). The first speaker profile may include one or more parameters indicating the first speaker's first speech. The first training session can be adjusted according to metrics or abilities that the first speaker must improve, for example, based on the first training criteria and / or the first speaker metric data. The first training session may include one or more exercises to improve one or more parameters and / or deficiencies of the first speech / first speaker. The first training session may be based on a first training criterion. For example, the first training session includes one or more exercises to improve one or more parameters that meet the first training criterion. The training session may include one or more training session types, for example, depending on the speech parameters / metrics / abilities to be trained. The first training session may include one or more exercises based on the priority of which speech parameters / metrics / abilities the first speaker should improve first. The first training session may include one or more speech training / guided exercises. The first training session can be selected from a list of training sessions, for example, the list of training sessions includes one or more standard training sessions. The training session may include a training session configuration or layout. In one or more exemplary methods, the first training session may be a personalized training session for the first speaker. For some speakers / users, the first speaker metric may be compensated by another speaker metric. For example, for some speakers / users, speech rate may have a direct impact on poor CSAT performance. For other speakers / users, speech rate may be compensated by another speaker metric. For example, a user / speaker who speaks very fast or very slow may still perform well on the CSAT. In one or more exemplary methods, the first training session may be determined based on a training model, for example, a machine learning model. The training model may, for example, utilize reinforcement learning. The training model can, for example, utilize reinforcement learning based on finding personalized target speech metrics and / or speaker metrics.

[0111] In one or more exemplary methods, a training session may include a training plan. In other words, a first training session may include a first training plan. The training plan may include training content, training complexity / level, training frequency, training time, follow-up plans / schemes, and / or training duration. The training duration of the first training session may range from 2 minutes to 45 minutes, for example, 10 minutes, 15 minutes, 20 minutes, or 30 minutes. The training plan may be based on the user's historical performance, experience, changes in role / function, such as moving to a different team, and / or previously completed exercises.

[0112] In one or more exemplary methods, determining a training session includes, for example, determining a training representation based on audio data AD, speech data VD, speaker metric data SPMD, defect data, and / or criteria, whereby the training representation is also denoted as TR. In other words, the method includes including the training representation TR in the training session TR. The training representation may indicate, for example, one or more exercises to be performed by a speaker. In one or more exemplary methods, determining a first training session includes, for example, determining a first training representation TR_1 based on first audio data, first speech data, first speaker metric data, first defect data, and / or a first criterion. In other words, the method includes including the first training representation TR_1 in the training session TR_1. Determining the first training representation TR_1 may include determining one or more practice representations, for example, including a first practice representation, whereby the first practice representation is also denoted as ER_1. The training representation may include one or more training representation types, for example, depending on the speech parameters / metrics / abilities to be trained. In other words, the first training representation may include a first training session type, for example, depending on the speech parameters / metrics / abilities to be trained by a first speaker. For example, the first training representation may be based on first defect data determined to indicate one or more defects in the first speaker's first speech. The first training representation may include one or more exercises selected from a list of exercises obtained, for example, from a database on a server device. The first training representation may be selected from a list of training representations, which may include one or more standard training representations. The training representation may include a training representation configuration or layout. The first training representation may include an audio representation, such as auditory feedback and / or audio icons.

[0113] In one or more exemplary methods, a training session TS includes one or more exercises, also denoted as EX_i, i = 1, 2, ..., X, where X is the number of exercises. The one or more exercises may represent defect data DD. In one or more exemplary methods, a first training session TS_1 includes one or more first exercises, also denoted as EX_1_i, i = 1, 2, ..., x, where x is the number of first exercises. Exercises may include durations, such as 10 minutes, 15 minutes, 20 minutes, or 30 minutes. Exercises may prompt a first speaker / user to utter phrases or sentences with specific speech metrics, such as speech features that the first training session and / or the exercise itself may target. The first speaker may be recorded during the exercise, for example, while uttering the phrase or sentence, to obtain audio / speech data. The audio / speech data from the exercise may be analyzed, and it may be determined whether the audio / speech data meets the training criteria. Based on the determination that the audio / speech data meets the training criteria, the method may proceed to another exercise, repeat the same exercise, or stop. In other words, determining the first training session may include determining one or more first exercises and including them in the first training session. One or more first exercises EX_1_i may represent first defect data DD_1. One or more first exercises EX_1_i may represent first audio data AD, first speech data VD, and / or first speaker metric data SPMD. In other words, one or more first exercises may be configured to train a first speaker's first voice to remedy or improve at least some of one or more defects in the first speech. One or more first exercises EX_1_i may be configured to train a first speaker's first speech to improve one or more metrics / parameters, such as the first speaker's first speech's voice metrics / parameters, such as the first speaker's first speech's pitch, speech features, and / or voice features. One or more first exercises EX_1_i may include exercises for improving understanding of speech, voice, and / or pitch.

[0114] In one or more exemplary methods, one or more first exercises include one or more of speech tone exercises, speech feature exercises, and voice feature exercises. In other words, determining one or more first exercises may include determining one or more of speech tone exercises, speech feature exercises, and voice feature exercises. Speech tone exercises may be exercises for improving, for example, speech tone defects in a first speaker's first voice. Speech feature exercises may be exercises for improving speech feature defects in a first speaker's first voice. Voice feature exercises may be exercises for improving voice feature defects in a first speaker's first voice.

[0115] The method includes outputting first training information indicating a first training session via an interface of a voice guidance device. In other words, outputting the first training information may include outputting the first training session via an interface of the voice guidance device. Outputting the first training information may include displaying a first user interface indicating the first training information. The user interface may include one or more (e.g., multiple) user interface objects. Outputting the first training information may, for example, instruct a first speaker / user on the voice guidance device to prepare to perform the first training session. For example, the first user interface may include one or more first user interface objects, such as a first primary user interface object and / or a first secondary user interface object. A user interface object may refer herein to a graphical representation of an object displayed on an interface (e.g., a display) of the voice guidance device. User interface objects may be user-interactive or selectable by user input. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may optionally constitute user interface objects. User interface objects may form part of a widget. A widget may be considered as a small application that a user can use.

[0116] In one or more exemplary methods, the first training information is a suggestion for performing a first training session, and wherein outputting the first training information includes displaying a training user interface object on a display of the interface. The training user interface object may indicate a first training session, a first training representation, and / or one or more first exercises. The training user interface object may be included in a first user interface indicating the first training information. The training user interface object may include one or more of images (e.g., icons), buttons, and text (e.g., hyperlinks) that can guide the user / first speaker to the first training session for training.

[0117] The first training information may include notifying the user / first speaker to be trained (e.g., email, SMS, and / or calendar invitation) to execute and / or to notify him / her that the first training session is available. In other words, the first training information may include populating a calendar with the first training session, such as a dedicated training calendar and / or the user / first speaker's calendar. The first training information may include populating the user / first speaker's task list so that the user / first speaker can see one or more training sessions he / she must execute.

[0118] In one or more exemplary methods, the method includes determining that a user has completed a first training session. Determining that a user has completed a first training session may include detecting the termination of the execution of the first training session, such as the termination of the execution of one or more first exercises, for example, termination, end, or sustaining the first training session.

[0119] In one or more exemplary methods, the method includes obtaining second audio data of the first speech, also denoted as AD_2, for example, based on a determination that the user has performed a first training session. In other words, the second audio data can be obtained after the first speaker / user has performed the first training session. The description of the first audio data AD_1 can also be applied to the second audio data AD_2. The acquisition of the second audio data may be later than that of the first audio data.

[0120] In one or more exemplary methods, the method includes obtaining second speech data based on second audio data AD_2, the second speech data also denoted as VD_2. The second speech data VD_2 may include data indicating the first speech, such as data of a first speaker. Obtaining the second speech data may include generating the second speech data of the first speech based on the second audio data. The second speech data may be second agent speech data, such as the second speech data of a first user / speaker / agent in a voice guidance system. The second speech data may include speaker metric data also denoted as SPMD, such as second speaker metric data SPMD_2.

[0121] In one or more exemplary methods, obtaining second speech data based on second audio data includes determining second speaker measurement data of a first speaker based on the second audio data. The second speaker measurement data includes second primary speaker measurement data, which is also denoted as SPMD_2_1.

[0122] In one or more exemplary methods, obtaining the second speech data includes determining second defect data DD_2 that indicates one or more defects in the first speech.

[0123] The descriptions of the first speech data VD_1, the first speaker measurement data, and the first defect data can also be applied to the second speech data VD_2, the second speaker measurement data, and the second defect data, respectively. The second speech data VD_2, the second speaker measurement data, and the second defect data can be different from or correspond to the first speech data VD_1, the first speaker measurement data, and the first defect data, respectively. The second speech data may be obtained later than the first speech data.

[0124] In one or more exemplary methods, the method includes determining feedback information, also referred to as FB, based on speech data VD that indicates feedback during a training session.

[0125] In one or more exemplary methods, the method includes determining first feedback information, which indicates feedback from a first training session, based on second speech data. The first training session feedback is also denoted as TFB_1, and the first feedback information is also denoted as FB_1.

[0126] Determining the first feedback information FB_1 may include determining the first training session feedback based on the first training session, the second audio data, and / or the second speech data.

[0127] The first feedback information may indicate the result of a first training session. The method may include, for example, outputting the first feedback information via the interface of a voice guidance device. Outputting the first feedback information may include displaying a first user interface indicating the first feedback information. The user interface may include one or more (e.g., multiple) user interface objects. For example, the first user interface may include one or more first user interface objects, such as a first primary user interface object and / or a first secondary user interface object. A user interface object may refer herein to a graphical representation of an object displayed on the interface of a voice guidance device, such as a display. User interface objects may be user-interactive or selectable by user input. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may optionally constitute user interface objects. User interface objects may form part of a widget. A widget may be considered as a small application that a user can use.

[0128] In one or more exemplary methods, the first feedback information is a suggestion to the user / speaker to view first training session feedback on the executed first training session and second voice data. The first feedback information may include notifying the user / first speaker to be trained (e.g., email, SMS, and / or calendar invitation) to view and / or notify that the first training session feedback is available to him / her. In other words, the first feedback information may include populating a calendar with the first training session feedback, such as a dedicated training calendar and / or the user / first speaker's calendar. The first feedback information may include populating the user / first speaker's task list so that the user / first speaker can see one or more training session feedbacks he / she has received.

[0129] The initial feedback information may be based on defect data and / or strength data. The initial feedback information may also depend on feedback from the first training session.

[0130] In other words, training information can provide feedback to the user / first speaker about his / her speech parameters / features, such as feedback related to defect data and / or intensity data. First feedback information based on defect data can provide the first speaker with feedback on which speech parameters / features need improvement. First feedback information based on intensity data can provide positive feedback to the first speaker, for example, by showing the improvement of one or more speech parameters / features and / or the intensity of one or more of those speech parameters / features over time, thus motivating and increasing the first speaker's awareness.

[0131] First training session feedback can provide the user / first speaker with feedback on his / her speech parameters / features, such as feedback related to defect data and / or intensity data. First training session feedback based on defect data can provide the first speaker with feedback on which speech parameters / features need improvement. First training session feedback based on intensity data can provide the first speaker with positive feedback, for example, by showing the improvement of the intensity of one or more speech parameters / features over time, thus motivating the first speaker. First training session feedback can include feedback related to the results or consequences of the first training session performed for the first speaker / user. In other words, first training session feedback can give the first speaker / user an indication of whether he / she has improved on one or more speech parameters / metrics.

[0132] In one or more exemplary methods, the method includes outputting first feedback information via an interface of a voice guidance device. The description of outputting first training information can also be applied to outputting the first feedback information.

[0133] In one or more exemplary methods, the method includes obtaining second speech data based on second audio data. The description of first speech data VD_1 can also be applied to second speech data VD_2. The acquisition of the second speech data may be later than the acquisition of the first speech data. For example, the second audio data and / or the second speech data may be acquired within a defined time period or time interval after the first speaker has performed a first training session. For example, the second audio data and / or the second speech data may be acquired days, weeks, and / or months after performing the first training session, for example, to determine the result or consequence of performing the first training session on the first speech.

[0134] In one or more exemplary methods, the method includes determining whether second speech data meets a second training criterion. The description of determining whether first speech data meets a first training criterion can also be applied to determining whether second speech data meets a second training criterion. The description of the first training criterion can be applied to the second training criterion.

[0135] In one or more exemplary methods, the second training criterion may include a second threshold greater than a first threshold of the first criterion. For example, a first speaker may have improved one or more speech parameters / features after performing a first training session. The threshold used to determine whether the first speaker needs to perform further training (e.g., a second training session) may increase proportionally to the first speaker's performance, for example, for continued improvement in the first speaker's first speech. In one or more exemplary methods, the first criterion may have been determined to be too high relative to the first speech. Therefore, the second training criterion may include a second threshold lower than the first threshold, for example, when the first speaker may have deteriorated one or more speech parameters / features after performing a first training session.

[0136] In one or more exemplary methods, the method includes determining a second training session based on determining that second speech data meets a second training criterion. The description of determining whether first speech data meets the first training criterion can also be applied to determining whether second speech data meets the second training criterion. After a period of time, a first speaker's first speech may have deteriorated; for example, one or more speech parameters / features may have deteriorated. Therefore, it can be determined whether the first speaker's first speech still meets a certain performance after that period of time. Therefore, after one or more training sessions (e.g., the first training session) have been performed, the first speaker can be continuously retrained. The second training criterion can indicate whether the user / speaker should be guided or trained for defects and / or speaker metrics in the first speech that differ from the first training criterion. In other words, the first training session can be configured to improve a first set of defects in the first speech, and the second training session can be configured to improve a second set of defects in the first speech. For example, the second training criterion may include a second speech pitch criterion / parameter, a second speech feature criterion / parameter, a second voice feature criterion / parameter, and / or a second time parameter / criterion (e.g., a time limit that the first speaker must perform before the second training session). In other words, the second training criterion may include one or more second conditions that the first speech data must meet.

[0137] In one or more exemplary methods, the method includes outputting second training information instructing a second training session via an interface of a voice guidance device. The description of outputting first training information can also be applied to outputting second training information. The second training information may be combined with first feedback information. For example, when the first feedback information instructs feedback for a first training session including suggestions for performing further training, the first feedback information may also include second training information instructing a second training session. In other words, if it is determined that a first speaker needs more training, the first feedback information may also instruct the first speaker that he / she should perform a second training session. In other words, when prior training information (e.g., first training information and / or first feedback information / reports) is available, the second training information may include second feedback information that includes a comparison showing changes relative to the first feedback information. For example, the second feedback information may include highlighting one or more improvements or deteriorations, such as the first speaker's / user's weaknesses and / or strengths.

[0138] In one or more exemplary methods, the method includes aggregating first speech data and second speech data into speech data. The speech data may include the first speech data and the second speech data. The speech data may include a comparison of the first speech data and the second speech data.

[0139] In one or more exemplary methods, the method includes determining whether speech data meets a third training criterion. The description of determining whether first speech data meets a first training criterion can also be applied to determining whether speech data meets a third training criterion. The description of the first training criterion can be applied to the third training criterion. The third criterion can be based on a comparison of the first speech data and the second speech data. For example, a first speaker may have improved one or more speech parameters / features after performing a first training session.

[0140] In one or more exemplary methods, the method includes determining a third training session based on determining that the speech data meets a third training criterion.

[0141] After a period of time, the first speaker's initial speech may have deteriorated; for example, one or more speech parameters / features may have worsened. Therefore, it can be determined whether the first speaker's initial voice still meets a certain performance standard after that time period. Thus, after one or more training sessions (e.g., the first training session) have been performed, the first speaker can be continuously retrained. A third training criterion can indicate whether the user / speaker should be guided or trained for different defects and / or speaker metrics in the first speech that differ from the first and second training criteria. In other words, the first training session can be configured to improve a first set of defects in the first speech, the second training session can be configured to improve a second set of defects in the first speech, and the third training session can be configured to improve a third set of defects in the first speech. For example, the third training criterion may include a third speech pitch criterion / parameter, a third speech feature criterion / parameter, a third voice feature criterion / parameter, and / or a third time parameter / criterion (e.g., a time limit that the first speaker must meet before performing the third training session). In other words, the third training criterion may include one or more third conditions that the first speech data must satisfy.

[0142] In one or more exemplary methods, the method includes outputting third training information instructing a third training session via an interface of a voice guidance device. The description of outputting first training information can also be applied to outputting third training information. The third training information may be combined with first feedback information. For example, when the first feedback information indicates first training session feedback including suggestions for performing further training, the first feedback information may also include third training information instructing a third training session. In other words, if it is determined that a first speaker needs more training, the first feedback information may also instruct the first speaker that he / she should perform a third training session.

[0143] A voice guidance system is disclosed. The voice guidance system includes a server device and a voice guidance device. The voice guidance device (e.g., a processor of the voice guidance device) is configured to perform any method according to this disclosure.

[0144] A voice guidance system is disclosed. The voice guidance system includes a server device and a voice guidance device. The voice guidance device includes a processor, memory, and an interface. The voice guidance system is configured to, for example, use the voice guidance device and / or the server device to obtain audio data representing one or more voices, the audio data including first audio data of a first voice.

[0145] Voice guidance devices may include, for example, one or more of mobile phones, computers, and tablets. A voice guidance device may be, for example, a user device, such as a mobile phone or computer, configured to perform a call between a user and another person. A voice guidance device may be configured to receive a first audio input, such as first audio input from a conversation or call between a user and another person. For example, a voice guidance device may act as a call agent device, where the user can be an agent, such as an agent of a call center (e.g., a support call center, after-sales call center, marketing call center, or sales call center). A voice guidance device may be, for example, a user device, such as a mobile phone or computer, configured to record the first audio input from a first speaker / user, such as recording the first speaker speaking or talking. A voice guidance device may be configured to receive one or more audio signals, such as those included in audio data. A voice guidance device may be configured to generate one or more audio signals, including, for example, the first audio signal included in the first audio data. The first audio signal may be based on the first audio input, and the first audio data may be based on the first audio signal.

[0146] The voice guidance system is configured to obtain first voice data based on first audio data, for example, using a voice guidance device and / or a server device.

[0147] The voice guidance system is configured to, for example, use a voice guidance device and / or a server device to determine whether the first voice data meets the first training criteria.

[0148] The voice guidance system is configured to determine a first training session based on the determination that the first voice data meets a first training criterion, for example, by using a voice guidance device and / or a server device.

[0149] The voice guidance system is configured to output first training information instructing a first training session via an interface of the voice guidance device. The interface of the voice guidance device may include a first display. The processor of the voice guidance device may be configured to display one or more user interfaces (e.g., user interface screens) on the interface (e.g., the first display), the user interfaces including a first user interface and / or a second user interface, for example, as a representation of the first training session and / or the first training (e.g., including one or more exercises). The user interface may include one or more (e.g., multiple) user interface objects. For example, the first user interface may include a first primary user interface object and / or a first secondary user interface object. The second user interface may include a second primary user interface object and / or a second secondary user interface object. The user interface may be referred to as a user interface screen.

[0150] Input (e.g., user input) may include touch (e.g., tap, forced touch, long press), click (e.g., mouse click), typing (e.g., typing on a keyboard), and / or movement of contact (e.g., swipe gestures, for example, for switching). Movement during contact may be detected by a touch-sensitive surface, for example, on a first display of a voice guidance device. Therefore, the first display may be a touch-sensitive display. The first input (e.g., the first user input) may include lifting. User input (e.g., the first primary user input, the second primary user input, and / or the second primary user input) may include touch and subsequent lifting movement.

[0151] The first display of the voice guidance device can be configured to detect user input, such as a first primary user input. For example, when the first display includes a touch-sensitive display, the user input can include touch input from the user. The user input can include touch points on the touch-sensitive display and / or a keyboard contained in or connected to the voice guidance device. The touch-sensitive display can provide a first user interface and / or a second user interface (e.g., an input interface) and an output interface between the voice guidance device and the user. The processor of the voice guidance device can be configured to receive electrical signals from and / or send electrical signals to the touch-sensitive display. The touch-sensitive display can be configured to display visual output to the user, such as a first training session and / or first training information. The visual output can optionally include graphics, text, icons, video, audio icons, and any combination thereof (collectively, "graphics"). For example, some, most, or all of the visual output can be considered to correspond to user interface objects. The voice guidance device can also be configured to output a first training representation including audio output, such as sound representation, audio icons, text output, graphic output, and / or auditory feedback.

[0152] A voice guidance system can be a system for guiding and / or training one or more users (e.g., one or more speakers). A voice guidance system can be a system for monitoring, processing, and / or analyzing one or more audio signals, such as a speaker speaking, for example, as a monologue or dialogue. The system can be a system for monitoring, processing, and / or analyzing one or more audio signals, such as a dialogue between two or more people, such as a conversation in a telephone or conference. The system can, for example, include or act as a call center system for monitoring, processing, and / or analyzing one or more audio signals, such as a dialogue between two or more people, for example, a telephone call between an agent of the call center system and a customer or caller.

[0153] It should be understood that the description of the method also applies to the corresponding features in the voice guidance device, server device and / or system.

[0154] Figure 1 An exemplary voice guidance system according to this disclosure is schematically illustrated, such as voice guidance system 2. System 2 includes a voice guidance device 10 and an optional server device 20. The voice guidance device 10 includes a memory 10A, one or more interfaces 10B, and a processor 10C. The server device 20 includes a memory 20A, one or more interfaces 20B, and one or more processors 20C. A user / first speaker 1A can use the voice guidance device 10 (e.g., as a mobile phone or computer) to perform or receive training or guidance. The voice guidance device may optionally be configured to perform or receive calls from a speaker 1B (e.g., a second speaker). Speaker 1B can communicate with user 1A using a speaker electronics device 30.

[0155] Voice guidance device 10 can be configured to act as a user device that user 1A can use for training or guidance (e.g., for performing one or more training sessions). Voice guidance device 10 can also be configured to act as a user device that user 1A can use to communicate with speaker 1B and / or monitor calls / conversations. Voice guidance device 10 (e.g., processor 10C) is configured to acquire audio data 4, 14 representing one or more voices, including first audio data of a first voice. The first voice may, for example, be the voice of user 1A. The audio data may include one or more audio signals including a first audio signal. The first audio signal 22 may be acquired, for example, via network 40 (e.g., a global network such as the Internet or a telecommunications network) from speaker electronic device 30. Audio data 14 (e.g., the first audio data) may be acquired, for example, via network 40 (e.g., a global network such as the Internet or a telecommunications network) from server device 20. Audio data (e.g., the first audio data) may be acquired from memory 10A. Voice guidance device 10 (e.g., processor 10C) is configured to acquire the first voice data 4, 14 based on the first audio data. The voice guidance device 10 (e.g., processor 10C) can be configured to acquire first voice data on the voice guidance device 10 itself, for example, to determine the first voice data. Optionally, one or more processors 20C are configured to acquire voice data (e.g., the first voice data), for example, to determine the voice data.

[0156] Speaker electronic device 30 can be configured to record audio input 32 (e.g., a first audio input) from speaker 1B, such as recording speaker 1B speaking or talking. Electronic device 30 can be configured to acquire one or more audio signals, such as generating one or more audio signals based on audio input 32, including a first audio signal based on the first audio input. Speaker electronic device 30 can be configured to send 22 audio data, such as the first audio signal, to voice guidance device 10, for example, via network 40. Speaker electronic device 30 can be configured to acquire 24 or more audio signals from voice guidance device 10, for example, based on user input 4 (e.g., user audio input). User input 4 can be user 1A speaking or talking; for example, voice guidance device 10 records user 1A speaking or talking. User 1A can be a first speaker and / or a second speaker.

[0157] The voice guidance system 2 is configured to determine whether the first voice data meets the first training criterion.

[0158] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine whether the first voice data meets the first training criteria.

[0159] Optionally, one or more processors 20C are configured to determine whether the first speech data meets the first training criteria.

[0160] The voice guidance system 2 is configured to determine a first training session based on the determination that the first voice data meets the first training criteria.

[0161] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine a first training session based on determining that the first voice data meets a first training criterion.

[0162] Optionally, one or more processors 20C are configured to determine a first training session based on determining that the first voice data meets a first training criterion. The one or more processors 20C may be configured to, for example, transmit the first training session to the voice guidance device 10 via network 40.

[0163] The voice guidance system 2 is configured to output 6 instructions on the first training information of the first training session via the interface 10B of the voice guidance device 10.

[0164] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to output 6 instructions for the first training session via the interface 10B of the voice guidance device 10.

[0165] Optionally, one or more processors 20C may be configured to transmit first training information 18, indicating the first training session, to the voice guidance device 10, for example, via network 40 (e.g., via interface 20B). Optionally, one or more processors 20C may be configured to output 6, 18, indicating the first training information of the first training session via interface 10B of the voice guidance device 10.

[0166] Optionally, the voice guidance system 2 is configured to determine first speaker measurement data of a first speaker based on first audio data, the first speaker measurement data including first primary speaker measurement data.

[0167] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine first speaker measurement data of a first speaker based on first audio data, the first speaker measurement data including first primary speaker measurement data.

[0168] Optionally, one or more processors 20C are configured to determine first speaker measurement data of a first speaker based on first audio data, the first speaker measurement data including first primary speaker measurement data. Optionally, one or more processors 20C may be configured to transmit, for example via network 40 (e.g., via interface 20B), the first speaker measurement data including the first primary speaker measurement data to the voice guidance device 10.

[0169] Optionally, the voice guidance system 2 is configured to determine first defect data indicating one or more defects in the first voice.

[0170] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine first defect data indicating one or more defects in the first voice.

[0171] Optionally, one or more processors 20C are configured to determine first defect data indicating one or more defects in the first voice. Optionally, one or more processors 20C may be configured to transmit the first defect data 18 to the voice guidance device 10, for example, via network 40 (e.g., via interface 20B).

[0172] Optionally, the voice guidance system 2 is configured to determine a first training representation.

[0173] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine a first training representation.

[0174] Optionally, one or more processors 20C are configured to determine a first training representation. Optionally, one or more processors 20C may be configured to transmit the first training representation 18 to the voice guidance device 10, for example, via network 40 (e.g., via interface 20B).

[0175] Optionally, the voice guidance system 2 is configured to display training user interface objects on the display 6 of the interface 10B.

[0176] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to display 6 training user interface objects on the display of interface 10B.

[0177] Optionally, one or more processors 20C are configured to transmit 18 training user interface objects 6 to be displayed on the display of interface 10B.

[0178] Optionally, the voice guidance system 2 is configured to determine that the user has performed a first training session. Optionally, the voice guidance system 2 is configured to obtain second audio data of the first speech. Optionally, the voice guidance system 2 is configured to obtain second speech data based on the second audio data. Optionally, the voice guidance system 2 is configured to determine first feedback information indicating feedback in the first training session based on the second speech data. Optionally, the voice guidance system 2 is configured to output the first feedback information via the interface 10B of the voice guidance device 10.

[0179] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine that the user has performed a first training session. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to obtain second audio data of the first speech. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to obtain second speech data based on the second audio data. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine first feedback information indicating feedback from the first training session based on the second speech data. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to output the first feedback information via the interface 10B of the voice guidance device 10.

[0180] Optionally, one or more processors 20C are configured to determine that the user has performed a first training session. Optionally, one or more processors 20C are configured to obtain second audio data of the first speech. Optionally, one or more processors 20C are configured to obtain second speech data based on the second audio data. Optionally, one or more processors 20C are configured to determine first feedback information indicating feedback from the first training session based on the second speech data. Optionally, one or more processors 20C are configured to output the first feedback information via the interface 10B of the voice guidance device 10.

[0181] Optionally, the voice guidance system 2 is configured to obtain second voice data based on the second audio data. Optionally, the voice guidance system 2 is configured to determine whether the second voice data meets a second training criterion. Optionally, the voice guidance system 2 is configured to determine a second training session based on the determination that the second voice data meets the second training criterion. Optionally, the voice guidance system 2 is configured to output second training information indicating the second training session via the interface of the voice guidance device.

[0182] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to obtain second voice data based on the second audio data. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine whether the second voice data meets a second training criterion. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine a second training session based on the determination that the second voice data meets the second training criterion.

[0183] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to output second training information indicating the second training session via the interface of the voice guidance device.

[0184] Optionally, one or more processors 20C are configured to obtain second speech data based on the second audio data. Optionally, one or more processors 20C are configured to determine whether the second speech data meets a second training criterion. Optionally, one or more processors 20C are configured to determine a second training session based on the determination that the second speech data meets the second training criterion. Optionally, one or more processors 20C are configured to output second training information indicating the second training session via the interface of the voice guidance device.

[0185] Optionally, the voice guidance system 2 is configured to aggregate the first voice data and the second voice data into voice data. Optionally, the voice guidance system 2 is configured to determine whether the voice data meets a third training criterion. Optionally, the voice guidance system 2 is configured to determine a third training session based on the determination that the voice data meets the third training criterion. Optionally, the voice guidance system 2 is configured to output third training information indicating the third training session via the interface of the voice guidance device.

[0186] Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to aggregate the first voice data and the second voice data into voice data. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine whether the voice data meets a third training criterion. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to determine a third training session based on the determination that the voice data meets the third training criterion. Optionally, the voice guidance device 10 (e.g., processor 10C) is configured to output third training information indicating the third training session via the interface of the voice guidance device.

[0187] Optionally, one or more processors 20C are configured to aggregate the first speech data and the second speech data into speech data. Optionally, one or more processors 20C are configured to determine whether the speech data meets a third training criterion. Optionally, one or more processors 20C are configured to determine a third training session based on the determination that the speech data meets the third training criterion. Optionally, one or more processors 20C are configured to output third training information indicating the third training session via the interface of the speech guidance device.

[0188] The voice guidance device 10 and / or server device 20 can be configured to perform in Figure 2A , Figure 2B Any method disclosed in the document.

[0189] The voice guidance device 10 (e.g., processor 10C) is optionally configured to execute Figures 2A to 2B Any operation disclosed herein (e.g., any one or more of S104A, S104B, S104C, S104D, S106, S110A, S110B, S111, S112, S114A, S116). The operation of the voice guidance device may be embodied in the form of an executable logic routine (e.g., lines of code, software program, etc.) stored on a non-transitory computer-readable medium (e.g., memory 10A) and executed by processor 10C.

[0190] Furthermore, the operation of the voice guidance device 10 can be considered as a method configured to be performed by the voice guidance device 10. Additionally, while the described functions and operations can be implemented in software, such functions can also be implemented via dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0191] The server device (e.g., processor 20C) is optionally configured to execute Figures 2A to 2BAny operation disclosed herein (e.g., any one or more of S104A, S104B, S104C, S104D, S106, S110A, S110B, S111, S112, S114A, S116). The operation of the server device may be embodied in the form of executable logical routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 20A) and executed by one or more processors 20C. Furthermore, the operation of the server device 20 can be considered as a method configured to be performed by the server device 20. Moreover, while the described functions and operations can be implemented in software, such functions can also be implemented via dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0192] Figure 2A and Figure 2B A flowchart of an exemplary method is shown. A method 100 for operating a voice guidance system including a voice guidance device is disclosed. The voice guidance device includes an interface, a processor, and a memory. Method 100 includes obtaining audio data AD in step S102, including first audio data AD_1 of a first speech (e.g., a first speaker) and optional second audio data AD_2. The method includes obtaining speech data VD based on the audio data AD in step S104. The method may include obtaining the first speech data VD_1 based on the first audio data AD_1. Optionally, the method includes obtaining the second speech data VD_2 based on the second audio data AD_2.

[0193] In one or more exemplary methods, obtaining S104 first speech data VD_1 based on first audio data AD_1 includes determining S104A first speaker measurement data SPMD_1 based on the first audio data AD_1, wherein the first speaker measurement data SPMD_1 includes first primary speaker measurement data SPMD_1_1.

[0194] In one or more exemplary methods, obtaining second speech data VD_2 based on second audio data AD_2 in S104 includes determining second speaker measurement data SPMD_2 of the first speaker in S104C based on the second audio data AD_2, wherein the second speaker measurement data SPMD_2 includes second primary speaker measurement data SPMD_2_1.

[0195] In one or more exemplary methods, obtaining the first speech data VD_1 in S104 includes determining first defect data DD_1 that indicates one or more defects in the first speech in S104B.

[0196] In one or more exemplary methods, obtaining S104 second speech data VD_2 includes determining S104D second defect data DD_2 indicating one or more defects in the first speech.

[0197] Method 100 includes determining whether the S108 speech data VD meets training criterion TC. Optionally, determining whether the S108 speech data VD meets training criterion TC includes determining whether the first speech data VD_1 meets the first training criterion TC_1. Optionally, determining whether the S108 speech data VD meets training criterion TC includes determining whether the second speech data VD_2 meets the second training criterion TC_2.

[0198] In one or more exemplary methods, method 100 includes aggregating first speech data VD_1 and second speech data VD_2 into speech data VD for S106.

[0199] Optionally, determining whether the speech data VD in S108 meets the training criterion TC includes determining whether the speech data VD meets the third training criterion TC_3. Optionally, when it is uncertain whether the speech data VD meets the training criterion TC, method 100 includes repeating method 100.

[0200] Method 100 includes determining the S110 training session TS based on the determined speech data VD satisfying the A training criterion TC.

[0201] Optionally, method 100 includes determining a first training session TS_1 in S110 based on determining that the first speech data VD_1 meets a first training criterion TC_1. Optionally, method 100 includes determining a second training session TS_2 in S110 based on determining that the second speech data VD_2 meets a second training criterion TC_2. Optionally, method 100 includes determining a third training session TS_3 in S110 based on determining that the speech data VD meets a third training criterion TC_3. Optionally, when it is determined that the first speech data VD_1 does not meet the first training criterion TC_1, method 100 includes repeating method 100 in B. Optionally, when it is determined that the second speech data VD_2 does not meet the second training criterion TC_2, method 100 includes repeating method 100 in B. Optionally, when it is determined that the speech data VD does not meet the third training criterion TC_3, method 100 includes repeating method 100 in B.

[0202] In one or more exemplary methods, determining S110 first training session TS_1 includes determining S110A first training representation TR_1, which includes, for example, one or more exercises, such as first exercise EX_1_i. Optionally, determining S110 second training session TS_2 includes determining S110A first training representation TR_2, which includes, for example, one or more exercises, such as second exercise EX_2_i. Optionally, determining S110 third training session TS_3 includes determining S110A first training representation TR_3, which includes, for example, one or more exercises, such as third exercise EX_3_i.

[0203] In one or more exemplary methods, method 100 includes determining that user S111 has performed a first training session TS_1.

[0204] In one or more exemplary methods, method 100 includes determining feedback information FB, which instructs training session feedback TFB, based on speech data VD. In one or more exemplary methods, method 100 includes determining first feedback information FB_1, which instructs first training session feedback TFB_1, based on second speech data VD_2.

[0205] Method 100 includes outputting training information TI indicating a training session TS via an interface of a voice guidance device, S116. Optionally, outputting training information TI via S116 includes outputting first training information TI_1 indicating a first training session TS_1. In one or more exemplary methods, outputting first training information TI_1 includes displaying a training user interface object on a display of the interface, S114A. Optionally, outputting training information TI via S116 includes outputting second training information TI_2 indicating a second training session TS_2. Optionally, outputting training information TI via S116 includes outputting third training information TI_3 indicating a third training session TS_3.

[0206] In one or more exemplary methods, method 100 includes outputting feedback information FB via an interface of a voice guidance device, S116. Optionally, outputting feedback information FB includes outputting first feedback information FB_1.

[0207] Figure 3An exemplary data structure according to this disclosure is illustrated schematically. Audio data AD includes first audio data AD_1 of a first speech. Audio data AD includes speech data VD. Optionally, audio data AD includes second audio data AD_2 of the first speech. First audio data AD_1 includes speech data VD containing the first speech data VD_1. Optionally, second audio data AD_2 includes speech data VD containing the second speech data VD_2. First speech data VD_1 optionally includes first speaker measurement data SPMD_1, which optionally includes first primary speaker measurement data SPMD_1_1. First speech data VD_1 optionally includes first defect data DD_1. Second speech data VD_2 optionally includes second speaker measurement data SPMD_2, which optionally includes second primary speaker measurement data SPMD_2_1. Second speech data VD_2 optionally includes second defect data DD_2.

[0208] Figure 4 An exemplary voice guidance device according to this disclosure is illustrated, such as voice guidance device 10. In this example, voice guidance device 10 is a laptop computer. Voice guidance device 10 may be, for example, a voice guidance device for training purposes by a user (e.g., a first speaker), such as a call center or a support call center agent. Voice guidance device 10 is configured to display a user interface indicating training information on an interface 10B of voice guidance device 10 (e.g., on the display 11 of the laptop computer). Voice guidance device 10 displays a first user interface UI_1, for example, having a first display area including a plurality of training user interface objects. After it has been determined that the first voice data meets a first training criterion based on the first audio data of the user / first voice, the user of voice guidance device 10 can visualize the first training information, such as a first training session including the first exercise. The first training information may provide the user with feedback on the first audio of the user's first voice, such as deficiencies to be improved for the user.

[0209] The first user interface (UI_1) indicates a first exercise, which is, for example, part of a first training session. The first exercise is designed to improve the user's understanding of speech, speech patterns, and / or intonation. The first user interface (UI_1) includes, for example, a first training user interface object 50 located in a first main display area. The first training user interface object 50 represents speech segments of a pre-recorded dialogue between, for example, a first speaker A acting as an agent and a second speaker B acting as a client / customer. The first training user interface object 50 includes three speech segments for the first speaker A and three speech segments for speaker B. The first user interface (UI_1) includes, for example, a second training user interface object 52 located in a first auxiliary display area. The second training user interface object 52 represents a timeline of the dialogue. The first user interface (UI_1) includes, for example, a third training user interface object 54 located in a first third display area. The third training user interface object 54 represents multiple sets of emotion type identifiers. Each speech segment in the first training user interface object 50 may have an associated set of emotion type identifiers in the third training user interface object 54. For example, the first speech segment of the first speaker A may have an associated first emotion type identifier set ST_ID_SET_1.

[0210] The user performing the first exercise can then, for example, label each speech segment they have listened to with an emotion type identifier selected from this set of emotion type identifiers. By labeling speech segments with emotion type identifiers, the user can identify one or more emotional states of the first speaker A and / or the second speaker B based on their perception of the speech segments in the pre-recorded dialogue. At the end of the round / dialogue, the voice guidance device 10 can determine first feedback information indicative of feedback for the first training session, for example, by comparing the user's answer / label with the model solution, and the voice guidance device 10 can then be configured to output the first feedback information, for example, by displaying the user's score in the first exercise on the display 11 via the interface 10B based on this comparison. Thus, the first exercise can improve the user's understanding of the speaker's voice, conversation, and / or tone.

[0211] Figure 5 A flowchart of an exemplary method (e.g., method 200 for operating a voice guidance system) is schematically shown, the voice guidance system including voice guidance means for performing training sessions, portions of training sessions and / or exercises.

[0212] According to this disclosure, method 200 can be performed on a voice guidance device (e.g., voice guidance device 10). Method 200 can be adapted for a user to perform a training session, a portion of a training session, and / or exercises included in the training session, such as a second exercise. In this example, voice guidance device 10 can be a laptop computer. Voice guidance device 10 can be, for example, a voice guidance device used by a user (e.g., an agent) for training purposes, such as a call center or an agent supporting a call center.

[0213] Method 200 includes outputting first audio data of the first speech in S202, for example, via an interface 10B (not shown) of a voice guidance device, such as proxy audio of the user / agent's speech. Outputting the first audio data of the first speech in S202 may include outputting a speech segment of the user / agent, wherein the first speech data based on the first audio data has been determined to meet a first training criterion. In other words, outputting the first audio data of the first speech in S202 may include outputting a speech segment of the user / agent for which first defect data (such as one or more defects) has been determined. Method 200 may optionally include outputting text data of S202A, for example, via the interface 10B of the voice guidance device, the text data being, for example, the corresponding text of the audio data, so that the user / agent can re-perform the speech segment. The user / agent of the voice guidance device can thereby listen to his / her own first audio data and optionally read the corresponding text. Optionally, method 200 may include obtaining the first audio data of the first speech, for example, via the interface 10B of the voice guidance device, before outputting the first audio data of S202.

[0214] Method 200 includes obtaining second audio data of the first speech in S204. The second audio data of the first speech may, for example, include a re-recording by the user / agent of the same speech segment output in S202, such as the same corresponding text, for example, a response to the customer in a recorded dialogue. In other words, when performing, for example, a second exercise, the user / agent can re-perform his / her detected defective speech segment with improved speech, manner, and / or tone. For example, the user / agent can re-perform the speech segment with better speed and / or a more aggressive tone.

[0215] Method 200 includes obtaining second speech data (S206) based on second audio data. In other words, obtaining second speech data (S206) based on second audio data may include determining speaker measurement data (SPMD) and / or defect data (DD) of the user / agent based on the second audio data.

[0216] Method 200 includes determining whether the second speech data S208 meets the second training criterion. In other words, determining whether the second speech data S208 meets the second training criterion may include determining whether the second speech data of the user / agent has been sufficiently improved. Alternatively, determining whether the second speech data S208 meets the second training criterion may include determining whether the user / agent needs further training.

[0217] Optionally, the method includes determining first feedback information indicating feedback from the first training session. In other words, determining the first feedback information indicating feedback from the first training session may include determining a score for the re-evaluated speech segment. When the score is above a certain threshold, the score is positive; when the score is below the threshold, the score is negative. When it is determined that the second speech data meets the second training criterion, the method includes repeating S210 to obtain the second audio data of the first speech in S204. In other words, the user / agent can re-perform the speech segment until his / her performance is satisfactory.

[0218] When it is determined that the second speech data does not meet the second training criteria, method 200 proceeds to another speech segment, proceeds to another exercise if the second exercise has ended, or stops if the training session has ended.

[0219] Therefore, the first exercise can improve the user's understanding of the speaker's voice, conversation, and / or tone.

[0220] The use of terms such as "first," "second," "third," "fourth," "primary," "secondary," and "third" does not imply any particular order, but rather serves to identify individual elements. Furthermore, the use of terms such as "first," "second," "third," "fourth," "primary," "secondary," and "third" does not indicate any order or importance; rather, it distinguishes one element from another. Note that the words "first," "second," "third," "fourth," "primary," "secondary," and "third" are used here and elsewhere solely for labelling purposes and are not intended to indicate any particular spatial or temporal order.

[0221] Memory can be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical setup, memory may include non-volatile memory for long-term data storage and volatile memory used as system memory for the processor. Memory can exchange data with the processor via a data bus. Memory can be considered a non-transitory computer-readable medium.

[0222] The memory can be configured to store information (e.g., information indicating one or more audio signals, one or more emotion metrics, one or more appearance metrics, speaker representations, emotion metric data, and / or appearance metric data) in a portion of the memory.

[0223] Furthermore, marking the first element does not imply the existence of a second element, and vice versa.

[0224] Understandable. Figures 1 to 5 This includes modules or operations shown in solid lines and modules or operations shown in dashed lines. Modules or operations included in solid lines are those included in the most widely used example embodiments. Modules or operations included in dashed lines are exemplary embodiments that may be included in or part of the modules or operations of the exemplary embodiments shown in solid lines, or may be other modules or operations that can be employed besides those of the exemplary embodiments shown in solid lines. It should be understood that these operations do not need to be performed in the order presented. Furthermore, it should be understood that not all operations need to be performed. Exemplary operations can be performed in any order and in any combination.

[0225] It should be noted that the word "including" does not necessarily exclude the presence of other elements or steps besides those listed.

[0226] It should be noted that the word "one" or "one" preceding a component does not preclude the existence of multiple such components.

[0227] It should also be noted that no reference numerals in the drawings limit the scope of the claims, exemplary embodiments may be implemented at least in part by hardware and software, and several “apparatus”, “units” or “devices” may be represented by the same hardware item.

[0228] The various exemplary methods, apparatuses, and systems described herein are described in the general context of the method steps and procedures. In one aspect, they can be implemented by a computer program product contained in a computer-readable medium, including computer-executable instructions, such as program code, that are executed by a computer in a networked environment. The computer-readable medium can include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), optical disc (CD), digital versatile disc (DVD), etc. Typically, program modules can include routines, programs, objects, components, data structures, etc., that perform specified tasks or implement specific abstract data types. The computer-executable instructions, associated data structures, and program modules represent examples of program code for performing steps of the methods disclosed herein. A particular sequence of such executable instructions or associated data structures represents examples of corresponding actions for implementing the functionality described in such steps or procedures.

[0229] Although features have been shown and described, it should be understood that they are not intended to limit the claimed invention, and it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the claimed invention. Therefore, the specification and drawings are considered illustrative rather than restrictive. The claimed invention is intended to cover all substitutions, modifications, and equivalents.

[0230] List of reference numerals

[0231] User 1A, First Speaker

[0232] 1B Speaker, Second Speaker, Caller

[0233] 2 systems

[0234] 4 User Input

[0235] 6. Training information, training representation, feedback information, training session, user output

[0236] 10 Voice guidance devices

[0237] 10A memory

[0238] 10B One or more interfaces

[0239] 10C processor

[0240] 12 transmissions

[0241] 14 obtained

[0242] 16 obtained

[0243] 18 transmissions

[0244] 20 server devices

[0245] 20A memory

[0246] 20B interface

[0247] 20C or one or more processors

[0248] 22 transmissions

[0249] 24 obtained

[0250] 30 speaker electronic devices

[0251] 32 Speaker input, audio input

[0252] 40 Network

[0253] 50 First training user interface objects

[0254] 52 Second Training User Interface Object

[0255] 54 Third Training User Interface Object

[0256] AD audio data

[0257] AD_1 First Audio Data

[0258] AD_2 Second Audio Data

[0259] AS audio signal

[0260] AS_1 First Audio Signal

[0261] AS_2 Second Audio Signal

[0262] A_ID Appearance Identifier

[0263] A_ID_1 First Appearance Identifier

[0264] A_ID_1_1 First Primary Appearance Identifier

[0265] A_ID_1_2 requires the appearance identifier for the first time.

[0266] A_ID_1_3 First and Third Appearance Identifiers

[0267] A_ID_2_1 Second Primary Appearance Identifier

[0268] A_ID_2_2 requires appearance identifier a second time.

[0269] A_ID_2_3 Second and Third Appearance Identifiers

[0270] A_ID_SET set of appearance identifiers

[0271] A_ID_SET_1 Primary appearance identifier set

[0272] A_ID_SET_2 Secondary appearance identifier set

[0273] ACS Appearance Confidence Score

[0274] ACS_1 First Appearance Confidence Score

[0275] ACS_1_1 First Primary Appearance Confidence Score

[0276] ACS_1_2 requires appearance confidence score for the first time.

[0277] ACS_2 Second Appearance Confidence Score

[0278] ACS_2_1 Second Primary Appearance Confidence Score

[0279] AL Appearance Grade

[0280] AL_1 First Appearance Level

[0281] AL_1_1 First Major Appearance Level

[0282] AL_1_2 requires appearance level for the first time.

[0283] AL_1_3 First and Third Appearance Levels

[0284] AL_2_1 Second Major Appearance Level

[0285] AL_2_2 requires appearance level for the second time.

[0286] AL_2_3 Second and Third Appearance Levels

[0287] AM Appearance Measurement

[0288] AMD Appearance Measurements

[0289] AM_1 First Appearance Measurement

[0290] AM_1_1 First Primary Appearance Measure

[0291] AM_1_2 First time requiring appearance measurement

[0292] AM_1_3 First and Third Appearance Measurements

[0293] AM_2_1 Second Primary Appearance Measure

[0294] AM_2_2 Second time, appearance measurement is required.

[0295] AM_2_3 Second and Third Appearance Measurements

[0296] DD Defect Data

[0297] DD_1 First Defect Data

[0298] DD_2 Second Defect Data

[0299] SCS confidence score

[0300] SCS_1 First Confidence Score

[0301] SCS_1_1 First Primary Confidence Score

[0302] SCS_1_2 requires confidence scores for the first time.

[0303] SCS_2_1 Second Primary Confidence Score

[0304] SL Emotional Level

[0305] SL_1_1 First Primary Affect Level

[0306] SL_1_2 requires an emotional level for the first time.

[0307] SL_2_1 Second Primary Affect Level

[0308] SM Emotional Measurement

[0309] SM_1 First Sentiment Measure

[0310] SM_1_1 First Primary Sentiment Measure

[0311] SM_1_2 requires emotional measurement for the first time

[0312] SM_2 Second Affect Measure

[0313] SM_2_1 Second Primary Affect Measure

[0314] SM_2_2 requires emotional measurement for the second time.

[0315] ST_ID_1_1 First Primary Sentiment Type Identifier

[0316] ST_ID_1_2 requires the sentiment type identifier for the first time.

[0317] ST_ID_2_1 Second Primary Sentiment Type Identifier

[0318] ST_ID_2_2 Second time needing sentiment type identifier

[0319] ST_ID_SET_1 is a set of primary sentiment type identifiers.

[0320] ST_ID_SET_2 Secondary Sentiment Type Identifier Set

[0321] SMD sentiment metrics data

[0322] SPMD Speaker Measurement Data

[0323] SPMD_1 First Speaker Measurement Data

[0324] SPMD_1_1 First Lead Speaker Measurement Data

[0325] SPMD_1_2 First Speaker Measurement Data

[0326] SPMD_1_3 First and Third Speaker Measurement Data

[0327] SPMD_2 Second Speaker Measurement Data

[0328] SPMD_2_1 Second Primary Speaker Measurement Data

[0329] ST_ID Sentiment Type Identifier

[0330] ST_ID_1 First sentiment type identifier

[0331] ST_ID_1_1 First Primary Sentiment Type Identifier

[0332] ST_ID_1_2 requires the sentiment type identifier for the first time.

[0333] ST_ID_SET set of sentiment type identifiers

[0334] ST_ID_SET_1 is a set of primary sentiment type identifiers.

[0335] ST_ID_SET_2 Secondary Sentiment Type Identifier Set

[0336] UI_1 First User Interface

[0337] VD Voice Data

[0338] VD_1 First Voice Data

[0339] VD_2 Second Voice Data

[0340] 100 Methods for Operating a Voice Guidance System

[0341] S102 obtains audio data

[0342] S104 acquires voice data

[0343] S104A determines the first speaker's measurement data.

[0344] S104B determines the first defect data.

[0345] S104C determines the second speaker measurement data.

[0346] S104D determines the second defect data.

[0347] S106 combines the first voice data and the second voice data into voice data.

[0348] S108 determines whether the speech data meets the training standards.

[0349] S110 determines the first training session.

[0350] S110A determines the first training representation

[0351] S111 determines that the user has completed a training session.

[0352] S112 Determine feedback information

[0353] S114 outputs training information

[0354] S114A displays user interface elements.

[0355] S116 outputs feedback information

[0356] Methods for operating a voice guidance system (200)

[0357] S202 outputs the first audio data of the first speech.

[0358] S202A outputs text data

[0359] S204 obtains the second audio data of the first speech.

[0360] S206 obtains second voice data based on second audio data.

[0361] S208 determines whether the second speech data meets the second training standard.

[0362] B repeats and restarts.

Claims

1. A method for operating a voice guidance system, the voice guidance system comprising a voice guidance device, the voice guidance device comprising an interface, a processor, and a memory, the method comprising: Obtain audio data representing one or more voices, the audio data including first audio data of a first voice of a first speaker, wherein the first audio data includes one or more first audio inputs from one or more dialogues performed by the first speaker, and at least one of the one or more dialogues is a dialogue monitored by the voice guidance system; First speech data is obtained based on the first audio data, wherein obtaining the first speech data includes determining first defect data indicating one or more defects in the first speech, and the first defect data includes one or more of speech tone defects, speech feature defects, and voice feature defects. Determine whether the first speech data meets a first training criterion, wherein meeting the first training criterion indicates that the first speaker will be guided or trained to address one or more defects in the first speech; The first training session is determined based on the determination that the first voice data meets the first training criteria; The first training information for the first training session is output via the interface of the voice guidance device.

2. The method according to claim 1, wherein, Obtaining first speech data based on the first audio data includes determining first speaker measurement data based on the first audio data, wherein the first speaker measurement data includes first primary speaker measurement data.

3. The method according to claim 2, wherein, The first training standard is based on first speaker measurement data.

4. The method according to any one of claims 1-3, wherein, The first training standard is based on the first standard.

5. The method according to any one of claims 1-3, wherein, Determining the first training session includes determining the first training representation.

6. The method according to claim 5, wherein, The first training session includes one or more first exercises representing the first defect data.

7. The method according to claim 6, wherein, The one or more first exercises include one or more of the following: speech intonation exercises, speech feature exercises, and voice feature exercises.

8. The method according to any one of claims 1-3, wherein, The first training information is a recommendation for performing the first training session, and wherein outputting the first training information includes displaying a training user interface object on the display of the interface.

9. The method according to any one of claims 1-3, the method comprising: Confirm that the user has completed the first training session; Obtain the second audio data of the first speech; Second voice data is obtained based on the second audio data; Based on the second voice data, a first feedback message indicating the feedback of the first training session is determined; and The first feedback information is output through the interface of the voice guidance device.

10. The method according to any one of claims 1-3, wherein the audio data includes second audio data of the first speech, the method comprising: Second voice data is obtained based on the second audio data; Determine whether the second speech data meets the second training criteria; The second training session is determined based on the determination that the second voice data meets the second training criteria; and The interface of the voice guidance device outputs second training information indicating the second training session.

11. The method according to claim 10, wherein the method comprises: The first voice data and the second voice data are aggregated into voice data; Determine whether the speech data meets the third training criterion; The third training session is determined based on the determination that the voice data meets the third training criterion; and The third training information, which instructs the third training session, is output through the interface of the voice guidance device.

12. A voice guidance system, the voice guidance system comprising a server device and a voice guidance device, wherein, The voice guidance device is configured to perform the method according to any one of claims 1-11.

13. A voice guidance system, the voice guidance system comprising a server device and a voice guidance device, the voice guidance device comprising an interface, a processor, and a memory, wherein, The voice guidance system is configured as follows: Obtain audio data representing one or more voices, the audio data including first audio data of a first voice of a first speaker, wherein the first audio data includes one or more first audio inputs from one or more dialogues performed by the first speaker, and at least one of the one or more dialogues is a dialogue monitored by the voice guidance system; First speech data is obtained based on the first audio data, wherein obtaining the first speech data includes determining first defect data indicating one or more defects in the first speech, and the first defect data includes one or more of speech tone defects, speech feature defects, and voice feature defects. Determine whether the first speech data meets a first training criterion, wherein meeting the first training criterion indicates that the first speaker will be guided or trained to address one or more defects in the first speech; The first training session is determined based on the determination that the first voice data meets the first training criteria; The first training information indicating the first training session is output through the interface of the voice guidance device.

Images