System, electronic device and related methods with speaker representation

Abstract

Systems, electronic devices, and related methods having a speaker representation are disclosed. Systems, electronic devices, and related methods are disclosed, and in particular, a method of operating a system including an electronic device is disclosed, the method including obtaining one or more audio signals including a first audio signal; determining, based on the first audio signal, one or more first emotion metrics indicative of a first speaker state, the one or more first emotion metrics including a first primary emotion metric indicative of a primary emotional state of a first speaker; determining one or more first appearance metrics indicative of an appearance of the first speaker, the one or more first appearance metrics including a first primary appearance metric indicative of a primary appearance of the first speaker; determining a first speaker representation based on the first primary emotion metric and the first primary appearance metric; and outputting, via an interface of the electronic device, the first speaker representation.

Description

Technical Field

[0001] This disclosure relates to speech processing and, more specifically, to tools and methods for analyzing, monitoring, and / or evaluating the speech of one or more speakers (e.g., callers and / or agents of calls and / or one or more participants in telephone conversations or meetings such as online conferences). Therefore, systems, electronic devices, and related methods are provided; more specifically, a method for operating a system including electronic devices is provided. Background Technology

[0002] Nowadays, interactions between people are increasingly taking place remotely and virtually, such as via telephone calls and conference calls. However, it may be difficult for people at both ends of a telephone conversation to grasp each other's emotions or feelings simply by listening to the voice.

[0003] For example, almost all support and after-sales service is conducted over the phone between call center agents who initiate and / or answer calls and the potential customers being contacted or who are contacting the call center agents about various issues. Call center agents working in call centers, support centers, or contact centers are performing a difficult, sometimes monotonous and repetitive task. This is a negative experience for the agents and can lead to a worse tone of voice, which in turn reduces customer satisfaction on the other end of the line and increases average call duration. After several hours on the line, it can be difficult to remember that someone on the other end of the line was eager to help resolve a problem. Summary of the Invention

[0004] Therefore, there is a need for systems, electronic devices, and methods for operating systems with improved speaker representation through speech processing.

[0005] A method for operating a system including an electronic device and / or a server device is disclosed, the method comprising: acquiring one or more audio signals including a first audio signal; determining one or more first emotion metrics indicating a first speaker state based on the first audio signal, the one or more first emotion metrics including a first primary emotion metric indicating a primary emotional state of the first speaker; determining one or more first appearance metrics indicating the appearance of the first speaker, the one or more first appearance metrics including a first primary appearance metric indicating a primary appearance of the first speaker; determining a first speaker representation based on the first primary emotion metric and the first primary appearance metric; and outputting the first speaker representation, for example, from the server device to the electronic device and / or via an interface of the electronic device.

[0006] In addition, an electronic device is disclosed, comprising a processor, a memory, and an interface, wherein the processor is configured to perform at least several portions of any of the methods according to the present disclosure.

[0007] An electronic device including a processor, memory, and an interface is also disclosed, wherein the processor is configured to: acquire one or more audio signals optionally including a first audio signal; optionally determine one or more first emotion metrics indicating a first speaker's state based on the first audio signal, the one or more first emotion metrics including a first primary emotion metric indicating the first speaker's primary emotional state; optionally determine one or more first appearance metrics indicating the first speaker's appearance, the one or more first appearance metrics including a first primary appearance metric indicating the first speaker's primary appearance; optionally determine a first speaker representation based on the first primary emotion metric and the first primary appearance metric; and output the first speaker representation, for example, from a server device to the electronic device and / or via the interface. Outputting the first speaker representation may include: receiving the first speaker representation from the server device; and outputting the first speaker representation.

[0008] A system comprising a server device and an electronic device is also disclosed, wherein the electronic device is an electronic device according to the present disclosure.

[0009] A server device including an interface, one or more processors, and memory is also disclosed. The one or more processors can be configured to perform at least several portions of any of the methods disclosed herein. For example, determining (several) emotion metrics and / or (several) appearance metrics can be performed at the server device. For example, determining (several) speaker representations can be performed at the server device. The speaker representations (several) can be transmitted to the electronic device for output via the electronic device, such as a display.

[0010] One advantage of this disclosure is that the speaker / user is able to monitor / evaluate their own or another speaker's (such as a client or another meeting) voice substantially in real time, thereby allowing the speaker / user to adapt their voice (such as the timbre of the voice) substantially in real time to the other party. This can improve the outcome of the speaker's statements and / or the outcome of conversations or meetings between the speaker and one or more other speakers. The speaker can further enhance their understanding of the conversation and / or another speaker, for example, by better grasping the other speaker's emotions.

[0011] Furthermore, this disclosure provides improved call feedback and monitoring by displaying one or more speaker representations at an increased level of detail. Additionally, this disclosure reduces the need for obtaining appearance data from another speaker, for example, by not actively obtaining information from them, thereby enabling more efficient calls / conversations and enhancing the user experience for the other speaker (such as a customer or caller).

[0012] Furthermore, this disclosure provides more expressive and improved incarnation support by significantly increasing the granularity and variety of available speaker representations, thereby allowing for improved representations and more specific feedback and / or personalization of speaker representations. By providing improved speaker representations, speakers / users can increase their engagement in conversations or meetings, for example, by increasing their engagement in their work and / or considering another speaker. Attached Figure Description

[0013] Those skilled in the art will more readily understand the above and other features and advantages of the invention through the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings, in which:

[0014] Figure 1 An exemplary system including a server device and electronic devices according to this disclosure is illustrated schematically.

[0015] Figures 2A to 2B This is a flowchart of an exemplary method according to this disclosure.

[0016] Figure 3 An exemplary electronic device according to this disclosure is illustrated schematically.

[0017] Figure 4 The user interface of the electronic device according to this disclosure is illustrated schematically.

[0018] Figure 5 An exemplary system including a server device and an electronic device according to this disclosure is illustrated schematically.

[0019] Figure 6 An exemplary data structure according to this disclosure is illustrated schematically. Detailed Implementation

[0020] Various exemplary embodiments and details are described below with reference to the associated figures. It should be noted that these figures may or may not be drawn to scale, and elements having similar structures or functions are represented by similar element symbols in these figures. It should also be noted that these figures are intended only to facilitate the description of embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. Furthermore, the illustrated embodiments are not required to show all aspects or advantages. Aspects or advantages described in connection with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment, even if not so illustrated or explicitly described.

[0021] A method for operating a system including electronic devices is disclosed. The system may optionally include a server device comprising an interface, one or more processors, and memory. The server device may be configured on a cloud, such as a cloud network. The server device may be remote from the rest of the system, such as remote from the electronic devices. The system may be a system for monitoring, processing, and / or analyzing one or more audio signals, such as audio signals of one or more speakers speaking (e.g., as a monologue or conversation, such as a conference conversation, video / telephone discussion) or calls (e.g., telephone or VoIP calls). The system may be a system for monitoring, processing, and / or analyzing one or more audio signals (such as conversations between two or more individuals / speakers, such as conversations in a telephone call or conference). The system may, for example, include or act as a call center system for monitoring, processing, and / or analyzing one or more audio signals (such as conversations between two or more individuals, such as telephone calls between an agent and a customer or caller in the call center system). The system may be configured to use edge processing between one or more electronic devices. Different operations and / or steps of the method and / or system can be performed at different devices (such as at the electronic device and / or at the server device).

[0022] The electronic device includes an interface, a processor, and memory. The electronic device may be, for example, a mobile phone, such as a smartphone, a computer (such as a laptop computer or PC), or a tablet computer. The electronic device may be, for example, a user device configured to perform a call between a user and one or more persons, such as a mobile phone or a computer. The electronic device may be configured to receive a first audio input, such as first audio input from the call between the user and another person. For example, the electronic device may act as: a call agent device, where the user can be an agent, such as an agent of a call center (such as a support call center, after-sales call center, marketing call center, receiving call center, sales call center); or a company where an important part of an employee's job is to talk to customers. The electronic device may be, for example, a user device configured to record first audio input from a first speaker (such as recording the first speaker speaking or talking), such as a mobile phone or a computer. The electronic device may be configured to receive one or more audio signals, such as generating one or more audio signals including the first audio signal. The first audio signal may be based on the first audio input.

[0023] The method includes obtaining one or more audio signals, also labeled AS_i (i = 1, 2, ..., N, where N is the number of speakers / audio signals), the one or more audio signals including a first audio signal also labeled AS_1. The first audio signal may represent a first voice / voice of a first speaker. The first speaker may be a caller, agent, or a first participant in a telephone conversation or conference. Optionally, the one or more audio signals may include a second audio signal also labeled AS_2. The second audio signal may represent a second voice / voice of a second speaker. The second speaker may be a caller, agent, or a second participant in a telephone conversation or conference.

[0024] Acquiring one or more audio signals may include acquiring one or more audio inputs, such as a first audio input. In other words, the first audio signal may be based on the first audio input. The first audio signal may be acquired from a call or conversation between a user and another person (e.g., a first call or first conversation). The first speaker may be the person who speaks / talks the most during the first call and / or the first conversation. The first speaker may be the first person to speak during the first call and / or the first conversation. The first speaker may be someone of interest that the user may want to express, such as the first speaker indicating. The first audio signal may be acquired from an electronic device that records the first audio input from the first speaker (such as recording the first speaker speaking or talking). Acquiring one or more audio signals may include generating one or more audio signals that include the first audio signal. The first audio signal may be based on the first audio input. The first audio signal may include the first speaker's audio signal. The first audio signal may be a proxy audio signal or a caller's audio signal, and the second audio signal is another signal.

[0025] The method includes identifying one or more first emotion measures, also labeled SM_1_i (i = 1, 2, ..., M, where M is the number of first emotion measures).

[0026] Emotion metrics indicate the speaker's emotional state. Emotion metrics may include one or more of the following: emotion type identifier, emotion level, and confidence score.

[0027] One or more first emotion metrics, such as SM_1_1 and / or SM_1_2, indicate the state of the first speaker, i.e., one or more first emotion states of the first speaker, and may be based on a first audio signal and / or a second audio signal. One or more first emotion metrics SM_1_i may include one or more parameters indicating the state of the first speaker.

[0028] One or more first emotion measures SM_1_i include a first primary emotion measure also labeled SM_1_1 and indicating the primary emotion state of the first speaker. In other words, SM_1_1 can indicate the primary emotion of the first speaker. SM_1_1 can indicate the dominant emotion state and / or primary emotion state of the first speaker over time. The primary emotion state can be one of the following: disappointment, boredom, fear, sadness, negativity, very negative, frustration, annoyance, fear, panic, hesitation, dissatisfaction, anger, surprise, worry, doubt, incomprehension, thinking, neutrality, positivity, very positive, happiness, friendliness, confidence, excitement, smugness, satisfaction, exhilaration, contentment, relaxation, energy, enthusiasm, service-oriented, helpfulness, interest, and joy. In one or more exemplary methods / systems, the primary emotion state of the first speaker can be selected from a set of primary emotion states.

[0029] The first emotion measure SM_1_i may include a first emotion type identifier also labeled ST_ID_1_i (where i is an index, and i = 1, 2, ..., H, where H is the number of first emotion type identifiers). In other words, determining one or more first emotion measures SM_1_i may include determining the first emotion type identifier ST_ID_1_i, for example, the first primary emotion type identifier ST_ID_1_1 of the first primary emotion measure SM_1_1. The emotion type identifier indicates the speaker's emotional state.

[0030] Emotion type identifiers can be, for example, labels, numbers, or values ​​(such as integers) corresponding to a specific emotion (state), emotion type, and / or emotion category. For example, ST_ID_1_i can be selected from the same or different sets of emotion type identifiers. For example, ST_ID_1_1 can be selected from a primary set of emotion type identifiers and / or ST_ID_1_2 can be selected from a secondary set of emotion type identifiers. The primary set of emotion type identifiers can be different from or the same as the secondary set of emotion type identifiers. The primary and secondary sets of emotion type identifiers can share one or more (such as multiple) emotion type identifiers.

[0031] In one or more exemplary methods, the first primary emotion measure SM_1_1 includes a first primary emotion type identifier ST_ID_1_1 selected from the primary emotion type identifier set ST_ID_SET_1, wherein ST_ID_SET_1 includes multiple emotion type identifiers, for example, including at least three, four, five or more emotion type identifiers.

[0032] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0033] ST_ID_SET_1 = {1,2,3,4,5}

[0034] "1" indicates a mood such as "very negative", "2" indicates a mood such as "negative", "3" indicates a mood such as "neutral", "4" is optional and indicates a mood such as "positive", and "5" is optional and indicates a mood such as "very positive".

[0035] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0036] ST_ID_SET_1 = {1,2,3,4,5}

[0037] The "1" indicates an emotion such as "anger", the "2" indicates an emotion such as "listlessness", the "3" indicates an emotion such as "good spirits", the optional "4" indicates an emotion such as "friendly, engaged, energetic", and the optional "5" indicates an emotion such as "highly engaged".

[0038] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0039] ST_ID_SET_1 = {"Very negative", "Negative", "Neutral", "Positive", "Very positive"}

[0040] For example, "very negative" and / or "very positive" are optional.

[0041] The set of emotion type identifiers, such as the set of primary emotion type identifiers and / or the set of secondary emotion type identifiers, may include at least three or at least four different emotion type identifiers, such as five, six, seven, eight, nine, or more emotion type identifiers. In other words, each emotion or emotion type may have a corresponding ST_ID_1_i. For example, the first primary emotion measure SM_1_1 may include a first primary emotion type identifier ST_ID_1_1 indicating or corresponding to the primary emotion state or the first speaker's state as "positive".

[0042] The first emotion measure SM_1_i may include an emotion level also labeled SL_1_i (i = 1, 2, ..., 0, where 0 is the number of emotion levels). In other words, determining SM_1_i may include determining SL_1_i; for example, determining SM_1_i may include determining a first primary emotion level SL_1_1. The emotion level SL_1_i may indicate the level of the i'th emotion type. In other words, SL_1_i may indicate the degree of the i'th emotion type. For example, when ST_ID_1_1 corresponds to the first speaker state "positive," the first primary emotion level SL_1_1 may indicate or correspond to the degree of the emotion "positive," for example, within a numerical range of, for example, from 0 to 1 or from 1 to 10, or selected from "medium," "low," and "high." In other words, the emotion level of the emotion measure may be within a numerical range of, for example, from 0 to 1 or from 1 to 10.

[0043] The first sentiment metric SM_1_i may include confidence scores labeled SCS_1_i (i = 1, 2, ..., P, where P is the number of confidence scores). In other words, determining SM_1_i may include determining the first confidence score SCS_1_i; for example, determining the first primary sentiment metric SM_1_1 may include determining the first primary confidence score SCS_1_1. The confidence score of the sentiment metric may indicate, for example, the score or probability that the determined sentiment metric, such as the sentiment type identifier and / or sentiment level, is correct (e.g., the sentiment state or sentiment type (as identified by the sentiment type identifier of the sentiment metric) is correct). For example, SCS_1_1 = 0.88 may indicate that there is an 88% probability that the determined ST_ID_1_1 is "positive".

[0044] Determining one or more first emotion measures indicative of a first speaker's state may include extracting one or more speaker features from a first audio signal, for example, where the one or more first emotion measures are based on one or more speaker features. The one or more speaker features may include paralinguistic features. The one or more speaker features may, for example, include speaker timbre features, speaker intonation features, speaker strength or volume features, speaker pitch features, speaker timbre features, speaker speech rate features, speech features, acoustic features, and / or speaker spectral energy features. Spectral energy features may include individual bins of a spectrogram indicating the signal energy level at a given frequency.

[0045] Linguistic features may include specific emotion-related words, such as positive and / or negative words. Linguistic features can be determined based on a text transcript of the audio signal. The text transcript can be obtained through a human annotator or using an automatic speech recognition (speech-to-text) algorithm or service. Linguistic features may include embedded features from deep neural networks (e.g., BERT transformer networks or other sequence-to-sequence autoencoders).

[0046] In one or more exemplary methods, one or more first emotion metrics can be determined based on a machine learning (ML) model, such as the output of an ML model. The input to the ML model can be speaker features or the audio signal itself. ML models can include linear regression models, support vector machines, decision tree classifiers (e.g., random forests, XGBoost), Gaussian mixture models, hidden Markov models, and / or neural networks. Neural networks can, for example, include one or more of linear feedforward layers, convolutional layers, recursive layers, and attentional layers. ML models can include weights of one or more speaker features. For example, an ML model can map, for example, speaker tone and / or vocal quality, to emotion metrics / types, emotion levels, and / or emotion confidence scores. ML models can include parameters ranging from 100,000 to 1,000,000, for example, from 500,000 to 1,000,000. ML models can include layers ranging from 5 to 20, for example, from 10 to 15 layers.

[0047] Instances of known ML models can be found at: "www.researchgate.net / publication / 222431291_Emotional_speech_recognition_Resources_features_and_methods", "https: / / mediatum.ub.tum.de / doc / 1523509 / 1523509.pdf" and "https: / / www.researchgate.net / publication / 319565810_End-to-end_learning_for_dimensional_emotion_recognition_from_physiological_signals".

[0048] For example, an emotion metric can be derived from a speaker speech regulation, also labeled S, which can be the sum of normalized variances of the fundamental frequency F0 / pitch (e.g., a range adjusted from 0 to 1 by multiplying by a factor determined with respect to the audio input, such as the training audio input, as the inverse of the range of F0 variance of the training data for the ML model). For example, an emotion metric can be derived from a speaker speech regulation S, which can be the normalized (normalized to the range 0 to 1) variance of the signal strength. To determine the emotion of “excitement” and “unexcitement,” a threshold of, for example, 1.0 can be applied to S, where excitement is detected when S is greater than or equal to 1.0 and unexcitement is detected when S is less than 1.0. Furthermore, an emotion level in the range 0 to 1 can be determined or obtained for the emotion of “excitement” or “unexcitement,” where S can be divided by two (e.g., the number of speaker features that are part of the sum). Furthermore, the sentiment confidence score can be determined or obtained based on the absolute value of the numerical difference between the normalized F0 variance and / or the normalized signal strength variance.

[0049] ML models can be trained based on, for example, call logs, where verifiers or supervisors, such as psychologists and / or human supervisors, have assigned emotion identifiers / labels to emotion metrics (e.g., based on their own subjective effort judgment) and / or assigned speaker feature labels to speaker features. Speaker features can be algorithmically determined via signal processing algorithms and / or identified as the output of another ML model. One or more first emotion metrics can be inferred using ML models. The inputs to an ML model can include one or more acoustic features such as loudness and / or pitch features. Timbre features can be determined using an ML model and can be, for example, negative or positive timbre. Furthermore, the inputs to an ML model can include spectrograms, latent (hidden) activation representations of (deep) neural networks. The inputs to an ML model can include static feature vectors ("fingerprints") such as mean, variance, slope, peak distance, and modulation spectrum. The input to the ML model can include frame-by-frame (low-level) acoustic features, such as the pitch, energy level, spectral parameters (Mel frequency cepstral, MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectrum envelope characteristics (e.g., formants, harmonics, harmonic-to-formant ratio), and / or sound quality measurements, such as harmonic noise ratio, HNR, jitter, and / or perturbation.

[0050] The method includes determining one or more first appearance metrics, also labeled AM_1_i (i = 1, 2, Q, where Q is the number of first appearance metrics of the first speaker or associated with the first speaker).

[0051] Appearance metrics indicate a speaker's appearance. Appearance metrics may include one or more of appearance identifiers, appearance levels, and confidence scores, and may indicate the appearance of a first speaker, for example, based on a first audio signal and / or a second audio signal. Determining one or more first appearance metrics may include, for example, retrieving one or more first appearance metrics from a database based on information about the first speaker stored in the database. One or more first appearance metrics AM_1_i may include one or more parameters indicating the appearance of the first speaker. One or more first appearance metrics AM_1_i include a first primary appearance metric also labeled AM_1_1 and indicating the primary appearance of the first speaker, and / or a first secondary appearance metric also labeled AM_1_2 and indicating the secondary appearance of the first speaker.

[0052] In other words, AM_1_1 can indicate the primary physical appearance of the first speaker. AM_1_1 can be selected from gender metrics (e.g., woman / female, man / male, or genderless), weight metrics, height metrics, age metrics, language metrics, language ability metrics, hearing ability metrics, dialect metrics, health metrics (e.g., breathing condition, speech defects, and / or speech disorders), personality metrics (e.g., extroverted or introverted), and comprehension metrics (e.g., based on age, health, and / or gender metrics). Comprehension metrics may be relevant when an older person has difficulty hearing conversations or when a foreigner is not accustomed to spoken language. Comprehension metrics can provide the user with instructions, for example, that he / she should speak more slowly and clearly.

[0053] The first appearance metric AM_1_i may include a first appearance identifier also labeled A_ID_1_i (where i is the index of the i'-th first appearance metric, i = 1, 2, ..., I, where I is the number of first appearance identifiers). The first appearance identifier may indicate one of the following: gender metric, weight metric, height metric, age metric, language metric, language ability metric, hearing metric, and comprehension metric. In other words, determining AM_1_i may include determining A_ID_1_i, including the first primary appearance identifier A_ID_1_1 of the first primary appearance metric AM_1_1 and / or the first secondary appearance identifier A_ID_1_2 of the first secondary appearance metric AM_1_2.

[0054] Appearance identifiers can be, for example, labels, numbers, or values ​​(such as integers) corresponding to a specific appearance metric, appearance type, and / or appearance category. For example, A_ID_1_i can be selected from a set of appearance types, including, for example, one or more of gender, height, weight, age, language, language ability, hearing ability, and comprehension ability. Appearance metric identifiers can be labels or numbers mapped to and / or indicating appearance metric types.

[0055] In one or more exemplary methods, the first primary appearance metric AM_1_1 includes a first primary appearance identifier A_ID_1_1 optionally selected from the primary appearance identifier set A_ID_SET_1, wherein A_ID_SET_1 includes multiple appearance identifiers, for example, including at least three, four, five or more appearance type identifiers. In one or more exemplary methods, the first primary appearance metric AM_1_1 is a gender metric, that is, the first primary metric identifier A_ID_1_1 indicates gender, for example, it can be mapped to gender via a table A_ID_1_1 = "gender" or A_ID_1_1 = 1.

[0056] In one or more exemplary methods, the first-level appearance metric AM_1_2 includes a first-level appearance identifier A_ID_1_2 optionally selected from the secondary appearance identifier set A_ID_SET_2, wherein A_ID_SET_2 includes multiple appearance identifiers, for example, including at least three, four, five or more appearance identifiers. In one or more exemplary methods, the first-level appearance metric AM_1_2 is an age metric, that is, the first-level metric identifier A_ID_1_2 indicates age, for example, it can be mapped to age via a table A_ID_1_2 = "age" or A_ID_1_2 = 2.

[0057] The set of appearance identifiers may include two or at least three or at least four different appearance identifiers, such as five, six, seven, eight, nine or more appearance identifiers. For example, a first primary appearance measure AM_1_1 may include a first primary appearance identifier A_ID_1_1 that indicates or corresponds to the primary appearance of the first speaker (e.g., one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability). For example, a first secondary appearance identifier A_ID_1_2 may indicate or correspond to the first secondary appearance of the first speaker, such as one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability. The first secondary appearance identifier may optionally be different from the first primary appearance identifier.

[0058] The first appearance metric AM_1_i may include appearance levels also labeled AL_1_i (i = 1, 2, ..., R, where R is the number of appearance levels). In other words, determining AM_1_i may include determining AL_1_i; for example, determining AM_1_1 may include determining a first primary appearance level AM_1_1. The first appearance level AL_1_i may indicate the level, value, range, or label of the appearance metric AM_1_i, as indicated by the appearance identifier A_ID_1_i. In other words, the first appearance level AL_1_i may indicate the level, value, range, or label of the first appearance metric AM_1_i. For example, when A_ID_1_1 corresponds to the first primary appearance of the first speaker as "gender," the first primary appearance level AL_1_1 may indicate or correspond to "male," "female," or optionally "genderless." For example, when the primary appearance identifier A_ID_1_2 corresponds to the primary appearance measure of the first speaker as "height", the primary appearance level AL_1_2 can indicate or correspond to "short", "medium", or "tall". For example, when the primary appearance identifier A_ID_1_2 corresponds to the primary appearance measure of the first speaker as "height", the primary appearance level AL_1_2 can indicate or correspond to "shorter than 160cm", "between 160cm and 185cm", or "taller than 185cm".

[0059] For example, when a first appearance identifier such as a first or third-level appearance identifier A_ID_1_3 corresponds to a first appearance measure such as a first or third-level appearance measure AM_1_3 of the first speaker being “age”, the first or third-level appearance level AL_1_3 may indicate or correspond to an age range such as “under 20 years old”, “20 to 40 years old”, “40 to 60 years old” or “over 60 years old” or an age label such as “young”, “middle-aged” or “old”.

[0060] The first appearance measure AM_1_i may include a confidence score also labeled ACS_1_i (i = 1, 2, ..., S, where S is the number of confidence scores). In other words, determining the first appearance measure AM_1_i may include determining the first appearance confidence score ACS_1_i. For example, determining the first primary appearance measure AM_1_1 may include determining the first primary appearance confidence score ACS_1_1. The first appearance confidence score ACS_1_i of the appearance measure AM_1_i may indicate the score or probability that the determined first appearance measure AM_1_i, such as the first appearance level AL_1_i, is correct (e.g., the appearance measure or appearance level is correct). For example, ACS_1_1 = 0.95 may indicate that there is a 95% probability that the determined AL_1_1 is "male".

[0061] Determining one or more first appearance metrics that indicate a first speaker may include extracting one or more speaker appearance features from a first audio signal. One or more speaker appearance features may include, for example, speaker timbre features, speaker intonation features, speaker strength features, speaker pitch features, speaker quality features, speaker speech rate features, speech features, acoustic features, and / or speaker spectral energy features.

[0062] Spectral energy characteristics can include individual grid cells of a spectrogram indicating the signal energy level at a given frequency.

[0063] Linguistic features may include specific appearance-related words, such as positive and / or negative words. Linguistic features can be determined based on text transcripts of audio signals. Text transcripts can be obtained through human annotators or using automatic speech recognition (speech-to-text) algorithms or services. Linguistic features may include embedded features from deep neural networks (e.g., BERT transformer networks or other sequence-to-sequence autoencoders).

[0064] In one or more exemplary methods, one or more first appearance metrics can be determined based on a machine learning (ML) model (such as the output of an ML model). One or more first appearance metrics can be inferred using an ML model. ML models can include linear regression models, support vector machines, decision tree classifiers (e.g., random forests, XGBoost), Gaussian mixture models, hidden Markov models, and / or neural networks. Neural networks can, for example, include one or more of linear feedforward layers, convolutional layers, recursive layers, and attention layers. ML models can include weights for one or more speaker features. For example, an ML model can map, for example, speaker tone and / or vocal quality, to emotion metrics / types, emotion levels, and / or emotion confidence scores. ML models can include parameters ranging from 100,000 to 1,000,000, for example, from 500,000 to 1,000,000 parameters. ML models can include layers ranging from 5 to 20, for example, from 10 to 15 layers.

[0065] ML models can be trained based on, for example, call recordings, where a verifier or supervisor, such as a human supervisor, has assigned emotion identifiers / labels to emotion metrics and / or speaker feature labels to speaker features. Speaker features can be algorithmically determined via signal processing algorithms. One or more first appearance metrics can be inferred using the ML model. The input to the ML model can include audio data, such as audio data stored in a database of known audio data matching one or more appearance metrics, such as appearance labels. Appearance labels can include labels assigned by people and / or basic facts, such as age or height from a passport or social registry. For example, audio data input can include recordings of phone calls, television programs, and / or film actors, etc.

[0066] The input to an ML model can include one or more acoustic features such as timbre. Timbre features can be, for example, negative or active timbre. Additionally, the input to an ML model can include spectrograms, latent (hidden layer activation) representations of (deep) neural networks. The input to an ML model can include static feature vectors (“fingerprints”) such as mean, variance, slope, peak distance, and modulation spectrum. The input to an ML model can include frame-by-frame (low-level) acoustic features such as speech pitch, energy levels, spectral parameters (Mel frequency cepstral, MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectrum envelope characteristics (e.g., formants, harmonics, harmonic-to-formant ratio), and / or phonological measurements such as harmonic-to-noise ratio, HNR, jitter, and / or perturbations. For example, acoustic features related to one or more appearance metrics such as body shape can include the ratio of vowel formants related to vocal tract length. For example, acoustic features may be related to one or more appearance metrics such as body size, timbre characteristics (e.g., age-related HNR, jitter, and / or perturbations) (e.g., more breathing, older age, more jitter), and pitch may be gender-related (e.g., males may have pitches below 150 Hz and females may have pitches above 150 Hz). Furthermore, acoustic features may include, for example, linguistic and dialectal features and / or mean spectral envelope features such as phoneme stocks / histograms for age and / or gender.

[0067] One or more first emotion metrics and one or more first appearance metrics may be part of the first speaker metrics data. The first speaker metrics data may also be represented as agent metrics data and / or caller metrics data.

[0068] The method includes determining a first speaker representation, also labeled SR_1, based on a first primary emotion measure SM_1_1 and / or a first primary appearance measure AM_1_1. The first speaker representation SR_1 may include the first primary speaker representation also labeled SR_1_1. Determining SR_1_1 may include generating the first primary speaker representation SR_1_1 based on SM_1_1 and AM_1_1. The first speaker representation may be determined based on public and / or customer registration. For example, for frequent callers / customers, the first primary emotion measure SM_1_1 and / or the first primary appearance measure AM_1_1 may be refined over multiple calls / conversations; for example, the more voices heard and the more audio data obtained, the more confidently one can determine that the speaker is, for example, male. One or more emotion and / or appearance measures may be known, for example, age from social registration and / or emotional state from previous conversations. One or more known emotion and / or appearance measures may be used to improve the accuracy of determining the speaker representation and / or to determine the speaker representation.

[0069] The first speaker representation SR_1 may include a first-level speaker representation also labeled SR_1_2. The first speaker representation SR_1 may include a first-third-level speaker representation also labeled SR_1_3. The first speaker representation SR_1 may include a first-fourth-level speaker representation also labeled SR_1_4. The first speaker representation SR_1 may include a first-fifth-level speaker representation also labeled SR_1_5. Therefore, determining the first speaker representation may include determining one or more of SR_1_2, SR_1_3, SR_1_4, and SR_1_5 based on a first audio signal (such as based on a first primary emotion measure SM_1_1 and / or a first primary appearance measure AM_1_1). Determining the first speaker representation may also include determining one or more of SR_1_2, SR_1_3, SR_1_4, and SR_1_5 based on a second audio signal (such as based on a second primary emotion measure SM_2_1 and / or a second primary appearance measure AM_2_1).

[0070] The first speaker representation can also be expressed as the first person representation.

[0071] The first speaker representation can indicate the first speaker's state and / or appearance substantially in real time (e.g., with a delay of less than 5 or 10 seconds). The first speaker representation can indicate segments being analyzed, such as speech segments. For example, a voice activity detection module can identify one or more speech / voice segments and discard noise. Segments can be, for example, speech segments of at least 5 or 10 seconds. The voice activity detection module can detect pauses longer than, for example, 400 ms, 500 ms, or 1 second. Speech segments can be detected at the following times: when a pause occurs, when another speaker begins to speak, or when the segment reaches a defined maximum length (e.g., up to 8 seconds) indicating the end of the speech segment. For each speech segment, one or more emotion metrics and / or one or more appearance metrics can be determined. In one or more exemplary methods / systems, the first speaker representation, such as the first primary speaker representation, is updated at an update frequency of at least 0.2 Hz (e.g., every second, every 5 seconds, every 7 seconds, or every 10 seconds). The update frequency of the first primary speaker representation can vary over time and can also depend on the conversation, such as the pace of conversation.

[0072] In other words, the first speaker representation, such as the first primary speaker representation, can be updated during the first speaker's speech and / or during a conversation between the first speaker and a second speaker (such as a telephone call). The first speaker representation can indicate or reflect the first speaker's present, real-time, immediate, and / or current emotions and / or appearance. For example, the first speaker representation, such as the first primary speaker representation, can be the first speaker's real-time physical and emotional expressions.

[0073] The advantages of having a real-time or near-real-time first speaker representation include: users of the electronic device can see or understand changes in emotion and / or the first speaker's appearance in real time. Furthermore, users of the electronic device can better imagine or envision the first speaker's emotions and / or appearance by seeing the first speaker representation, including the first primary speaker representation. In other words, users can have the experience of video speaking or video calling without receiving a video signal.

[0074] First-speaker representation can provide users with real-time feedback on the first speaker's speech, such as feedback on the first speaker's characteristics, like the first speaker's state and / or appearance. First-speaker representation can provide a realistic representation of the first speaker. It can provide anthropomorphism, portraits, appearances, sketches, and / or gamification of the first speaker.

[0075] First speaker representation can include sound representations, such as auditory feedback and / or audio icons.

[0076] The method includes outputting a first speaker representation SR_1 via an interface of an electronic device. Outputting the first speaker representation SR_1 may include displaying a first user interface indicating the first speaker representation.

[0077] A user interface may include one or more (such as a plurality of) user interface objects. For example, a first user interface may include one or more first user interface objects, such as a first primary user interface object and / or a first-level user interface object. A user interface object may refer herein to a graphical representation of an object displayed on an interface of an electronic device, such as a display. User interface objects may be interactive with the user or selectable through user input. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may each optionally constitute a user interface object. User interface objects may form part of a widget. A widget can be considered a mini-application that a user can use.

[0078] In one or more exemplary methods, one or more first emotion measures SM_1_i include a first secondary emotion measure also labeled SM_1_2 and indicating the secondary emotional state of the first speaker.

[0079] Secondary emotional states can differ from primary emotional states. In other words, SM_1_2 can indicate the primary speaker's secondary feelings or emotions. SM_1_2 can be a less dominant emotional state and / or a second emotional state in time than the primary emotional state. Secondary emotional states can be one of the following: disappointment, boredom, fear, sadness, negativity, dread, panic, dissatisfaction, anger, surprise, worry, doubt, incomprehension, contemplation, neutrality, positivity, very positivity, happiness, friendliness, confidence, excitement, smugness, satisfaction, exhilaration, contentment, relaxation, energy, enthusiasm, service-oriented, helpfulness, interest, and joy.

[0080] In one or more exemplary methods / systems, the secondary emotional state of the first speaker can be selected from a set of secondary emotional states.

[0081] The first emotion measure SM_1_i may include a first emotion type identifier also labeled ST_ID_1_i (where i is an index). In other words, determining one or more first emotion measures SM_1_i may include determining the first emotion type identifier ST_ID_1_i, for example, the first-level emotion type identifier ST_ID_1_2 of the first-level emotion measure SM_1_2. The emotion type identifier indicates the speaker's emotional state.

[0082] Emotion type identifiers can be, for example, labels, numbers, or values ​​(such as integers) corresponding to a specific emotion (state), emotion type, and / or emotion category. For example, ST_ID_1_i can be selected from the same or different sets of emotion type identifiers. For example, ST_ID_1_2 can be selected from a set of secondary emotion type identifiers.

[0083] In one or more exemplary methods, the primary emotion measure SM_1_2 includes a primary emotion type identifier ST_ID_1_2 selected from the secondary emotion type identifier set ST_ID_SET_2, wherein ST_ID_SET_2 includes multiple emotion type identifiers, for example, including at least three, four, five or more emotion type identifiers.

[0084] In one or more exemplary methods, the set of secondary emotion type identifiers ST_ID_SET_2 is given in the following manner:

[0085] ST_ID_SET_2 = {1,2,3,4,5}

[0086] "1" indicates a mood such as "very negative", "2" indicates a mood such as "negative", "3" indicates a mood such as "neutral", "4" is optional and indicates a mood such as "positive", and "5" is optional and indicates a mood such as "very positive".

[0087] In one or more exemplary methods, the set of secondary emotion type identifiers ST_ID_SET_2 is given in the following manner:

[0088] ST_ID_SET_2 = {"Very negative", "Negative", "Neutral", "Positive", "Very positive"}

[0089] For example, "very negative" and / or "very positive" are optional.

[0090] In one or more exemplary methods, the set of secondary emotion type identifiers ST_ID_SET_2 is given in the following manner:

[0091] ST_ID_SET_2 = {1,2,3,4,5}

[0092] The "1" indicates an emotion such as "anger", the "2" indicates an emotion such as "listlessness", the "3" indicates an emotion such as "good spirits", the optional "4" indicates an emotion such as "friendly, engaged, energetic", and the optional "5" indicates an emotion such as "highly engaged".

[0093] The set of emotion type identifiers, such as the set of secondary emotion type identifiers and / or the set of secondary emotion type identifiers, may include at least three or at least four different emotion type identifiers, such as five, six, seven, eight, nine, or more emotion type identifiers. In other words, each emotion or emotion type may have a corresponding ST_ID_1_i. For example, the primary emotion measure SM_1_2 may include a primary emotion type identifier ST_ID_1_2 indicating or corresponding to a secondary emotion state or the first speaker's state as "positive".

[0094] The first emotion measure SM_1_i may include an emotion level also labeled SL_1_i (i = 1, 2, ..., 0, where 0 is the number of emotion levels). In other words, determining SM_1_i may include determining SL_1_i; for example, determining SM_1_2 may include determining the first-level emotion level SL_1_2. The emotion level SL_1_i may indicate the level of the i'th emotion type. In other words, SL_1_i may indicate the degree of the i'th emotion type. For example, when ST_ID_1_2 corresponds to the first speaker state "positive", the first-level emotion level SL_1_2 may indicate or correspond to the degree of the emotion "positive", for example, within a numerical range of, for example, from 0 to 1 or from 1 to 10, or selected from "low", "medium", and "high". In other words, the emotion level of the emotion measure may be within a numerical range of, for example, from 0 to 1 or from 1 to 10.

[0095] The first sentiment measure SM_1_i may include confidence scores labeled SCS_1_i (i = 1, 2, ..., P, where P is the number of confidence scores). In other words, determining SM_1_i may include determining the first confidence score SCS_1_i; for example, determining the first-level sentiment measure SM_1_2 may include determining the first-level confidence score SCS_1_2. The confidence score of the sentiment measure may indicate the score or probability that the determined sentiment measure, such as the sentiment type identifier and / or sentiment level, is correct (e.g., the sentiment state or sentiment type (as identified by the sentiment type identifier of the sentiment measure) is correct). For example, SCS_1_2 = 0.88 may indicate that the determined ST_ID_1_2, for example, "positive," has an 88% probability of being correct.

[0096] In one or more exemplary methods, one or more first appearance metrics AM_1_i include a first secondary appearance metric also labeled AM_1_2 and indicating the secondary appearance of the first speaker.

[0097] The first appearance measure AM_1_i may include a first appearance identifier also labeled A_ID_1_i (where i is the index of the i'-th first appearance measure). The first appearance identifier may indicate one of the following: gender measure, weight measure, height measure, age measure, language measure, language ability measure, hearing ability measure, and comprehension ability measure. In other words, determining AM_1_i may include determining A_ID_1_i, including the first-level appearance identifier A_ID_1_2 of the first-level appearance measure AM_1_2.

[0098] In one or more exemplary methods, the primary appearance metric AM_1_2 includes a primary appearance identifier A_ID_1_2 optionally selected from the secondary appearance identifier set A_ID_SET_1, wherein A_ID_SET_1 includes, for example, a plurality of appearance identifiers including at least three, four, five or more appearance type identifiers. In one or more exemplary methods, the primary appearance metric AM_1_2 is a gender metric, that is, the primary metric identifier A_ID_1_2 indicates gender, for example, it can be mapped to gender via a table A_ID_1_2 = "gender" or A_ID_1_2 = 1.

[0099] In one or more exemplary methods, the first-level appearance metric AM_1_2 includes a first-level appearance identifier A_ID_1_2 optionally selected from the secondary appearance identifier set A_ID_SET_2, wherein A_ID_SET_2 includes multiple appearance identifiers, for example, including at least three, four, five or more appearance identifiers. In one or more exemplary methods, the first-level appearance metric AM_1_2 is an age metric, that is, the first-level metric identifier A_ID_1_2 indicates age, for example, it can be mapped to age via a table A_ID_1_2 = "age" or A_ID_1_2 = 2.

[0100] The set of appearance identifiers may include two or at least three or at least four different appearance identifiers, such as five, six, seven, eight, nine or more appearance identifiers. For example, a primary appearance measure AM_1_2 may include a primary appearance identifier A_ID_1_2 that indicates or corresponds to a secondary appearance of the first speaker (e.g., one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability). For example, a primary appearance identifier A_ID_1_2 may indicate or correspond to a primary appearance of the first speaker, such as one of gender, weight, height, age, language, language ability, hearing ability, and comprehension ability. Primary appearance identifiers may optionally differ from primary appearance identifiers.

[0101] The first appearance metric AM_1_i may include appearance levels also labeled AL_1_i (i = 1, 2, ..., R, where R is the number of appearance levels). In other words, determining AM_1_i may include determining AL_1_i; for example, determining AM_1_2 may include determining the first-level appearance level AM_1_2. The first appearance level AL_1_i may indicate the level, value, range, or label of the appearance metric AM_1_i, as indicated by the appearance identifier A_ID_1_i. In other words, the first appearance level AL_1_i may indicate the level, value, range, or label of the first appearance metric AM_1_i. For example, when A_ID_1_2 corresponds to the first-level appearance of the first speaker as "gender," the first-level appearance level AL_1_2 may indicate or correspond to "male," "female," or optionally "genderless." For example, when the primary appearance identifier A_ID_1_2 corresponds to the primary appearance measure of the first speaker as "height", the primary appearance level AL_1_2 can indicate or correspond to "short", "medium", or "tall". For example, when the primary appearance identifier A_ID_1_2 corresponds to the primary appearance measure of the first speaker as "height", the primary appearance level AL_1_2 can indicate or correspond to "shorter than 160cm", "between 160cm and 185cm", or "taller than 185cm".

[0102] For example, when a first appearance identifier such as a first or third-level appearance identifier A_ID_1_3 corresponds to a first appearance measure such as a first or third-level appearance measure AM_1_3 of the first speaker being “age”, the first or third-level appearance level AL_1_3 may indicate or correspond to an age range such as “under 20 years old”, “20 to 40 years old”, “40 to 60 years old” or “over 60 years old” or an age label such as “young”, “middle-aged” or “old”.

[0103] The first appearance measure AM_1_i may include a confidence score also labeled ACS_1_i (i = 1, 2, ..., S, where S is the number of confidence scores). In other words, determining the first appearance measure AM_1_i may include determining the first appearance confidence score ACS_1_i. For example, determining the first-level appearance measure AM_1_2 may include determining the first-level appearance confidence score ACS_1_2. The first appearance confidence score ACS_1_i of the appearance measure AM_1_i may indicate the score or probability that the determined first appearance measure AM_1_i, such as the first appearance level AL_1_i, is correct (e.g., the appearance measure or appearance level is correct). For example, ACS_1_2 = 0.95 may indicate that there is a 95% probability that the determined AL_1_2 is "male".

[0104] In one or more exemplary methods, the first speaker representation is the caller representation. The first speaker representation could be, for example, the representation of the first participant in or during a meeting. The first speaker representation could be, for example, the representation of the first presenter during or in a speech or moderation session. The caller representation could be the representation of someone calling a call center, such as someone supporting a call center.

[0105] In one or more exemplary methods, the first speaker representation is an agent representation. The agent representation could be the representation of an agent answering calls in a call center, such as a call center support system.

[0106] In one or more exemplary methods, determining the first speaker representation SR_1 (such as determining the first principal speaker representation SR_1_1 of the first speaker representation SR_1) includes determining one or more first features F_1_i, i = 1, ..., K, where K is the number of first features. The one or more first features may include a first principal feature of the first principal speaker representation SR_1_1, also labeled F_1_1, and / or a first-level feature, also labeled F_1_2. The number L of first features may be 2, 3, 4, 5, or more. The first principal speaker representation S_1_1 may be or include a first avatar, a first emoji, a first smiley face, a first icon, a first image, a first animation, and / or a first visual graphic representation simulation.

[0107] In one or more exemplary methods, determining a first speaker representation SR_1 includes determining a first primary feature, also labeled F_1_1, and / or a first secondary feature, also labeled F_1_2, based on a first primary emotion measure SM_1_1 and / or a first primary appearance measure AM_1_1. Optionally, a first speaker representation such as a first primary speaker representation SR_1_1 in a first speaker representation SR_1 includes a first incarnation. Determining SR_1 (such as determining SR_1_1) may include determining one or more features, such as a first feature, based on one or more emotion measures, such as a first emotion measure. Determining SR_1 (such as determining SR_1_1) may include determining one or more features, such as a first feature, based on one or more emotion measures, such as a first emotion measure and / or one or more appearance measures. Determining SR_1 (such as determining SR_1_1) may include determining F_1_1 based on SM_1_1 and / or AM_1_1. In other words, the first speaker representation SR_1, such as the first primary speaker representation SR_1_1, can be based on one or more first features, for example, based on F_1_1 and F_1_2.

[0108] The first primary feature F_1_1 can indicate the first primary emotion measure SM_1_1. In other words, F_1_1 can indicate the primary emotional state indicated by SM_1_1. For example, when the primary emotional state indicated by SM_1_1 is negative, F_1_1 can indicate a negative feature, such as negative eyes or negative mouth.

[0109] F_1_1 can be selected from a list of features and / or a class of features. F_1_1 can be selected from or chosen from a set of features (e.g., a set of feature types), and a number or value can be assigned to each feature type in the set of feature types.

[0110] A primary representation, such as a first avatar, can indicate the primary emotional state of the first speaker. A first avatar can be a real-time physical and / or emotional expression of the first speaker. A first avatar can be a facial expression indicating the speaker's emotional state and / or the first speaker's appearance. The term "representation" can be understood as one or more of avatar, smiley face, emoji, emoticon, portrait, personification, sketch, animation, visual graphic representation, simulation, and appearance. A primary representation, such as a first avatar, can be the sum of one or more primary features representing one or more emotions or emotional states of the first speaker and / or one or more appearances of the first speaker. A primary representation, such as a first avatar, can include at least one feature, at least two features, at least five features, or at least ten features.

[0111] In one or more exemplary methods, the first primary feature F_1_1 is selected from mouth features, eye features, nose features, forehead features, eyebrow features, hair features, ear features, beard features, gender features, cheek features, accessory features, skin features, body features, trunk features, leg features, height features, foot features, and head size features.

[0112] A first feature F_1_i, such as F_1_1, may include a feature identifier also labeled F_ID_i (i = 1, 2, ..., L, where L is the number of feature identifiers). In other words, one or more first features that determine the first primary representation may include a first primary identifier F_ID_1_1 that determines the first primary feature.

[0113] For example, the feature identifier of the first feature can be, for instance, a number, value (such as an integer), or label corresponding to or indicating the feature type. For example, F_ID_1_i can be selected from a set of feature types, and a number or value can be assigned to each feature type in that set. The set of feature types can include at least five different feature types, for example, L=5; at least ten different feature types, for example, L=10; or at least twenty feature types, for example, L=20. For example, the first feature type identifier F_ID_1 can indicate or correspond to the feature type “eyes,” “mouth,” “nose,” “forehead,” “eyebrows,” “hair,” “ears,” “beard,” “gender,” “cheeks,” “accessories,” “skin,” “body,” or “head size.”

[0114] A first feature F_1_i, such as F_1_1, may include feature levels also labeled FL_i (i = 1, 2, ..., F, where F is the number of feature levels). In other words, determining one or more first features that define the first primary representation may include determining the first primary feature level FL_1_1. The first feature level FL_1_i may indicate the level, value, range, or label of the first feature F_1_i, for example, as indicated by the feature identifier F_ID_1_i. In other words, the first feature level FL_1_i may indicate the level, value, range, or label of the first feature F_1_i. For example, when F_ID_1_1 corresponds to the first feature of the first speaker as “head size”, the first primary feature level FL_1_1 may, for example, indicate or correspond to “male head size”, “female head size”, or optionally “genderless head size” based on one or more emotion measures and / or one or more appearance measures. A first feature F_1_i, such as F_1_1, may include multiple feature levels also labeled F_1_i_j. Therefore, the first feature can be based on multiple metrics, such as emotion metrics and appearance metrics. For example, a first feature (such as F_1_2) with feature identifier F_ID_1_2 = "eyes" can include first feature levels FL_1_2_1 = "angry" and FL_1_2_2 = "feminine", thereby allowing the eye feature represented by the first speaker to indicate an angry woman.

[0115] In one or more exemplary methods / systems, one or more first features F_1_i are determined based on one or more first emotion measures and / or one or more first appearance measures, such as being mapped from one or more first emotion measures and / or one or more first appearance measures.

[0116] Determining the first primary speaker representation may include, for example, selecting a first avatar from an avatar library based on one or more of a first feature, a first emotion measure, and a first appearance measure. Determining the first primary speaker representation may include constructing and / or generating a first avatar that includes one or more first feature icons, optionally wherein one or more (such as each) first feature icon is based on one or more first features. In other words, determining the first primary speaker representation may include optionally selecting a first feature icon from a first feature icon library based on one or more of a first feature, a first emotion measure, and a first appearance measure, and including that first feature icon in the first primary speaker representation (such as in the first avatar).

[0117] In one or more exemplary methods, determining the first speaker representation SR_1 (such as determining the first primary speaker representation SR_1_1) includes determining the first primary feature F_1_2 of the first primary speaker representation SR_1_1, such as the first incarnation, based on the first primary emotion metric SM_1_1 and / or based on the first primary appearance metric AM_1_1.

[0118] Determining SR_1 may include determining F_1_2 based on SM_1_1 and / or AM_1_1. In one or more exemplary methods, determining SR_1 may include determining F_1_2 based on SM_1_1, SM_1_2, AM_1_1 and / or AM_1_2.

[0119] The first-level feature F_1_2 can indicate the first primary appearance measure AM_1_1. In other words, F_1_2 can indicate the primary appearance indicated by AM_1_1. For example, when the primary appearance indicated by AM_1_1 is "old age", F_1_2 can indicate aging features, such as wrinkled eyes or wrinkled mouth.

[0120] F_1_2 can be selected from a list of features and / or a class of features. F_1_2 can be selected from or chosen from a set of features (e.g., a set of feature types), and each feature type in the set of feature types can be assigned a number or value.

[0121] In one or more exemplary methods, the first-level feature is different from the first primary feature and is selected from mouth features, eye features, nose features, forehead features, eyebrow features, hair features, ear features, beard features, gender features, cheek features, accessory features, skin features, body features, trunk features, leg features, height features, foot features, and head size features.

[0122] A first feature F_1_i, such as F_1_2, may include a feature identifier also labeled F_ID_i (i = 1, 2, ..., L). In other words, determining one or more first features of the first primary representation may include a first-level identifier F_ID_1_2 that determines the first-level feature.

[0123] For example, the feature identifier of the first feature can be, for instance, a number, value (such as an integer), or label corresponding to or indicating the feature type. For example, F_ID_1_i can be selected from a set of feature types, and a number or value can be assigned to each feature type in that set. The set of feature types can include at least five different feature types, for example, L=5; at least ten different feature types, for example, L=10; or at least twenty feature types, for example, L=20. For example, the first feature type identifier F_ID_1 can indicate or correspond to the feature type “eyes,” “mouth,” “nose,” “forehead,” “eyebrows,” “hair,” “ears,” “beard,” “gender,” “cheeks,” “accessories,” “skin,” “body,” or “head size.”

[0124] A first feature F_1_i, such as F_1_2, may include feature levels also labeled FL_i (i = 1, 2, ..., F). In other words, determining one or more first features of a first primary representation may include determining a first-level feature level FL_1_2 for the first-level feature. The first feature level FL_1_i may indicate the level, value, range, or label of the first feature F_1_i, for example, indicated by the feature identifier F_ID_1_i. In other words, the first feature level FL_1_i may indicate the level, value, range, or label of the first feature F_1_i. For example, when F_ID_1_2 corresponds to the first feature of the first speaker as “head size”, the first-level feature level FL_1_2 may indicate or correspond to “male head size”, “female head size”, or optionally “genderless head size”, for example, based on one or more emotion measures and / or one or more appearance measures. A first feature F_1_i, such as F_1_2, may include multiple feature levels also labeled F_1_i_j. Thus, a first feature may be based on multiple measures, such as emotion measures and appearance measures. For example, a first feature (such as F_1_2) with the feature identifier F_ID_1_2 = "eyes" can include first feature levels FL_1_2_1 = "angry" and FL_1_2_2 = "female", thereby allowing the eye feature represented by the first speaker to indicate an angry female.

[0125] In one or more exemplary methods, obtaining one or more audio signals includes obtaining a second audio signal, also labeled AS_2. The second audio signal may represent a second speech / voice of a second speaker. The second speaker may be a caller, an agent, or a second participant in a telephone conversation or conference.

[0126] Acquiring one or more audio signals may include acquiring one or more audio inputs, such as a second audio input. In other words, the second audio signal may be based on a second audio input. A second audio signal may be acquired from a call or conversation between a user and another person (e.g., a first call or first conversation). The second speaker may be the person who speaks / talks the second most during the first call and / or first conversation. The second speaker may be the second person who speaks during the first call and / or first conversation. The second speaker may be the person speaking to the person of interest (e.g., the first speaker). The second speaker may be the user who wants the representation of the first speaker (e.g., the first speaker's representation). The second audio signal may be acquired from an electronic device that records second audio input from the second speaker (e.g., recording the second speaker speaking or talking). Acquiring one or more audio signals may include generating one or more audio signals that include the second audio signal. The second audio signal may be based on a second audio input. The second audio signal may include the second speaker's audio signal. The second audio signal may be a proxy audio signal or a caller's audio signal, and the second audio signal is another signal.

[0127] In one or more exemplary methods, the method includes determining one or more second emotion measures, also labeled SM_2_i (i = 1, 2, ..., A, where A is the number of second emotion measures).

[0128] One or more second emotion metrics, such as SM_2_1 and / or SM_2_2, indicate the state of the second speaker, i.e., one or more first emotion states of the second speaker, and may be based on a first audio signal and / or a second audio signal. One or more second emotion metrics SM_2_i may include one or more parameters indicating the state of the second speaker.

[0129] One or more second emotion measures SM_2_i include a second primary emotion measure also labeled SM_2_1 and indicating the primary emotional state of the second speaker.

[0130] In other words, SM_2_1 can indicate the second speaker's primary emotion. SM_2_1 can indicate the second speaker's dominant emotional state and / or secondary emotional state over time. The dominant emotional state can be one of the following: disappointment, boredom, fear, sadness, negativity, very negative, fear, frustration, annoyance, panic, hesitation, dissatisfaction, anger, surprise, worry, doubt, incomprehension, contemplation, neutrality, positivity, very positive, happiness, friendliness, confidence, excitement, triumph, satisfaction, exhilaration, contentment, relaxation, energy, enthusiasm, service-oriented, helpfulness, interest, and joy. In one or more exemplary methods / systems, the second speaker's dominant emotional state can be selected from a set of dominant emotional states.

[0131] The second emotion measure SM_2_i may include a second emotion type identifier also labeled ST_ID_2_i (i = 1, 2, ..., B, where B is the number of second emotion type identifiers and i is the index). In other words, determining one or more second emotion measures SM_2_i may include determining the second emotion type identifier ST_ID_2_i, for example, the second primary emotion type identifier ST_ID_2_1 of the second primary emotion measure SM_2_1. The emotion type identifier indicates the speaker's emotional state.

[0132] Emotion type identifiers can be, for example, labels, numbers, or values ​​(such as integers) corresponding to a specific emotion (state), emotion type, and / or emotion category. For example, ST_ID_2_i can be selected from the same or different sets of emotion type identifiers. For example, ST_ID_2_1 can be selected from a primary set of emotion type identifiers and / or ST_ID_2_2 can be selected from a secondary set of emotion type identifiers. The primary set of emotion type identifiers can be different from or the same as the secondary set of emotion type identifiers. The primary and secondary sets of emotion type identifiers can share one or more (such as multiple) emotion type identifiers.

[0133] In one or more exemplary methods, the second primary emotion measure SM_2_1 includes a second primary emotion type identifier ST_ID_2_1 selected from the primary emotion type identifier set ST_ID_SET_1, wherein ST_ID_SET_1 includes multiple emotion type identifiers, for example, including at least three, four, five or more emotion type identifiers.

[0134] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0135] ST_ID_SET_1 = {1,2,3,4,5}

[0136] "1" indicates a mood such as "very negative", "2" indicates a mood such as "negative", "3" indicates a mood such as "neutral", "4" is optional and indicates a mood such as "positive", and "5" is optional and indicates a mood such as "very positive".

[0137] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0138] ST_ID_SET_1 = {"Very negative", "Negative", "Neutral", "Positive", "Very positive"}

[0139] For example, "very negative" and / or "very positive" are optional.

[0140] In one or more exemplary methods, the set of primary emotion type identifiers ST_ID_SET_1 is given in the following manner:

[0141] ST_ID_SET_1 = {1,2,3,4,5}

[0142] The "1" indicates an emotion such as "anger", the "2" indicates an emotion such as "listlessness", the "3" indicates an emotion such as "good spirits", the optional "4" indicates an emotion such as "friendly, engaged, energetic", and the optional "5" indicates an emotion such as "highly engaged".

[0143] The set of emotion type identifiers, such as the set of primary emotion type identifiers and / or the set of secondary emotion type identifiers, may include at least three or at least four different emotion type identifiers, such as five, six, seven, eight, nine, or more emotion type identifiers. In other words, each emotion or emotion type may have a corresponding ST_ID_2_i. For example, the second primary emotion measure SM_2_1 may include a second primary emotion type identifier ST_ID_2_1 indicating or corresponding to the primary emotion state or the second speaker state as “positive”.

[0144] The second emotion measure SM_2_i may include emotion levels also labeled SL_2_i (i = 1, 2, ..., C, where C is the number of emotion levels). In other words, determining SM_2_i may include determining SL_2_i; for example, determining SM_2_1 may include determining the second primary emotion level SL_2_1. The emotion level SL_2_i may indicate the level of the i'th emotion type. In other words, SL_2_i may indicate the degree of the i'th emotion type. For example, when ST_ID_2_1 corresponds to the second speaker state "positive," the second primary emotion level SL_2_1 may indicate or correspond to the degree of the emotion "positive," for example, within a numerical range of, for example, from 0 to 1 or from 1 to 10, or selected from "low," "medium," and "high." In other words, the emotion level of the emotion measure may be within a numerical range of, for example, from 0 to 1 or from 1 to 10.

[0145] The second sentiment metric SM_2_i may include confidence scores labeled SCS_2_i (i = 1, 2, ..., C, where C is the number of confidence scores). In other words, determining SM_2_i may include determining the second confidence score SCS_2_i; for example, determining the second primary sentiment metric SM_2_1 may include determining the second primary confidence score SCS_2_1. The confidence score of the sentiment metric may indicate the score or probability that a determined sentiment metric, such as a sentiment type identifier and / or sentiment level, is correct (e.g., the sentiment state or sentiment type (as identified by the sentiment type identifier of the sentiment metric) is correct). For example, SCS_2_1 = 0.88 may indicate that there is an 88% probability that the determined ST_ID_2_1 is, for example, "positive".

[0146] Determining one or more second emotion measures indicative of a second speaker's state may include extracting one or more speaker features from a second audio signal, for example, wherein the one or more second emotion measures are based on one or more speaker features. The one or more speaker features may include, for example, speaker timbre features, speaker intonation features, speaker strength or volume features, speaker pitch features, speaker timbre features, speaker speech rate features, linguistic features, acoustic features, and / or speaker spectral energy features. Spectral energy features may include individual grids of a spectrogram indicating the signal energy level at a given frequency.

[0147] Linguistic features may include specific emotion-related words, such as positive and / or negative words. Linguistic features can be determined based on a text transcript of the audio signal. The text transcript can be obtained through a human annotator or using an automatic speech recognition (speech-to-text) algorithm or service. Linguistic features may include embedded features from deep neural networks (e.g., BERT transformer networks or other sequence-to-sequence autoencoders).

[0148] In one or more exemplary methods, one or more second emotion metrics can be determined based on a machine learning (ML) model (such as the output of the ML model). The ML model can include linear regression models, support vector machines, decision tree classifiers (e.g., random forests, XGBoost), Gaussian mixture models, hidden Markov models, and / or neural networks. The neural network can, for example, include one or more of linear feedforward layers, convolutional layers, recursive layers, and attentional layers. The ML model can include weights for one or more speaker features. For example, the ML model can map, for example, speaker tone and / or vocal quality, to emotion metrics / types, emotion levels, and / or emotion confidence scores. The ML model can include parameters ranging from 100,000 to 1,000,000, for example, from 500,000 to 1,000,000 parameters. The ML model can include layers ranging from 5 to 20, for example, from 10 to 15 layers.

[0149] ML models can be trained based on, for example, call logs, where a verifier or supervisor, such as a human supervisor, has assigned emotion identifiers / labels to emotion metrics and / or speaker feature labels to speaker features. Speaker features can be algorithmically determined via signal processing algorithms. One or more second emotion metrics can be inferred using the ML model. The input to the ML model can include one or more acoustic features such as timbre features. Timbre features can be, for example, negative or positive timbre. Furthermore, the input to the ML model can include spectrograms, latent (hidden) activation representations of (deep) neural networks. The input to the ML model can include static feature vectors ("fingerprints") such as mean, variance, slope, peak distance, and modulation spectrum. The input to the ML model can include frame-by-frame (low-level) acoustic features, such as the pitch, energy level, spectral parameters (Mel frequency cepstral, MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectrum envelope characteristics (e.g., formants, harmonics, harmonic-to-formant ratio), and / or sound quality measurements, such as harmonic noise ratio, HNR, jitter, and / or perturbation.

[0150] In one or more exemplary methods, the method includes obtaining one or more second appearance metrics, also labeled AM_2_i (i = 1, 2, ..., D, where D is the number of second appearance metrics of the second speaker or associated with the second speaker). Obtaining one or more second appearance metrics may include determining one or more second appearance metrics.

[0151] Appearance metrics indicate the speaker's appearance. Appearance metrics may include one or more of appearance identifiers, appearance levels, and confidence scores, and may indicate the appearance of a second speaker, for example, based on a first audio signal and / or a second audio signal. Obtaining one or more second appearance metrics may include, for example, retrieving one or more second appearance metrics from a database based on information about the second speaker stored in the database. When the second speaker is a user / agent of the system, obtaining one or more second appearance metrics may include retrieving one or more second appearance metrics from an employee database, for example, that includes information about the appearance of employees. Alternatively or additionally, the second speaker (e.g., an agent) may select or choose one or more second primary features and / or one or more second speaker representations, such as from a database of second primary features and / or second speaker representations. One or more second appearance metrics AM_2_i may include one or more parameters indicating the appearance of the second speaker. One or more second appearance metrics AM_2_i include a second primary appearance metric also labeled AM_2_1 and indicating the primary appearance of the second speaker and / or a second secondary appearance metric also labeled AM_2_2 and indicating the secondary appearance of the second speaker.

[0152] In other words, AM_2_1 can indicate the primary physical appearance of the second speaker. AM_2_1 can be selected from gender metrics, weight metrics, height metrics, age metrics, language metrics, language ability metrics, hearing metrics, dialect metrics, health metrics (e.g., breathing condition, speech defects and / or speech disorders), personality metrics (e.g., extroverted or introverted), and comprehension metrics (e.g., based on age, health, and / or gender metrics). Comprehension metrics may be relevant when an older person has difficulty hearing conversations or when a foreigner is not accustomed to spoken language. Comprehension metrics can provide the user with instructions, for example, that he / she should speak more slowly and clearly.

[0153] The second appearance metric AM_2_i may include a second appearance identifier also labeled A_ID_2_i (where i is the index of the i'-th second appearance metric). The second appearance identifier may indicate one of the following: gender metric, weight metric, height metric, age metric, language metric, language ability metric, hearing metric, and comprehension metric. In other words, determining AM_2_i may include determining A_ID_2_i, including the second primary appearance identifier A_ID_2_1 of the second primary appearance metric AM_2_1 and / or the second secondary appearance identifier A_ID_2_2 of the second secondary appearance metric AM_2_2.

[0154] Appearance identifiers can be, for example, labels, numbers, or values ​​(such as integers) corresponding to a specific appearance metric, appearance type, and / or appearance category. For example, A_ID_2_i can be selected from a set of appearance types, including, for example, one or more of gender, height, weight, age, language, language ability, hearing, and comprehension. Appearance metric identifiers can be labels or numbers mapped to and / or indicating appearance metric types.

[0155] In one or more exemplary methods, the second primary appearance metric AM_2_1 includes a second primary appearance identifier A_ID_2_1 optionally selected from the primary appearance identifier set A_ID_SET_1, wherein A_ID_SET_1 includes multiple appearance identifiers, for example, including at least three, four, five or more appearance type identifiers. In one or more exemplary methods, the second primary appearance metric AM_2_1 is a gender metric, that is, the second primary metric identifier A_ID_2_1 indicates gender, for example, it can be mapped to gender via a table A_ID_2_1 = "gender" or A_ID_2_1 = 1.

[0156] In one or more exemplary methods, the secondary appearance metric AM_2_2 includes a secondary appearance identifier A_ID_2_2 optionally selected from the secondary appearance identifier set A_ID_SET_2, wherein A_ID_SET_2 includes multiple appearance identifiers, for example, including at least three, four, five or more appearance identifiers. In one or more exemplary methods, the secondary appearance metric AM_2_2 is an age metric, that is, the secondary metric identifier A_ID_2_2 indicates age, for example, it can be mapped to age via a table A_ID_2_2 = "age" or A_ID_2_2 = 2.

[0157] The set of appearance identifiers may include two or at least three or at least four different appearance identifiers, such as five, six, seven, eight, nine or more appearance identifiers. For example, the second primary appearance measure AM_2_1 may include a second primary appearance identifier A_ID_2_1 that indicates or corresponds to the primary appearance of the second speaker (e.g., one of gender, weight, height, age, language, language ability, hearing, and comprehension). For example, the secondary appearance identifier A_ID_2_2 may indicate or correspond to the secondary appearance of the second speaker, such as one of gender, weight, height, age, language, language ability, hearing, and comprehension. The secondary appearance identifier may optionally be different from the second primary appearance identifier.

[0158] The second appearance metric AM_2_i may include appearance levels also labeled AL_2_i (i = 1, 2, ..., E, where E is the number of appearance levels). In other words, determining AM_2_i may include determining AL_2_i; for example, determining AM_2_1 may include determining the second primary appearance level AM_2_1. The second appearance level AL_2_i may indicate the level, value, range, or label of the appearance metric AM_2_i, as indicated by the appearance identifier A_ID_2_i. In other words, the second appearance level AL_2_i may indicate the level, value, range, or label of the second appearance metric AM_2_i. For example, when A_ID_2_1 corresponds to the second primary appearance of the second speaker as "gender," the second primary appearance level AL_2_1 may indicate or correspond to "male," "female," or optionally "genderless." For example, when the secondary appearance identifier A_ID_2_2 corresponds to the secondary appearance measure of the second speaker as "height", the secondary appearance level AL_2_2 can indicate or correspond to "short", "medium", or "tall". For example, when the secondary appearance identifier A_ID_2_2 corresponds to the secondary appearance measure of the second speaker as "height", the secondary appearance level AL_2_2 can indicate or correspond to "shorter than 160cm", "between 160cm and 185cm", or "taller than 185cm".

[0159] For example, when a second appearance identifier such as a second or third-level appearance identifier A_ID_2_3 corresponds to a second appearance metric such as a second or third-level appearance metric AM_2_3 for the second speaker, which is “age”, the second or third-level appearance level AL_2_3 can indicate or correspond to an age range such as “under 20 years old”, “20 to 40 years old”, “40 to 60 years old”, or “over 60 years old”, or an age label such as “young”, “middle-aged”, or “old”.

[0160] The second appearance measure AM_2_i may include a confidence score also labeled ACS_2_i (i = 1, 2, ..., G, where G is the number of confidence scores). In other words, determining the second appearance measure AM_2_i may include determining the second appearance confidence score ACS_2_i; for example, determining the second primary appearance measure AM_2_1 may include determining the second primary appearance confidence score ACS_2_1. The second appearance confidence score ACS_2_i of the appearance measure AM_2_i may indicate the score or probability that the determined second appearance measure AM_2_i, such as the second appearance level AL_2_i, is correct (e.g., the appearance measure or appearance level is correct). For example, ACS_2_1 = 0.95 may indicate that there is a 95% probability that the determined AL_2_1 is "female".

[0161] Determining one or more second appearance metrics that indicate a second speaker may include extracting one or more speaker appearance features from a second audio signal. One or more speaker appearance features may include, for example, speaker timbre features, speaker intonation features, speaker power features, speaker pitch features, speaker quality features, speaker speech rate features, speech features, acoustic features, and / or speaker spectral energy features.

[0162] Spectral energy characteristics can include individual grid cells of a spectrogram indicating the signal energy level at a given frequency.

[0163] Linguistic features may include specific appearance-related words, such as positive and / or negative words. Linguistic features can be determined based on text transcripts of audio signals. Text transcripts can be obtained through human annotators or using automatic speech recognition (speech-to-text) algorithms or services. Linguistic features may include embedded features from deep neural networks (e.g., BERT transformer networks or other sequence-to-sequence autoencoders).

[0164] In one or more exemplary methods, one or more second appearance metrics can be determined based on a machine learning (ML) model (such as the output of an ML model). One or more second appearance metrics can be inferred using an ML model. ML models can include linear regression models, support vector machines, decision tree classifiers (e.g., random forests, XGBoost), Gaussian mixture models, hidden Markov models, and / or neural networks. Neural networks can, for example, include one or more of linear feedforward layers, convolutional layers, recursive layers, and attention layers. ML models can include weights for one or more speaker features. For example, an ML model can map, for example, speaker tone and / or vocal quality, to emotion metrics / types, emotion levels, and / or emotion confidence scores. ML models can include parameters ranging from 100,000 to 1,000,000, for example, from 500,000 to 1,000,000 parameters. ML models can include layers ranging from 5 to 20, for example, from 10 to 15 layers.

[0165] ML models can be trained based on, for example, call recordings, where a verifier or supervisor, such as a human supervisor, has assigned emotion identifiers / labels to emotion metrics and / or speaker feature labels to speaker features. Speaker features can be algorithmically determined via signal processing algorithms. One or more second appearance metrics can be inferred using the ML model. The input to the ML model can include audio data, such as audio data stored in a database of known audio data that matches one or more appearance metrics, such as appearance labels. Appearance labels can include labels assigned by people and / or basic facts, such as age or height from a passport or social registry. For example, audio data input can include recordings of phone calls, television programs, and / or film actors, etc.

[0166] The input to an ML model can include one or more acoustic features such as timbre. Timbre features can be, for example, negative or active timbre. Additionally, the input to an ML model can include spectrograms, latent (hidden layer activation) representations of (deep) neural networks. The input to an ML model can include static feature vectors (“fingerprints”) such as mean, variance, slope, peak distance, and modulation spectrum. The input to an ML model can include frame-by-frame (low-level) acoustic features such as speech pitch, energy levels, spectral parameters (Mel frequency cepstral, MFCC; e.g., logMelSpec), spectral statistics (slope, roll-off point), speech spectrum envelope characteristics (e.g., formants, harmonics, harmonic-to-formant ratio), and / or phonological measurements such as harmonic-to-noise ratio, HNR, jitter, and / or perturbations. For example, acoustic features related to one or more appearance metrics such as body shape can include the ratio of vowel formants related to vocal tract length. For example, acoustic features may be related to one or more appearance metrics such as body size, timbre characteristics (e.g., age-related HNR, jitter, and / or perturbations) (e.g., more breathing, older age, more jitter), and pitch may be gender-related (e.g., males may have pitches below 150 Hz and females may have pitches above 150 Hz). Furthermore, acoustic features may include, for example, linguistic and dialectal features and / or mean spectral envelope features such as phoneme stocks / histograms for age and / or gender.

[0167] One or more second emotion metrics and one or more second appearance metrics may be part of the second speaker metrics data. The second speaker metrics data may also be represented as agent metrics data and / or caller metrics data.

[0168] In one or more exemplary methods, the method includes determining a second speaker representation, also labeled SR_2, based on a second primary emotion measure SM_2_1 and / or a second appearance measure AM_2_1.

[0169] The second speaker representation SR_2 may include a second primary speaker representation also labeled SR_2_1. Determining SR_2_1 may include generating the second primary speaker representation SR_2_1 based on SM_2_1 and AM_2_1. The second speaker representation may be determined based on public and / or customer registration. For example, for frequent callers / customers, the second primary emotion measure SM_2_1 and / or the second primary appearance measure AM_2_1 may be refined over multiple calls / conversations; for example, the more voices heard and the more audio data obtained, the more confidently one can determine that the speaker is, for example, male. One or more emotion and / or appearance measures may be known, for example, age from social registration and / or emotional state from previous conversations. One or more known emotion and / or appearance measures may be used to improve the accuracy of speaker representation determination and / or to determine speaker representation.

[0170] The second speaker representation SR_2 may include a second-level speaker representation also labeled SR_2_2. The second speaker representation SR_2 may include a second-third-level speaker representation also labeled SR_2_3. The second speaker representation SR_2 may include a second-fourth-level speaker representation also labeled SR_2_4. The second speaker representation SR_2 may include a second-fifth-level speaker representation also labeled SR_2_5. Therefore, determining the second speaker representation may include determining one or more of SR_2_2, SR_2_3, SR_2_4, and SR_2_5 based on a second audio signal (such as based on a second primary emotion measure SM_2_1 and / or a second primary appearance measure AM_2_1).

[0171] The second speaker can also be represented as the second person.

[0172] The second speaker representation can indicate the second speaker's state and / or appearance substantially in real time (e.g., with a delay of less than 5 or 10 seconds). The second speaker representation can indicate segments being analyzed, such as speech fragments or utterances. For example, a voice activity detection module can identify one or more speech / utterance segments and discard noise. Segments can be, for example, speech segments of at least 5 or 10 seconds. The voice activity detection module can detect pauses longer than, for example, 400 ms, 500 ms, or 1 second. Speech segments can be detected at the following times: when a pause occurs, when another speaker begins to speak, or when the segment reaches a defined maximum length (e.g., up to 8 seconds), indicating the end of the speech segment. For each speech segment, one or more emotion metrics and / or one or more appearance metrics can be determined. In one or more exemplary methods / systems, the second speaker representation, such as the second primary speaker representation, is updated at an update frequency of at least 0.2 Hz (e.g., every second, every 5 seconds, every 7 seconds, or every 10 seconds). The frequency of updates indicated by the primary speaker can vary over time and can also depend on factors such as the pace of the conversation.

[0173] In other words, second speaker representations, such as those of the second primary speaker, can be updated during the second speaker's speech and / or during conversations between the second speakers (such as telephone calls). Second speaker representations can indicate or reflect the second speaker's present, real-time, immediate, and / or current emotions and / or appearance. For example, a second speaker representation such as that of the second primary speaker could be a real-time physical and emotional expression of the second speaker.

[0174] The advantages of having a real-time or near-real-time second speaker representation include: users of the electronic device can see or understand changes in emotion and / or the second speaker's appearance in real time. Furthermore, users of the electronic device can better imagine or envision the second speaker's emotions and / or appearance by seeing the second speaker representation, which includes the second primary speaker representation. In other words, users can have the experience of video speaking or video calling without receiving a video signal.

[0175] Second speaker representation can provide users with real-time feedback on the second speaker's speech, such as feedback on the second speaker's characteristics, like the second speaker's state and / or appearance. Second speaker representation can provide a realistic representation of the second speaker. It can provide anthropomorphism, portraits, appearances, sketches, and / or gamification of the second speaker.

[0176] Second speaker representations may include sound representations, such as auditory feedback and / or audio icons.

[0177] In one or more exemplary methods, determining a second speaker representation SR_2 (such as determining a second primary speaker representation SR_2_1 of the second speaker representation SR_2) includes determining one or more second features F_2_i, i = 1, ..., J, where J is the number of second features. The one or more second features may include a second primary feature of the second primary speaker representation SR_2_1 also labeled as F_2_1 and / or a second-level feature also labeled as F_2_2. The number L of second features may be 2, 3, 4, 5, or more. The second primary speaker representation S_2_1 may be or include a second avatar, a second emoji, a second smiley face, a second icon, or a second image.

[0178] In one or more exemplary methods, determining a second speaker representation SR_2 includes determining a second primary feature, also labeled F_2_1, and / or a second secondary feature, also labeled F_2_2, of a second incarnation based on a second primary emotion measure SM_2_1 and / or a second primary appearance measure AM_2_1. Optionally, a second speaker representation such as a second primary speaker representation SR_2_1 in a second speaker representation SR_2 includes a second incarnation. Determining SR_2 (such as determining SR_2_1) may include determining one or more features, such as a second feature, based on one or more emotion measures, such as a second emotion measure. Determining SR_2 (such as determining SR_2_1) may include determining one or more features, such as a second feature, based on one or more emotion measures, such as a second emotion measure and / or one or more appearance measures. Determining SR_2 (such as determining SR_2_1) may include determining F_2_1 based on SM_2_1 and / or AM_2_1. In other words, a second speaker representation SR_2, such as a second primary speaker representation SR_2_1, can be based on one or more second features, for example, based on F_2_1 and F_2_2.

[0179] The second primary feature F_2_1 can indicate the second primary emotion measure SM_2_1. In other words, F_2_1 can indicate the primary emotional state indicated by SM_2_1. For example, when the primary emotional state indicated by SM_2_1 is negative, F_2_1 can indicate negative features, such as negative eye or negative mouth.

[0180] F_2_1 can be selected from a list of features and / or a class of features. F_2_1 can be selected from or chosen from a set of features (e.g., a set of feature types), and each feature type in the set of feature types can be assigned a number or value.

[0181] A second primary representation, such as a second incarnation, can indicate the primary emotional state of the second speaker. A second incarnation can be a real-time physical and / or emotional expression of the second speaker. A second incarnation can be a facial expression indicating the speaker's emotional state and / or the second speaker's appearance. The term "representation" can be understood as one or more of incarnation, smiley face, emoji, emoticon, portrait, personification, sketch, and appearance. A second primary representation, such as a second incarnation, can be the sum of one or more secondary features indicating one or more emotions or emotional states of the second speaker and / or one or more appearances of the second speaker. A second primary representation, such as a second incarnation, can include at least one feature, at least two features, at least five features, or at least ten features.

[0182] In one or more exemplary methods, the method includes outputting a second speaker representation SR_2 via an interface of an electronic device. Outputting the second speaker representation SR_1 may include displaying a second user interface indicating the second speaker representation.

[0183] A user interface may include one or more (such as a plurality of) user interface objects. For example, a second user interface may include one or more second user interface objects, such as a second primary user interface object and / or a second-level user interface object. A user interface object may refer herein to a graphical representation of an object displayed on an interface of an electronic device, such as a display. User interface objects may be interactive with the user or selectable through user input. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may each optionally constitute a user interface object. User interface objects may form part of a widget. A widget can be considered a mini-application that a user can use.

[0184] In one or more exemplary methods, one or more second emotion measures SM_2_i include a second-level emotion measure also labeled SM_2_2 and indicating the secondary emotional state of the second speaker. For a description of the second-level emotion measure SM_2_2 for the second speaker, refer back to the description of the first-level emotion measure SM_1_2 for the first speaker. The description of the second-level emotion measure SM_2_2 for the first speaker is equivalent to the description of the first-level emotion measure SM_1_2, except that the first-level emotion measure SM_1_2 is replaced by the second-level emotion measure SM_2_2.

[0185] In one or more exemplary methods, one or more second appearance metrics AM_2_i include a second-level appearance metric also labeled AM_2_2 and indicating the secondary appearance of the second speaker. For a description of the second-level appearance metric AM_2_2 for the second speaker, refer back to the description of the first-level appearance metric AM_1_2 for the first speaker. The description of the secondary appearance metric AM_2_2 for the first speaker is equivalent to the description of the first-level appearance metric AM_1_2, except that the first-level appearance metric AM_1_2 is replaced by the second-level appearance metric AM_2_2.

[0186] In one or more exemplary methods, the second speaker representation is an agent representation. The agent representation could be the representation of an agent answering calls in a call center, such as a support call center.

[0187] In one or more exemplary methods, the method includes detecting speech termination. Detecting speech termination may include detecting the termination of speech or utterance by a first speaker and / or a second speaker. Speech termination may be, for example, the termination of a conversation, the termination of a sentence, or the termination of a monologue.

[0188] In one or more exemplary methods, the method includes: storing a speaker record in a memory and / or transmitting the speaker record to a server device of the system based on the detection of a speech termination. In one or more exemplary methods, the speaker record includes a first speaker record indicating one or more of first appearance measurement data and first emotion measurement data of a first speaker.

[0189] An electronic device is disclosed. The electronic device includes a processor, a memory, and an interface. The processor is configured to perform any of the methods according to this disclosure.

[0190] An electronic device is disclosed. The electronic device includes a processor, a memory, and an interface. The processor is configured to receive one or more audio signals, including a first audio signal.

[0191] Electronic devices may include, for example, one or more of mobile phones, computers, and tablets. An electronic device may be, for example, a user device configured to perform a call between a user and another person, such as a mobile phone or computer. An electronic device may be configured to receive a first audio input, such as first audio input from a conversation between a user and another person. For example, an electronic device may act as a call agent device, where the user can be an agent, such as an agent of a call center (such as a support call center, after-sales call center, marketing call center, or sales call center). An electronic device may be, for example, a user device configured to record first audio input from a first speaker (such as recording the first speaker speaking or talking), such as a mobile phone or computer. An electronic device may be configured to receive one or more audio signals, such as generating one or more audio signals including a first audio signal. The first audio signal may be based on the first audio input.

[0192] The processor is configured to determine one or more first emotion metrics indicating the state of a first speaker based on a first audio signal. The one or more first emotion metrics include a first primary emotion metric indicating the primary emotional state of the first speaker. The processor is configured to determine one or more first appearance metrics indicating the appearance of the first speaker. The one or more first appearance metrics include a first primary appearance metric indicating the primary appearance of the first speaker. The processor is configured to determine a first speaker representation based on the first primary emotion metrics and the first primary appearance metrics. The processor is configured to output the first speaker representation via an interface.

[0193] The interface of the electronic device may include a first display. Optionally, the system may include a second interface separate from the electronic device, such as a second display (e.g., a wing). The second interface may act as the display rather than the interface of the electronic device. The first display of the electronic device may be configured to detect user input, such as a first primary user input. For example, when the first display includes a touch-sensitive display, the user input may include touch input from the user. User input may include touch on the touch-sensitive display and / or a keyboard included in or connected to the electronic device. The touch-sensitive display may provide a first user interface and / or a second user interface (such as an input interface) and an output interface between the electronic device and the user. The processor of the electronic device may be configured to receive electrical signals from and / or send electrical signals to the touch-sensitive display. The touch-sensitive display may be configured to display visual output to the user, such as a first speaker representation and / or a second speaker representation. The visual output may optionally include graphics, text, icons, video, audio icons, and any combination thereof (collectively, "graphics"). For example, some, most, or all of the visual output may be considered to correspond to user interface objects. The electronic device can also be configured to output a first speaker representation, including audio output, such as a sound representation, audio icon, and / or auditory feedback.

[0194] The processor of an electronic device can be configured to display one or more user interfaces, such as a user interface screen, on an interface (e.g., a first display). The user interface may include one or more (such as a plurality of) user interface objects. For example, the first user interface may include a first primary user interface object and / or a first-level user interface object. The second user interface may include a second primary user interface object and / or a second-level user interface object. The user interface may be referred to as a user interface screen.

[0195] Inputs such as user inputs can include touch (e.g., tapping, pressure touch, long press), click (e.g., mouse click), typing (e.g., typing on a keyboard), audio input (e.g., voice assistant), and / or touch movement (e.g., swipe gestures for switching). Touch movement can be detected via a touch-sensitive surface (e.g., on a first display of an electronic device). Therefore, the first display can be a touch-sensitive display. A first input (such as a first user input) can include lifting off. User inputs such as a first primary user input, a second primary user input, and / or a secondary user input can include touching followed by movement and then lifting off.

[0196] A system is disclosed. The system includes a server device and an electronic device. The electronic device is the electronic device according to this disclosure.

[0197] The system can be a system for monitoring, processing, and / or analyzing one or more audio signals, such as a speaker's speech (e.g., as a monologue). The system can be a system for monitoring, processing, and / or analyzing one or more audio signals, such as conversations between two or more individuals, such as conversations in a telephone call or conference. The system can, for example, include or act as a call center system for monitoring, processing, and / or analyzing one or more audio signals, such as conversations between two or more individuals, for example, telephone conversations between an agent and a customer or caller in a call center system.

[0198] It should be understood that the description of the features of (several) methods can also be applied to corresponding features in electronic devices, server devices, and electronic circuit / audio devices.

[0199] Figure 1An exemplary system, such as system 2, with speaker representation according to the present disclosure is illustrated schematically. System 2 includes an electronic device 10 and optionally a server device 20. The electronic device 10 includes a memory 10A, one or more interfaces 10B, and a processor 10C. The server device 20 includes a memory 20A, one or more interfaces 20B, and one or more processors 20C. User 1A can use the electronic device 10 (e.g., a mobile phone or computer) to make or receive calls from speaker 1B (e.g., a first speaker). Speaker 1B can use speaker electronics 30 to communicate with user 1A.

[0200] Electronic device 10 can be configured to act as a user device, which user 1A can use to communicate with speaker 1B and / or monitor conversations / conversations with speaker 1B. Electronic device / processor 10C is configured to acquire 14 one or more audio signals, including a first audio signal. 22 The first audio signal can be acquired from speaker electronic device 30, for example, via network 40 such as a global network (e.g., the Internet or a telecommunications network). 14 The first audio signal can be acquired from server device 20, for example, via network 40 such as a global network (e.g., the Internet or a telecommunications network).

[0201] Speaker electronic device 30 can be configured to record audio input 32 from speaker 1B, such as a first audio input, for example, recording speaker 1B speaking or talking. Speaker electronic device 30 can be configured to acquire one or more audio signals, such as generating one or more audio signals based on audio input 32, including generating a first audio signal based on the first audio input. Speaker electronic device 30 can be configured to transmit the first audio signal 22 to electronic device 10, for example, via network 40. Speaker electronic device 30 can be configured to acquire one or more audio signals from electronic device 10, for example, based on user input 4, such as user audio input. User input 4 can be user 1A speaking or talking; for example, electronic device 10 records user 1A speaking or talking. User 1A can be a first speaker and / or a second speaker.

[0202] Electronic device / processor 10C is configured to determine one or more first emotion measures indicating the state of a first speaker based on a first audio signal. The one or more first emotion measures include a first primary emotion measure indicating the primary emotional state of the first speaker 1B.

[0203] Optionally, one or more processors 20C are configured to determine one or more first emotion measures indicative of the state of the first speaker based on the first audio signal. The processor 10C can then be configured to obtain, for example via network 40, one or more emotion measures indicative of the state of the first speaker from server device 20 based on the first audio signal. The processor 20C can be configured to transmit, for example via network 40, one or more emotion measures to electronic device 10.

[0204] The processor 10C is optionally configured to determine one or more first appearance metrics that indicate the appearance of the first speaker 1B. The one or more first appearance metrics include a first primary appearance metric that indicates the primary appearance of the speaker 1B.

[0205] Optionally, processor 20C is configured to determine one or more first appearance metrics indicative of the appearance of speaker 1B. Processor 20C may then be configured to obtain, for example, the one or more appearance metrics indicative of the appearance of speaker 1B from server device 20 via network 40. Processor 20C may be configured to transmit, for example, the one or more appearance metrics to electronic device 10 via network 40.

[0206] The processor 10C is configured to determine the first speaker representation based on a first primary emotion metric and a first primary appearance metric.

[0207] Optionally, processor 20C is configured to determine a first speaker representation based on a first primary emotion metric and a first primary appearance metric. Processor 20C may then be configured to obtain the first speaker representation from server device 20, for example, via network 40. Processor 20C may be configured to transmit the first speaker representation / second speaker representation to electronic device 10, for example, via network 40.

[0208] The processor 10C is configured to output, for example, a first speaker representation via interface 10B.

[0209] Processor 10C can be configured to detect speech termination, such as termination, end, or sustaining of a call, and to store speaker recordings in memory 10A and / or transmit speaker recordings to server device 20 based on the detected speech / call termination. Speaker recordings indicate, include, and / or are based on one or more emotion and / or appearance metrics determined during speech. Processor 10C can be configured, for example, to transmit speaker recordings 12 to server device 20 of system 2 via network 40, depending on the detected speech termination. Speaker recordings may include a first speaker record indicating one or more of first appearance metrics and first emotion metrics for a first speaker 1B.

[0210] Electronic device 10 can be configured to perform in Figure 2A , 2B Any of the methods disclosed in the document.

[0211] Processor 10C is optionally configured to execute in Figures 2A to 2B Any operation disclosed in the above (such as any one or more of S104A, S104B, S106A, S106B, S108A, S108A, S108AA, S108BA, S110A, S110B, S110AA, S110BA, S112, S114, S116). The operation of the electronic device may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 10A) and executed by the processor 10C.

[0212] Processor 20C is optionally configured to execute in Figures 2A to 2B Any operation disclosed in the above (such as any one or more of S104A, S104B, S106A, S106B, S108A, S108A, S108AA, S108BA, S110A, S110B, S110AA, S110BA, S112, S114, S116). The operation of the server device may be embodied in the form of an executable logical routine (e.g., lines of code, software program, etc.) stored on a non-transitory computer-readable medium (e.g., memory 20A) and executed by the processor 20C.

[0213] Furthermore, the operation of electronic device 10 can be viewed as a method configured to be implemented by electronic device 10. Moreover, although the described functions and operations can be implemented in software, this functionality can also be implemented via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0214] Figure 2A and Figure 2BA flowchart illustrating an exemplary method is shown. A method 100 for operating a system including an electronic device is disclosed. The electronic device includes an interface, a processor, and a memory. Method 100 includes obtaining, in step S102, one or more audio signals including a first audio signal AS_1 and optionally a second audio signal AS_2. Method 100 includes determining, in step S104, one or more emotion measures indicative of a speaker's state based on the one or more audio signals. Determining the one or more emotion measures in step S104 may include, for example, determining, in step S104A, one or more first emotion measures including a first primary emotion measure indicative of a first speaker's primary emotional state based on AS_1 and / or AS_2. Determining the one or more emotion measures in step S104 may include, for example, determining, in step S104B, one or more second emotion measures indicative of a second speaker's primary emotional state based on AS_1 and / or AS_2.

[0215] Method 100 includes determining one or more appearance measures in S106 that indicate the appearance of a speaker. Determining one or more appearance measures in S106 may include, for example, determining one or more first appearance measures in S106A that include a first primary appearance measure indicating the primary appearance of a first speaker, based on AS_1 and / or AS_2. Determining one or more appearance measures in S106 may include, for example, determining one or more second appearance measures in S106B that include a second primary appearance measure indicating the primary appearance of a second speaker, based on AS_1 and / or AS_2.

[0216] Method 100 includes determining a speaker representation S108 based on one or more emotion metrics and one or more appearance metrics. Determining the speaker representation S108 may include determining a first speaker representation S108A based on a first primary emotion metric and / or a first primary appearance metric.

[0217] Determining the first speaker representation of S108A may include determining one or more first features of the first incarnation of S108AA. Determining one or more first features of the first incarnation of S108AA may include determining the first principal feature of that first incarnation.

[0218] Determining the speaker representation in S108 may include determining the second speaker representation in S108B based on a second primary emotion measure and a second primary appearance measure.

[0219] Determining the second speaker representation of S108B may include determining one or more second features of the second incarnation of S108BA. Determining one or more second features of the second incarnation of S108BA may include determining a second primary feature of that second incarnation.

[0220] Method 100 includes outputting one or more speaker representations via an interface of an electronic device, S110. The output of one or more speaker representations may include, for example, output S110A, a first speaker representation including a first avatar.

[0221] The output S110A first speaker representation may include output S110AA, for example, the first main speaker representation of the first main incarnation.

[0222] Output S110 may include one or more speaker representations, such as output S110B, which may include a second speaker representation of a second incarnation.

[0223] The output S110B second speaker representation may include output S110BA, for example, the second main speaker representation including the second main incarnation.

[0224] In one or more exemplary methods, method 100 includes detecting S112 voice termination, such as termination, end, or sustaining of a call.

[0225] In one or more exemplary methods, based on the detection S112 to the end of speech, the method includes storing the speaker record S114 in a memory.

[0226] In one or more exemplary methods, upon detection S112 to the end of speech, the method includes transmitting a speaker record S116 to a server device of the system. The speaker record includes a first speaker record indicating one or more of first physical appearance measurement data and first emotional measurement data of a first speaker.

[0227] In one or more exemplary methods, when no speech termination is detected, the method includes repeating / restarting method B 100, for example, at a frequency of at least 0.2 Hz (e.g., every second, every 5 seconds, every 7 seconds, or every 10 seconds), where no pause is detected. The update frequency of the first primary speaker representation may vary over time and may also depend on the conversation, such as the conversation speed.

[0228] Figure 3 An exemplary electronic device according to the present disclosure, such as electronic device 10, is illustrated schematically. In this example, electronic device 10 is a laptop computer. Electronic device 10 is configured to display a user interface including a speaker representation SR on an interface 10B of the electronic device (e.g., on the display 11 of the laptop computer). The user interface includes a plurality of user interface objects. Electronic device 10 displays a first speaker representation as a first user interface object SR_1, for example, having a first display area. The first speaker representation SR_1 includes, for example, a first primary speaker representation SR_1_1 arranged in the first display area.

[0229] The first primary speaker representation is based on a first audio signal, for example, based on a determined first primary emotion measure SM_1_1 and / or a determined first primary appearance measure AM_1_1. The determination of the first primary speaker representation SR_1_1 is based on the determination of a first primary emotion measure SM_1_1 indicating the state of the first speaker; for example, selecting a first emotion measure from five different emotion measures to indicate a negative state. The determination of the first primary speaker representation SR_1_1 is based on the determination of a first primary appearance measure AM_1_1 indicating the primary appearance of the first speaker; for example, selecting a first primary appearance measure from two different appearance measures to indicate a male primary appearance. Therefore, the first primary speaker representation SR_1_1 is determined to be negative male; for example, the first incarnation is determined to be negative male. Determining the first primary speaker representation may include determining a first primary feature of the first incarnation based on the first primary emotion measure. For example, in the case of a negative first primary emotion measure, a first primary feature, such as an eye feature, may be selected from five different types of eyes as negative male eyes. For example, if the primary emotional measure is negative, the first-level feature of the mouth can be selected from four different types of mouths as negative male mouths.

[0230] The second primary speaker representation is based on a second audio signal, for example, based on a determined second primary emotion measure SM_2_1 and / or a determined second primary appearance measure AM_2_1. The determination of the second primary speaker representation SR_2_1 is based on the determination of a second primary emotion measure SM_2_1 indicating the second speaker's state; for example, selecting a second emotion measure from four different emotion measures to indicate a positive state. The determination of the second primary speaker representation SR_2_1 is based on the determination of a second primary appearance measure AM_2_1 indicating the second speaker's primary appearance; for example, selecting a second primary appearance measure from two different appearance measures to indicate a female primary appearance. Therefore, the second primary speaker representation SR_2_1 is determined to be positive female; for example, the second incarnation is determined to be positive female. Determining the second primary speaker representation may include determining a second primary feature of the second incarnation based on the second primary emotion measure. For example, in the case where the second primary emotion measure is positive, a second primary feature, such as an eye feature, may be selected from five different types of eyes as positive female eyes. For example, if the second primary emotion measure is positive, the second-level feature of the mouth can be selected from four different types of mouths as positive female mouths, such as the smiling incarnation.

[0231] Electronic device 10 displays a first-level speaker representation as a first-level user interface object SR_1_2, for example, having a first-level display area. The first speaker representation SR_1 may include, for example, a first-level speaker representation SR_1_2 arranged in the first-level display area. The first-level speaker representation may indicate the speech ratio between a first speaker and a second speaker. The first-level speaker representation may be based on a first audio signal and a second audio signal. The first-level speaker representation may be determined as a comparison between the first audio signal and the second audio signal. The second speaker representation SR_2 may include a similar second-level speaker representation SR_2_2.

[0232] Electronic device 10 displays a first third-level speaker representation as a first third-level user interface object SR_1_3, for example, having a first third-level display area. The first speaker representation SR_1 may include, for example, a first third-level speaker representation SR_1_3 arranged in the first third-level display area. The first third-level speaker representation may indicate interruptions and excessive talking between the first and second speakers. The first third-level speaker representation may be based on a first audio signal and a second audio signal. The first third-level speaker representation may be determined as a comparison between the first audio signal and the second audio signal. The second speaker representation SR_2 may include a similar second third-level speaker representation SR_2_3.

[0233] Electronic device 10 displays a first fourth-level speaker representation as a first fourth-level user interface object SR_1_4, for example, having a first fourth-level display area. The first speaker representation SR_1 may include, for example, a first fourth-level speaker representation SR_1_4 arranged in the first fourth-level display area. The first fourth-level speaker representation may indicate a prolonged pause by the first speaker. The first fourth-level speaker representation may be based on a first audio signal. The second speaker representation SR_2 may include a similar second fourth-level speaker representation SR_2_4.

[0234] Electronic device 10 displays a first fifth-level speaker representation as a first fifth-level user interface object SR_1_5, for example, having a first fifth-level display area. The first speaker representation SR_1 may include, for example, a first fifth-level speaker representation SR_1_5 arranged in the first fifth-level display area. The first fifth-level speaker representation may indicate the speech rate of the first speaker. The first fifth-level speaker representation may be based on a first audio signal. The second speaker representation SR_2 may include a similar second fifth-level speaker representation SR_2_5.

[0235] Electronic device 10 displays a first sixth-level speaker representation as a first sixth-level user interface object SR_1_6, for example, having a first sixth-level display area. The first speaker representation SR_1 may include, for example, a first sixth-level speaker representation SR_1_6 arranged in the first sixth-level display area. The first sixth-level speaker representation may indicate the tone of the first speaker. The first sixth-level speaker representation may be based on a first audio signal. The second speaker representation SR_2 may include a similar second sixth-level speaker representation SR_2_6.

[0236] Electronic device 10 displays a first seventh-level speaker representation as a first seventh-level user interface object SR_1_7, for example, having a first seventh-level display area. The first speaker representation SR_1 may include, for example, a first seventh-level speaker representation SR_1_7 arranged within the first seventh-level display area. The first seventh-level speaker representation may indicate a timeline of conversation between a first speaker and a second speaker. The timeline may also indicate speech segments from the first and second speakers. The first seventh-level speaker representation may be based on a first audio signal and a second audio signal.

[0237] Figure 4 An exemplary user interface including a speaker representation (SR) is illustrated schematically. The user interface includes multiple user interface objects. The speaker representation (SR) includes a first primary speaker representation (SR_1_1).

[0238] The first primary speaker representation is based on a first audio signal, for example, based on a determined first primary emotion measure SM_1_1 and / or a determined first primary appearance measure AM_1_1. The determination of the first primary speaker representation SR_1_1 is based on the determination of a first primary emotion measure SM_1_1 indicating the state of the first speaker; for example, selecting a first emotion measure from four different emotion measures to indicate a neutral state. The determination of the first primary speaker representation SR_1_1 is based on the determination of a first primary appearance measure AM_1_1 indicating the primary appearance of the first speaker; for example, selecting a first primary appearance measure from two different appearance measures to indicate a male primary appearance. Therefore, the first primary speaker representation SR_1_1 is determined to be a neutral male; for example, the first incarnation is determined to be a neutral male. Determining the first primary speaker representation may include determining a first primary feature of the first incarnation based on the first primary emotion measure. For example, in the case where the first primary emotion measure is neutral, a first primary feature, such as an eye feature, may be selected from five different types of eyes as a neutral male eye. For example, if the first primary emotion measure is neutral, the first-level feature of the mouth feature can be selected from four different types of mouths as the neutral male mouth.

[0239] Speaker representation SR can include the first eighth-level speaker representation SR_1_8. The first eighth-level speaker representation can indicate the first primary emotion level SL_1_1 of the first primary emotion measure SM_1_1. The first primary emotion level is selected from 1 to 10 and is 7.4. Therefore, when the first primary emotion measure SM_1_1 is neutral, the first primary emotion level is 7.4 (neutral).

[0240] The speaker representation SR can include a second primary speaker representation SR_1_1. The second primary speaker representation is based on a second audio signal, for example, based on a determined second primary emotion measure SM_2_1 and / or a determined second primary appearance measure AM_2_1 of the second speaker. The second primary speaker representation SR_2_1 is determined based on a determined second primary emotion measure SM_2_1 indicating the second speaker's state; for example, selecting a second emotion measure from four different emotion measures to indicate a very positive state. The second primary speaker representation SR_2_1 is also determined based on a determined second primary appearance measure AM_2_1 indicating the second speaker's primary appearance; for example, selecting a second primary appearance measure from two different appearance measures to indicate a female primary appearance. Therefore, the second primary speaker representation SR_2_1 is determined to be a very positive female, for example, the second incarnation is determined to be a very positive female. Determining the second primary speaker representation can include determining a second primary feature of the second incarnation based on the second primary emotion measure. For example, in the case where the second primary emotion measure is very positive, a second primary feature, such as eye features, can be selected from five different types of eyes as very positive female eyes. For example, if the second primary emotion measure is very positive, then the second-level feature of the mouth feature can be selected from four different types of mouths as very positive female mouths.

[0241] Speaker representation SR can include a second eighth-level speaker representation SR_2_8. This second eighth-level speaker representation can indicate the second primary emotion level SL_2_1 of the second primary emotion measure SM_2_1. The second primary emotion level is selected from 1 to 10, and is 9.2. Therefore, when the second primary emotion measure SM_2_1 is very positive, the second primary emotion level is 9.2, which is very positive.

[0242] exist Figure 3 and Figure 4 The speaker representations SR_1, SR_1_1, SR_2, and SR_2_1 are exemplified using the representations from "www.iconfinder.com / UsersInsights".

[0243] Figure 5 An exemplary system with speaker representation according to this disclosure, such as System 2, is illustrated schematically. System 2 is similar to... Figure 1 The system is shown, but in which user 1A and speaker or speaker group 1C are in the same room or place, for example. User 1A and speaker 1C can hold meetings, such as seminars, in-person meetings, or job interviews. Electronic device 10, such as interface 10B, may include or be connected to a microphone, through which user 1A and / or speaker 1C can speak to provide audio input 32.

[0244] Optionally, the speaker electronics 30 may include a microphone, into which the speaker 1C may speak to provide audio input 32. The speaker electronics 30 may be locally connected to the electronics 10, for example, via interface 10B. The connection may be wired or wireless, such as Bluetooth. The speaker electronics 30 may transmit 22 of one or more audio signals, including a first audio signal, to the electronics 10 via, for example, the connection via interface 10B. The speaker electronics 30 may receive 24 of one or more audio signals from the electronics 10 via the connection.

[0245] Figure 6 An exemplary data structure according to this disclosure is illustrated schematically. Speaker measurement data SPMD may include first speaker measurement data SPMD_1. First speaker measurement data SPMD_1 includes a first emotion measurement SM_1_i. First emotion measurement SM_1_i includes a first primary emotion measurement SM_1_1 and optionally a first secondary emotion measurement SM_1_2. First speaker measurement data SPMD_1 includes a first appearance measurement AM_1_i. First appearance measurement AM_1_i includes a first primary appearance measurement AM_1_1 and optionally a first secondary appearance measurement AM_1_2.

[0246] Speaker measurement data SPMD may optionally include second speaker measurement data SPMD_2. Second speaker measurement data SPMD_2 may include a second emotion measure SM_2_i. Second emotion measure SM_2_i may include a second primary emotion measure SM_2_1 and optionally a second secondary emotion measure SM_2_2. Second speaker measurement data SPMD_2 may include a second appearance measure AM_2_i. Second appearance measure AM_2_i may include a second primary appearance measure AM_2_1 and optionally a second secondary appearance measure AM_2_2.

[0247] The use of the terms "first," "second," "third," and "fourth," "primary," "secondary," and "third-level," etc., does not imply any particular order, but is included to identify individual elements. Furthermore, the use of the terms "first," "second," "third," and "fourth," "primary," "secondary," and "third-level," etc., does not indicate any order or importance, but is used to distinguish one element from another. Note that the terms "first," "second," "third," and "fourth," "primary," "secondary," and "third-level," etc., are used here and elsewhere solely for marking purposes and are not intended to indicate any specific spatial or temporal order.

[0248] Memory can be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical arrangement, memory may include non-volatile memory for long-term data storage and volatile memory used as system memory for the processor. Memory can exchange data with the processor via a data bus. Memory can be considered a non-transitory computer-readable medium.

[0249] The memory can be configured to store information (such as information indicating one or more audio signals, one or more emotion metrics, one or more appearance metrics, speaker representation, emotion metric data and / or appearance metric data) in a portion of the memory.

[0250] Furthermore, marking the first element does not imply the existence of a second element, and vice versa.

[0251] It is understandable. Figures 1 to 6 This includes modules or operations illustrated with solid lines and modules or operations illustrated with dashed lines. Modules or operations included in solid lines are those included in the most generalized example embodiment. Modules or operations included in dashed lines are those that can be included in, are part of, or are example embodiments of other modules or operations that may be taken in addition to those in the solid-line example embodiments. It should be understood that these operations do not need to be performed in the presented order. Furthermore, it should be understood that not all operations need to be performed. Exemplary operations can be performed in any order and in any combination.

[0252] It should be noted that the word "including" does not necessarily exclude the presence of elements or steps other than those listed.

[0253] It should be noted that the word "one" or "a" preceding a component does not preclude the existence of multiple such components.

[0254] It should be further noted that any reference numerals do not limit the scope of the claims, and exemplary embodiments may be implemented at least in part by means of both hardware and software, and several “components,” “units,” or “devices” may be represented by the same hardware item.

[0255] The various exemplary methods, apparatuses, and systems described herein are described in the general context of method steps and procedures, which may be implemented in one aspect by a computer program product embodied in a computer-readable medium comprising computer-executable instructions such as program code, and executed by a computer in a networked environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), optical disc (CD), digital versatile optical disc (DVD), etc. Generally, program modules may include routines, programs, objects, components, data structures, etc., that perform specified tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent instances of program code for performing steps of the methods disclosed herein. Specific sequences of such executable instructions or associated data structures represent instances of corresponding actions for implementing the functions described in such steps or procedures.

[0256] Although features have been shown and described, it will be understood that these features are not intended to limit the claimed invention, and those skilled in the art will appreciate that various changes and modifications can be made without departing from the spirit and scope of the claimed invention. Therefore, the specification and drawings are considered illustrative rather than restrictive. The claimed invention is intended to cover all alternatives, modifications, and equivalents.

[0257] Reference tag list

[0258] 1A User, Second Speaker

[0259] 1B Speaker, First Speaker, Caller

[0260] 1C Speakers, Speaker Groups

[0261] 2 System

[0262] 4 User Input

[0263] 6. Speaker representation and user output

[0264] 10 Electronic devices

[0265] 10A memory

[0266] 10B One or more interfaces

[0267] 10C processor

[0268] 12 Transmission

[0269] 14 obtained

[0270] 16 obtained

[0271] 18 Transmission

[0272] 20 server devices

[0273] 20A Memory

[0274] 20B interface

[0275] 20C One or more processors

[0276] 22 Transmission

[0277] 24 obtained

[0278] 30 Speaker Electronic Devices

[0279] 32. Speaker input, audio input

[0280] 40 Network

[0281] AS audio signal

[0282] AS_1 First Audio Signal

[0283] AS_2 First Audio Signal

[0284] A_ID Appearance Identifier

[0285] A_ID_1 First appearance identifier

[0286] A_ID_1_1 First primary appearance identifier

[0287] A_ID_1_2 First-level appearance identifier

[0288] A_ID_1_3 First and third level appearance identifier

[0289] A_ID_2_1 Second primary appearance identifier

[0290] A_ID_2_2 Second-level appearance identifier

[0291] A_ID_2_3 Second and third level appearance identifiers

[0292] A_ID_SET Appearance Identifier Set

[0293] A_ID_SET_1 Primary Appearance Identifier Set

[0294] A_ID_SET_2 Secondary Appearance Identifier Set

[0295] ACS Appearance Confidence Score

[0296] ACS_1 First Appearance Confidence Score

[0297] ACS_1_1 First Primary Appearance Confidence Score

[0298] ACS_1_2 First-order appearance confidence score

[0299] ACS_2 Second Appearance Confidence Score

[0300] ACS_2_1 Second Primary Appearance Confidence Score

[0301] AL Appearance Level

[0302] AL_1 First level of appearance

[0303] AL_1_1 First primary appearance level

[0304] AL_1_2 First-level appearance level

[0305] AL_1_3 First and third level of appearance

[0306] AL_2_1 Second primary appearance level

[0307] AL_2_2 Second-level appearance level

[0308] AL_2_3 Second and third level of appearance

[0309] AM Appearance Measurement

[0310] AM_1_1 First Primary Appearance Measure

[0311] AM_1_2 First-level appearance measurement

[0312] AM_2_1 Secondary Primary Appearance Measurement

[0313] AM_2_2 Second-order appearance measurement

[0314] F_1 First Feature

[0315] F_1_1 First primary feature

[0316] F_1_2 First-level feature

[0317] F_2 Second Feature

[0318] F_2_1 Second Main Feature

[0319] F_2_2 Second-order features

[0320] F_ID Feature Identifier

[0321] F_ID_1 Feature type identifier

[0322] F_ID_1_1 First primary characteristic identifier

[0323] F_ID_1_2 First-level feature identifier

[0324] FL feature level

[0325] FL_1_1 First Principal Feature Level

[0326] FL_1_2 First-level feature level

[0327] PCR was indicated after the conversation.

[0328] PCR_1 indicates after the first conversation

[0329] SCS Confidence Score

[0330] SCS_1 First Confidence Score

[0331] SCS_1_1 First Primary Confidence Score

[0332] SCS_1_2 First Level Confidence Score

[0333] SCS_2_1 Second Primary Confidence Score

[0334] SL Mood Level

[0335] SL_1_1 First Primary Emotional Level

[0336] SL_1_2 First Level of Emotion

[0337] SL_2_1 Second Primary Emotional Level

[0338] SM Emotional Measurement

[0339] SM_1 First Sentiment Measure

[0340] SM_1_1 First Primary Sentiment Measure

[0341] SM_1_2 First-Level Emotion Measurement

[0342] SM_2 Second Emotion Measure

[0343] SM_2_1 Second Primary Emotion Measure

[0344] SM_2_2 Second-Level Emotion Measurement

[0345] ST_ID_1_1 First Primary Emotion Type Identifier

[0346] ST_ID_1_2 First-level emotion type identifier

[0347] ST_ID_2_1 Second Primary Emotion Type Identifier

[0348] ST_ID_2_2 Secondary Emotion Type Identifier

[0349] ST_ID_SET_1 is the set of primary emotion type identifiers.

[0350] ST_ID_SET_2 Secondary Emotion Type Identifier Set

[0351] SPMD Speaker Measurement Data

[0352] SPMD_1 First Speaker Measurement Data

[0353] SPMD_1_1 First Lead Speaker Measurement Data

[0354] SPMD_1_2 First-level speaker measurement data

[0355] SPMD_1_3 Level 1 and 3 Speaker Measurement Data

[0356] SPMD_2 Second Speaker Measurement Data

[0357] SPMD_2_1 Second Primary Speaker Measurement Data

[0358] SR speaker indicates

[0359] SR_1 First speaker representation

[0360] SR_1_1 First Principal Speaker Representation

[0361] SR_1_2 First-level speaker representation

[0362] SR_1_3 First and Third Level Speaker Representation

[0363] SR_1_4 First and Fourth Level Speaker Representation

[0364] SR_1_5 Speaker representation of Level 1 and 5

[0365] SR_1_6 Level 1-6 Speaker Representation

[0366] SR_1_7 Level 1-7 Speaker Representation

[0367] SR_1_8 Level 1 / 8 Speaker Representation

[0368] SR_2 Second speaker representation

[0369] SR_2_1 Second primary speaker representation

[0370] SR_2_2 Second-level speaker representation

[0371] SR_2_3 Second and Third Level Speaker Representation

[0372] SR_2_4 Second and Fourth Level Speaker Representation

[0373] SR_2_5 Second and Fifth Level Speaker Representation

[0374] SR_2_6 Second and Sixth Level Speaker Representation

[0375] SR_2_7 Second and seventh level speaker representation

[0376] 100 Methods for operating systems including electronic devices

[0377] S102 obtains audio signal

[0378] S104 Determine the measure of emotion

[0379] S104A Determine the primary sentiment measure

[0380] S104B Determine the Second Emotional Measure

[0381] S106 Determine appearance measurements

[0382] S106A Determine the first appearance measurement

[0383] S106B Determines the Second Appearance Measurement

[0384] S108 Determine the speaker's representation

[0385] S108A Determine the first speaker's representation

[0386] S108B Determine the second speaker's representation

[0387] S110 Output Speaker Representation

[0388] S110A outputs the first speaker's name.

[0389] S110B outputs the second speaker representation.

[0390] S112 Detects voice termination

[0391] S114 Store speaker records

[0392] S116 Transmits the speaker recording to the system's server device

[0393] B. Reiterate, restart.

Claims

1. A method of operating a system including an electronic device, the electronic device including an interface, a processor, and a memory, the method comprising: The system acquires one or more audio signals, including a first audio signal; wherein the first audio signal is acquired from a call or conversation between a user of the electronic device and a caller, and wherein the user is an agent; using the processor and based on a machine learning model, the system determines one or more first emotion metrics indicative of a first speaker's state based on the first audio signal, the one or more first emotion metrics including a first primary emotion metric indicative of the first speaker's primary emotional state; wherein determining the one or more first emotion metrics includes extracting one or more speaker features from the first audio signal, the one or more speaker features including paralinguistic features, and wherein the one or more first emotion metrics are based on the paralinguistic features; Using the processor and based on a machine learning model, based on the first audio signal, determine one or more first appearance metrics that indicate the appearance of the first speaker, the one or more first appearance metrics including a first primary appearance metric that indicates the primary appearance of the first speaker; Determine the first speaker representation based on the first primary emotion measure and the first primary appearance measure; and The first speaker's statement is output via the interface of the electronic device.

2. The method according to claim 1, wherein, The one or more first emotion measures include a first-level emotion measure that indicates the second-level emotional state of the first speaker.

3. The method according to any one of claims 1 to 2, wherein, The one or more first appearance metrics include a first-level appearance metric that indicates the secondary appearance of the first speaker.

4. The method according to any one of claims 1 to 3, wherein, The first speaker representation is the caller representation.

5. The method according to any one of claims 1 to 3, wherein, The first speaker is represented by a proxy.

6. The method according to any one of claims 1 to 5, wherein, Determining the first speaker representation includes: determining a first primary feature of the first avatar based on the first primary emotion metric, wherein the first speaker representation includes the first avatar.

7. The method according to claim 6, wherein, The first primary feature is selected from: mouth features, eye features, nose features, forehead features, eyebrow features, hair features, ear features, beard features, gender features, cheek features, accessory features, skin features, body features, and head size features.

8. The method according to claim 6, wherein, Determining the first speaker representation includes: determining the first-level features of the first avatar based on the first primary appearance metric.

9. The method according to claim 8, wherein, The first-level feature is different from the first primary feature, and the first-level feature is selected from: mouth features, eye features, nose features, forehead features, eyebrow features, hair features, ear features, beard features, gender features, cheek features, accessory features, skin features, body features, and head size features.

10. The method according to claim 8 or 9, wherein, Obtaining one or more audio signals includes obtaining a second audio signal; the method includes: Based on the second audio signal, one or more second emotion measures indicating the state of the second speaker are determined, the one or more second emotion measures including a second primary emotion measure indicating the primary emotional state of the second speaker; Obtain one or more second appearance metrics that indicate the appearance of the second speaker, the one or more second appearance metrics including a second primary appearance metric that indicates the primary appearance of the second speaker; A second speaker representation is determined based on the second primary emotion measure and the second appearance measure; and The second speaker's statement is output via the interface of the electronic device.

11. The method according to claim 10, wherein, The second speaker is represented by a proxy.

12. The method according to any one of claims 1 to 11, wherein the method comprises: The system detects the termination of speech and, based on the detection of the termination of speech, stores the speaker record in the memory and / or transmits the speaker record to the server device of the system, the speaker record including a first speaker record indicating one or more of a first appearance measurement data and a first emotion measurement data of the first speaker.

13. An electronic device comprising a processor, a memory, and an interface, wherein, The processor is configured to perform any step of the method according to any one of claims 1 to 12.

14. An electronic device comprising a processor, a memory, and an interface, wherein, The processor is configured to: acquire one or more audio signals including a first audio signal; wherein the first audio signal is acquired from a call or conversation between a user of the electronic device and a caller, and wherein the user is an agent; using the processor and based on a machine learning model, determine one or more first emotion metrics indicating a first speaker's state based on the first audio signal, the one or more first emotion metrics including a first primary emotion metric indicating the first speaker's primary emotional state; wherein determining the one or more first emotion metrics includes extracting one or more speaker features from the first audio signal, the one or more speaker features including paralinguistic features, and wherein the one or more first emotion metrics are based on the paralinguistic features; using the processor and based on a machine learning model, determine one or more first appearance metrics indicating the first speaker's appearance based on the first audio signal, the one or more first appearance metrics including a first primary appearance metric indicating the first speaker's primary appearance; determine a first speaker representation based on the first primary emotion metric and the first primary appearance metric; and output the first speaker representation via the interface.

15. A system comprising a server device and electronic devices, wherein, The electronic device is the electronic device according to any one of claims 13 to 14.

Images