A voice self-adjustment method and system

Abstract

The embodiment of the specification provides a voice self-adjusting method and system, the method comprises: extracting multi-modal features of a user based on multi-modal data of the user; identifying an emotional state of the user based on the multi-modal features; determining a parameter adjustment amount based on the emotional state and an emotional influence matrix of the user; generating real-time output parameters of a virtual digital person based on basic output parameters of the virtual digital person and the parameter adjustment amount; and playing voice to the user based on the real-time output parameters.

Description

Technical Field

[0001] This manual relates to the field of speech processing, and in particular to a method and system for autonomous speech adjustment. Background Technology

[0002] With the rapid development of artificial intelligence and virtual reality technologies, virtual digital humans are increasingly being used in customer service, entertainment interaction, and educational companionship, placing higher demands on the naturalness and human-likeness of voice interaction. However, existing virtual digital human voice systems mostly rely on general or preset fixed voice models, lacking the ability to deeply perceive and dynamically respond to users' real-time behavior, emotional state, and personalized preferences. This results in relatively monotonous and stiff voice output in terms of tone, speed, and emotional expression, making it difficult to adaptively adjust to changes in the interactive scenario or user feedback, and also unable to continuously optimize through long-term learning. Consequently, this restricts the immersive experience and personalized service level of the user experience.

[0003] Therefore, there is an urgent need for a method for autonomous voice adjustment to overcome the limitations of existing virtual digital human voice output in terms of personalization and adaptability, and to achieve intelligent voice generation and optimization based on multi-dimensional user perception, real-time emotion analysis, dynamic scene adaptation, and continuous self-learning. Summary of the Invention

[0004] This specification provides one or more embodiments of a voice autonomous adjustment method, the method comprising: extracting multimodal features of a user based on the user's multimodal data; identifying the user's emotional state based on the multimodal features; determining parameter adjustment amounts based on the emotional state and the user's emotional influence matrix; generating real-time output parameters of the virtual digital human based on the virtual digital human's basic output parameters and the parameter adjustment amounts; and playing voice to the user based on the real-time output parameters.

[0005] One embodiment of this specification provides a voice autonomous adjustment system, the system comprising: an emotion perception module configured to: extract multimodal features of a user based on the user's multimodal data; and identify the user's emotional state based on the multimodal features; a parameter adjustment module configured to: determine parameter adjustment amounts based on the emotional state and the user's emotion influence matrix; and generate real-time output parameters of the virtual digital human based on the virtual digital human's basic output parameters and the parameter adjustment amounts; and a voice output module configured to: play voice to the user based on the real-time output parameters. Attached Figure Description

[0006] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is a schematic diagram illustrating application scenarios of the autonomous voice adjustment system according to some embodiments of this specification; Figure 2 This is an exemplary block diagram of a voice autonomous adjustment system according to some embodiments of this specification; Figure 3 This is an exemplary flowchart of a voice autonomous adjustment method according to some embodiments of this specification; Figure 4 This is a flowchart illustrating the generation and adjustment of basic output parameters according to some embodiments of this specification; Figure 5 This is a flowchart illustrating the process of determining emotional state according to some embodiments of this specification. Detailed Implementation

[0007] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0008] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0009] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0010] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of a voice autonomous adjustment system according to some embodiments of this specification.

[0012] In some embodiments, the autonomous voice adjustment system can meet the demand for flexible voice interaction in diverse scenarios, and its application scope is wide, including customer service, entertainment interaction, and educational companionship. The autonomous voice adjustment system integrates natural language processing and acoustic environment perception technologies, enabling dynamic optimization and adaptive adjustment of voice output volume, speech rate, and dialect accents. This improves the naturalness of human-computer interaction and the comfort of the user experience, ensuring efficient, accurate, and user-friendly voice services in different application environments.

[0013] In some embodiments, such as Figure 1 As shown, the application scenario 100 of the voice autonomous adjustment system (hereinafter referred to as application scenario 100) may include a user terminal 110, a network 120, a processor 130, a storage device 140, a user 150, and a virtual digital human 160.

[0014] User terminal 110 can be used to interact with users.

[0015] In some embodiments, the user terminal 110 includes an acquisition device and a playback device. The acquisition device is a device that acquires user multimodal data. Examples include a camera, microphone, etc. The playback device is a device that outputs voice to the user. Examples include a speaker, digital media player, and audio equipment.

[0016] Network 120 may include any suitable network capable of facilitating information and / or data exchange. In some embodiments, at least one component of application scenario 100 (e.g., a user terminal, processor, storage device, etc.) may exchange information and / or data with at least one other component in the application scenario via the network. For example, a processor may retrieve historical behavioral data of a user from a storage device via the network.

[0017] Processor 130 can process data and / or information obtained from other devices or system components. Based on this data, information, and / or processing results, the processor can execute program instructions to perform one or more functions described in this application. For example, when playing voice to a user, the processor can adjust real-time output parameters based on real-time feedback data from the user. In some embodiments, the processor may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-chip processing device). By way of example only, the processor may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), a microprocessor, or any combination thereof.

[0018] Storage device 140 may store data, instructions, and / or any other information related to the voice autonomous adjustment system. In some embodiments, storage device may store data and / or information acquired by user terminal 110, user terminal, and processor (e.g., user's multimodal characteristics, emotional state, parameter adjustment amounts, voice preference characteristics, etc.).

[0019] In some embodiments, a storage device may include one or more storage units, each of which may be a separate device or part of another device. In some embodiments, the storage device may be implemented on a cloud platform. In some embodiments, the storage device may be part of a user terminal or a processor.

[0020] User 150 refers to an individual who uses a voice-activated self-adjustment system.

[0021] In some embodiments, user needs vary depending on the application scenario. For example, in a customer service scenario, a user might be an elderly person calling a bank to inquire about services, whose need is for the voice self-adjustment system to slow down the speech and increase the volume so that the user can clearly hear the information. In an entertainment and interactive scenario, a user might be a young person using a voice assistant late at night, whose need is for the voice self-adjustment system to automatically lower the volume so as not to disturb family members. In an educational and companionship scenario, a user might be a child learning a foreign language, whose need is for the voice self-adjustment system to adjust the speech rate according to their pronunciation level to improve speech comprehension and learning efficiency.

[0022] A virtual digital human (DHuman) is a computer-generated virtual avatar capable of interacting with users. For example, a virtual digital human can be a virtual customer service representative, a virtual broadcaster, a virtual teacher, or a character in an interactive game.

[0023] In some embodiments, users interact with a virtual digital human in real time through a user terminal. The user terminal collects the user's multimodal data and sends it to a processor. Based on this data, the processor dynamically adjusts the virtual digital human's real-time voice output parameters. Finally, the virtual digital human in the user terminal plays the adjusted voice to the user, forming a closed-loop personalized interaction. In some embodiments, the processor and storage device can be independent devices or part of the user terminal.

[0024] Figure 2 This is an exemplary block diagram of a voice autonomous adjustment system according to some embodiments of this specification.

[0025] In some embodiments, such as Figure 2 As shown, the autonomous voice adjustment system 200 may include an emotion perception module 210, a parameter adjustment module 220, and a voice output module 230. In some embodiments, multiple modules of the autonomous voice adjustment system may run in a processor. For more information about processors, see [link to relevant documentation]. Figure 1 And its related descriptions.

[0026] In some embodiments, the emotion sensing module 210, parameter adjustment module 220, and voice output module 230 can establish a communication connection via a network or other means. In some embodiments, the emotion sensing module 210, parameter adjustment module 220, and voice output module 230 can be deployed in the same physical device or distributed across different terminals or servers, and work collaboratively through a network.

[0027] In some embodiments, the voice autonomous adjustment system 200 can be implemented in various forms. For example, the voice autonomous adjustment system 200 can be a standalone hardware entity connected to the user terminal 110 via an interface. Alternatively, the voice autonomous adjustment system 200 can be a software system, integrated into the processor of the user terminal 110 as software or firmware.

[0028] The emotion perception module 210 refers to a functional unit used to perceive, identify, or infer the user's emotional state. In some embodiments, the emotion perception module can process data from different sensing devices. Sensing devices include cameras, microphones, and speakers, etc.

[0029] In some embodiments, the emotion perception module is configured to: extract multimodal features of the user based on the user's multimodal data; and identify the user's emotional state based on the multimodal features.

[0030] The parameter adjustment module 220 is a functional unit used to determine the adjustment value of the parameter and generate the adjusted parameter based on the adjustment value and the basic parameter. In some embodiments, the parameter adjustment module 220 is communicatively connected to the emotion perception module 210 and the voice output module 230.

[0031] In some embodiments, the parameter adjustment module is configured to: determine the parameter adjustment amount based on the emotional state and the user's emotional influence matrix; and generate the virtual digital human's real-time output parameters based on the virtual digital human's basic output parameters and parameter adjustment amount.

[0032] In some embodiments, the parameter adjustment module is further configured to: construct a user preference model based on the user's historical behavior data; determine the user's voice preference features based on the preference model; and generate primary output parameters for the virtual digital human based on the voice preference features, as basic output parameters.

[0033] In some embodiments, the parameter adjustment module is further configured to adjust real-time output parameters based on real-time feedback data from the user when playing voice to the user.

[0034] In some embodiments, the parameter adjustment module is further configured to adjust real-time output parameters based on external environment data and interactive scenarios before playing voice to the user.

[0035] The voice output module 230 is a functional unit used to output corresponding voice based on the input real-time output parameters.

[0036] In some embodiments, the voice output module is configured to play voice to the user based on real-time output parameters.

[0037] For more information on the above modules, please refer to [link / reference]. Figures 3-5 And related explanations.

[0038] It should be noted that the above description of the voice autonomous adjustment system and its modules is for ease of description only and should not be construed as limiting this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 1 The emotion perception module, parameter adjustment module, and voice output module disclosed herein can be different modules within a single system, or a single module can implement the functions of two or more of the aforementioned modules. Such variations are all within the scope of protection of this specification.

[0039] Figure 3 This is an exemplary flowchart of a voice autonomous adjustment method according to some embodiments of this specification. Figure 3 As shown, process 300 includes steps 310-350 as described below. In some embodiments, process 300 is executed by a processor.

[0040] Step 310: Extract the user's multimodal features based on the user's multimodal data.

[0041] For more information about users, please see [link / details]. Figure 1 And its related descriptions.

[0042] Multimodal data refers to the raw data of multiple information modalities collected by the emotion perception module during real-time interaction between a user and a virtual digital human via a user terminal. In some embodiments, multimodal data may include voice data, semantic text data, and visual image data. Voice data refers to audio data containing the user's voice information. For example, voice data can be the original speech of the user when interacting with the virtual digital human. Semantic text data refers to text-based data that carries the semantic content of the user's speech. For example, semantic text data can be text content converted from the user's voice data through speech recognition. Visual image data refers to image or video data containing the user's facial information. For example, visual image data can be facial images or video frames captured by a camera.

[0043] In some embodiments, voice data may be collected by the microphone of the user terminal; semantic text data may be generated by the processor after performing speech recognition on the voice data collected by the microphone; and visual image data may be collected by the camera or video camera in the user terminal.

[0044] Multimodal features refer to feature vectors that characterize a user's emotional state and interaction preferences. In some embodiments, multimodal features include three modalities: semantic, audio, and facial expression features, represented by semantic sentiment vectors, audio sentiment vectors, and facial expression sentiment vectors, respectively.

[0045] Semantic sentiment vectors (SVCs) are vectorized data that represent the semantic and sentiment features inherent in semantic text data. In some embodiments, SVCs can be semantic representation vectors output by a natural language processing model after encoding the input text. In some embodiments, SVCs can consist of sentiment polarity intensity, sentiment category confidence, sentiment intensity, and subjective rating. Sentiment polarity intensity indicates the tendency and degree of the input text in the "positive-negative" dimension, with positive values ​​indicating positive and negative values ​​indicating negative; sentiment category confidence indicates the probability distribution of the input text corresponding to different sentiment categories; sentiment intensity indicates the arousal level or intensity of the emotion carried by the input text; and subjective rating assesses the degree to which the input text is an objective statement of fact or an expression of subjective opinion.

[0046] For example, the semantic text data is "Your answer is too professional", and the corresponding semantic sentiment vector can be: [sentiment polarity intensity +0.65, emotion category confidence_happy 0.10, emotion category confidence_angry 0.55, emotion category confidence_sad 0.35, emotion intensity 0.78, subjectivity score 0.92].

[0047] Audio emotion vectors are vectorized data that represent the emotional features contained in speech data. In some embodiments, audio emotion vectors can be feature vectors composed of prosodic feature parameters of speech data. Prosodic feature parameters are feature parameters that represent the rhythm and prosodic variation of speech. For example, prosodic feature parameters may include fundamental frequency, energy, Mel-Frequency Cepstral Coefficients (MFCC), speech rate, pauses, etc.

[0048] An expression emotion vector is a vectorized data representing the emotional features contained in visual image data. In some embodiments, an expression emotion vector may consist of facial action units and their corresponding activation intensity features. For example, an expression emotion vector may be: [inner eyebrow raised 3.0, eyebrow lowered 0.2, corner of mouth lowered 2.2].

[0049] In some embodiments, the processor extracts prosodic feature parameters from speech data using signal processing techniques (such as pitch detection algorithms and energy normalization methods) to construct an audio emotion vector; it converts speech data into semantic text data using speech recognition technology, and extracts semantic and emotion features from the semantic text data using natural language processing models (such as Bidirectional Encoder Representations from Transformers, BERT) to construct a semantic emotion vector; and it extracts the activation intensity of facial action units from visual image data using facial action unit detection techniques (such as facial keypoint detection models based on convolutional neural networks) to construct an expression emotion vector.

[0050] Step 320: Identify the user's emotional state based on multimodal features.

[0051] Emotional state refers to the emotional tendency exhibited by a user at the current moment of interaction. In some embodiments, a user may have multiple emotion categories, and the emotional state can be represented by a probability distribution vector consisting of the probabilities of multiple emotion categories. For example, an emotional state of [anger 0.3, sadness 0.3, calm 0.4] means that the probability of the user being angry is 0.3, the probability of being sad is 0.3, and the probability of being calm is 0.4.

[0052] In some embodiments, emotional states can be divided into single emotion-dominant states and complex emotion states.

[0053] A single dominant emotion state refers to a situation where, among the emotional states exhibited by a user, the probability corresponding to a single emotion is greater than a first threshold. A single emotion refers to a state containing only one basic emotion. Basic emotions include, but are not limited to, anger, sadness, calmness, and joy. The first threshold can be set by a person skilled in the art based on experience. For example, the first threshold can be 0.9.

[0054] For example, an emotional state of [anger 0.91, sadness 0.06, calm 0.03] indicates that the user is in a state dominated by the emotion of "anger", which is a single emotion-dominated state.

[0055] A complex emotional state refers to a situation where a user exhibits multiple basic emotions simultaneously exceeding a second threshold. The second threshold can be set by someone skilled in the art based on experience, and it is less than the first threshold. For example, the second threshold could be 0.4. Complex emotional states can include irony, forced smiles, and happiness masked by tension. For instance, an emotional state of [happiness 0.50, tension 0.45, anger 0.05] indicates that the user is experiencing "happiness masked by tension," which is a complex emotional state.

[0056] In some embodiments, the processor can analyze and identify a user's emotional state based on multimodal features using clustering algorithms. These clustering algorithms include K-means clustering, among others.

[0057] In some embodiments, the processor maps the semantic sentiment vector in the multimodal features to a preset emotion feature space; calculates multiple similarities between the semantic sentiment vector and multiple cluster centers in the emotion feature space and normalizes the multiple similarities to obtain the corresponding probability distribution vectors; similarly, it calculates the probability distribution vectors corresponding to the audio sentiment vector and the facial expression sentiment vector, and then, according to the preset modal weights, weights the probability distribution vectors corresponding to the semantic sentiment vector, the audio sentiment vector, and the facial expression sentiment vector to obtain the user's emotional state.

[0058] For example, the semantic sentiment vector includes three categories of emotions: happiness, sadness, and anger, which means that the emotion feature space includes three cluster centers. The similarity between the semantic sentiment vector and the corresponding three cluster centers is calculated, and the result is [0.85, 0.65, 0.30]. Then, the result is normalized to obtain the semantic sentiment vector [0.55, 0.33, 0.12]. Similarly, the probability distribution vectors corresponding to the audio emotion vector and the facial expression emotion vector are calculated. The probability distribution vector corresponding to the audio emotion vector is [0.25, 0.50, 0.25], and the probability distribution vector corresponding to the facial expression emotion vector is [0.13, 0.13, 0.74]. The user's emotional state can be obtained by weighting the vectors according to the preset modal weights (such as semantic emotion vector 0.2, audio emotion vector 0.3, and facial expression emotion vector 0.5). For example, happy = 0.55×0.2+0.25×0.3+0.13×0.5=0.250, sad = 0.33×0.2+0.50×0.3+0.13×0.5=0.281, angry = 0.12×0.2+0.25×0.3+0.74×0.5=0.469. The final user's emotional state is: [happy 0.25, sad 0.281, angry 0.469].

[0059] The emotion feature space refers to the feature space that represents the emotion features of different modalities.

[0060] Similarity is a numerical metric that indicates the degree of proximity between multimodal features and cluster centers. In some embodiments, the processor can calculate similarity using Euclidean distance.

[0061] Cluster centers are feature vectors in the emotion feature space used to represent the distribution of features of a certain emotion category. In some embodiments, cluster centers can be obtained by clustering training on historical data of manually labeled emotion states.

[0062] In some embodiments, the processor determines the user's multimodal emotion distribution based on multimodal features and an emotion classification model; wherein the emotion classification model is a machine learning model; and the emotional state is determined based on the multimodal emotion distribution and multiple emotion weights.

[0063] For more information on recognizing emotional states, see [link to relevant information]. Figure 5 And its related descriptions.

[0064] Step 330: Determine the parameter adjustment amount based on the emotional state and the user's emotional influence matrix.

[0065] The emotion influence matrix is ​​a matrix that represents the relationship between different emotion categories and the adjustment of numerical parameters in speech output parameters.

[0066] Voice output parameters refer to the acoustic parameters when a virtual digital human plays speech to a user. In some embodiments, voice output parameters may include numerical parameters and categorical parameters. Numerical parameters include pitch, speech rate, pauses, and volume. Categorical parameters include timbre and regional accent.

[0067] In some embodiments, different rows of the emotion influence matrix represent different emotions, different columns represent different speech output parameter types, and each element represents the emotion corresponding to the row in which the element is located, and the adjustment amount of the speech output parameter corresponding to the column in which the element is located.

[0068] For example, if the emotion influence matrix is ​​a 2-row, 4-column matrix, it consists of row vectors [-0.2, -0.3, +0.4, -0,2] and [+0.3, +0.2, -0.1, +0.5]. The row vector [-0.2, -0.3, +0.4, -0.2] corresponds to the emotion "sadness". The speech output parameters from column 1 to column 4 are pitch, speech rate, pause, and volume, respectively. This row vector means that when the emotion category is "sadness", the pitch is reduced by 0.2 times, the speech rate is reduced by 0.3 times, the pause is increased by 0.4 times, and the volume is reduced by 0.2 times. The row vector [+0.3, +0.2, -0.1, +0.5] corresponds to the emotion category "anger". The speech output parameters from column 1 to column 4 are pitch, speech rate, pause, and volume, respectively. This row vector means that when the emotion is "anger", the pitch is increased by 0.3 times, the speech rate is increased by 0.2 times, the pause is reduced by 0.1 times, and the volume is increased by 0.3 times.

[0069] In some embodiments, the emotion influence matrix differs for different users. In some embodiments, the emotion influence matrix for different users can be determined based on the user's historical data with the virtual digital human.

[0070] Parameter adjustment refers to the amount of adjustment of the speech output parameters. In some embodiments, the parameter adjustment can be represented by a vector consisting of the adjustment amounts of each numerical parameter. The adjustment amount can be an adjustment factor (including an increase factor and a decrease factor, represented by positive and negative values, respectively).

[0071] In some embodiments, for each emotion in an emotional state, the processor can multiply the probability corresponding to the emotion category in the emotional state with the row vector corresponding to the emotion category in the emotion influence matrix, and record the resulting vector as the influence vector of the emotion; the influence vectors corresponding to each emotion are summed column by column to obtain the parameter adjustment amount.

[0072] For example, the emotion influence matrix consists of row vectors [-0.2, -0.3, +0.4, -0,2] and [+0.3, +0.2, -0.1, +0.5], with the emotions corresponding to the two row vectors being [sadness, anger] and the emotional state being [sadness 0.7, anger 0.3]. Then the parameter adjustment is 0.7×[-0.2,-0.3, +0.4,-0.2]+0.3×[+0.3, +0.2,-0.1, +0.5]=[-0.05,-0.15, +0.25, +0.01], that is, the parameter adjustment is [pitch -0.05, speech rate -0.15, pause +0.25, volume +0.01].

[0073] Step 340: Based on the basic output parameters and parameter adjustment amounts of the virtual digital human, generate the real-time output parameters of the virtual digital human.

[0074] For more information on virtual digital humans, see [link to relevant content]. Figure 1 And its related descriptions.

[0075] Basic output parameters refer to the default speech output parameters recorded in the user's personalized profile. For example, basic output parameters may include baseline pitch, baseline speech rate, baseline pause, baseline volume, default timbre, and default dialect accent. In some embodiments, basic output parameters can be represented by a vector composed of the various basic output parameters. For example, the basic output parameters are [Pitch 1.0, Speech Rate 1.0, Pause 1.0, Volume 1.0, Timbre A, Dialect Accent B]. Pitch 1.0, Speech Rate 1.0, Pause 1.0, and Volume 1.0 represent the baseline pitch, baseline speech rate, baseline pause, and baseline volume, respectively. Timbre A is the default timbre, and Dialect Accent B is the default dialect accent. In some embodiments, the baseline pitch, baseline speech rate, baseline pause, baseline volume, default timbre, and default dialect accent can be preset by the user or set by the system default.

[0076] In some embodiments, the processor can construct a user preference model based on the user's historical behavior data; determine the user's voice preference features based on the preference model; and generate basic output parameters based on the voice preference features.

[0077] In some embodiments, the processor can also determine the user's dialect accent based on a preference model and adjust the basic output parameters based on the dialect accent.

[0078] For more information on generating and adjusting basic output parameters, see [link to relevant documentation]. Figure 4 And its related descriptions.

[0079] Real-time output parameters refer to the voice output parameters used by the virtual digital human for the current voice playback.

[0080] In some embodiments, the processor can adjust the numerical parameters in the basic output parameters according to the corresponding parameter adjustment amounts based on the basic output parameters and parameter adjustment amounts of the virtual digital human, while keeping the categorical parameters unchanged, thus forming the real-time output parameters. For example, if the basic output parameters are [pitch 1.0, speech rate 1.0, pause 1.0, volume 1.0, timbre A, dialect accent B], and the parameter adjustment amounts are [pitch -0.05, speech rate -0.15, pause +0.25, volume +0.01], then the real-time output parameters are [pitch 1.0 -0.05, speech rate 1.0 -0.15, pause 1.0 +0.25, volume 1.0 +0.01, timbre A, dialect accent B], that is, the real-time output parameters are [pitch 0.95, speech rate 0.85, pause 1.25, volume 1.01, timbre A, dialect accent B]. [Dialect Accent B] indicates that the real-time output parameters are adjusted as follows: tone to 0.95 times the baseline tone, speech rate to 0.85 times the baseline speech rate, pauses to 1.25 times the baseline pauses, volume to 1.01 times the baseline volume, timbre and dialect accent to remain at their default values. The actual values ​​corresponding to the basic output parameters can be preset by the user or set by the system default. For example, volume 1.0 represents 50 dB, speech rate 1.0 represents 200 words per minute, etc.

[0081] Step 350: Play voice messages to the user based on real-time output parameters.

[0082] In some embodiments, the processor inputs preset broadcast content and real-time output parameters into the speech synthesis engine to generate speech with emotional coloring and personalized features, which is then played to the user through a playback device. The preset broadcast content refers to standard text characters that are manually pre-set.

[0083] A speech synthesis engine is software that converts standard text characters into audible speech data. Examples include CoquiTTS and HTS.

[0084] In some embodiments of this specification, multimodal features are extracted from the user's multimodal data, and the user's emotional state is identified based on the multimodal features. The parameter adjustment amount is determined by combining the user's emotional state and the emotional influence matrix. Then, the basic output parameters of the virtual digital human are dynamically adjusted to generate corresponding real-time output parameters and play the voice. This realizes the adaptive change of the virtual digital human's voice output parameters with the user's emotional state, thereby improving the matching degree between the voice output and the user's current emotional state and enhancing the naturalness and immersion in the human-computer interaction process.

[0085] Since the generated real-time output parameters may exceed the user's personalized preferences or acceptable range, further constraint processing of the real-time output parameters is required before voice playback to avoid causing discomfort to the user or violating the user's long-term usage habits. Therefore, in some embodiments, before playing voice to the user, the processor may adjust the user-sensitive items in the real-time output parameters to the user-preferred range if the user-sensitive items are not within the user's preferred range.

[0086] User-sensitive parameters refer to voice output parameters that users are sensitive to changes in. For example, user-sensitive parameters can be any one or more of various voice output parameters (pitch, speech rate, pauses, volume, timbre, dialect accent, etc.).

[0087] In some embodiments, the processor can determine user-sensitive items by statistically analyzing the user's historical data. For example, if the proportion of times the user adjusts the volume in multiple historical interactions with the virtual digital human exceeds a preset threshold, it indicates that the user is sensitive to the voice output parameter "volume," meaning that the user-sensitive item is volume, the voice output parameter.

[0088] User preference range refers to the range of commonly used settings for voice output parameters, reflecting the user's acceptable range for these parameters. Different voice output parameters can have corresponding user preference ranges. For example, when volume 1.0 represents 50 dB, the corresponding user preference range for volume can be [0.8, 1.4], meaning the user's acceptable volume range is 40 dB-70 dB; when speech rate 1.0 represents 200 words / minute, the corresponding user preference range for speech rate can be [0.6, 1.5], meaning the user's acceptable speech rate range is 120-300 words / minute.

[0089] In some embodiments, the processor can determine the range of user preferences using historical data or ranges manually entered by the user. For example, the processor can extract the numerical values ​​of the volume that the user manually adjusted each time over the past 30 days, and determine the user preference range corresponding to the volume by calculating the average of these values ​​or the range with the highest frequency. As another example, for the user preference range corresponding to speech rate, the processor can statistically analyze the playback speed selected by the user when watching or listening to content over a long period.

[0090] In some embodiments, in response to a user-sensitive item in the real-time output parameters not being within the user's preferred range, the user-sensitive item is adjusted to the user's preferred range. For example, if a user's user-sensitive item is volume, and their user preference range is set to [0.8, 1.4], when the volume in the real-time output parameters generated by the processor is 1.6, it will be automatically adjusted to 1.4 to ensure that the real-time output parameters conform to the user's long-established listening habits and comfort range.

[0091] In some embodiments of this specification, by detecting user-sensitive items before playing voice messages to the user and determining whether they are within the user's preference range, and if not, adjusting them to the user's preference range, it is possible to avoid significant deviations between the voice output results and the user's long-term preferences, thereby improving the comfort of the interaction process and user satisfaction.

[0092] Since the external environment and current interactive scenario of the user are dynamic, in order to make the voice output of the virtual digital human more in line with the actual use situation, it is necessary to adjust the real-time output parameters for scenario adaptation before the voice is played. Therefore, in some embodiments, the processor can adjust the real-time output parameters based on the external environment data and the interactive scenario before playing the voice to the user.

[0093] External environment data refers to data related to the objective environment in which a user interacts with a virtual digital human. For example, external environment data may include the current time period, weather conditions, etc. In some embodiments, the processor may obtain external environment data through network services (such as weather APIs, time servers) or based on third-party platforms (such as clocks, weather apps).

[0094] An interactive scenario refers to the type or theme of interaction between a user and a virtual digital human. For example, an interactive scenario may include casual conversation, news broadcasting, emotional support, or command execution. In some embodiments, the processor may determine the interactive scenario based on the user's multimodal data to identify intent. Intent identification can be achieved in various ways, such as through natural language processing techniques.

[0095] In some embodiments, the processor can determine the corresponding voice output parameters by querying a first preset table based on external environment data and interactive scenarios, and then adjust the real-time output parameters to the voice output parameters.

[0096] The first preset table includes the correspondence between different external environmental data, interactive scenarios, and voice output parameters. For example, the first preset table may include: when the time period is "night", the weather condition is "rainy", the interactive scenario is "emotional comfort", and the corresponding voice output parameters are: [pitch 0.85, speech rate 0.80, pause 1.30, volume 0.85, timbre A, dialect accent B]; when the time period is "day", the weather condition is "sunny", the interactive scenario is "consultation broadcast", and the corresponding voice output parameters are: [pitch 1.05, speech rate 0.9, pause 0.9, volume 1.0, timbre A, dialect accent B]; when the time period is "evening", the weather condition is "cloudy", the interactive scenario is "casual chat and entertainment", and the corresponding voice output parameters are: [pitch 1.10, speech rate 1.20, pause 0.85, volume 1.10, timbre A, dialect accent B].

[0097] In some embodiments, the processor may construct a first preset table based on historical data. For example, the processor collects historical data including historical external environment data, historical interaction scenarios, and historical voice output parameters, and selects historical data with positive user feedback results as samples to fill into a table to generate the first preset table.

[0098] In some embodiments of this specification, by introducing a perception and adaptation mechanism for the external environment and interactive scenarios before playing voice to the user, the real-time output parameters can be dynamically adjusted according to different situations, making the expression of the virtual digital human closer to the needs of the context, improving the naturalness and realism of voice interaction, and enhancing the user's immersive experience.

[0099] Since users' attention to the voice content, comprehension, or emotional response may change during the interaction, the real-time output parameters need to be adaptively adjusted. Therefore, in some embodiments, when playing voice to the user, the processor can adjust the real-time output parameters based on the user's real-time feedback data.

[0100] Real-time feedback data refers to the immediate information provided by the user during the virtual digital human's speech playback, used to instruct adjustments to the speech output. For example, real-time feedback data could be voice commands issued by the user during speech playback, such as "speak louder," "speak slower," or "change your voice."

[0101] In some embodiments, if the processor receives real-time feedback data from the user during the virtual digital human's speech playback, it can use a large language model to parse the real-time feedback data and adjust the corresponding real-time output parameters accordingly.

[0102] In some embodiments, in response to fuzzy feedback data, after playing voice based on adjusted real-time output parameters: the processor can extract the user's facial expression features based on the user's visual modality data; determine the user's feedback emotional features based on the facial expression features; and roll back the real-time output parameters in response to the feedback emotional features satisfying a negative condition.

[0103] Vague feedback refers to non-instructive feedback signals from users that do not contain explicit adjustment instructions or specific semantic content. For example, vague feedback can be a user's interjection or a period of silence without a clear intention. Vague feedback usually expresses the user's confusion or dissatisfaction but does not specify the problem.

[0104] Visual modal data refers to facial images or video data of users collected through a camera after user authorization.

[0105] Facial features refer to the characteristics that represent a user's facial expressions. For example, the state of the eyes when they are focused or wandering, the position and curvature of the corners of the mouth, and the degree of opening and closing of the eyelids.

[0106] In some embodiments, the processor can use facial expression analysis algorithms (such as OpenFace) to locate facial regions in visual modality data. Based on this, it uses facial landmark detection algorithms (such as Convolutional Pose Machines (CPM) and Practical Facial Landmark Detector (PFLD)) to identify and track positional changes of facial landmarks such as eyes, eyebrows, mouth, and nose. Then, based on the positional changes of the landmarks, and in conjunction with a Facial Action Coding System (FACS), it calculates the activation intensity of each facial action unit (AU) and uses the activation intensity as an expression feature. A facial action unit refers to the smallest action unit in a facial action coding system used to describe the basic movement patterns of facial muscles or muscle groups. Each facial action unit corresponds to a facial muscle contraction or muscle combination action, and the activation intensity is used to characterize the intensity of the action.

[0107] Feedback emotional characteristics refer to qualitative labels representing a user's true emotional state after hearing the output of a virtual digital human. For example, feedback emotional characteristics can include satisfaction, dissatisfaction, and confusion.

[0108] In some embodiments, the processor determines the user's feedback emotional characteristics by querying a second preset table based on facial expression features.

[0109] The second preset table includes the correspondence between different facial expression features and feedback emotional features.

[0110] In some embodiments, the second preset table can be constructed based on historical data containing various user emotions. During the construction process, the processor can collect historical interaction records between the user and the virtual digital human multiple times. Each historical interaction record includes at least the user's historical facial expression features and historical feedback emotional features after hearing the virtual digital human's output speech. The historical facial expression features can be represented by the historical activation intensity of each facial action unit. By statistically analyzing the historical interaction records, different facial expression features and the corresponding feedback emotional features are determined and filled into a table to generate the second preset table. In some embodiments, the historical feedback emotional features can be manually labeled.

[0111] For example, the second preset table may include: when the activation intensity of facial expression features AU01 is greater than or equal to 3.0, the activation intensity of AU04 is greater than or equal to 3.0, the activation intensity of AU15 is greater than or equal to 2.5, and the activation intensity of AU12 is less than or equal to 1.0, the feedback emotional feature is dissatisfaction; when the activation intensity of AU01 is less than or equal to 1.0, the activation intensity of AU04 is less than or equal to 1.0, the activation intensity of AU15 is less than or equal to 1.0, and the activation intensity of AU12 is greater than or equal to 3.0, the feedback emotional feature is satisfaction; when the activation intensity of AU01 is greater than or equal to 2.5, the activation intensity of AU04 is greater than or equal to 2.5, the activation intensity of AU15 is less than or equal to 1.0, and the activation intensity of AU12 is less than or equal to 1.0, the feedback emotional feature is confusion. Here, AU01, AU04, AU12, and AU15 are multiple different facial action units in the facial action coding system, corresponding to inner eyebrow raising, eyebrow lowering, corner of mouth drooping, and smiling, respectively.

[0112] A negative condition refers to a situation where the feedback emotional characteristics indicate that the user is currently in a negative emotional state, and the intensity of the negative emotion exceeds a preset threshold. The preset threshold can be set by someone skilled in the art based on experience. A negative emotional state refers to the user being in a negative emotional category. Negative emotional categories can include dissatisfaction, confusion, disappointment, irritability, resistance, and boredom, etc.

[0113] In some embodiments, the intensity of negative emotion can be a weighted average of the activation intensity of individual facial movement units. For example, the intensity of negative emotion... , where w1~w4 are weighting coefficients, and when S> a preset threshold, the negative condition is satisfied.

[0114] Rollback of real-time output parameters refers to restoring the real-time output parameters, which were adjusted based on real-time user feedback data, to the original voice output parameters before the adjustment.

[0115] In some embodiments of this specification, when the user does not actively provide feedback, the user's emotional state is obtained and judged through visual perception, which effectively makes up for the blind spot in the case of lack of feedback, realizes timely correction of abnormal voice output parameters, and improves the interactive intelligence of the virtual digital human.

[0116] In some embodiments of this specification, user feedback data is received and parsed in real time during voice playback, and voice output parameters are adjusted accordingly. This method allows the voice output to dynamically adapt to the user's immediate needs. For example, the user can request "louder" or "slower" at any time, and the virtual digital person can respond immediately. This effectively avoids the voice output continuously deviating from the user's current preferences, significantly improves the fluency and naturalness of human-computer interaction, and enhances the user's sense of control and satisfaction with the system.

[0117] It should be noted that the above description of process 300 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 300 under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0118] Figure 4 This is a flowchart illustrating the generation and adjustment of basic output parameters according to some embodiments of this specification.

[0119] In some embodiments, such as Figure 4 As shown, the processor can construct a user preference model 420 based on the user's historical behavior data 410; determine the user's voice preference features 430 based on the preference model 420; and generate primary output parameters 440 for the virtual digital human based on the voice preference features 430, which serve as basic output parameters 450.

[0120] For more information on basic output parameters, please refer to [link / reference]. Figure 3 And related descriptions. For more information on virtual digital humans, see [link to relevant documentation / description]. Figure 1 And its related descriptions.

[0121] Historical behavioral data refers to historical data that reflects user habits and preferences. In some embodiments, historical behavioral data may include the user's historical interaction habits, historical preference settings, and historical feedback data on various voice output parameters during historical interactions.

[0122] A preference model is a model used to predict a user's voice preference characteristics. In some embodiments, the preference model can be a probabilistic statistical model to describe the distribution of a user's preferences across various voice output parameters.

[0123] In some embodiments, the preference model includes preference center values ​​and preference acceptance ranges for various speech output parameters.

[0124] In some embodiments, the processor can construct a preference model based on historical behavioral data and statistical analysis algorithms to analyze the user's preference center values ​​and acceptable ranges for various speech output parameters. Statistical analysis algorithms can include Gaussian distribution fitting, mean-variance analysis, or probability distribution estimation. For example, for numerical parameters (including tone, speech rate, pauses, and volume), the preference model can be a multivariate Gaussian distribution model, where each speech output parameter corresponds to one dimension. The elements of the mean vector of the multivariate Gaussian distribution model are used as the preference center values ​​for various speech output parameters, and the diagonal elements of the covariance matrix of the multivariate Gaussian distribution model are used as the acceptable ranges for various speech output parameters. For categorical parameters (including timbre and dialect accent), the preference model can be a probabilistic statistical model. The processor can statistically analyze the most frequently occurring value for each speech output parameter in historical behavioral data, using it as the preference center value for each speech output parameter, and using the two most frequently occurring values ​​as the acceptable ranges for preferences. For example, if the value corresponding to the voice output parameter is "Sichuan timbre" which appears the most, then "Sichuan timbre" is the preferred center value of the timbre. If the value of "Northeast timbre" appears the second most, then "Sichuan timbre" and "Northeast timbre" are the preferred acceptable range of the timbre.

[0125] In some embodiments, such as Figure 4 As shown, the processor can adjust the preference model 420 every preset period based on the user's feedback record 460 and interaction record 470.

[0126] A preset period refers to a predetermined time interval. For example, a preset period can be a day, a week, or a month. In some embodiments, the preset period can be manually preset.

[0127] Feedback logs refer to the feedback information a user provides to the output voice of a virtual digital human during interaction. In some embodiments, feedback logs include explicit and implicit feedback. Explicit feedback refers to evaluations or instructions actively and clearly expressed by the user. For example, explicit feedback may include user input such as "the voice is too soft," "I don't like this tone," or instructions such as "save this setting." Implicit feedback refers to user evaluations or intentions inferred by analyzing the user's behavioral responses. For example, implicit feedback may be the user's immediate behavioral response after listening to the voice, such as immediately interrupting (potentially indicating dissatisfaction) or listening to the entire voice (potentially indicating satisfaction).

[0128] Interaction records refer to objective data used to record the interaction process between a user and a virtual digital human. In some embodiments, an interaction record may include the specific values ​​of the voice output parameters used in a complete interaction session, the interaction time, the number of dialogue turns, etc. Interaction time refers to the duration of interaction between the user and the virtual digital human in a complete interaction session. The number of dialogue turns refers to the number of times information is exchanged between the user and the virtual digital human in a complete interaction session. For example, if the user speaks a sentence and the virtual digital human replies with a sentence, it is counted as one dialogue turn.

[0129] In some embodiments, the processor collects all interaction records from multiple interactive sessions within a preset period and assigns weights to the voice output parameters used in each interactive session based on feedback records within the same preset period. For example, for an interactive record determined to be positive feedback, its corresponding voice output parameter is assigned a first weight; for an interactive record determined to be negative feedback, its corresponding voice output parameter is assigned a second weight. The second weight is lower than the first weight. Finally, based on the assigned first and second weights, the user's preferences for each voice output parameter are recalculated using weighted averages. Then, the elements of the mean vector of the multivariate Gaussian distribution model are used as the preference center values ​​for each voice output parameter, and the diagonal elements of the covariance matrix of the multivariate Gaussian distribution model are used as the acceptable preference ranges for each voice output parameter, thereby updating and obtaining the adjusted preference model.

[0130] In some embodiments, the processor can determine positive and negative feedback in various ways based on feedback records and interaction records. For example, those skilled in the art can preset various positive feedback behaviors (including listening completely, liking, saving settings, etc.) and negative feedback behaviors (including interrupting immediately, giving a bad review, frequently modifying, etc.) based on historical experience. When a user's feedback record meets the positive feedback behavior, it is considered positive feedback; when a user's feedback record meets the negative feedback behavior, it is considered negative feedback.

[0131] In some embodiments of this specification, the processor periodically adjusts the preference model based on user feedback and interaction records, thereby achieving dynamic adaptive optimization of the preference model. This enables the virtual digital human to respond promptly to changes in user voice preferences, continuously optimize personalized voice styles, and enhance user stickiness and satisfaction.

[0132] Voice preference features refer to the voice output parameters preferred by the user. In some embodiments, voice preference features may include user-preferred speech rate, timbre, volume, etc. For example, voice preference features may be: [pitch 1.00, speech rate 1.10, pause 0.90, volume 1.10, timbre A, dialect accent B].

[0133] In some embodiments, for numerical parameters such as speech rate and volume, the processor can extract their preference center values ​​as the corresponding speech preference features. For categorical parameters such as timbre, the processor selects the timbre and dialect accent that appear most frequently in historical behavioral data as the corresponding speech preference features.

[0134] Primary output parameters refer to the speech output parameters initially configured for the virtual digital human. In some embodiments, primary output parameters can be represented by a vector consisting of individual primary output parameters. For example, [pitch 1.0, speech rate 1.1, pause 0.90, volume 1.10, timbre A, dialect accent B].

[0135] In some embodiments, the processor assigns the determined voice preference features to the corresponding values ​​of each initial output parameter. For example, if the determined voice preference features are [pitch 1.05, speech rate 1.10, pause 0.90, volume 1.10, steady male voice, dialect accent B], the processor will set the pitch to 1.05 times the baseline pitch, the speech rate to 1.1 times the baseline speech rate, the pause to 0.90 times the baseline pause, the volume to 1.10 times the baseline volume, the voice to "steady male voice", and the dialect accent B to be maintained in the basic output parameters of the virtual digital human.

[0136] In some embodiments, the processor can directly use the primary output parameters as the basic output parameters.

[0137] In some embodiments of this specification, a preference model is constructed based on the user's historical behavioral data, which can accurately capture the user's voice preferences. Then, the initial output parameters of the virtual digital human are generated based on the voice preference features determined by the preference model. This allows the virtual digital human's default voice style to highly match the user's personal habits and preferences even without explicit emotional input, avoiding generic and rigid initial settings, and improving the stability of the personalized interactive experience and user satisfaction.

[0138] In some embodiments, such as Figure 4 As shown, the processor can determine the user's dialect accent 480 based on the preference model 420; based on the dialect accent 480, adjust the primary output parameters 440, and use the adjusted primary output parameters as the basic output parameters 450.

[0139] Dialect accent refers to the regional speech characteristics that users naturally use during interaction, which differ from standard Mandarin. For example, dialect accents can be divided into various types according to region, such as Sichuan accent and Northeastern accent.

[0140] In some embodiments, the processor can retrieve the most frequently occurring dialect accent type by searching a preference model and identify it as the user's dialect accent.

[0141] In some embodiments, the processor can retrieve the corresponding dialect acoustic model from a preset speech library based on the user's dialect accent; adjust the primary output parameters according to the dialect acoustic model, and use the adjusted primary output parameters as the basic output parameters. The preset speech library refers to a pre-stored database used to store various dialect acoustic models.

[0142] Dialect acoustic models are acoustic models trained and optimized to suit the speech characteristics of different dialect accents. Examples include VITS or Your TTS. A dialect acoustic model includes the tone and range features of the dialect, which can be used to adjust the tone in the primary output parameters; a dialect acoustic model also includes the average pronunciation speed of the dialect, which can be used to adjust the speech rate in the primary output parameters.

[0143] Some embodiments in this specification determine the user's dialect accent based on a preference model and adjust the primary output parameters accordingly, so that the voice played by the virtual digital human has the user's familiar accent characteristics, which goes beyond the limitation of a single standard voice and provides a highly personalized and friendly interactive experience. It significantly enhances the user's emotional connection and personality consistency, and is suitable for scenarios with high requirements for personality consistency and voice affinity, such as long-term companionship and emotional care.

[0144] Figure 5 This is a flowchart illustrating the process of determining emotional state according to some embodiments of this specification.

[0145] In some embodiments, such as Figure 5 As shown, the processor can determine the user's multimodal emotion distribution 530 based on multimodal features 510 and an emotion classification model 520; wherein the emotion classification model is a machine learning model; and determine the emotion state 550 based on the multimodal emotion distribution 530 and multiple emotion weights 540.

[0146] For more information on multimodal features and emotional states, see [link to relevant documentation]. Figure 3 And its related descriptions.

[0147] An emotion classification model is a model used to determine a user's multimodal emotion distribution. In some embodiments, the emotion classification model is a machine learning model. For example, an emotion classification model may include one or more combinations of Deep Neural Network (DNN) models, Convolutional Neural Network (CNN) models, or other custom models.

[0148] Multimodal emotion distribution refers to the probability distribution used to characterize different emotion categories of users in different modalities. In some embodiments, multimodal emotion distribution includes semantic emotion distribution, audio emotion distribution, and facial expression emotion distribution. Semantic emotion distribution refers to the probability distribution of different emotion categories obtained through an emotion classification model based on semantic sentiment vectors; audio emotion distribution refers to the probability distribution of different emotion categories obtained through an emotion classification model based on audio sentiment vectors; and facial expression emotion distribution refers to the probability distribution of different emotion categories obtained through an emotion classification model based on facial expression sentiment vectors.

[0149] For more information on semantic sentiment vectors, audio sentiment vectors, and facial expression sentiment vectors, please see [link to relevant documentation]. Figure 3 And its related descriptions.

[0150] For example, a multimodal emotion distribution may include a semantic emotion distribution [happiness 0.20, anger 0.50, sadness 0.30], an audio emotion distribution [happiness 0.10, anger 0.70, sadness 0.20], and an facial expression emotion distribution [happiness 0.05, anger 0.8, sadness 0.15].

[0151] In some embodiments, the emotion classification model includes a semantic classification layer, an audio classification layer, an facial expression classification layer, and a Softmax activation function.

[0152] Semantic classification layers refer to models used to determine the distribution of semantic sentiment. Examples include CNNs and DNNs.

[0153] In some embodiments, the input to the semantic classification layer can be a semantic sentiment vector, and the output of the semantic classification layer can be a semantic sentiment distribution.

[0154] In some embodiments, the semantic classification layer can be obtained by training an initial semantic classification layer using multiple sets of first samples with a first label. The first training samples may include sample semantic sentiment vectors obtained after processing sample semantic text data using a natural language processing model. The method for determining the sample semantic sentiment vector is the same as the method for determining the semantic sentiment vector; see the figure and related content for more details. The first label may be the true emotion category label corresponding to the sample semantic text data. The first training samples may be obtained based on historical data. The true emotion category label serving as the first label may be manually labeled. In some embodiments, the processor can input multiple first training samples with the first label into the initial semantic classification layer, construct a loss function using the first label and the output of the initial semantic classification layer, and iteratively update the parameters of the initial semantic classification layer based on the loss function using gradient descent or other methods. When preset conditions are met, the model training is complete, and a trained semantic classification layer is obtained. The preset conditions may include loss function convergence, the number of iterations reaching a threshold, etc.

[0155] An audio classification layer refers to a model used to determine the emotional distribution of audio. Examples include CNNs and DNNs.

[0156] In some embodiments, the input to the audio classification layer can be an audio emotion vector, and the output of the semantic classification layer can be an audio emotion distribution.

[0157] In some embodiments, the audio classification layer can be obtained by training the initial audio classification layer with multiple sets of second samples bearing second labels. The second training samples may include sample audio sentiment vectors obtained after signal processing of the sample speech data. The method for determining the sample audio sentiment vectors is the same as the method for determining the audio sentiment vectors; for more details, see [link to documentation]. Figure 3 And related content. The second label can be the true emotion category label corresponding to the sample speech data. The second training sample can be obtained based on historical data. The true emotion category label used as the second label can be manually labeled. In some embodiments, the training process of the audio classification layer is similar to that of the semantic classification layer, and will not be elaborated here.

[0158] An expression classification layer refers to a model used to determine the distribution of emotions in facial expressions. Examples include CNNs and DNNs.

[0159] In some embodiments, the input to the expression classification layer can be an expression emotion vector, and the output of the expression classification layer can be an expression emotion distribution.

[0160] In some embodiments, the expression classification layer can be obtained by training the initial expression classification layer with multiple sets of third samples bearing third labels. The third training samples may include sample expression emotion vectors obtained from sample visual image data using facial motion unit monitoring technology. The third label may be the true emotion category label corresponding to the sample visual image data. The third training samples can be obtained based on historical data. The true emotion category label serving as the third label can be manually annotated. In some embodiments, the training process of the expression classification layer is similar to that of the semantic classification layer, and will not be elaborated here.

[0161] In some embodiments, the processor processes the semantic emotion vector, audio emotion vector, and facial emotion vector based on the semantic classification layer, audio classification layer, and facial expression classification layer of the emotion classification model, respectively. Then, based on the Softmax activation function, it outputs the semantic emotion distribution of the semantic emotion vector in the semantic modality, the audio emotion distribution of the audio emotion vector in the audio modality, and the audio emotion distribution of the facial emotion vector in the facial expression modality, representing the probability values ​​corresponding to each emotion category in different modalities, so as to determine the multimodal emotion distribution.

[0162] Emotion weights are numerical values ​​that characterize the importance of emotion categories from different modalities in determining the final emotional state. In some embodiments, multiple emotion weights may include semantic weights, audio weights, and facial expression weights, which are used to represent the importance of the semantic modality, audio modality, and facial expression modality in emotion recognition, respectively. In some embodiments, the values ​​of the multiple emotion weights may be set by those skilled in the art based on experience.

[0163] In some embodiments, the processor multiplies the semantic emotion distribution, audio emotion distribution, and facial expression emotion distribution output by the emotion classification model by their corresponding semantic weights, audio weights, and facial expression weights, respectively. Then, it sums the probabilities of the same emotion category in the emotion distributions of each modality to obtain the emotion state. For example, the semantic emotion distribution [happy 0.20, angry 0.50, sad 0.30], the audio emotion distribution [happy 0.10, angry 0.70, sad 0.20], and the facial expression emotion distribution [happy 0.05, angry 0.80, sad 0.15], with semantic weights, audio weights, and facial expression weights of 0.2, 0.3, and 0.5 respectively, yields the emotion state [happy 0.095, angry 0.710, sad 0.195] after weighted calculation.

[0164] In some embodiments, such as Figure 5 As shown, the processor can determine the user's emotional conflict level 560 based on the multimodal emotion distribution 530; in response to the emotional conflict level 560 being greater than the conflict threshold 570, the processor can determine the emotional state 550 based on the user's historical multimodal features 580 over a historical period and through the emotion recognition model 590, wherein the emotion recognition model is a machine learning model.

[0165] Emotional conflict level refers to a numerical indicator used to characterize the degree of inconsistency in sentiment judgment results across the three modalities of semantic, audio, and facial expressions. It reflects the degree of inconsistency between semantic sentiment distribution, audio sentiment distribution, and facial sentiment distribution. Sentiment judgment result refers to the probability value of each emotion category under different modalities.

[0166] In some embodiments, for each emotion category in the multimodal emotion distribution, the processor calculates the variance of the probability values ​​of the three modalities (semantics, audio, and facial expressions) for that emotion category; then, it sums or averages the variances of all emotion categories to obtain the degree of emotional conflict.

[0167] For example, for the emotion category "happiness," the variances of the probability values ​​for "happiness" (0.2) in the semantic emotion distribution, "happiness" (0.1) in the audio emotion distribution, and "happiness" (0.05) in the facial expression emotion distribution are calculated. Similarly, the variances of the corresponding probability values ​​for "anger" and "sadness" in the semantic, audio, and facial expression emotion distributions are calculated. Then, the variances calculated for all emotion categories are summed, and the total sum represents the user's degree of emotional conflict. In some embodiments, a higher degree of emotional conflict indicates a more dispersed emotional judgment across different modalities.

[0168] A conflict threshold is a preset value used to determine whether the level of emotional conflict has reached a critical point requiring attention, intervention, or triggering subsequent procedures. In some embodiments, the conflict threshold can be a value set based on experience, such as 0.5.

[0169] Historical time periods refer to the time periods preceding the current moment.

[0170] In some embodiments, the processor can determine historical time periods in various ways. For example, the processor can define a complete interaction session's time interval as a historical time period based on the start and end times of each historical interaction session. Alternatively, the processor can dynamically extract historical interaction sessions as historical time periods based on a preset time window (e.g., 5 minutes).

[0171] In some embodiments, the length of the historical period is related to the difference between the degree of emotional conflict and the conflict threshold. The length of the historical period refers to the length of the time series samples input into the emotion recognition model. A time series sample is a sequence of multimodal features from multiple historical moments arranged in chronological order.

[0172] In some embodiments, the length of the historical period is positively correlated with the difference between the degree of emotional conflict and the conflict threshold. That is, the greater the difference between the degree of emotional conflict and the conflict threshold, the longer the historical period, in order to provide more information to assist in the judgment.

[0173] In some embodiments of this specification, the processor dynamically adjusts the length of the historical period based on the difference between the degree of emotional conflict and the conflict threshold, which can provide more comprehensive contextual support when the expression of conflicting emotions is stronger, improve the ability to identify complex emotional states, and avoid judgment errors caused by input segments that are too short.

[0174] Historical multimodal features refer to multimodal features from historical time periods, including semantic, audio, and facial expression features, which are represented by historical semantic sentiment vectors, historical audio sentiment vectors, and historical facial expression sentiment vectors from the historical time periods, respectively. The method for determining historical multimodal features is similar to the method for extracting user multimodal features from user multimodal data, and will not be elaborated here.

[0175] An emotion recognition model is a model used to determine a user's emotional state. In some embodiments, the emotion recognition model is a machine learning model. For example, an emotion recognition model may include one or more combinations of Long Short-Term Memory (LSTM) networks, Deep Neural Network (DNN) models, or other custom models.

[0176] In some embodiments, the input to the emotion recognition model is the user's historical multimodal features over a historical period.

[0177] In some embodiments, the output of the emotion recognition model is the user's current emotional state.

[0178] In some embodiments, the emotion recognition model can be determined using multiple sets of fourth training samples with a fourth label. The fourth training sample can be a multimodal feature of a sample time period containing modal conflict segments extracted from historical data. The fourth label can be the true emotion category label corresponding to the end of the sample time period. A modal conflict segment refers to a data segment that is manually determined to have inconsistent emotion categories across different modalities during manual annotation. For example, a data segment whose emotion category is happiness in the semantic modality and irony in the facial expression modality is a "modal conflict segment". The determination method for sample multimodal features is similar to that for historical multimodal features and will not be elaborated here.

[0179] The training process for the emotion recognition model is similar to that for the semantic classification layer; see the relevant description above for details.

[0180] In some embodiments, when training an emotion recognition model, the loss weight of composite training samples is greater than the loss weight of single training samples. Composite training samples are training samples corresponding to composite emotion labels, while single training samples are training samples corresponding to single emotion labels. For more information on single and composite emotions, see [link to relevant documentation]. Figure 3 And its related descriptions.

[0181] Composite training samples refer to fourth training samples with a fourth label of composite emotion.

[0182] A single training sample refers to a fourth training sample whose fourth label is a single emotion.

[0183] Emotion labels refer to the true emotion category labels given when manually labeling the fourth training sample.

[0184] In some embodiments of this specification, by assigning higher loss weights to composite emotion samples, the emotion recognition model can focus more on complex and atypical emotional expression scenarios, thereby improving the recognition accuracy of virtual digital persons in composite emotional states such as irony and forced smiles.

[0185] In some embodiments of this specification, by introducing historical multimodal features from historical periods when there are conflicts in the multimodal emotion distribution, it is possible to identify complex emotional states such as irony and concealment, which significantly improves the robustness and accuracy of numerical virtual humans in scenarios with inconsistent emotional expression.

[0186] In some embodiments of this specification, by modeling and weighted fusion of multimodal emotion features, it is possible to more accurately identify the user's emotional state when there are differences in multi-source information, thereby improving the stability and reliability of virtual digital human voice adjustment.

[0187] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0188] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0189] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0190] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0191] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0192] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0193] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for autonomous speech adjustment, characterized in that, The method includes: Based on the user's multimodal data, extract the user's multimodal features; Based on the multimodal features, the user's emotional state is identified; Based on the emotional state and the user's emotional influence matrix, the parameter adjustment amount is determined; Based on the basic output parameters of the virtual digital human and the parameter adjustment amount, the real-time output parameters of the virtual digital human are generated. Based on the real-time output parameters, the voice is played to the user.

2. The method according to claim 1, characterized in that, The method further includes: Based on the user's historical behavior data, construct the user's preference model; Based on the preference model, the user's voice preference characteristics are determined; Based on the aforementioned voice preference features, the initial output parameters of the virtual digital human are generated, which serve as the basic output parameters.

3. The method according to claim 2, characterized in that, The method further includes: The preference model is adjusted every preset period based on the user's feedback and interaction records.

4. The method according to claim 2, characterized in that, The method further includes: Based on the preference model, the user's dialect accent is determined; Based on the dialect accent, the primary output parameters are adjusted, and the adjusted primary output parameters are used as the basic output parameters.

5. The method according to claim 1, characterized in that, The method further includes: When playing voice messages to the user, the real-time output parameters are adjusted based on the user's real-time feedback data.

6. The method according to claim 1, characterized in that, The method further includes: Before playing the voice message to the user, the real-time output parameters are adjusted based on external environmental data and the interaction scenario.

7. A voice autonomous adjustment system, characterized in that, The system includes; The emotion perception module is configured as follows: Based on the user's multimodal data, extract the user's multimodal features; Based on the multimodal features, the user's emotional state is identified; The parameter adjustment module is configured as follows: Based on the emotional state and the user's emotional influence matrix, the parameter adjustment amount is determined; Based on the basic output parameters of the virtual digital human and the parameter adjustment amount, the real-time output parameters of the virtual digital human are generated. The voice output module is configured as follows: Based on the real-time output parameters, the voice is played to the user.

8. The system according to claim 7, characterized in that, The parameter adjustment module is further configured to: Based on the user's historical behavior data, construct the user's preference model; Based on the preference model, the user's voice preference characteristics are determined; Based on the aforementioned voice preference features, the initial output parameters of the virtual digital human are generated, which serve as the basic output parameters.

9. The system according to claim 7, characterized in that, The parameter adjustment module is further configured to: When playing voice messages to the user, the real-time output parameters are adjusted based on the user's real-time feedback data.

10. The system according to claim 7, characterized in that, The parameter adjustment module is further configured to: Before playing the voice message to the user, the real-time output parameters are adjusted based on external environmental data and the interaction scenario.

Images