A method and system for non-semantic paralinguistic synthesis based on affective drive

Abstract

The embodiment of the specification provides a non-semantic phonetic synthesis method and system based on emotion driving. The method comprises the following steps: receiving input instructions comprising an emotion type, an emotion intensity and a pronunciation duration; determining an acoustic parameter based on the emotion type and the emotion intensity; determining a pseudo-prosody curve based on the emotion type and the pronunciation duration; and determining a non-semantic audio waveform based on the acoustic parameter and the pseudo-prosody curve.

Description

Technical Field

[0001] This specification relates to the field of emotional expression technology for smart devices, and in particular to an emotion-driven non-semantic onomatopoeia synthesis method and system. Background Technology

[0002] In human-computer interaction scenarios of smart devices (such as smart speakers and robots), smart devices mainly rely on the playback of fixed materials (such as playing pre-recorded audio files) when expressing emotions. However, for diverse scenarios, to achieve voices conveying a variety of emotions, a massive amount of audio files must be pre-stored, resulting in high storage pressure; moreover, the audio is highly repetitive and lacks flexibility.

[0003] Therefore, there is a need to provide an emotion-driven non-semantic onomatopoeia synthesis method and system that can simulate non-semantic onomatopoeia with human emotional characteristics through instructions, taking into account both expressive richness and real-time synthesis capabilities of the terminal, while reducing audio pre-storage and transmission overhead. Summary of the Invention

[0004] One embodiment of this specification provides an emotion-driven non-semantic onomatopoeia synthesis method, comprising: receiving an input instruction, the input instruction including an emotion type, an emotion intensity, and a pronunciation duration; determining acoustic parameters based on the emotion type and the emotion intensity; determining a pseudo-intonation curve based on the emotion type and the pronunciation duration; and determining a non-semantic audio waveform based on the acoustic parameters and the pseudo-intonation curve.

[0005] One embodiment of this specification provides an emotion-driven non-semantic onomatopoeia synthesis system, comprising: a receiving module configured to receive input instructions, the input instructions including emotion type, emotion intensity, and pronunciation duration; a parameter determination module configured to determine acoustic parameters based on the emotion type and the emotion intensity; a intonation determination module configured to determine a pseudo-intonation curve based on the emotion type and the pronunciation duration; and an audio determination module configured to determine a non-semantic audio waveform based on the acoustic parameters and the pseudo-intonation curve. Attached Figure Description

[0006] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0007] Figure 1 This is a schematic diagram illustrating application scenarios of an emotion-driven non-semantic onomatopoeia synthesis method according to some embodiments of this specification; Figure 2 This is an exemplary block diagram of an emotion-driven non-semantic onomatopoeia synthesis system according to some embodiments of this specification; Figure 3 This is an exemplary flowchart of an emotion-driven non-semantic onomatopoeia synthesis method according to some embodiments of this specification; Figure 4 This is an exemplary flowchart illustrating the determination of acoustic parameters according to some embodiments of this specification; Figure 5 This is an exemplary schematic diagram illustrating the determination of output parameters according to some embodiments of this specification. Detailed Implementation

[0008] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0009] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0010] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0011] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0012] Figure 1 This is a schematic diagram illustrating application scenarios of the emotion-driven non-semantic onomatopoeia synthesis method according to some embodiments of this specification. The emotion-driven non-semantic onomatopoeia synthesis method provided in the embodiments of this specification can be applied to, for example... Figure 1 The application scenario shown is 100.

[0013] In some embodiments, such as Figure 1As shown, application scenario 100 may include smart device 110, user 120, processor 130, storage device 140, and network 150.

[0014] In some embodiments, one or more components of application scenario 100 may transmit data to other components of application scenario 100 via network 150. For example, processor 130 may obtain relevant information and / or data from smart device 110 and storage device 140 via network 150, or may send information and / or data to smart device 110 and storage device 140 via network 150.

[0015] In some embodiments, the smart device 110 can receive input commands sent by the user 120 and perform non-semantic onomatopoeia synthesis, that is, synthesize simulated speech with emotion or feeling. For example, the smart device 110 can be at least one of a smart speaker, a robot, etc. The smart device includes a data interface, such as local process communication, UART serial port, or Socket network interface. The smart device 110 can receive input commands uploaded by the user through the data interface.

[0016] User 120 is a user of smart device 110. User 120 can directly input input commands containing emotion type, emotion intensity, and pronunciation duration through the display screen or operation screen of smart device 110; in addition, input commands can be uploaded through user terminals (such as mobile phones, laptops, desktop computers, etc.).

[0017] In some embodiments, the smart device 110 sends input instructions to the processor 130 via the network 150 for processing. In some embodiments, the smart device 110 obtains non-semantic audio waveforms from the processor 130 and performs onomatopoeic synthesis.

[0018] In some embodiments, the smart device 110 includes at least one of a sound rendering component, a digital oscillator, an envelope generator (ADSR), a digital filter, a digital-to-analog converter (DAC), and a power amplifier driving a speaker. The sound rendering component receives acoustic parameters and a pseudo-pitch curve to render the simulated sound to be synthesized. The sound rendering component may include at least one of a built-in DSP, an audio codec chip, a microcontroller, etc. The digital oscillator generates a reference waveform for the simulated sound. The envelope generator controls the amplitude of the simulated sound. The digital filter shapes the frequency band of the simulated sound. The digital-to-analog converter (DAC) and the power amplifier drive the speaker to emit the simulated sound.

[0019] The above description of the smart device 110 is for illustrative purposes only and is not intended to limit the scope of this specification.

[0020] The processor 130 can be used to process data from one or more components or external data sources in application scenario 100. For example, the processor 130 can be used to receive data and information from the smart device 110. As another example, the processor 130 can be used to send control commands to the smart device 110.

[0021] In some embodiments, processor 130 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 130 may be local or remote. In some embodiments, processor 130 may include a central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set computer device (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal computer device (DSP), computer device, microcomputer device unit, reduced instruction set computer (RISC), microcomputer device, etc., or any combination thereof. In some embodiments, processor 130 may be local or remote. In some embodiments, processor 130 may be integrated into the console of smart device 110. In some embodiments, processor 130 may be implemented on a cloud platform. By way of example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-layered cloud, etc., or any combination thereof.

[0022] Storage device 140 is used to store data, instructions, and / or any other information. In some embodiments, storage device 140 may store data and / or information (e.g., input instructions, non-semantic audio waveforms, etc.) sent or received by smart device 110 and processor 130. In some embodiments, storage device 140 may store data and / or instructions used by processor 130 to perform or use in order to complete the exemplary methods described herein. In some embodiments, storage device 140 may include mass storage, removable memory, etc., or any combination thereof. In some embodiments, storage device 140 may be integrated into processor 130.

[0023] Network 150 can be any suitable network capable of facilitating information and / or data exchange. In some embodiments, network 150 can be any one or more of a wired network or a wireless network. Network 150 may include one or more network access points. Through these network access points, one or more components of application scenario 100 can connect to network 150 to exchange data and / or information. In some embodiments, network 150 can be any one or more of a wired network or a wireless network. In some embodiments, network 150 can be various topologies or combinations of multiple topologies, such as point-to-point, shared, centralized, etc.

[0024] Figure 2This is an exemplary block diagram of an emotion-driven, non-semantic onomatopoeia synthesis system according to some embodiments of this specification. Figure 2 As shown, the emotion-driven non-semantic onomatopoeia synthesis system 200 may include: a receiving module 210, a parameter determination module 220, a intonation determination module 230, and an audio determination module 240.

[0025] In some embodiments, the receiving module 210 is configured to receive input instructions, including emotion type, emotion intensity, and pronunciation duration. The receiving module 210 can receive input instructions via the data interface of the smart device 110.

[0026] In some embodiments, the parameter determination module 220 can be configured to determine acoustic parameters based on emotion type and emotion intensity.

[0027] In some embodiments, the parameter determination module 220 is further configured to: determine emotional features based on the emotional type and the emotional intensity; determine basic acoustic parameters based on the emotional features; determine feedback signal values ​​based on user interaction behavior; determine correction parameters based on the feedback signal values; and determine the acoustic parameters based on the basic acoustic parameters and the correction parameters.

[0028] In some embodiments, the input instruction includes a mixed input instruction, which contains multiple emotion types and corresponding weight information. The parameter determination module 220 is further configured to: generate the composite acoustic parameters based on the basic acoustic parameters corresponding to the multiple emotion types and the corresponding weight information; and determine the acoustic parameters based on the composite acoustic parameters and the correction parameters.

[0029] In some embodiments, the intonation determination module 230 is configured to determine a pseudo-intonation curve based on the emotion type and the pronunciation duration.

[0030] In some embodiments, the audio determination module 240 is configured to determine a non-semantic audio waveform based on the acoustic parameters and the pseudo-pitch curve.

[0031] In some embodiments, the audio determination module 240 is further configured to: acquire background noise information; and adjust the output parameters of the non-semantic audio waveform based on the background noise information.

[0032] In some embodiments, the output parameters include filtering parameters, and the audio determination module 240 is further configured to: determine the frequency domain distribution characteristics of the background noise information; and adjust the filtering parameters according to the frequency domain distribution characteristics.

[0033] In some embodiments, the receiving module 210, the parameter determining module 220, the tone determining module 230, and the audio determining module 240 may be integrated onto the processor 130.

[0034] For more information on the emotion-driven non-semantic onomatopoeia synthesis system described above, please refer to other parts of this manual (such as...). Figure 1 The relevant description of ).

[0035] Figure 3 This is an exemplary flowchart of an emotion-driven non-semantic onomatopoeia synthesis method according to some embodiments of this specification. Flow 300 is an exemplary flowchart of an emotion-driven non-semantic onomatopoeia synthesis method. In some embodiments, such as Figure 3 As shown, process 300 includes steps 310-340. Steps 310-340 are as follows.

[0036] Step 310: Receive input instructions.

[0037] Input instructions refer to instruction data used to describe the user's needs for the onomatopoeic synthesis of the smart device 110. In some embodiments, input instructions include emotion type, emotion intensity, and pronunciation duration.

[0038] Emotion type refers to a category label used to describe the onomatopoeic emotional tendencies synthesized by the smart device 110. For example, emotion types may include happiness, sadness, sympathy, curiosity, or affirmation.

[0039] Emotional intensity refers to numerical information used to quantitatively describe the emotional tension, excitement, or exaggeration of onomatopoeia. For example, emotional intensity can be represented by a normalized value between 0 and 1; the greater the emotional intensity, the larger the value.

[0040] The duration of a sound production refers to the time span from the start to the end of a single onomatopoeic sound production.

[0041] In some embodiments, the receiving module 210 can receive input instructions uploaded or entered by the user through the data interface of the smart device 110; and extract the emotion type field, emotion intensity field and pronunciation duration field from the input instructions based on natural language processing (such as NLP, NLU) and other methods.

[0042] Step 320: Determine the acoustic parameters based on emotion type and emotion intensity.

[0043] Acoustic parameters refer to the parameters used to control the synthesis of sound effects. For example, acoustic parameters may include at least one of the following: reference frequency, waveform type, and noise ratio. The reference frequency is the frequency parameter that determines the overall fluctuation of the sound effect; the higher the reference frequency, the higher the pitch of the sound effect. The waveform type reflects the waveform type of the sound's spectral structure, such as sine wave, square wave, sawtooth wave, etc. The waveform type determines the timbre of the sound effect. The noise ratio refers to the proportion of noise (non-periodic vibration) in the total signal energy of the sound effect signal. A suitable amount of noise can make the sound effect resemble a natural human voice, while excessive noise will make the sound effect sound hoarse or indistinct.

[0044] In some embodiments, the parameter determination module 220 can determine the acoustic parameters by searching an acoustic parameter table based on the emotion type and emotion intensity. The acoustic parameter table contains different combinations of emotion types and emotions and their corresponding acoustic parameters. The acoustic parameter table can be preset based on experience or needs.

[0045] In some embodiments, the parameter determination module 220 can also be configured to: determine emotional features based on emotion type and emotion intensity; determine basic acoustic parameters based on the emotional features; determine feedback signal values ​​based on user interaction behavior; determine correction parameters based on the feedback signal values; and determine acoustic parameters based on the basic acoustic parameters and the correction parameters. For more details, please refer to... Figure 3 And its related descriptions.

[0046] Step 330: Determine the pseudo-intonation curve based on emotion type and pronunciation duration.

[0047] A pseudo-intonation curve is a curve used to characterize and control the trend of pitch and / or volume changes over time in onomatopoeia.

[0048] In some embodiments, the intonation determination module 230 can determine the corresponding function based on the emotion type by searching a preset function library; and determine a pseudo-intonation curve based on the function and the pronunciation duration. The preset function library includes different emotion types and their corresponding functions. The preset function library can be set based on experience. The intonation determination module 230 can use the pronunciation duration as the range of values ​​for the independent variable and substitute it into the corresponding function to generate a pseudo-intonation curve. For example, the emotion type "curiosity" corresponds to a quadratic exponential rise function; the emotion type "affirmation" corresponds to a first-order linear decay function.

[0049] For example, the emotion type "curiosity" corresponds to a quadratic exponential upward function, and the corresponding pseudo-intonation curve is a frequency control curve with a sharp increase in frequency in the latter half of the pronunciation duration. The quadratic exponential upward function can be expressed as formula (1). Formula (1) is shown below: (1) in, Indicates time The instantaneous frequency value of the output, in Hz; The value of is in the range of 0-T, where T represents the duration of pronunciation; The frequency stretching factor is... The steepness coefficient is the upward slope coefficient. and It can be preset based on experience; As the reference frequency, It can be determined using an acoustic parameter table.

[0050] For example, the emotion type "affirmative" corresponds to a first-order linear decay function, and the corresponding pseudo-intonation curve is a frequency control curve that starts with a high pitch and then drops sharply. The first-order linear decay function can be expressed as formula (2). Formula (2) is shown below: (2) in, The pitch drop slope factor. It can be preset based on experience; The starting frequency is the frequency at which the sound begins to beat. This starting frequency can be determined based on the reference frequency in the acoustic parameters, such as directly using the initial reference frequency as the starting frequency. The pitch drop slope factor is a parameter characterizing how quickly the pitch drops.

[0051] In some embodiments, the intonation determination module 230 may use the pronunciation duration as the effective range of the independent variable, and determine the corresponding part of the function within the effective range as a pseudo-intonation curve.

[0052] Step 340: Determine the non-semantic audio waveform based on acoustic parameters and pseudo-intonation curves.

[0053] Non-semantic audio waveforms are digital audio waveforms that can be used to directly play or output onomatopoeic sounds. For example, a non-semantic audio waveform can be a PCM audio waveform. Non-semantic audio waveforms can drive a digital-to-analog converter (DAC) to generate analog electrical signals.

[0054] In some embodiments, the audio determination module 240 can drive a digital oscillator to generate a reference waveform based on the waveform type and reference frequency in the acoustic parameters; control the envelope generator to control the frequency within the pronunciation duration according to the pseudo-pitch curve, thereby controlling the volume of the onomatopoeia; control the digital filter to perform frequency band shaping, output a non-semantic audio waveform, and drive a speaker to emit onomatopoeia via a digital-to-analog converter and a power amplifier.

[0055] In some embodiments, the audio determination module 240 may also select an excitation source and a resonant filter based on acoustic parameters, and then update the excitation frequency and amplitude frame by frame according to the pseudo-pitch curve to output a non-semantic audio waveform.

[0056] In some embodiments, the audio determination module 240 can also acquire background noise information; and adjust the output parameters of the non-semantic audio waveform based on the background noise information. For more details, please refer to... Figure 5 And its related descriptions.

[0057] Some embodiments in this specification decompose emotion into acoustic parameter links and pseudo-intonation links, and then combine them to generate waveforms, thereby improving the emotion recognition and real-time synthesis capabilities of onomatopoeic audio without relying on text semantics, and reducing pre-storage and transmission overhead.

[0058] Figure 4 This is an exemplary flowchart illustrating the determination of acoustic parameters according to some embodiments of this specification. Flow 400 is an exemplary flowchart for determining acoustic parameters. In some embodiments, such as Figure 4 As shown, process 400 includes steps 410-450. Steps 410-450 are as follows.

[0059] Step 410: Determine the emotional characteristics based on the emotional type and emotional intensity.

[0060] Emotional features refer to data expressions that quantify emotional states. Emotional features can be composed of multiple dimensions. For example, an emotional feature can be a two-dimensional feature vector consisting of valence and arousal values. Valence represents pleasure; the higher the valence value, the higher the pleasure. Arousal represents excitement; the higher the arousal value, the higher the excitement.

[0061] In some embodiments, the parameter determination module 220 can determine the corresponding baseline vector based on the emotion type by searching an emotion feature table; the fusion result of the baseline vector and the emotion intensity is used as the emotion feature. The emotion feature table includes different emotion types and corresponding baseline vectors, where the baseline vector is composed of an initial valence value and an initial arousal value. The emotion feature table can be preset based on experience. The product of the initial valence value and the emotion intensity is the valence value, and the product of the initial arousal value and the emotion intensity is the arousal value. The emotion feature can be represented as a vector composed of the valence value and the arousal value. The fusion result of the baseline vector and the emotion intensity can be the product of the baseline vector and the emotion intensity. For example, the parameter determination module 220 searches the emotion feature table and obtains the baseline vector for the emotion type "curiosity" as [valence baseline: 0.2, arousal baseline: 0.4]. Combined with an emotion intensity of 0.8 for scaling calculation, the final emotion feature is obtained as [0.2, 0.4] × 0.8 = [0.16, 0.32].

[0062] Step 420: Determine the basic acoustic parameters based on emotional characteristics.

[0063] Basic acoustic parameters refer to the default control values ​​for acoustic parameters. For example, basic acoustic parameters may include the reference frequency, waveform type, and basic noise ratio of the initial sound.

[0064] In some embodiments, the parameter determination module 220 can determine the basic acoustic parameters based on emotional features using a preset algorithm. The preset algorithm is a mathematical operation relationship between emotional features and basic acoustic parameters, such as a basic mapping matrix. The basic mapping matrix defines the linear operation relationship between the emotional feature vector and the acoustic parameters. For example, the basic mapping matrix can be expressed as formula (3), and the parameter determination module 220 can calculate the reference frequency using formula (3). Formula (3) is shown below: (3) in, This indicates the reference frequency, measured in Hz. , It is a constant greater than 0, which can be set according to needs or experience; Indicates the valence value; This represents the wake-up level value.

[0065] In some embodiments, the parameter determination module 220 can also, based on the emotional feature (a vector composed of valence and arousal values), find the historical emotional feature with the highest similarity to the emotional feature in historical data, and use the historical acoustic parameter corresponding to the historical emotional feature as the basic acoustic parameter. Similarity is negatively correlated with vector distance, which can be cosine distance or Euclidean distance.

[0066] Step 430: Determine the feedback signal value based on user interaction behavior.

[0067] User interaction behavior refers to the feedback operations performed by a user during or after the smart device emits sound. For example, user interaction behavior may include swiping (such as swiping up, down, left, or right), double-tapping, or re-entering on the smart device's operating screen. In some embodiments, the parameter determination module 220 may acquire user interaction behavior through sensors (such as an inertial measurement unit, displacement sensor, etc.) built into the smart device 110.

[0068] Feedback signal values ​​refer to information used to characterize the direction and magnitude of parameter adjustments. For example, feedback signal values ​​can be positive, negative, or zero.

[0069] In some embodiments, the parameter determination module 220 can determine the feedback signal value based on user interaction behavior through a preset feedback channel. The preset feedback channel can be preset according to needs or experience.

[0070] The preset feedback channels can include frequency adjustment channels, time adjustment channels, and noise adjustment channels. The frequency adjustment channel is an algorithmic channel used to adjust the reference frequency. The time adjustment channel is an algorithmic channel used to adjust the onset time. Onset time refers to the time when the non-semantic audio waveform begins to be emitted. The noise adjustment channel is an algorithmic channel used to adjust the noise ratio.

[0071] In some embodiments, in response to a user interaction behavior of swiping up or down on the operation screen, the parameter determination module 220 can determine the feedback signal value of the reference frequency based on the displacement of the swipe up or down. The absolute value of the feedback signal value is positively correlated with the displacement of the swipe up or down. The sign of the feedback signal value is related to the operation direction (e.g., a positive value corresponds to swiping up, and a negative value corresponds to swiping down).

[0072] In some embodiments, in response to a user interaction behavior of swiping left or right on the operation screen, the parameter determination module 220 can determine the feedback signal value of the start time based on the displacement of the left or right swipe; the absolute value of the feedback signal value is positively correlated with the displacement of the left or right swipe; the sign of the feedback signal value is related to the operation direction (e.g., left and right swipes correspond to positive values, and left swipes correspond to negative values).

[0073] In some embodiments, in response to a double-click user interaction, the parameter determination module 220 can determine the feedback signal value of the noise ratio as a negative value (e.g., -1), thereby reducing the noise ratio.

[0074] Step 440: Determine the correction parameters based on the feedback signal value.

[0075] Correction parameters refer to the parameter information used to adjust the basic acoustic parameters. For example, correction parameters may include adjustments to the reference frequency, the attack time, or the noise ratio.

[0076] In some embodiments, the parameter determination module 220 may determine the correction parameters based on a preset feedback channel and the corresponding feedback signal value.

[0077] For example, in the frequency adjustment channel, the parameter determination module 220 can use the product of the feedback signal value, the frequency adjustment gain, the wake-up value corresponding to the basic acoustic parameters, and the learning rate as the adjustment amount of the reference frequency. The frequency adjustment gain refers to the multiple by which the absolute value of the reference frequency is increased, and can be preset based on experience. For more information on the wake-up value, see step 410 and related explanations. The learning rate is positively correlated with the frequency of user interaction actions.

[0078] For example, in the time adjustment channel, the parameter determination module 220 can use the product of the feedback signal value, the time adjustment gain, the wake-up value corresponding to the basic acoustic parameters, and the learning rate as the adjustment amount for the onset time. Here, the time adjustment gain refers to the multiple by which the absolute value of the onset time is increased, and can be preset based on experience.

[0079] For example, in the noise adjustment channel, the parameter determination module 220 can use the product of the feedback signal value, noise adjustment gain, the negative contribution of the valence value, and the learning rate as the adjustment amount for the noise proportion. Here, the noise adjustment gain refers to the multiple by which the noise proportion is adjusted, and can be preset based on experience. The negative contribution of the valence value is negatively correlated with the valence value; for example, for a normalized valence value, the negative contribution can be expressed as the absolute value of the difference between the valence value and 1.

[0080] In some embodiments, the ratio of an element in the correction parameters to the corresponding element in the basic acoustic parameters (such as reference frequency, attack time, noise ratio, etc.) is less than or equal to a preset threshold (e.g., 10%), thereby preventing over-adjustment. When the ratio of an element in the correction parameters to that element in the basic acoustic parameters is greater than the preset threshold, the element is adjusted according to the correction parameters corresponding to the preset threshold.

[0081] In some embodiments, the parameter determination module 220 can determine the average feedback value corresponding to the input command based on multiple historical feedback signal values ​​corresponding to the input command; and determine the correction parameter based on the average feedback value.

[0082] Historical feedback signal values ​​refer to the historical values ​​of feedback signals corresponding to the same input command during multiple historical interactions. In some embodiments, during the operation of the smart device 110, after each interaction is completed, the parameter determination module 220 can write the input command and the corresponding feedback signal value into the buffer storage area in the storage device 140 without triggering the update of the correction parameters.

[0083] The average feedback value is the average of multiple historical feedback signal values, used to characterize the overall feedback tendency for the same input command. For example, the average feedback value can be the arithmetic mean of multiple historical feedback signal values.

[0084] In some embodiments, the parameter determination module 220 may, in response to the number of historical feedback signal values ​​corresponding to the input instruction being greater than or equal to a counting threshold (e.g., 3), use the arithmetic mean of multiple historical feedback signal values ​​as the average feedback value. In some embodiments, the parameter determination module 220 may, in response to the number of historical feedback signal values ​​being less than the counting threshold, determine the correction parameter based on the feedback signal value using the method described in step 440.

[0085] In some embodiments, the parameter determination module 220 can also directly obtain all historical feedback signal values ​​corresponding to the input command, sum all historical feedback signal values ​​and divide by the number of samples to obtain the average feedback value.

[0086] In some embodiments, the parameter determination module 220 can determine a preset feedback channel based on user interaction behaviors corresponding to multiple historical feedback signal values; and determine correction parameters for the basic acoustic parameters corresponding to the preset feedback channel based on the preset feedback channel and the average feedback value. The method for determining the correction parameters is described in step 440 and related instructions above, except that the feedback signal values ​​are replaced with the average feedback value.

[0087] Some embodiments in this specification, by aggregating similar historical feedback and updating correction parameters based on average feedback values, can filter out abnormal fluctuations caused by occasional accidental touches, reduce parameter drift, and improve the stability and long-term reliability of personalized corrections.

[0088] Step 450: Determine the acoustic parameters based on the basic acoustic parameters and the correction parameters.

[0089] In some embodiments, the parameter determination module 220 can directly modify the basic acoustic parameters based on the modification parameters to obtain the acoustic parameters. For example, the parameter determination module 220 can use the vector sum of the modification parameters and the basic acoustic parameters as the final acoustic parameters.

[0090] In some embodiments, the parameter determination module 220 can also retrieve historical correction parameters corresponding to historical input commands that are identical to the current input command from historical data; the vector sum of the historical correction parameters and the basic acoustic parameters is used as the final acoustic parameters. The above method can be based on existing data for correction, thereby reducing the need for repeated feedback processes.

[0091] In some embodiments, the input instruction includes a mixed input instruction, which contains multiple emotion types and corresponding weight information. The parameter determination module 220 can also generate composite acoustic parameters based on the basic acoustic parameters corresponding to the multiple emotion types and the corresponding weight information; and determine the acoustic parameters based on the composite acoustic parameters and correction parameters.

[0092] Mixed input instructions are composite instructions that simultaneously contain multiple requirements in a single instruction.

[0093] Weighting information refers to numerical information used to characterize the proportion of each emotion type in mixed input commands.

[0094] In some embodiments, the parameter determination module 220 can obtain the initial weight uploaded by the user and normalize the initial weight to obtain weight information for multiple emotion types.

[0095] Composite acoustic parameters refer to the comprehensive acoustic parameters formed by combining acoustic parameters corresponding to multiple emotion types in an acoustic dimension, and are used to characterize the acoustic performance of composite emotions.

[0096] In some embodiments, for each emotion type, the parameter determination module 220 can determine the emotion characteristics based on the emotion type and the corresponding emotion intensity; and determine the basic acoustic parameters corresponding to the emotion type based on the emotion characteristics. For the specific method of determining the basic acoustic parameters, please refer to steps 410-420 and related explanations.

[0097] In some embodiments, the parameter determination module 220 can perform a weighted summation of the basic acoustic parameters corresponding to multiple emotion types based on the weight information corresponding to multiple emotion types to obtain composite acoustic parameters.

[0098] In some embodiments, the composite acoustic parameters can also be generated in other ways. For example, the parameter determination module 220 can first align the multiple basic acoustic parameters corresponding to each of the multiple emotion types, and then perform linear interpolation dimension by dimension based on the weight information to obtain the composite acoustic parameters.

[0099] In some embodiments, the parameter determination module 220 may use the statistical values ​​(such as offset increments) of the composite acoustic parameters and the correction parameters corresponding to multiple emotion types as acoustic parameters.

[0100] For example, the emotion types include "happiness" and "surprise." The base frequency for "happiness" is 224Hz with a weight of 0.6; the base frequency for "surprise" is 368Hz with a weight of 0.4; the weighted sum gives a composite base frequency parameter of 281.6Hz; the adjustment amount for the base frequency corresponding to "happiness" is... The adjustment amount of the reference frequency corresponding to "surprise" is 2Hz, which is 1.6Hz. The adjustment amounts of the reference frequencies corresponding to "happy" and "surprise" are both superimposed on 281.6Hz to obtain the final reference frequency of 282Hz, which is used as an acoustic parameter and output.

[0101] Some embodiments in this specification, by weighting and mixing multiple emotion types in the physical acoustic dimension and superimposing them with correction parameters to generate acoustic parameters, can express complex or transitional emotions, improve the subtlety of onomatopoeia, and take into account personalized output.

[0102] In some embodiments, when a mixed input instruction is received, the parameter determination module 220 may: determine whether there is a conflict relationship between multiple emotion types; and when there is a conflict relationship between multiple emotion types and the conflict coefficients of multiple emotion types meet preset conditions, perform asymmetric compensation processing on the basic acoustic parameters corresponding to the emotion types with conflict relationships.

[0103] A conflict relationship refers to a relationship in which multiple emotion types are mutually exclusive and opposed in perception and expression. For example, there is a conflict between sadness and happiness.

[0104] In some embodiments, when the conflict coefficient of two emotion types is greater than a conflict coefficient threshold, the parameter determination module 220 can determine that the two emotion types have a conflicting relationship. The conflict coefficient threshold can be preset by technicians based on specific application requirements or experience data.

[0105] The conflict coefficient is a numerical value used to quantify the degree of conflict between different emotion types. In some embodiments, the parameter determination module 220 can determine the conflict coefficient between any two emotion types by searching a conflict coefficient matrix based on multiple emotion types. The conflict coefficient matrix includes different emotion types and their corresponding conflict coefficients. The conflict coefficient matrix can be set as an N×N matrix, where N represents the number of emotion types. The values ​​in the matrix (i.e., the conflict coefficients) are used to quantify the degree of opposition between any two emotion types, and the value range is [0,1]. For example, the conflict coefficient between "happiness" and "sadness" is set to the highest value of 1. The conflict coefficient matrix can be preset based on experience.

[0106] Preset conditions refer to pre-defined conditions used to determine whether to perform asymmetric compensation processing. For example, preset conditions may include a preset number of conflict coefficients greater than a conflict coefficient threshold, and the weights of conflicting emotion types greater than a weight threshold. The weight threshold can be preset based on needs or experience. In some embodiments, the parameter determination module 220 may perform asymmetric compensation processing on the basic acoustic parameters corresponding to at least two emotion types based on the fact that the conflict coefficients of at least two emotion types are greater than the conflict coefficient threshold, and the weights of the at least two emotion types are greater than the weight threshold. When the preset conditions are met, a high conflict relationship is determined, triggering asymmetric compensation processing.

[0107] Asymmetric compensation processing refers to parameter processing methods that employ different compensation strategies for different emotion types, so that the roles of each emotion in complex expression are not entirely equal. For example, different parameter retention or superposition methods can be used for dominant and secondary emotions. In some embodiments, asymmetric compensation processing includes: retaining the core basic acoustic parameters in the basic acoustic parameters of the dominant emotion, and performing restricted superposition of the basic acoustic parameters of the secondary emotions to determine the composite acoustic parameters.

[0108] Dominant emotion refers to the emotion type with a higher weight (e.g., above a preset weight threshold) among emotion types with conflicting relationships. Secondary emotion refers to the emotion type with a lower weight (e.g., not above a preset weight threshold) among emotion types with conflicting relationships.

[0109] Core fundamental acoustic parameters refer to the basic acoustic parameters that play a major role in determining the characteristics of sound perception. For example, core fundamental acoustic parameters may include a reference frequency. In some embodiments, the core fundamental acoustic parameters corresponding to different emotion types may be different. The core fundamental acoustic parameters corresponding to different emotion types can be determined by consulting a preset core parameter table. The preset core parameter table includes different emotion types and the types of corresponding core fundamental acoustic parameters. The preset core parameter table can be set based on experience. For example, the core fundamental parameters for sadness and depression are the reference frequency, attack time, and decay time; the core fundamental acoustic parameters for anger may be the noise ratio and the waveform type of the digital oscillator. The decay time is the time when the frequency of the simulated sound begins to decrease. The waveform type of the digital oscillator is the type of reference waveform generated by the digital oscillator (such as a sine wave, sawtooth wave, etc.).

[0110] Restricted stacking refers to a parameter stacking method that is carried out within a limited range, intensity, or proportion, in order to avoid excessive stacking that could cause distortion of expression.

[0111] In some embodiments, the parameter determination module 220 can, for at least two dominant emotions with conflicting relationships, retain the core basic acoustic parameters of the dominant emotion with higher weight; for the secondary emotion with lower weight, it can use a built-in limiter (such as a multiplier) to superimpose the product of the acoustic parameters corresponding to the secondary emotion and the limiting coefficient onto the core basic acoustic parameters to obtain composite acoustic parameters. The limiting coefficient can be set empirically, and its value is less than 1.

[0112] Some embodiments in this specification, by retaining the core parameters of the dominant emotion and limiting the intensity of secondary emotion superposition, reduce the mutual cancellation of conflicting emotions and improve the recognizability and impact of complex emotion expression.

[0113] Some embodiments in this specification generate acoustic parameters through the linkage of emotion mapping and interactive feedback, so that the output simultaneously matches emotional expression and user preferences, thereby improving the acceptability, recognition and naturalness of non-semantic onomatopoeia.

[0114] Figure 5 This is an exemplary schematic diagram illustrating the determination of output parameters according to some embodiments of this specification. In some embodiments, such as Figure 5 As shown, the audio determination module 240 is configured to: acquire background noise information; and adjust the output parameters of the non-semantic audio waveform based on the background noise information.

[0115] Background noise information refers to sound information that characterizes the noise interference in the environment in which the smart device is located. For example, background noise information may include the volume of the environment, which can be expressed in decibels. In some embodiments, the audio determination module 240 can acquire a sound wave signal of a preset duration (e.g., 100ms) based on the acquisition device (e.g., a microphone) of the smart device, and extract the decibel value (unit: dB) from the sound wave signal as background noise information using a preset algorithm (e.g., a logarithmic formula). The sound wave signal is an electrical signal that characterizes environmental noise, and includes sound pressure, frequency, etc.

[0116] In some embodiments, background noise information can also be obtained through other means. For example, the audio determination module 240 can directly read the decibel value output by the preset ambient noise detection module of the smart device.

[0117] Output parameters refer to the information used to control the final output effect of the non-semantic audio waveform. For example, output parameters may include loudness gain coefficient, safe amplitude range, and filtering parameters. The loudness gain coefficient is the coefficient that enhances the loudness of the final synthesized onomatopoeia. Onomatopoeia loudness is positively correlated with the output amplitude of the non-semantic audio waveform. The safe amplitude range refers to the safe range of output amplitude of the non-semantic audio waveform allowed by the smart device. The safe amplitude range can be preset based on experience. For more information on filtering parameters, please see the relevant content below.

[0118] In some embodiments, the audio determination module 240 can determine the loudness gain coefficient based on the decibel value in the background noise information using a logarithmic curve equation; the product of the amplitude of each sampling point in the non-semantic audio waveform and the loudness gain coefficient is used as the adjusted output amplitude corresponding to the non-semantic audio waveform. The logarithmic curve equation can be expressed as the following formula (4): (4) in Indicates the loudness gain coefficient; Indicates the decibel value; and This is a scaling constant, which can be preset based on experience.

[0119] In some embodiments, the adjusted output amplitude must be less than or equal to the upper limit of the safe amplitude range; the audio determination module 240 may adjust the output amplitude of the non-semantic audio waveform to the upper limit of the safe amplitude range in response to the adjusted output amplitude being greater than the upper limit of the safe amplitude range.

[0120] Some embodiments in this specification improve audibility and enhance interaction stability by acquiring background noise information and adaptively adjusting output parameters to maintain appropriate loudness of non-semantic audio waveforms under different noise environments.

[0121] In some embodiments, the output parameters include filtering parameters. Filtering parameters refer to parameter information that affects the filtering characteristics of a digital filter. For example, filtering parameters may include cutoff frequency, bandwidth, center frequency, or quality factor.

[0122] In some embodiments, the audio determination module 240 can determine the frequency domain distribution characteristics of background noise information and adjust the filtering parameters according to the frequency domain distribution characteristics.

[0123] Frequency domain distribution characteristics refer to the characteristic information used to reflect the energy distribution of a sound signal within different frequency ranges. For example, frequency domain distribution characteristics can be expressed as the energy intensity distribution of each frequency band.

[0124] In some embodiments, the audio determination module 240 can perform spectral analysis on the digital background noise information using a Fast Fourier Transform (FFT) mathematical accelerator, calculate the power spectral density map composed of different preset frequency bands, and use the energy values ​​of each frequency point or frequency band as frequency domain distribution characteristics. The preset frequency bands can be set empirically. For example, the preset frequency bands may include low-frequency bands (such as bands with frequencies of 100Hz-300Hz), mid-frequency bands (such as bands with frequencies of 300Hz-900Hz), and high-frequency bands (such as bands with frequencies greater than 900Hz).

[0125] In some embodiments, the method for determining frequency domain distribution characteristics may further include: dividing the background noise digital signal into frames and windowing it according to time frames, calculating the short-time spectrum of each time frame, and then averaging or weighting the energy of multiple time frames in each frequency band to generate a frequency domain digital feature map reflecting the stable noise distribution.

[0126] In some embodiments, the audio determination module 240 can retrieve the lowest-energy idle bandwidth as a frequency band recess from the power spectral density map, and update the cutoff frequency and quality factor Q of the digital filter based on the center frequency and bandwidth of the frequency band recess. The frequency band recess is a frequency band characterized by the weakest noise interference. The cutoff frequency is positively correlated with the centroid frequency. The quality factor Q is positively correlated with the center frequency and negatively correlated with the bandwidth. In some embodiments, the audio determination module 240 can control the digital filter to subtract all frequency energies except for the frequency band recess based on the updated cutoff frequency and quality factor Q, thereby reshaping and concentrating all acoustic energy of the non-semantic audio waveform into the frequency band recess.

[0127] In some embodiments, the audio determination module 240 may further include: dividing the frequency domain distribution characteristics according to preset frequency bands, comparing the noise energy of each preset frequency band, selecting target frequency bands with energy below a threshold and continuous bandwidth meeting the requirements, and mapping the target frequency bands to a combination of center frequency, bandwidth and quality factor through a parameter mapping table or calculation formula as the current filtering parameters.

[0128] Some embodiments in this specification improve penetration and reduce harshness by extracting the background noise spectrum and resetting the filtering parameters so that non-semantic audio avoids concentrated output in high-noise frequency bands.

[0129] In some embodiments, the audio determination module 240 is configured to: determine the target acoustic feature based on correction parameters when the volume of the background noise information exceeds a set threshold; and perform compensation processing on the acoustic parameters corresponding to the target acoustic feature based on the background noise information.

[0130] The threshold is set as the volume threshold at which the ambient noise level is deemed too high and requires compensation. This threshold can be set empirically, such as 65 dB.

[0131] Target acoustic features refer to acoustic characteristics that influence the current user's subjective preference. For example, target acoustic features may include crispness and onset speed. Crispness is a parameter reflecting the crispness of the onomatopoeic sound; crispness includes high-frequency crispness and low-frequency crispness. Onset speed refers to the speed at which the onomatopoeic sound is emitted.

[0132] In some embodiments, multiple acoustic parameters (such as reference frequency, onset time, noise ratio, etc.) correspond to multiple correction parameters. The audio determination module 240 can obtain at least one acoustic parameter corresponding to the correction parameter with the largest absolute value, and determine the target acoustic feature based on a preset feature table using at least one acoustic parameter. The preset feature table includes the mapping relationship between different acoustic parameters and their corresponding target acoustic features. The preset feature table can be set based on experience; for example, reference frequency corresponds to crispness, and onset time corresponds to onset velocity.

[0133] In some embodiments, the processor can also aggregate multiple correction parameters by acoustic dimension (i.e., different types of acoustic parameters) to obtain the cumulative correction amount or weighted score of each acoustic dimension, and determine the one or more acoustic dimensions with the largest cumulative correction amount or the highest weighted score as the target acoustic feature preferred by the user. For example, when the reference frequency consistently shows a large positive correction in multiple input instructions, the processor can determine high-frequency crispness as the target acoustic feature.

[0134] In some embodiments, the audio determination module 240 can determine a compensation gain coefficient based on the decibel value in the background noise information using a preset algorithm (such as a piecewise function or a linear function), and multiply the acoustic parameters corresponding to the target acoustic features by the compensation gain coefficient to obtain the compensated acoustic parameters. The compensation gain coefficient is positively correlated with the decibel value. The compensation gain coefficient is less than or equal to a preset gain threshold. When the compensation gain coefficient is greater than the preset gain threshold, the acoustic parameters are amplified according to the preset gain threshold to avoid overcompensation.

[0135] In some embodiments, the processor can also map background noise information to a preset compensation table to obtain parameter compensation amounts corresponding to the current noise range, and then superimpose the parameter compensation amounts onto the acoustic parameters corresponding to the target acoustic features. For example, the processor can increase high-frequency parameters in the 65 dB to 75 dB range and simultaneously shorten the onset time in higher noise ranges to enhance timbre crispness and response speed.

[0136] Some embodiments in this specification provide targeted compensation for the parameters corresponding to the target acoustic features of user preferences in high-noise scenarios. By using a noise adaptive compensation mechanism, the overall amplification distortion is reduced and the emotional recognizability is improved.

[0137] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0138] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0139] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0140] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0141] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0142] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0143] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A non-semantic onomatopoeia synthesis method based on emotion-driven approach, characterized in that, include: Receive input instructions, which include emotion type, emotion intensity, and pronunciation duration; Based on the emotion type and the emotion intensity, determine the acoustic parameters; Based on the emotion type and the pronunciation duration, a pseudo-intonation curve is determined; Based on the acoustic parameters and the pseudo-intonation curve, the non-semantic audio waveform is determined.

2. The method according to claim 1, characterized in that, The determination of acoustic parameters based on the emotion type and the emotion intensity further includes: Based on the emotion type and the emotion intensity, determine the emotion characteristics; based on the emotion characteristics, determine the basic acoustic parameters; Determine the feedback signal value based on user interaction behavior; Based on the feedback signal value, determine the correction parameters; The acoustic parameters are determined based on the basic acoustic parameters and the correction parameters.

3. The method according to claim 2, characterized in that, The input instructions include mixed input instructions, which contain multiple emotion types and corresponding weight information. Determining the acoustic parameters based on the basic acoustic parameters and the correction parameters includes: The composite acoustic parameters are generated based on the basic acoustic parameters corresponding to multiple emotion types and the corresponding weight information. The acoustic parameters are determined based on the composite acoustic parameters and the correction parameters.

4. The method according to claim 1, characterized in that, The method further includes: Obtain background noise information; Based on the background noise information, the output parameters of the non-semantic audio waveform are adjusted.

5. The method according to claim 4, characterized in that, The output parameters include filtering parameters, and adjusting the output parameters of the non-semantic audio waveform based on the background noise information includes: Determine the frequency domain distribution characteristics of the background noise information; The filtering parameters are adjusted based on the frequency domain distribution characteristics.

6. An emotion-driven non-semantic onomatopoeia synthesis system, characterized in that, include: The receiving module is configured to receive input instructions, which include emotion type, emotion intensity, and pronunciation duration. The parameter determination module is configured to determine acoustic parameters based on the emotion type and the emotion intensity; The intonation determination module is configured to determine a pseudo-intonation curve based on the emotion type and the pronunciation duration; The audio determination module is configured to determine non-semantic audio waveforms based on the acoustic parameters and the pseudo-intonation curve.

7. The system according to claim 6, characterized in that, The parameter determination module is further configured to: Based on the emotion type and the emotion intensity, the emotion characteristics are determined; Based on the aforementioned emotional characteristics, the basic acoustic parameters are determined; Determine the feedback signal value based on user interaction behavior; Based on the feedback signal value, determine the correction parameters; The acoustic parameters are determined based on the basic acoustic parameters and the correction parameters.

8. The system according to claim 7, characterized in that, The input instructions include mixed input instructions, which contain multiple emotion types and corresponding weight information. The parameter determination module is further configured to: The composite acoustic parameters are generated based on the basic acoustic parameters corresponding to multiple emotion types and the corresponding weight information. The acoustic parameters are determined based on the composite acoustic parameters and the correction parameters.

9. The system according to claim 6, characterized in that, The audio determination module is further configured to: Obtain background noise information; Based on the background noise information, the output parameters of the non-semantic audio waveform are adjusted.

10. The system according to claim 9, characterized in that, The output parameters include filtering parameters, and the audio determination module is further configured to: Determine the frequency domain distribution characteristics of the background noise information; The filtering parameters are adjusted based on the frequency domain distribution characteristics.

Images