Emotional audio real-time driven synthesis method

Abstract

This invention discloses a real-time driven synthesis method for emotional audio, belonging to the field of digital signal processing and emotion computing. The method first acquires emotional description parameters and timbre selection parameters S in real time, where the emotional description parameters include multi-dimensional vectors representing emotional dimensions or categories; it then generates a vocal frequency parameter F, a pitch parameter T, and a volume parameter V through a mapping algorithm; simultaneously, it extracts the sampling length parameter L from the original audio material. In the synthesis stage, the incremental step size of the phase accumulator is updated in real time using parameters L and T, and the phase accumulator is reset and started by parameter F, combined with a logic halting mechanism to achieve precise waveform addressing. Furthermore, the system introduces sampling switching latch logic to ensure the stability of dynamic sampling set rotation, and performs random perturbation and smoothing processing on F and T, and smoothing processing on V. This invention achieves dynamic closed-loop control of pitch, rhythm, and volume by emotional parameters, enhancing the expressive power of sound while also considering biological naturalness and real-time response efficiency in embedded environments.

Description

Technical Field

[0001] This invention relates to the fields of digital signal processing and emotion computing, specifically to a real-time driven synthesis method for nonverbal emotional audio based on the PAD model. Background Technology

[0002] Currently, achieving natural and dynamic anthropomorphic voices in the nonverbal emotional expression of intelligent interactive devices (such as emotional robots and intelligent dolls) remains a technical challenge. Traditional solutions are mainly based on fixed sampling triggering mechanisms: pre-recording audio clips in various emotional states and playing them according to interactive commands. While simple to implement, this approach has inherent flaws: the physical characteristics of the sound, such as pitch, volume, and rhythm, are locked after recording and cannot change in real time with the robot system's emotional state. This results in the output sound of the device often having a distinctly fragmented and mechanical feel, making it difficult to simulate the subtle and random acoustic changes that occur when a living being speaks with emotional fluctuations. Therefore, an ideal solution not only needs to be able to dynamically switch between different audio sampling materials like traditional solutions (e.g., dynamically rotating through a series of samples with similar timbres but subtle physiological differences), but also needs to be able to modulate dynamic sounds in various emotional states in real time based on underlying algorithms. By organically combining sampling rotation with real-time emotional modulation, the richness and anthropomorphism of nonverbal emotional voices can be greatly enhanced. Therefore, how to provide a real-time driven synthesis scheme that can automatically adapt to diverse audio materials, has highly decoupled control parameters, and has low computational overhead is a problem that urgently needs to be solved in this field. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time driven synthesis method for emotional audio based on multidimensional emotional mapping and phase accumulator. By deconstructing abstract emotional parameters into multidimensional feature vectors and mapping them into three core driving components—vocal frequency parameter F, pitch parameter T, and volume parameter V—and combining this with adaptive sampling length technology and a logic halting mechanism based on a phase accumulator, real-time dynamic shaping of the physical characteristics of non-verbal sounds is achieved.

[0004] The key technical points of this invention are as follows: The system constructs a multi-dimensional parameter parallel input model, receiving timbre selection parameters S and emotion description parameters; the emotion description parameters include a three-dimensional emotion vector consisting of pleasure level P, activation level A, and dominance level D, or a discrete emotion tag model. Through a preset mapping algorithm, the emotion description parameters are coupled and mapped in real time to a control sequence of vocal frequency parameters F, pitch parameters T, and volume parameters V. Simultaneously, the system obtains the length parameter L of the driven audio material. The parameter L serves as the physical reference for the phase accumulator, used together with the pitch parameter T to determine the increment step size of the normalized phase accumulator, thereby ensuring that the driving logic can adapt to audio primitives of different durations or periods; simultaneously, the real-time output value of the phase accumulator is used for linear addressing within the physical interval defined by the length parameter L to output a continuous emotional audio signal.

[0005] As a further development of the above scheme, in the real-time dynamic synthesis stage, this invention constructs a phase accumulator adaptive waveform driving module. This module uses the pitch parameter T as an adjustment factor and combines it with the sampling length parameter L to update the increment step size of the normalized phase accumulator in real time. In specific operation, the system uses the sound interval trigger signal generated by mapping the sound frequency parameter F to drive the phase accumulator to reset and start; the normalized phase sequence output by the phase accumulator is converted into the reading index address of the original audio material through a mapping operator. The operating logic of the phase accumulator adaptive waveform driving module includes: responding to the trigger signal to reset the phase value to zero and performing waveform reading according to the determined increment step size; when the phase value reaches the preset termination threshold, i.e., the normalized value is 1, the system automatically sets the increment step size to zero and enters a standby lock state, thereby realizing the logical shutdown of a single sound event until the next trigger signal is received. Regarding control parameter optimization, this invention introduces a sampling switching synchronization mechanism. By setting state latching logic, it ensures that the system does not respond to new timbre selection parameters S during the current audio sample loading and parameter L update calculation. The switching channel is only opened after the physical parameter latching is stable to avoid logical operation disorder. At the same time, independent random perturbation processing is introduced for the F and T sequences to simulate the natural perturbation characteristics of biological vocalization; linear interpolation smoothing processing is performed on the volume parameter V to obtain a continuous and stable gain control envelope and apply it to the output signal.

[0006] The technical solution of this invention also includes a signal post-processing stage: before waveform output, the system performs real-time adjustment of the synthesized audio based on the gain control envelope generated by the aforementioned volume parameter V. To ensure the stability of the signal during high-speed dynamic changes, the system introduces an adaptive amplitude limiting algorithm to limit the amplitude of the audio sample within a preset dynamic threshold range. In its specific implementation, this algorithm serves two purposes: firstly, it prevents truncation distortion that may occur due to rapid gain adjustment of the audio signal, ensuring the physical integrity of the output signal under different emotional intensities; secondly, by dynamically controlling the waveform peak value, it improves the loudness and clarity of the synthesized audio on the low-power speaker of the embedded device, ultimately outputting an emotional audio signal that is synchronized with the emotional parameters in real time and has a stable signal.

[0007] The beneficial effects of this invention are as follows: (1) Achieved dynamic sound shaping driven by high-dimensional emotional parameters: By real-time coupling and mapping of multi-dimensional emotional description parameters (such as P, A, D components) to acoustic features such as vocal frequency, pitch and volume, non-verbal sounds can be adaptively adjusted according to the continuous changes in emotional state, significantly enhancing the emotional expression ability and anthropomorphism of synthesized sounds. (2) Ensured the stability of vocal logic in complex interactive environments: By introducing adaptive compensation of sampling length L, logic shutdown mechanism and state latch synchronization control, the problem of signal truncation and reading errors caused by frequent switching of dynamic instructions or differences in material length was solved. While achieving deep modulation of parameters, the continuity and physical integrity of audio output were ensured. (3) Balanced biological naturalness and embedded engineering efficiency: By introducing random perturbation processing to key acoustic parameters, the natural fluctuation characteristics of biological vocalization were effectively simulated; at the same time, the overall architecture is based on the streaming processing logic of phase accumulator, with low computational complexity and small memory footprint, especially suitable for the low-latency real-time response requirements in embedded hardware environments. Attached Figure Description

[0008] Figure 1 This is a block diagram of the overall architecture of a real-time driven synthesis system for emotional audio provided in an embodiment of the present invention.

[0009] In the diagram: 101, Multi-dimensional parameter input module; 102, Emotional multi-dimensional mapping module; 103, Audio material acquisition and length parsing module; 104, Parameter perturbation and smoothing module; 105, Phase accumulator adaptive waveform driving module; 106, Audio post-processing and limiting output module. Detailed Implementation

[0010] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the core technical concept of the present invention, all equivalent implementation methods obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0011] This invention discloses an adaptive voice shaping and synthesis system based on multidimensional emotion mapping. Addressing the limitation of traditional nonverbal voice expression in conveying limited emotion, it achieves natural, stable, and highly emotion-adaptive nonverbal audio synthesis output through core technologies such as differentiated processing of multiple acoustic parameters, a timbre parameter latching synchronization mechanism, pitch and material length coupling step size calculation, and frequency-triggered phase closed-loop reset. The overall working steps of this invention are as follows:

[0012] The first step involves system parameter input, distribution, and timbre switching latch control based on the multi-dimensional parameter input module 101. The system synchronously receives two independent parameters from the external upper-layer algorithm via module 101: the timbre selection parameter S and the three-dimensional emotion description parameters P, A, and D. To achieve a high-rich expression combining dynamic sampling set rotation and real-time emotion modulation, the system supports real-time changes to the timbre selection parameter S during vocalization. To avoid computational conflicts between the dynamic changes in parameter S and the material length L refresh and the high real-time phase addressing logic (such as addressing errors or waveform truncation pops caused by unstable parameter coverage), module 101 internally sets up sampling switching synchronization and state latching logic: when the system is loading the current audio material and iteratively updating the sampling length L, the system locks its working state, no longer responding to or refreshing newly input timbre selection parameters S. It automatically unlocks after the current frame material is loaded and the L parameter calculation is stable, allowing it to respond to and activate new timbre switching commands and output to module 103. The emotion description parameters P, A, and D are not constrained by latching logic. They are input and distributed in real time throughout the process and directly output to the 102 module, realizing real-time dynamic updates of the emotion state. All parameters operate synchronously, independently, and without interference.

[0013] The second step involves performing multidimensional mapping of emotional parameters based on the emotional multidimensional mapping module 102 and generating three sets of independent acoustic control parameters. Module 102 receives the three-dimensional emotional parameters P, A, and D, and through a preset emotional-acoustic correlation mapping model, deconstructs, calculates, and independently outputs three sets of core control parameters: the frequency parameter F, the pitch parameter T, and the volume parameter V. The frequency parameter F controls the trigger interval and rhythm of the audio loop playback; the pitch parameter T controls the pitch of the synthesized sound; and the volume parameter V controls the volume of the output audio. These three sets of parameters are independent of each other, providing a precise control basis for subsequent differential algorithm processing and phase driving.

[0014] The third step involves the audio material acquisition and sampling length analysis calculation performed by the audio material acquisition and length analysis module 103. Module 103 receives the timbre selection parameter S from module 101, retrieves the corresponding target audio material preset by the system based on S, and performs sampling analysis on the material to accurately calculate the unique sampling length L corresponding to the current material. The sampling length L has a dual core function: First, it participates in the phase step mixing calculation of the subsequent pitch parameter T to adapt to the basic playback rate of different materials; second, after the phase operation is completed, it is multiplied with the phase result to calibrate and obtain the final actual audio playback position, ensuring that materials of different lengths can adaptively match the phase iteration logic.

[0015] The fourth step involves performing multi-parameter independent differentiated perturbation and smoothing preprocessing based on the parameter perturbation and smoothing processing module 104. This invention differs from the traditional, coarse-grained approach of uniformly packaging parameters. Within module 104, three completely independent parameter processing links are set up for F, T, and V. Each link operates in isolation and does not interfere with the others. Furthermore, differentiated processing logic is employed to precisely match the control attributes of different acoustic parameters: for the phonation frequency parameter F and the pitch parameter T, two dedicated and independent random micro-perturbation submodules are configured, with different random seeds, perturbation ranges, and operational parameters for each submodule. The F parameter undergoes independent random perturbation followed by smoothing filtering to optimize the natural randomness of the sound trigger interval and avoid a mechanically fixed rhythm. The T parameter also undergoes independent random perturbation followed by smoothing filtering to eliminate the stiffness of the sound and achieve biomimetic pitch instability and jitter. The volume parameter V is only smoothed to eliminate volume jumps and pops, ensuring a stable output loudness throughout. After preprocessing, the F and T parameters are independently output to the 105 phase accumulator adaptive waveform driver module to participate in phase algorithm calculations. The V parameter, after smoothing filtering, is directly output to the 106 audio post-processing and limiting output module for real-time control of the final playback volume.

[0016] The fifth step involves the phase accumulator adaptive waveform driving module 105 performing real-time dynamic calculations and frequency-triggered closed-loop driving. Module 105 has a built-in normalized closed-loop phase accumulator (Phasor) ranging from 0 to 1. It receives preprocessed parameters F and T from input 104 and the sampling length L from output 103 to perform phase calculations. Each parameter performs its specific function and works together to complete the waveform driving logic, as follows: First, the pitch parameter T and sampling length L are coupled and calculated in real-time. The system pre-calculates the normalized reference step size using the system sampling rate and sampling length L, and uses the preprocessed pitch parameter T as an adjustment factor to scale the reference step size, thereby dynamically updating the phase increment step size of the Phaser to achieve dynamic adaptive adjustment of the playback pitch. Relying on the random perturbation characteristics of the T parameter, the pitch change is natural and fluid, without a mechanical feel. Second, the sound frequency parameter F is responsible for triggering the phase cycle start and stop. The phase value continuously accumulates within the range of 0 to 1. When the phase value reaches the upper limit of 1, the step size is automatically reset to zero, playback pauses, and the system enters standby mode. The system continuously monitors the timing cycle of the F parameter, which precisely triggers a new round of phase reset and restart playback, determining the time interval and rhythm of each sound output. After the phase calculation is completed, the system multiplies the phase result by the sampling length L to obtain accurate actual audio playback waveform data, which is then output to the audio post-processing and limiting output module 106.

[0017] The sixth step involves audio post-processing, volume adaptation, and final output based on the audio post-processing and limiting output module 106. Module 106 simultaneously receives the shaped waveform data output from module 105 and the smoothed and filtered volume parameter V output from module 104. It adaptively adjusts the loudness of the audio waveform, and in conjunction with waveform gain and limiting processing, it amplifies the waveform signal while suppressing popping and distortion problems caused by waveform peaks exceeding the threshold. The final output is a high-quality non-verbal audio signal with natural rhythm, rich emotion, stable volume, and no acoustic defects.

[0018] In summary, this invention solves the problem of disordered timbre switching through the S-parameter latching synchronization mechanism of module 101, achieves accurate conversion of emotional parameters to acoustic parameters through multi-dimensional mapping of module 102, adapts material characteristics through dual-dimensional calculation of L-parameters of module 103, and achieves zoned control with random rhythm optimization, natural pitch fluctuation, and stable volume output through the completely independent differentiated preprocessing and branch output logic of F, T, and V channels within module 104. At the same time, relying on the refined phase-driven mechanism of F frequency-triggered playback and T real-time dynamic adjustment step size within module 105, combined with the collaborative architecture of V parameter real-time volume control input to module 106, this invention completely solves the problem of robots (such as intelligent interactive devices) having mechanical voices, lacking expressiveness, and unable to adjust acoustic characteristics in real time according to emotional state when expressing non-verbal emotions. The greatest technical advantage of this invention lies in its lightweight real-time synthesis algorithm, which achieves dynamic closed-loop control of pitch (T), rhythm (F), and volume (V) using multi-dimensional emotional parameters. This not only supports the extreme enrichment of physiological details and expressive layers of sound through dynamic rotation of sample sets, but also endows the sound with a natural, dynamic sense of randomness and emotional fluctuation. The overall architecture has extremely low operating overhead and strong real-time performance, perfectly adapting to the low-latency operation requirements of embedded hardware.

Claims

1. An affective audio real-time driven synthesis method, characterized by, The method includes the following steps: (1) Emotional feature acquisition: real-time acquisition of the emotional description parameters and timbre selection parameters S of the target object; the emotional description parameters include a three-dimensional emotion vector consisting of pleasure level P, activation level A, and dominance level D; (2) Multidimensional mapping coupling: The emotion description parameters are associated and mapped to a set of audio driving parameter sequences through a mapping algorithm. The sequence includes at least the vocal frequency parameter F, the pitch parameter T, and the volume parameter V. (3) Sampling adaptive processing: Load the corresponding original sampled audio from the preset sampled material set according to the timbre selection parameter S, and extract the sampling length parameter L of the original sampled audio; (4) Sequence enhancement optimization: Random perturbation algorithm and linear smoothing are performed on the vocal frequency parameter F and pitch parameter T to generate a driving sequence with biological wave characteristics; linear smoothing is performed on the volume parameter V; (5) Dynamic waveform synthesis: The sampling length parameter L and the optimized pitch parameter are input into the phase accumulator to match the scanning frequency, and the phase accumulator is driven by the trigger signal generated based on the sound frequency parameter F to read and change the pitch of the original sampled audio in real time. (6) Output gain control: Perform dynamic gain adjustment and clipping processing on the synthesized audio signal based on the volume parameter V.

2. The method of claim 1, wherein: In the adaptive sampling process, the total number of sampling frames of the original sampled audio is obtained and defined as the sampling length parameter L. This L is used as the frequency compensation factor of the phase accumulator, so that the resampling rate corresponding to the pitch parameter T is dynamically self-calibrated as L changes.

3. The method of claim 1, wherein: In the sequence vividness optimization, independent random perturbation algorithms are superimposed on the basic sequences of vocal frequency parameter F and pitch parameter T, which are generated in real time by mapping the emotion description parameters. Specifically, a trigger pulse sequence with non-periodic characteristics is generated based on the superimposed perturbation F, which serves as the trigger condition for driving the phase accumulator to start scanning. A dynamically fluctuating pitch control signal is generated based on the superimposed perturbation T, which is used to jointly simulate the randomness characteristics of biological vocalization that fluctuate with emotions and have natural micro-perturbations.

4. The method of claim 1, wherein: The frequency parameter F is determined by the activation degree A and the offset of the dominance degree D relative to the preset steady-state value; wherein, F increases with the increase of activation degree A and increases with the increase of the degree of dominance degree D deviating from the preset steady-state value.

5. The method of claim 1, wherein: The calculation logic of the pitch parameter T uses the pleasure level P as the dominant factor, and performs linear or nonlinear compensation adjustment based on the activation level A and the dominance level D.

6. The method of claim 1, wherein: In the dynamic waveform synthesis, the phase value generated by the phase accumulator is used to index the address of the original sampled audio. By combining the sampling length parameter L to change the increment step of the read pointer in real time, dynamic pitch shifting at the sampling frame level is achieved.

7. The method of claim 1, wherein: The output gain control includes real-time power amplification of the signal and the use of a limiting algorithm to limit the amplitude of the audio sample within a preset dynamic range threshold determined based on the volume parameter V, in order to prevent audio clipping distortion and enhance the aggressiveness of the sound.

8. The method of claim 1, wherein: The method is encapsulated in a computing component adapted to the aarch64 architecture. It receives external input S-parameters and P, A, and D data streams through the application programming interface (API), achieving millisecond-level synchronization between parameter changes and audio waveform adjustments.

9. The method of claim 8, wherein: The computing component supports controlled switching of the timbre selection parameter S during audio playback via state latching logic. That is, the switching channel is locked during the update calculation of the current sample length parameter L, and the new timbre selection parameter S is responded to after the physical parameter latching is stable, so as to ensure the stability of the logic operation during the dynamic rotation process in the sample material set with subtle feature differences.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that: When the computer instructions are executed by the processor, they implement the method for real-time driven synthesis of nonverbal emotional audio based on the PAD model as described in any one of claims 1 to 9.

Images