Real-time music generation and audio-visual linkage method and system based on speech feature analysis

Abstract

This invention discloses a real-time music generation and audio-visual linkage method and system based on speech feature analysis, including: acquiring user speech and verifying its validity; extracting acoustic features such as fundamental frequency, energy, and rhythm from valid speech, mapping them to music composition parameters such as mode, tempo, note density, and dynamics; generating structured music data based on the music composition parameters and rendering and playing it; analyzing the audio signal during playback in real time, extracting the beat signal and full-band RMS power; controlling particle emission density, diffusion speed, and glow brightness based on the beat signal, and controlling the color shift of the screen based on the cumulative phase of the full-band RMS power, driving the visual content to dynamically and synchronously change with the music. This invention achieves real-time generation of personalized music based on user speech features, and uses the generated music to drive the visuals, forming an immersive interactive experience with deep audio-visual linkage; the entire process is performed locally, without a network, meeting the requirements of real-time response and stable offline operation.

Description

Technical Field

[0001] This invention relates to the field of digital media technology, specifically to a method and system for real-time music generation and audio-visual linkage based on speech feature analysis. Background Technology

[0002] Existing interactive music installations are mainly divided into two categories: preset music playback type and sound response type. Preset music playback type devices play pre-recorded or generated background music, accompanied by preset animations or lighting effects; users cannot integrate their own characteristics into the displayed content, lacking a personalized sense of participation. Sound response type devices collect ambient sound through microphones, and although they can provide immediate visual or lighting feedback on the volume or spectrum of real-time audio, they only react to sound pressure level (loudness) and cannot extract the melodic features of speech (such as fundamental frequency and pitch), let alone generate structured musical works based on this.

[0003] In recent years, AI music generation services based on large language models (such as Suno and Udio) have emerged, capable of generating high-quality music from text descriptions. However, these services require cloud servers and a stable network connection, with response times typically ranging from tens of seconds to several minutes, failing to meet the real-time (second-level response) and stable operation requirements of public exhibition halls in offline environments. Furthermore, while platforms like TouchDesigner, used for real-time visual programming, are widely applied in interactive installation development, their built-in audio operators primarily provide real-time spectrum analysis and signal processing functions, lacking the ability to create structured music based on speech features.

[0004] Therefore, existing technologies have the following problems: they cannot generate personalized music in real time based on the user's voice characteristics; cloud solutions have high latency and cannot run offline; the visual driving signal is only the real-time sound pressure level and has no relation to the structural characteristics of the generated music, resulting in a disconnect between audio and video; at the same time, they lack an automatic judgment mechanism for recording validity, cannot filter environmental noise or excessively short recordings, and affect output quality. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time music generation and audio-visual linkage method and system based on speech feature analysis, so as to solve the problems mentioned in the background art, such as lack of personalization, high response delay, audio-visual disconnect, and lack of recording validity judgment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: According to one aspect of the present invention, a method for real-time music generation and audio-visual linkage based on speech feature analysis is provided, the method comprising: Collect user voice input and verify its validity; if invalid, prompt the user to re-enter. Acoustic features are extracted from verified speech and mapped to music composition parameters, including fundamental frequency, energy, and rhythm, and music composition parameters including root tone, mode, tempo, note density, and note dynamics. Structured music data is generated based on the music creation parameters and rendered as an audio signal for playback; During playback, the audio signal is analyzed in real time to extract dynamic features, which include low-frequency energy, beat signal, and full-band RMS power. Based on the aforementioned dynamic features, visual content changes are generated and driven in real time, synchronizing the visual changes with the dynamic features of the music to output an interactive audio-visual display.

[0007] Based on the aforementioned scheme, the validity verification includes: comparing the actual recording duration with a preset minimum valid recording duration threshold; if the recording duration is less than the threshold, it is determined to be invalid; and determining the presence of human voices in the recorded audio; if no valid human voices are present, it is determined to be invalid.

[0008] Based on the aforementioned scheme, the determination of the presence of human voice includes: calculating the root mean square energy of the recorded audio frame by frame, counting the proportion of frames with root mean square energy exceeding a preset energy threshold as the proportion of audible frames, and calculating the global root mean square energy average; if the global root mean square energy average is less than the energy threshold, or the proportion of audible frames is less than the preset audible frame proportion threshold, then it is determined that there is no valid human voice.

[0009] Based on the aforementioned scheme, mapping the acoustic features to the music composition parameters includes: The fundamental frequency mean is linearly mapped to the root tone MIDI number, and a major or minor scale is selected based on the normalized fundamental frequency value. The extracted rhythm values ​​are constrained within a preset speed range to obtain the music tempo; The fundamental frequency standard deviation is normalized and then mapped to note density; After normalizing the energy standard deviation, it is mapped to the note dynamics.

[0010] Based on the aforementioned scheme, the generation of structured music data includes: creating a MIDI object and generating a main melody part and a bass part; wherein, the main melody part uses a sequencer to determine whether to fill notes at each step based on the note density, the note pitch is randomly selected from the root note according to the selected key scale, and the note length is randomly selected within a preset number of steps; the bass part is filled with an octave lower than the root note according to the measure cycle; the total music duration is determined by the recording duration and is constrained within a preset number of measures.

[0011] Based on the aforementioned scheme, the steps of extracting acoustic features from the verified speech and mapping them to music composition parameters, and generating structured music data and rendering it as an audio signal, are executed by a background daemon thread. The background daemon thread sends progress and completion signals to the main thread through a thread-safe message queue, and the main thread updates the interface state by polling the message queue, thereby achieving decoupling from the main rendering thread.

[0012] Based on the aforementioned scheme, the real-time analysis of the audio signal to extract dynamic features includes: extracting low-frequency energy channels through multi-band filters, extracting beat signals through transient impulse detection, and extracting full-band RMS power through spectrum analysis; the low-frequency energy is a normalized low-frequency root mean square energy value, the beat signal is a normalized impulse signal, and the full-band RMS power is the root mean square energy value of the audio signal in the current frame.

[0013] Based on the aforementioned scheme, the step of generating and driving visual content changes in real time based on the dynamic features includes: The emission density, initial diffusion velocity, and rendering opacity of the particle system are controlled according to the beat signal, so that the particle dynamics are synchronized with the music beat pulse. The sampling coordinate offset of the color noise field is shifted according to the cumulative phase control of the RMS power across the entire frequency band, so that the color tone of the picture continuously drifts within the preset color range as the music energy accumulates. The brightness of the glow effect in the post-processing is controlled according to the beat signal, so that the glow intensity changes synchronously with the beat.

[0014] Based on the aforementioned scheme, the method also includes a reset process: when a return command is received, audio playback is stopped, the playback interface is hidden and the main recording interface is displayed, the countdown timer and button states are reset, and the system returns to the initial recording standby state, waiting for the next user input.

[0015] According to another aspect of the present invention, a real-time music generation and audio-visual linkage system based on speech feature analysis is provided, the system comprising: The data acquisition and verification module is used to collect user voice input and verify its validity. If it is invalid, the user is prompted to re-enter the input. The feature mapping module is used to extract acoustic features from the verified speech and map them to music composition parameters; The music generation and rendering module is used to generate structured music data based on the music creation parameters and render it as an audio signal for playback. The audio analysis module is used to analyze the audio signal in real time during playback to extract dynamic features, including beat signals, low-frequency energy, and full-band RMS power. The visual driving module is used to generate and drive changes in visual content in real time based on the dynamic features, so that the visual changes are synchronized with the dynamic features of the music, and output an interactive audio-visual display.

[0016] As can be seen from the above technical solution, the present invention has at least the following advantages and positive effects compared with the prior art: The present invention first generates structured music, and then uses the dynamic features of the generated music to drive changes in generated visual content. During the playback of the audio signal rendered from the structured music data, the system extracts the dynamic features of the audio signal in real time (including beat, frequency band energy, and full-band RMS power), and drives changes in visual content according to the dynamic features, so that the visual content is synchronized with the music result generated by the user's voice features. By extracting the acoustic features of the user's voice, such as fundamental frequency, energy, and rhythm, and mapping them to structured music parameters such as mode, tempo, note density, and dynamics, personalized background music can be generated in real time according to the voice characteristics of each user, enhancing the uniqueness of the interaction.

[0017] Audio analysis, MIDI generation, software synthesis, and visual rendering are completed on a local computer without relying on a network connection or cloud server; the end-to-end processing latency is usually within 10 seconds (of which audio analysis and music generation take about 5 to 8 seconds, and rendering and playback are instantaneous), meeting the dual requirements of real-time response and stable offline operation in scenarios such as public exhibition halls, science and technology museums, and commercial spaces.

[0018] An asynchronous processing mechanism combining a background daemon thread and a thread-safe message queue is adopted to decouple time-consuming computational tasks such as audio feature extraction, MIDI generation, and FluidSynth rendering from the TouchDesigner main rendering thread. The main thread receives progress and completion signals by polling the message queue every frame in a non-blocking manner, maintaining a smooth 60FPS interface rendering during background task execution. The progress bar updates smoothly, and the user interface is free from lag or freezing, significantly improving the continuity of the interactive experience.

[0019] By using particle systems and Bloom post-processing effects to precisely map beat signals and audio energy, particle dynamics are synchronized with the beat, hues drift with energy accumulation, and glow changes with the beat, creating an immersive experience with deep audio-visual coupling. This overcomes the problem in existing systems where the visual only reacts to real-time sound pressure levels and is disconnected from the music content.

[0020] This invention incorporates a dual validity verification mechanism: recording duration detection (compared to a preset minimum duration) and multi-dimensional voice presence judgment (global RMS threshold and sound frame ratio threshold). When the recording is too short or the input is ambient noise or silence, the system automatically refuses to enter the music generation process, provides a text prompt, and resets the interface to wait for the user to retry. This mechanism effectively filters out invalid input, ensuring that each generated music has audible and personalized characteristics, while avoiding the waste of computing resources and the decline in display effect caused by low-quality input.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a schematic diagram of the real-time music generation and audio-visual linkage method based on speech feature analysis of the present invention. Figure 2 This is a schematic diagram illustrating the process of collecting user voice input and verifying its validity according to the present invention. Figure 3 This is a schematic diagram of the real-time music generation and audio-visual linkage system based on speech feature analysis of the present invention. Detailed Implementation

[0023] To more clearly illustrate the purpose, technical solutions, and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein. On the contrary, these embodiments are provided so that the present invention will be more comprehensive and complete, and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] The present invention will now be described in detail with reference to specific embodiments.

[0028] Example 1

[0029] like Figure 1 As shown, this embodiment provides a real-time music generation and audio-visual linkage method based on speech feature analysis. The specific steps of this method are as follows: S1: Collect user voice input and verify its validity. If invalid, prompt the user to re-enter.

[0030] After the system starts, it initializes global parameters by reading the configuration file (config.json) in the project root directory, including: maximum recording time (back_time, default 30 seconds), minimum effective recording time (short_time, default 2 seconds), and voice detection RMS threshold (voice_threshold, default 0.005). The user graphical interface is composed of the COMP component of TouchDesigner, including the main recording interface (p1) and the music playback interface (p2), where the main recording interface has a record button (button1).

[0031] like Figure 2As shown, when the user presses the record button, the `onOffToOn` event of the `CHOP Execute DAT` component is triggered. This event performs the following operations: activates the microphone acquisition module (`audioodevin1.par.active=1`); starts audio file recording (`audiofileout1.par.record=1`), writing the PCM audio data input from the microphone to the local temporary file `input.0.wav` in real time; switches the interface background to recording mode (`bg_switch.par.index=1`); and starts a countdown timer (`timer1.par.start.pulse()`), with the countdown duration determined by the configured `back_time`. During the acquisition of user voice input, the interface provides real-time feedback on the recorded duration and remaining available time using a circular progress bar and a digital countdown. When the user releases the record button or the countdown reaches zero, the `onOnToOff` event of `chopexec1` is triggered, automatically stopping recording (`audiofileout1.par.record=0`, `audioodevin1.par.active=0`) and closing the audio file writing operation.

[0032] After recording stops, the current value of `real_time CHOP` is read to obtain the actual recording duration in seconds, denoted as `recorded_duration`. This duration is compared with the configured shortest valid recording duration `short_time` (default 2 seconds). If `recorded_duration < short_time`, the recording duration is deemed invalid. At this point, the system does not proceed with the subsequent music generation process. Instead, it resets the countdown, restores the button state, and displays "Recording too short, please try again" on the interface via a fade_timer animation. The interface then returns to the initial recording standby state.

[0033] For audio files that pass the recording duration validity verification, the `start_voice_to_music()` function is further called to create a background daemon thread. This thread calls the `extract_features()` function to perform a voice presence determination. Specifically, `input.0.wav` is loaded at a 16kHz sampling rate, and the short-time root mean square energy (RMS) is calculated frame by frame to obtain an energy sequence. An energy threshold `voice_threshold` (default 0.005) is defined, and the proportion of frames with RMS values ​​exceeding this threshold is counted as the voiced frame ratio `voiced_ratio`. The mean RMS global root mean square energy of the entire audio file, `mean_rms`, is calculated. If either of the following conditions is met: (a) `mean_rms` < `voice_threshold`; (b) `voiced_ratio` < 0.08 (the preset voiced frame ratio threshold), it is determined that there is no valid voice in the current recording, triggering the `no_speech` event. The interface returns to its initial state and displays "No valid voice detected, please re-record". At this point, the subsequent music generation process is interrupted.

[0034] Only after the recording time reaches the target and passes the human voice presence verification will the next step be taken, which is to continue calling extract_features() to extract acoustic features such as fundamental frequency, energy, and rhythm, and then perform subsequent music generation.

[0035] S2: Extract acoustic features from the verified speech and map them to music composition parameters, wherein the acoustic features include fundamental frequency, energy and rhythm, and the music composition parameters include root note, mode, tempo, note density and note dynamics.

[0036] After passing the dual verification of recording duration and human voice presence in step S1, the acoustic feature extraction and music parameter mapping stage begins. This stage is executed by a background daemon thread to avoid blocking the main rendering thread.

[0037] The extract_features() function is called to load the audio file input.0.wav recorded in step S1 at a sampling rate of 16kHz, and the following acoustic features are extracted sequentially: Fundamental frequency: The librosa.pyin() algorithm (probabilistic fundamental frequency estimation) was used, and the frequency range for analysis was limited to C2 (65Hz) to C6 (1047Hz). The fundamental frequency value was estimated for each audio frame, and the mean (mean_f0) and standard deviation (std_f0) of non-empty frames were calculated. The mean fundamental frequency reflects the average pitch of the user's speech, and the standard deviation of the fundamental frequency reflects the degree of pitch fluctuation (i.e., intonation variation).

[0038] Energy: Calculate the short-time root mean square energy frame by frame to obtain the energy sequence, and calculate the global mean (mean_rms) and standard deviation (std_rms) of the sequence; the energy mean reflects the overall loudness, and the energy standard deviation reflects the dynamic range of loudness.

[0039] Tempo: The librosa.beat.beat_track() function is called to estimate the number of beats per minute (BPM) of the music, denoted as tempo.

[0040] Voiced Ratio: The proportion of frames with RMS values ​​exceeding a preset energy threshold (0.015) out of the total number of frames. This is used to verify the validity of human voices and to help determine the intensity of speech activity.

[0041] Furthermore, the extracted features are judged a second time. If mean_rms is less than voice_threshold (default 0.005) configured in step S1 or voiced_ratio is less than 0.08, it is determined that there is no valid human voice in the recording, triggering the no_speech event, the interface returns to the initial state and displays a friendly prompt, and the subsequent process is terminated; otherwise, the music parameter mapping stage is entered.

[0042] The `map_to_music_params()` function is called to map the above acoustic features into structured music composition parameters. These parameters include root MIDI, scale, tempo (BPM), activity, and energy. The function implements the following mapping rules: (1) Linearly map the fundamental frequency mean_f0 to the root note MIDI note numbers 48-72 (corresponding to C3-C5). Specifically, calculate the normalized fundamental frequency value: Then the root note MIDI number is root_midi = 48 + t * 24; where, This represents a linear normalization calculation, mapping mean_f0 from the original range [120, 800] Hz to the interval [0,1]; clip(..., 0, 1) is a clipping function that restricts the calculation result to between 0 and 1. The higher the fundamental frequency, the higher the pitch of the root note.

[0043] (2) The mode type is determined based on the normalized fundamental frequency value t; if t ≥ 0.5, a major scale ([0,2,4,5,7,9,11]) is selected; if t < 0.5, a natural minor scale ([0,2,3,5,7,8,10]) is selected. This mapping causes sounds with a higher fundamental frequency mean (i.e., a higher average pitch) to generate bright major music, and sounds with a lower fundamental frequency mean to generate melancholic minor music; the scale array corresponding to the mode is stored using scale.

[0044] (3) Constrain the extracted tempo value within the preset music tempo range, i.e. bpm_min and bpm_max are read from the configuration file (default 60 and 130), and STYLE_SPEED is the style adjustment factor (default 1.0).

[0045] (4) The activity density is obtained by normalizing the fundamental frequency standard deviation std_f0: The STYLE_ACTIVITY parameter is the density adjustment factor (default 1.0). The greater the fundamental frequency fluctuation (the more varied the speech), the higher the note density.

[0046] (5) The note dynamics (energy) is obtained by normalizing the energy standard deviation std_rms: energy = clip(std_rms × 10.0 × STYLE_ENERGY, 0, 1), where STYLE_ENERGY is the dynamics adjustment factor (default 1.0). The greater the fluctuation of sound energy, the stronger the note is played.

[0047] After mapping is complete, the function returns a dictionary of music parameters containing root_midi, scale, activity, energy, and bpm, and writes it to the log.

[0048] S3: Generate structured music data based on the music creation parameters and render it as an audio signal for playback.

[0049] To avoid blocking the main rendering thread during audio analysis and music generation, a background daemon thread is created to perform computationally intensive tasks such as feature extraction, parameter mapping, and MIDI synthesis rendering. The background thread sends progress and completion signals to the main thread via a thread-safe message queue (queue.Queue). The main thread polls this message queue non-blockingly during each frame rendering cycle, updating the UI loading progress or switching playback interfaces based on the received signals. By combining the background daemon thread with the asynchronous processing mechanism of the thread-safe message queue, computational tasks and graphics rendering are decoupled, ensuring a smooth and responsive user interface.

[0050] S3.1: Generate structured music data The MIDI generation module is invoked, and a MIDI object is created using the pretty_midi.PrettyMIDI library; based on the music parameters obtained in step S2, two voice parts are generated: Main Melody (Program 80, synthesized string timbre): Employs a 16-step sequencer, progressively filling notes in 16th-note increments. At each step, a random number r ∈ [0,1] is generated. If r < activity, a note is generated; otherwise, the step is skipped. activity is the note density parameter (the probability of generating a note), ranging from [0,1]. If generated, a pitch offset is randomly selected from the current scale, and the actual MIDI note number is calculated using the root note root_midi. The note length is randomly selected within a preset number of steps, such as 1 step (1 / 16 beat) or 2 steps. The note velocity is obtained by multiplying the energy parameter by the velocity base value (default 80) and then rounding down.

[0051] The bass part (Program 38, synthesized bass tone): fills the root note in a measure cycle, with the pitch lowered by an octave (root_midi-12), providing harmonic support.

[0052] The total music duration is determined by the recording duration in step S1 and is constrained within a preset range of bars, such as 4 to 16 bars (MIN_LOOP_DURATION=4s, MAX_LOOP_DURATION=16s), to ensure that the generated music is suitable for loop playback and has a complete rhythmic structure. During the generation process, a background thread periodically sends progress messages (0-100%) through a message queue, and the main thread smoothly updates the interface loading progress bar upon receiving the messages.

[0053] S3.2: Audio rendering of structured music data Using the FluidSynth software synthesizer, the project's built-in synthgms.sf2 sound library file (located in the soundfonts / directory) is loaded. The MIDI data is rendered into 44100Hz sampling rate, 16-bit PCM audio data and written to the file voice_loop.wav. After rendering, a background thread sends a done signal to the main thread via a thread-safe queue (queue.Queue), along with the output file path voice_loop.wav. The main thread calls the _pump_main_queue() function every frame to poll the message queue (driven by run(..., delayFrames=1)). Upon receiving the done signal, it triggers the audio playback module (audiofilein2) to reload the voice_loop.wav file (reloadpulse) and begin playback. The interface switches from the recording main interface (p1) to the playback interface (p2), hiding the recording button and displaying the audio-visual synchronized view.

[0054] S4: During playback, the audio signal is analyzed in real time to extract dynamic features, including low-frequency energy, beat signal and full-band RMS power.

[0055] During music playback, the output audio signal is continuously analyzed in real time to extract dynamic features that drive the visual content.

[0056] After the playback interface (p2) is activated, the real-time audio analysis component (audioAnalysis) located in image_container processes the audio stream output by audiofilein2 frame by frame. This component has a built-in multi-band filter and outputs low-frequency energy and beat signals in real time. Low-frequency energy is calculated by applying a low-pass filter to the audio signal (the cutoff frequency is usually set to 80-120Hz), calculating the root mean square energy of the filtered signal, and normalizing it to the [0,1] interval; this value reflects the instantaneous intensity of the low-frequency components of the music. Beat signal is output by detecting transient impulse components (beat points) in the audio; at the instant the beat is triggered, the signal value is 1, and it decays exponentially to 0 at other times; this signal is continuously output in a frame-by-frame update manner.

[0057] The two signals mentioned above are output in real time via a CHOP channel for the visual rendering module in step S5. The audio analysis component operates synchronously with music playback to ensure that the visual driving signal strictly corresponds to the rhythm and energy changes of the currently playing music. Furthermore, the full-band audio RMS power is extracted through spectrum analysis, denoted as... The full-band RMS power is the root mean square energy value of the current frame audio signal. In this embodiment, it is obtained through the Analyze (RMS) link of the AudioSpectrum component, reflecting the overall energy intensity of the current frame audio signal.

[0058] S5: Based on the dynamic features, generate and drive changes in visual content in real time, so that the visual changes are synchronized with the dynamic features of the music, and output an interactive audio-visual display.

[0059] During music playback, the real-time audio dynamic features extracted in step S4, including beat signals, low-frequency energy, and full-band RMS power, drive the three-dimensional particle system and post-processing effects to achieve synchronous changes in visual content and music dynamic features.

[0060] S5.1: Particle System Parameter Mapping A three-dimensional particle system (geo1 / render1) is constructed, whose emission behavior, motion state, and visual attributes are controlled in real time by a beat signal (Kick). A first-order hysteresis filter (exponential smoothing) is applied to the beat impulse signal to avoid visual flickering caused by steep jumps in the beat impulse signal, thus obtaining a smooth beat envelope signal. ; Wherein, Kick(t)∈{0,1} is the beat impulse signal output in step S4, which is output by the transient detection of the audio analysis module (AudioAnalysis component). It takes a value of 1 at the moment the beat occurs and 0 at other times. The filtered smooth beat envelope signal has a value range of [0,1]; α is the filtering coefficient, and in this embodiment, α=0.713.

[0061] The emission density of the particle system per frame is controlled by the amplitude of the particle instantiation noise field, which is proportional to the smooth beat envelope signal: ; In this embodiment, the particle emission gain coefficient is driven by the beat. ; The particle emission density controls the amplitude of the particle instantiation noise field, determining the spatial diffusion density of particles in a single frame; when a beat is triggered, As the particle density rises, it increases instantaneously, creating a pulse-like burst effect; during the interval between beats, the particle density smoothly falls back as the envelope decays.

[0062] The initial velocity of the particle as it diffuses spherically outward from the emission point has a linear mapping relationship with the original beat impulse signal: ; The initial diffusion velocity, and These represent the minimum and maximum values ​​of the particle diffusion velocity, respectively, in this embodiment. , (Dimensionless unit of speed); As the benchmark for normalizing beat amplitude, take ; This is a limiting function that constrains the input value to the interval [0,1]. When hour, The particles diffuse at their highest speed; when hour, The particles maintain their basic diffusion motion. This mapping ensures that strong beats produce violent particle ejections, while weak beats or non-beat moments result in relatively gentle particle motion.

[0063] The rendering opacity of particles is proportional to the smoothing beat envelope signal: ; To render opacity, This is the opacity gain coefficient in this embodiment. When a beat is triggered, the particles become more vivid and bright; during the interval between beats, the particles become translucent or fade away, thus enhancing the pulsating feeling of the visual rhythm.

[0064] S5.2: Post-processing effect mapping A Bloom post-processing component (Bloom_ver3) is used to apply a GLSL halo effect to the rendered image, and adaptive hue shifting is achieved through a color noise field. The sampling coordinate offset of the color noise field is controlled by the cumulative phase of the RMS power across the entire frequency band, allowing the image hue to continuously shift within a preset color range as the music energy accumulates. A phase offset that monotonically increases with the accumulation of audio energy is defined to drive the sampling coordinate shift of the color noise field. ; For color shift phase, The full-band audio RMS power extracted in step S4 reflects the overall energy intensity of the audio signal in the current frame. This is the drift gain coefficient, in this embodiment ; The time interval between adjacent frames; initial phase During music playback, P(t) increases monotonically with time, and the phase accumulates faster during periods of higher energy.

[0065] Furthermore, a three-dimensional color noise field is constructed, and color values ​​are obtained in the fragment shader based on the sampling coordinates; the offset of the sampling coordinates is defined as: ; tx(t) and tz(t) are the X-axis and Z-axis sampling coordinate offsets of the color noise field, respectively. As P(t) increases, the sampling coordinates move continuously along a specific direction of the noise field, resulting in a visual effect of continuous blue-purple hue drift. In this embodiment, the color range is determined by a color scale mapping table, with the red component (R) ranging from [0, 0.717], the green component (G) ranging from [0, 0.771], and the blue component (B) ranging from [0.582, 0.826]. In the low-energy stage, the image presents a purplish-blue hue (with a higher blue component); as P(t) increases, the sampling coordinates move to the area in the noise field where the red and green components increase, and the overall hue drifts towards blue-white, thus achieving a smooth transition of visual color temperature while accumulating auditory energy.

[0066] In this embodiment, the Bloom post-processing module uses fixed parameters: Intensity = 1.66, Exponent = 3, and Blacklevel = 1.14. The glow brightness is not directly driven by audio energy, but indirectly modulated through particle opacity, and the equivalent brightness output is: ; Glow brightness Beat envelope signal Linear variation: the halo intensifies when the beat is triggered and weakens during the interval between beats, forming a visual pulse synchronized with the rhythm.

[0067] All of the above visual parameters (particle emission density) Initial diffusion velocity V(t), rendering opacity Opacity(t), color shift phase P(t), glow brightness Each frame is based on the latest beat signal Smooth beat envelope and spectral energy It updates in real time and outputs the results to the playback interface, achieving an immersive and interactive display with deep audio-visual integration.

[0068] When the user clicks the back button in the lower right corner of the playback interface, panelexec DAT triggers a reset operation, stops audio playback (audiofilein2.par.play=0), hides the playback interface (p2) and displays the main recording interface (p1), resets the countdown timer and button states, and the system returns to the initial recording standby state in step S1, waiting for the next round of interaction.

[0069] Example 2 like Figure 3 As shown in the figure, this embodiment exemplarily presents a real-time music generation and audio-visual linkage system based on speech feature analysis, including a user interface layer, a data acquisition and verification module, a feature mapping module, a music generation and rendering module, an audio analysis module, and a visual driving module.

[0070] The user interface layer consists of TouchDesigner's COMP component, including: the main recording interface (p1), which provides a recording button, countdown display, circular progress bar, and prompt information area; and the playback interface (p2), which displays a full-screen audio-visual interactive visual effect and includes a semi-transparent back button. The interface layer is responsible for receiving user interaction events (button press / release, back click) and switching the display according to the system status.

[0071] The acquisition and verification module is used to acquire user voice input and verify its validity. If invalid, the user is prompted to re-enter. This module connects to the microphone device, responds to recording trigger events on the user interface, and writes the acquired audio data to a temporary file. When the user presses and holds the record button on the main recording interface, the module executes the following: activates microphone acquisition (audioodevin1.par.active = 1); starts audio file recording (audiofileout1.par.record = 1), writing the PCM audio data input from the microphone to the temporary file input.0.wav in real time; and starts a countdown timer, displaying the remaining duration and a circular progress bar on the interface.

[0072] When the user releases the record button or the countdown reaches zero, the module stops recording and closes the file. It performs dual validity verification: recording duration verification and voice presence verification. For recording duration verification, the actual recording duration is compared to the preset minimum valid recording duration threshold `short_time`. If it is insufficient, the recording is deemed invalid, and the interface displays "Recording too short, please try again" and resets. For voice presence verification, for recordings of the specified duration, `input.0.wav` is loaded, and the short-time root mean square energy (RMS) is calculated frame by frame to obtain the global mean `mean_rms` and the proportion of frames with sound `voiced_ratio` (the percentage of frames with RMS exceeding `voice_threshold`). If `mean_rms < voice_threshold` or `voiced_ratio < 0.08`, it is determined that no valid voice is detected, and the interface displays "No valid voice detected, please re-record" and resets. When both verifications pass, the module passes the audio file path to the feature mapping module.

[0073] The feature mapping module extracts acoustic features from verified speech and maps them to music composition parameters. This module runs in a background daemon thread to avoid blocking the main rendering thread. It calls an audio analysis library (such as librosa) to perform the following: load input.0.wav; extract the fundamental frequency using the librosa.pyin() algorithm, limiting the frequency range to 65Hz–1047Hz, and calculate the mean_f0 and standard deviation_f0 of non-empty frames; extract energy by calculating the RMS frame-by-frame to obtain the global mean_rms and standard deviation_rms; extract tempo by estimating the BPM using librosa.beat.beat_track(), denoted as tempo. Based on preset mapping rules, the acoustic features are mapped to music composition parameters, including converting the fundamental frequency mean_f0 to the root note MIDI number and key (major / minor), constraining the tempo to the tempo range, normalizing the fundamental frequency standard deviation to note density, and normalizing the energy standard deviation to note velocity. After mapping is complete, the music parameter dictionary is passed to the music generation and rendering module, and a progress message is sent to the main thread via a message queue.

[0074] The music generation and rendering module is used to generate structured music data based on music composition parameters and render it as an audio signal for playback. This module receives music parameters and performs the following actions: creating a MIDI object, generating the main melody and bass parts, with the main melody using a 16-step sequencer in 16th-note increments; at each step, determining whether to generate a note based on probability (activity); if generated, randomly selecting a pitch offset from the current key scale and calculating the actual MIDI note number based on the root note; the note length is randomly selected between 1 and 2 steps; the note velocity is calculated by multiplying energy by the velocity base value and rounding down; the bass part is filled with an octave lower than the root note according to a measure cycle; a software synthesizer (such as FluidSynth) is called to load the sound library, rendering the MIDI data into a PCM audio file voice_loop.wav, and triggering playback through the audio output component. After rendering, the module sends a "done" signal to the main thread via a thread-safe message queue, along with the output file path.

[0075] The audio analysis module is used to analyze audio signals in real time and extract dynamic features during playback. It starts upon activation of the playback interface; this module uses the `audioAnalysis` component in `image_container` to acquire the audio stream output from `audiofilein2` in real time, performs multi-band filtering and transient detection on the playing audio stream, and outputs normalized beat signals (Kick) and low-frequency energy signals (Low) in real time. These two signals are output in real time per frame in CHOP channel format for use by the vision-driven module. The module also extracts full-band RMS power in real time through the `Analyze(RMS)` link of the `AudioSpectrum` component. This signal reflects the overall energy intensity of the audio signal in the current frame and is used for color shifting in the vision-driven module.

[0076] The visual driving module generates and drives changes in visual content in real time based on the dynamic features, synchronizing visual changes with the dynamic features of the music to output an interactive audio-visual display. This module constructs a three-dimensional particle system and a Bloom post-processing component, driving visual changes in real time based on the dynamic features output by the audio analysis module. A first-order hysteresis filter is applied to the beat impulse signal to obtain a smooth beat envelope signal. The emission density and particle rendering opacity of the particle system are controlled based on the smooth beat envelope signal, allowing particle bursts and decays to smoothly change with the beat envelope. The initial velocity of particle diffusion is controlled by nonlinear mapping based on the amplitude of the beat impulse signal, so that stronger beats correspond to more intense particle ejection. The sampling coordinate shift of the color noise field is driven by the cumulative phase of the full-band audio power, causing the image hue to continuously drift within a preset color range (e.g., from purplish-blue to blue-white) as the music energy accumulates. The equivalent glow brightness of the Bloom post-processing component is controlled by gain modulation based on the smooth beat envelope signal, ensuring that the glow intensity synchronizes with the beat pulse. All visual parameters are updated in real time for each frame based on the latest beat impulse signal, smooth envelope signal and full-band audio power, and output to the playback interface (p2) to form an interactive display with deep audio-visual coupling.

[0077] When the user clicks the back button in the lower right corner of the playback interface, the system performs a reset operation: stops audio playback (audiofilein2.par.play = 0), hides the playback interface (p2) and displays the main recording interface (p1), resets the countdown timer and button states, and the system returns to the standby state after initialization, waiting for the next round of interaction.

[0078] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims. It should be understood that the invention is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A real-time music generation and audio-visual linkage method based on speech feature analysis, characterized in that, The method includes: Collect user voice input and verify its validity; if invalid, prompt the user to re-enter. Acoustic features are extracted from verified speech and mapped to music composition parameters, including fundamental frequency, energy, and rhythm, and music composition parameters including root tone, mode, tempo, note density, and note dynamics. Structured music data is generated based on the music creation parameters and rendered as an audio signal for playback; During playback, the audio signal is analyzed in real time to extract dynamic features, which include low-frequency energy, beat signal, and full-band RMS power. Based on the aforementioned dynamic features, visual content changes are generated and driven in real time, synchronizing the visual changes with the dynamic features of the music to output an interactive audio-visual display.

2. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The validity verification includes: comparing the actual recording duration with a preset minimum valid recording duration threshold; if the recording duration is less than the threshold, it is determined to be invalid; and determining the presence of human voices in the recorded audio; if no valid human voices are present, it is determined to be invalid.

3. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 2, characterized in that, The determination of the presence of human voice includes: calculating the root mean square energy of the recorded audio frame by frame, counting the proportion of frames with root mean square energy exceeding a preset energy threshold as the proportion of audible frames, and calculating the global root mean square energy average; if the global root mean square energy average is less than the energy threshold, or the proportion of audible frames is less than the preset audible frame proportion threshold, then it is determined that there is no valid human voice.

4. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, Mapping the acoustic features to the music composition parameters includes: The fundamental frequency mean is linearly mapped to the root tone MIDI number, and a major or minor scale is selected based on the normalized fundamental frequency value. The extracted rhythm values ​​are constrained within a preset speed range to obtain the music tempo; The fundamental frequency standard deviation is normalized and then mapped to note density; After normalizing the energy standard deviation, it is mapped to the note dynamics.

5. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The generation of structured music data includes: creating a MIDI object, generating a main melody part and a bass part; wherein, the main melody part uses a sequencer to determine whether to fill notes at each step based on the note density, the note pitch is randomly selected from the root note according to the selected key scale, and the note length is randomly selected within a preset number of steps; the bass part is filled with the octave lower than the root note according to the measure cycle; the total music duration is determined by the recording duration and is constrained within a preset number of measures.

6. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The steps of extracting acoustic features from the verified speech and mapping them to music composition parameters, and generating structured music data and rendering it as an audio signal, are executed by a background daemon thread. The background daemon thread sends progress and completion signals to the main thread through a thread-safe message queue. The main thread updates the interface state by polling the message queue, thereby decoupling from the main rendering thread.

7. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The real-time analysis of the audio signal to extract dynamic features includes: extracting low-frequency energy channels through multi-band filters, extracting beat signals through transient impulse detection, and extracting full-band RMS power through spectrum analysis; the low-frequency energy is a normalized low-frequency root mean square energy value, the beat signal is a normalized impulse signal, and the full-band RMS power is the root mean square energy value of the audio signal in the current frame.

8. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The generation and real-time driving of visual content changes based on the dynamic features includes: The emission density, initial diffusion velocity, and rendering opacity of the particle system are controlled according to the beat signal, so that the particle dynamics are synchronized with the music beat pulse. The sampling coordinate offset of the color noise field is shifted according to the cumulative phase control of the RMS power across the entire frequency band, so that the color tone of the picture continuously drifts within the preset color range as the music energy accumulates. The brightness of the glow effect in the post-processing is controlled according to the beat signal, so that the glow intensity changes synchronously with the beat.

9. The real-time music generation and audio-visual linkage method based on speech feature analysis according to claim 1, characterized in that, The method also includes a reset process: when a return command is received, audio playback is stopped, the playback interface is hidden and the main recording interface is displayed, the countdown timer and button states are reset, and the system returns to the initial recording standby state, waiting for the next user input.

10. A real-time music generation and audio-visual linkage system based on speech feature analysis, used to implement the method as described in any one of claims 1-9, characterized in that, include: The data acquisition and verification module is used to collect user voice input and verify its validity. If it is invalid, the user is prompted to re-enter the input. The feature mapping module is used to extract acoustic features from the verified speech and map them to music composition parameters; The music generation and rendering module is used to generate structured music data based on the music creation parameters and render it as an audio signal for playback. The audio analysis module is used to analyze the audio signal in real time during playback to extract dynamic features, including beat signals, low-frequency energy, and full-band RMS power. The visual driving module is used to generate and drive changes in visual content in real time based on the dynamic features, so that the visual changes are synchronized with the dynamic features of the music, and output an interactive audio-visual display.

Images