A method for music and light visualization based on acousto-optic mapping

Abstract

The application belongs to the technical field of music propagation and artistic algorithm, and specifically discloses a music and light visualization method based on sound-light mapping, which comprises the following steps: collecting and pre-analyzing and extracting audio signals through real-time microphone collection or a preset audio input mode; filtering out noise in the audio signals, improving the uniformity and stability of the audio signals, and amplifying audio details through an adaptive filtering algorithm, a normalization algorithm and a dynamic compression method, so that preprocessed audio signals are obtained; a multi-layer convolutional neural network model is constructed, the preprocessed audio signals are input into the multi-layer convolutional neural network model, and music features are extracted based on a mel-frequency scale; a multi-dimensional sound-light parameter mapping relationship is established based on the music features and light feature parameters; and light signals are generated and output according to the mapping relationship, so that music and light visualization is realized. The application solves the problems of single nature, insufficient stability, high delay and application limitations in traditional music visualization technology.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of music dissemination and artistic algorithm, and specifically relates to a music and light visualization method based on sound-light mapping. BACKGROUND

[0002] Music and light, as two important forms of artistic expression, play an important role in the history of art and modern multimedia art. With the development of technology, more and more artists and researchers have begun to explore the deep integration of music and light to create more immersive and interactive artistic experiences. In modern concerts and stage performances, lighting design often changes in synchronization with the rhythm of music to enhance the atmosphere of the scene; in art exhibitions and immersive installations, the combination of music and light has also become a common means of expression. However, most current ways of combining music and light are still relatively simple, mainly relying on preset programs or simple mapping rules, lacking multi-element accurate analysis and intelligent light control mapping of music details (such as volume, duration, timbre, synthesis, reverb, etc.), making it difficult to achieve more expressive and interactive experiences.

[0003] In the field of synesthesia research, researchers have found that there is a certain correlation between music and visual perception, and some synesthetes can naturally map pitch or timbre to specific colors or shapes. This phenomenon provides an important theoretical basis for the visualization design of music and light. At the same time, with the development of digital media, artificial intelligence and Internet of Things technology, real-time analysis and visualization of audio data has become more feasible, making music-driven light interaction systems not only based on traditional mapping rules, but also able to achieve more dynamic responses with machine learning and complex algorithms.

[0004] Therefore, it is necessary to explore a deep integration method of music and light, to accurately analyze sound signals and light signal elements, to map sound and light physical properties, to establish accurate sound-light multi-element mapping rules, to introduce sound-light parameter mapping algorithms, to complete multi-dimensional and multi-level fast sound-light signal conversion, to realize an intelligent and interactive music visualization system, to help expand the forms of music and light, and to provide new possibilities for multi-sensory artistic interaction in fields such as performing arts, exhibitions, entertainment and immersive experiences. SUMMARY

[0005] The purpose of the present application is to solve the problems of single, insufficient stability, high delay and application limitations in traditional music visualization technology, and to propose a music and light visualization method based on sound-light mapping.

[0006] The technical solution of the present application is: a music and light visualization method based on sound-light mapping, comprising the following steps:

[0007] Collect and pre-analyze audio signals;

[0008] The noise in the audio signal is filtered out, the uniformity and stability of the audio signal are improved, and the audio details are amplified through an adaptive filtering algorithm, a normalization algorithm and a dynamic compression method to obtain a preprocessed audio signal.

[0009] A multi-layer convolutional neural network model is constructed, the preprocessed audio signal is input into the multi-layer convolutional neural network model, and music features are extracted based on a mel frequency scale.

[0010] A multi-dimensional sound-light parameter mapping relationship is established based on the music features and the light feature parameters.

[0011] According to the mapping relationship, a light signal is generated and output, and music and light visualization are realized.

[0012] As an optimization, the noise in the audio signal is filtered out, the uniformity and stability of the audio signal are improved, and the audio details are amplified through an adaptive filtering algorithm, a normalization algorithm and a dynamic compression method to obtain a preprocessed audio signal, which specifically includes the following steps:

[0013] The adaptive filtering algorithm is used to analyze the environmental noise characteristics of the audio signal in real time, and a waveform opposite to the environmental noise is automatically generated to offset the environmental noise, thereby obtaining a filtered audio signal.

[0014] The normalized algorithm is used to adjust the amplitude of the filtered audio signal to a fixed standard interval, thereby obtaining an amplitude-adjusted audio signal.

[0015] The adjustable compression ratio and threshold are set, the amplitude-adjusted audio signal is dynamically compressed, the audio signal stronger than the preset threshold is attenuated, the audio signal weaker than the preset threshold is enhanced, and the audio details are amplified, thereby obtaining a preprocessed audio signal.

[0016] As an optimization, the method for constructing the multi-layer convolutional neural network model is specifically as follows:

[0017] The AudioTransformer model is used to construct the multi-layer convolutional neural network model, and the cross-entropy loss function is minimized as a training target to train the multi-layer convolutional neural network model.

[0018]

[0019] wherein, represents the cross-entropy loss function, represents the true label corresponding to the input audio signal, represents the probability distribution of the prediction result of each category in the timbre classification, represents the one-hot encoding value of the true label of the input audio signal, represents a natural base logarithm function, represents a prediction probability that an input audio signal belongs to the i-th represents the total number of categories;

[0020] The is calculated by the output layer of the multi-layer convolutional neural network model through a softmax function, and the specific calculation formula is:

[0021]

[0022] wherein, represents a natural base logarithm function, represents the original output value of the i-th category by the multi-layer convolutional neural network model, represents the original output value of the i-th category by the multi-layer convolutional neural network model.

[0023] As a preferred, the preprocessed audio signal is input to the multi-layer convolutional neural network model, and music features are extracted based on the mel frequency scale, specifically including the following steps:

[0024] The preprocessed audio signal is subjected to short-time Fourier transform to obtain a spectrum graph;

[0025] High-dimensional timbre features are extracted from the audio signal according to the spectrum graph, and the high-dimensional timbre features include a mel spectrum graph, a mel frequency cepstrum coefficient, a chroma frequency, and a frequency band energy distribution difference;

[0026] The high-dimensional timbre features are input to the multi-layer convolutional neural network model, and music features are output;

[0027] The music features include volume, tone length, pitch, and timbre.

[0028] As a preferred, the multi-dimensional sound and light parameter mapping relationship includes: volume corresponds to light intensity, tone length corresponds to light duration, pitch corresponds to light frequency / hue, and timbre corresponds to light signal flicker.

[0029] As a preferred, the volume is linearly mapped to the light intensity through a root mean square algorithm;

[0030] The pitch is mapped to the light frequency / hue after Fourier transform analysis of the fundamental frequency distribution;

[0031] The tone length is mapped to the light duration through peak detection and threshold gating;

[0032] The timbre is mapped to the light signal flicker after spectral energy distribution analysis of the database source.

[0033] ​As a preferred, the volume is linearly mapped with the light intensity by the method of root mean square algorithm, specifically:

[0034] The RMS value of the audio signal is taken as the volume size of the audio signal, and the specific calculation formula is:

[0035]

[0036] Wherein, The root mean square value is represented, The number of samples in the frame of the audio signal is represented, The first Sampling point;

[0037] If , the audio signal is regarded as environmental noise, and the light intensity is set to 0;

[0038] If , the light intensity is mapped in proportion to: , wherein, The light intensity is represented, The minimum value is represented, To realize the dynamic effect of the larger the volume, the brighter the light.

[0039] As a preferred, the pitch is mapped with the light frequency / hue after Fourier analysis of the fundamental frequency distribution, specifically: the audio signal is subjected to fast Fourier transform, the direct current component is ignored, and the frequency corresponding to the maximum fast Fourier transform amplitude is taken as the dominant frequency;

[0040] The dominant frequency is mapped to the light frequency by using logarithmic mapping, and the specific mapping formula is:

[0041]

[0042] Wherein, The natural base logarithmic function is represented;

[0043] The light frequency is linearly mapped to the hue value , and the specific mapping formula is:

[0044] ;

[0045] The hue value realizes the visual feedback rule of "the higher the pitch, the more blue-purple the hue".

[0046] As a preferred, the sound length is mapped with the light duration by peak detection and threshold gating, specifically:

[0047] Set the time window of the audio signal analysis frame, and update the visual content at a fixed frame rate, and the change of the statistical sound length of the duration of a certain frequency component in the audio signal is counted;

[0048] If a certain frequency component lasts for at least two audio signal analysis frames, it is considered as a "long sound", and the light color corresponding to the frequency component is maintained unchanged until the frequency component changes;

[0049] By controlling the holding time of the color block in the animation frame, the visual effects of "long sound→long light" and "short sound→flicker" are realized.

[0050] As preferred, the light signal is generated and output according to the mapping relationship, realizing music and light visualization, specifically:

[0051] In the HSV color space, the light signal configuration instruction is generated according to the mapped light parameter;

[0052] According to the light signal configuration instruction, the pulse modulation signal is output through the general input and output interface GPIO to control the brightness, color, duration and flicker frequency of the light output device;

[0053] According to the light signal configuration instruction, the WS2812B LED light strip is used for visual presentation.

[0054] The beneficial effects of the present application are:

[0055] 1. The music and light visualization method based on sound-light mapping disclosed in the present application realizes intelligent and dynamic light and shadow interaction based on music characteristics (such as volume, sound length, tone, synthesis, reverberation, etc.) compared with traditional fixed light effects, enriches artistic creation means, provides new inspiration and expression for artists and creators, and promotes the development of multi-sensory art.

[0056] 2. The present application develops a music visualization system based on accurate sound-light signal mapping relationship, realizes real-time and rapid output of music to light system, and provides rapid and stable guarantee for high timeliness scenes such as large-scale performances, exhibitions and concerts.

[0057] 3. The music and light visualization method based on sound-light mapping disclosed in the present application can help students to better understand rhythm, melody and harmony, etc. through observing the change mode of light, thereby improving learning interest and understanding ability. In addition, this research can also be extended to special education to provide visual music experience for the hearing-impaired, and make up for the limitations of auditory perception. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1The diagram shows a flowchart of a music and light visualization method based on acoustic-optical mapping provided in Embodiment 1 of the present invention.

[0059] Figure 2 The diagram shows how the dominant frequency and pitch of the real-time monitoring audio data with a frequency of 301.5 Hz provided in Embodiment 3 of the present invention are extracted and mapped into color signals after Fourier transform.

[0060] Figure 3 The diagram shows how the dominant frequency and tone of the real-time monitoring audio data with a frequency of 258.4 Hz provided in Embodiment 3 of the present invention are extracted and mapped into color signals after Fourier transform.

[0061] Figure 4 The diagram shown is a real-time volume analysis and monitoring diagram provided in Embodiment 3 of the present invention.

[0062] Figure 5 The diagram shown is a real-time analysis and monitoring diagram of the extracted timbre provided in Embodiment 3 of the present invention.

[0063] Figure 6 The diagram shown is a real-time analysis and monitoring diagram of the extracted pitch provided in Embodiment 3 of the present invention.

[0064] Figure 7 The diagram shown is a real-time analysis and monitoring diagram of the extracted pitch length provided in Embodiment 3 of the present invention.

[0065] Figure 8 The figure shown is a schematic diagram of real-time monitoring of input signal waveform provided in Embodiment 3 of the present invention.

[0066] Figure 9 The figure shown is a schematic diagram of FFT spectrum analysis provided in Embodiment 3 of the present invention.

[0067] Figure 10 The diagram shown is a schematic diagram of the main frequency color mapping provided in Embodiment 3 of the present invention.

[0068] Figure 11 The figure shown is a statistical result of actual audio analysis provided in Embodiment 3 of the present invention. Detailed Implementation

[0069] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the drawings are merely exemplary and are intended to illustrate the principles and spirit of the invention, and are not intended to limit the scope of the invention.

[0070] Example 1:

[0071] like Figure 1 As shown, a method for visualizing music and light based on acousto-optic mapping includes the following steps:

[0072] S1. Collect and pre-analyze audio signals by real-time microphone acquisition or preset audio input mode;

[0073] In the embodiment of the application, the initial data can be obtained from music signals in two ways, namely real-time acquisition and music file import. The sound signals in the environment can be accurately collected by a high-sensitivity microphone. Or directly collect and import audio files, set the sampling rate and pre-analyze the audio duration to achieve different audio data extraction.

[0074] S2. Filter out the noise in the audio signal, improve the uniformity and stability of the audio signal, and amplify the audio details by using adaptive filtering algorithm, normalization algorithm and dynamic compression method, to obtain the preprocessed audio signal;

[0075] The adaptive filtering algorithm is used to analyze the environmental noise characteristics of the audio signal in real time, and automatically generates a waveform opposite to the environmental noise to offset it, effectively eliminating environmental interference such as fan noise, background human voice, etc., to obtain the filtered audio signal;

[0076] The normalization algorithm is used to unify the signal amplitude range, and the amplitude of the filtered audio signal is adjusted to a fixed standard interval to ensure the consistency and stability of subsequent processing, avoid algorithm errors caused by large differences in signal amplitude, and obtain the amplitude-adjusted audio signal;

[0077] By setting an adjustable compression ratio and threshold, the amplitude-adjusted audio signal is dynamically compressed, the signal stronger than the preset threshold is attenuated, and the signal weaker than the preset threshold is enhanced, to obtain the preprocessed audio signal, thereby significantly improving the adaptability of the signal dynamic range and making the audio details clearer and more discernible.

[0078] S3. Construct a multi-layer convolutional neural network model, input the preprocessed audio signal into the multi-layer convolutional neural network model, and extract music features based on the Mel frequency scale;

[0079] S4. Establish a multi-dimensional sound-light parameter mapping relationship based on music features and light feature parameters;

[0080] S5. Generate and output light signals according to the mapping relationship to realize music and light visualization.

[0081] In this embodiment, the method for constructing a multi-layer convolutional neural network model is specifically:

[0082] A multi-layer convolutional neural network model is constructed using an AudioTransformer model, and a minimum cross-entropy loss function is used as a training target to train the multi-layer convolutional neural network model, wherein the cross-entropy loss function is:

[0083]

[0084] wherein, represents a cross-entropy loss function, represents the true label corresponding to the input audio signal, represents the probability distribution of the prediction result of each category in timbre classification, represents the one-hot encoded value of the true label of the input audio signal, if the input audio signal belongs to the i-th category, , otherwise , represents a natural base logarithm function, represents the prediction probability that the input audio signal belongs to the i-th category, represents the total number of categories;

[0085] The is calculated by the output layer of the multi-layer convolutional neural network model through a softmax function, and the specific calculation formula is:

[0086]

[0087] wherein, represents a natural base logarithm function, represents the original output value of the i-th category of the multi-layer convolutional neural network model, represents the original output value of the i-th category of the multi-layer convolutional neural network model; During the training process, by minimizing the cross-entropy loss function, the model gradually adjusts its parameters to make the predicted probability distribution consistent with the true label

[0088] , so as to realize accurate instrument classification. The minimization process of the loss function is carried out through a back propagation algorithm to optimize the model weights. In the embodiment, the preprocessed audio signal is input into the multi-layer convolutional neural network model, and music features are extracted based on the mel frequency scale, specifically including the following steps:

[0089] Performing short-time Fourier transform on the preprocessed audio signal to obtain a spectrum graph;

[0090] Extracting high-dimensional timbre features in the audio signal according to the spectrum graph, the high-dimensional timbre features including a mel spectrum graph, a mel frequency cepstrum coefficient, a chroma frequency, and a frequency band energy distribution difference;

[0091] Performing short-time Fourier transform on the preprocessed audio signal to obtain a spectrum graph;

[0092] ​​The high-dimensional timbre feature is input into a multi-layer convolutional neural network model, and music features are output.

[0093] In the embodiment, the multi-dimensional sound-light parameter mapping relationship includes: volume corresponding to light intensity, tone length corresponding to light duration, pitch corresponding to light frequency / phase, timbre corresponding to light signal flicker degree, and specific mapping is shown in Table 1.

[0094] Table 1 Multi-dimensional sound-light parameter mapping relationship

[0095]

[0096] The volume is linearly mapped with the light intensity by the root mean square algorithm, and specifically:

[0097] The root mean square (RMS) value of the audio signal is taken as the volume size of the audio signal, and the specific calculation formula is:

[0098]

[0099] wherein, the root mean square value is represented by RMS, the number of samples in the frame of the audio signal is represented by N, the i-th sampling point is represented by xi, If RMS < 0, the audio signal is regarded as environmental noise, and the light intensity is set to 0.

[0100] If RMS > 0, the light intensity is proportionally mapped as: wherein,

[0101] the light intensity is represented by I, the minimum value is represented by Imin, which is used to control the L component in the HSL color model to realize the dynamic effect that the larger the volume is, the brighter the light is. The pitch is mapped with the light frequency / phase after Fourier change analysis of the fundamental frequency distribution, and specifically: the audio signal is subjected to fast Fourier transform, the direct current component is ignored, and the frequency corresponding to the maximum fast Fourier transform amplitude is taken as the dominant frequency; The dominant frequency f is mapped to the light frequency f by using logarithmic mapping, and the specific mapping formula is:

[0102]

[0103]

[0104]

[0105] wherein, the natural base logarithm function is represented by ln; ​​​​​

[0106] Linearly mapping the light frequency to the hue value The specific mapping formula is:

[0107] ;

[0108] The hue value is used to construct the H component in the HSL color model, realizing the visual feedback rule of "the higher the pitch, the more blue-purple the hue".

[0109] The sound length is mapped to the light duration through peak detection and threshold gating, specifically:

[0110] Set the time window of the audio signal analysis frame (for example, 20ms), and update the visual content at a fixed frame rate, and calculate the change of the sound length by the duration of a certain frequency component in the audio signal;

[0111] If a frequency component lasts for at least two audio signal analysis frames, it is considered as a "long sound", and the light color corresponding to the frequency component is maintained unchanged until the frequency component changes;

[0112] By controlling the holding time of the color block in the animation frame, the visual effects of "long sound→long light" and "short sound→flicker" are realized.

[0113] The timbre is mapped to the light signal flickering degree by analyzing the spectrum energy distribution through the database, and the data set is selected: select a publicly available data set with rich instrument annotation, including:

[0114] NSynth (provided by Google, covering a variety of synthetic and real instruments);

[0115] IRMAS (suitable for multi-label multi-instrument identification);

[0116] UrbanSound8K (can be used to expand the identification of common sound sources).

[0117] In this embodiment, the light signal is generated and output according to the mapping relationship, realizing the visualization of music and light, specifically:

[0118] In the HSV color space, detailed light signal configuration instructions are generated according to the mapped parameters, and the instructions are output through the general input and output interface GPIO to generate pulse modulation signals, which are used to accurately control the brightness, color, duration, flickering frequency and other parameters of the light output device. The generated control instructions are visualized through WS2812B LED light strips, and the output light signal configuration instructions are verified with the characteristics of high brightness, low power consumption and high programming control, and the output light signal configuration instructions are accurately presented with a variety of dynamic light effects, realizing the synchronization conversion of music features and dynamic light effects.

[0119] In the embodiment, the sound-light mapping based music and light visualization method further iteratively optimizes the music visualization system through a genetic algorithm in software and collects work feedback in practice, further improves the music and light performance form, specifically: through the genetic algorithm or gradient descent algorithm, the feature extraction, parameter mapping and control logic are adaptively adjusted, the data processing, algorithm and instruction generation for light signal generation are iteratively optimized, the conversion speed and conversion stability of the music real-time conversion to the light signal are improved, and the delay control is within 5 ms; for the completed visualized music works, feedback opinions of users on the light effect performance are collected, whether the light effect conforms to the music emotion, whether it is too dazzling or not is analyzed, and the light effect performance form is further optimized according to the feedback opinions, and the user interactive experience is improved.

[0120] The sound-light mapping based music and light visualization method provided by the application aims to deeply analyze the multi-element mapping of music and light signals, on the basis of sound-light physical properties, along the conversion path of sound-digital-light signals, a precise mapping relationship from sound signals (volume, length, tone, synthesis, reverberation, etc.) to digital signals (time, amplitude, frequency, phase, etc.) to light signals (light intensity, frequency, phase, pulse time, spatial distribution, etc.) is established, and a sound-light parameter mapping algorithm is developed to optimize the stability and speed of the mapping process, realizing multi-dimensional and multi-level fast sound-light signal conversion, providing assistance for intelligent and interactive music visualization systems.

[0121] Embodiment 2:

[0122] On the basis of embodiment 1, the music visualization interactive art device is built in the embodiment of the application, which is used to execute the sound-light mapping based music and light visualization method provided in embodiment 1, realizes the precise conversion of music elements (volume, length, tone, rhythm, etc.) and light parameters (brightness, color, dynamic change, etc.), extracts music features by using audio signal processing technology (such as FFT, MFCC), designs mapping rules combined with synesthesia theory and artistic aesthetic principles, and develops real-time visualization algorithms through Python programming and hardware devices (Arduino / Raspberry Pi), provides technical support for digital art, immersive performance, intelligent education and other fields, and promotes the innovative application of cross-sensory experience.

[0123] The music visual interactive art device specifically acquires music signals in two ways. A high-sensitivity condenser microphone is installed to accurately collect music signals in the environment, including interference noise from musical instruments, music or the environment. Alternatively, different audio files can be imported into the system through a storage device. In the control software, the audio sampling rate is set to 44.1 kHz to ensure audio quality while adapting to the complex frequency changes of different music types. The pre-analysis audio duration is set to 2.5 ms to ensure stable audio data and determine the music speed during the analysis process.

[0124] The Python programming environment is used to analyze the environmental noise characteristics of the collected audio signals in real time through an adaptive filtering algorithm program. The algorithm automatically generates waveforms opposite to the noise to cancel it out, effectively reducing the interference of environmental noise on the music signal. A normalization algorithm is used to process the audio signal amplitude and adjust it to the standard interval of [0, 1] to ensure consistency and stability in subsequent processing and avoid algorithm errors caused by large differences in signal amplitude. Additionally, a dynamic compression ratio of 3:1 and a threshold of -20 dB are set for dynamic compression of the audio signal. After processing, stronger audio signals are appropriately attenuated, and weaker signals are enhanced, making the details of the music, such as subtle drum beats and string tremolo, more clear and distinguishable.

[0125] A Transformer model is built based on the deep learning framework Pytoch. The model includes multiple self-attention layers, normalization layers, and fully connected layers. The pre-processed audio signal is input into the model. The self-attention mechanism is responsible for extracting local features of the audio, the normalization layer normalizes the features to improve the stability of the model, and the fully connected layer integrates and accurately extracts key music features such as spectral features, time domain characteristics, and timbre components. Meanwhile, the librosa library in Python is used to convert the audio signal to a mel spectrum based on the mel frequency scale, highlighting the dominant frequency components in the audio signal. Then, the mel frequency cepstral coefficients are calculated by performing a discrete cosine transform on the mel spectrum, further accurately extracting the timbre features of the audio signal, providing rich music data for subsequent sound-light mapping.

[0126] Determine volume, length, pitch, tone as characteristic parameters of music data, and brightness, duration, frequency / hue, flicker as characteristic parameters of light signal data. The root mean square (RMS) algorithm is used to detect the dynamic loudness change of the audio signal in real time, and a linear mapping relationship between volume and brightness is established; by monitoring the time domain waveform of the audio signal, a suitable peak detection and threshold gate judgment mechanism is set to accurately identify the start and end points of the musical notes or rhythm, and correspond them to the duration of the light signal, and the LED flashes in accordance with the rhythm and is closely synchronized with the music rhythm; the frequency spectrum of the audio signal is analyzed by using the Fourier transform to determine the change of the pitch, and the association between the fundamental frequency and the hue value is established to realize the conversion of the pitch to the light signal frequency / hue, and the high pitch corresponds to the purple light and the low pitch corresponds to the red light; by analyzing the energy distribution of the spectrum, the energy of different frequency bands is counted, and the sound source instrument is judged by combining the pre-established sound characteristic database of the instrument, the flicker frequency of the light signal is adjusted according to the tone source, and the light and dark distribution of the LED lamp strip is adjusted to distinguish different tone sources.

[0127] In the HSV color space, detailed light signal configuration instructions are generated according to the mapped parameters. The control instructions are converted into pulse modulation signals recognizable by the general input and output interface (GPIO) through Python code, and the brightness, color, duration, and flicker frequency of the WS2812B LED lamp strip are accurately controlled by the Arduino development board according to the received signals. When the music enters the exciting climax part, the volume increases, the pitch rises, and the rhythm speeds up, the LED lamp strip quickly adjusts to high brightness, bright colors (such as purple) and fast flashing dynamic light effects; when the music enters the slow paragraph, the lamp strip switches to soft colors (such as light blue) and lower brightness, and the flicker frequency is reduced, accurately presenting various dynamic light effects, realizing the synchronous conversion of music characteristics and dynamic light effects.

[0128] Adopt genetic algorithm to adaptively adjust feature extraction, parameter mapping and control logic. In the process of device operation, the genetic algorithm continuously optimizes the related parameters, improves the speed and stability of the real-time conversion of music to light signal, and ensures that the optimized delay control is within 5 ms. Based on the completed visual music works, the audience's opinions on the experience feedback are collected, and the mapping rules and algorithms are iteratively optimized in terms of light effect changes, color combinations, visual effects, and sound-light integration, interactivity and other issues, so as to improve the user's interactive experience and enable the audience to better appreciate the artistic charm of the integration of music and light.

[0129] Embodiment 3:

[0130] On the basis of embodiment 1, the sound collecting module, the control unit and the light output device are used to build the hardware and system used in the music and light visualization method based on sound-light mapping in embodiment 1.

[0131] The sound acquisition device consists of a high-sensitivity microphone to achieve real-time music acquisition. Simultaneously, for the acquired audio files, an external storage device containing the files is connected to the control unit via a USB interface. A Raspberry Pi is selected as the core control unit to meet the requirements of complex algorithm operation and multi-device connection. An Arduino Uno development board is used to assist in processing high-real-time conversion data. The light output device is a WS2812B LED strip, whose data pins receive high-speed output signals from the Arduino to control the light signals.

[0132] A Python script was written to configure both microphone acquisition and audio input methods. A 44.1kHz audio sampling rate was used to ensure audio data quality, with a single audio analysis step size of 0.025s, the value of which determines the music tempo. The audio spectrum was calculated using Fast Fourier Transform, ignoring DC components and extracting the dominant frequencies. Based on the A4=440Hz standard pitch system, the frequencies were mapped to MIDI note numbers. An adaptive filtering algorithm was used to remove environmental noise, and a normalization algorithm was applied to unify the audio signal amplitude to the [0,1] range. Dynamic compression was used, with a threshold set to -20dB and a compression ratio of 2:1, enhancing the capture of details in weak signals. Based on the HSL color model, hue was evenly distributed at 12, with each semitone corresponding to a 30° hue ring. Brightness increased linearly with pitch. Color blocks were updated in real-time by setting square color blocks, and the waveform displayed the time-domain signal in real-time. The volume distribution is displayed in the form of a spectrum graph. The spectrum distribution after Fourier transform is used for subsequent comparison and analysis with the database to determine the timbre source of the audio signal. The pitch and duration are statistically marked on the output color block in the form of data for real-time monitoring of music conversion.

[0133] Based on the hardware system used in Example 1, a complete audio file is input and used in the conversion algorithm to verify the algorithm's stability, such as... Figure 2 and Figure 3 The image shows examples of different audio signals after FFT processing, where the main frequency components are extracted, mapped to standard pitch, and the resulting color block diagrams. Figure 4 , Figure 5 , Figure 6 and Figure 7 The image shows the real-time analysis and monitoring interface for extracted volume, timbre, pitch, and duration. It dynamically generates sine and cosine waves from analog signals and performs FFT transformations in real-time (cyclically). The interface monitors the real-time input signal waveform, FFT spectrum, and color visualization of key frequencies, with different colors mapping to different frequencies. Figure 8 , Figure 9 and Figure 10 The diagram illustrates real-time monitoring of multiple frequency components. (As shown...) Figure 11The statistical analysis of the actual audio shows that the average volume is 0.206 RMS, the frequency range is 20-1771 Hz, the dominant pitch is D#1, the system delay is 2.4 ms, and the note interval is 0.72 s.

[0134] Based on the Python language environment, Raspberry Pi is used as the control unit, Python development environment is built, numpy, pandas, matplotlib and other data analysis and visualization libraries are installed, librosa audio processing library is installed, TensorFlow and PyTorch frameworks are used for neural network model running, and Adafruit_NeoPixel is installed in Arduino for controlling WS2812B LED strip.

[0135] Taking Raspberry Pi-Arduino communication-LED strip control as the path, through USB serial conversion link, Python environment is used to write scripts, 44.1 kHz single-channel audio input audio signal is used, threshold is set to-20 dB with dynamic compression, compression ratio is 2:1, PREEMPT_RT is used to reduce audio processing delay, and real-time conversion time is effectively guaranteed to be less than 5 ms. YIN algorithm is used to extract the fundamental frequency, the pitch is corresponded to the hue, and the volume is corresponded to the brightness, Arduino control instructions are generated and then input to the LED strip, control instructions are sent to the LED strip through the establishment of serial protocol, and Arduino receives the instructions, analyzes the corresponding hue and brightness of the LED, and generates light effects.

[0136] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader understand the principles of the present application, and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A method for music and light visualization based on acoustic-optical mapping, characterized by, The method comprises the following steps: Collect and pre-analyze the extracted audio signal; Filter out noise in the audio signal, improve the uniformity and stability of the audio signal, and amplify the audio details by using an adaptive filtering algorithm, a normalization algorithm, and a dynamic compression method to obtain a preprocessed audio signal; A multi-layer convolutional neural network model is constructed, the preprocessed audio signal is input into the multi-layer convolutional neural network model, and music features are extracted based on a mel frequency scale, specifically including the following steps: Performing short-time Fourier transform on the preprocessed audio signal to obtain a frequency spectrum; Extracting high-dimensional timbre features from the audio signal based on the frequency spectrum, wherein the high-dimensional timbre features include a mel spectrum, mel frequency cepstral coefficients, chroma frequency, and frequency band energy distribution difference; Inputting the high-dimensional timbre features into the multi-layer convolutional neural network model to output the music features; The music features include volume, length, pitch, and timbre; A multi-dimensional sound-light parameter mapping relationship is established based on the music features and light feature parameters; According to the mapping relationship, a light signal is generated and output, realizing music and light visualization.

2. The music and light visualization method based on the sound-light mapping according to claim 1, characterized in that, The method for filtering out noise in the audio signal, improving the uniformity and stability of the audio signal, and amplifying the audio details by using an adaptive filtering algorithm, a normalization algorithm, and a dynamic compression method to obtain a preprocessed audio signal specifically includes the following steps: Using an adaptive filtering algorithm to analyze the environmental noise characteristics in the audio signal in real time, and automatically generating a waveform opposite to the environmental noise to offset it to obtain a filtered audio signal; Using a normalization algorithm to adjust the amplitude of the filtered audio signal to a fixed standard interval to obtain an amplitude-adjusted audio signal; Setting an adjustable compression ratio and a threshold, dynamically compressing the amplitude-adjusted audio signal, attenuating the audio signal stronger than the preset threshold, enhancing the audio signal weaker than the preset threshold, and amplifying the audio details to obtain a preprocessed audio signal.

3. The music and light visualization method based on the sound-light mapping according to claim 1, characterized in that, The method for constructing a multi-layer convolutional neural network model specifically includes: Using an AudioTransformer model to construct a multi-layer convolutional neural network model, using a minimum cross-entropy loss function as a training target to train the multi-layer convolutional neural network model, and the cross-entropy loss function is: wherein, represents a cross-entropy loss function, represents a true label corresponding to the input audio signal, represents a probability distribution of the prediction result of each category in timbre classification, represents a one-hot encoding value of the true label of the input audio signal, represents a logarithm function with a natural base, represents a prediction probability that the input audio signal belongs to the i-th category, represents a prediction probability that the input audio signal belongs to the i-th category, represents the total number of categories; The The output of the multi-layer convolutional neural network model is calculated by a softmax function, and the specific calculation formula is as follows: wherein, denotes the natural logarithm function, denotes the raw output value of the multi-layer convolutional neural network model for the class, denotes the raw output value of the multi-layer convolutional neural network model for the class.

4. The music and light visualization method based on the sound-light mapping according to claim 1, characterized in that, The multi-dimensional sound-light parameter mapping relationship includes: volume corresponds to light intensity, length corresponds to light duration, pitch corresponds to light frequency / hue, and timbre corresponds to light signal flicker.

5. The music and light visualization method based on the sound-to-light mapping according to claim 4, characterized in that, The volume is linearly mapped to the light intensity by using a root mean square algorithm; The pitch is mapped to the light frequency / hue after Fourier analysis of the fundamental frequency distribution; The length is mapped to the light duration by peak detection and threshold gating; The timbre is mapped to the light signal flicker after database analysis of the spectral energy distribution.

6. The music and light visualization method based on the sound-to-light mapping according to claim 5, characterized in that, The method for linearly mapping the volume to the light intensity by using a root mean square algorithm specifically includes: The RMS value of the audio signal is used as the volume size of the audio signal, and the specific calculation formula is: wherein denotes the root mean square value, denotes the number of samples within a frame of the audio signal, denotes the th sample point; If then the audio signal is considered ambient noise and the light intensity is set to 0; If then the light intensity is mapped proportionally to: where represents the light intensity, represents the minimum value, is used to implement the dynamic effect that the larger the volume, the brighter the light.

7. The music and light visualization method based on the sound-to-light mapping according to claim 5, characterized in that, The method for mapping the pitch to the light frequency / hue after Fourier analysis of the fundamental frequency distribution specifically includes: performing fast Fourier transform on the audio signal, ignoring the direct current component, and taking the frequency corresponding to the maximum fast Fourier transform amplitude as the dominant frequency; The primary frequency is mapped to the optical frequency using a logarithmic mapping with the specific mapping formula:​ wherein represents a logarithmic function with natural base; Linear mapping of light frequencies to hue values The specific mapping formula is: ； The hue value is used to realize the visual feedback rule of "the higher the pitch, the more blue-violet the hue".

8. The music and light visualization method based on the sound-to-light mapping according to claim 5, characterized in that, The method for mapping the sound length through peak detection and threshold gating and light duration is specifically: The time window of the audio signal analysis frame is set, and the visual content is updated at a fixed frame rate, and the change of the sound length is counted by the duration of a certain frequency component in the audio signal; If a frequency component continuously crosses at least two audio signal analysis frames, it is considered as "long sound", and the light color corresponding to the frequency component is maintained unchanged until the frequency component changes; By controlling the holding time of the color block in the animation frame, the visual effects of "long sound → long light" and "short sound → flicker" are realized.

9. The music and light visualization method based on the sound-to-light mapping according to claim 1, characterized in that, The light signal is generated and output according to the mapping relationship, and the music and light visualization are realized, specifically: In the HSV color space, light signal configuration instructions are generated according to the mapped light parameters; According to the light signal configuration instructions, pulse modulation signals are output through the general input and output interface GPIO to control the brightness, color, duration and flicker frequency of the light output device; According to the light signal configuration instructions, the WS2812B LED light strip is used for visual presentation.

Images