Lighting scene generation method based on acousto-optic interaction and intelligent control system for realizing acousto-optic interaction

Abstract

The invention provides a lighting scene generation method based on acousto-optic interaction and an intelligent control system for realizing acousto-optic interaction. The lighting scene generation method based on acousto-optic interaction comprises the following steps: acquiring audio track information of target music; processing the audio track information to obtain audio data; identifying the audio data to determine features of the target music; and controlling the lamp to generate an illumination scene matched with the target music according to the characteristics. Compared with the prior art, perfect fusion of music and light can be realized, and the immersion of audiences is enhanced.

Description

Technical Field

[0001] This invention relates to a method for generating lighting scenes based on sound and light interaction and an intelligent control system for realizing sound and light interaction, belonging to the field of sound and light linkage technology. Background Technology

[0002] As society develops, people's demands for entertainment are gradually increasing. They are no longer satisfied with simple auditory experiences and are beginning to require comprehensive sensory entertainment. As one of the channels for conveying visual effects, lighting is becoming increasingly important.

[0003] In homes or entertainment venues, music is often played with only simple lighting effects. The lighting does not change with the music, resulting in a lack of consistency and an overall poor effect that fails to immerse the audience.

[0004] In view of this, it is indeed necessary to improve the existing methods and control systems for audio-visual interaction in order to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a lighting scene generation method based on sound and light interaction, which can achieve a perfect integration of music and light and enhance the audience's immersion.

[0006] To achieve the above objectives, the present invention provides a method for generating lighting scenes based on audio-visual interaction, comprising:

[0007] Obtain the audio track information of the target music;

[0008] The audio track information is processed to obtain audio data;

[0009] Identify audio data to determine the characteristics of the target music;

[0010] Based on the characteristics, the lighting fixtures are controlled to generate a lighting scene that matches the target music.

[0011] As a further improvement of the present invention, the features include: musical style and melodic characteristics.

[0012] As a further improvement of the present invention, identifying the musical style of the target music by recognizing audio data includes: using a signal processing algorithm to detect periodic pulses in the audio data and identifying the strong beats therein;

[0013] Calculate the time interval between adjacent strong beats, determine the beat type and the beat combination methods between different beat types, in order to determine the musical style of the target music.

[0014] As a further improvement of the present invention, identifying the musical style of the target music by recognizing audio data also includes: recognizing the timbre characteristics of the audio data, and obtaining the type of instrument playing the target music from the timbre characteristics;

[0015] The musical style of the target music is determined based on the rhythm combination and the type of instrument.

[0016] As a further improvement of the present invention, identifying the melodic features of the target music by recognizing audio data includes: using a fundamental frequency estimation algorithm to obtain the fundamental frequency information of high or low points based on the audio data, and recognizing its pitch features;

[0017] Identify drum beat features in audio data, and combine pitch features and drum beat features to obtain melody features.

[0018] As a further improvement of the present invention, the melody features also include harmonic features, which are configured to be identified and determined by a pre-trained chord recognition model.

[0019] As a further improvement of the present invention, the lighting scene that matches the target music is generated by controlling the lighting fixtures according to the features, including: dynamically configuring the corresponding light type and light emission mode.

[0020] As a further improvement of the present invention, dynamically configuring the corresponding light type and light emission mode includes:

[0021] The lighting scene is configured according to the music style, and the color temperature and brightness of the lights are dynamically adjusted according to the music style and melody characteristics to complete the configuration process.

[0022] Another objective of this invention is to provide an intelligent control system for realizing audio-visual interaction.

[0023] To achieve the above objectives, the present invention provides an intelligent control system for realizing audio-visual interaction, comprising:

[0024] Lighting fixtures are used for indoor lighting and creating ambiance.

[0025] The audio acquisition module is used to acquire the audio track information of the target music and process the audio track information to obtain audio data;

[0026] The audio processing module is used to identify audio data and determine the musical style and melodic characteristics of the target music;

[0027] The system outputs instructions to the lighting fixtures based on the music style and melody characteristics to control the dynamic configuration of the lighting fixtures and the corresponding light output mode, thereby generating a lighting scene that matches the target music.

[0028] As a further improvement of the present invention, a control module is also included. The control module is used to control the dynamic configuration of the lamps according to the music style and melody characteristics to generate a lighting scene that matches the target music.

[0029] The beneficial effects of this invention are as follows: By acquiring the audio track information of a target music track and processing it to obtain audio data, this invention identifies the characteristics of the target music based on the audio data, thereby enabling the control of lighting fixtures to generate a lighting scene that matches the target music. Compared to existing technologies, this invention achieves a perfect fusion of music and light, enhancing the audience's immersive experience. Attached Figure Description

[0030] Figure 1 This is a flowchart of the lighting scene generation method based on sound and light interaction of the present invention.

[0031] Figure 2 This is a block diagram of the overall structure of the intelligent control system for realizing audio-visual interaction according to the present invention.

[0032] Figure 3 This is a flowchart of a specific embodiment of the present invention.

[0033] Figure label:

[0034] 100 - Control module; 200 - Lighting fixtures; 300 - Audio acquisition module; 400 - Audio processing module; 500 - Lighting scene library. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Please see Figures 1-3 As shown, this invention discloses a lighting scene generation method based on sound and light interaction, applied to a lamp 200, which mainly includes the following steps:

[0037] Obtain the audio track information of the target music;

[0038] The audio track information is processed to obtain audio data;

[0039] Identify audio data to determine the characteristics of the target music;

[0040] Based on the characteristics, the lighting fixtures 200 generate a lighting scene that matches the target music.

[0041] Based on this method, the present invention can achieve a perfect fusion of music and light, enhancing the audience's immersion.

[0042] Specifically, the audio track information is obtained from pre-imported music files. After acquiring the audio track information of the target music, the audio track information is preprocessed, including noise reduction, sampling rate adjustment, and format conversion, to obtain audio data. This ensures the quality of the audio data and avoids affecting the accuracy of subsequent analysis.

[0043] In this embodiment, preprocessing the audio track information to achieve noise reduction can be done using professional audio editing software, such as Adobe Audition. The audio track information file to be processed is imported into the audio editing software, the audio is played, a section of pure noise (i.e., a part without speech or music) is found, and the noise reduction tool or effects in the software are used to collect this noise sample. The collected noise sample is then applied to the entire audio track information file, and the software automatically analyzes and reduces the noise in the audio. Preprocessing the audio track information to adjust the sampling rate can be done using the `-ar` parameter of FFmpeg. Preprocessing the audio track information to perform format conversion can be done using professional audio editing software, such as Adobe Audition, or an online conversion tool can be selected to convert the audio track signal file from one format to another so that it can be played on different devices or software.

[0044] In this embodiment, the features of the target music determined from the audio data mainly include musical style and melodic features. Musical style is determined by the beat pattern and instrument type. Melodic features are determined by pitch features (i.e., fundamental frequency features), drum beat features, and harmonic features.

[0045] Specifically, the steps for identifying the musical style of a target music by recognizing audio data include:

[0046] Signal processing algorithms are used to detect periodic pulses in audio data and identify strong beats within them;

[0047] Calculate the time interval between adjacent strong beats, determine the beat type and the beat combination methods between different beat types, in order to determine the musical style of the target music.

[0048] The strong beats and their combinations determine the rhythm and tempo of music. Common identification / extraction methods include: 1. Using signal processing algorithms, such as autocorrelation functions or template matching, to detect periodic pulses in audio data and identify the strong beats. Alternatively, machine learning models, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), can be used to automatically learn and identify strong beats; there are no limitations on this. 2. Determining the beat type and the combinations of different beat types by calculating the average time interval between adjacent strong beats or estimating the beats per minute (BPM), thereby determining the musical style of the target music.

[0049] We know that time signatures can be 2 / 4, 4 / 4, etc. The musical style of a song usually corresponds to different combinations of different time signatures. That is, we can identify the corresponding musical style based on different combinations. Generally, it's a many-to-many relationship, but the number of combinations far exceeds the number of musical styles.

[0050] Identifying the musical style of target music by recognizing audio data also includes: identifying the timbre features of the audio data and determining the type of instrument playing the target music from these features; and determining the musical style of the target music based on the rhythm combination and instrument type. In other words, timbre features include instrument type and timbre texture, which add rich color to the music. The specific steps for identifying the timbre features of audio data are as follows: using machine learning models, such as Support Vector Machines (SVM), Random Forests, or deep learning models, to train a timbre classifier to identify the instrument type or timbre texture in the audio data; and after identifying the timbre, analyzing its time-domain and frequency-domain characteristics, such as spectral envelope and harmonic structure, to gain a deeper understanding of the characteristics and expressiveness of the timbre. In this way, the instrument type can be determined through timbre features.

[0051] Musical styles typically include genres such as heavy metal, light music, and pop. The lighting scene generation method based on sound and light interaction of this invention uses a trained classification model to classify music. The classification model categorizes music into specific musical styles based on beat combinations and instrument types. The specific steps of the classification model include:

[0052] Data preparation involves extracting effective beat combinations and instrument types from known audio track information, and combining these beat combinations and instrument types with known category labels to construct a labeled dataset for training a classification model.

[0053] Model selection involves choosing a suitable classification model, including logistic regression, support vector machine (SVM), random forest, gradient boosting tree (GBDT), or deep learning models. Deep learning models can include convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants, long short-term memory networks (LSTM), gated recurrent units (GRU), etc.

[0054] Model training involves training a classification model using the extracted features and label dataset, adjusting model parameters, and performing cross-validation to optimize model performance. The classification performance of the model can also be evaluated using a test set, including metrics such as accuracy, recall, and F1 score.

[0055] Model application involves applying the trained classification model to new audio data to automatically classify and determine the music style.

[0056] Optionally, when the trained classification model cannot determine the music style, the present invention can also use a clustering algorithm to group similar audio data, and classify the music by observing the clustering results to determine potential music styles. The specific steps of the clustering algorithm include:

[0057] Data preparation involves extracting effective beat combinations and instrument types from known audio track information. Since clustering algorithms are sensitive to the scale of features, the extracted features need to be standardized.

[0058] Clustering algorithm selection: Choose a suitable clustering algorithm based on the characteristics and needs of the data, including K-means, hierarchical clustering, DBSCAN, or spectral clustering, and set the corresponding parameters. For algorithms such as K-means that require a preset number of clusters, select an appropriate number of clusters based on the actual situation. For other algorithms, you also need to set the corresponding parameters.

[0059] Perform clustering, using the selected clustering algorithm and parameters to cluster the audio data, and analyze the clustering results to understand the commonalities and differences of the audio tracks in each cluster. Visualization tools can be used to assist in the analysis of the clustering results and identify potential musical styles.

[0060] After determining the music style, the audio data is simultaneously identified to determine the melodic features of the target music. In this embodiment, identifying the audio data to determine the melodic features of the target music includes:

[0061] A fundamental frequency estimation algorithm is used to obtain the fundamental frequency information of high or low points from audio data and to identify its pitch characteristics;

[0062] Identify drum beat features in audio data, and combine pitch features and drum beat features to obtain melody features.

[0063] Pitch, intervals, and tone constitute the melodic line of music. In this invention, a fundamental frequency estimation algorithm, such as the YIN algorithm, AMDF algorithm, or autocorrelation method, is used to detect the fundamental frequency in audio data to obtain the pitch. Alternatively, the interval relationships between adjacent notes and the overall tonal trend can be analyzed to determine melodic characteristics.

[0064] The specific steps for obtaining the fundamental frequency information of high or low points from audio data are as follows:

[0065] Audio data is analyzed to obtain audio signals and extract time-frequency features of the audio signals, including spectral centroids and spectral roll-off points.

[0066] Next, peak detection is performed on the audio signal. The peak usually corresponds to the part with higher pitch, that is, the region in the spectrum where the high-frequency components are more prominent. By comparing the spectral energy or spectral centroid within adjacent time windows, the peak of the pitch can be identified, and the fundamental frequency information of the peak can be obtained.

[0067] You can also set high and low thresholds to more accurately locate high and low points. Peaks above the high threshold are high points, and areas below the low threshold are low points.

[0068] For drum beat identification, rhythm analysis algorithms or drum beat detectors can be used to identify drum beats in audio data. Drum beats are located and marked by analyzing the transient characteristics, spectral characteristics, and rhythmic patterns of the audio signal. Specifically, in the waveform of an audio signal, drum beats typically appear as distinct pulses or spikes. The location of drum beats can be manually marked by magnifying the waveform and observing the transient changes in the audio signal. Of course, some professional audio editing software or plugins offer automatic drum beat marking functions, automatically analyzing the audio signal and marking drum beats based on rhythmic patterns and spectral characteristics.

[0069] The melodic features also include harmonic features. Harmonic features include chord progressions, harmonic structures, etc., which reflect the harmonious relationships in music. The harmonic features are configured to be identified and determined by a pre-trained chord recognition model. Specifically, the steps for identifying harmonic features are as follows: using a pre-trained chord recognition model, such as a deep learning-based model, to automatically identify and label chords in audio data; and after identifying the chords, analyzing the progression relationships between chords and their roles in the musical structure, such as chord progressions and harmonic progressions, to determine the harmonic features.

[0070] The present invention controls the lamps 200 to generate a lighting scene that matches the target music based on the music style and melody characteristics. The main steps include: dynamically configuring the corresponding light type and light output mode to complete the configuration process of the lighting scene.

[0071] Specifically, the dynamic configuration of the corresponding lighting types includes: configuring the ambient scene of the luminaire 200 according to the music style. Music styles typically include heavy metal, light music, pop, etc., and the type of skylight can be changed according to different music styles, including clear sky, rainbow, etc., to create different ambient scenes for the luminaire 200. The dynamic configuration of the corresponding light output methods includes: dynamically adjusting the color temperature and brightness of the luminaire 200 according to the music style and melody characteristics. For example, the color temperature of the light can be changed according to the beat combination, instrument type, and pitch and harmony characteristics in the melody; the brightness of the light can be dynamically changed according to the drum beat characteristics.

[0072] like Figure 2 As shown, this invention also provides an intelligent control system for realizing sound and light interaction, applying the aforementioned method for generating lighting scenes based on sound and light interaction, including: a control module 100, a lamp 200, an audio acquisition module 300, an audio processing module 400, and a lighting scene library 500. Through the cooperation of the control module 100, the lamp 200, the audio acquisition module 300, the audio processing module 400, and the lighting scene library 500, sound and light linkage is achieved.

[0073] The control module 100 is used to control the playback of the target music.

[0074] The luminaire 200 is connected to the control module 100 and is used for indoor lighting and atmosphere rendering.

[0075] The audio acquisition module 300 is connected to the control module 100 and is used to acquire the audio track information of the target music. The acquired audio track information is then preprocessed to obtain audio data.

[0076] The audio processing module 400 is connected to the audio acquisition module 300 and the control module 100 respectively. It is used to identify the audio data, determine the musical style and melody characteristics of the target music, and transmit them to the control module 100.

[0077] The lighting scene library 500 is connected to the control module 100 and the luminaire 200, providing the luminaire 200 with a variety of lighting scenes.

[0078] The control module 100 is also used to output commands to the lighting fixtures 200 according to the music style and melody characteristics, so as to control the lighting fixtures 200 to dynamically configure the corresponding light type and light output mode, and generate a lighting scene that matches the target music. In order to better integrate music and light, realize the synchronization of light changes with dynamic music changes, and further enhance the audience's immersion, the control module 100 can also realize functions such as music acceleration, pause, stop, next track, and previous track according to voice commands.

[0079] The lighting scene library 500 includes lighting scenes with light fixtures 200 that match the music style, as well as light emission methods that match the music style and melody characteristics. Preferably, the lighting scenes include color temperatures and brightness that correspond one-to-one with high points, low points, and drum beats.

[0080] The lighting should align with the emotional expression of the music. Specifically, at the climax of a piece of music, which is usually accompanied by intense and uplifting emotions, a higher color temperature can be chosen to create a bright and vibrant atmosphere. A color temperature of 5000K-6500K is preferred, as this cool tone enhances the brightness and dynamism of the music. Simultaneously, the brightness should be increased accordingly to highlight the climax of the music. The brightness should ideally be set at 100%-150% of the base brightness to ensure that the audience can clearly perceive the music's peak moment.

[0081] At the lower points of music, emotions often become softer and deeper, so a lower color temperature should be chosen to create a warm and comfortable atmosphere. The preferred color temperature is between 2700K and 3500K, as this warm tone can increase the intimacy and warmth of the space. At the same time, brightness should be appropriately reduced to match the somber emotional expression of the music. The preferred brightness setting is 50%-75% of the base brightness to avoid overly glaring light affecting the audience's immersion.

[0082] As a crucial rhythmic element in music, drum beats require lighting that creates a powerful impact. When a drum beat occurs, a brief switch to a higher color temperature is recommended, ideally between 5000K and 6000K, to emphasize its power and rhythm. Alternatively, a color temperature that harmonizes with the overall musical style can be maintained to avoid jarring effects; there are no strict limitations on this. Brightness should be dynamically adjusted based on the intensity of the drum beat. When the drum beat is strong, brightness can be increased appropriately, ideally set to 100%-120% of the base brightness to enhance its impact; when the drum beat is weak, brightness should be maintained or slightly reduced to preserve the continuity of the music.

[0083] In this embodiment, the control module 100 can be music playback software or an intelligent control system. After controlling the lamps 200 to generate a lighting scene matching the target music, the control module 100 also generates and displays a playlist of the music, synchronizing music playback with changes in the lighting scene. When the music reaches a specific part, i.e., the target music, the corresponding lighting scene is automatically triggered. The lighting effect of the lighting scene of the lamps 200 can also be viewed by calling the playlist. In practical applications, the lighting scene can also be adjusted in real time according to the on-site effect and audience feedback to achieve the best audiovisual experience. That is, the control module 100 can adjust the type of light that matches the music style, or the color temperature and brightness that correspond to the high points, low points, and drum beats, to achieve emotional resonance and dynamic synchronization.

[0084] In summary, this invention acquires the audio track information of a target music piece, processes the audio track information to obtain audio data, and then identifies the characteristics of the target music based on the audio data. This allows the invention to control the lighting fixtures 200 to generate a lighting scene that matches the target music. Compared to existing technologies, this invention achieves a perfect fusion of music and light, enhancing the audience's immersive experience.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating lighting scenes based on audio-visual interaction, characterized in that, include: Obtain the audio track information of the target music; The audio track information is processed to obtain audio data; Identify the audio data to determine the characteristics of the target music; The lighting fixtures (200) are controlled according to the features to generate a lighting scene that matches the target music.

2. The lighting scene generation method based on audio-visual interaction according to claim 1, characterized in that, The features include: musical style and melodic characteristics.

3. The lighting scene generation method based on audio-visual interaction according to claim 2, characterized in that, Identifying the audio data to determine the musical style of the target music includes: The signal processing algorithm is used to detect periodic pulses in the audio data and identify the strong beats within them; Calculate the time interval between adjacent strong beats, determine the beat type and the beat combination method between different beat types, so as to determine the musical style of the target music.

4. The lighting scene generation method based on audio-visual interaction according to claim 3, characterized in that, Identifying the audio data to determine the musical style of the target music further includes: Identify the timbre characteristics of the audio data, and determine the type of instrument playing the target music based on the timbre characteristics; The musical style of the target music is determined based on the rhythm combination and the instrument type.

5. The lighting scene generation method based on audio-visual interaction according to claim 2, characterized in that, Identifying the audio data to determine the melodic features of the target music includes: A fundamental frequency estimation algorithm is used to obtain the fundamental frequency information of high or low points based on the audio data, and its pitch characteristics are identified. The drum beat features in the audio data are identified, and the melody features are obtained by combining the pitch features and the drum beat features.

6. The lighting scene generation method based on audio-visual interaction according to claim 5, characterized in that, The melody features also include harmonic features, which are configured to be identified and determined by a pre-trained chord recognition model.

7. The lighting scene generation method based on audio-visual interaction according to claim 2, characterized in that, The lighting control unit (200) generates a lighting scene that matches the target music based on the features, including: dynamically configuring the corresponding light type and light emission mode.

8. The lighting scene generation method based on audio-visual interaction according to claim 7, characterized in that, Dynamically configure the corresponding light type and light emission mode, including: The lighting scene is configured according to the music style, and the color temperature and brightness of the lighting fixtures (200) are dynamically adjusted according to the music style and the melody characteristics to complete the configuration process of the lighting scene.

9. An intelligent control system for realizing audio-visual interaction, characterized in that, include: Lighting fixtures (200), used for indoor lighting and atmosphere rendering; The audio acquisition module (300) is used to acquire the audio track information of the target music and process the audio track information to obtain audio data; An audio processing module (400) is used to identify the audio data and determine the musical style and melodic characteristics of the target music; According to the music style and the melody characteristics, the output instructions are sent to the lamp (200) to control the lamp (200) to dynamically configure the corresponding light type and light output mode, and generate a lighting scene that matches the target music.

10. The intelligent control system for realizing audio-visual interaction according to claim 9, characterized in that, It also includes a control module (100), which is used to control the lamps (200) to dynamically configure the corresponding light type and light output mode according to the music style and the melody characteristics, so as to generate a lighting scene that matches the target music.

Images