A music feature-based visual effect generation method and terminal

Abstract

This invention discloses a method and terminal for generating visual effects based on music features. The method includes the following steps: acquiring music data and extracting music features from the music data, the music features including basic music features, emotional features, and musical melody features; establishing a mapping rule library; matching corresponding audio-visual mapping rules from the mapping rule library according to the music features; generating corresponding visual effects using an image rendering engine according to the audio-visual mapping rules; and outputting the visual effects to a display device. This invention extracts multi-dimensional features from music data and maps them to corresponding visual effects to generate multi-layered dynamic visual content, providing real-time audio-visual interactive effects, improving the richness and adaptability of visual effects, and enhancing the user experience.

Description

Technical Field

[0001] This invention relates to the field of computer applications, and in particular to a method and terminal for generating visual effects based on music features. Background Technology

[0002] Audiovisual interaction technology, as an art form that integrates visual and auditory elements, has been widely used in music playback and stage performances in recent years. Existing audiovisual interaction technologies are mainly applied to music visualization software and stage performance effect generation systems. These technologies typically generate basic animation effects, such as waveforms, spectrums, or geometric patterns, by analyzing the rhythm and frequency of music, and are widely used in the visualization functions of music playback devices or players. Simultaneously, in stage performances, some professional systems use music to control lighting, projection, and LED screens to enhance the visual expressiveness of the performance.

[0003] However, current audio-visual interactive technologies have the following drawbacks: 1. Lack of multi-dimensional interactivity: Existing audio-visual interactive technologies are mostly one-way outputs, relying only on basic audio analysis. The generated visual effects are relatively simple, lacking flexibility and interactivity. Viewers or users cannot customize or interact according to their own preferences, making it difficult to achieve a personalized immersive experience. 2. Incomplete recognition of musical features: Most technologies only focus on basic parameters such as frequency and rhythm of music, ignoring richer features such as emotion, timbre, and dynamic changes, resulting in a single level of visual expression and an inability to accurately represent the diverse emotions of music. 3. Limited application scenarios: Existing audio-visual interactive technologies are mostly used for static music visualization and preset stage effect generation, which is difficult to meet the needs of real-time changes, especially in immersive scenarios such as VR and AR, and lacks flexible design that supports multiple platforms. 4. Weak user customization: Traditional technologies do not provide user-friendly customization interfaces, making it difficult for users to customize the mapping rules between sound and image or select specific visual style modes, which cannot meet personalized needs and limits their widespread application in areas such as personal creation and social media.

[0004] In summary, while existing audio-visual interactive technologies have enriched the expressive forms of music and stage performances to some extent, there is still considerable room for improvement in terms of multi-dimensional interactivity, music feature recognition, application scenarios, and user customization functions. Therefore, there is an urgent need for a music feature-based visual effects generation method and terminal that can solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and terminal for generating visual effects based on music features, which can provide real-time audio-visual interactive effects, improve the richness and adaptability of the generated visual effects, thereby meeting user needs and improving user experience.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for generating visual effects based on musical features, comprising the following steps: S1. Acquire music data and extract music features from the music data, including basic music features, emotional features, and music melody features; S2. Establish a mapping rule library, match the corresponding audio-visual mapping rule from the mapping rule library according to the music features, generate the corresponding visual effect using the image rendering engine according to the audio-visual mapping rule, and output the visual effect to the display device.

[0007] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A visual effects generation terminal based on music features includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the aforementioned visual effects generation method based on music features.

[0008] The beneficial effects of this invention are as follows: This invention provides a method and terminal for generating visual effects based on music features. By extracting music features from acquired music data, the method can understand and analyze the music data, laying the foundation for subsequent audio-visual interactive effects. A mapping rule library is established, and corresponding audio-visual mapping rules are matched from the library based on the music features, improving the accuracy and diversity of audio-visual mapping. This results in richer and more adaptable generated visual effects. The corresponding visual effects are generated using an image rendering engine based on the audio-visual mapping rules and output to a display device. The real-time generated dynamic visual effects change synchronously with the music features, increasing the visual dynamism and layering, further enhancing the user experience. Attached Figure Description

[0009] Figure 1 This is a flowchart of a method for generating visual effects based on music features according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a visual effects generation terminal based on music features according to an embodiment of the present invention; Label Explanation: 1. A visual effects generation terminal based on music features; 2. Memory; 3. Processor. Detailed Implementation

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] Please refer to Figure 1 This invention provides a method for generating visual effects based on musical features, comprising the following steps: S1. Acquire music data and extract the music features of the music data, including basic music features, emotional features and music melody features; S2. Establish a mapping rule library, match the corresponding audio-visual mapping rule from the mapping rule library according to the music features, generate the corresponding visual effect using the image rendering engine according to the audio-visual mapping rule, and output the visual effect to the display device.

[0012] As can be seen from the above description, the beneficial effects of the present invention are as follows: by extracting music features from the acquired music data, the music data can be understood and analyzed, laying the foundation for subsequent audio-visual interactive effects; a mapping rule library is established, and corresponding audio-visual mapping rules are matched from the mapping rule library according to the music features, improving the accuracy and diversity of audio-visual mapping, so that the generated visual effects are richer and more adaptable; corresponding visual effects are generated using an image rendering engine according to the audio-visual mapping rules, and the visual effects are output to the display device; the dynamic visual effects generated in real time change synchronously with the music features, increasing the sense of dynamism and layering of the visuals, further enhancing the user experience.

[0013] Furthermore, the extraction of musical features from the music data includes: The music data is subjected to spectral decomposition using an audio analysis algorithm to extract basic music features; The trained emotion recognition model is used to analyze the music data and extract emotion features; A beat detection algorithm is used to identify the beat points and melody changes in the music data and extract the melody features.

[0014] As described above, by employing audio analysis algorithms for spectral decomposition, music data is effectively converted into a frequency domain representation, which helps to analyze and process audio signals more intuitively, extract basic music features, and use a trained emotion recognition model to analyze music data, extract emotional features, identify the emotion type of the music, and improve the efficiency of emotion recognition. By using a beat detection algorithm to identify the beat points and melodic changes in music data and extract musical melody features, it is helpful to analyze the dynamics and rhythm of the music, thus providing a basis for the dynamic expression of subsequent visual effects. Furthermore, the training process of the emotion recognition model includes: Collect a music dataset with sentiment annotations and preprocess the music dataset; The emotion recognition model is trained using a preprocessed music dataset, and a loss function and optimization method are defined to adjust the parameters of the emotion recognition model. The adjusted emotion recognition model is then retrained. Evaluate the performance of the retrained emotion recognition model. When the evaluation metric reaches the preset value, stop training the emotion recognition model and obtain the trained emotion recognition model.

[0015] As described above, by collecting music datasets with sentiment annotations and preprocessing them, the music datasets become more suitable for subsequent analysis, providing rich training data for the sentiment recognition model. Training the sentiment recognition model using the preprocessed music dataset allows it to better capture the emotional features in the music data. Defining loss functions and optimization methods to adjust the model's parameters quantifies the difference between the model's predictions and the true labels, guiding parameter updates, accelerating the convergence process, and improving training efficiency. Retraining the adjusted sentiment recognition model further enhances its performance, and the performance of the retrained model is evaluated. When the evaluation metric reaches a preset value, training is stopped to avoid overfitting and improve the model's generalization ability.

[0016] Furthermore, the establishment of the mapping rule base includes: Establish preset audio-visual mapping rules, and generate a mapping rule library based on the parameters that associate music features and visual effects according to the preset audio-visual mapping rules; When the musical characteristics change, the parameters of the corresponding visual effects in the mapping rule base are modified and updated.

[0017] As described above, by pre-setting audio-visual mapping rules, the parameters of music features and visual effects are associated to generate a mapping rule library, ensuring the synchronization of music features and visual effects. When music features change, the parameters of the corresponding visual effects in the mapping rule library are modified and updated, so that the visual effects can change with the changes in music features, ensuring the real-time performance and dynamic response capability of the system, and enhancing the user's immersion and interactive experience.

[0018] Furthermore, the step of generating corresponding visual effects using an image rendering engine based on the audio-visual mapping rules includes: Use the rendering engine to set up visual layers, and apply different musical features to different visual layers according to preset priorities; The system monitors changes in musical features and switches visual layers based on these changes to generate corresponding visual effects. The visual layers include an emotional feature layer, a rhythmic feature layer, and a melody feature layer.

[0019] As described above, by using the rendering engine to set visual layers and applying different musical features to different visual layers according to preset priorities, resources are effectively allocated and utilized to achieve a more detailed and richer visual expression. By monitoring changes in musical features and switching visual layers according to these changes, the visual effects closely follow the dynamic changes in musical features, enhancing the attractiveness and expressiveness of the visual effects. This allows users to experience the emotions and rhythm of the music more deeply, improving the user experience.

[0020] Please refer to Figure 2 Another embodiment of the present invention provides a visual effects generation terminal based on music features, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the above-described visual effects generation method based on music features.

[0021] The above-described method and terminal for generating visual effects based on music features are suitable for real-time audio-visual interaction, improving the richness and adaptability of visual effects, thereby meeting user needs and enhancing user experience. The following describes specific implementation methods: Please refer to Figure 1 Embodiment 1 of the present invention is: a method for generating visual effects based on musical features, comprising the following steps: S1. Acquire music data and extract the music features of the music data, including basic music features, emotional features and music melody features.

[0022] In this embodiment, an audio input module is designed to build an input interface that supports multiple audio formats, so as to acquire music data in real time, extract music data, obtain music features, understand and analyze music data, and lay the foundation for subsequent realization of audio-visual interactive effects.

[0023] Furthermore, in this embodiment, the extraction of musical features from the music data includes: performing spectral decomposition on the music data using an audio analysis algorithm to extract basic musical features, wherein the audio analysis algorithm includes a Fourier transform algorithm, and the basic musical features include frequency, pitch, and intensity; analyzing the music data using a trained emotion recognition model to extract emotion features, wherein the emotion features include emotion type and emotion dimension; and identifying the beat points and melodic changes in the music data using a beat detection algorithm to extract musical melody features. By analyzing and extracting the music data using the audio analysis algorithm, the emotion recognition model, and the beat detection algorithm respectively, basic musical features, emotion features, and musical melody features are obtained, improving data processing efficiency and providing a basis for the dynamic expression of subsequent visual effects.

[0024] Furthermore, in this embodiment, the training process of the emotion recognition model includes: collecting a music dataset with emotion annotations and preprocessing the music dataset; training the emotion recognition model using the preprocessed music dataset, defining a loss function and optimization method to adjust the parameters of the emotion recognition model, and retraining the emotion recognition model with adjusted parameters; evaluating the performance of the retrained emotion recognition model, and stopping the emotion recognition model training when the evaluation index reaches a preset value, thus obtaining a trained emotion recognition model. By collecting a music dataset with emotion annotations and preprocessing the music dataset, the music dataset is made more suitable for subsequent analysis, providing rich training data for the emotion recognition model. The process involves training an emotion recognition model using a preprocessed music dataset to better capture emotional features from the music data. A loss function and optimization method are defined to adjust the model's parameters, quantifying the difference between the model's predictions and the true labels, guiding parameter updates, accelerating convergence, and improving training efficiency. The adjusted emotion recognition model is then retrained to further enhance its performance. The performance of the retrained model is evaluated, and training is stopped when the evaluation metrics reach preset values ​​to avoid overfitting and improve generalization ability. Preferably, deep learning models are used to train the emotion recognition model. The specific process includes: 1. Data preparation: Data sources: Collect music datasets that include sentiment annotations, such as movie scores, pop music, or specialized sentiment classification datasets (such as the DEAM dataset). Data preprocessing: The music signal is divided into short frames of fixed length (e.g., 10 seconds). Short-time Fourier transform (STFT) is applied to each frame to generate a spectrogram as model input, and the corresponding emotion type (e.g., excitement, joy, melancholy) or emotion dimension (e.g., activity level, pleasure level) for each frame is labeled. 2. Model Selection and Design: Model architecture: Convolutional Neural Networks (CNNs) are used to extract local features from the spectrogram, such as frequency distribution and temporal dynamics; Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) are combined to capture the emotional patterns of music over time; Emotional classification results (such as excitement, melancholy) are output through fully connected layers and activation function layers, or continuous emotional dimensions (such as pleasure) are output through regression layers. Model optimization: Data augmentation techniques (such as time scaling and pitch adjustment) are used to expand the training data and improve the model's generalization ability; attention mechanisms are combined to make the model focus on key parts of the music signal and improve the accuracy of emotion recognition. 3. Model Training: Define loss functions: cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. Optimization method: The Adan optimizer is used to iteratively optimize the model parameters and the learning rate is adjusted to accelerate convergence; Training and validation separation: The dataset is divided into training, validation and test sets to ensure the reliability of model performance; 4. Model Evaluation and Deployment: Evaluation metrics: Use accuracy, F1 score, or mean squared error (MSE) to evaluate model performance; Real-time performance testing: Ensure that the model's sentiment classification latency in real-time audio streams is less than 100ms; Deployment optimization: Convert the model to a lightweight version (such as TensorFlow Lite) to adapt to low-performance devices.

[0025] S2. Establish a mapping rule library, match the corresponding audio-visual mapping rule from the mapping rule library according to the music features, generate the corresponding visual effect using the image rendering engine according to the audio-visual mapping rule, and output the visual effect to the display device.

[0026] In this embodiment, by establishing a mapping rule library, corresponding audio-visual mapping rules are matched from the library based on music features, improving the accuracy and diversity of audio-visual mapping. This results in richer and more adaptable generated visual effects. The corresponding visual effects are generated using an image rendering engine based on the audio-visual mapping rules and output to a display device (such as a projector, screen, or AR / VR device). The real-time generated dynamic visual effects change synchronously with the music features, increasing the visual dynamism and layering. Users can fine-tune the real-time effects via mobile devices or a dedicated control interface, such as adjusting hue, graphics speed, and particle density. Video recording and saving functions are also provided, allowing users to export the generated video effects for later playback or sharing, further enhancing the user experience. Specific implementation methods include: 1. Display Device Adaptation: Provides multiple output interfaces to adapt to different display devices, including projectors, LED screens, AR / VR devices, etc. Ensures that the generated visual content plays synchronously across various resolutions and display environments; 2. Mobile App and Control Interface Support: Users can fine-tune visual effect parameters in real time, such as hue, speed, and particle density, via a mobile application or dedicated control interface. The control interface communicates synchronously with the main system via wireless connection (such as Wi-Fi or Bluetooth).

[0027] 3. Recording and Saving Functions: Enables the recording of real-time generated visual effects into high-quality video files. Encoding options (such as resolution and frame rate) are provided to meet different storage needs, facilitating later sharing and playback.

[0028] Furthermore, in this embodiment, establishing a mapping rule base includes: establishing preset audio-visual mapping rules; generating a mapping rule base by associating music features with visual effect parameters according to the preset audio-visual mapping rules, such as mapping frequency to hue, volume to graphic size, and rhythm to dynamic motion frequency; when music features change, modifying and updating the corresponding visual effect parameters in the mapping rule base; by associating music features and visual effect parameters through preset audio-visual mapping rules, generating a mapping rule base, ensuring the synchronization of music features and visual effects; when music features change, modifying and updating the corresponding visual effect parameters in the mapping rule base, so that the visual effect can change with the change of music features, ensuring the real-time performance and dynamic response capability of the system, and enhancing the user's immersion and interactive experience; wherein, the method of associating music features with visual effect parameters includes: 1. Feature Parameter Standardization: After extraction, musical features (such as frequency, volume, and rhythm) need to be standardized to ensure they are uniformly mapped to visual effect parameters. Where X is the original feature value, and These are the minimum and maximum values ​​of the feature, respectively. These are the standardized feature values ​​(ranging from 0 to 1). 2. Define mapping functions: Define corresponding mapping functions based on different music feature types, including: Frequency to Color (Hue) Mapping: Using the HSV color model, frequency values ​​are mapped to hue ranges. ,in, and These are the upper and lower limits of the hue range, respectively; Volume and graphic size mapping: Enhancing dynamics through exponential function mapping: Where k is a dynamic adjustment factor. and These are the upper and lower limits of the graphic size range, respectively; Rhythm and Dynamic Frequency Mapping: Based on the beat detection results, rhythmic intensity is mapped to movement frequency. ,in, and X represents the upper and lower limits of the dynamic frequency range, respectively; X′ represents the normalized rhythm intensity.

[0029] 3. Multi-dimensional feature fusion: Applying multiple features simultaneously to a single visual parameter enhances the visual depth. For example, combining frequency and rhythm features to calculate particle motion speed using a weighted average. ,in, For frequency normalization, For rhythm standardization value, and As weight; 4. Dynamic parameter adjustment mechanism: Introducing a real-time adjustment factor to adjust the response speed and range of visual parameters according to changes in musical characteristics. ,in, As a smoothing factor, The current parameter value. The parameter value is from the previous frame. This represents the current standardized feature value.

[0030] Furthermore, in this embodiment, the mapping rule base not only includes fixed preset rules, but also supports dynamically adjusting the priority and parameter range of rules based on music characteristics, including: 1. Emotion-driven dynamic adjustment: When the emotional characteristics of music change (such as from "excitement" to "melancholy"), the system automatically switches to the corresponding rule set. For example, when the music is excited, warm colors and high particle density rules are used first; when the music is melancholy, cool colors and slow motion rules are used first. 2. Rhythm-driven dynamic changes: The rule parameters are dynamically modified according to changes in the music rhythm. For example, when the rhythm speeds up, the particle speed range is increased and the transition time is shortened; when the rhythm slows down, the number of particles is reduced and the transition time is extended. 3. Feature Value Significance Adjustment: The priority of rules is dynamically adjusted based on the significance of the current feature values. For example, when the frequency feature changes significantly, frequency-related rules are applied first; when the sentiment feature is significant, sentiment-related rules are applied first. In addition, this embodiment also provides a user-customizable interface, which supports users to manually select and adjust mapping rules. After the user adjusts the rules, the system saves them as new templates and dynamically sorts them according to usage frequency, realizing the preset and saving functions of adjustable parameters. This allows users to reuse custom settings in different scenarios and provides a one-click visual effect generation function for beginners. By combining AI model analysis of music feature data, the system recommends the rule template that best matches the current music features. By matching the default rule library with music features, the system automatically generates the most suitable visual effect. The system calculates the visual effect parameters in real time through parameter interpolation, so that it is seamlessly connected with the music features.

[0031] Furthermore, in this embodiment, the step of generating corresponding visual effects using an image rendering engine according to the audio-visual mapping rules includes: setting visual layers using the rendering engine; applying different musical features to different visual layers according to a preset priority; monitoring changes in musical features; switching visual layers according to the changes in musical features; and generating corresponding visual effects. The rendering engine includes OpenGL, Unity, and Unreal Engine; the visual layers include an emotion feature layer, a rhythm feature layer, and a melody feature layer; and the visual effects include particle diffusion, gradient colors, and geometric pattern movement. The rendering engine is used and its parameters are configured to support real-time rendering of dynamic visual effects. The system utilizes a rendering engine to set up visual layers, applying different musical features to different visual layers according to preset priorities. This effectively allocates and utilizes resources to achieve more detailed and richer visual presentations. It monitors changes in musical features and switches visual layers accordingly, ensuring the visual effects closely follow the dynamic changes in musical features. This enhances the visual appeal and expressiveness, allowing users to more deeply experience the emotions and rhythm of the music and improving the user experience. The visual effects include blinking, diffusion, and trailing effects achieved through a particle system. Random algorithms and a physics engine are used to simulate particle movement, combined with musical rhythm parameters to achieve synchronized changes. The particle effect generation process is as follows: 1. Particle initialization: Define the basic properties of a particle, including position. ,speed Size (s) and lifespan (l); Initialize particle properties using a random algorithm: Where r is a random number in the range [0,1], These represent the positions of the particle in two-dimensional space in the horizontal and vertical directions, respectively. These represent the minimum and maximum position ranges of the particle in the horizontal direction, respectively. These represent the minimum and maximum position ranges of the particle in the vertical direction, respectively. These represent the particle's velocities in the horizontal and vertical directions, respectively. r is a random number ranging from [0,1], used to generate the randomness of the particle's initial position. A reference value representing the initial velocity of a particle. It represents the range of particle velocities and is used to control the randomness of particle velocities; 2. The combination of particle motion parameters and musical characteristics: Adjust particle speed using the music rhythm (R): Where R is the music rhythm parameter, representing the intensity or frequency of the music rhythm, used to control the speed of particle movement, and k is an adjustment factor used to adjust the speed of particle speed response to rhythm. Adjust the number of particles according to the volume (V): ,in, This represents the basic number of particles, i.e., the default number when there is no change in volume. V is an adjustment factor used to control the degree to which the number of particles responds to the volume, and V is a volume parameter that represents the strength of the music volume and is used to adjust the dynamic changes in the number of particles. 3. Implementation of particle effects: Flickering: Particle transparency ( It changes periodically over time: ,in, The blinking frequency controls the periodicity of particle transparency changes, while t is a time variable representing the passage of time for the particle effect. Diffusion: Particle position updates with velocity change: ,in, This is the time increment, used to control the particle's motion update within each time step; Trailing: Records particle trajectories and gradually reduces opacity. ,in, This indicates the transparency of a particle as it travels along its trajectory. The initial transparency of the particle represents its transparency immediately after it is generated. t is the decay factor, used to control the rate at which the particle transparency decays over time, where t is a time variable representing the passage of time in the particle decay process. 4. Particle Destruction: When the particle lifetime (l) is less than 0 or exceeds the display area, the particle is destroyed and a new particle is generated.

[0032] In addition, the visual effects include gradient colors and geometric patterns. By constructing a color gradient algorithm, the visual effects present different colors according to the pitch. Dynamic geometric shapes are generated using a geometric pattern generation module, and motion parameters (such as expansion, contraction, and rotation) are used to simulate patterns that change with the music. Gradient color generation includes: 1. Color mapping rules: Use musical pitch (F) to control the primary hue (H) of the color: ; Volume (V) controls color saturation (S): ; in, For pitch characteristics, H is the primary hue value. Based on the base hue value, S represents the volume characteristic value, and S represents the color saturation. Based on saturation, This is the adjustment coefficient; 2. Gradient Calculation: Interpolate between the two colors (H1, H2): ,in, This is the hue value of the first color (hue 1), representing the starting color of the gradient, ranging from [0, 360]. The hue value of the second color (hue 2) represents the ending color of the gradient, ranging from [0, 360]. t is an interpolation parameter used to control the gradient from... arrive Gradual progress : Indicates the interpolation parameters The hue value at that point, i.e., the current color in the gradient, as... From 0 to 1, It will gradually come from Transition to Specifically, when t=0, the result is: The initial color; when t=1, the result is... The final color is t; when t is between 0 and 1, it indicates a smooth transition between the two colors.

[0033] Dynamic generation of geometric patterns includes: 1. Basic shape generation: Initialize geometric shape parameters: radius (r), rotational angular velocity (r / r). ) and scaling factor(s); 2. Integration with musical characteristics: Use rhythm (R) to control shape scaling: ,in, This represents the base scaling factor, which is preferably set to the default value in the initial state. This is the scaling factor, used to adjust the scaling range. This is the rhythm intensity value, usually obtained from audio analysis, representing the strength of the music's rhythm, and typically ranges from [0,1]. Rotation speed is controlled using frequency (F): ,in, The base rotation speed is preferably set to the default value in the initial state. This is the rotational speed adjustment coefficient, used to increase or decrease the rotational speed by a certain amount. These are frequency values, typically derived from spectral analysis of audio signals, reflecting low- or high-frequency components in music, and usually ranging from [F]. min ,F max ], F min For the lowest frequency value, F max This is the highest frequency value; 3. Dynamic Change Rules: Update geometry parameters every frame: ,in, The current rotation angle, Angular velocity of rotation For time intervals.

[0034] In this embodiment, by implementing a layer grouping mechanism in the rendering engine, different musical features are applied to different visual layers, resulting in rich visual presentations and allowing multiple visual effects to be overlaid. The implementation process includes: 1. Layer grouping mechanism: Grouping strategy: Visual effects mapped from different musical features are assigned to independent layers, mainly divided into: Emotional Feature Layer: Used to control overall tone and background dynamics; Rhythm feature layer: used to realize particle motion or shape changes; Melodic feature layer: used to generate complex geometric patterns and their motion trajectories; 2. Order of music feature processing: Define priority: Prioritize emotional characteristics to set the basic visual style (such as color tone). Rhythmic features are secondary, used to control dynamic effects (such as particle count and motion frequency). Melodic features are used to refine content (such as shape transformation and trajectory generation); Processing completion judgment: Once the color tone stabilizes, the processing of sentiment features is complete. The number and velocity of particles are adjusted within each beat cycle; Once the complete melody segment analysis is complete, update the corresponding pattern generation rules; 3. Dynamic switching mechanism: A dynamic machine controller is introduced to trigger dynamic layer switching by listening to changes in music characteristics. For example, when the rhythm accelerates, particle dynamic effects are enhanced first; when the melody changes, the shape and movement of geometric patterns are adjusted.

[0035] This embodiment supports multi-platform output, including desktop, mobile, and VR / AR devices, and is suitable for various display and interactive scenarios. It also provides API interfaces for easy integration with social media or other content creation platforms, enabling cross-platform content sharing. Specific implementation methods include: 1. Multi-platform output adaptation: Adaptation layers are built for desktop, mobile, and VR / AR devices respectively, enabling the system to run seamlessly on various devices. System resource allocation is optimized to ensure real-time performance and stability.

[0036] 2. API Interface Design: Design an open API interface that allows third-party applications or creation platforms to integrate the audio-visual interaction system, facilitating content sharing and cross-platform dissemination.

[0037] 3. Social Media Integration Support: Through the social media sharing module, users can quickly publish generated dynamic video content to various social media platforms, providing direct share buttons and API calls to simplify the content dissemination process.

[0038] This embodiment also provides the following specific application scenarios: Scenario 1: In concerts and live performances, this system can combine music and visual effects in real time to bring the audience a more stunning multi-sensory experience. In music festivals, concerts and other occasions, this system can generate corresponding dynamic visual effects, such as lighting and LED screen content, based on changes in the music being played on site, to enhance the performance effect. At the same time, the audience can participate in the interaction through mobile devices, adjust the visual style they see, and increase the sense of immersion.

[0039] Scenario 2: This system is ideal for contemporary art exhibitions, providing viewers with an immersive art experience through audio-visual interaction. The artworks in the exhibition can automatically adjust their visual effects, such as color and form changes, in response to changes in the background music, creating different visual sensations. Especially in installation art exhibitions, viewers can participate interactively, for example, by changing the display effects through voice or gestures, exploring the unique relationship between sound and image, and enhancing the interactivity of the artworks.

[0040] Scenario 3: In virtual reality (VR) and augmented reality (AR) applications, users can enjoy an immersive audiovisual experience based on music by wearing VR headsets or AR glasses. The system can generate dynamic visual content in real time in a virtual environment based on background music, such as virtual fireworks and dynamic scenes. It is suitable for virtual concerts, interactive entertainment, virtual art exhibitions, etc., and especially provides players with multi-dimensional immersive audiovisual interaction in VR / AR games.

[0041] Scenario 4: In art education, the system can be used for teaching demonstrations in aesthetic education courses. Teachers can use the audio-visual interactive system to demonstrate the relationship between sound and image, helping students understand and explore the multidimensional expression of music and visual art. For example, in the classroom, students can change the melody or rhythm of the music and observe the changes in visual effects, thereby gaining a more intuitive understanding of the expressiveness of sound and the way emotions are conveyed.

[0042] Scenario 5: In the creation of social media content such as short videos and live streams, creators can use this system to add visual effects to video content to enhance its expressiveness and appeal. Creators can automatically generate matching visual effects based on the emotion and rhythm of the background music, thereby creating creative content that attracts audience attention. In addition, the system supports real-time video output, which can be used to provide viewers with a dynamic audio-visual interactive experience during live streams, increasing interactivity.

[0043] Scenario Six: Music creators can use this system to preview the visual effects corresponding to their music in real time during the creative process, inspiring them and enriching their creative expression. Individual users can also use the system as a music playback tool to experience the real-time interactive effects of music and visuals, enjoying a rich audio-visual experience. Furthermore, users can save the generated effects as videos for personal collection or sharing.

[0044] Scenario 7: The system is also suitable for theme parks, entertainment venues and other scenarios, serving as an interactive display tool for background music and decorative effects. For example, in amusement parks, cafes or bars, it can generate corresponding visual effects based on different background music, such as wall projection or holographic projection, to create an atmosphere. The system can also be adapted to interactive modules, allowing customers to adjust the visual effects according to their own preferences, providing a more attractive interactive experience for the venue.

[0045] Please refer to Figure 2 Embodiment two of the present invention is as follows: A visual effects generation terminal 1 based on music features includes a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, it implements the various steps of the visual effects generation method based on music features according to Embodiment 1.

[0046] In summary, the present invention provides a method and terminal for generating visual effects based on music features. This method acquires music data and extracts its musical features, including: performing spectral decomposition on the music data using an audio analysis algorithm to extract basic musical features; analyzing the music data using a trained emotion recognition model to extract emotional features; and identifying the beat points and melodic changes in the music data using a beat detection algorithm to extract musical melody features. This analysis and extraction of music data provides a basis for the dynamic representation of subsequent visual effects. The training process of the emotion recognition model includes: collecting a music dataset with emotion annotations and preprocessing the music dataset to make the music... The dataset is more suitable for subsequent analysis, providing rich training data for the emotion recognition model. The emotion recognition model is trained using a preprocessed music dataset, enabling it to better capture emotional features in the music data. A loss function and optimization method are defined to adjust the model's parameters, quantifying the difference between the model's predictions and the true labels, guiding parameter updates, accelerating the convergence process, and improving training efficiency. The adjusted emotion recognition model is then retrained to further enhance its performance. The performance of the retrained model is evaluated; when the evaluation metrics reach preset values, training is stopped, resulting in a fully trained emotion recognition model, thus avoiding overfitting. This approach combines various methods to improve the model's generalization ability. It establishes preset audio-visual mapping rules, and generates a mapping rule library by associating music features with visual effects parameters based on these rules. This ensures the synchronization between music features and visual effects. When music features change, the corresponding visual effect parameters in the mapping rule library are modified and updated, allowing the visual effects to change with the music features. This guarantees the system's real-time performance and dynamic response capabilities, enhancing user immersion and interactive experience. Furthermore, it matches corresponding audio-visual mapping rules from the mapping rule library based on the music features, improving the accuracy and diversity of audio-visual mapping to generate richer and more adaptable visual effects. The audio-visual mapping rule utilizes an image rendering engine to generate corresponding visual effects and outputs these effects to a display device. The real-time generated dynamic visual effects change synchronously with the musical characteristics, increasing the visual dynamism and layering. By using the rendering engine to set visual layers and applying different musical characteristics to different visual layers according to preset priorities, resources are effectively allocated and utilized to achieve a more detailed and richer visual presentation. By monitoring changes in musical characteristics and switching visual layers accordingly, the visual effects closely follow the dynamic changes in musical characteristics, enhancing the attractiveness and expressiveness of the visual effects. This allows users to more deeply experience the emotions and rhythm of the music, improving the user experience.

[0047] Therefore, the embodiments of the present invention can achieve the following beneficial effects: 1. Enhanced interactive experience and immersion: By dynamically combining music and visual effects, the system brings a brand-new interactive experience to users and audiences, allowing music and images to change synchronously, enhancing immersion. Audiences can adjust visual effects through the real-time control interface, making the audiovisual interactive experience more personalized and suitable for multi-sensory interaction in scenarios such as concerts and art exhibitions. 2. Multi-level music feature analysis and emotional expression: The system not only extracts basic music features (such as rhythm and frequency), but also identifies music emotions and melodic trends, generating richer visual effects. This makes the visual expression more in line with the emotional changes in the music, bringing the audience a more emotionally engaging audiovisual experience. It is suitable for scenarios such as virtual idols and interactive entertainment. 3. High efficiency in cross-platform adaptability: This system supports multi-platform adaptation and can be applied to desktop, mobile and VR / AR devices, with a wide range of application scenarios. This high efficiency in cross-platform adaptability meets the needs of different users and is suitable for diverse applications such as personal entertainment, education, art education courses and themed venues, improving the system's flexibility and practicality. 4. Powerful user customization features: The system provides personalized settings for audio-visual mapping rules through a user customization interface. Users can freely adjust the style, color, and dynamics of visual effects to meet different aesthetic needs and creative intentions. This feature allows users to customize suitable audio-visual interactive effects according to specific situations or personal preferences, increasing the fun and interactivity of content creation. 5. Wide range of applications and creative possibilities: This system can be applied in a variety of scenarios, including concerts, art exhibitions, education, and social media, and has great potential for creative development. Especially in the fields of modern art and virtual reality, this system is not only a technical tool for audiovisual interaction, but also an innovative platform for artistic creation, providing flexible creative support for artists, educators and content creators.

[0048] 6. Easy to share and spread: The system supports saving real-time generated audio-visual interactive effects as videos or sharing them to social media platforms, enabling users to quickly use visual effects in short video, live streaming and other applications. This is suitable for widespread dissemination. This sharing function not only increases the applicability of the system, but also enhances the attractiveness of user-created content, making it easier to form a larger user base and audience feedback.

[0049] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating visual effects based on musical features, characterized in that, Including the following steps: S1. Acquire music data and extract the music features of the music data, including basic music features, emotional features and music melody features; S2. Establish a mapping rule library, match the corresponding audio-visual mapping rule from the mapping rule library according to the music features, generate the corresponding visual effect using the image rendering engine according to the audio-visual mapping rule, and output the visual effect to the display device.

2. The method for generating visual effects based on musical features according to claim 1, characterized in that, The extraction of musical features from the music data includes: The music data is subjected to spectral decomposition using an audio analysis algorithm to extract basic music features; The trained emotion recognition model is used to analyze the music data and extract emotion features; A beat detection algorithm is used to identify the beat points and melody changes in the music data and extract the melody features.

3. The method for generating visual effects based on musical features according to claim 2, characterized in that, The training process of the emotion recognition model includes: Collect a music dataset with sentiment annotations and preprocess the music dataset; The emotion recognition model is trained using a preprocessed music dataset, and a loss function and optimization method are defined to adjust the parameters of the emotion recognition model. The adjusted emotion recognition model is then retrained. Evaluate the performance of the retrained emotion recognition model. When the evaluation metric reaches the preset value, stop training the emotion recognition model and obtain the trained emotion recognition model.

4. The method for generating visual effects based on musical features according to claim 1, characterized in that, The establishment of the mapping rule base includes: Establish preset audio-visual mapping rules, and generate a mapping rule library based on the parameters that associate music features and visual effects according to the preset audio-visual mapping rules; When the musical characteristics change, the parameters of the corresponding visual effects in the mapping rule base are modified and updated.

5. The method for generating visual effects based on musical features according to claim 1, characterized in that, The step of generating corresponding visual effects using an image rendering engine based on the audio-visual mapping rules includes: Use the rendering engine to set up visual layers, and apply different musical features to different visual layers according to preset priorities; The system monitors changes in musical features and switches visual layers based on these changes to generate corresponding visual effects. The visual layers include an emotional feature layer, a rhythmic feature layer, and a melody feature layer.

6. A visual effects generation terminal based on music features, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: S1. Acquire music data and extract the music features of the music data, including basic music features, emotional features and music melody features; S2. Establish a mapping rule library, match the corresponding audio-visual mapping rule from the mapping rule library according to the music features, generate the corresponding visual effect using the image rendering engine according to the audio-visual mapping rule, and output the visual effect to the display device.

7. A visual effects generation terminal based on music features according to claim 6, characterized in that, The extraction of musical features from the music data includes: The music data is subjected to spectral decomposition using an audio analysis algorithm to extract basic music features; The trained emotion recognition model is used to analyze the music data and extract emotion features; A beat detection algorithm is used to identify the beat points and melody changes in the music data and extract the melody features.

8. A visual effects generation terminal based on music features according to claim 7, characterized in that, The training process of the emotion recognition model includes: Collect a music dataset with sentiment annotations and preprocess the music dataset; The emotion recognition model is trained using a preprocessed music dataset, and a loss function and optimization method are defined to adjust the parameters of the emotion recognition model. The adjusted emotion recognition model is then retrained. Evaluate the performance of the retrained emotion recognition model. When the evaluation metric reaches the preset value, stop training the emotion recognition model and obtain the trained emotion recognition model.

9. A visual effects generation terminal based on music features according to claim 6, characterized in that, The establishment of the mapping rule base includes: Establish preset audio-visual mapping rules, and generate a mapping rule library based on the parameters that associate music features and visual effects according to the preset audio-visual mapping rules; When the musical characteristics change, the parameters of the corresponding visual effects in the mapping rule base are modified and updated.

10. A visual effects generation terminal based on music features according to claim 6, characterized in that, The step of generating corresponding visual effects using an image rendering engine based on the audio-visual mapping rules includes: Use the rendering engine to set up visual layers, and apply different musical features to different visual layers according to preset priorities; The system monitors changes in musical features and switches visual layers based on these changes to generate corresponding visual effects. The visual layers include an emotional feature layer, a rhythmic feature layer, and a melody feature layer.

Images