Music inspiration concrete interaction system

Abstract

The invention relates to the technical field of music generation and artificial intelligence crossing, and discloses a music inspiration concrete interaction system which comprises a gesture acquisition module, a data processing module, a melody contour extraction module, an AI phrase complementation module and a music synthesis output module. Generating high-quality music; real-time performance and creativity are balanced; the self-adaptive capability is strong; the system adopts mature computer vision, a sensor technology and a deep learning algorithm, does not need hardware or algorithm support exceeding the prior art level, and has a good industrialization prospect.

Description

Technical Field

[0001] This invention relates to the field of music generation and artificial intelligence, and in particular to an interactive system for visualizing musical inspiration by generating melody outlines based on gesture drawing and then completing them into complete musical phrases in real time using artificial intelligence. Background Technology

[0002] With the development of computer technology and artificial intelligence, music creation methods are gradually evolving from traditional instruments to digital and interactive approaches. Several existing technologies attempt to convert human movements into music.

[0003] In addition, most existing music generation systems suffer from the following common defects: 1. Unnatural interaction: Most systems rely on preset gestures or fixed actions, lack the ability to analyze continuous and complex gesture trajectories, and cannot achieve subtle expressions in music composition.

[0004] 2. Music generation machinery: Existing systems mostly use mapping to generate music, resulting in insufficient musical expressiveness, lack of artistry and creativity, and inability to meet the needs of professional music creation.

[0005] 3. Lack of real-time structured completion: Existing systems often remain at the level of simple sound effect triggering or style switching, and cannot complete the musical inspiration fragments (such as melody outlines) provided by users into structurally complete and musically rich musical phrases in real time.

[0006] 4. Poor system flexibility: Existing solutions are mostly designed for specific scenarios or devices, such as vehicles or mobile phones, and lack universality and scalability. Summary of the Invention

[0007] The purpose of this invention is to provide an interactive system for visualizing musical inspiration, addressing the problems of unnatural interaction, mechanical music generation, and lack of real-time structured completion capabilities in existing music generation systems. This invention aims to generate melody outlines through gesture drawing and utilize AI to complete them into full musical phrases in real time, thereby making the music creation process more intuitive, interactive, and intelligent.

[0008] To achieve the above objectives, the present invention provides a music inspiration visualization interactive system, characterized by comprising a gesture acquisition module, a data processing module, a melody contour extraction module, an AI musical phrase completion module, and a music synthesis output module: A music-inspired interactive system includes a gesture acquisition module, a data processing module, a melody contour extraction module, an AI phrase completion module, a music synthesis and output module, and extended functional components to enhance system flexibility. The gesture acquisition module is used to collect continuous and complex spatial trajectory data of the user's hand, supporting natural interaction without preset fixed actions, and realizing subtle gesture expression without relying on specific trigger actions; The data processing module processes the collected raw data and maps the gesture movement trajectory to a two-dimensional music parameter space. The horizontal axis of the two-dimensional music parameter space represents time, and the vertical axis represents pitch. It also supports the association mapping of multi-dimensional music parameters (timbre, volume, harmony) to adapt to different creative needs. The melody contour extraction module extracts key feature points from the processed gesture trajectory to generate melody contour data containing pitch sequence and rhythm information, providing a foundation for structured music generation. The AI ​​phrase completion module takes the melody outline data as input and generates complete and musically rich phrase music data in real time, rather than simple sound effect triggers or style switching, to meet the artistic and creative needs of professional music creation. The music synthesis output module converts the generated musical phrase data into audio signals and outputs them. It also supports the loading and switching of various music style templates, adapting to different application scenarios such as music creation, education, and entertainment, thereby improving the system's versatility and scalability.

[0009] Furthermore, the gesture acquisition module includes a depth camera and a wearable IMU sensor, which improves trajectory tracking accuracy through sensor fusion algorithms.

[0010] Furthermore, the data processing module uses a Kalman filter algorithm to smooth the raw data and uses a normalization method to map the gesture coordinates to a preset pitch range and time window.

[0011] Furthermore, the melody contour extraction module simplifies the gesture trajectory using the Douglas-Puk algorithm, extracts key turning points as melody contour feature points, and converts these feature points into MIDI note sequences.

[0012] Furthermore, the AI ​​musical phrase completion module adopts a sequence-to-sequence model based on an attention mechanism, including an encoder, an attention mechanism, and a decoder.

[0013] Furthermore, the encoder is a bidirectional LSTM network used to encode the input melody contour; the attention mechanism calculates the correlation weights between the encoder's hidden state and the decoder's current state; the decoder is a unidirectional LSTM network that autoregressively generates a complete note sequence based on the attention-weighted context vector and historical outputs.

[0014] Furthermore, the AI ​​musical phrase completion module uses an adversarial training strategy, introducing a discriminator to distinguish between model-generated musical phrases and real musical phrases.

[0015] Furthermore, it also includes a user feedback module, where users can adjust the generated musical phrases through gestures or a graphical interface, and the system updates the AI ​​model parameters based on user feedback.

[0016] Furthermore, it supports multi-dimensional music parameter control, allowing users to control the timbre, volume, and harmony parameters of music through specific gestures, where the direction of the gesture movement is consistent with the direction of parameter change.

[0017] Furthermore, it provides a variety of music style templates. Users can select a style template before generating musical phrases, and the AI ​​musical phrase completion module will generate musical phrases of the corresponding style based on the selected style.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Natural and intuitive interaction: The melody outline is generated by drawing with gestures, which conforms to the natural way humans express musical inspiration, lowers the technical threshold of music creation, and realizes a "what you see is what you get" creative experience.

[0019] 2. High-quality music generation: The attention-based Seq2Seq model can understand the musical intent of the melody outline and generate complete and musically rich musical phrases, rather than simple sound effect triggers or style switching.

[0020] 3. Balance between real-time performance and creativity: While maintaining real-time interaction, the system can generate creative music content to meet the needs of professional music creation.

[0021] 4. Strong Adaptability: Through user feedback mechanisms, the system can learn users' creative preferences, continuously optimize the generated results, and form a personalized music creation assistant.

[0022] 5. Technical feasibility: The system adopts mature computer vision, sensor technology and deep learning algorithms, and does not require hardware or algorithm support that exceeds the current level of technology, and has good industrialization prospects. Attached Figure Description

[0023] Figure 1 This is a block diagram of the overall architecture of the interactive system for visualizing musical inspiration according to the present invention.

[0024] Figure 2 This is a schematic diagram of the components of the gesture acquisition module of the present invention.

[0025] Figure 3 This is a schematic diagram of how the data processing module of this invention maps gesture trajectories to the music parameter space.

[0026] Figure 4 This is a schematic diagram of the melody contour extraction process of the present invention.

[0027] Figure 5 This is a schematic diagram of the structure of the AI ​​musical phrase completion model of the present invention.

[0028] Figure 6 This is a schematic diagram of the interactive interface of the user feedback module of the present invention.

[0029] Figure 7 This is a schematic diagram of the system workflow of the present invention.

[0030] Figure 8 This is a schematic diagram of a multi-user collaborative creation scenario in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail with reference to the accompanying drawings and specific embodiments, but the implementation of the present invention is not limited thereto.

[0032] This embodiment includes a gesture acquisition module, a data processing module, a melody contour extraction module, an AI musical phrase completion module, a music synthesis output module, and extended functional components to enhance system flexibility: The gesture acquisition module is used to collect continuous and complex spatial trajectory data of the user's hand, supporting natural interaction without preset fixed actions, and realizing subtle gesture expression without relying on specific trigger actions; The data processing module processes the collected raw data and maps the gesture movement trajectory to a two-dimensional music parameter space. The horizontal axis of the two-dimensional music parameter space represents time, and the vertical axis represents pitch. It also supports the association mapping of multi-dimensional music parameters (timbre, volume, harmony) to adapt to different creative needs. The melody contour extraction module extracts key feature points from the processed gesture trajectory to generate melody contour data containing pitch sequence and rhythm information, providing a foundation for structured music generation. The AI ​​phrase completion module takes the melody outline data as input and generates complete and musically rich phrase music data in real time, rather than simple sound effect triggers or style switching, to meet the artistic and creative needs of professional music creation. The music synthesis output module converts the generated musical phrase data into audio signals and outputs them. It also supports the loading and switching of various music style templates, adapting to different application scenarios such as music creation, education, and entertainment, thereby improving the system's versatility and scalability.

[0033] System Overall Architecture like Figure 1As shown, the music inspiration visualization interactive system provided by this invention includes a gesture acquisition module 101, a data processing module 102, a melody outline extraction module 103, an AI musical phrase completion module 104, and a music synthesis output module 105. Additionally, the system may optionally include a user feedback module 106 and a music style selection module 107.

[0034] When the system is working, the user "draws" melody lines in space using gestures. The gesture acquisition module 101 captures the hand movement trajectory in real time. The data processing module 102 preprocesses the trajectory and maps it to the music parameter space. The melody contour extraction module 103 extracts key feature points from the trajectory to generate a melody contour. The AI ​​musical phrase completion module 104 completes the melody contour into a complete musical phrase. The music synthesis output module 105 converts the musical phrase data into audio and plays it. The user feedback module 106 receives user adjustment opinions and optimizes the generated results.

[0035] Gesture acquisition module implementation method like Figure 2 As shown, the gesture acquisition module 101 includes a depth camera 201 and a wearable IMU sensor 202. The depth camera 201 uses devices such as Microsoft Kinect V2 or Intel RealSense D435, with a acquisition frequency of 30Hz and a resolution of 640x480 pixels, capable of acquiring the three-dimensional coordinates of 21 key points on the user's hand in real time (including the palm, finger joints, etc.). The wearable IMU sensor 202 uses an MPU-9250 chip, integrating a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer, fixed to the back of the user's hand, with a acquisition frequency of 100Hz.

[0036] The sensor fusion algorithm employs complementary filtering combined with extended Kalman filtering to fuse position data from the depth camera and attitude data from the IMU sensor, addressing the noise, drift, and occlusion issues inherent in single-sensor systems. The specific steps are as follows: 1. IMU sensor data preprocessing: Calibrate and temperature compensate the raw data from the accelerometer and gyroscope to eliminate system errors.

[0037] 2. Complementary filtering: Combining the low-frequency characteristics of the accelerometer and the high-frequency characteristics of the gyroscope to estimate the hand attitude angle.

[0038] 3. Extended Kalman Filter: The state vector includes position, velocity, attitude angle, and sensor bias. Through prediction and update steps, it optimizes hand trajectory estimation.

[0039] After sensor fusion, the hand trajectory tracking accuracy reaches ±1mm within a 10cm working range, and the angle accuracy reaches ±0.5°, meeting the requirements of music generation for fine gesture recognition.

[0040] In this embodiment, the gesture acquisition module includes a depth camera and a wearable IMU sensor, and improves trajectory tracking accuracy through a sensor fusion algorithm.

[0041] Example 1: Basic Gesture Acquisition Configuration In this embodiment, the system uses a single depth camera to achieve basic gesture capture. The depth camera is positioned 1.5-2 meters in front of the user, covering the area of ​​the user's gesture activity. The system uses OpenPose or MediaPipe hand keypoint detection algorithms to track the two-dimensional coordinates of 21 hand keypoints in real time. Although this configuration has slightly lower accuracy compared to a dual-sensor solution, it is less expensive and suitable for applications where high accuracy is not required, such as music education or entertainment applications.

[0042] Data processing module implementation method The data processing module 102 receives the raw trajectory data from the gesture acquisition module 101. First, it uses a Kalman filter for smoothing and noise reduction to minimize the impact of hand tremors and environmental noise. The state equation and observation equation for the Kalman filter are as follows: ``` State equation: X(k) = A·X(k-1) + B·u(k) + w(k) Observation equation: Z(k) = H·X(k) + v(k) ``` Where X(k) represents the state vector (position, velocity) at time k, Z(k) represents the observation vector, A, B, and H are coefficient matrices, u(k) is the control variable, and w(k) and v(k) are the process noise and observation noise, respectively.

[0043] Subsequently, the data processing module performs normalization processing, mapping the gesture coordinates to a preset music parameter space. For example... Figure 3 As shown, in the two-dimensional music parameter space, the horizontal axis 301 represents time, and the vertical axis 302 represents pitch. Let the displacement of the gesture trajectory in the X direction be mapped to time variation, and the displacement in the Y direction be mapped to pitch variation. The specific mapping formula is as follows: ``` Pitch = pitch_min + (Y - Y_min) / (Y_max - Y_min) * (pitch_max - pitch_min) Time position = Time start point + (X - X_min) / (X_max - X_min) * Time window length ``` Among them, pitch_min and pitch_max are set according to the user-selected pitch range (such as C4 to C6), and the time window length is 4 bars by default (set according to the current tempo, such as 120 BPM, about 8 seconds).

[0044] In addition, the data processing module also recognizes hand gesture pressure characteristics and reflects the music intensity (volume) through the speed of hand movements. The faster the movement, the greater the intensity of the generated notes.

[0045] In this embodiment, the data processing module uses the Kalman filter algorithm to smooth the raw data and uses a normalization method to map the gesture coordinates to a preset pitch range and time window.

[0046] Example 2: Multi-parameter mapping configuration In this embodiment, the system supports simultaneous mapping of multi-dimensional music parameters. In addition to basic pitch-time mapping, the displacement of a gesture along the Z-axis is mapped to volume changes, the hand rotation angle to timbre brightness changes, and the curvature of the gesture's trajectory to vibrato intensity. This multi-parameter mapping allows a single gesture to control multiple dimensions of the music, greatly enriching its expressive power.

[0047] Melody contour extraction module implementation method like Figure 4 As shown, the melody contour extraction module 103 receives the processed gesture trajectory and first simplifies the trajectory using the Douglas-Puk algorithm to extract key feature points 401. This algorithm retains points in the trajectory whose deviation from the line connecting the start and end points is greater than ε by setting a threshold ε (default value is 2cm). These points correspond to moments when the direction of the gesture movement changes significantly.

[0048] Next, the module converts the key feature points into a MIDI note sequence. The Y-coordinate of each feature point corresponds to the note pitch, and the difference in the X-coordinate between adjacent feature points corresponds to the note duration. Considering the musical scale system, the system quantizes continuous pitches to the nearest semitone, conforming to the twelve-tone equal temperament system of Western music.

[0049] In addition, the melody contour extraction module also identifies the relationship between gesture features and musical expression: The smoothness of gesture movements corresponds to the legato or staccato effect. Gesture acceleration corresponds to note intensity (dynamics). Hand rotation angle corresponds to tone brightness Gesture trajectory width corresponds to harmonic thickness These subtle gestures greatly enrich the possibilities of musical expression, making the interaction more refined and natural.

[0050] In this embodiment, the melody contour extraction module simplifies the gesture trajectory using the Douglas-Puk algorithm, extracts key turning points as melody contour feature points, and converts these feature points into MIDI note sequences.

[0051] Example 3: Melody Contour Quantization Scheme In this embodiment, the system provides multiple scale quantization options, including natural major, harmonic minor, pentatonic scales, and blues scales. Users can select the appropriate scale template according to their creative needs, and the system will automatically quantize the extracted continuous pitches to the closest note of the selected scale. This function ensures that the generated melody conforms to the scale system of a specific musical style, improving the professionalism of the music.

[0052] AI Phrase Completion Module Implementation like Figure 5 As shown, the AI ​​musical phrase completion module 104 adopts a sequence-to-sequence model based on an attention mechanism, including an encoder 501, an attention mechanism 502, and a decoder 503.

[0053] The encoder 501 is a 3-layer bidirectional LSTM network with 512 hidden units per layer. The encoder receives a MIDI note sequence (including pitch, duration, and velocity) from the melody contour extraction module, converts each note into a 128-dimensional embedding vector, and inputs it into the LSTM network to generate the encoder hidden state sequence {h1, h2, ..., h...}. n}

[0054] The attention mechanism 502 employs additive attention, calculating the correlation weight α between the current state of the decoder and each hidden state of the encoder, as shown in the following formula: ``` ``` Where s_t is the current state of the decoder, and v, W1, and W2 are trainable parameters.

[0055] The decoder 503 is a 3-layer unidirectional LSTM network with 512 hidden units per layer. The decoder generates complete note sequences autoregressively based on attention-weighted context vectors and historical outputs until a terminator is generated or the maximum length (32 notes by default) is reached.

[0056] Training process: The model was pre-trained using a large-scale symbolic music dataset (such as the Lakh MIDI dataset), with cross-entropy loss as the loss function, Adam as the optimizer, an initial learning rate of 0.001, and a batch size of 64. During training, a teacher-forced policy was used, with 85% of the historical steps using the true values ​​as input.

[0057] Adversarial Training: To further improve the quality of generated music, a discriminator network is introduced. This network is a 5-layer one-dimensional convolutional neural network used to distinguish generated musical phrases from real musical phrases. The generator (Seq2Seq model) and the discriminator undergo adversarial training. The objective function is as follows: ``` min_θ max_φ [E_{x∼p_data} log D_φ(x) + E_{z∼p_z} log(1 - D_φ(G_θ(z)))] ``` Where G_θ is the generator, D_φ is the discriminator, z is the input melody contour, and x is the actual musical phrase.

[0058] Style Adaptation: The AI ​​phrase completion module supports the generation of multiple music styles. A style conditional vector is added to the last layer of the model. This vector is concatenated with the decoder input to guide the generation process to match the music style selected by the user.

[0059] In this embodiment, the AI ​​musical phrase completion module adopts a sequence-to-sequence model based on an attention mechanism, including an encoder, an attention mechanism, and a decoder.

[0060] Example 4: Lightweight AI Completion Solution In this embodiment, to meet the deployment requirements of mobile devices or resource-constrained environments, the system provides a lightweight AI completion solution. This solution uses a single-layer LSTM network, reducing the number of hidden units to 128, and employs knowledge distillation technology to allow the lightweight model to learn the output distribution of the larger model. Although the quality of the generated music is slightly reduced, the inference speed is improved by 3-5 times, and memory usage is reduced by 70%, making it suitable for application scenarios with higher real-time requirements.

[0061] In this embodiment, the encoder is a bidirectional LSTM network used to encode the input melody contour; the attention mechanism calculates the correlation weight between the encoder's hidden state and the decoder's current state; the decoder is a unidirectional LSTM network that autoregressively generates a complete note sequence based on the attention-weighted context vector and historical outputs.

[0062] Example 5: Personalized Optimization Scheme In this embodiment, the system supports personalized tuning by the user. Users can provide their own musical compositions as training data, and the system fine-tunes the pre-trained model based on this data to make the generated music more in line with the user's creative style. The fine-tuning process uses transfer learning technology, updating only the parameters of the last few layers of the model, achieving personalized adaptation while maintaining the model's generality.

[0063] Music Synthesis Output Module Implementation Method The music synthesis output module 105 receives the MIDI sequence generated by the AI ​​phrase completion module and converts it into an audio signal through a wavetable synthesis or sampling synthesis engine. By default, it uses the FluidSynth software synthesizer in conjunction with the SoundFont sound library (such as General User GS SoundFont) to provide high-quality instrument sounds.

[0064] The audio signal undergoes digital signal processing, including dynamic compression, equalization, and reverberation, to enhance the listening experience. The processed audio signal is then played through audio equipment, providing visual feedback by displaying generated sheet music, piano rolls, or sound wave animations on a screen.

[0065] In addition, the system supports multi-channel output, allowing users to control the main melody, harmonic layers, and rhythmic parts separately through different gestures, thus enabling multi-part music creation.

[0066] In this embodiment, the AI ​​musical phrase completion module uses an adversarial training strategy, introducing a discriminator to distinguish between model-generated musical phrases and real musical phrases.

[0067] Example 6: Multi-tone output configuration In this embodiment, the system supports simultaneous output of multiple timbres. Users can select the timbres of the main melody instruments, harmonic instruments, and percussion instruments before generating musical phrases. The AI ​​phrase completion module will then generate a complete musical segment containing multiple tracks. For example, if the user selects the piano as the main melody, strings as the harmony, and drums as the rhythm, the system will generate a complete arrangement containing these three tracks.

[0068] User Feedback Module Implementation like Figure 6 As shown, the user feedback module 106 provides a graphical interface, including a rhythm fine-tuning slider 601, a pitch correction button 602, and a timbre selection menu 603. Users can adjust the generated musical phrases using these controls.

[0069] The system records user adjustment behaviors as reinforcement learning signals and uses the Proximal Policy Optimization (PPO) algorithm to fine-tune the parameters of the AI ​​phrase completion module. Specifically, user satisfaction is used as the reward function, and the optimization direction is to maximize user satisfaction. ``` Objective function: max E[Σ γ^t R(s_t, a_t)] ``` Where R(s_t, a_t) represents the reward (user satisfaction) obtained by taking action a_t (generating a musical phrase) in state s_t, and γ is the discount factor.

[0070] Through repeated interactions, the system gradually learns the user's music preferences and forms a personalized music generation model.

[0071] System Workflow like Figure 7 As shown, the workflow of the system of the present invention includes the following steps: Step S701: System initialization, loading AI model parameters, setting music style and timbre library.

[0072] Step S702: The gesture acquisition module detects user gesture input. If a creative gesture is detected (such as raising a hand and moving it in the air), the module begins to capture the hand movement trajectory.

[0073] Step S703: The data processing module filters and normalizes the trajectory data and maps it to the music parameter space.

[0074] Step S704: The melody contour extraction module extracts key feature points of the trajectory and generates melody contour data.

[0075] Step S705: The AI ​​phrase completion module generates complete musical phrase data based on the melody outline data.

[0076] Step S706: The music synthesis output module converts the musical phrase data into audio and plays it, while providing visual feedback.

[0077] Step S707: The system checks whether the user is satisfied. If the user makes adjustments through the feedback module, then proceed to step S708; otherwise, end the current creation cycle.

[0078] Step S708: The user feedback module records the adjustment information, updates the AI ​​model parameters, and returns to step S705 to regenerate the musical phrase.

[0079] In this embodiment, a user feedback module is also included, allowing users to adjust the generated musical phrases through gestures or a graphical interface, and the system updates the AI ​​model parameters based on user feedback.

[0080] Example 7: Multi-user collaborative creation scenario like Figure 8 As shown, in multi-user collaborative creation scenarios, the system supports simultaneous recognition of multiple gestures, allowing up to four users to participate in creation simultaneously. Different users can control different musical elements: user A801 ​​controls the main melody, user B802 controls the harmony parts, user C803 controls the rhythm mode, and user D804 controls special sound effects.

[0081] The system distinguishes different users' gestures through hand icons and spatial positioning, assigning each user a different music creation role to achieve collaborative music creation. This multi-user interaction mode is particularly suitable for scenarios such as music education, team building, and artistic performance.

[0082] Comparative Example: Traditional Gesture Music System In contrast, traditional gesture-based music systems typically employ a preset gesture-sound effect mapping mechanism. For example, waving an upward gesture triggers an ascending scale, waving a downward gesture triggers a descending scale, and drawing a circle triggers a repetitive rhythmic pattern. The music generation in such systems relies entirely on preset rules, lacking creativity and flexibility, and cannot generate corresponding musical changes based on subtle variations in gestures, resulting in limited musical expressiveness.

[0083] In this embodiment, multi-dimensional music parameter control is supported. Users can control the timbre, volume, and harmony parameters of music through specific gestures, wherein the direction of the gesture movement is consistent with the direction of parameter change.

[0084] In this embodiment, multiple music style templates are provided. Users can select a style template before generating musical phrases, and the AI ​​musical phrase completion module generates musical phrases of the corresponding style based on the selected style.

[0085] This invention generates melody outlines through gesture drawing and combines them with AI to complete the musical phrases in real time, making the music creation process more intuitive, intelligent, and interactive. The system utilizes mature hardware and deep learning algorithms, requiring no groundbreaking technology, and possesses promising industrialization prospects and commercial value.

[0086] This system can be applied to multiple fields: Music creation: Providing musicians with tools for capturing inspiration and rapid creation. Music Education: Helping students understand music theory through intuitive interaction Entertainment Industry: Providing dynamic music generation capabilities for games and VR / AR environments Music therapy: Achieving musical expression through non-instrumental interaction. In-car entertainment: Providing a safe and interactive in-car music experience. Experimental data and effect verification To verify the effectiveness of this invention, we conducted a user experiment, inviting 30 participants with different musical backgrounds to use the system for music creation. The experimental results show: 95% of participants found the system's interaction method intuitive and easy to use. 88% of participants were satisfied with the quality of the AI-generated music. The average composition time for each piece of music is 2.3 minutes, which is far shorter than traditional composition methods. The system achieves a recognition accuracy of 92.5% and a response latency of less than 150ms. Experimental results show that the present invention is superior to existing technical solutions in terms of naturalness of interaction, music quality, and system performance.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A musical inspiration visualization interactive system, characterized in that, It includes a gesture acquisition module, a data processing module, a melody contour extraction module, an AI phrase completion module, a music synthesis and output module, and extended functional components to enhance system flexibility: The gesture acquisition module is used to collect continuous and complex spatial trajectory data of the user's hand, supporting natural interaction without preset fixed actions, and realizing subtle gesture expression without relying on specific trigger actions; The data processing module processes the collected raw data and maps the gesture movement trajectory to a two-dimensional music parameter space. The horizontal axis of the two-dimensional music parameter space represents time, and the vertical axis represents pitch. It also supports the association mapping of multi-dimensional music parameters. The melody contour extraction module extracts key feature points from the processed gesture trajectory to generate melody contour data containing pitch sequence and rhythm information. The AI ​​phrase completion module uses the melody outline data as input to generate complete and musically rich phrase data in real time. The music synthesis output module converts the generated musical phrase data into audio signals and outputs them, while also supporting the loading and switching of various music style templates.

2. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, The gesture acquisition module includes a depth camera and a wearable IMU sensor, and improves trajectory tracking accuracy through sensor fusion algorithms.

3. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, The data processing module uses a Kalman filter algorithm to smooth the raw data and uses a normalization method to map the gesture coordinates to a preset pitch range and time window.

4. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, The melody contour extraction module simplifies the gesture trajectory using the Douglas-Puk algorithm, extracts key turning points as melody contour feature points, and converts these feature points into MIDI note sequences.

5. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, The AI ​​musical phrase completion module adopts a sequence-to-sequence model based on an attention mechanism, which includes an encoder, an attention mechanism, and a decoder.

6. The interactive system for visualizing musical inspiration according to claim 5, characterized in that, The encoder is a bidirectional LSTM network used to encode the input melody contour; the attention mechanism calculates the correlation weights between the encoder's hidden state and the decoder's current state; the decoder is a unidirectional LSTM network that autoregressively generates a complete note sequence based on the attention-weighted context vector and historical outputs.

7. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, The AI ​​phrase completion module uses an adversarial training strategy, introducing a discriminator to distinguish between model-generated phrases and real musical phrases.

8. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, It also includes a user feedback module, where users can adjust the generated musical phrases through gestures or a graphical interface, and the system updates the AI ​​model parameters based on user feedback.

9. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, It supports multi-dimensional music parameter control, allowing users to control the timbre, volume, and harmony parameters of music through specific gestures, where the direction of the gesture movement is consistent with the direction of parameter change.

10. The interactive system for visualizing musical inspiration according to claim 1, characterized in that, It offers a variety of music style templates. Users can select a style template before generating musical phrases, and the AI ​​musical phrase completion module will generate musical phrases of the corresponding style based on the selected style.

Images