Multi-mode melody and vibration mapping generation method based on AI

Abstract

The invention discloses a multi-mode melody and vibration mapping generation method based on AI. The multi-mode melody and vibration mapping generation method comprises the following steps: S1, constructing an AI melody generation model; s2, realizing real-time mapping from melody to vibration; s3, establishing a personalized curve control and learning mechanism; and S4, supporting multi-modal interactive input. The invention relates to the technical field of artificial intelligence and multimedia interaction, in particular to an AI-based multi-mode melody and vibration mapping generation method, which realizes multi-mode melody generation through an AI melody generation algorithm, and can generate rich and diversified melodies according to user preferences and emotional requirements. A melody-to-vibration real-time mapping mechanism realizes accurate multi-dimensional vibration parameter mapping, and vibration response is more vivid and diverse in combination with a rhythm template; a personalized curve control and learning mechanism can continuously optimize and generate results according to user feedback, and more personalized experience is provided; a multi-mode interaction input mode facilitates interaction between a user and the system, and usability and flexibility are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and multimedia interaction technology, and in particular to an AI-based multi-modal melody and vibration mapping generation method. Background Art

[0002] In existing technologies, the combination of melody generation and vibration feedback is relatively simple, lacking flexible and diverse generation modes and personalized mapping mechanisms. Traditional melody generation methods struggle to generate rich melodies tailored to users' diverse preferences and emotional needs. The melody-to-vibration mapping is typically a simple, fixed correspondence that fails to fully capture the melody's details and emotional dynamics. Furthermore, the method lacks learning and optimization based on user feedback, making it difficult to meet users' increasingly diverse needs. Therefore, there is an urgent need for a method that can achieve multimodal melody generation, real-time, accurate vibration mapping, and personalized learning and optimization. Summary of the Invention

[0003] In view of this, the present invention aims to provide an AI-based multi-modal melody and vibration mapping generation method to achieve real-time and accurate interaction between the physical entity and virtual model of the weak current system, thereby improving the intelligence level and operation and maintenance efficiency of the system.

[0004] The technical solution of the embodiment of the present invention is achieved as follows:

[0005] An AI-based multimodal melody and vibration mapping generation method includes the following steps:

[0006] S1. Build an AI melody generation model;

[0007] S2, achieve real-time mapping of melody to vibration;

[0008] S3. Establish personalized curve control and learning mechanism;

[0009] S4. Support multimodal interactive input

[0010] Wherein: S1, building an AI melody generation model: based on the Transformer, RNN, and VAE deep learning models, inputs user preference parameters, including emotion, rhythm, and pitch range, and outputs melody sequences in MIDI and audio formats;

[0011] S2, achieving real-time mapping of melody to vibration: mapping the AI-generated melody into multi-dimensional vibration parameters, including vibration frequency, intensity, duration, partitioning, etc., to achieve 1:N mapping, so that one note drives the progressive vibration of multiple motors;

[0012] S3, establishing a personalized curve control and learning mechanism: the system records the user's vibration feedback each time, including satisfaction, usage time, common rhythm, etc., and introduces reinforcement learning or recommendation system to personalize the generated melody;

[0013] S4 supports multimodal interactive input: supports users to input voice commands, which are converted into AI generation conditions by the natural language processing module, and also supports image / emotion panel selection for setting the vibration rhythm and emotional tone.

[0014] Preferably, in the AI ​​melody generation model constructed in S1, when achieving rhythm-emotion consistency control, the attention weight is defined Among them E i is the i-th emotion feature vector, R j is the th rhythm feature vector, and f is the function for calculating the correlation between emotion and rhythm.

[0015] Preferably, the S2 realizes the real-time mapping of melody to vibration, and the pitch H and vibration frequency f v The mapping relationship is f v =k1·H+b1, rhythm R and vibration duration t v The mapping relationship is Force D and vibration intensity I v The mapping relationship is I v =k3·D+b3, where k1, b1, k2, b2, k3, and b3 are mapping coefficients.

[0016] Preferably, in the personalized curve control and learning mechanism established in S3, in the reinforcement learning, the reward function R is defined reward =ω1·S+ω2·T+ω3·similarity(R user ,R generated ), where ω1, ω2, and ω3 are weight coefficients, S is satisfaction, T is usage time, and similarity(R user ,R generated ) represents the similarity between the user's commonly used rhythm and the generated rhythm.

[0017] Preferably, in the AI ​​melody generation model constructed in S1, the loss function used in model training is the weighted sum of the cross entropy loss function and the mean square error loss function, so as to optimize the model parameters to make it more in line with the relationship between the user preference input and the melody sequence output.

[0018] Preferably, the S2 realizes real-time mapping of melody to vibration. Different vibration style libraries are preset for different types of music styles, such as classical, pop, rock, etc. Each style library contains a set of specific vibration frequency, intensity, duration and partition combination patterns. The system can automatically match the corresponding vibration style library according to the music style of the generated melody.

[0019] Preferably, in the personalized curve control and learning mechanism established in S3, the system analyzes user feedback data through a clustering algorithm, divides users with similar usage habits and preferences into the same cluster, and adopts different reinforcement learning strategies or recommendation algorithms for users in different clusters to achieve more accurate personalized tuning.

[0020] Preferably, the S4 supports multimodal interactive input. In the process of voice command recognition, a speech recognition technology based on a combination of a hidden Markov model (HMM) and a deep neural network (DNN) is adopted to improve the accuracy of converting voice commands into AI generation conditions; the image / emotion panel selection operation extracts and classifies image features or user-selected emotion panel features through a convolutional neural network (CNN) and converts them into corresponding emotion feature vectors.

[0021] Preferably, in the AI ​​melody generation model constructed by S1, in the thematic generation mode, the model uses knowledge graph technology to construct an association network between emotions and musical elements, and generates melodies that meet specific emotional tags by traversing the knowledge graph.

[0022] Preferably, the S2 realizes real-time mapping of melody to vibration. In 1:N mapping, the number of driving motors and the distribution weight of vibration intensity are adjusted according to the importance of the notes. Notes with high importance drive more motors and have greater vibration intensity.

[0023] The embodiment of the present invention adopts the above technical solution, which has the following advantages:

[0024] The present invention realizes the generation of melody in various modes through the AI ​​melody generation algorithm, and can generate rich and diverse melodies according to user preferences and emotional needs; the real-time mapping mechanism of melody to vibration realizes accurate multi-dimensional vibration parameter mapping, and the combination of rhythmic templates makes the vibration response more vivid and diverse; the personalized curve control and learning mechanism can continuously optimize the generated results according to user feedback, providing a more personalized experience; the multimodal interactive input method facilitates the interaction between the user and the system, and improves the ease of use and flexibility of the system. In summary, the present invention can bring users a richer and more personalized melody and vibration interaction experience, and has broad application prospects.

[0025] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a flow chart of the overall architecture of the present invention. DETAILED DESCRIPTION

[0028] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and description are to be considered as illustrative in nature and not restrictive.

[0029] It should be noted that the terms "first," "second," "symmetrical," "array," etc. are used only to distinguish descriptions from positional descriptions and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, limitations on features such as "first" and "symmetrical" may explicitly or implicitly include one or more of these features; similarly, when the number of certain features is not limited in the form of words such as "two" or "three," it should be noted that these features also explicitly or implicitly include one or more of the number of features.

[0030] In the present invention, unless otherwise expressly specified or limited, terms such as "installation," "connection," and "fixation" should be understood broadly; for example, they may refer to fixed connection, detachable connection, or integral molding; they may refer to mechanical connection, direct connection, welding, or indirect connection through an intermediate medium; they may refer to internal communication between two components or interaction between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on the specification and drawings in conjunction with specific circumstances.

[0031] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] like Figure 1The present invention provides an AI-based multi-modal melody and vibration mapping generation method, comprising the following steps:

[0033] S1. Build an AI melody generation model;

[0034] S2, achieve real-time mapping of melody to vibration;

[0035] S3. Establish personalized curve control and learning mechanism;

[0036] S4. Support multimodal interactive input

[0037] Among them: S1. Build an AI melody generation model: Based on deep learning models such as Transformer, RNN, VAE, input user preference parameters, including emotions, rhythm, pitch range, etc., and output melody sequences in MIDI format or audio format; support random generation, thematic generation (emotional labels), and sample-based imitation generation of multiple modes, and introduce attention mechanisms to achieve rhythm-emotion consistency control; S2. Realize real-time mapping of melody to vibration: Map the AI-generated melody (MIDI or spectrum data) into multi-dimensional vibration parameters, including vibration frequency, intensity, duration, partition, etc., to achieve 1:N mapping, so that one note drives multiple motors Progressive vibration, and pitch, rhythm, and strength affect different vibration dimensions respectively, and a "rhythm template" can be embedded as a vibration response primitive; S3. Establish a personalized curve control and learning mechanism: The system records the user's vibration feedback each time, including satisfaction, usage time, common rhythm, etc., and introduces reinforcement learning or recommendation system to personalize the generated melody. Users can manually fine-tune the AI-generated melody and vibration response mapping curve, and save it as a custom solution; S4. Support multimodal interactive input: Support users to input voice commands, which are converted into AI generation conditions by the natural language processing module. It also supports image / emotion panel selection for setting the emotional tone of the vibration rhythm.

[0038] Specifically, in the AI ​​melody generation model constructed by S1, when achieving rhythm-emotion consistency control, attention weight is defined Among them E i is the i-th emotion feature vector, R j is the rhythm feature vector, f is the function for calculating the correlation between emotion and rhythm, S2 realizes the real-time mapping from melody to vibration, the pitch H and vibration frequency f v The mapping relationship is f v =k1·H+b1, rhythm R and vibration duration t v The mapping relationship is Force D and vibration intensity I v The mapping relationship is I v=k3·D+b3, where k1, b1, k2, b2, k3, and b3 are mapping coefficients.

[0039] Specifically, in the present invention, S3 establishes a personalized curve control and learning mechanism, and in reinforcement learning, defines the reward function R reward =ω1·S+ω2·T+ω3·similarity(R user ,R generated ), where ω1, ω2, and ω3 are weight coefficients, S is satisfaction, T is usage time, and similarity(R user ,R generated ) represents the similarity between the user's commonly used rhythm and the generated rhythm.

[0040] Specifically, in S1, when constructing an AI melody generation model, the loss function used in model training is the weighted sum of the cross entropy loss function and the mean square error loss function, so as to optimize the model parameters to make it more in line with the relationship between the user preference input and the melody sequence output.

[0041] Specifically, in the present invention, S2 realizes real-time mapping of melody to vibration. Different vibration style libraries are preset for different types of music styles, such as classical, pop, rock, etc. Each style library contains a set of specific vibration frequency, intensity, duration and partition combination patterns. The system can automatically match the corresponding vibration style library according to the music style of the generated melody.

[0042] Specifically, in the present invention, S3 establishes a personalized curve control and learning mechanism. The system analyzes user feedback data through a clustering algorithm, divides users with similar usage habits and preferences into the same cluster, and adopts different reinforcement learning strategies or recommendation algorithms for users in different clusters to achieve more accurate personalized tuning.

[0043] Specifically, the present invention supports multimodal interactive input, and in the process of voice command recognition, a speech recognition technology based on the combination of hidden Markov model (HMM) and deep neural network (DNN) is adopted to improve the accuracy of converting voice commands into AI generation conditions; the image / emotion panel selection operation uses a convolutional neural network (CNN) to extract and classify image features or user-selected emotion panel features, and convert them into corresponding emotion feature vectors.

[0044] Specifically, in the present invention, S1 constructs an AI melody generation model. In the thematic generation mode, the model uses knowledge graph technology to construct an association network between emotions and musical elements, and generates melodies that meet specific emotional labels by traversing the knowledge graph. S2 realizes real-time mapping of melody to vibration. In 1:N mapping, the number of driving motors and the distribution weight of vibration intensity are adjusted according to the importance of the notes. Notes with high importance drive more motors and have greater vibration intensity.

[0045] In this embodiment, the present invention is specifically designed to work as follows:

[0046] 1. Building an AI Melody Generation Model

[0047] (1) Model construction

[0048] Taking the Transformer model as an example, a network consisting of an encoder and decoder is constructed. User preference parameters such as emotion and tempo are converted into vectors through an embedding layer. Feature extraction is performed using a multi-head attention mechanism and a feedforward neural network. The RNN can use either an LSTM or GRU structure, capturing sequential dependencies of input parameters based on time steps to generate melodies. The VAE encodes user preferences into a latent vector, which is then sampled and decoded to generate melodies, increasing diversity.

[0049] (2) Model training

[0050] We collected music datasets annotated with sentiment and rhythm to train the model. We used the weighted sum of the cross-entropy and mean squared error loss functions as the loss. By adjusting the parameters through backpropagation, we reduced the loss value and improved the model's understanding of user preferences and the accuracy of melody generation.

[0051] (3) Generation mode implementation

[0052] Random Generation: Based on user-defined parameters, notes are randomly sampled from the model output space and combined into a melody according to musical rules.

[0053] Thematic generation: Users input emotional labels, such as "happy," and the model converts the labels into quantities. Combined with other parameter inputs, the model generates corresponding melodies based on the learned mapping relationship, often using bright notes, fast tempos, and harmonious chords.

[0054] Sample-based imitation generation: Users upload sample melodies, and the model extracts features such as pitch and rhythm, and generates similar melodies based on user preferences, such as incorporating the unique rhythmic pattern of the sample.

[0055] (4) Rhythm-emotion consistency control

[0056] In the Transformer model, the formula Calculate the attention weights and let the model generate melodies based on the weights, paying attention to rhythmic features according to emotional dynamics, such as "sad" emotions corresponding to slow melodies.

[0057] 2. Realize real-time mapping of melody to vibration

[0058] (1) Hardware preparation

[0059] Smart wearable or multimedia devices are equipped with multiple vibration motors, arranged in zones, such as in smart wristbands. The devices also include data processing and communication modules to receive and process melody data and control the motors.

[0060] (2) Mapping relationship calculation

[0061] Pitch and vibration frequency mapping: extract the melody pitch, press f v =k1·H+b1 to calculate the vibration frequency, k1 and b1 are predefined coefficients, and high notes correspond to high-frequency vibrations.

[0062] Rhythm and vibration duration mapping: extract rhythm, Calculate the vibration duration, fast rhythm corresponds to short vibration duration.

[0063] Velocity and vibration intensity mapping: Get the melody intensity, press I v =k3·D+b3 to calculate the vibration intensity, and a larger force corresponds to a stronger vibration.

[0064] (3) 1:N mapping implementation

[0065] Depending on the importance of the notes, the core notes drive more and stronger vibration motors, while the non-core notes drive fewer, achieving progressive vibration.

[0066] (4) Rhythm template application

[0067] Select a rhythm template based on the emotional style of the melody. For example, for a romantic melody, choose a wavy template, in which the vibration intensity changes in a wave-like manner; for a rock melody, choose a stab template to produce an instantaneous strong vibration. Templates can also be combined.

[0068] 3. Establishing a personalized curve control and learning mechanism

[0069] (1) User feedback data collection

[0070] When users use the device, the system records satisfaction (through the application evaluation button) and usage time, and analyzes the rhythm parameters set by the user multiple times to obtain the commonly used rhythm.

[0071] (2) Personalized Tuning Implementation

[0072] Reinforcement learning method: based on the reward function R reward =ω1·S+ω2·T+ω3·similarity(R user ,R generated ), using user feedback as a reward, adjusting model parameters, high satisfaction corresponds to high rewards, and the model generates melodies of similar styles.

[0073] Recommendation system approach: Use clustering algorithms to divide users into clusters, use different recommendation algorithms for different clusters, and update strategies based on real-time user feedback.

[0074] (3) Manual fine-tuning and saving by the user

[0075] The app provides an interface for users to adjust the slope and offset of the mapping curve, such as increasing the slope of the pitch and vibration frequency mapping, and save it as a custom scheme for direct use next time.

[0076] 4. Support multimodal interactive input

[0077] (1) Voice command interaction

[0078] Application of speech recognition technology: The device integrates a speech recognition module that combines HMM and DNN, pre-processes the speech signal and then inputs it into the model to recognize text.

[0079] Converting instructions into AI-generated conditions: The natural language processing module parses text, extracts key information such as emotions and rhythm, and converts it into an AI-generated condition input model.

[0080] (2) Image / Emotion Panel Interaction

[0081] Image feature extraction and classification: The user selects an image that represents emotion, and the device uses CNN to extract features, classify them, and convert them into emotion feature vectors.

[0082] Emotion panel selection processing: When the user selects an emotion panel label, the system directly converts it into the corresponding emotion vector and inputs it into the model together with other parameters.

[0083] The following are several specific embodiments of the present invention:

[0084] Example 1

[0085] Combined with the immersive experience of the smart home system: In the smart home environment, the melody and vibration mapping generation method of the present invention is integrated into the home audio and smart lighting system. When the user uses the voice command to "create a relaxing night atmosphere", the system first uses the natural language processing module to parse the command and extract the emotional keyword "relaxed". Based on this emotional label, the AI ​​melody generation model combines the system's preset night scene rhythm preferences (such as a soothing medium-slow rhythm) and the appropriate tone range (such as a soft mid-low range) to generate a melody that meets the requirements;

[0086] The mapping of melody to vibration is linked with the smart lighting system. At this time, the vibration is no longer limited to the audio equipment itself, but controls the flashing frequency and brightness changes of the lights through the smart socket. For example, notes with higher pitches correspond to faster flashing and higher brightness of the lights, and vice versa for notes with lower pitches; the speed of the rhythm determines the interval time of the light flashing; the strength affects the amplitude of the light brightness change. In this way, a multi-modal immersive experience of vision, hearing, and touch (vibration is indirectly perceived through light changes) is achieved. At the same time, the smart home system records feedback data such as the user's usage time in different scenarios, adjustments to the atmosphere, etc., for subsequent personalized optimization.

[0087] Example 2

[0088] Interactive feedback in virtual reality games: This invention is applied in virtual reality (VR) game scenarios. When a player enters the game, the game system inputs the corresponding emotions and rhythm requirements to the AI ​​melody generation model based on the current game scene and plot. For example, in a tense and exciting battle scene, the model generates a melody with a tight rhythm, high pitch, and strong impact;

[0089] The mapping of melody to vibration is achieved through the VR gloves and vests worn by the players. The micro-vibration motors in the VR gloves generate vibrations of different frequencies and durations according to the pitch and rhythm of the melody, simulating the various "tactile" feedback felt by the player's hands. For example, the vibration during an attack is related to the rhythm and strength of the weapon swing. The VR vest uses vibration modules distributed in different positions to provide players with vibration feedback on different parts of the body according to the changes in the intensity of the melody and partition mapping, thereby enhancing the player's immersion and interactivity in the game. Data such as the player's operating performance and dwell time during the game are collected and used to optimize the generation of melodies and vibrations in subsequent game scenes to better match the player's gaming experience needs.

[0090] Example 3

[0091] Rehabilitation therapy auxiliary application: In the field of rehabilitation therapy, embodiments are designed for patients undergoing limb motor function rehabilitation. When patients undergo rehabilitation training, the sensors on the treatment equipment collect the patient's movement data in real time, including the rhythm and strength of the movements. These data are converted into one of the input parameters of the AI ​​melody generation model, and combined with the patient's desired emotional state (such as relaxation, positivity, etc.). For example, when a patient is performing slow joint movement training, the model generates a soothing melody that matches the rhythm and is in line with the relaxing mood;

[0092] The mapping of melody to vibration is achieved through a small vibration device fixed to the patient's rehabilitation limb. The pitch corresponds to the amplitude of joint movement. The larger the amplitude, the higher the vibration frequency; the rhythm is consistent with the patient's movement rhythm, and the intensity reflects the degree of force applied in the rehabilitation training movement. The greater the force, the greater the vibration intensity. In this way, patients can more intuitively feel their own movement status through auditory and tactile feedback during rehabilitation training. At the same time, the emotional regulation effect brought by music and vibration helps to enhance the patient's enthusiasm for rehabilitation. The system continuously records various data of the patient during the rehabilitation training process, including training effect evaluation, usage time, etc., to provide a basis for the adjustment of the rehabilitation plan and the personalized optimization of melody vibration generation.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various modifications and substitutions within the technical scope disclosed in the present invention, and such modifications and substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An AI-based multi-modal melody and vibration mapping generation method, characterized in that: The following steps are involved: S1. Build an AI melody generation model; S2, realize real-time mapping of melody to vibration; S3. Establish personalized curve control and learning mechanism; S4. Support multimodal interactive input Wherein: S1, building an AI melody generation model: based on the Transformer, RNN, and VAE deep learning models, inputs user preference parameters, including emotion, rhythm, and pitch range, and outputs melody sequences in MIDI and audio formats; S2, achieving real-time mapping of melody to vibration: mapping the AI-generated melody into multi-dimensional vibration parameters, including vibration frequency, intensity, duration, partitioning, etc., to achieve 1:N mapping, so that one note drives the progressive vibration of multiple motors; S3, establishing a personalized curve control and learning mechanism: the system records the user's vibration feedback each time, including satisfaction, usage time, common rhythm, etc., and introduces reinforcement learning or recommendation system to personalize the generated melody; S4 supports multimodal interactive input: supports users to input voice commands, which are converted into AI generation conditions by the natural language processing module, and also supports image / emotion panel selection for setting the vibration rhythm and emotional tone.

2. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: In the S1 AI melody generation model, attention weights are defined when achieving rhythm-emotion consistency control. Among them E i is the i-th emotion feature vector, R j is the th rhythm feature vector, and f is the function for calculating the correlation between emotion and rhythm.

3. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: The S2 realizes the real-time mapping of melody to vibration, and the pitch H and vibration frequency f v The mapping relationship is f v =k1·H+b1, rhythm R and vibration duration t v The mapping relationship is Force D and vibration intensity I v The mapping relationship is I v =k3·D+b3, where k1, b1, k2, b2, k3, and b3 are mapping coefficients.

4. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: In the S3 personalized curve control and learning mechanism, in reinforcement learning, the reward function R is defined reward =ω1·S+ω2·T+ω3·similarity(R user ,R generated ), where ω1, ω2, and ω3 are weight coefficients, S is satisfaction, T is usage time, and similarity(R user ,R generated ) represents the similarity between the user's commonly used rhythm and the generated rhythm.

5. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: In the AI ​​melody generation model constructed in S1, the loss function used in model training is the weighted sum of the cross entropy loss function and the mean square error loss function, so as to optimize the model parameters to make it more in line with the relationship between the user preference input and the melody sequence output.

6. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: The S2 realizes real-time mapping of melody to vibration. Different vibration style libraries are preset for different types of music styles, such as classical, pop, rock, etc. Each style library contains a set of specific vibration frequency, intensity, duration and partition combination patterns. The system can automatically match the corresponding vibration style library according to the music style of the generated melody.

7. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: In the personalized curve control and learning mechanism established in S3, the system analyzes user feedback data through a clustering algorithm, divides users with similar usage habits and preferences into the same cluster, and adopts different reinforcement learning strategies or recommendation algorithms for users in different clusters to achieve more accurate personalized tuning.

8. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: The S4 supports multimodal interactive input. During the voice command recognition process, it adopts voice recognition technology based on the combination of hidden Markov model (HMM) and deep neural network (DNN) to improve the accuracy of converting voice commands into AI generation conditions; the image / emotion panel selection operation uses a convolutional neural network (CNN) to extract and classify image features or user-selected emotion panel features and convert them into corresponding emotion feature vectors.

9. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: In the AI ​​melody generation model constructed by S1, in the thematic generation mode, the model uses knowledge graph technology to construct an association network between emotions and musical elements, and generates melodies that meet specific emotional tags by traversing the knowledge graph.

10. The AI-based multi-modal melody and vibration mapping generation method according to claim 1, characterized in that: The S2 realizes real-time mapping of melody to vibration. In 1:N mapping, the number of driven motors and the distribution weight of vibration intensity are adjusted according to the importance of the notes. Notes with high importance drive more motors and have greater vibration intensity.

Images