Interactive music intelligent generation method and system for adaptive scene space

Abstract

The present application relates to an interactive music intelligent generation method and system for adaptive scene space. The method constructs a conditional diffusion model framework by sensing the user's behavior, emotional state and spatial environment changes, generates music matching the current scene in real time, achieves a deep integration of music and user experience, and dynamically adjusts the generated music to ensure that the generated music can not only flexibly respond to scene changes, but also meet the user's personalized needs, thereby enhancing the user's interactive experience with the music generation system in the scene and improving the diversity and intelligence of scene solution generation.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method and system for intelligently generating interactive music in an adaptive scene space. Background Art

[0002] Music is an art composed of harmony, melody and rhythm, which is deeply integrated into every aspect of human life. In recent years, with the rapid development of artificial intelligence and perception technology, the demand for personalized and dynamic content generation has increased. Especially in immersive experience scenarios such as augmented reality, smart home and interactive display, music plays a key role as an important element in atmosphere shaping. Traditional music playback methods are often fixed and cannot be dynamically adjusted according to the environment, user's actions, emotions and spatial layout, resulting in a low fit between music and scenes.

[0003] Although existing technologies have tried to adapt to different scenarios through preset music libraries, parameter mapping, symbol generation, and audio generation, they still lack high real-time adaptability and interactivity. In addition, these systems need to be improved in terms of music diversity, personalization, and emotional expression. Users in different scenarios or activities, such as games, exhibitions, fitness, meditation, etc., expect music to be generated according to their current state and scene changes, thereby providing a more immersive and personalized experience. Summary of the invention

[0004] Based on this, it is necessary to provide an interactive music intelligent generation method and system for adaptive scene space to address the above technical problems.

[0005] An interactive music intelligent generation method for adaptive scene space, the method comprising:

[0006] User information data, scene space data and music data are obtained, and the user information data and the scene space data are merged to obtain multi-scene data, wherein the multi-scene data is text data.

[0007] The trained CSMP model is used to process the multivariate scene data to obtain high-dimensional features of the multivariate scene data, and the high-dimensional features of the multivariate scene data are spliced ​​according to the time sequence to obtain a multivariate scene data sequence; the CSMP model is a dual encoder architecture, including a multivariate scene data encoder and a music data encoder.

[0008] The multivariate scene data sequence is processed by interacting with the GPT model to obtain the interactive high-dimensional features of the interactive data.

[0009] The Mel-spectrogram feature map of the music data is extracted, and the Mel-spectrogram feature map is processed using the trained CSMP model to obtain the music feature vector of the music data.

[0010] The music feature vector, Mel-spectrogram feature map and interactive high-dimensional features are input into the trained conditional controlled diffusion model to generate the Mel-spectrogram feature map of the reconstructed music data.

[0011] The Mel-spectrogram feature map of the reconstructed music data is processed using the HiFi-GAN pre-trained model to generate the corresponding scene music, which is then fed back into the scene space and added to the music data.

[0012] In one embodiment, the scene space data includes: layout information, state information, geographic information, weather information, lighting information, time information, cultural information, visual information, auditory information, physical information, smell information and story information in the scene space.

[0013] User information data includes: user basic information, user physiological information, user behavior information and user voice information.

[0014] The music data is obtained from open source music datasets.

[0015] In one embodiment, the training process of the CSMP model includes:

[0016] Collect music data from open source music datasets and extract the Mel-spectrogram feature map of the music data.

[0017] The scene space data and user information data are integrated to obtain multi-dimensional scene data.

[0018] The mel-spectrogram features of the multivariate scene data and the music data are used to construct a multivariate scene data-music dataset according to the pre-defined correspondence between the multivariate scene data and the music data.

[0019] The multivariate scene data and the corresponding music data are respectively input into the multivariate scene data encoder and the music data encoder to obtain the scene feature vector of the multivariate scene data and the music feature vector of the music data.

[0020] Map the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m .

[0021] Construct the total loss function of the CSMP model.

[0022] According to the normalized vector s and the normalized vector mAs well as the total loss function of the CSMP model, through contrast learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

[0023] In one embodiment, the total loss function of the CSMP model includes: a loss function for retrieving music data using multi-scenario data and a loss function for retrieving multi-scenario data using music data.

[0024] Among them, the loss function of multi-scenario data retrieval music data is:

[0025] ;

[0026] in, In order to utilize the loss of multi-scenario data to retrieve music data, N is the amount of music data, For the i Multi-scenario data and j The similarity between music data, For the j The characteristic data of music, For the i Feature data of multiple scenarios.

[0027] The loss function for music data retrieval of multi-scenario data is:

[0028] ;

[0029] in, In order to use music data to retrieve the loss of multi-scene data, For the i Music data and j The similarity between multiple scene data, Z is the number of multivariate scene data.

[0030] In one embodiment, the conditional controlled diffusion model is a latent diffusion model, including a VAE encoder, a UNet+Attention model, and a VAE decoder.

[0031] The scene feature vector, music feature vector, Mel-spectrogram feature map and interactive high-dimensional features are input into the trained conditional controlled diffusion model to generate a Mel-spectrogram feature map of the reconstructed music data, including:

[0032] The Mel-spectrogram feature map is input into the VAE encoder to obtain the latent feature vector of the music data.

[0033] The latent features of the music data are forward diffused through the conditional controlled diffusion model to obtain a low-dimensional latent vector with noise.

[0034] According to the noisy low-dimensional latent vector, the interactive high-dimensional features and music feature vectors are used as control conditions, and the UNet+Attention model is used to predict noise to obtain the predicted noise.

[0035] After the predicted noise is processed by the VAE decoder, a Mel-spectrogram feature map of the reconstructed music data is generated.

[0036] An interactive music intelligent generation system for adaptive scene space, the system comprises: an information acquisition module, an interactive data processing and feature extraction module, an adaptive scene music generation module and a multimodal feedback module based on music generation.

[0037] The information acquisition module is used to obtain scene space data and user information data through the interactive device, and transmit the scene space data and user information data to the scene data processing and feature extraction module.

[0038] The interactive data processing and feature extraction module is used to pre-process the scene space data and user information data and then fuse them to obtain multi-scene data; the multi-scene data is processed by the trained CSMP model, and the generated high-dimensional features of the multi-scene data are spliced ​​according to the time sequence to obtain the multi-scene data sequence; the multi-scene data sequence is processed by interacting with the GPT model to obtain the interactive high-dimensional features of the interactive data; the CSMP model is a dual encoder architecture, including a multi-scene data encoder and a music data encoder; the multi-scene data is text data.

[0039] The adaptive scene music generation module is used to obtain music data and extract the Mel-spectrogram feature map of the music data, and process the Mel-spectrogram feature map using the trained CSMP model to obtain the scene feature vector of the multivariate scene data and the music feature vector of the music data; the scene feature vector, the music feature vector, the Mel-spectrogram feature map and the interactive high-dimensional features are input into the trained conditional controlled diffusion model to generate the Mel-spectrogram feature map of the reconstructed music data; the Mel-spectrogram feature map of the reconstructed music data is processed using the HiFi-GAN pre-trained model to generate the corresponding scene music.

[0040] The multimodal feedback module based on music generation is used to feed back the scene music generated by the adaptive scene music generation module into the scene space and add it to the music data.

[0041] In one embodiment, the information collection module includes: a scene space data collection module, a user information data collection module and a data set construction module.

[0042] The scene space data collection module is used to collect the scene's layout information, status information, geographic information, weather information, lighting information, time information, astronomical information, visual information, auditory information, physical information, smell information and story information using the corresponding interactive device.

[0043] The user information data collection module is used to collect the user's basic information, physiological information, behavioral information and voice information using the corresponding interactive device.

[0044] The dataset construction module is used to collect data from different scenarios using interactive devices. By analyzing the melody, rhythm and emotion of the music, the collected scene data is matched one-to-one with the music data to construct a multi-scenario data-music dataset.

[0045] In one of the embodiments, the multimodal feedback module generated based on music is also used to wake up the output device close to the user according to the scene space data and user information data, match the best output device, and select a multimodal output interactive device linked to the music; the multimodal output interactive device is used to display multimodal content synchronized with the music, and is linked to the generated scene music.

[0046] In one embodiment, the training process of the CSMP model in the interactive data processing and feature extraction module includes:

[0047] Collect music data from open source music datasets and extract the Mel-spectrogram feature map of the music data.

[0048] The scene space data and user information data are integrated to obtain multi-dimensional scene data.

[0049] The mel-spectrogram features of the multivariate scene data and the music data are used to construct a multivariate scene data-music dataset according to the pre-defined correspondence between the multivariate scene data and the music data.

[0050] The multivariate scene data and the corresponding music data are respectively input into the multivariate scene data encoder and the music data encoder to obtain the scene feature vector of the multivariate scene data and the music feature vector of the music data.

[0051] Map the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m .

[0052] Construct the total loss function of the CSMP model.

[0053] According to the normalized vector s and the normalized vector mAs well as the total loss function of the CSMP model, through contrastive learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

[0054] In one embodiment, the total loss function of the CSMP model includes: a loss function for retrieving music data using multi-scenario data and a loss function for retrieving multi-scenario data using music data.

[0055] Among them, the loss function of multi-scenario data retrieval music data is:

[0056] ;

[0057] in, In order to utilize the loss of multi-scenario data to retrieve music data, N is the amount of music data, For the i Multi-scenario data and j The similarity between music data, For the j The characteristic data of music, For the i Feature data of multiple scenarios.

[0058] The loss function for music data retrieval of multi-scenario data is:

[0059] ;

[0060] in, In order to utilize the loss of multi-scenario data to retrieve music data, For the i Music data and j The similarity between multiple scene data, Z is the number of multivariate scene data.

[0061] The above-mentioned interactive music intelligent generation method and system for adaptive scene space, the method constructs a conditional diffusion model framework by sensing the user's behavior, emotional state and spatial environment changes, generates music that matches the current scene, realizes the deep integration of music and user experience, and dynamically adjusts the generated music to ensure that the generated music can not only flexibly respond to scene changes, but also meet the user's personalized needs, thereby enhancing the user's interactive experience with the music generation system in the scene, and improving the diversity and intelligence of scene solution generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A schematic diagram of a flow chart of an interactive music intelligent generation method in an adaptive scene space in one embodiment;

[0063] Figure 2 A flowchart for data feature extraction and diffusion model intelligent generation in one embodiment;

[0064] Figure 3 The structure block diagram of an interactive music intelligent generation system for adaptive scene space in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In one embodiment, Figure 1 As shown, a method for intelligently generating interactive music in an adaptive scene space is provided, and the method comprises the following steps:

[0067] Step 100: Obtain user information data, scene space data and music data, and fuse the user information data and the scene space data to obtain multi-scenario data; wherein the multi-scenario data takes text data as an example.

[0068] Specifically, music data is obtained. The initial scene music data is empty. Once the system generates music, it is set as the scene music data and updated in each loop iteration.

[0069] Get multi-scenario data. After the system generates music for the first time, the multi-scenario data will contain user feedback and will be updated in each cycle to better understand the user's needs in that scenario.

[0070] Obtain user information data, scene space data and music data through interactive devices.

[0071] Interactive devices include sensor devices, touch devices, audio devices, visual devices, mobile devices, environmental adjustment devices, and security and monitoring devices.

[0072] Sensor devices include but are not limited to environmental sensors (such as temperature sensors, humidity sensors, infrared sensors), physiological signal sensors (such as skin electrical feedback sensors, pulse sensors, eye tracking sensors, breathing sensors, iris sensors, ultrasonic sensors), motion sensors (accelerometers, gyroscopes, vibration sensors), etc.

[0073] Touch devices include but are not limited to touch screens, touch pads, touch gloves, etc.

[0074] Audio devices include but are not limited to speakers, headphones, microphones, etc.

[0075] Visual devices include but are not limited to cameras, projectors, AR devices, VR devices, etc.

[0076] Olfactory devices include but are not limited to odor generators, etc.

[0077] Mobile devices include but are not limited to smartphones, tablets, smart watches, etc.

[0078] Environmental conditioning equipment includes but is not limited to temperature and humidity control systems, air purification systems, intelligent fresh air systems, etc.

[0079] Security and monitoring equipment include but are not limited to smart cameras, smart door locks, etc.

[0080] Step 102: Use the trained CSMP model to process the multivariate scene data to obtain high-dimensional features of the multivariate scene data, and splice the high-dimensional features of the multivariate scene data according to time sequence to obtain a multivariate scene data sequence; the CSMP model is a dual encoder architecture, including a multivariate scene data encoder and a music data encoder.

[0081] Specifically, the full name of the CSMP (Contrastive Scene Music Pre-training) model is: Contrastive Scene Music Pre-training Model.

[0082] The trained CSMP model is trained using a pre-constructed multivariate scene data-music data set. During the training process, the distance between matching multivariate scene data-music data pairs in the vector space is made closer, while the distance between unmatched multivariate scene data-music data pairs is made farther.

[0083] The steps for constructing the multivariate scene data-music dataset include: first, collecting music data from the music open source dataset; then fusing the scene space data and user information data to obtain the multivariate scene data; then, further determining the correspondence between the Mel spectrum features of the multivariate scene data and the music data to construct the multivariate scene data-music dataset.

[0084] Preferably, the multi-scenario data encoder uses the BERT pre-trained model to encode the multi-scenario data; the music data encoder uses the HTS-AT (Hierarchical Token-Semantic AudioTransformer) pre-trained model to encode the music data.

[0085] Step 104: Process the multivariate scene data sequence in an interactive manner with the GPT model to obtain interactive high-dimensional features of the interactive data.

[0086] As a preferred method, the large language model uses the GPT model based on the Transformer structure. Since the Transformer is better than LSTM in extracting contextual information and long-distance dependencies, the GPT model based on the Transformer structure is used when processing multi-scenario data sequences containing time information. The GPT model is trained using multi-scenario data sequences to obtain a GPT pre-trained model.

[0087] The process of acquiring interactive high-dimensional features of interactive data: If the current moment is the initial moment, the interactive data feature vector is the same as the multivariate scene data feature vector of the current moment; if the current moment is not the initial moment, the multivariate scene data feature vector of the current moment and the multivariate scene data feature vectors of all previous moments are concatenated in chronological order into a multivariate scene data sequence feature vector, and then the feature vector is input into the GPT pre-training model to obtain the interactive data feature vector that combines the multivariate scene data feature vectors of the current moment and all previous moments.

[0088] The training steps of the GPT model include:

[0089] Step 1: Determine multi-scenario data , select multi-scenario data at different time points from the multi-scenario data-music dataset MusicScenceDataset.

[0090] Step 2: Determine the multivariate scenario data sequence in chronological order . Multi-scenario data at different time points Arrange in chronological order to form a multi-scenario data sequence ,in: .

[0091] Step 3: Build a GPT model. Here, we take GPT1 as an example.

[0092] Step 4: Train the GPT model. The data in the data have a chronological relationship on the timeline. Therefore, the GPT model can be used to predict the information at the next moment based on the information at the current moment and the information at the previous moment.

[0093] Since the GPT model cannot capture multi-scenario data sequences The position information / timing information in the Embedding Multivariate scene data sequence The position information of is introduced into the GPT model, recorded as:

[0094] (1)

[0095] (2)

[0096] in, position Indicates that a token (multi-scenario data at a certain moment) is in a multi-scenario data sequence location, Embedding is the position vector of the token, Indicates the position vector corresponding to the token i vectors ( i indicates odd dimensions, 2 i denotes an even dimension), Indicates the dimension of the token.

[0097] Multivariate scene data sequence and position encoding Embedding Add together to get the input of the GPT model , written as:

[0098] (3)

[0099] enter After being processed by multiple layers of Transformer modules, the output is obtained , written as:

[0100] (4)

[0101] in, n is the number of layers of the Transformer module.

[0102] Will output After Softmax processing, the frequency corresponding to the next moment information is obtained , written as:

[0103] (5)

[0104] in, is a multivariate scene data sequence The feature vectors corresponding to the tokens in .

[0105] Loss function of the GPT model , written as:

[0106] (6)

[0107] in, iis the position of the token, are the parameters of the GPT model.

[0108] Step 5: After the training is completed, the pre-trained GPT model is obtained.

[0109] Step 106: extract the Mel-frequency spectrum feature map of the music data, and process the Mel-frequency spectrum feature map using the trained CSMP model to obtain a music feature vector of the music data.

[0110] Specifically, a specific method for extracting the Mel spectrum features of the music data is to process the music data using short-time Fourier transform, Mel frequency scale and Mel filter bank to obtain the Mel spectrum features of the music data.

[0111] If music data is empty, ignore step.

[0112] If the music data is not empty, the Mel spectrum features of the music data are extracted, and the Mel spectrum feature map is processed using the trained CSMP model.

[0113] Step 108: Input the scene feature vector, the music feature vector, the Mel-spectrogram feature map, and the interactive high-dimensional features into the trained conditional controlled diffusion model to generate a reconstructed music data Mel-spectrogram.

[0114] Specifically, the conditional controlled diffusion model includes: VAE pre-training model and UNet+Attention model.

[0115] The Mel-spectrogram features of the music data are mapped to a low-dimensional latent feature vector through the encoder in the VAE pre-trained model.

[0116] Under the condition that the music data is not empty, the forward diffusion process of the conditional diffusion model is performed, and Gaussian noise is added to the low-dimensional latent feature vector at each time step, and finally a latent feature vector containing noise is generated. If the music data is empty, a random Gaussian noise vector is generated.

[0117] The music data feature vector and the interaction data feature vector are used as control conditions to guide the reverse diffusion process of the UNet+Attention model in the conditional control diffusion model, and finally a reconstructed low-dimensional latent feature vector is generated after noise is removed.

[0118] During the inverse diffusion process of the conditional controlled diffusion model, the error between the predicted noise and Gaussian noise is calculated at each time step.

[0119] The reconstructed low-dimensional latent feature vector is mapped to the Mel-spectrogram features of the reconstructed music data through the decoder in the VAE pre-trained model.

[0120] Step 110: The reconstructed mel-spectrogram features of the music data are processed using the HiFi-GAN pre-trained model to generate corresponding scene music, which is fed back into the scene space and added to the music data.

[0121] In the above-mentioned interactive music intelligent generation method for adaptive scene space, the method constructs a conditional diffusion model framework by sensing the user's behavior, emotional state and spatial environment changes, generates music matching the current scene, realizes the deep integration of music and user experience, and dynamically adjusts the generated music to ensure that the generated music can not only flexibly respond to scene changes, but also meet the user's personalized needs, thereby enhancing the user's interactive experience with the music generation system in the scene and improving the diversity and intelligence of scene solution generation.

[0122] In one embodiment, the scene space data in step 100 includes: layout information, state information, geographic information, weather information, lighting information, time information, cultural information, visual information, auditory information, physical information, smell information and story information in the scene space.

[0123] User information data includes: user basic information, user physiological information, user behavior information and user voice information.

[0124] The music data is obtained from open source music datasets.

[0125] Specifically, (1) scene space data includes:

[0126] Layout information includes: buildings, man-made facilities (such as street lights, seats, trash cans), transportation facilities (such as parking lots, bus stops, roads), vegetation and natural elements (such as trees, flowers, water bodies), public spaces (such as squares, parks, etc.), etc.

[0127] Status information includes: exercise, sleep, study, office, entertainment and other status.

[0128] Geographic information includes: spatial location: longitude and latitude, altitude / height, country; terrain features: such as mountains, plains, valleys, etc.; land features: such as forests, grasslands, farmlands, cities, etc.; ecological features: such as vegetation, animals, soil, etc.

[0129] Weather information includes: temperature, humidity, air pressure, cloud cover, visibility, air quality, ultraviolet rays, pollutant concentration, weather conditions (such as sunny, cloudy, rainy, foggy, snowy), wind speed and direction, etc.

[0130] Lighting information includes: natural light (such as sunlight, moonlight, etc.), artificial light (such as lamplight, firelight, etc.) and other lighting information.

[0131] Time information includes: different times of the day (such as morning, noon, and evening), seasons (such as spring, summer, autumn, and winter), and festivals (such as the Spring Festival, the Mid-Autumn Festival, and the Double Ninth Festival).

[0132] Cultural information includes: historical background, religion, traditional festivals, architectural style (the design style, structural form and decorative elements reflected in the buildings in the scene), daily living habits, food culture, language, writing, painting, sculpture, music, dance, handicrafts and other cultural information.

[0133] Visual information includes: appearance features of objects (such as shape and size, color and texture, edge characteristics), spatial information of objects (such as position, direction, posture) and semantic information of objects (object categories, such as cars and trees; relationships, such as contact and distance in space) and other visual information of objects in the scene.

[0134] Auditory information includes: sound characteristics (such as volume and frequency), spatial information of sound (such as the location of the sound source), temporal information of sound (such as duration, rhythm), semantic information of sound (sound category, such as background music, environmental sound, character sound, noise) and other auditory information.

[0135] Physical information includes: static characteristics: mass, density, temperature, humidity, material, rigidity, elasticity, viscosity, strength, hardness, roughness, friction, vibration, etc.; dynamic characteristics: speed, acceleration, angular velocity / rotational speed, etc.

[0136] Odor information includes: odor type (such as floral, fruity, and rancid), odor intensity / concentration, odor source location, odor change, odor duration, and semantic information of the odor (the specific source of the odor, such as plants, food, chemicals, etc.).

[0137] Story information includes: characters, environment, events, emotions, themes and other story information.

[0138] (2) User information data includes:

[0139] Basic information includes: height, weight, gender, age, type (such as anxiety patients, people with disabilities, depressed patients, pregnant women), music preferences, hobbies, body fat percentage, and metabolism.

[0140] Physiological information includes: brain electromagnetic physiological signals (such as electroencephalogram signals, magnetoencephalogram signals), skin electrical signals, skin electrical response, skin electrical activity; electromyography signals, electrocardiogram signals, electrooculography signals, blood pressure data, blood oxygen data, pulse signals, eye movement data, breathing signals, iris images, body temperature data, blood sugar data, and fetal heart rate data.

[0141] Behavioral information includes: static data of exercise: exercise type, exercise calorie consumption, exercise distance / height and climb, exercise duration, exercise records; dynamic data of exercise: heart rate during exercise, stride length during exercise (the distance between two adjacent steps), step frequency during exercise (the number of landings per minute), gait during exercise (posture during exercise), exercise acceleration, exercise speed / pace; gestures, body movements and facial expressions.

[0142] Voice information includes: pitch, volume, speaking speed, and emotion.

[0143] In one embodiment, the training process of the CSMP model in step 102 includes: collecting music data from an open source music data set and extracting the Mel-spectrogram features of the music data; fusing the scene space data and the user information data to obtain multivariate scene data; using the Mel-spectrogram features of the multivariate scene data and the music data to construct a multivariate scene data-music data set according to a predefined correspondence between the multivariate scene data and the music data; inputting the multivariate scene data and the corresponding music data into a multivariate scene data encoder and a music data encoder respectively to obtain a scene feature vector of the multivariate scene data and a music feature vector of the music data; mapping the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m ; Construct the total loss function of the CSMP model; According to the normalized vector s and the normalized vector m As well as the total loss function of the CSMP model, through contrast learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

[0144] Specifically, we use interactive devices to collect rich data from different scenes, and through in-depth analysis of the music's melody, rhythm, emotion and other elements, we match the collected scene data with the music data one by one to ensure that the music data and scene data can echo each other in terms of emotional expression and atmosphere creation, and finally establish a multi-scene data - music data set. For example, fast-paced music is more suitable for busy city street scenes, while slow and melodious melodies are more suitable for tranquil natural scenery.

[0145] In one embodiment, the total loss function of the CSMP model includes: a loss function for retrieving music data using multi-scenario data and a loss function for retrieving multi-scenario data using music data.

[0146] Among them, the loss function of multi-scenario data retrieval music data is:

[0147] (7)

[0148] in, In order to utilize the loss of multi-scenario data to retrieve music data, N is the amount of music data, For the i Multi-scenario data and j The similarity between music data, is the similarity of the positive sample, and Indicates i The sum of the similarities between the multi-scenario data and all the music data, For the j The characteristic data of music, For the i Feature data of multiple scenarios.

[0149] The loss function for music data retrieval of multi-scenario data is:

[0150] (8)

[0151] in, In order to utilize the loss of multi-scenario data to retrieve music data, For the i Music data and j The similarity between the multivariate scene data, Indicates i The sum of the similarities between music data and all multi-scenario data, Z is the number of multivariate scene data.

[0152] Specifically, the training process of the CSMP model includes:

[0153] Step 1: Determine scene space data and user information data Use interactive devices to collect user information data and scene space data respectively, remove noise and invalid data through data cleaning and feature extraction, and determine the scene space data and user information data .

[0154] Step 2: Determine multi-scenario data . The scene space data and user information data Fusion to obtain multi-scene data , where multivariate scene data takes text data as an example.

[0155] Step 3: Confirm the music data . Screen and select from open source music data sets to determine the music data .

[0156] Step 4: Construct multi-scenario data - MusicScene Dataset. And the calculated Mel spectrum features of the music data , based on the pre-defined correspondence between multi-scene data and music data, a multi-scene data-music dataset MusicScene Dataset is constructed.

[0157] Step 5: Build a comparative scene music model CSMP. The CSMP model is a dual encoder architecture, in which one of the multi-scene data encoders and a music data encoder . Extract features from multi-scenario data, Extract features from music data.

[0158] Given a multivariate scene data , after the multi-scene data encoder After encoding, we get a Dimensional vector , written as:

[0159] (9)

[0160] Given a music data , after the music data encoder After encoding, we get a Dimensional vector , written as:

[0161] (10)

[0162] Using multivariate scene data transformation matrix Vector Mapping to the latent space yields The normalized vector s of dimension is denoted as:

[0163] (11)

[0164] Transforming the Matrix Using Music Data Vector Mapping to the latent space yields The normalized vector of dimension m , written as:

[0165] (12)

[0166] Step 6: Train the comparative scene music model CSMP. During the training process, CSMP uses comparative learning to make the distance between matching multivariate scene data and music data pairs in the vector space closer, while the distance between mismatched multivariate scene data and music data pairs is farther. The music data can be data in the time domain or data in the frequency domain. Here, the music data takes data in the frequency domain as an example.

[0167] The entire training process has two loss functions, one of which is the loss function of retrieving music data using multi-scenario data. , and its calculation expression is shown in formula (7).

[0168] The loss function in another direction is to use music data to retrieve the loss function of multi-scene data. , and its calculation expression is shown in formula (8).

[0169] Step 7: After the training is completed, the comparison scene music pre-training model CSMP is obtained.

[0170] In one embodiment, the conditional controlled diffusion model is a latent diffusion model, including a VAE encoder, a UNet+Attention model and a VAE decoder; step 108 includes: inputting the Mel-frequency spectrum feature map into the VAE encoder to obtain the latent feature vector of the music data; forward diffusing the latent features of the music data through the conditional controlled diffusion model to obtain a noisy low-dimensional latent vector; based on the noisy low-dimensional latent vector, using the interactive high-dimensional features and the music feature vector as control conditions, using the UNet+Attention model to perform noise prediction to obtain predicted noise; after processing the predicted noise through the VAE decoder, a reconstructed Mel-frequency spectrum map of the music data is generated.

[0171] Specifically, the training process of the diffusion model includes:

[0172] Step 1: Determine the music data . Select music data from the multi-scene data-music dataset MusicSceneDataset .

[0173] Step 2: Determine multi-scenario data . Select music data from the multi-scene data-music dataset MusicSceneDataset Corresponding multi-scenario data .

[0174] Step 3: Confirm the music data Mel spectrum features . Use short-time Fourier transform, Mel frequency scale and Mel filter bank to analyze music data Processing to obtain the Mel spectrum features of music data .

[0175] Before applying short-time Fourier transform to music data, it is necessary to perform pre-emphasis, frame division and windowing on the music data. The pre-emphasis process is to pass the music data through a high-pass filter, which is recorded as:

[0176] (14)

[0177] in, is the filter coefficient, which is usually between 0.9 and 1.

[0178] Framing processing is to divide the pre-emphasized, continuous music data into frame signals.

[0179] Windowing is to multiply each frame of signal by the window function , reduce the phenomenon of "spectrum leakage", window function Denoted as:

[0180] (15)

[0181] in, is the index of the signal frame, is the length of the frame, =0.46.

[0182] For each frame of preprocessed signal, a length Fourier transform of Denoted as:

[0183] (16)

[0184] in, For the i Frame No. k frequency components, For the i Frame No. The signal value of the time domain sampling point, is the number of sampling points in the time domain, k is the frequency index, and its value range is , is the complex rotation factor in Fourier transform.

[0185] The spectrum line energy of the music data is obtained by taking the modulus square of the spectrum of the music data processed by Fourier transform. , written as:

[0186] (17)

[0187] in, For music data No. i All frequency components of a frame.

[0188] According to the Hertz frequency f Computing music data In each frame of the signal M + Mel-scale values ​​of 2 frequency components , that is, uniformly generated on the Mel scale value M +2 points, recorded as:

[0189] (18)

[0190] in, M is the number of filters, f is the frequency in Hertz.

[0191] Convert the Mel scale value corresponding to the music data back to Hertz frequency, recorded as:

[0192] (19)

[0193] in, is the Mel scale value.

[0194] According to music data Hertz frequency Define Mel filter bank (Take the triangular filter bank as an example), denoted as:

[0195] (20)

[0196] in, M is the number of filters, f ( m ) is the m The center frequency of the filter, k is the frequency.

[0197] The power spectrum of the music data is passed through the Mel filter bank to obtain the music data Mel spectrum features .

[0198] Step 4: Build a conditional controlled diffusion model, which includes the VAE pre-trained model and the UNet+attention model.

[0199] Step 5: Train the conditional controlled diffusion model. First, use the encoder in the VAE pre-trained model The music data Mel spectrum features Mapping to a low-dimensional latent vector y , and in the forward diffusion process of the conditional controlled diffusion model, the latent vector y Add to T time steps of Gaussian noise.

[0200] The forward diffusion process is defined as a Markov chain process, denoted as:

[0201] (twenty one)

[0202] (twenty two)

[0203] in, , , is the noise diffusion coefficient at each time step t, is the low-dimensional latent vector at the 0th time step, Indicates t The low-dimensional latent vector after noise addition at each time step, is standard normal Gaussian noise.

[0204] After the forward diffusion process is completed, the reverse diffusion process is carried out. Here, the reverse diffusion process of the DDPM model is taken as an example.

[0205] The purpose of the back diffusion process is to Denoising is performed step by step, using interactive high-dimensional features and music feature vectors as control conditions, and using the UNet+Attention model to predict noise data , where the scene feature vectors of multivariate scene data are obtained through the CSMP model and the music feature vector of the music data .

[0206] The entire training process of the conditional controlled diffusion model uses the predicted noise Fitting at each time step t Added Noise , the loss function of the conditional controlled diffusion model , written as:

[0207] (twenty three)

[0208] The final sampling formula for the reverse diffusion process is written as:

[0209] (twenty four)

[0210] in, Indicatest -1 time step of noisy low-dimensional latent vector.

[0211] It should be understood that although Figure 1 , Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0212] In one embodiment, Figure 3 As shown, an interactive music intelligent generation system for adaptive scene space is provided, which includes: an information acquisition module, an interactive data processing and feature extraction module, an adaptive scene music generation module and a multimodal feedback module based on music generation; wherein:

[0213] The information acquisition module is used to obtain scene space data and user information data through the interactive device, and transmit the scene space data and user information data to the scene data processing and feature extraction module.

[0214] The interactive data processing and feature extraction module is used to pre-process the scene space data and user information data and then fuse them to obtain multi-scene data; the multi-scene data is processed by the trained CSMP model, and the generated high-dimensional features of the multi-scene data are spliced ​​according to the time sequence to obtain the multi-scene data sequence; the multi-scene data sequence is processed by interacting with the GPT model to obtain the interactive high-dimensional features of the interactive data; the CSMP model is a dual encoder architecture, including a multi-scene data encoder and a music data encoder; the multi-scene data is text data.

[0215] The adaptive scene music generation module is used to obtain music data and extract the Mel-spectrogram feature map of the music data, process the Mel-spectrogram feature map using the trained CSMP model to obtain the music feature vector of the music data; input the music feature vector, Mel-spectrogram feature map and interactive high-dimensional features into the trained conditional controlled diffusion model to generate the Mel-spectrogram feature map of the reconstructed music data; process the Mel-spectrogram feature map of the reconstructed music data using the HiFi-GAN pre-trained model to generate the corresponding scene music.

[0216] The multimodal feedback module based on music generation is used to feed back the scene music generated by the adaptive scene music generation module into the scene space and add it to the music data.

[0217] In one embodiment, the information collection module includes: a scene space data collection module, an interactive device data module and a data set construction module.

[0218] The scene space data collection module is used to collect the scene's layout information, status information, geographic information, weather information, lighting information, time information, astronomical information, visual information, auditory information, physical information, smell information and story information using the corresponding interactive device.

[0219] The user information data collection module is used to collect the user's basic information, physiological information, behavioral information and voice information using the corresponding interactive device.

[0220] The dataset construction module is used to collect data from different scenarios using interactive devices. By analyzing the melody, rhythm and emotion of the music, the collected scene data is matched one-to-one with the music data to construct a multi-scenario data-music dataset.

[0221] Specifically, we use interactive devices to collect rich data from different scenes, and through in-depth analysis of the music's melody, rhythm, emotion and other elements, we match the collected scene data with the music data one by one to ensure that the music data and scene data can echo each other in terms of emotional expression and atmosphere creation, and finally establish a multi-scene data - music data set. For example, fast-paced music is more suitable for busy city street scenes, while slow and melodious melodies are more suitable for tranquil natural scenery.

[0222] In one of the embodiments, the multimodal feedback module generated based on music is also used to wake up the output device close to the user according to the scene space data and user information data, match the best output device, and select a multimodal output interactive device linked to the music; the multimodal output interactive device is used to display multimodal content synchronized with the music, and is linked to the generated scene music.

[0223] Specifically, the working process of the multimodal feedback module based on music generation includes:

[0224] (1) Identify the output device close to the user through the user data and scene data obtained by the above information collection module.

[0225] (2) Match the best output device. Ensure that the content of the adaptive scene music generation module can be fed back to the user through the appropriate output device. For example, if the user is close to multiple sound systems, select the sound system with the highest quality, play music through this system first, and adjust the volume and sound effect settings according to the acoustic characteristics of the room.

[0226] (3) Select interactive devices that are linked to music. Through the user data and scene data collected by the system, analyze the interactive devices that are suitable for user needs and the current scene, and realize adaptive selection of multimodal interactive devices. For example, KTV scenes are more suitable for playing by linking speakers and screens, while study rooms are more suitable for playing with headphones.

[0227] (4) Output multimodal content based on the generated music. Use interactive devices with multimodal output to display multimodal content synchronized with the music, and interact with the generated music to provide a richer multisensory experience. For example, use visual output devices, including but not limited to music visualization graphics, rhythmic lighting, etc., to enhance the experience by interacting with the music; use olfactory output devices, the system can release corresponding fragrances according to the style and rhythm of the music, synchronized with the music, to provide a richer multisensory experience. For example, when playing jazz, the fragrance of tobacco and leather is released; when playing beach-style music, the fragrance of coconut and sea salt is released; using tactile output devices, the system can provide synchronized tactile feedback when playing music. For example, when playing heavy metal music, the seat can provide strong vibration feedback to simulate the rhythm and intensity of the music.

[0228] (5) Integrate the management of input and output devices into the interactive platform. The specific steps are: the feedback module provides an interactive platform for user-controlled devices, integrates the management of input and output devices into the interactive platform, and allows users to select and configure these devices according to personal preferences and scenario requirements, so as to facilitate users to change the interactive content and meet users' real-time personalized needs.

[0229] (6) Allow users to select and configure multiple input data through the interactive platform according to personal preferences and scene requirements. For example, users can control the data they want to be monitored. When exercising, they can choose their own heart rate data to be monitored by the input device, choose the scene's weather data to be monitored by the input device, and finally choose the output device to be the headphones they wear.

[0230] (7) Allow users to actively select or change feedback channels through the interactive interface according to personal preferences and scene requirements. For example, when resting, users can choose the feedback mode. If they want their data to be fed back in the form of music and lights, they can choose the feedback mode as hearing and vision.

[0231] (8) Allow users to set the above as the default feedback channel for quick selection next time.

[0232] (9) The system provides preset scene modes and appropriate feedback modes for different scenarios (such as fitness, work, leisure, etc.), which users can quickly select in the interactive platform.

[0233] (10) The interactive platform control system responds accordingly based on the input data, output devices, and feedback modes (through touch screen controls, commands, or other forms) defined by the user in the interactive platform.

[0234] (11) During the immersive experience, users can change the feedback channel at any time as needed.

[0235] (12) Real-time update and adjustment of user feedback to respond to changes in user needs and provide a highly personalized and immersive experience.

[0236] (13) After the experience, users can evaluate different feedback channels so that the system can be continuously optimized to enhance the flexibility and adaptability of the interactive system.

[0237] In one embodiment, the training process of the CSMP model in the interactive data processing and feature extraction module includes:

[0238] Collect music data from open source music datasets and extract the Mel-spectrogram feature map of the music data.

[0239] The scene space data and user information data are integrated to obtain multi-dimensional scene data.

[0240] The mel-spectrogram features of the multivariate scene data and the music data are used to construct a multivariate scene data-music dataset according to the pre-defined correspondence between the multivariate scene data and the music data.

[0241] The multivariate scene data and the corresponding music data are respectively input into the multivariate scene data encoder and the music data encoder to obtain the scene feature vector of the multivariate scene data and the music feature vector of the music data.

[0242] Map the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m .

[0243] Construct the total loss function of the CSMP model.

[0244] According to the normalized vector s and the normalized vector mAs well as the total loss function of the CSMP model, through contrastive learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

[0245] Specifically, the working process of the interactive data processing module includes:

[0246] Use the multi-scene data-music dataset MusicScene Dataset to train the contrast scene music model and obtain the contrast scene music pre-training model CSMP.

[0247] The multivariate scene data is input into CSMP to generate a multivariate scene data feature vector that is similar to the music data feature vector.

[0248] In the training phase of the conditional control diffusion model, the multivariate scene feature vector and the music feature vector are directly used as control conditions to train the conditional control diffusion model, without considering the multivariate scene data sequence that satisfies the time order. In the sampling phase of the conditional control diffusion model, if the current moment is the initial moment, the interaction data feature vector is the same as the multivariate scene data feature vector at the current moment, and the scene music data is empty; if the current moment is not the initial moment, the multivariate scene data feature vector at the current moment and the multivariate scene data feature vectors at all previous moments are spliced ​​into a multivariate scene data sequence feature vector in time order, and then the feature vector is input into the GPT pre-training model to obtain the interaction data feature vector that combines the multivariate scene data feature vectors at the current moment and all previous moments, and the scene music data is the music generated by the conditional control diffusion model at the previous moment. If the scene music data is empty, a Gaussian noise vector is randomly generated; if the scene music data is not empty, the scene music data is passed through the encoder of the VAE pre-training model to obtain the potential feature vector of the scene music data, and Gaussian noise is added to the potential feature vector of the scene music data in the forward diffusion process of the conditional control diffusion model to obtain a noise vector that satisfies the Gaussian distribution. The scene music data feature vector / noise vector and the interaction data feature vector are input into the conditional controlled diffusion model for reverse diffusion to generate the Mel spectrum features of the scene music data. The Mel spectrum features are input into the HiFi-GAN model to obtain the corresponding scene music.

[0249] In one embodiment, the total loss function of the CSMP model includes: a loss function for retrieving music data using multi-scenario data and a loss function for retrieving multi-scenario data using music data.

[0250] The loss function of music data retrieval from multi-scenario data is shown in formula (7); the loss function of music data retrieval from multi-scenario data is shown in formula (8).

[0251] For the specific definition of the interactive music intelligent generation system for adaptive scene space, please refer to the definition of the interactive music intelligent generation method for adaptive scene space above, which will not be repeated here. Each module in the above-mentioned interactive music intelligent generation system for adaptive scene space can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0252] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0253] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An interactive music intelligent generation method for adaptive scene space, characterized in that: The method comprises: Acquire user information data, scene space data, and music data, and fuse the user information data and the scene space data to obtain multi-scenario data, wherein the multi-scenario data is text data; The multi-scenario data are processed using a trained CSMP model to obtain high-dimensional features of the multi-scenario data, and the high-dimensional features of the multi-scenario data are spliced ​​according to a time sequence to obtain a multi-scenario data sequence; the CSMP model is a dual encoder architecture, including a multi-scenario data encoder and a music data encoder; The multi-scenario data sequence is processed by interacting with the GPT model to obtain interactive high-dimensional features of the interactive data; Extracting a Mel-frequency spectrum feature map of the music data, and processing the Mel-frequency spectrum feature map using a trained CSMP model to obtain a music feature vector of the music data; Inputting the music feature vector, the Mel-spectrogram feature map and the interactive high-dimensional features into a trained conditional controlled diffusion model to generate a Mel-spectrogram feature map of reconstructed music data; The Mel-spectrogram feature map of the reconstructed music data is processed using the HiFi-GAN pre-trained model to generate corresponding scene music, and the scene music is fed back into the scene space and added to the music data; The conditional controlled diffusion model is a potential diffusion model, including a VAE encoder, a UNet+Attention model, and a VAE decoder; Inputting the music feature vector, the Mel-spectrogram feature map and the interactive high-dimensional features into a trained conditional controlled diffusion model to generate a Mel-spectrogram feature map of reconstructed music data, including: Inputting the Mel frequency spectrum feature map into the VAE encoder to obtain a potential feature vector of the music data; The latent features of the music data are forward diffused through a conditional controlled diffusion model to obtain a low-dimensional latent vector after adding noise; According to the noisy low-dimensional latent vector, the interactive high-dimensional features and the music feature vector are used as control conditions, and the UNet+Attention model is used to perform noise prediction to obtain predicted noise; After the predicted noise is processed by the VAE decoder, a Mel-spectrogram feature map of the reconstructed music data is generated.

2. The method for intelligently generating interactive music in an adaptive scene space according to claim 1, characterized in that: The scene space data includes: layout information, state information, geographic information, weather information, lighting information, time information, cultural information, visual information, auditory information, physical information, smell information and story information in the scene space; The user information data includes: user basic information, user physiological information, user behavior information and user voice information; The music data is obtained from an open source music dataset.

3. The method for intelligently generating interactive music in an adaptive scene space according to claim 1, characterized in that: The training process of the CSMP model includes: Collect music data from open source music datasets and extract Mel-spectrogram feature maps of the music data; The scene space data and user information data are integrated to obtain multi-dimensional scene data; Using the Mel-spectrogram features of the multivariate scene data and the music data, a multivariate scene data-music dataset is constructed according to the pre-defined correspondence between the multivariate scene data and the music data; Inputting the multi-scenario data and the corresponding music data into the multi-scenario data encoder and the music data encoder respectively, obtaining the scene feature vector of the multi-scenario data and the music feature vector of the music data; Map the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m ; Construct the total loss function of the CSMP model; According to the normalized vector s and the normalized vector m As well as the total loss function of the CSMP model, through contrast learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

4. The method for intelligently generating interactive music in an adaptive scene space according to claim 3, characterized in that: The total loss function of the CSMP model includes: the loss function of retrieving music data using multi-scenario data and the loss function of retrieving multi-scenario data using music data; Among them, the loss function of multi-scenario data retrieval music is: in, In order to utilize the loss of multi-scenario data to retrieve music data, N is the amount of music data, For the i Multi-scenario data and j The similarity between music data, For the j The characteristic data of music, For the i Feature data of multiple scenarios; The loss function for music data retrieval of multi-scenario data is: in, In order to use music data to retrieve the loss of multi-scene data, For the i Music data and j The similarity between multiple scene data, Z is the number of multivariate scene data.

5. An interactive music intelligent generation system for adaptive scene space, characterized in that: The system includes: an information acquisition module, an interactive data processing and feature extraction module, an adaptive scene music generation module, and a multi-modal feedback module based on music generation; The information acquisition module is used to obtain scene space data and user information data through an interactive device, and transmit the scene space data and the user information data to the interactive data processing and feature extraction module; The interactive data processing and feature extraction module is used to pre-process the scene space data and the user information data and then fuse them to obtain multivariate scene data; the multivariate scene data is processed by a trained CSMP model, and the generated high-dimensional features of the multivariate scene data are spliced ​​according to the time sequence to obtain a multivariate scene data sequence; the multivariate scene data sequence is processed in an interactive manner with the GPT model to obtain interactive high-dimensional features of the interactive data; the CSMP model is a dual encoder architecture, including a multivariate scene data encoder and a music data encoder; the multivariate scene data is text data; The adaptive scene music generation module is used to obtain music data, extract the Mel-spectrogram feature map of the music data, process the Mel-spectrogram feature map using a trained CSMP model to obtain a music feature vector of the music data; input the music feature vector, the Mel-spectrogram feature map and the interactive high-dimensional features into a trained conditional controlled diffusion model to generate a Mel-spectrogram feature map of the reconstructed music data; process the Mel-spectrogram feature map of the reconstructed music data using a HiFi-GAN pre-trained model to generate corresponding scene music; The multimodal feedback module based on music generation is used to feed back the scene music generated by the adaptive scene music generation module to the scene space and add it to the music data; Among them, the conditional controlled diffusion model is a latent diffusion model, including a VAE encoder, a UNet+Attention model and a VAE decoder; the adaptive scene music generation module is also used to input the Mel frequency spectrum feature map into the VAE encoder to obtain the latent feature vector of the music data; the latent features of the music data are forward diffused through the conditional controlled diffusion model to obtain a noisy low-dimensional latent vector; based on the noisy low-dimensional latent vector, the interactive high-dimensional features and the music feature vector are used as control conditions, and the UNet+Attention model is used to perform noise prediction to obtain predicted noise; after the predicted noise is processed through the VAE decoder, a Mel frequency spectrum feature map of the reconstructed music data is generated.

6. The interactive music intelligent generation system of adaptive scene space according to claim 5 is characterized in that: The information collection module includes: a scene space data collection module, a user information data collection module and a data set construction module; The scene space data collection module is used to collect layout information, state information, geographic information, weather information, lighting information, time information, astronomical information, visual information, auditory information, physical information, smell information and story information of the scene using the corresponding interactive device; The user information data collection module is used to collect the user's basic information, physiological information, behavioral information and voice information using the corresponding interactive device; The data set construction module is used to collect data from different scenes using interactive devices, and by analyzing the melody, rhythm and emotion of the music, the collected scene data is matched one-to-one with the music data to construct a multi-scene data-music data set.

7. The interactive music intelligent generation system of adaptive scene space according to claim 5 is characterized in that: The multimodal feedback module generated based on music is also used to wake up the output device close to the user according to the scene space data and the user information data, match the best output device, and select the multimodal output interactive device linked with the music; the multimodal output interactive device is used to display the multimodal content synchronized with the music, and is linked with the generated scene music.

8. The interactive music intelligent generation system of adaptive scene space according to claim 5 is characterized in that: The training process of the CSMP model in the interactive data processing and feature extraction module includes: Collect music data from open source music datasets and extract Mel-spectrogram feature maps of the music data; The scene space data and user information data are integrated to obtain multi-dimensional scene data; Using the Mel-spectrogram features of the multivariate scene data and the music data, a multivariate scene data-music dataset is constructed according to the pre-defined correspondence between the multivariate scene data and the music data; Inputting the multi-scenario data and the corresponding music data into the multi-scenario data encoder and the music data encoder respectively, obtaining the scene feature vector of the multi-scenario data and the music feature vector of the music data; Map the scene feature vector to a normalized vector in the latent space s , mapping the music feature vector to a normalized vector in the latent space m ; Construct the total loss function of the CSMP model; According to the normalized vector s and the normalized vector m As well as the total loss function of the CSMP model, through contrast learning, the distance between the matching multi-scene data-music data pairs in the vector space is made closer, while the distance between the unmatched multi-scene data-music data pairs is made farther, thus obtaining a trained CSMP model.

9. The interactive music intelligent generation system of adaptive scene space according to claim 8 is characterized in that: The total loss function of the CSMP model includes: the loss function of retrieving music data using multi-scenario data and the loss function of retrieving multi-scenario data using music data; Among them, the loss function of multi-scenario data retrieval music data is: in, In order to utilize the loss of multi-scenario data to retrieve music data, N is the amount of music data, For the i Multi-scenario data and j The similarity between music data, For the j The characteristic data of music, For the i Feature data of multiple scenarios; The loss function for music data retrieval of multi-scenario data is: in, In order to utilize the loss of multi-scenario data to retrieve music data, For the i Music data and j The similarity between multiple scene data, is the number of multivariate scene data.

Images