Method for realizing high-dimension alignment and conversion of multi-modal features (vision and music)

Abstract

The invention provides a method for realizing high-dimensional alignment and conversion of multi-modal features (vision and music), and aims to solve the problems of insufficient alignment of the multi-modal features, lack of consistency of styles and emotions and insufficient optimization of generated contents in the prior art. According to the method, through a multi-modal feature extraction module, a multi-modal feature alignment and mapping module, a dynamic generation and optimization module and a user interaction and feedback module, consistency of vision and music in style, emotion and dynamic characteristics is achieved, and accurate conversion from one mode to another mode is supported. Experimental results show that the method is remarkably superior to the prior art in feature alignment precision and generated content consistency, and has wide application prospects and technical innovation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence (AI) and multimodal interaction technology, and specifically relates to a technical method based on multimodal feature extraction and mapping, which aims to achieve high-dimensional alignment and dynamic conversion of visual art style and musical art style for cross-modal content generation and multi-scene immersive experience. Background Art

[0002] Definition of Key Terms Multimodal features: refers to data from different modalities (such as vision and music), which are converted into high-dimensional vector representations through feature extraction methods for subsequent alignment and mapping; High-dimensional feature space: refers to the multidimensional space where the feature vectors are located, which is used to uniformly represent the features of different modes for easy alignment and mapping; Cycle consistency loss: A loss function used to ensure consistency when generating from one modality to another and then back to the original modality. It is often used in models such as CycleGAN. Contrastive learning: A self-supervised learning method that learns effective feature representations by comparing similar and dissimilar pairs of samples; KL divergence (Kullback-Leibler Divergence): An indicator used to measure the degree of difference between two probability distributions.

[0003] Limitations of existing technologies With the development of artificial intelligence and multimodal learning technology, the interaction and conversion between different modalities (such as text, images, and audio) has gradually become a research hotspot. Especially in the fields of artistic creation, metaverse scenes, education, and cultural creation, how to achieve artistic style conversion between vision and hearing has become an important direction of multimodal research.

[0004] Existing technologies mainly focus on: Image generation technology: such as DALL-E and MidJourney, which can generate high-quality images based on text descriptions; Music generation technology: Such as Suno and Udio, which can generate dynamic music content based on descriptions.

[0005] However, the existing technology lacks a high-dimensional alignment method for multimodal features between vision and music, and cannot achieve accurate style mapping and consistency in emotional expression. Therefore, there are the following technical problems that need to be solved urgently: Cross-modal alignment of multimodal features: How to map the key features of vision and music into a unified high-dimensional feature space to achieve accurate alignment; Style and emotional consistency: How to ensure consistency in artistic style and emotional expression between visual content (e.g., color, composition) and musical content (e.g., melody, rhythm); Dynamic generation and optimization: How to support the generation of content from one modal to another and allow users to dynamically adjust the generated content. Summary of the invention

[0006] Technical issues The present invention aims to solve the problems of insufficient multimodal feature alignment, lack of consistency between style and emotion, and insufficient optimization of generated content in the prior art, specifically including: Cross-modal alignment of multimodal features: Extract key features of vision and music, map different modalities into a unified high-dimensional feature space, and achieve precise alignment; Stylistic and emotional consistency: Ensure consistency in artistic style and emotional expression between visual content (e.g., color, composition) and musical content (e.g., melody, rhythm); Dynamic generation and optimization: Supports generating content from one modality to another, and also supports users to adjust the generated content.

[0007] Technical Solution In order to solve the above technical problems, the present invention provides a method for realizing high-dimensional alignment and conversion of multimodal features (visual and musical), and its technical solution includes the following steps: Multimodal feature extraction module: extracts high-dimensional features of images and music, including style, emotion, dynamic characteristics, etc.; Multimodal feature alignment and mapping module: Establishing mapping rules between visual and music features to ensure alignment of features in high-dimensional space; Dynamic generation and optimization module: Based on feature mapping, it calls the image generation or music generation interface to generate multimodal content consistent with the input; User interaction and feedback module: Provides interactive tools for users to adjust style and emotional weight, and optimizes generation logic.

[0008] Beneficial Effects The present invention has the following beneficial effects: High-precision feature alignment: Through high-dimensional feature alignment methods, accurate mapping of visual and musical features is achieved for the first time, significantly improving the quality of cross-modal generated content; Consistency guarantee: Introduce a cycle consistency loss mechanism to ensure the stability and consistency of bidirectional generated content; Interactivity optimization: Supports users to adjust emotion and style parameters in real time, providing personalized and customized generation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to clearly illustrate the present invention, the following description is given with reference to the accompanying drawings.

[0010] Figure 1 It is a schematic diagram of the system architecture of the present invention, showing the five core modules of the system and their connection relationships.

[0011] Figure 2 This is the flow chart of the multimodal feature extraction module, showing the specific steps of visual feature extraction and music feature extraction.

[0012] Figure 3 Schematic diagram of the implementation of the multimodal feature alignment and mapping module, illustrating the process of feature mapping and sentiment consistency modeling.

[0013] Figure 4 It is a workflow diagram for the dynamic generation and optimization module, describing the specific steps of content generation and optimization.

[0014] Figure 5 This is a schematic diagram of the user interaction and feedback module interface, showing the interface design for user parameter adjustment and feedback collection. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] The system architecture of the present invention is as follows Figure 1 As shown in the figure, it mainly consists of five core modules, namely user input module, multimodal feature extraction module, multimodal feature alignment and mapping module, dynamic generation and optimization module, and user interaction and feedback module. In practical applications, users first provide raw data through the user input module. These data can be images, music or text descriptions as the starting point for multimodal content generation.

[0017] Specifically, the user input module supports multiple input forms, including common image formats (such as JPEG, PNG), audio formats (such as MP3, WAV) and natural language text descriptions. Users can upload image files, audio files or enter text descriptions through the graphical user interface (GUI). After the system receives the input, the data will be passed to the multimodal feature extraction module for processing.

[0018] Furthermore, the multimodal feature extraction module is Figure 2As shown in Figure 1, it is responsible for extracting high-dimensional feature vectors from images and music provided by users. These feature vectors cover multiple dimensions such as style, emotion, and dynamic characteristics, providing a basis for subsequent feature alignment and mapping. For image input, the system uses a pre-trained ResNet-50 convolutional neural network (CNN) architecture for feature extraction. First, the input image is preprocessed, including uniformly adjusting the image to a fixed size (such as 224x224 pixels), converting the color space (from BGR to RGB), and normalizing it so that the image adapts to the input requirements of ResNet-50. The preprocessed image is extracted through the middle layer of the ResNet-50 network, followed by global average pooling to obtain a feature vector of fixed length.

[0019] During the image feature extraction process, the system further classifies color features, composition features, and style features. Color features include color-related statistical data such as RGB distribution and saturation; composition features involve image symmetry, geometric complexity, and other composition-related parameters; style features use style transfer networks or style classifiers to identify the artistic style of an image, such as impressionism, cubism, etc. In addition, the visual sentiment analysis model is used to identify the emotional state expressed by the image, such as tranquility, excitement, etc. The final generated visual feature vector is output.

[0020] On the other hand, for music input, the multimodal feature extraction module uses the BERT-based Transformer model to perform feature analysis on the audio. First, the input music is preprocessed, including frame processing, fast Fourier transform (FFT), and Mel-spectrogram generation to simulate the human ear's perception of different frequencies. The preprocessed audio is extracted through the Transformer model to extract high-level audio features, and further classified into rhythm features (such as beats per minute BPM), pitch features, and emotional features (such as excitement, tranquility). The generated music feature vector Is output for subsequent feature alignment and mapping.

[0021] After feature extraction is completed, the data will enter the multimodal feature alignment and mapping module, such as Figure 3 As shown. The main task of this module is to transform the visual features and music features Mapping to a unified high-dimensional feature space to achieve cross-modal feature alignment and consistency. Specifically, the system constructs a mapping function through a deep neural network , using contrastive learning method and KL divergence minimization strategy to optimize the distribution consistency between visual features and music features. Contrastive learning improves the accuracy of feature alignment by maximizing the similarity between similar feature pairs and minimizing the similarity between dissimilar feature pairs. At the same time, KL divergence (Kullback-Leibler Divergence) is used to measure the difference in probability distribution between visual features and music features. By minimizing KL divergence, the consistency of feature distribution is further promoted.

[0022] On this basis, the emotional consistency model ensures that the visual content and the music content are in the emotional vector and The system designs the sentiment loss function To minimize the difference between visual and music emotion vectors, thus ensuring the consistency of cross-modal emotion expression. This process not only improves the accuracy of feature alignment, but also enhances the coordination of generated content at the emotional level.

[0023] Dynamically generate and optimize modules such as Figure 4 As shown, the corresponding multimodal content is generated based on the mapped features, and the consistency and high quality of the generated content with the input are ensured through the optimization mechanism. In terms of image generation, the system calls the MidJourney API to generate images based on text descriptions or music features. The specific steps include converting the mapped music features or the text description provided by the user into the input format required by the API (such as JSON), and then passing the input features through HTTP requests or using the SDK provided by MidJourney to generate images of the corresponding style. The generated images undergo necessary post-processing, such as denoising, color correction, and resizing, to improve image quality and visual effects.

[0024] In terms of music generation, the system calls the Suno API to generate music based on image features. The specific steps include converting the mapped visual features into the input format required by the API (such as serialization of feature vectors), passing the input features through HTTP requests or using the SDK provided by Suno, and generating music of the corresponding style. The generated music undergoes necessary post-processing, such as volume adjustment, editing, and format conversion, to ensure the fluency and playback quality of the music.

[0025] To further improve the generation quality, the system introduces the CycleGAN architecture, which uses the cycle consistency loss function Optimize the generated content. Specifically, the generating function GGG maps visual features to music features to generate music content; the generating function The music features are mapped back to the visual features to generate the image content. The cycle consistency loss ensures that after generating one modality from another, the reverse generation back to the original modality remains consistent. Through the back-propagation algorithm, the system minimizes the cycle consistency loss and optimizes the generation function and parameters to gradually improve the consistency and quality of generated content.

[0026] User interaction and feedback modules such as Figure 5 As shown in the figure, an interface is provided for users to adjust the style and emotional parameters of the generated content in real time. Users can adjust parameters such as the warmth and coldness of the tone, the speed of the melody, and the intensity of the emotion through sliders or knobs. The system displays the generated images and plays the generated music in real time, allowing users to instantly view and listen to the adjustment effects. After the user completes the adjustment, he can click the Submit button, and the system will record the adjustment data and optimize the feature mapping rules through the reinforcement learning mechanism to achieve a personalized and customized generation experience.

[0027] Specifically, the system builds a user preference database by recording each parameter adjustment and feedback information of the user. Subsequently, the reinforcement learning algorithm is used to optimize the feature mapping rules based on the user feedback data, so that the generated content is more in line with the user's preferences. The reward mechanism promotes the generation model to learn the user's preferred patterns by giving high rewards for positive feedback and low rewards or penalties for negative feedback. Policy optimization optimizes the generation strategy through methods such as policy gradient algorithms or Q-learning, so that the generation model gradually adapts to the user's personalized needs. In addition, the system dynamically adjusts the parameters of the generation model according to the feedback data of different users to achieve a personalized and customized generation experience.

[0028] In a specific application, Example 1 shows the conversion process from vision to music. The user uploads an impressionist forest picture through the user input module, and the system extracts visual features through the multimodal feature extraction module. , and then the visual features are mapped to the music feature space through the multimodal feature alignment and mapping module Next, the dynamic generation and optimization module calls the Suno interface and generates the music based on the mapped music features. Generate a matching calm melody. Users can adjust the rhythm and emotional weight of the melody through the interactive module. The system records these adjustments and optimizes the mapping rules through a reinforcement learning mechanism, ultimately generating music that is highly consistent with the input visual content.

[0029] Example 2 shows the conversion process from music to vision. The user inputs a piece of fast-paced, high-pitched electronic music through the user input module, and the system extracts music features through the multimodal feature extraction module. , and then the music features are mapped to the visual feature space through the multimodal feature alignment and mapping module The dynamic generation and optimization module calls the MidJourney interface and maps the visual features. Generate futuristic images. Users can adjust the color and composition parameters of the image through the interactive module. The system records these adjustments and optimizes the mapping rules through the reinforcement learning mechanism, and finally generates images that are consistent with the input music content in terms of emotion and style.

[0030] In order to verify the effectiveness and superiority of the present invention, a series of experiments were conducted. First, through user surveys and sentiment analysis models, the consistency of emotions and styles between the generated music and the input visual content was scored. The experimental results show that the emotional consistency score between the music generated by the present invention and the input visual content reached more than 90%, which is significantly higher than the 60% of the prior art. Secondly, through comparative experiments, the method of the present invention was compared and evaluated with the prior art in terms of feature alignment accuracy and generated content consistency. The results show that the present invention has improved both feature alignment accuracy and content consistency by 30%, greatly enhancing the artistic consistency of multimodal content generation.

[0031] In summary, the present invention successfully achieves the consistent conversion of style and emotion between vision and music through multimodal feature extraction, high-dimensional mapping, emotional consistency modeling and dynamic generation optimization. This method not only improves the quality and consistency of multimodal content generation, but also optimizes the generation results through user interaction, and has broad application prospects and significant technical innovation. The system and method proposed in the invention show superior performance and wide applicability in many fields such as artistic creation, metaverse, education, psychotherapy, film and television production, game development, advertising creativity, etc., and have high market value and promotion potential.

Claims

1. A method for high-dimensional alignment and conversion of multimodal features (visual and musical), characterized in that: The following steps are involved: Multimodal feature extraction: Extract high-dimensional visual features and music features from input images and music. The visual features include color, texture, composition, and emotional features, and the music features include rhythm, pitch, beat, and emotional features. Feature mapping and alignment: Map the visual features and music features to a unified high-dimensional feature space, construct a mapping function through a deep neural network to achieve the conversion of visual features to music features, and optimize the consistency of features through contrastive learning and KL divergence minimization; Content generation: Based on the mapped features, the corresponding image generation interface or music generation interface is called to generate multimodal content that is highly consistent with the input content in style and emotion; Cycle consistency optimization: A cycle consistency loss function is used to ensure that the content is consistent when generating from one modality to another and then back to the original modality; User Interaction and Feedback: Provide a user interface that allows users to adjust the style and sentiment parameters of generated content in real time, and optimize feature mapping rules based on user feedback.

2. The method according to claim 1, characterized in that The multimodal feature extraction step extracts the color, texture, and composition features of the image using a pre-trained convolutional neural network (CNN), and extracts the rhythm, pitch, and emotion features of the music using a BERT-based Transformer model.

3. The method according to claim 2, characterized in that The convolutional neural network is specifically a ResNet-50 architecture.

4. The method according to claim 1, characterized in that In the feature mapping and alignment steps, the mapping function is implemented by a multi-layer perceptron (MLP), and a Siamese network structure is used for contrastive learning.

5. The method according to claim 1, characterized in that In the cycle consistency optimization step, the CycleGAN architecture is used to minimize the cycle consistency loss To optimize generated content.

6. The method according to claim 1, characterized in that In the user interaction and feedback step, the user interface includes a parameter adjustment area, a preview area and a feedback submission button, and the parameter adjustment area allows the user to adjust the warmth and coldness of the tone, the speed of the melody and the intensity of the emotion.

7. The method according to claim 1, characterized in that The user interaction and feedback step further includes dynamically optimizing feature mapping rules according to user feedback data through a reinforcement learning mechanism.

8. A system for high-dimensional alignment and conversion of multimodal features (visual and musical), characterized in that: include: User input module: used to receive images, music or text descriptions provided by users as input; Multimodal feature extraction module: used to extract high-dimensional visual features and music features from user input data; Multimodal feature alignment and mapping module: used to map the extracted visual features and music features into a unified high-dimensional feature space, and optimize the consistency of features through contrastive learning and KL divergence minimization; Dynamic generation and optimization module: used to call the image generation interface or music generation interface based on the mapped features, generate multimodal content that is highly consistent with the input content in style and emotion, and optimize it using cycle consistency loss; User interaction and feedback module: Used to provide a user interface that allows users to adjust the style and emotional parameters of generated content in real time, and optimize feature mapping rules based on user feedback.

9. The system according to claim 8, characterized in that The multimodal feature extraction module uses a convolutional neural network (CNN) with a ResNet-50 architecture to extract the color, texture, and composition features of the image, and uses a BERT-based Transformer model to extract the rhythm, pitch, and emotional features of the music.

10. The system according to claim 8, characterized in that The multimodal feature alignment and mapping module constructs a mapping function through a multi-layer perceptron (MLP) and adopts a Siamese network structure for contrastive learning, while optimizing the consistency between visual features and music features by minimizing the KL divergence.

Images