Music auxiliary image generation method based on musical elements extraction

Abstract

This invention discloses a music-assisted image generation method based on music theory element extraction. The method specifically includes: extracting six-dimensional fine-grained structured music theory semantics from audio using a multimodal large language model and performing cross-modal consistency verification; constructing a neutral scene description based on the NSA (Neutral, Concise, White Space) principle to provide a semantic whiteboard; intelligently translating music theory elements into visual modification instructions using the large language model, adaptively rewriting the neutral scene to generate fusion prompts; and using the FastVAR model for efficient image generation. Compared with existing technologies, this invention effectively solves the problems of black-box feature extraction and semantic alignment ambiguity in audio-driven generation, eliminates logical conflicts caused by cross-modal splicing, and achieves high-quality generation while ensuring high-fidelity image texture details and semantic consistency. The method is intelligent and efficient, and has good application prospects in the fields of intelligent art creation and music visualization.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cross-modal generation technology, specifically to a music-assisted image generation method based on fine-grained music theory extraction. Background Technology

[0002] With the generational leap in generative artificial intelligence (AIGC) technology, text-to-image (T2I) technology has made groundbreaking progress, especially the large-scale application of latent diffusion models, which has greatly expanded the boundaries of visual creation. Driven by this technological wave, the research focus of academia and industry is gradually evolving towards more challenging cross-modal generation. Among them, audio-driven image generation aims to break down the semantic barriers between auditory and visual senses, realizing the interaction and transformation of multimodal information, and has become a cutting-edge hot topic in the fields of multimodal cognitive computing and intelligent art creation.

[0003] However, unlike ambient sounds which typically correspond to specific physical sound sources (e.g., "dog barking" corresponds to "dog"), music, as a highly abstract, time-flowing, and emotionally rich symbolic system, faces significant challenges in its visualization. Early audio-driven image generation methods primarily employed end-to-end mapping strategies, attempting to directly establish a conditional probability model between audio features and image pixel distribution using Generative Adversarial Networks (GANs) or simple diffusion models, compressing complex audio signals into a single latent vector. This existing technology suffers from the following significant technical drawbacks in practical applications:

[0004] Feature extraction is fragmented and lacks fine-grained music theory understanding. Existing cross-modal models, when processing input audio, often only extract global acoustic features (such as Mel spectrograms) or rely on simple classification labels (such as "sad" and "rock"). This "black box" feature extraction approach causes the model's understanding of the music itself to remain superficial, making it difficult to capture subtle music theory elements such as microscopic orchestration texture, pitch changes, and macroscopic tonal color and formal structure. As a result, the final generated visual image inherently lacks information in terms of the richness and structural sense of the corresponding music.

[0005] The generated content is heavily coupled with emotion, resulting in semantic confusion. Traditional end-to-end generative models lack explicit control mechanisms over the generated content, making it difficult to distinguish between the abstract imagery conveyed by music and the concrete entities in images. For example, when inputting music with "sacred" or "ethereal" qualities, the model often incorrectly concretizes this emotion directly into specific physical entities (such as generating "angels" or "churches"), instead of rendering the corresponding atmosphere in the user's desired scene. This "semantic confusion" phenomenon, which forcibly maps abstract auditory experiences to fixed concrete objects, not only disrupts the physical logic of the original scene but also severely limits the accuracy and controllability of the generated results in artistic expression.

[0006] In summary, existing audio-driven image generation technologies often suffer from "audio-visual asynchrony" and "loss of atmosphere" during cross-modal alignment due to the lack of a unified, fine-grained music theory modeling paradigm and explicit content control mechanisms. Therefore, breaking down the black-box mechanism, establishing an interpretable music theory feature extraction system, and achieving explicit decoupling and precise fusion of subjective musical emotion and objective scene content are key technical problems that urgently need to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a music-assisted image generation method based on music theory element extraction. It employs an expert-guided large language model analysis architecture to explicitly deconstruct continuous audio signals into a multi-dimensional structured music theory description including orchestration, style, tonality, time signature, tempo, and mood. A pre-trained cross-modal contrastive model is used to perform consistency checks to accurately filter semantic illusions. A standardized scene description based on the NSA (Neutral, Simple, White Space) principle is constructed to provide a semantic whiteboard for cross-modal mapping. An adaptive synesthetic fusion mechanism is built using the large language model to intelligently translate the extracted music theory elements into visual modification instructions. The neutral scene is adaptively rewritten through attribute slot filling and conflict resolution strategies to generate logically consistent enhanced fusion prompts. Finally, the fusion prompts are input into a visual generation model to complete the rendering from text to a high-fidelity image. This invention effectively solves the problems of black-box feature extraction and lack of fine-grained music theory cognition in traditional audio-driven image generation by establishing a full-link framework of fine-grained music theory extraction and adaptive synesthetic fusion. It fundamentally overcomes the semantic confusion caused by the severe coupling between generated content and emotion, realizes the paradigm shift from implicit acoustic features to explicit structured semantics, eliminates the logical conflicts caused by multimodal semantic splicing, and significantly improves the controllability and interpretability of the generation process while ensuring the consistency of image texture details and cross-modal semantics. The method is intelligent, efficient, and effective, and has good application prospects in fields such as intelligent art creation, immersive media design, and music visualization.

[0008] The objective of this invention is achieved as follows:

[0009] A method for generating music-assisted images based on music theory element extraction, the method comprising the following steps:

[0010] Step 1: Input the preprocessed audio data into an expert-guided micro-descriptor to explicitly deconstruct the audio signal into fine-grained structured music theory elements;

[0011] Step 2: Introduce a cross-modal consistency verification mechanism to calculate and filter the similarity of structured music theory elements, and eliminate semantic bias and hallucination noise;

[0012] Step 3: Build and provide standardized scene descriptions based on NSA principles as a neutral semantic whiteboard for cross-modal mapping;

[0013] Step 4: Use a large language model to perform adaptive synesthetic fusion, interpret the extracted music theory elements into visual modification instructions and rewrite the neutral scene description to obtain enhanced fusion prompt words;

[0014] Step 5: Input the enhanced fusion prompt into the visual generation model to complete the generation and rendering of the high-fidelity image from the text.

[0015] Furthermore, step 1 specifically includes:

[0016] 1-1: Construct an expert-oriented analysis architecture, constrain the behavior patterns of the multimodal large language model through system-level instructions, and define its music analysis role;

[0017] 1-2: Design a structured thinking chain guidance template to force the model to focus on and extract music theory elements in six dimensions: orchestration, style, tonality, time signature, tempo, and mood.

[0018] 1-3: Constrain the output format of the model, normalize the natural language prose description into a machine-readable structured list and JSON object, and output the initial six-dimensional structured music theory description.

[0019] By constructing an expert-guided analysis architecture based on a multimodal large language model, and combining structured thinking chain prompts and output format constraints, the preprocessed continuous audio signal is explicitly deconstructed into fine-grained structured music theory elements containing six dimensions.

[0020] Furthermore, the six dimensions of music theory elements follow the following synesthetic isomorphism mechanism in cross-modal mapping:

[0021] Among them, the instrumentation dimension is used to map the texture and material of the visual image; the style dimension is used to define the overall artistic style and refine the brushstrokes; the tonality dimension is used to control the color temperature and light of the image; the time signature dimension is used to determine the spatial composition and geometric layout; the speed dimension is used to map the dynamic effect and visual tension of the image; and the emotion dimension, as the highest level of semantics, governs the overall atmosphere and tone of the image.

[0022] Furthermore, step 2 specifically includes:

[0023] 2-1: Extract the global acoustic feature vector of the input audio using a pre-trained contrastive language-audio model (CLAP) audio encoder;

[0024] 2-2: Using the text encoder of the contrastive language-audio model, the fine-grained music theory descriptions generated in step 1 are encoded into text feature vectors respectively;

[0025] 2-3: Calculate the cosine similarity between the global acoustic feature vector and each text feature vector to quantify the degree of semantic consistency;

[0026] 2-4: Set a confidence threshold, retain text descriptions with similarity scores higher than the threshold, remove noisy descriptions with scores lower than the threshold, and output high-purity structured music theory semantics.

[0027] A cross-modal consistency verification mechanism based on a contrastive language-audio model is introduced. By calculating the cosine similarity between acoustic and text feature vectors and setting a threshold for filtering, semantic bias and hallucination noise are eliminated, and high-purity structured music theory semantics are output.

[0028] Furthermore, step 3 specifically includes:

[0029] 3-1: In accordance with the principle of neutrality, remove modifiers with subjective emotional coloring from the scene description and retain only the objective statements of physical entities;

[0030] 3-2: Following the principle of simplicity, reduce visual distractions and adopt a minimalist descriptive strategy to establish a single visual focus;

[0031] 3-3: In accordance with the principle of ambiguity, semantic generalization processing is performed on the scene's lighting conditions, weather conditions, and object materials to inject reserved programmable semantic slots into music theory features.

[0032] Based on the NSA principles of neutrality, simplicity, and white space, a standardized scene description is constructed as a neutral semantic whiteboard for cross-modal feature injection by stripping away emotional embellishments, simplifying visual focus, and performing attribute generalization processing.

[0033] Furthermore, step 4 specifically includes:

[0034] 4-1: Activate the synesthetic translation mechanism of the large language model, and map discrete music theory elements into visual modification instructions that include light and shadow, color tone, material details and atmosphere through logical reasoning;

[0035] 4-2: By adopting the attribute slot filling strategy, visual modification instructions are seamlessly embedded into the neutral semantic whiteboard, so as to preserve the core physical structure of the original scene without tampering while rendering the musical atmosphere;

[0036] 4-3: Activate the logical conflict resolution mechanism, process the potential semantic paradox between abstract musical emotions and concrete scene physical attributes through stylistic transfer, and output logically consistent enhanced fusion prompts.

[0037] An adaptive synesthetic fusion mechanism is constructed using a large language model. Through synesthetic translation mapping of visual instructions, attribute slot filling, and logical conflict resolution, structured music theory elements are intelligently rewritten into a neutral semantic whiteboard, generating enhanced fusion prompts that are logically consistent and retain the original physical structure.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0039] 1) This invention proposes a multi-agent collaborative analysis mechanism based on a large language model, which explicitly deconstructs chaotic continuous audio into a structured music theory description with six dimensions: orchestration, style, tonality, time signature, tempo, and mood. This approach not only fills the gap in cross-modal alignment of fine-grained music theory features, but also enables the generated images to obtain accurate and interpretable semantic guidance in terms of structural tension, color warmth and coolness, and texture.

[0040] 2) The Neutral, Simple, and White Space (NSA) principle proposed in this invention strips away the emotional noise of the initial scene and utilizes the logical reasoning capabilities of a large language model to adaptively translate auditory features into visual modification instructions and fill them into attribute slots. This mechanism successfully decouples subjective musical emotions from objective physical scenes, achieving a deep immersion and stylized reconstruction of the visual imagery through musical atmosphere, ensuring a high degree of unity between "sound" and "image" in both aesthetics and logic. Attached Figure Description

[0041] Figure 1 This is a flowchart of the present invention;

[0042] Figure 2 is a schematic diagram of the specific operation of Example 1. Detailed Implementation

[0043] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0044] To address the issues of black-box feature extraction and cross-modal semantic confusion in traditional audio-driven image generation, this invention proposes a music-assisted image generation method based on music theory element extraction. Utilizing a multi-agent collaborative analysis framework based on a large language model, continuous audio signals are explicitly deconstructed into six-dimensional structured music theory semantics. A cross-modal consistency verification mechanism based on the Contrastive Language-Audio Model (CLAP) is then used to rigorously filter semantic illusions. Next, a cross-modal semantic whiteboard based on the NSA (Neutral, Simple, White Space) principle is constructed to remove the emotional priors of the initial scene. A large language model-driven synesthetic translation mechanism is designed to adaptively map auditory features to visual modification instructions. Attribute slot filling and semantic conflict resolution are performed sequentially to generate logically consistent enhanced fusion prompts. Finally, these enhanced fusion prompts are used as conditional guidance signals input into a visual generation network, sequentially performing low-resolution spatial structure construction and high-resolution material texture filling, ultimately generating a high-fidelity image highly aligned with the emotional content of the input audio.

[0045] See Figure 1 The present invention provides a music-assisted image generation method based on music theory element extraction, which specifically includes the following steps:

[0046] 1) Construct a multi-agent collaborative analysis framework to extract initial music theory semantics.

[0047] 1.1: Receive preprocessed audio data and input it into a group of intelligent agents with specialized division of labor based on a large language model to perform modular extraction of fine-grained features;

[0048] 1.2: Construct independent analysis agents for different music theory dimensions, and strictly constrain the areas of interest of each agent through system instructions to avoid semantic interference between dimensions;

[0049] 1.3: Explicitly deconstruct the continuous audio signal and output an initial structured music theory description containing six dimensions: orchestration, style, tonality, time signature, tempo, and mood.

[0050] 2) Perform cross-modal consistency checks and semantic filtering

[0051] 2.1: Use a pre-trained contrastive language-audio model (CLAP) to extract the global feature vector of the original audio, and encode the initial structured music theory description of each dimension output in step 1) into text feature vectors respectively;

[0052] 2.2: Calculate the cosine similarity between the global audio feature vector and each text feature vector to quantify the degree of cross-modal semantic consistency;

[0053] 2.3: Combining the consistency score of multi-agent collaboration with the cosine similarity score, a comprehensive confidence threshold is set for filtering, eliminating semantic bias and hallucination noise with scores below the threshold, and outputting the final music theory semantics with high purity.

[0054] 3) Construct a cross-modal semantic whiteboard based on NSA principles

[0055] 3.1: Obtain the preset original image scene description, following the NSA principles of neutrality, simplicity, and ambiguity, and use the text parsing module to remove all adjectives with subjective emotional color and specific environmental rendering attributes from the scene description;

[0056] 3.2: Only objective statements of physical entities in the description are retained, and semantic generalization processing is performed on visual attributes such as lighting conditions, weather conditions and object materials in the picture using extremely concise sentence structure to generate a neutral semantic whiteboard with programmable attribute slots.

[0057] 4) Auditory-visual synesthetic translation based on a large language model

[0058] 4.1: Input the high-purity final music theory semantics output from step 2) into the large language model and activate the synesthetic translation mechanism;

[0059] 4.2: Through logical reasoning of the large language model, discrete auditory symbols are adaptively mapped to corresponding visual modification instructions; among them, tone is mapped to hue warmth and coolness parameters, orchestration is mapped to texture parameters, tempo and time signature are mapped to dynamic tension and composition parameters, and emotion is mapped to environmental atmosphere parameters.

[0060] 5) Perform attribute slot filling and semantic conflict resolution

[0061] 5.1: Using the attribute slot filling strategy, the various visual modification instructions generated in step 4) are precisely embedded into the corresponding empty slots of the neutral semantic whiteboard obtained in step 3, so as to achieve the initial immersion of the musical atmosphere in the visual scene.

[0062] 5.2: Activate the logic conflict resolution module to detect potential semantic paradoxes between abstract musical emotion commands and concrete scene physical structures;

[0063] 5.3: When a logical conflict is detected, the modification instructions are soft-landed and corrected through an artistic stylization transfer algorithm, strictly preserving the core physical structure of the original scene without tampering, and finally outputting a logically consistent enhanced fusion prompt.

[0064] 6) Cross-modal high-fidelity image generation based on enhanced fusion prompts

[0065] 6.1: Encode the enhanced fusion prompt words output in step 5) into high-dimensional semantic vectors, and input them into the visual generative model as conditional guidance signals;

[0066] 6.2: In the low-resolution structure building stage, the visual generation model prioritizes responding to the macro-instructions and structural music theory elements in the fusion prompts to establish the spatial composition and geometric layout of the generated image;

[0067] 6.3: In the high-resolution texture filling stage, the model focuses on responding to the music theory elements of the style layer and aesthetic layer, and performs material texture filling and lighting atmosphere rendering for high-frequency areas that have not yet converged, and finally generates and outputs a high-fidelity image that is highly aligned with the emotional content of the input audio.

[0068] Example

[0069] See Figure 2 In this embodiment, preprocessed audio data is input into a multi-agent collaborative analysis framework based on a large language model, explicitly deconstructing it to obtain an initial structured music theory description containing six dimensions: orchestration, style, tonality, time signature, tempo, and mood. Then, a pre-trained contrastive language-audio model (CLAP) is used to extract audio and text feature vectors and calculate cosine similarity. Semantic illusions are filtered out using a comprehensive confidence threshold to obtain a high-purity final music theory semantic. On the other hand, a preset original image scene description is obtained, and text parsing and semantic generalization are performed following the principles of neutrality, simplicity, and white space (NSA) to construct... A neutral semantic whiteboard with programmable attribute slots is constructed. Then, an auditory-visual synesthetic translation mechanism is activated using a large language model to adaptively map high-purity music theory semantics into visual modification instructions, and accurately fill the corresponding empty slots in the neutral semantic whiteboard. Combined with the resolution of potential logical conflicts and artistic stylization correction, logically consistent enhanced fusion prompts are generated. Finally, the enhanced fusion prompts are encoded into high-dimensional semantic vectors and input into the visual generation model, which performs low-resolution spatial structure construction and high-resolution material texture filling in sequence, and finally outputs a high-fidelity image that is highly aligned with the emotion of the input audio.

[0070] The above description is only a preferred embodiment of the present invention. Modifications may be made within the scope defined by the claims of the present invention, but all such modifications shall fall within the protection scope of the present invention.

Claims

1. A method for generating music-assisted images based on music theory element extraction, characterized in that, The method includes the following steps: Step 1: Input the preprocessed audio data into an expert-guided micro-descriptor to explicitly deconstruct the audio signal into fine-grained structured music theory elements; Step 2: Introduce a cross-modal consistency verification mechanism to calculate and filter the similarity of structured music theory elements, and eliminate semantic bias and hallucination noise; Step 3: Build and provide standardized scene descriptions based on NSA principles as a neutral semantic whiteboard for cross-modal mapping; Step 4: Use a large language model to perform adaptive synesthetic fusion, interpret the extracted music theory elements into visual modification instructions and rewrite the neutral scene description to obtain enhanced fusion prompt words; Step 5: Input the enhanced fusion prompt into the visual generation model to complete the generation and rendering of the high-fidelity image from the text.

2. The music-assisted image generation method according to claim 1, characterized in that, Step 1 specifically includes: 1-1: Construct an expert-oriented analysis architecture, constrain the behavior patterns of the multimodal large language model through system-level instructions, and define its music analysis role; 1-2: Design a structured thinking chain guidance template to force the model to focus on and extract music theory elements in six dimensions: orchestration, style, tonality, time signature, tempo, and mood. 1-3: Constrain the output format of the model, normalize the natural language prose description into a machine-readable structured list and JSON object, and output the initial six-dimensional structured music theory description.

3. The method for generating music-assisted images based on music theory element extraction according to claim 2, characterized in that, The six dimensions of music theory elements follow the following synesthetic isomorphism mechanism in cross-modal mapping: Among them, the instrumentation dimension is used to map the texture and material of the visual image; the style dimension is used to define the overall artistic style and refine the brushstrokes; the tonality dimension is used to control the color temperature and light of the image; the time signature dimension is used to determine the spatial composition and geometric layout; the speed dimension is used to map the dynamic effect and visual tension of the image; and the emotion dimension, as the highest level of semantics, governs the overall atmosphere and tone of the image.

4. The method for generating music-assisted images based on music theory element extraction according to claim 1, characterized in that, Step 2 specifically includes: 2-1: Extract the global acoustic feature vector of the input audio using the audio encoder of a pre-trained contrastive language-audio model; 2-2: Using the text encoder of the contrastive language-audio model, the fine-grained music theory descriptions generated in step 1 are encoded into text feature vectors respectively; 2-3: Calculate the cosine similarity between the global acoustic feature vector and each text feature vector to quantify the degree of semantic consistency; 2-4: Set a confidence threshold, retain text descriptions with similarity scores higher than the threshold, remove noisy descriptions with scores lower than the threshold, and output high-purity structured music theory semantics.

5. The method for generating music-assisted images based on music theory element extraction according to claim 1, characterized in that, Step 3 specifically includes: 3-1: In accordance with the principle of neutrality, remove modifiers with subjective emotional coloring from the scene description and retain only the objective statements of physical entities; 3-2: Following the principle of simplicity, reduce visual distractions and adopt a minimalist descriptive strategy to establish a single visual focus; 3-3: In accordance with the principle of leaving blank space, semantic generalization processing is performed on the lighting conditions, weather conditions and object materials of the scene to inject reserved programmable semantic slots into music theory features.

6. Based on the NSA principles of neutrality, simplicity, and white space, standardized scene descriptions are constructed as neutral semantic whiteboards for cross-modal feature injection by stripping away emotional embellishments, simplifying visual focus, and performing attribute generalization processing. A method for generating music-assisted images based on music theory element extraction according to claim 1, characterized in that, Step 4 specifically includes: 4-1: Activate the synesthetic translation mechanism of the large language model, and map discrete music theory elements into visual modification instructions that include light and shadow, color tone, material details and atmosphere through logical reasoning; 4-2: By adopting the attribute slot filling strategy, visual modification instructions are seamlessly embedded into the neutral semantic whiteboard, so as to preserve the core physical structure of the original scene without tampering while rendering the musical atmosphere; 4-3: Activate the logical conflict resolution mechanism, process the potential semantic paradox between abstract musical emotions and concrete scene physical attributes through stylistic transfer, and output logically consistent enhanced fusion prompts.

Images