Music-driven image generation method and system based on cross-modal alignment and computer device
By directly generating images consistent with thematic semantics and emotions from music audio through cross-modal alignment networks and diffusion models, the problem of unstable music generation in existing technologies is solved, and high-precision music visualization is achieved.
Patent Information
- Application Number
- CN202511637290.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-02-06
AI Technical Summary
Existing music-to-image generation methods rely on intermediate text descriptions, which makes it difficult to fully express the continuity and fine-grained features of music. Furthermore, they are prone to introducing semantic noise during cross-modal conversion, resulting in unstable generation results in terms of thematic consistency and emotional expression. In particular, they are difficult to reflect timbre, rhythm, and cultural imagery in traditional music.
The music audio features are directly mapped to the pseudo-text sentiment feature space through a cross-modal alignment network, and then fused with the text title features. Finally, a diffusion model is used to generate an image consistent with the music theme and sentiment. The method employs audio embedding feature extraction, cross-modal alignment, feature fusion, and diffusion generation.
It achieves intuitive visualization and cultural expression of musical emotions, improves the emotional consistency and semantic expression accuracy between music and images, and ensures that the generated images are consistent with the emotional changes of the input music, thus significantly improving the semantic accuracy and artistic expressiveness of the generated results.
Smart Images
Figure CN121479012A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology in the arts, and more specifically, to a method, system, and computer device for generating music-driven images based on cross-modal alignment. Background Technology
[0002] Music, as one of the most expressive art forms, conveys emotional and semantic information over time through melody, rhythm, and timbre. Unlike visual arts such as painting and photography, which use space as a medium, music is an abstract auditory signal. Its emotional expression relies on auditory experience and subjective feelings, lacking a direct visual representation. When listening to music, listeners often form images in their minds that resonate with the melody or emotion, but this perception is highly individualized and difficult to share or objectively present.
[0003] In recent years, with the development of artificial intelligence, cross-modal generation technology has become a research hotspot. Scholars have attempted to establish a mapping between auditory and visual perception using deep learning models to achieve automatic music-to-image generation. However, most existing music-to-image generation methods rely on intermediate text descriptions as a bridge. That is, they first use language models or sentiment analysis modules to convert music into text or tags, and then use text-to-image generation models (such as Stable Diffusion) to generate visual images. This approach has two main problems: first, text is difficult to fully express the continuity and fine-grained features of music; second, semantic noise is easily introduced during cross-modal conversion, leading to instability in the generated results in terms of thematic consistency and emotional expression.
[0004] On the other hand, existing research on sound-to-image generation mainly focuses on natural sound sources, such as ocean waves, wind, or birdsong, emphasizing the physical matching of sound with the scene rather than the abstract emotions and cultural imagery of music. This makes it difficult to generate artistic images that reflect the musical atmosphere. Especially in traditional music (such as Chinese folk music), the timbre, rhythm, and playing style of different instruments contain rich cultural symbols and emotional layers, and existing models struggle to effectively model these subtle changes.
[0005] Therefore, how to establish a generation method that can directly extract multi-dimensional features from music audio and achieve dual alignment of song title semantics and emotion in visual space has become a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0006] In view of this, the present invention proposes a music-driven image generation method, system and computer device based on cross-modal alignment, aiming to solve the problems existing in the current technology.
[0007] This invention proposes a music-driven image generation method based on cross-modal alignment, comprising the following steps: S1. Obtain the music audio segment and its corresponding text title and sentiment classification; S2. Perform feature extraction and normalization on the music audio segment to obtain audio embedding features; S3. Map the audio embedding features to the pseudo-text embedding space, and obtain the audio pseudo-text sentiment features through a cross-modal alignment network; S4. The audio pseudo-text sentiment features and text title features are fused to obtain joint conditional features; S5. Input the joint conditional features into the diffusion model to generate an image that matches the musical theme and emotional features in the music audio segment.
[0008] Preferably, step S2 specifically includes: S21. Perform Mel spectrum transformation on the music audio segment to obtain a two-dimensional spectrum that represents the melody, rhythm and energy distribution in the time and frequency domain, i.e., Mel spectrum. S22. Input the Mel spectrogram into the pre-trained audio encoder AudioCLIP to extract multi-dimensional audio features including rhythm, pitch, timbre and energy; S23. Perform L2 normalization on the extracted multi-dimensional audio features to obtain standardized audio embedding features. The normalization formula is as follows: ; in, Let i be the i-th audio embedding feature vector. for The normalized vector.
[0009] Preferably, step S3 specifically includes: S31. Input the standardized audio embedding features into the cross-modal alignment network, map them to the pseudo-text sentiment embedding space that is compatible with the text title features, and output the audio pseudo-text sentiment features. S32, The cross-modal alignment network includes an audio mapping sub-network, a text encoding sub-network, a discriminator, and a reconstruction sub-network; S33. Jointly optimize and train the cross-modal alignment module. By jointly optimizing the weighted feature alignment loss, adversarial loss and reconstruction constraint loss, the distribution consistency of audio and text sentiment modalities in the latent semantic space is achieved, and audio pseudo-text sentiment features are obtained. S34. The joint loss function is defined as follows: ; in, For weighted feature alignment loss, To combat the losses, To reconstruct the constraint loss. Weighting coefficients; for Weighting coefficients; for Weighting coefficients.
[0010] Preferably, the formula for calculating the weighted feature alignment loss is: ; in, The features of the determined sentiment token obtained from sentiment classification, To correspond to the pseudo-emotional features mapped in the audio, For semantic weight coefficients, is the sample index; i is the token index; N is the total number of samples; L is the total number of tokens in each sample.
[0011] Preferably, step S4 specifically includes: S41. The audio pseudo-text sentiment features and text title features are weighted and fused at the token level. The tokens are then selected based on their importance. Tokens with high semantic contribution are retained, while irrelevant tokens are discarded. S42. Perform weighted averaging on the effective features of the edges; S43. The weighted edge features are concatenated with the retained important tokens to form the fused joint conditional features.
[0012] Preferably, the diffusion model in step S5 adopts a latent space diffusion structure, and its reverse denoising process satisfies: ; in, Let c be the latent variable at time step t, and c be the joint conditional feature. For noise prediction function, This is a Gaussian noise term. These are noise adjustment parameters. is the coefficient of standard deviation.
[0013] Preferably, the audio mapping sub-network in step S32 specifically includes: The audio embedding features are encoded to form a (16, 64) token sequence, and the text title features are represented by a (77, 768) token. The two are semantically aligned at the token level through a multi-head self-attention-based mapping network. The attention calculation formula is as follows: ; in, , , These are query, key, and value matrices, respectively. For feature dimensions.
[0014] Preferably, the joint optimization training process of the cross-modal alignment network in step S33 employs the Adam optimizer combined with a segmented learning rate decay strategy.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a music-driven image generation method based on cross-modal alignment, which achieves intuitive visualization and cultural expression of musical emotions through end-to-end audio feature extraction, semantic mapping, and diffusion generation mechanisms. This method aligns audio and text features in the latent semantic space, enabling the model to establish stable semantic relationships across different modalities, thereby effectively improving the emotional consistency and semantic expression accuracy between music and images. Audio embedding features provide dynamic information such as melody, rhythm, and timbre, while text features supplement musical themes and cultural imagery. The combination of these two features forms a complementary multimodal representation in the latent space, providing more emotionally and semantically constrained input for subsequent image generation.
[0016] This invention proposes a token-level semantic fusion strategy that evaluates the importance of each feature token in semantic expression, retaining high-weight features and suppressing low-relevance or redundant features, thereby achieving efficient fusion of multimodal information. Building upon this, a conditionally controlled latent space diffusion model is introduced, employing reverse denoising sampling to achieve emotion-driven image generation. This ensures that the generated image maintains consistency with the emotional changes of the input music in terms of color, composition, and atmosphere. This method establishes a stable correspondence between image content and musical emotion, significantly improving the semantic accuracy and artistic expressiveness of the generated results, and providing a new technical path for music visualization and digital art creation.
[0017] On the other hand, this application also provides a music-driven image generation system based on a cross-modal alignment-based music-driven image generation method, comprising: The audio feature extraction module is used to extract multi-dimensional audio features from music audio clips; The cross-modal alignment module is used to map multi-dimensional audio features to the pseudo-text embedding space and perform semantic alignment. The feature fusion module is used to fuse audio pseudo-text sentiment features with text title features; The image generation module is used to generate images that are semantically and emotionally consistent with music audio clips based on a diffusion model.
[0018] Furthermore, this application also provides a computer device, comprising: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor implements the music-driven image generation method based on cross-modal alignment.
[0019] It is understood that the music-driven image generation system and computer device based on cross-modal alignment provided in this application have the same beneficial effects as the music-driven image generation method based on cross-modal alignment described above, and will not be repeated here. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of a music-driven image generation method based on cross-modal alignment provided in an embodiment of the present invention; Figure 2 A structural block diagram of a music-driven image generation system based on cross-modal alignment provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-modal alignment network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the fusion of audio pseudo-text sentiment features and text title features provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of music input and image generation provided in an embodiment of the present invention. Detailed Implementation
[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey its scope to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0022] Example 1 See Figure 1 This embodiment provides a music-driven image generation method based on cross-modal alignment, including the following steps: S1. Obtain the music audio segment and its corresponding text title and sentiment classification.
[0023] Specifically, in step S1 of this embodiment, the input music audio segments are segmented, denoised, normalized, and resampled to ensure the time-frequency consistency of the signal, and paired with the corresponding text titles. Subsequently, each audio segment is labeled with sentiment based on the valence-arousal two-dimensional sentiment model, classifying music emotions into four categories: joy, anger, sadness, and peace. This constructs a multimodal training dataset containing audio signals, corresponding text titles, and sentiment labels.
[0024] S2. Perform feature extraction and normalization on the music audio segments to obtain audio embedding features.
[0025] Preferably, step S2 specifically includes: S21. Perform Mel spectrum transformation on the music audio clip to obtain a two-dimensional spectrum that represents the melody, rhythm and energy distribution in the time and frequency domain, namely the Mel spectrum.
[0026] Specifically, in step S21, the time-domain signal is mapped to the time-frequency domain space by performing Mel spectrum transformation on the input music audio segment to obtain a two-dimensional spectrum that can characterize the melody, rhythm and energy distribution, namely the Mel spectrum.
[0027] S22. Input the Mel spectrogram into the pre-trained audio encoder AudioCLIP to extract multi-dimensional audio features including rhythm, pitch, timbre, and energy.
[0028] Specifically, the Mel spectrogram is normalized and then input into the audio encoder AudioCLIP, which consists of a convolutional coding unit and a Transformer module. It is used to extract the acoustic feature vector sequence and obtain the semantic-level audio embedding through global average pooling. The output feature dimension is (16, 64) so as to align with the text title features.
[0029] Understandably, in step S22, the Mel spectrogram is input into the pre-trained audio encoder AudioCLIP, local time-frequency patterns are extracted through convolutional structures, and global semantic associations are captured by combining multi-head attention mechanisms, thereby obtaining audio embedding features that combine local details and overall semantics.
[0030] S23. Perform L2 normalization on the extracted multi-dimensional audio features to obtain standardized audio embedding features. The normalization formula is as follows: ; in, Let i be the i-th audio embedding feature vector. for The normalized vector.
[0031] Understandably, in step S23, L2 normalization is performed on the extracted multi-dimensional audio features to distribute the features at a uniform scale, reducing amplitude differences between samples and improving the consistency and stability of the feature space. The normalized audio embedding features can maintain a distributional match with the text title features in the subsequent cross-modal alignment stage, thereby ensuring the numerical stability and semantic consistency of the model during training and inference.
[0032] It is understood that in step S2 of this embodiment, the time-domain signal is converted into a time-frequency domain representation through Mel spectrum transformation, and then input into the pre-trained AudioCLIP encoder to extract multi-dimensional features such as rhythm, pitch, and timbre. After standardization processing, stable audio embedding features are obtained.
[0033] Combination Figure 3 As shown in Figure S3, the audio embedding features are mapped to the pseudo-text embedding space, and the audio pseudo-text sentiment features are obtained through a cross-modal alignment network.
[0034] Preferably, step S3 specifically includes: S31. Input the standardized audio embedding features into the cross-modal alignment network, map them to the pseudo-text sentiment embedding space that is compatible with the text title features, and output the audio pseudo-text sentiment features.
[0035] Specifically, step 31 uses a mapping network to realize the semantic correspondence between audio and text, so that the two modalities have unified feature representation in the same latent space.
[0036] S32. The cross-modal alignment network includes an audio mapping subnetwork, a text encoding subnetwork, a discriminator, and a reconstruction subnetwork.
[0037] Preferably, the audio mapping sub-network in step S32 specifically includes: The audio embedding features are encoded to form a (16, 64) token sequence, and the text title features are represented as a (77, 768) token. The two are semantically aligned at the token level through a multi-head self-attention-based mapping network. The attention calculation formula is as follows: ; in, , , These are query, key, and value matrices, respectively. For feature dimensions.
[0038] Specifically, the audio mapping subnetwork employs a multi-layer transformer structure to perform nonlinear feature transformation and high-dimensional projection on the input audio embedding features, generating audio pseudo-text sentiment features of the audio embedding features. The audio mapping subnetwork captures global dependencies through a self-attention mechanism and utilizes a multi-head attention mechanism to interact with information in different feature subspaces. The mapping output maintains the same dimension as the text title feature (e.g., 768 dimensions) to facilitate subsequent modality fusion.
[0039] Specifically, the text encoding subnetwork uses a pre-trained language model, such as the Clip model, to extract semantic features from the text title.
[0040] Specifically, the discriminator uses a multilayer perceptron structure to perform binary classification on the input features, which is used to distinguish the distribution differences between the sentiment features of audio pseudo-text and the title features of real text.
[0041] Specifically, the reconstruction subnetwork reverses the reconstruction of the audio pseudo-text sentiment features to recover the audio embedding features.
[0042] S33. Jointly optimize and train the cross-modal alignment module. By jointly optimizing the weighted feature alignment loss, adversarial loss and reconstruction constraint loss, the distribution consistency of audio and text sentiment modalities in the latent semantic space is achieved, and audio pseudo-text sentiment features are obtained. S34. The joint loss function is defined as follows: ; in, For weighted feature alignment loss, To combat the losses, To reconstruct the constraint loss. Weighting coefficients; for Weighting coefficients; for Weighting coefficients. The aforementioned... , , All are non-negative real numbers, used to balance the relative contributions of each loss term.
[0043] The preferred formula for calculating the weighted feature alignment loss is: ; in, The features of the determined sentiment token obtained from sentiment classification, To correspond to the pseudo-sentiment features mapped from audio, i.e., pseudo-textual sentiment features in audio, For semantic weight coefficients, is the sample index; i is the token index; N is the total number of samples; L is the total number of tokens in each sample.
[0044] Preferably, the joint optimization training process of the cross-modal alignment network in step S33 employs the Adam optimizer combined with a piecewise learning rate decay strategy.
[0045] Specifically, in step S33, the cross-modal alignment network is jointly optimized and trained to achieve consistency between the audio pseudo-text sentiment features and the text title features in the latent semantic space. This step, through the joint constraints of weighted feature alignment loss, adversarial loss, and reconstruction loss, enables the model to simultaneously possess semantic matching, distribution consistency, and structural fidelity capabilities during the alignment process, thereby obtaining stable audio pseudo-text sentiment features.
[0046] Understandably, in step S3, the standardized audio embedding features are input into the cross-modal alignment network. Through joint optimization of audio mapping, text encoding, and discrimination and reconstruction networks, the semantic alignment of audio and text features in the latent space is achieved, and pseudo-text sentiment embedding is obtained.
[0047] Combination Figure 4 As shown in Figure S4, the audio pseudo-text sentiment features and text title features are fused to obtain joint conditional features.
[0048] Preferably, step S4 specifically includes: S41. The audio pseudo-text sentiment features and text title features are weighted and fused at the token level. The tokens are then selected based on their importance. Tokens with high semantic contribution are retained, while irrelevant tokens are discarded. S42. Perform weighted averaging on the effective features of the edges; S43. The weighted edge features are concatenated with the retained important tokens to form the fused joint conditional features.
[0049] Understandably, in step S4, the audio pseudo-text sentiment features are fused with the text title features, key information is filtered based on token importance and weighted and concatenated to generate joint conditional features that combine semantic integrity and sentiment consistency.
[0050] S5. Input the joint conditional features into the diffusion model to generate an image that matches the musical theme and emotional features in the music audio clip.
[0051] Preferably, the diffusion model in step S5 adopts a latent space diffusion structure, and its reverse denoising process satisfies: ; in, Let c be the latent variable at time step t, and c be the joint conditional feature. For noise prediction function, This is a Gaussian noise term. These are noise adjustment parameters. is the coefficient of standard deviation.
[0052] Understandably, in step S5, the joint conditional features are input into the diffusion model, and an image consistent with the music theme, tone and emotional atmosphere is generated through reverse denoising sampling, thereby achieving high-quality music-to-image generation.
[0053] It is understandable that the music-driven image generation method provided in this embodiment solves the problems of insufficient semantic expression and unstable emotional mapping in existing technologies. This method, by introducing a cross-modal alignment mechanism, achieves high-precision fusion of audio and text features in the latent space, enabling the direct generation of images consistent with thematic semantics and emotional atmosphere from music without the need for intermediate text description. The method includes four core stages: audio feature extraction, pseudo-text mapping, feature fusion, and diffusion generation, which can fully capture the melody, rhythm, and emotional changes of music, achieving a unified mapping of semantics and vision. Figure 5 As shown in the experimental results, the images generated by the present invention are significantly better than those generated by existing comparison methods in terms of semantic consistency and emotional expressiveness.
[0054] It is understood that the music-driven image generation method of this invention not only improves the accuracy and stability of cross-modal feature learning, but also expands the application scope of music visualization. This technology can be widely applied in fields such as digital art creation, intelligent music visualization, virtual reality, augmented reality, and cultural dissemination, and is particularly suitable for the artistic expression and cultural display of traditional Chinese music. Furthermore, the cross-modal alignment and fusion mechanism proposed in this invention can also be extended to other multimodal generation tasks, providing a new technical path for intelligent perception and generation.
[0055] Example 2 Combination Figure 2 As shown, this embodiment provides a music-driven image generation system based on the method in Embodiment 1, including: The audio feature extraction module is used to extract multi-dimensional audio features from music audio clips; The cross-modal alignment module is used to map multi-dimensional audio features to the pseudo-text embedding space and perform semantic alignment. The feature fusion module is used to fuse audio pseudo-text sentiment features with text title features; The image generation module is used to generate images that are semantically and emotionally consistent with music audio clips based on a diffusion model.
[0056] It is understood that the music-driven image generation system based on cross-modal alignment provided in this embodiment has the same beneficial effects as the music-driven image generation method based on cross-modal alignment in Embodiment 1 above, and will not be repeated here.
[0057] Example 3 This embodiment provides a computer device, including: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor implements the method in Embodiment 1.
[0058] It is understood that the computer device provided in this embodiment has the same beneficial effects as the music-driven image generation method based on cross-modal alignment in Embodiment 1 above, and will not be described again here.
[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0060] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A music-driven image generation method based on cross-modal alignment, characterized in that, Includes the following steps: S1. Obtain the music audio segment and its corresponding text title and sentiment classification; S2. Perform feature extraction and normalization on the music audio segment to obtain audio embedding features; S3. Map the audio embedding features to the pseudo-text embedding space, and obtain the audio pseudo-text sentiment features through a cross-modal alignment network; S4. The audio pseudo-text sentiment features and text title features are fused to obtain joint conditional features; S5. Input the joint conditional features into the diffusion model to generate an image that matches the musical theme and emotional features in the music audio segment.
2. The music-driven image generation method based on cross-modal alignment according to claim 1, characterized in that, Step S2 specifically includes: S21. Perform Mel spectrum transformation on the music audio segment to obtain a two-dimensional spectrum that represents the melody, rhythm and energy distribution in the time and frequency domain, i.e., Mel spectrum. S22. Input the Mel spectrogram into the pre-trained audio encoder AudioCLIP to extract multi-dimensional audio features including rhythm, pitch, timbre and energy; S23. Perform L2 normalization on the extracted multi-dimensional audio features to obtain standardized audio embedding features. The normalization formula is as follows: ； in, Let i be the i-th audio embedding feature vector. for The normalized vector.
3. The music-driven image generation method based on cross-modal alignment according to claim 1, characterized in that, Step S3 specifically includes: S31. Input the standardized audio embedding features into the cross-modal alignment network, map them to the pseudo-text sentiment embedding space that is compatible with the text title features, and output the audio pseudo-text sentiment features. S32, The cross-modal alignment network includes an audio mapping sub-network, a text encoding sub-network, a discriminator, and a reconstruction sub-network; S33. Jointly optimize and train the cross-modal alignment module. By jointly optimizing the weighted feature alignment loss, adversarial loss and reconstruction constraint loss, the distribution consistency of audio and text sentiment modalities in the latent semantic space is achieved, and audio pseudo-text sentiment features are obtained. S34. The joint loss function is defined as follows: ； in, For weighted feature alignment loss, To combat the losses, To reconstruct the constraint loss. Weighting coefficients; for Weighting coefficients; for Weighting coefficients.
4. The music-driven image generation method based on cross-modal alignment according to claim 3, characterized in that, The formula for calculating the weighted feature alignment loss is as follows: ； in, The features of the determined sentiment token obtained from sentiment classification, To correspond to the pseudo-emotional features mapped in the audio, These are semantic weight coefficients; is the sample index; i is the token index; N is the total number of samples; L is the total number of tokens in each sample.
5. The music-driven image generation method based on cross-modal alignment according to claim 1, characterized in that, Step S4 specifically includes: S41. The audio pseudo-text sentiment features and text title features are weighted and fused at the token level. The tokens are then selected based on their importance. Tokens with high semantic contribution are retained, while irrelevant tokens are discarded. S42. Perform weighted averaging on the effective features of the edges; S43. The weighted edge features are concatenated with the retained important tokens to form the fused joint conditional features.
6. The music-driven image generation method based on cross-modal alignment according to claim 1, characterized in that, The diffusion model in step S5 adopts a latent space diffusion structure, and its reverse denoising process satisfies: ； in, Let c be the latent variable at time step t, and c be the joint conditional feature. For noise prediction function, This is a Gaussian noise term. These are noise adjustment parameters. is the coefficient of standard deviation.
7. The music-driven image generation method based on cross-modal alignment according to claim 3, characterized in that, The audio mapping sub-network in step S32 specifically includes: The audio embedding features are encoded to form a (16, 64) token sequence, and the text title features are represented by a (77, 768) token. The two are semantically aligned at the token level through a multi-head self-attention-based mapping network. The attention calculation formula is as follows: ； in, , , These are query, key, and value matrices, respectively. For feature dimensions.
8. The music-driven image generation method based on cross-modal alignment according to claim 1, characterized in that, The joint optimization training process of the cross-modal alignment network in step S33 uses the Adam optimizer combined with a piecewise learning rate decay strategy.
9. A music-driven image generation system based on the music-driven image generation method based on cross-modal alignment as described in any one of claims 1 to 8, characterized in that, include: The audio feature extraction module is used to extract multi-dimensional audio features from music audio clips; The cross-modal alignment module is used to map multi-dimensional audio features to the pseudo-text embedding space and perform semantic alignment. The feature fusion module is used to fuse audio pseudo-text sentiment features with text title features; The image generation module is used to generate images that are semantically and emotionally consistent with music audio clips based on a diffusion model.
10. A computer device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the program is executed by the processor, the processor implements the music-driven image generation method based on cross-modal alignment as described in any one of claims 1 to 8.