Multi-view semantic alignment Chinese painting emotion perception audio generation method

Abstract

The invention belongs to the technical field of cross-modal emotion calculation and audio generation, and particularly discloses a Chinese painting emotion perception audio generation method based on multi-view semantic alignment, and the method comprises the steps: generating an audio cue word through a large voice model; visual semantic feature vectors of image information are extracted through a CLIP model, fine-grained emotion feature vectors are obtained through an EmoColor model, spatial mapping of visual semantics and emotion attributes is established through a multi-view semantic mapping matrix, multi-modal fusion is carried out on the visual semantic feature vectors and the fine-grained emotion feature vectors through a cross-modal fusion module, and the visual semantic feature vectors and the fine-grained emotion feature vectors are fused. Consistent mapping of visual elements and emotional semantics is realized; a progressive audio generation network is adopted, a rough prior and refined prior double-stage structure is adopted, and high-fidelity audio with consistent emotion is generated through a decoder; through cooperation of multi-view semantic alignment and progressive prior generation, the generated audio is more natural and harmonious in the aspect of emotional expression and content correspondence.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and cross-modal generation technology, specifically involving a method for generating emotional audio of Chinese paintings with multi-perspective semantic alignment. This method achieves automatic generation of emotional audio of traditional Chinese landscape paintings by integrating visual semantics and emotional features, and can be applied to digital art creation, digital display of cultural heritage, and immersive multimedia systems. Background Technology

[0002] With the rapid development of deep learning and multimodal generation technologies, cross-modal generation from vision to hearing has become an important direction in artificial intelligence research. Existing image-to-audio generation models mainly focus on natural scene images or general datasets, such as AudioLDM and SpecVQGAN. These methods can generate corresponding sound effects from images, but they have significant shortcomings in terms of emotional consistency and artistic context adaptation.

[0003] Chinese landscape painting, as a traditional art form carrying profound cultural connotations, differs from ordinary images in its mode of expression. Landscape paintings often use techniques such as "combining emptiness and fullness" and "leaving blank space and applying ink" to convey poetic imagery and emotional tension. Therefore, it is difficult to express the aesthetic connotations and psychological emotions of the painting by generating sound solely through visual features.

[0004] However, existing technologies have the following main problems: (1) Lack of emotional expression: Traditional image-to-audio models often generate audio based only on pixel or semantic feature mapping, lacking quantitative modeling of the emotion and artistic conception of the picture, resulting in the generated sound not matching the emotion of the image; (2) Semantic-audio mismatch problem: During the generation process, the semantic elements of the image (such as mountains, clouds, and water flow) fail to establish a corresponding relationship with the rhythm and timbre of the audio, causing a separation between "pictorial meaning" and "sound sense"; (3) Lack of cultural context adaptability: Existing models are mostly trained on Western art or modern photography datasets, lacking understanding of the composition rules and aesthetic semantics of Chinese landscape painting, and the generated results often have problems such as "emotional misalignment" or "abrupt timbre".

[0005] Therefore, it is necessary to design a multi-perspective semantic alignment method for generating emotionally perceptive audio of Chinese landscape paintings that can solve the above problems. Summary of the Invention

[0006] To overcome the aforementioned shortcomings in the prior art, this invention provides a method for generating Chinese painting emotion-aware audio with multi-perspective semantic alignment, comprising the following steps: Step 1: Obtain Chinese painting images and corresponding artistic description texts; use the visual language model Qwen2.5-VL to semantically optimize the descriptive texts and generate audio prompts; construct a multimodal dataset containing "image-text-emotion"; Step 2: Construct a semantic feature extraction module, which includes a CLIP model. Image information from the multimodal dataset is input into the CLIP model to extract visual semantic feature vectors from the image information. ; Step 3: Construct an emotion feature extraction module, which includes an EmoColor model based on the Transformer architecture. Image information from the multimodal dataset is input into the EmoColor model to extract fine-grained emotion feature vectors. ; Step 4: Convert the visual semantic feature vector With fine-grained sentiment feature vectors Projected onto a unified embedding space, a spatial mapping between visual semantics and emotional attributes is established through a multi-view semantic mapping matrix, generating an object-emotion mapping relationship. The visual semantic feature vector and fine-grained emotional feature vector are then fused in a multimodal manner to generate emotion-enhanced contextual fusion features. ; Step 5: Construct an audio generation model, which includes a Lite-KAN-based progressive audio generation network that integrates the fused features. The input progressive audio generation network includes a coarse prior stage and a refined prior stage, and the fused features... After being processed sequentially through a coarse prior stage and a refined prior stage, high-fidelity audio is output. ; Step 6: During the audio generation process, a joint loss function is used to optimize the audio generation to obtain emotion-aware audio.

[0007] Furthermore, the expression for extracting the visual semantic feature vector in step 2 is: (1); in, This represents the input landscape painting image; These are the sets of semantic model parameters. The CLIP model's visual-language feature extraction function is a multimodal mapping function obtained from contrastive language-image pre-training. The CLIP model optimizes the embedding space of the visual encoder and text encoder through cross-modal contrastive loss, so that visual features and semantic descriptions have a high degree of correlation in the same semantic space.

[0008] Furthermore, in step 3, the expression for extracting the fine-grained sentiment feature vector is: (2); The set of parameters representing the sentiment model; This represents the color sentiment feature extraction function of the EmoColor model, where the EmoColor model obtains high-resolution sentiment representations by supervised learning of the Transformer through sentiment-labeled data. This represents the primary image features extracted by the convolutional network; This represents a multi-head self-attention module based on the Transformer architecture, used to model the sentiment relevance between different regions within an image; The weight matrix is ​​a learnable matrix; Nonlinear activation function; This is a global average pooling operation used to generate the final fine-grained sentiment vector. .

[0009] Furthermore, in step 4, while establishing the spatial mapping between visual semantics and emotional attributes, the multi-view semantic mapping matrix is ​​also used to align cross-modal features. The alignment process is driven by contrastive learning, and its expression is: (3); in, and These represent the linear mapping matrix of the visual semantic feature vector and the linear mapping matrix of the fine-grained sentiment feature vector, respectively. and The projected uniform embedding vector; These are the positive sample feature vectors of the semantic modality; This represents the correct sample feature vector corresponding to the sentiment modality. The negative sample feature vector of the semantic modality; is the interval coefficient between positive and negative samples; N is the number of samples; + indicates taking the non-negative part; This represents the semantic-sentiment alignment loss; By minimizing Achieve multi-perspective coupling between semantic space and emotional space.

[0010] Furthermore, in step 4, the visual semantic feature vector is... With fine-grained sentiment feature vectors The input is processed by a cross-modal fusion module, which uses a cross-modal attention mechanism to fuse multi-channel information and generate emotion-enhanced contextual fusion features. ,include: For the visual semantic feature vector With the fine-grained emotion feature vector Perform linear mappings to obtain the query Q, key K, and value V matrices; Calculate the similarity matrix between the query Q and the key K, and obtain the attention distribution weights using the softmax function. ; The attention-weighted result is concatenated with the visual semantic feature vector residual, and then deep feature fusion is performed through convolution to obtain the context fusion feature. .

[0011] Furthermore, in step 5, the method for audio generation specifically includes: First, in the rough prior stage, the fused features are... The input is a Transformer encoder, which models long-range dependencies through a multi-layer self-attention mechanism to generate low-resolution audio features. The features described can be used to describe the overall acoustic emotion and acoustic layout; Secondly, in the refinement of the prior stage, the low-resolution audio features output from the coarse prior stage are used. As input, a progressive audio generation network is used to reconstruct high-resolution details from low-resolution audio features, generating high-fidelity audio. ; The high-fidelity audio Map back to the time-domain audio signal to generate audio. Complete audio generation.

[0012] Furthermore, the joint loss function in step 6 As shown below: (12); in, This represents the audio reconstruction loss, used to maintain waveform fidelity; This represents semantic-sentiment alignment loss, ensuring cross-modal consistency. This indicates a loss of emotional consistency, constraining the emotional resonance between the generated audio and the input image; These are the weighting coefficients; During audio generation, a reconstruction loss function is used to ensure the consistency of sound quality and emotion in the output signal. The expression for the reconstruction loss function is as follows: (13); in, The audio is generated by the model; It is a genuine reference audio file; The average of the first k largest squared errors is taken to improve the model's sensitivity to abnormal frequency bands; These are the weighting coefficients for each loss; the reconstruction loss function is used to ensure the consistency of the output signal in terms of sound quality and smoothness, effectively reducing artifact noise; The generated audio is constrained by cosine similarity between its sentiment embedding space and the sentiment embedding space of the input image. Sentiment consistency loss is employed to maintain directional consistency between the output audio and the input image in the high-dimensional sentiment space, ensuring the synergy of cross-modal sentiment perception. The expression is as follows: (14); in, This represents the embedding vector of fine-grained emotional features in audio. This represents the embedding vector of fine-grained emotional features of an image.

[0013] Beneficial effects: This invention proposes a multi-view semantic alignment method for generating emotionally perceptive audio for Chinese paintings. The method projects the visual semantic feature vectors and fine-grained emotional feature vectors of the image information into a unified embedding space. Through a multi-view semantic mapping matrix, a spatial mapping between visual semantics and emotional attributes is established, generating an object-emotion mapping relationship. This ensures a high degree of consistency between the semantic content and emotional tone of the generated audio, enabling the audio to accurately reflect the artistic atmosphere and spiritual realm of the painting. By introducing a dual-branch fusion structure of an emotional feature extraction module based on the EmoColor model and a semantic encoding module based on the CLIP model, both objective semantic and subjective emotional information of the painting are captured. Emotion-driven audio generation is achieved through a Lite-KAN progressive generative network, allowing the sound to naturally correspond to the emotions of the painting in terms of rhythm, timbre, and dynamic changes, enhancing the artistry and immersiveness of the auditory performance. The progressive generative structure gradually constructs the audio hierarchy in two stages: coarse prior and refined prior, coordinating and optimizing the global acoustic layout and local details. The generated audio significantly outperforms traditional methods in terms of fidelity, continuity, and dynamic performance. The design of the joint loss function comprehensively constrains reconstruction error, semantic alignment error, and sentiment consistency, thereby achieving a dynamic balance between semantic fidelity and emotional expression, effectively improving the stability and convergence performance of the training process. The overall generated results possess both high-quality technical output and embody the artistic characteristics of Chinese landscape painting—"vibrant spirit and expressive form"—creating a harmonious unity between auditory output and visual imagery, showcasing a profound Eastern aesthetic. This invention not only enhances the emotional expression capabilities of cross-modal generation but also provides a new technical path for the digital preservation and innovative dissemination of traditional art, possessing significant research value and broad application prospects. Attached Figure Description

[0014] Figure 1 This is a flowchart of a method for generating audio representations of Chinese paintings with multi-perspective semantic alignment, according to the present invention. Figure 2This is a schematic diagram of the EmoSpace2Wav model, a method for generating Chinese painting emotion perception audio with multi-perspective semantic alignment according to the present invention. Figure 3 This is a comparison of the audio generated by this invention with 2-second, 3-second, and 4-second audio samples. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1-3 This invention provides a method for generating emotionally perceptive audio of landscape paintings based on multi-perspective semantic alignment, comprising the following steps: Step 1: Obtain Chinese painting images and corresponding artistic description texts. Use the visual language model Qwen2.5-VL to semantically optimize the descriptive texts and generate audio prompts. At the same time, perform size normalization, color space standardization, and brightness balance processing on the Chinese painting images to ensure data quality and input consistency, and construct a multimodal dataset containing "image-text-emotion". Step 2: Construct a semantic feature extraction module, which includes a Contrastive Language-Image Pre-Training (CLIP) model. Image information from the multimodal dataset is input into the CLIP model to extract visual semantic feature vectors from the image information. ; Step 3: Construct an emotion feature extraction module, which includes an EmoColor (Emotion Color-based Recognition Model) based on the Transformer architecture. Image information from the multimodal dataset is input into the EmoColor model to extract fine-grained emotion feature vectors. ; Step 4: Convert the visual semantic feature vector With fine-grained sentiment feature vectors Projected onto a unified embedding space, a spatial mapping between visual semantics and emotional attributes is established through a multi-view semantic mapping matrix, generating an object-emotion mapping relationship. The visual semantic feature vector and fine-grained emotional feature vector are then fused in a multimodal manner to generate emotion-enhanced contextual features. ; Step 5: Construct an audio generation model, which includes a Lite-KAN-based progressive audio generation network that integrates the fused features. The input progressive audio generation network includes a coarse prior stage and a refined prior stage, wherein the fused features After being processed sequentially through a coarse prior stage and a refined prior stage, high-fidelity audio is output. ; Step 6: During the audio generation process, a joint loss function is used to optimize the audio generation to obtain emotion-aware audio.

[0017] Furthermore, the expression for extracting the visual semantic feature vector in step 2 is: (1); in, This represents the input landscape painting image; These are the sets of semantic model parameters. The CLIP model's visual-language feature extraction function is a multimodal mapping function obtained from contrastive language-image pre-training. The CLIP model optimizes the embedding space of the visual encoder and text encoder through cross-modal contrastive loss, so that visual features and semantic descriptions have a high degree of correlation in the same semantic space.

[0018] Furthermore, in step 3, the expression for extracting the fine-grained sentiment feature vector is: (2); in, The set of parameters representing the sentiment model; This represents the color sentiment feature extraction function of the EmoColor model, where the EmoColor model obtains high-resolution sentiment representations by supervised learning of the Transformer through sentiment-labeled data. This represents the primary image features extracted by the convolutional network; This represents a multi-head self-attention module based on the Transformer architecture, used to model the sentiment relevance between different regions within an image; The weight matrix is ​​a learnable matrix; Nonlinear activation function; This is a global average pooling operation used to generate the final fine-grained sentiment vector. .

[0019] Furthermore, in step 4, while establishing the spatial mapping between visual semantics and emotional attributes, the multi-view semantic mapping matrix is ​​also used to align cross-modal features. The alignment process is driven by contrastive learning, and its expression is: (3); in, and These represent the linear mapping matrix of the visual semantic feature vector and the linear mapping matrix of the fine-grained sentiment feature vector, respectively. and The projected uniform embedding vector; These are the positive sample feature vectors of the semantic modality; This represents the correct sample feature vector corresponding to the sentiment modality. The negative sample feature vector of the semantic modality; is the interval coefficient between positive and negative samples; N is the number of samples; + indicates taking the non-negative part; Represents semantic-sentiment alignment loss; where and Generated by a pre-trained cross-modal projection network, whose parameters are obtained through end-to-end training on a semantic-sentiment alignment dataset. During training, the semantic embedding and sentiment embedding are constrained by the feature similarity between them, and the semantic-sentiment alignment loss function is minimized. Continuously updated and The weight parameters are chosen to ensure an optimal correspondence between visual semantic feature vectors and fine-grained sentiment feature vectors within a unified embedding space. In implementation... and Using linear transformation matrix form: , ,in , These are the learnable weight matrices. , This is the bias term. The mapping matrix obtained after training can capture the high-dimensional semantic correspondence between visual semantic feature vectors and fine-grained sentiment feature vectors, achieving adaptive alignment of feature spaces of different modalities.

[0020] By minimizing Achieve multi-perspective coupling between semantic space and emotional space.

[0021] Furthermore, in step 4, the visual semantic feature vector is... With fine-grained sentiment feature vectors The input is processed by a cross-modal fusion module, which uses a cross-modal attention mechanism to fuse multi-channel information and generate emotion-enhanced contextual fusion features. ,include: First, the visual semantic feature vector With the fine-grained emotion feature vector Performing linear mappings separately yields the query Q, key K, and value V matrix, expressed as: (4); Next, the similarity matrix between the query Q and the key K is calculated, and the attention distribution weights are obtained using the softmax function. The expression is: (5); in, The scaling constant for the dimensions of the key vector; This represents the similarity matrix between the query and the key. Finally, the attention-weighted result is concatenated with the semantic feature residuals, and then... Convolutional processing performs deep feature fusion to obtain the context-fused features. The expression is: (6); Furthermore, in step 5, the method for audio generation specifically includes: First, in the rough prior stage, the fused features are... The input is a Transformer encoder, which models long-range dependencies through a multi-layer self-attention mechanism to generate low-resolution audio features. The features described can be used to describe the overall acoustic emotion and acoustic layout, and this process can be formalized as follows: (7); Secondly, in the refinement of the prior stage, the low-resolution audio features output from the coarse prior stage are used. As input, a progressive audio generation network is used to reconstruct high-resolution details from low-resolution audio features, generating high-fidelity audio. The generation process is represented as follows: (8); in Represents the Lite-KAN nonlinear mapping function; spline cardinality; Linear combination weights are used to integrate the outputs of sub-functions; This represents the dimension of the j-th input feature; The weight matrix that maps input dimensions to basis functions; This represents the transpose of the weight vector; Nonlinear activation function; Indicates the input dimension. p-th order B-spline basis function; B-spline basis functions are defined recursively: (9); (10); in, Represents a spline node vector, which can be learned or adapted; The k-th, p-th order B-spline basis function; p represents the spline order; k represents the index of the spline basis function; Node vectors; x input features; The Lite-KAN network is based on the Kolmogorov-Arnold representation theorem, and its function mapping form is as follows: (11); In the formula, Objective function with multidimensional input; The B-spline basis functions are learnable and control the smoothness of the function. These are linear combination weights used to integrate the outputs of each sub-function. This indicates the decomposition stage, where q represents the index of the q-th sub-mapping component; The high-fidelity audio Map back to the time-domain audio signal to generate audio. Complete audio generation.

[0022] Furthermore, the joint loss function in step 6 As shown below: (12); in, This represents the audio reconstruction loss, used to maintain waveform fidelity; This represents semantic-sentiment alignment loss, ensuring cross-modal consistency. This indicates a loss of emotional consistency, constraining the emotional resonance between the generated audio and the input image; These are the weighting coefficients; During audio generation, a reconstruction loss function is used to ensure the consistency of sound quality and emotion in the output signal. The expression for the reconstruction loss function is as follows: (13); in, The audio is generated by the model; It is a genuine reference audio file; The average of the first k largest squared errors is taken to improve the model's sensitivity to abnormal frequency bands; These are the weighting coefficients for each loss; the reconstruction loss function is used to ensure the consistency of the output signal in terms of sound quality and smoothness, effectively reducing artifact noise; The generated audio is constrained by cosine similarity between its sentiment embedding space and the sentiment embedding space of the input image. Sentiment consistency loss is employed to maintain directional consistency between the output audio and the input image in the high-dimensional sentiment space, ensuring the synergy of cross-modal sentiment perception. The expression is as follows: (14); in, This represents the embedding vector of fine-grained emotional features in audio. This represents the embedding vector of fine-grained emotional features of an image.

[0023] Performance Testing: To verify the effectiveness and superiority of the proposed method for generating emotionally perceptive audio of Chinese landscape paintings based on multi-view semantic alignment, a systematic experiment and comparative evaluation were conducted on the constructed Emotion Space-to-Waveform Audio Generation Model (EmoSpace2Wav). The experimental portion included quantitative index testing, visualization comparison, module ablation experiments, and cross-dataset generalization performance analysis, combined with multi-dimensional subjective and objective evaluations for comprehensive verification. The experimental results fully demonstrate that the present invention exhibits significant advantages in emotional consistency, audio generation quality, and cross-modal semantic alignment.

[0024] First, quantitative evaluation: This invention was systematically tested on the self-constructed multimodal landscape painting dataset EmoACLP. This dataset contains three modalities: image, text, and audio, covering rich emotional tags and artistic scene information, and is suitable for research on visual-to-audio emotion generation. To verify the model performance, three mainstream international image-to-audio generation models were selected as comparison objects: AudioLDM2 (a multimodal audio generation method based on a diffusion model), AudioLCM (a fast generation network based on latent space consistency), and Tango2 (an audio generation model optimized based on human preferences). The evaluation metrics used in this verification included: (1) Fréchet AudioDistance (FAD), which measures the difference between the generated audio and the real sample distribution; (2) Fréchet InceptionDistance (FID), which is used to evaluate the semantic consistency and realism of the generated results; (3) InceptionScore (IS), which reflects the diversity and clarity of the generated audio; and (4) Kullback–Leibler Divergence (KL), which quantifies the consistency between the model output and the real distribution. The experimental results are shown in [link to experimental results]. Figure 3 The summary is shown in Table 1: Table 1: Comparison of Optimal Methods

[0025] Our method achieved state-of-the-art performance across all metrics. Specifically, its FID score was 20.680, a 1.5% improvement over the second-best method. For KL, our model scored 2.451, a 1.8% improvement over the runner-up. These results demonstrate that incorporating fine-grained sentiment features and an "object-sentiment" mapping mechanism enables the model to better capture the consistency between visual content and sentiment semantics.

[0026] Second, qualitative analysis: In order to further understand the differences in behavior and semantic consistency of different methods, we analyzed the visualization results of the generated audio. Figure 2 The results show a comparison of metrics for audio generated at durations of 2 seconds, 3 seconds, and 4 seconds. We observe that AudioLCM and Tango2 exhibit significant performance fluctuations in complex emotional scenes, indicating that they are not stable enough in visual-audio emotional semantic modeling.

[0027] In contrast, our method demonstrates a more consistent advantage across all emotion types, with significant improvements in both abstract and subtle emotional expressions. This is attributed to the effective guidance of the "object-emotion" mapping mechanism, which enables the model to identify key objects in an image.

[0028] In summary, both quantitative and qualitative analyses demonstrate that our framework not only performs well in terms of numerical performance, but also exhibits greater controllability and stability in terms of semantic alignment and sentiment consistency.

[0029] Third, ablation experiments: To analyze the impact of each module on overall performance, this invention designed ablation experiments for the system, removing and comparing key components one by one. These include: a spatial alignment module (Align): responsible for the unified embedding of semantic and sentiment spaces; a cross-modal fusion module (Fusion): performing cross-modal attention fusion of visual and sentiment features; a Lite-KAN structure: enhancing the model's nonlinear expression and high-frequency feature modeling; and a hybrid loss term (Lmix): balancing global smoothness and local detail fidelity. The results are shown in Table 2. Table 2 Ablation test results for each module

[0030] As shown in Table 2, the ablation experiments further verified the effectiveness of each component.

[0031] Component effectiveness. Removing the spatial alignment module resulted in a 5.3% decrease in FID, confirming its crucial role in aligning visual and emotional semantic features. Disabling the feature fusion mechanism increased KL by 0.43, demonstrating that our fusion strategy effectively enhances multimodal interaction. Furthermore, removing Lite-KAN decreased IS by 0.063, validating its contribution to generative stability and semantic consistency.

[0032] The effectiveness of hyperparameters. We further analyzed the impact of key hyperparameters. The results show that when =0, =1, The best performance can be obtained when the value is 0.02, which shows that balancing the loss term is crucial for high-quality generation.

[0033] Effectiveness of loss functions. We compared different loss functions. Only using... The loss ensures overall waveform alignment but lacks high-frequency detail. Only using... The loss is sensitive to outliers. In contrast, our... The strategy achieved the best results on FID, improving by at least 5.3 points compared to any single loss, which confirms its advantage in balancing smoothness and local fidelity.

[0034] Fourth, generalization performance testing: To verify the generalization ability of the method of this invention, extended tests were conducted on different types of visual datasets, including the ImageNet natural image dataset, animation image dataset, Chinese landscape painting dataset, and natural landscape dataset.

[0035] Test results show that the FID is below 23 on all datasets, the IS is above 8.2, and the KL is stable within 2.5. Under different styles of data input, the model can still maintain the consistency of semantic-emotion mapping. In non-artistic images (such as ImageNet natural scenes), the generated audio still maintains the rhythm and emotion balance, indicating that the present invention has strong robustness and cross-domain adaptability.

[0036] This demonstrates that the present invention is not only applicable to landscape painting scenes, but can also generate audio results consistent with emotions under multi-source visual inputs such as animation and natural landscapes, reflecting strong generalization and high transferability.

[0037] Fifth, results analysis and summary: The following conclusions can be drawn from the combined results of quantitative and qualitative analysis: (1) The multi-perspective semantic-emotion alignment mechanism significantly improves cross-modal consistency. By constructing an "object-emotion" mapping, this invention achieves high-dimensional fusion of visual semantic features and emotional features, making the generated audio highly consistent with the original image in both semantic expression and emotional flow.

[0038] (2) The Lite-KAN progressive generation structure enhances the generation hierarchy and detail. The two-stage generation mechanism (coarse prior and refined prior) ensures the integrity of the rhythmic structure and the fidelity of acoustic details, making the generated audio more artistically expressive in terms of rhythm, melody and spatial depth.

[0039] (3) Experimental results demonstrate the comprehensive superiority of the method. EmoSpace2Wav achieves the best level in the industry in all indicators such as FID, IS, KL and FAD; in terms of visualization and subjective perception, the generated audio has consistent emotion, natural semantics, rich sound field and strong artistry.

[0040] In summary, the multi-perspective semantic alignment and emotion-aware audio generation method proposed in this invention can achieve cross-modal art mapping that "generates sound from image and guides sound with emotion." Its generation results significantly outperform existing technologies in terms of structural rationality, emotional authenticity, and artistic expressiveness, providing a novel technical path and theoretical support for the digital creation and intelligent dissemination of traditional Chinese art.

[0041] It should be noted that this invention establishes a spatial mapping relationship between visual semantics and emotional attributes, solving the problems of missing emotional expression and semantic-audio mismatch, and achieving a high degree of unity between audio-visual emotional semantics. This spatial mapping relationship is based on bi-branch feature extraction. The CLIP model accurately captures core semantic objects in landscape paintings (such as mountains, flowing water, clouds, and lone boats), generating structured semantic feature vectors to ensure that audio generation accurately corresponds to the objective content of the image. Simultaneously, leveraging the Transformer architecture of the EmoColor model, it mines fine-grained emotions (such as tranquility, grandeur, remoteness, and desolation) conveyed by elements like color, composition, and white space in the image, generating richly layered emotional feature vectors and emotional labels, thus overcoming the coarseness of traditional models in emotional modeling. This invention projects semantic and emotional features onto a unified embedding space using a contrastive learning-driven linear mapping matrix, constructing a spatial mapping relationship between visual semantics and emotional attributes: for example, mountains correspond to a steady and profound audio texture, flowing water to a soothing and continuous rhythmic change, and clouds to an ethereal and ethereal timbre expression, ensuring that each semantic object can find a matching emotional acoustic mapping. This alignment is not a simple feature splicing, but rather achieves multi-perspective coupling between semantic space and emotional space by minimizing semantic-emotional alignment loss. This ensures that the generated audio accurately conveys the "emotion in the picture" (emotional tone) while describing the "scene in the picture" (semantic content), perfectly presenting the artistic effect of "generating sound from picture and guiding sound with emotion", and completely solving the technical pain points of misalignment between sound and picture and semantic mismatch. By employing a Lite-KAN-based progressive audio generation network, the problem of coarse audio quality is solved, achieving high-fidelity and highly expressive audio output. This network, through a two-stage generation logic and a non-linear enhancement structure, achieves a significant improvement in audio quality. In the generation process, the network first generates low-resolution audio based on contextual fusion features (semantics + emotion), focusing on building the overall acoustic emotional atmosphere and acoustic layout—for example, constructing a tranquil and solitary basic acoustic environment for landscape paintings like "Fishing Alone on a Cold River," and establishing a majestic and expansive acoustic tone for works like "A Thousand Miles of Rivers and Mountains," ensuring that the overall direction of the audio is consistent with the artistic conception of the scene. Subsequently, in the prior refinement stage, the strong non-linear expressive power of the Lite-KAN structure is used for detailed reconstruction. Lite-KAN, based on the Kolmogorov-Arnold representation theorem, accurately models acoustic details using learnable B-spline basis functions. It captures high-frequency dynamic changes in audio, such as the babbling of water, the alternation of wind sounds, and the subtle nuances of instrument timbre, overcoming the shortcomings of traditional linear models in detail representation. Simultaneously, the application of inverse short-time Fourier transform ensures accurate mapping of high-fidelity audio to time-domain signals, resulting in audio with significantly higher fidelity than traditional methods, free from artifacts, noise, and disconnections. In terms of continuity, rhythmic transitions are natural and smooth, conforming to auditory perception principles. In terms of dynamic expression, it conveys emotional tension through layered changes in volume, timbre, and rhythm, making audio not merely a "superposition of sounds" but an "extension of mood," completely solving the technical problems of rough audio quality and insufficient expressiveness in traditional models. The joint loss function design addresses the issues of training instability and performance imbalance, achieving efficient convergence and multi-objective optimization. Through the weighted fusion of multiple loss terms and precise constraints, the joint loss function achieves dual optimization of training stability and performance balance. This invention presents a specialized architecture adapted to the artistic characteristics of landscape painting, overcoming the bottleneck of insufficient cultural context adaptability and conveying the aesthetic conception of the East. The invention constructs a specialized architecture adapted to the artistic characteristics of landscape painting from three levels: data, features, and models, completely solving the technical problem of insufficient cultural context adaptability. At the data level, the constructed "image-text-emotion" multimodal dataset focuses on Chinese landscape painting, with artistic description text covering Eastern aesthetic dimensions such as composition, artistic conception, and emotion. The audio prompts, optimized by Qwen2.5-VL, accurately match the cultural expression habits of landscape painting. At the feature level, the semantic extraction module of the CLIP model, through targeted fine-tuning, can recognize features such as "texture strokes," "blank space," and "ink rhyme." By incorporating semantic elements specific to landscape paintings, the EmoColor model's emotion extraction module, through supervised learning using emotion-labeled data, can capture emotion types with Eastern cultural characteristics such as "detachment," "broad-mindedness," and "melancholy." At the model level, the multi-view alignment mechanism and cross-modal fusion module can understand the artistic pursuit of "capturing the spirit through form," deeply binding concrete semantic objects with abstract emotional realms. The generated audio not only matches the visual elements of the painting but also conveys its underlying cultural connotations and aesthetic implications—for example, the "ethereal" quality of ink wash landscapes corresponds to a simple and elegant timbre, the "magnificent" quality of green landscapes corresponds to a full and rich audio texture, and the "combination of reality and illusion" in the composition corresponds to the rhythmic alternation of "sound and silence." This deep adaptation imbues the audio generated by this invention with a strong Eastern aesthetic realm, completely breaking free from the constraints of the "Westernized context" of existing models, solving the core technical problem of insufficient cultural adaptability, and providing precise technical support for the cross-modal digital dissemination of traditional art. This invention solves the problems existing in the prior art, achieves a high degree of unity between emotion and semantics, excellent audio quality, and deep adaptation to cultural context, promotes the breakthrough of cross-modal generation technology in the field of art, and opens up a new path for the digital protection and innovative dissemination of traditional Chinese landscape painting. It has important technical value and application prospects.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating audio representations of Chinese paintings based on multi-perspective semantic alignment, characterized in that, Includes the following steps: Step 1: Obtain Chinese painting images and corresponding artistic description texts; use the visual language model Qwen2.5-VL to semantically optimize the descriptive texts and generate audio prompts; construct a multimodal dataset containing "image-text-emotion"; Step 2: Construct a semantic feature extraction module, which includes a CLIP model. The image information from the multimodal dataset is input into the CLIP model to extract visual semantic feature vectors from the image information. ; Step 3: Construct an emotion feature extraction module, which includes an EmoColor model based on the Transformer architecture. Image information from the multimodal dataset is input into the EmoColor model to extract fine-grained emotion feature vectors. ; Step 4: Convert the visual semantic feature vector With fine-grained sentiment feature vectors Projected onto a unified embedding space, a spatial mapping between visual semantics and emotional attributes is established through a multi-view semantic mapping matrix, generating an object-emotion mapping relationship. The visual semantic feature vector and fine-grained emotional feature vector are then fused in a multimodal manner to generate emotion-enhanced contextual fusion features. ; Step 5: Construct an audio generation model, which includes a progressive audio generation network based on Lite-KAN, and integrates the fused features. The input progressive audio generation network includes a coarse prior stage and a refined prior stage, and the fused features... After being processed sequentially through a coarse prior stage and a refined prior stage, high-fidelity audio is output. ; Step 6: During the audio generation process, a joint loss function is used to optimize the audio generation to obtain emotion-aware audio.

2. The method for generating Chinese painting emotion-perceived audio based on multi-perspective semantic alignment according to claim 1, characterized in that: The expression for extracting the visual semantic feature vector in step 2 is: （1）； in, This represents the input landscape painting image; These are the sets of semantic model parameters. The CLIP model's visual-language feature extraction function is a multimodal mapping function obtained from contrastive language-image pre-training. The CLIP model optimizes the embedding space of the visual encoder and text encoder through cross-modal contrastive loss, so that visual features and semantic descriptions have a high degree of correlation in the same semantic space.

3. The method for generating Chinese painting emotion-perception audio based on multi-perspective semantic alignment according to claim 1, characterized in that: In step 3, the expression for fine-grained sentiment feature vector extraction is: （2）； in, The set of parameters representing the sentiment model; This represents the color sentiment feature extraction function of the EmoColor model, where the EmoColor model obtains high-resolution sentiment representations by supervised learning of the Transformer through sentiment-labeled data. This represents the primary image features extracted by the convolutional network; This represents a multi-head self-attention module based on the Transformer architecture, used to model the sentiment relevance between different regions within an image; The weight matrix is ​​a learnable matrix; Nonlinear activation function; This is a global average pooling operation used to generate the final fine-grained sentiment vector. .

4. The method for generating Chinese painting emotion-perceived audio based on multi-perspective semantic alignment according to claim 1, characterized in that: In step 4, while establishing the spatial mapping between visual semantics and emotional attributes, the multi-view semantic mapping matrix is ​​also used to align cross-modal features. The alignment process is driven by contrastive learning, and its expression is: （3）； in, and These represent the linear mapping matrix of the visual semantic feature vector and the linear mapping matrix of the fine-grained sentiment feature vector, respectively. and The projected uniform embedding vector; These are the positive sample feature vectors of the semantic modality; This represents the correct sample feature vector corresponding to the sentiment modality. The negative sample feature vector of the semantic modality; is the interval coefficient between positive and negative samples; N is the number of samples; + indicates taking the non-negative part; This represents the semantic-sentiment alignment loss; By minimizing Achieve multi-perspective coupling between semantic space and emotional space.

5. The method for generating Chinese painting emotion-perceived audio based on multi-perspective semantic alignment according to claim 1, characterized in that: In step 4, the visual semantic feature vector is... With fine-grained sentiment feature vectors The input is processed by a cross-modal fusion module, which uses a cross-modal attention mechanism to fuse multi-channel information and generate emotion-enhanced contextual fusion features. ,include: For the visual semantic feature vector With the fine-grained emotion feature vector Perform linear mappings to obtain the query Q, key K, and value V matrices; Calculate the similarity matrix between the query Q and the key K, and obtain the attention distribution weights using the softmax function. ; The attention-weighted result is concatenated with the visual semantic feature vector residual, and then deep feature fusion is performed through convolution to obtain the context fusion feature. .

6. The method for generating Chinese painting emotion-perceived audio based on multi-perspective semantic alignment according to claim 1, characterized in that: In step 5, the audio generation method specifically includes: First, in the rough prior stage, the fused features are... The input is a Transformer encoder, which models long-range dependencies through a multi-layer self-attention mechanism to generate low-resolution audio features. The features described can be used to describe the overall acoustic emotion and acoustic layout; Secondly, in the refinement of the prior stage, the low-resolution audio features output from the coarse prior stage are used. As input, a progressive audio generation network is used to reconstruct high-resolution details from low-resolution audio features, generating high-fidelity audio. ; The high-fidelity audio Map back to the time-domain audio signal to generate audio. Complete audio generation.

7. The method for generating Chinese painting emotion-perceived audio based on multi-perspective semantic alignment according to claim 1, characterized in that: Joint loss function in step 6 As shown below: （12）； in, This represents the audio reconstruction loss, used to maintain waveform fidelity; This represents semantic-sentiment alignment loss, ensuring cross-modal consistency. This indicates a loss of emotional consistency, constraining the emotional resonance between the generated audio and the input image; These are the weighting coefficients; During audio generation, a reconstruction loss function is used to ensure the consistency of sound quality and emotion in the output signal. The expression for the reconstruction loss function is as follows: （13）； in, The model generates the audio; It is a genuine reference audio file; The average of the first k largest squared errors is taken to improve the model's sensitivity to anomalous frequency bands; These are the weighting coefficients for each loss; the reconstruction loss function is used to ensure the consistency of the output signal in terms of sound quality and smoothness, effectively reducing artifact noise; The generated audio is constrained by cosine similarity between its sentiment embedding space and the sentiment embedding space of the input image. Sentiment consistency loss is employed to maintain directional consistency between the output audio and the input image in the high-dimensional sentiment space, ensuring the synergy of cross-modal sentiment perception. The expression is as follows: （14）； in, This represents the embedding vector of fine-grained emotional features in audio. This represents the embedding vector of fine-grained emotional features of an image.

Images