A multi-modal feature fusion-based cultural and creative image generation method

Abstract

The application discloses a kind of cultural and creative image generation methods based on multi-modal feature fusion, to solve the problem that existing generation method ignores the deep mapping of cultural connotation and visual symbol, leading to the lack of cultural recognition of image.The method of the present application comprises: constructing the cultural gene map of target IP, wherein each gene node is associated with cultural connotation label and visual feature vector;Adopting graph neural network to learn the nonlinear mapping function of cultural connotation semantic space to visual feature space, and mining the transformation rule of "cultural symbol-visual symbol";Receive user text and reference image, convert the cultural keywords in the text into expected visual features through the mapping function, and perform dynamic fusion with the reference image features by semantic similarity weighting;The fusion features are input into the diffusion model, and the cultural gene regularization term is introduced to constrain the generated image not to deviate from the core style of IP;Finally, through brand consistency check, qualified images are output and incrementally updated atlas.The present application significantly improves the cultural recognition and brand consistency of the generated image by explicitly modeling the culture-visual association and introducing active constraints.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for generating cultural and creative images based on multimodal feature fusion. Background Technology

[0002] As a core component of the digital creative economy, the development of visual content for IP derivatives in the cultural and creative industry has become a key path for extending brand value and disseminating culture. With the rapid development of AI-generated content technology, automatically generating cultural and creative images using technologies such as diffusion models and multimodal large language models has become an important trend for reducing design costs and improving creative efficiency. Current mainstream methods for generating cultural and creative images typically employ multimodal feature fusion technology. This involves an encoder extracting visual features from the user-input text description and reference images, then simply concatenating or weighting these features before inputting them into a generation model. The final output is a cultural and creative image that integrates textual intent and image style. For example, existing technologies combine text encoding with image style features to achieve stylized image generation based on prompts. However, existing methods reveal significant technical shortcomings when applied to the design of cultural and creative IP derivatives with specific cultural connotations. These methods only focus on… While focusing on visual fusion at the element level, such as mechanically splicing specific patterns with specific shapes, this approach neglects the deep mapping relationship between cultural connotations and visual symbols. The same cultural connotation may correspond to multiple visual symbols, such as specific patterns, colors, or compositional proportions. This mapping is complex and non-linear. Existing technologies lack the ability to model such deep semantic connections, resulting in images that, although visually integrating multiple elements, perform poorly in terms of cultural recognition and brand consistency. They often suffer from a superficial resemblance but lack the essence, making it difficult to accurately convey the core cultural value of the IP. In fact, style drift may even damage the brand image. Furthermore, existing methods lack proactive constraint mechanisms and closed-loop verification feedback for the generated results, failing to ensure that the generated images continuously conform to the core visual specifications of the IP. Therefore, there is an urgent need for a cultural and creative image generation method that can deeply explore and faithfully reproduce the mapping relationship between the IP's cultural connotations and visual symbols, and possess proactive constraint and closed-loop verification capabilities. Summary of the Invention

[0003] A method for generating cultural and creative images based on multimodal feature fusion includes the following steps; S1. Cultural Gene Map Construction: Obtain the historical works of the target cultural and creative IP, extract high-frequency visual elements, color features and composition patterns, and construct a cultural gene map containing multiple gene nodes. Each gene node contains a cultural connotation label and a corresponding visual feature vector. S2. Semantic-Visual Association Mining: Graph neural networks are used to embed cultural gene maps, establish a nonlinear mapping function between the semantic space of cultural connotation and the visual feature space, and explore the transformation rules of cultural symbols into visual symbols. S3. Multi-source input parsing: Receives text descriptions and reference images input by the user, extracts cultural connotation keywords from the text and visual features from the reference images respectively, and maps the cultural connotation keywords into the desired visual feature vectors through a mapping function; S4. Consistency Constraint Fusion: The desired visual feature vector is weighted and fused with the visual features of the reference image. The fusion weight is dynamically determined by the semantic similarity between the cultural connotation keywords and the reference image to obtain the fused visual features. S5. Stylized Generation: The fused visual features are input into the pre-trained diffusion model. During the generation process, a cultural gene regularization term is introduced to constrain the generated image within the feature space spanned by the cultural gene map. S6. Brand Consistency Verification and Iterative Optimization: Extract key visual features of the generated image, calculate its matching degree with each node in the cultural gene map. If the matching degree is lower than the preset threshold, trigger a rollback correction and re-execute S4 to S5. Add the verified image visual features and corresponding cultural connotation tags to the incremental update library of the cultural gene map. The corresponding cultural connotation tags are determined by the cultural connotation keywords extracted from the image generation conditions, or by selecting the tag with the highest matching degree from multiple candidate tags, so as to realize the continuous evolution of IP style.

[0004] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S1 includes the following specific steps: S11. Data Collection and Feature Extraction: Collect historical works of target cultural and creative IPs covering different periods and types, use target detection algorithms to extract high-frequency visual elements in the dataset, including patterns, designs and object outlines, use color quantization algorithms to extract the main color tone and color scheme to form color feature vectors, and use spatial layout analysis algorithms to identify the composition patterns of the works, including symmetrical composition, central composition and scattered composition. S12. Gene Node Construction: Cluster visual features with the same cultural connotation labels to construct gene nodes containing cultural connotation labels and corresponding visual feature vectors, forming a cultural gene map.

[0005] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S2 includes the following specific steps: S21. Graph structure initialization: Each gene node in the cultural gene map is used as an initial node of the graph neural network. The initial features of the node are the text embedding vectors of cultural connotation labels. The edges between the nodes are constructed based on the co-occurrence relationship, semantic relevance, or combination frequency in historical works to form a graph structure. S22. Graph Embedding and Mapping Learning: A graph attention network is used to embed the graph structure. The information of neighboring nodes is aggregated through the attention mechanism to update the feature representation of each node. The updated node features are input into a multilayer perceptron. The mean square error between the predicted visual feature vector and the corresponding real visual feature vector of the node is used as the loss function for training to obtain a nonlinear mapping function.

[0006] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S3 includes the following specific steps: S31. Text Parsing and Mapping: Receive text descriptions input by users, use natural language processing techniques for word segmentation, part-of-speech tagging, and named entity recognition, extract keywords related to cultural connotations, input the extracted cultural connotation keywords into a pre-trained language model to convert them into fixed-dimensional semantic feature vectors, and input the semantic feature vectors into a nonlinear mapping function to calculate the corresponding expected visual feature vectors. S32. Reference Image Analysis and Similarity Calculation: Receive the reference image input by the user, use a pre-trained visual feature extraction network to extract its global and local features to form the visual feature vector of the reference image, and calculate the cosine similarity between the semantic feature vector of the cultural connotation keywords and the visual feature vector of the reference image as the basis for dynamic fusion weights.

[0007] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, the calculation formula for the fusion weight in step S4 is as follows: , , where sim is the semantic similarity, and its value ranges from [-1, 1]; Scaling factor and >0; It is the sigmoid function; For the desired visual feature fusion weights, The fusion weights are the visual features of the reference image; The formula for calculating the fused visual feature vector is as follows: in For the desired visual feature vector, This is the visual feature vector of the reference image.

[0008] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, in step S5, the cultural gene regularization term is defined as the minimum Euclidean distance between the representation of the currently generated intermediate image in the latent feature space and the feature vectors of all gene nodes in the cultural gene map: ,in Here, E is the image generated at the current time, and E is a pre-trained image encoder used to map the input image to the visual feature vector. Feature spaces of the same dimension Let be the visual feature vector of the i-th gene node; in the inverse denoising process of the diffusion model, the update formula for each step is: Where c represents conditional embedding. It is random noise. Let be the noise variance at step t. The weight coefficients for the regularization term. For regularization terms The gradient.

[0009] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, in step S6, the matching degree is the weighted cosine similarity between the key visual features of the generated image and the visual feature vectors of each gene node in the cultural gene map. The preset threshold for the matching degree is 0.75, and a backoff correction is triggered when the value is lower than the threshold.

[0010] The beneficial effects of this invention are: By constructing a cultural gene map, the abstract cultural connotations of cultural and creative IPs are structurally associated with specific visual symbols. The nonlinear mapping between the two is learned through graph neural networks, so that the generated images can accurately reflect the typical visual features corresponding to the cultural connotations. This solves the technical problem of insufficient cultural recognition caused by the lack of mapping relationship between cultural connotations and visual symbols in existing technologies. By introducing a cultural gene regularization term, the Euclidean distance between the latent feature representation of the generated image and the nearest neighbor gene node in the map is continuously calculated during the inverse denoising process of the diffusion model. The gradient of this distance is used to actively guide the denoising direction toward the map feature space, ensuring that the visual features of the generated image fall within the feature space spanned by the cultural gene map, thereby effectively avoiding the style drift problem caused by the bias of the model training data. By using a dynamic fusion mechanism with semantic similarity weighting, the sigmoid function is used to map the normalized cosine similarity to a fusion weight in the range of [0,1]. This allows for adaptive adjustment of the contribution ratio of the user-input text and the reference image in the generation process based on their actual correlation, thus avoiding the visual feature distortion problem caused by simple equal-weighted fusion in semantic conflict scenarios. The closed-loop mechanism formed by brand consistency verification and incremental map update enables each generated and verified image feature to continuously enrich the cultural gene map, realizing the adaptive evolution and self-reinforcement of IP style in the generation process. Experimental verification shows that the cultural and creative images generated by the method of this invention have improved the cultural connotation expression index by more than 65% and the visual style fit index by more than 85% compared with the traditional multimodal feature splicing method, significantly improving the cultural recognition and brand consistency of the generated images. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a method for generating cultural and creative images based on multimodal feature fusion. Detailed Implementation

[0012] Example 1: A method for generating cultural and creative images based on multimodal feature fusion includes the following steps: S1. Cultural Gene Map Construction: Obtain the historical works of the target cultural and creative IP, extract high-frequency visual elements, color features and composition patterns, and construct a cultural gene map containing multiple gene nodes. Each gene node contains a cultural connotation label and a corresponding visual feature vector. S2. Semantic-Visual Association Mining: Graph neural networks are used to embed cultural gene maps, establish a nonlinear mapping function between the semantic space of cultural connotation and the visual feature space, and explore the transformation rules of cultural symbols into visual symbols. S3. Multi-source input parsing: Receives text descriptions and reference images input by the user, extracts cultural connotation keywords from the text and visual features from the reference images respectively, and maps the cultural connotation keywords into the desired visual feature vectors through a mapping function; S4. Consistency Constraint Fusion: The desired visual feature vector is weighted and fused with the visual features of the reference image. The fusion weight is dynamically determined by the semantic similarity between the cultural connotation keywords and the reference image to obtain the fused visual features. S5. Stylized Generation: The fused visual features are input into the pre-trained diffusion model. During the generation process, a cultural gene regularization term is introduced to constrain the generated image within the feature space spanned by the cultural gene map. S6. Brand Consistency Verification and Iterative Optimization: Extract key visual features of the generated image, calculate its matching degree with each node in the cultural gene map. If the matching degree is lower than the preset threshold, trigger a rollback correction and re-execute S4 to S5. Add the verified image visual features and corresponding cultural connotation tags to the incremental update library of the cultural gene map. The corresponding cultural connotation tags are determined by the cultural connotation keywords extracted from the image generation conditions, or by selecting the tag with the highest matching degree from multiple candidate tags, so as to realize the continuous evolution of IP style.

[0013] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S1 involves clustering visual features with the same cultural connotation label to summarize multiple visual expressions under the same cultural connotation and form a gene node. The clustering uses the K-means algorithm, and the number of clusters K is determined based on the elbow rule, that is, calculating the sum of squared errors within the cluster under different K values ​​and selecting the K value corresponding to the inflection point. Experiments have verified that the clustering effect is best when K is in the range of 5 to 15. Taking the cultural connotation label of auspiciousness as an example, its corresponding visual features include different visual vectors such as auspicious cloud patterns, lingzhi patterns, and auspicious beast patterns. By clustering these vectors, a visual feature distribution space representing auspiciousness is constructed. Any point in this space is regarded as a certain visual expression of the cultural connotation of auspiciousness.

[0014] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S2 involves using a graph neural network to learn a nonlinear mapping function F:S 文化 →V 视觉 Let the cultural gene map be defined as graph G=(V,E), where V is the set of gene nodes and E is the set of edges. initial features The text embedding vectors represent the cultural connotation labels. The text embedding uses a pre-trained BERT model to convert each cultural connotation label into a 768-dimensional semantic feature vector. A graph attention network is used for embedding learning, where for a central node i, the attention coefficients of its neighboring nodes j are... The calculation is as follows: Where W is the shared linear transformation weight matrix, a is the weight vector of the attention mechanism, || denotes the vector concatenation operation, the negative slope of LeakyReLU is set to 0.2, and the attention coefficients are normalized using the softmax function: ; The updated feature representation of node i is as follows: ,in The ELU activation function is used, and the graph attention network has two layers. The output feature dimension of each layer is set to 512 dimensions. After passing through multiple layers of the graph attention network, the features of each node are... It not only includes its own cultural connotations but also integrates information from its neighboring nodes, thus obtaining richer contextual semantics and ultimately acquiring the node's embedding features. Then, a four-layer multilayer perceptron is used as the mapping function F, and its input is... The hidden layer dimensions are set to 1024, 512, and 256 respectively, and the output is the predicted visual feature vector. The visual feature vector has a dimension of 512, and the training objective is to minimize the difference between the predicted value and the true visual feature vector. Mean squared error loss between: ; The parameters of the graph attention network and MLP are optimized by backpropagation algorithm. The optimizer adopts the Adam algorithm, the learning rate is set to 0.001, and the number of training rounds is set to 200 rounds, finally obtaining a well trained mapping function F.

[0015] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, in step S3, when parsing the user-inputted text description, natural language processing techniques are first used for word segmentation, part-of-speech tagging, and named entity recognition to extract keywords related to cultural connotations. The extracted keywords are then input into a pre-trained BERT language model to obtain a fixed-dimensional semantic feature vector. This semantic feature vector is subsequently input into the nonlinear mapping function F trained in step S2, which maps it to the corresponding expected visual feature vector. For the user-inputted reference image, a pre-trained ResNet-50 visual feature extraction network is used to extract its global pooling layer features and intermediate layer local features. The global features and local features are then concatenated to form the reference image visual feature vector. Then, the cosine similarity between the semantic feature vector of the cultural connotation keywords and the visual feature vector of the reference image is calculated as the basis for dynamic fusion weights. The formula for calculating the cosine similarity sim is: ,in This represents the semantic feature vector of the text keywords. This is the visual feature vector of the reference image.

[0016] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, the formula for calculating the dynamic fusion weight in step S4 is as follows: , , where sim is the cosine similarity between the semantic feature vector of the cultural connotation keyword and the visual feature vector of the reference image, and its value ranges from [-1, 1]. Scaling factor The sigmoid function. ,when The range of values ​​is obtained after the sigmoid function maps the value to the interval [0,1]. , The complement is then used accordingly. A higher semantic similarity (sim) between the user-input text description and the reference image indicates a greater consistency between the text intent and the style of the reference image. In this case, higher weights are given to the desired visual features to reinforce consistent expression. Conversely, a lower similarity suggests that the user may prefer the style of the reference image to dominate, thus reducing the weight of the text features. The value of determines the sensitivity of similarity to the weight. Extensive experimental verification shows that when... When sim=1, the weight changes smoothly; when sim=0, the weight changes are smooth. =0.5 achieves balanced fusion. When sim approaches -1 or 1, the weights also approach 0 or 1, achieving a clear switch of dominant weights. L2 norm normalization is used when normalizing the fused visual feature vectors, with the formula: ,in for The L2 norm is used to normalize the eigenvectors to a magnitude of 1, ensuring that the eigenvalues ​​remain stable within a preset range to facilitate the stable generation of subsequent diffusion models.

[0017] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, the logic for setting the cultural gene regularization term R in step S5 is as follows: the set of visual feature vectors of all gene nodes in the cultural gene map is defined as follows. For the image generated at the current moment It is mapped to a signal by a pre-trained image encoder E. The same visual feature space is obtained The image encoder E uses a pre-trained ResNet-50 network, taking its 2048-dimensional features after global pooling and reducing them to 512 dimensions through a fully connected layer. The regularization term is defined as... Euclidean distance between the feature vectors of the nearest neighbor nodes in the graph: In the reverse denoising process, the standard diffusion model's reverse update formula is: Where c represents conditional embedding. For random noise sampled from a standard normal distribution, For the noise variance at step t, the update formula becomes after introducing a regularization term: ,in These are the weighting coefficients of the regularization term, used to control the strength of the constraints. It is a regular term about The gradient, whose direction points towards the point that makes... The direction closer to the feature space of the map The value of has been verified experimentally. When the constraint effect and generation diversity are at 0.1, the optimal balance is achieved. When the value is less than 0.05, the constraint effect is not obvious; when... When the value is greater than 0.3, the diversity of generated images is severely reduced, and the total number of iterations for reverse denoising is set to 1000 steps.

[0018] Furthermore, in a method for generating cultural and creative images based on multimodal feature fusion, step S6 uses weighted cosine similarity to calculate the matching degree, assuming the key visual feature vector of the generated image is... The visual feature vector of the i-th gene node in the cultural gene map is The matching degree is calculated as follows: ,in For vector dot product, and The L2 norm of the vectors are used for matching, and the preset threshold for matching is 0.75. This threshold is established based on statistical analysis of 500 samples randomly selected from the historical works of the target cultural and creative IP. The maximum cosine similarity between each sample and each gene node in the map is calculated, and the lower quartile of its distribution is taken as the threshold. When the MatchScore is lower than 0.75, a backtracking correction is triggered. The backtracking correction mechanism discards the currently generated image and re-executes steps S4 to S5. The maximum number of backtrackings is set to 3. If it is still unqualified after 3 times, the user is prompted to adjust the input parameters. The qualified image is used as the final output. At the same time, its visual features are extracted and associated with the corresponding cultural connotation tags and added to the incremental update library. According to the preset update cycle, the data in the incremental update library is merged into the cultural gene map to realize the continuous evolution of IP style.

[0019] Example 2: The method of the present invention will be illustrated by taking Jingdezhen ceramic IP as a specific application scenario.

[0020] In step S1, the specific implementation in this embodiment is as follows: 1000 representative works of Jingdezhen ceramics IP from Song Dynasty celadon to modern creative porcelain are obtained. High-frequency visual elements in the works are extracted using a target detection algorithm, including intertwined branches, lotus patterns, fish and algae patterns, and children playing scenes. The main color tones are extracted using a color quantization algorithm, including celadon, shadow blue, underglaze red, and famille rose. The composition patterns are identified using a spatial layout analysis algorithm, including full-work, openwork, and continuous scene. Gene nodes with the cultural connotation label of auspiciousness and wealth are constructed. The associated visual feature vectors include the outline features of intertwined branches, the morphological features of lotus patterns, and the color features of famille rose. Gene nodes with the cultural connotation label of literati elegance are constructed. The visual feature vectors include the composition features of landscape patterns and figure paintings, as well as the color tones of celadon. Edges are constructed between nodes based on the co-occurrence frequency in historical works. Intertwined branches and lotus patterns often appear simultaneously on the same piece of porcelain. There are strong correlation edges between the two nodes. A total of 12 gene nodes and 23 edge relationships are constructed.

[0021] In step S2, the specific implementation in this embodiment is as follows: The constructed graph is input into the graph attention network for training. After 200 rounds of training, the network learns the typical mapping rules of the cultural connotation of auspiciousness and wealth in the visual style of ceramics. When the text embedding of auspiciousness and wealth is input, the trained mapping function F outputs a 512-dimensional visual feature vector. The Euclidean distance between this vector and the feature vector of the intertwined branch pattern is 0.23, the Euclidean distance between this vector and the feature vector of the lotus pattern is 0.31, the Euclidean distance between this vector and the feature vector of the pastel tone is 0.18, the distance from the feature vector of the blue-white pattern is 0.89, and the distance from the feature vector of the landscape pattern is 0.94.

[0022] In step S3, the specific implementation in this embodiment is as follows: The user input text description is "auspicious and prosperous wedding tea set," and the user input reference image is a teacup image with a modern minimalist style. The cultural connotation keywords "auspicious and prosperous" and "wedding" are extracted from the text. These are converted into semantic feature vectors using a pre-trained language model, and then mapped to the desired visual features using a mapping function F. This feature tends towards the traditional lotus scrolling and pastel style. Visual features such as its modern minimalist outline and low-saturation colors are extracted from the reference image using a pre-trained visual feature extraction network to construct the visual feature vector of the reference image. The semantic similarity between the auspicious and prosperous image and the reference image is calculated to be sim=0.30.

[0023] In step S4, the specific implementation in this embodiment is as follows: Calculate the dynamic fusion weight based on semantic similarity. =1, then = =1 / (1+e^(-0.30))≈0.57, =1- The weighted sum of the expected visual feature vector and the reference image visual feature vector is approximately 0.43, which yields the fused visual feature vector. The fused visual feature vectors are normalized using the L2 norm and then output to the diffusion model.

[0024] In step S5, the specific implementation in this embodiment is as follows: The fused visual feature vector is used as the conditional embedding input to pre-train the diffusion model. During the inverse denoising process of the diffusion model, the cultural gene regularization term... It continues to function effectively, even at step 500 of the reverse denoising process. The value is 2.31, which drops to 0.87 at step 800 and to 0.34 at step 1000. If the image deviates from the style of Jingdezhen ceramics during the generation process, such as generating exotic patterns that should not appear on blue and white porcelain, the gradient value of the regularization term will reach more than 5.6 and forcibly correct the generation direction, guiding it back to the feature space spanned by the Jingdezhen ceramic atlas, and finally generating a simple style porcelain image with intertwined branches.

[0025] In step S6, the specific implementation in this embodiment is as follows: extract the key visual features of the generated image, calculate its matching degree with each node in the cultural gene map. The matching degree with the auspicious and prosperous node is 0.87, and the matching degree with the literati and refined taste node is 0.34. The highest matching degree of 0.87 is higher than the preset threshold of 0.75, so no backtracking correction is required. The image that passes the verification is used as the output, and its visual features and corresponding cultural connotation tags are added to the incremental update library of the cultural gene map to realize the continuous evolution of the Jingdezhen ceramic IP style.

[0026] Performance Verification: To verify the effectiveness of the method of this invention, a combination of objective evaluation and subjective assessment was adopted. The objective metrics used were FID and CLIP-Score. FID measures the distributional similarity between the generated image and the target IP's real portfolio, while CLIP-Score measures the semantic consistency between the generated image and the input text description. On the test set of Jingdezhen ceramic IPs, the FID value of the method of this invention was 32.8, significantly better than the 52.7 of the traditional multimodal feature stitching method and the 41.3 of the method without regularization constraints. The CLIP-Score... The value of 0.314 is better than the comparison methods of 0.291 and 0.278, indicating that the cultural gene regularization term effectively constrains the generated image to fall within the real visual distribution space of the target IP, and the dynamic fusion mechanism improves the fidelity of the textual intent. In a blind test comparison of 20 cultural and creative designers, the present invention showed a preference rate of 73.0% and 70.5% in the dimensions of cultural fit and style consistency, respectively, compared with the traditional splicing method. This indicates that the proposed cultural gene map guidance mechanism also significantly improves the cultural recognition and style fidelity of the generated image at the subjective perception level.

[0027] Example 3: The method of this invention is illustrated by taking the Palace Museum's cultural IP as a specific application scenario.

[0028] In step S1, the specific implementation in this embodiment is as follows: 800 representative cultural relics from the Ming and Qing dynasties, encompassing categories such as porcelain, calligraphy and painting, textiles and embroidery, and enamelware, are obtained from the Palace Museum's cultural IP. High-frequency visual elements are extracted from the works using a target detection algorithm, including dragon patterns, phoenix patterns, cloud patterns, sea and cliff patterns, and lotus scroll patterns. The main color tones are extracted using a color quantization algorithm, including bright yellow, vermilion, lapis lazuli, gold, and lotus root color. Spatial layout analysis algorithms are used to identify composition patterns, including symmetrical composition, central radial composition, and full-coverage composition, thus constructing a cultural connotation marker. The gene node with the signature of supreme imperial power has visual feature vectors including the outline of the dragon pattern on the front, the color of bright yellow, and the layout of the symmetrical composition. The gene node with the cultural connotation label of auspiciousness and good fortune has visual feature vectors including the combination of cloud and bat patterns, the color of red, and the layout of the central radial composition. Edges are constructed between nodes based on the co-occurrence frequency in cultural relics. Dragon patterns and cloud patterns often appear simultaneously on the same court artifact. There are strong correlation edges between two nodes. A total of 15 gene nodes and 28 edge relationships are constructed.

[0029] In step S2, the specific implementation in this embodiment is as follows: The constructed graph is input into the graph attention network for training. After 200 rounds of training, the network learns the typical mapping rules of the cultural connotation of imperial supremacy in the visual style of the palace. When the text embedding of imperial supremacy is input, the visual feature vector output by the trained mapping function F has a Euclidean distance of 0.19 with the frontal dragon pattern feature vector, a Euclidean distance of 0.22 with the bright yellow tone feature vector, a Euclidean distance of 0.27 with the symmetrical composition feature vector, a distance of 0.78 from the phoenix pattern feature vector, and a distance of 0.91 from the lotus root color feature vector.

[0030] In step S3, the specific implementation in this embodiment is as follows: The user input text description is a New Year's greeting card for the Emperor's Supreme Power; the user input reference image is a red greeting card image with a modern minimalist style; the cultural connotation keywords "Emperor's Supreme Power" and "New Year" are extracted from the text; the text is converted into a semantic feature vector using a pre-trained language model; and "Emperor's Supreme Power" is mapped to the desired visual features using a mapping function F. This feature tends towards a frontal dragon pattern, bright yellow tones, and a symmetrical composition style. Visual features such as the minimalist layout and dominant red color scheme are extracted from the reference image using a pre-trained visual feature extraction network to construct the reference image's visual feature vector. The semantic similarity between the imperial image and the reference image is calculated to be sim=0.25.

[0031] In step S4, the specific implementation in this embodiment is as follows: Calculate the dynamic fusion weight based on semantic similarity. =1, then = =1 / (1+e^(-0.25))≈0.56, =1- The weighted sum of the expected visual feature vector and the reference image visual feature vector is approximately 0.44, which yields the fused visual feature vector. This feature balances traditional court style with modern minimalist style. The fused visual feature vector is normalized by L2 norm and then output to the diffusion model.

[0032] In step S5, the specific implementation in this embodiment is as follows: The fused visual feature vector is used as the conditional embedding input to pre-train the diffusion model. During the inverse denoising process of the diffusion model, the cultural gene regularization term... It continues to function effectively, even at step 500 of the reverse denoising process. The value is 2.67, which drops to 0.95 at step 800 and to 0.41 at step 1000. If the image deviates from the style of the Palace Museum's court culture during the generation process, such as generating content that is too complicated or does not conform to the court regulations, the gradient of the regularization term will forcefully correct the generation direction, guiding it back to the feature space spanned by the Palace Museum's court culture map, and finally generating a greeting card image that integrates modern minimalist layout and court pattern elements.

[0033] In step S6, the specific implementation in this embodiment is as follows: extract the key visual features of the generated image, calculate its matching degree with each node in the cultural gene map. The matching degree with the Imperial Power Supreme node is 0.82, and the matching degree with the Auspicious and Prosperous node is 0.56. The highest matching degree of 0.82 is higher than the preset threshold of 0.75, so no backtracking correction is required. The image that passes the verification is used as the output, and its visual features and corresponding cultural connotation tags are added to the incremental update library of the cultural gene map to realize the continuous evolution of the Palace Museum's court culture IP style.

[0034] Performance Verification: On the test set of Palace Museum cultural IP, the FID value of the method of this invention is 35.2 and the CLIP-Score value is 0.306, both of which are better than the comparison method. The ablation experiment results show that the FID reduction when the graph neural network mapping function and the cultural gene regularization term work together is 23.5, which is greater than the sum of their individual contributions of 11.5 and 9.4, respectively, verifying that there is a significant synergistic enhancement effect between semantic-visual mapping and regularization constraints. In addition, quantitative statistics on the core cultural elements of the palace in the generated images show that the detection rate of dragon patterns in the images generated by the method of this invention reaches 76%, the average proportion of bright yellow areas is 16.5%, and the average SSIM value of symmetry is 0.67, all of which are significantly higher than those of various ablation variants, further quantitatively proving the effectiveness of this invention in accurately reproducing the typical visual features of specific cultural IP.

[0035] Example 4: The method of the present invention will be illustrated by taking the Dunhuang mural cultural IP as a specific application scenario.

[0036] In step S1, the specific implementation in this embodiment is as follows: Obtain 600 representative mural works from different periods, from the Northern Wei to the Yuan Dynasty, representing the Dunhuang mural cultural IP. Extract high-frequency visual elements from the works using a target detection algorithm, including flying apsaras, lotus patterns, scroll patterns, honeysuckle patterns, donor figures, and the nine-colored deer. Extract the main color tones using a color quantization algorithm, including earth red, malachite green, ochre, lapis lazuli blue, and yellow ochre. Identify composition patterns using a spatial layout analysis algorithm, including centripetal composition, horizontal scroll composition, and symmetrical composition. Construct a cultural connotation label of "Buddhist Pure Land". Gene nodes, whose associated visual feature vectors include the ribbon shape of flying apsaras, the shape of lotus flowers, and the color combination of earth red and malachite green, construct gene nodes with the cultural connotation label of Silk Road integration. Their associated visual feature vectors include the mixed pattern of scroll and honeysuckle patterns, the color of lapis lazuli blue, and the layout of horizontal scroll composition. Edges are constructed between nodes based on the co-occurrence frequency in the murals. Flying apsaras and lotus patterns often appear simultaneously in the same mural, and there are strong correlation edges between two nodes. A total of 10 gene nodes and 18 edge relationships are constructed.

[0037] In step S2, the specific implementation in this embodiment is as follows: The constructed graph is input into the graph attention network for training. After 200 rounds of training, the network learns the typical mapping rules of the cultural connotation of the Pure Land of Buddhism in the visual style of Dunhuang murals. When the text embedding of the Pure Land of Buddhism is input, the visual feature vector output by the trained mapping function F has an Euclidean distance of 0.21 with the flying ribbon feature vector, an Euclidean distance of 0.28 with the lotus pattern feature vector, an Euclidean distance of 0.25 with the combination feature vector of earth red and malachite green, a distance of 0.83 from the nine-colored deer feature vector, and a distance of 0.87 from the earth yellow tone feature vector.

[0038] In step S3, the specific implementation in this embodiment is as follows: The user input text description is a silk scarf design representing a Buddhist Pure Land; the user input reference image is a square silk scarf image with a contemporary minimalist style; the cultural connotation keywords "Buddhist Pure Land" and "silk scarf" are extracted from the text; this is converted into a semantic feature vector using a pre-trained language model; and "Buddhist Pure Land" is mapped to the desired visual features using a mapping function F. This feature tends towards a style reminiscent of flying ribbons, lotus patterns, and earthy red and stone green color schemes. Visual features such as a simple square composition and soft neutral tones are extracted from the reference image using a pre-trained visual feature extraction network to construct the reference image's visual feature vector. The semantic similarity between the Buddhist Pure Land and the reference image was calculated to be sim=0.20.

[0039] In step S4, the specific implementation in this embodiment is as follows: Calculate the dynamic fusion weight based on semantic similarity. =1, then = = =1 / (1+e^(-0.20))≈0.55, =1- The weighted sum of the expected visual feature vector and the reference image visual feature vector is approximately 0.45, which yields the fused visual feature vector. This feature leans towards a contemporary minimalist style but retains the core visual elements of the Buddhist Pure Land. The fused visual feature vector is normalized using the L2 norm and then output to the diffusion model.

[0040] In step S5, the specific implementation in this embodiment is as follows: The fused visual feature vector is used as the conditional embedding input to pre-train the diffusion model. During the inverse denoising process of the diffusion model, the cultural gene regularization term... It continues to function effectively, even at step 500 of the reverse denoising process. The value is 2.54, which drops to 0.91 at step 800 and to 0.38 at step 1000. If the image deviates from the style of Dunhuang murals during the generation process, such as generating content that does not conform to the Dunhuang color system or pattern rules, the gradient of the regularization term will forcefully correct the generation direction, guiding it back to the feature space spanned by the Dunhuang mural atlas, and finally generating a silk scarf design image that integrates contemporary minimalist composition and Dunhuang pattern elements.

[0041] In step S6, the specific implementation in this embodiment is as follows: extract the key visual features of the generated image, calculate its matching degree with each node in the cultural gene map. The matching degree with the Buddhist Pure Land node is calculated to be 0.79, and the matching degree with the Silk Road Integration node is 0.52. The highest matching degree of 0.79 is higher than the preset threshold of 0.75, so no backtracking correction is required. The image that passes the verification is used as the output, and its visual features and corresponding cultural connotation tags are added to the incremental update library of the cultural gene map to realize the continuous evolution of the Dunhuang mural cultural IP style.

[0042] Performance Verification: On the test set of Dunhuang mural IP, the FID value of the method of this invention was 37.5, and the CLIP-Score value was 0.298. The FID improvement rates in the three IP scenarios were 37.8%, 36.3%, and 36.0%, respectively, with a standard deviation of only 0.94%, indicating that the invention has a highly consistent performance improvement effect and good scene generalization ability in different cultural IP scenarios. In the blind test comparison, the invention achieved a cultural fit preference ratio of 74.0% compared with the traditional splicing method, further verifying that the proposed cultural gene map construction method and semantic-visual mapping mechanism have good adaptability to different types of visual cultural symbol systems.

Claims

1. A method for generating cultural and creative images based on multimodal feature fusion, characterized in that, Includes the following steps: S1. Cultural Gene Map Construction: Obtain the historical works of the target cultural and creative IP, extract high-frequency visual elements, color features and composition patterns, and construct a cultural gene map containing multiple gene nodes. Each gene node contains a cultural connotation label and a corresponding visual feature vector. S2. Semantic-Visual Association Mining: Graph neural networks are used to embed cultural gene maps, establish a nonlinear mapping function between the semantic space of cultural connotation and the visual feature space, and explore the transformation rules of cultural symbols into visual symbols. S3. Multi-source input parsing: Receives text descriptions and reference images input by the user, extracts cultural connotation keywords from the text and visual features from the reference images respectively, and maps the cultural connotation keywords into the desired visual feature vectors through a mapping function; S4. Consistency Constraint Fusion: The desired visual feature vector is weighted and fused with the visual features of the reference image. The fusion weight is dynamically determined by the semantic similarity between the cultural connotation keywords and the reference image to obtain the fused visual features. S5. Stylized Generation: The fused visual features are input into the pre-trained diffusion model. During the generation process, a cultural gene regularization term is introduced to constrain the generated image within the feature space spanned by the cultural gene map. S6. Brand Consistency Verification and Iterative Optimization: Extract key visual features of the generated image, calculate its matching degree with each node in the cultural gene map, and if the matching degree is lower than the preset threshold, trigger a rollback correction and re-execute S4 to S5. Add the verified image visual features and corresponding cultural connotation tags to the incremental update library of the cultural gene map to achieve the continuous evolution of IP style.

2. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S1, the specific steps are as follows: S11. Data Collection and Feature Extraction: Collect historical works of target cultural and creative IPs covering different periods and types, use target detection algorithms to extract high-frequency visual elements in the dataset, use color quantization algorithms to extract the main color tone and color scheme to form color feature vectors, and use spatial layout analysis algorithms to identify the composition patterns of the works. S12. Gene Node Construction: Cluster visual features with the same cultural connotation labels to construct gene nodes containing cultural connotation labels and corresponding visual feature vectors, forming a cultural gene map.

3. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S2, the specific steps are as follows: S21. Graph structure initialization: Each gene node in the cultural gene map is used as an initial node of the graph neural network. The initial features of the node are the text embedding vectors of cultural connotation labels. The edges between the nodes are constructed based on the co-occurrence relationship, semantic relevance, or combination frequency in historical works to form a graph structure. S22. Graph Embedding and Mapping Learning: A graph attention network is used to embed the graph structure. The information of neighboring nodes is aggregated through the attention mechanism to update the feature representation of each node. The updated node features are input into a multilayer perceptron. The mean square error between the predicted visual feature vector and the corresponding real visual feature vector of the node is used as the loss function for training to obtain a nonlinear mapping function.

4. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S3, the specific steps are as follows: S31. Text Parsing and Mapping: Receive text descriptions input by users, extract cultural connotation keywords using natural language processing technology, input the extracted cultural connotation keywords into a pre-trained language model to convert them into fixed-dimensional semantic feature vectors, and input the semantic feature vectors into a nonlinear mapping function to calculate the corresponding expected visual feature vectors. S32. Reference Image Analysis and Similarity Calculation: Receive the reference image input by the user, use a pre-trained visual feature extraction network to extract its global and local features to form the visual feature vector of the reference image, and calculate the cosine similarity between the semantic feature vector of the cultural connotation keywords and the visual feature vector of the reference image as the basis for dynamic fusion weights.

5. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S4, the formula for calculating the fusion weight is: , ,in, For semantic similarity, Scaling factor and >0, For the sigmoid function, For the desired visual feature fusion weights, The fusion weights are used as references to the visual features of the image; the formula for calculating the fused visual feature vector is: ,in For the desired visual feature vector, This is the visual feature vector of the reference image.

6. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S5, the cultural gene regularization term is defined as the minimum Euclidean distance between the representation of the currently generated intermediate image in the latent feature space and the feature vectors of all gene nodes in the cultural gene map: ,in Here, E is the image generated at the current time, and E is a pre-trained image encoder used to map the input image to the visual feature vector. Feature spaces of the same dimension Let be the visual feature vector of the i-th gene node; in the inverse denoising process of the diffusion model, the update formula for each step is: Where c represents conditional embedding. It is random noise. Let be the noise variance at step t. The weight coefficients for the regularization term. For regularization terms The gradient.

7. The method for generating cultural and creative images based on multimodal feature fusion as described in claim 1, characterized in that, In step S6, the matching degree is the weighted cosine similarity between the key visual features of the generated image and the visual feature vectors of each gene node in the cultural gene map. The preset threshold for the matching degree is 0.

75. If it is lower than the threshold, a backoff correction is triggered.

Images