A modeling method for Chinese traditional New Year pictures based on multi-modal feature fusion

Abstract

The present application relates to the field of intelligent classification modeling, and specifically to a Chinese traditional New Year picture modeling method based on multi-modal feature fusion, the present application increases the style perception attention module in the visual feature extraction network of New Year picture style, identifies and emphasizes the unique visual features of different New Year picture schools, and at the same time, identifies the visual form of text symbols to understand cultural implications; the present application introduces the analysis of the cultural metaphor of New Year pictures, establishes fine correspondence between pixel-level visual areas and word-level text concepts through a visual-text bidirectional attention mechanism, and increases the understanding of text information; the present application constructs a New Year picture cultural knowledge graph, models visual, text and context features as three types of nodes in a heterogeneous graph, and explicitly models complex relationships; on this basis, a cultural alignment matrix is introduced to encode inter-modal correlation constraints under the guidance of cultural knowledge, a spatial semantic bias matrix is introduced to increase the understanding of the regional composition of New Year pictures, and the optimization classification modeling of traditional New Year pictures is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent classification modeling, specifically a modeling method for traditional Chinese New Year paintings based on multimodal feature fusion. Background Technology

[0002] The multimodal feature fusion-based modeling method for traditional Chinese New Year paintings can combine image and textual descriptions of multiple modalities to more comprehensively and deeply depict the characteristics and cultural connotations of traditional Chinese New Year paintings, enabling scientific classification and modeling of these paintings. Existing image classification modeling methods, when applied to traditional New Year paintings, often suffer from limitations in visual feature extraction, insufficient textual information processing, a lack of cultural contextual information, and oversimplification of multimodal fusion. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a modeling method for traditional Chinese New Year paintings based on multimodal feature fusion. Addressing the limitations of existing image classification modeling methods when applied to traditional New Year paintings in terms of visual feature extraction, this invention designs a visual feature extraction network specifically for New Year painting styles, adding a style-aware attention module to identify and emphasize the unique visual features of different New Year painting schools. In the visual processing flow, a symbol detection and recognition sub-network is set up to not only recognize the visual form of text symbols but also understand their cultural connotations. A hierarchical feature extraction strategy is adopted to extract high-level semantic features while preserving low-level detailed texture features. Addressing the shortcomings of existing image modeling techniques in text information processing, this invention uses a dedicated semantic understanding module to analyze the cultural metaphors and symbolic meanings of New Year paintings through visual-textual... This bidirectional attention mechanism establishes a fine correspondence between pixel-level visual regions and word-level textual concepts. Alignment is based not only on visual similarity but also on cultural knowledge, enhancing the understanding of textual information. Addressing the lack of cultural contextual information and the simplification of multimodal fusion, this invention designs a comprehensive contextual feature extraction framework and constructs a cultural knowledge graph of New Year paintings. During training and inference, the model can query the knowledge graph to obtain relevant cultural background knowledge. Simultaneously, visual, textual, and contextual features are modeled as three types of nodes in a heterogeneous graph. The complex relationships between them are explicitly modeled through a graph attention network. Based on the standard attention mechanism, a cultural alignment matrix is ​​introduced to encode intermodal association constraints guided by cultural knowledge. A spatial semantic bias matrix is ​​also introduced to enhance the understanding of the regional composition of New Year paintings, ultimately achieving optimized classification and modeling of traditional New Year paintings.

[0004] This invention provides a method for modeling traditional Chinese New Year paintings based on multimodal feature fusion, which specifically includes the following steps: Step S1: New Year picture data collection, which involves collecting a dataset of New Year pictures using big data analytics. Step S2: Visual feature extraction. Visual directional features are extracted from the New Year picture, and multi-scale fusion optimization is performed to obtain visual features. This specifically includes the following steps: Step S21: Use ResNet-50 to extract basic features of the overall composition of the New Year picture; Step S22: Use DEPnet to extract detailed features of brushstrokes and textures from the New Year paintings; Step S23: Draw the color histogram of the New Year picture and extract features from the histogram to obtain color features; Step S24: Classify all New Year paintings into different styles and schools, and design a style-aware attention module to extract style features based on the significant differences in visual style among different New Year painting schools. ; ; In the formula, This represents all the characteristics of the predefined j-th style of New Year paintings. This represents the multi-scale features of New Year paintings used for stylistic feature extraction, including basic features, detail features, and color features. Represents the cosine similarity function. The number of representative styles and schools, Represents the attention coefficient. The weight matrix representing the j-th style of New Year paintings. Represents basic characteristics, Representative style characteristics; Step S25: Integrate basic features, detail features, color features, and style features to obtain visual features; Step S3: Text feature extraction from New Year pictures. This involves detecting, recognizing, and extracting features from the text in the New Year pictures to obtain text features. Specifically, this includes the following steps: Step S31: Use TextDetector to detect text regions in the New Year picture; Step S32: Use TextRecognizer to perform text recognition and extraction on the text regions in the New Year picture; Step S33: Use the large model of CultureLLM to perform semantic analysis on the text and obtain text features; Step S4: Cultural context feature extraction. This involves extracting cultural context features related to the historical background, regional culture, and usage scenarios of the New Year paintings. Specifically, this includes the following steps: Step S41: Metadata feature extraction, the metadata of the New Year picture is feature-encoded to obtain metadata features, the metadata includes the year, place of origin, and author; Step S42: Embedding the cultural knowledge graph, constructing a New Year painting cultural knowledge graph G=(V,E,R), where V represents entities, including New Year paintings, figures, symbols, and themes; E represents relations, including belonging, symbolizing, created from, and influencing; and R represents relation type. The New Year painting cultural knowledge graph is embedded using a graph neural network to obtain cultural knowledge features. Step S43: Use scene feature extraction to encode the usage scenes of the New Year picture based on text to obtain usage scene features; Step S44: Cultural context feature fusion, which integrates metadata features, cultural knowledge features, and usage scenario features to obtain cultural context features; Step S5: Processing based on heterogeneous graph cross-modal attention mechanism. Visual features, text features, and cultural context features are processed using a heterogeneous graph-based cross-modal attention mechanism to obtain multimodal fusion features. This specifically includes the following steps: Step S51: Intramodal feature self-enhancement, performing self-attention optimization on visual and text features; Step S52: Fine-grained visual-text alignment. An attention mechanism is used to interact with the optimized visual and text features. A fine correspondence is established between pixel-level visual features and word-level text features based on the dimension of cultural context features. This captures the semantic association between the image region and textual concept of the New Year picture, resulting in aligned visual and text features. Specifically, this includes the following steps: Step S521: Construction of the cultural semantic alignment matrix. This involves constructing a cultural semantic alignment matrix to introduce cultural knowledge to guide attention allocation. ; In the formula, Represents cultural similarity, Represents the optimized visual and textual features. Represents the preset weight matrix. Represents the ReLU activation function. The cultural semantic alignment matrix represents the cultural similarity, which is calculated as follows: ; In the formula, The function representing the mapping from visual features to cultural contextual features. The mapping function represents the mapping from text features to cultural context features, where i and j represent the i-th position of the optimized visual feature and the j-th position of the optimized text feature, respectively. Step S522: Constructing the spatial semantic bias matrix. Based on the specific cultural meanings of different regions in the New Year picture, construct a region-based spatial semantic bias matrix for the New Year picture: ; In the formula, The sine code representing the image position (i,j) of the New Year picture. The classification vector representing the 9-square grid area of ​​the New Year picture. Represents spatial location weight. Represents the regional classification weight. Represents the spatial semantic bias matrix; Step S523: Cross-attention calculation: Construct a query vector based on the optimized visual features, construct key and value vectors based on the optimized text features, and calculate the cross-attention score. ; In the formula, These represent the query vector and the key vector, respectively. Represents the activation function. Represents the cross-attention score; Step S524: Cross-modal feature fusion, aligning the optimized visual features and text features according to the cross attention score to obtain aligned visual features and aligned text features; Step S53: Heterogeneous graph attention network processing. The multimodal features, consisting of aligned visual features, aligned text features, and cultural context features, are treated as heterogeneous graphs. A heterogeneous graph attention network is used for processing to obtain multimodal fusion features. This specifically includes the following steps: Step S531: Node construction. Construct visual nodes, where each node is aligned with a spatial location of a visual feature; construct text nodes, where each node corresponds to a word in the text sequence of the New Year picture; construct cultural context nodes, where each node corresponds to a cultural context dimension. Step S532: Edge construction, construct bidirectional edges between each pair of visual nodes, text nodes, and cultural context nodes; Step S533: Graph composition, all visual nodes, text nodes, cultural context nodes, and bidirectional edges constitute a heterogeneous graph; Step S534: Graph attention network processing. The heterogeneous graph is processed by a heterogeneous graph attention network to obtain multimodal fusion features. Step S6: New Year picture classification modeling, activate and classify the multimodal fusion features, and assign classification labels to different New Year picture categories to obtain the classification model of New Year pictures.

[0005] The beneficial results achieved by the present invention using the above solution are as follows: (1) In view of the limitations of existing image classification modeling when used for traditional New Year paintings, this invention designs a visual feature extraction network specifically for New Year painting styles, adds a style perception attention module, identifies and emphasizes the unique visual features of different New Year painting schools, sets up a symbol detection and recognition sub-network in the visual processing flow, not only recognizes the visual form of text symbols, but also understands their cultural connotations, and adopts a hierarchical feature extraction strategy to extract high-level semantic features while retaining low-level detailed texture features. (2) In view of the shortcomings of existing image modeling technology in text information processing, this invention analyzes the cultural metaphors and symbolic meanings of New Year pictures through a dedicated semantic understanding module. Through the visual-text bidirectional attention mechanism, it establishes a fine correspondence between pixel-level visual regions and word-level text concepts. The alignment is not only based on visual similarity, but also takes into account cultural knowledge, thereby increasing the understanding of text information. (3) In response to the lack of cultural context information and the simplification of multimodal fusion, this invention designs a comprehensive context feature extraction framework and constructs a cultural knowledge graph of New Year paintings. During training and inference, the model can query the knowledge graph to obtain relevant cultural background knowledge. At the same time, visual, text, and context features are modeled as three types of nodes in a heterogeneous graph. The complex relationship between them is explicitly modeled through a graph attention network. On the basis of the standard attention mechanism, a cultural alignment matrix is ​​introduced to encode the intermodal association constraints under the guidance of cultural knowledge. A spatial semantic bias matrix is ​​introduced to increase the understanding of the regional composition of New Year paintings. Finally, the optimized classification modeling of traditional New Year paintings is realized. Attached Figure Description

[0006] Figure 1 The flowchart illustrates a method for modeling traditional Chinese New Year paintings based on multimodal feature fusion, as provided in this invention.

[0007] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0008] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0009] Example 1, see Figure 1 A modeling method for traditional Chinese New Year paintings based on multimodal feature fusion, specifically including the following steps: Step S1: New Year picture data collection, which involves collecting a dataset of New Year pictures using big data analytics. Step S2: Visual feature extraction. Visual directional features are extracted from the New Year picture and multi-scale fusion optimization is performed to obtain visual features. Step S3: New Year picture text feature extraction. The text in the New Year picture is detected, recognized and its features are extracted to obtain the text features; Step S4: Cultural context feature extraction, extracting cultural context features related to the historical background, regional culture and usage scenarios of the New Year picture; Step S5: Processing based on heterogeneous graph cross-modal attention mechanism, visual features, text features and cultural context features are processed by heterogeneous graph cross-modal attention mechanism to obtain multimodal fusion features; Step S6: New Year picture classification modeling, activate and classify the multimodal fusion features, and assign classification labels to different New Year picture categories to obtain the classification model of New Year pictures.

[0010] Example 2, this example is based on the above example, step S2 specifically includes the following steps: Step S21: Use ResNet-50 to extract basic features of the overall composition of the New Year picture; Step S22: Use DEPnet to extract detailed features of brushstrokes and textures from the New Year paintings; Step S23: Draw the color histogram of the New Year picture and extract features from the histogram to obtain color features; Step S24: Classify all New Year paintings into different styles and schools, and design a style-aware attention module to extract style features based on the significant differences in visual style among different New Year painting schools. ; ; In the formula, This represents all the characteristics of the predefined j-th style of New Year paintings. This represents the multi-scale features of New Year paintings used for stylistic feature extraction, including basic features, detail features, and color features. Represents the cosine similarity function. The number of representative styles and schools, Represents the attention coefficient. The weight matrix representing the j-th style of New Year paintings. Represents basic characteristics, Representative style characteristics; Step S25: Integrate the basic features, detailed features, color features, and style features to obtain the visual features.

[0011] Example 3, this example is based on the above example, step S3 specifically includes the following steps: Step S31: Use TextDetector to detect text regions in the New Year picture; Step S32: Use TextRecognizer to perform text recognition and extraction on the text regions in the New Year picture; Step S33: Use the large model of CultureLLM to perform semantic analysis on the text and obtain text features.

[0012] Example 4: This example is based on the above examples. Step S5 specifically includes the following steps: Step S51: Intramodal feature self-enhancement, performing self-attention optimization on visual and text features; Step S52: Visual-text fine-grained alignment. The optimized visual features and text features interact with each other through an attention mechanism. Based on the dimension of cultural context features, a fine correspondence is established between pixel-level visual features and word-level text features. The semantic association between the image region and text concept of the New Year picture is captured, and the aligned visual features and aligned text features are obtained. Step S53: Heterogeneous graph attention network processing: Construct a heterogeneous graph from the multimodal features consisting of aligned visual features, aligned text features, and cultural context features, and process it using a heterogeneous graph attention network to obtain multimodal fusion features.

[0013] Example 5, this example is based on the above examples, step S52 specifically includes the following steps: Step S521: Construction of the cultural semantic alignment matrix. This involves constructing a cultural semantic alignment matrix to introduce cultural knowledge to guide attention allocation. ; In the formula, Represents cultural similarity, Represents the optimized visual and textual features. Represents the preset weight matrix. Represents the ReLU activation function. The cultural semantic alignment matrix represents the cultural similarity, which is calculated as follows: ; In the formula, The function representing the mapping from visual features to cultural contextual features. The mapping function represents the mapping from text features to cultural context features, where i and j represent the i-th position of the optimized visual feature and the j-th position of the optimized text feature, respectively. Step S522: Constructing the spatial semantic bias matrix. Based on the specific cultural meanings of different regions in the New Year picture, construct a region-based spatial semantic bias matrix for the New Year picture: ; In the formula, The sine code representing the image position (i,j) of the New Year picture. The classification vector representing the 9-square grid area of ​​the New Year picture. Represents spatial location weight. Represents the regional classification weight. Represents the spatial semantic bias matrix; Step S523: Cross-attention calculation: Construct a query vector based on the optimized visual features, construct key and value vectors based on the optimized text features, and calculate the cross-attention score. ; In the formula, These represent the query vector and the key vector, respectively. Represents the activation function. Represents the cross-attention score; Step S524: Cross-modal feature fusion. Align the optimized visual features and text features according to the cross-attention scores to obtain aligned visual features and aligned text features.

[0014] Example 6, this example is based on the above example, step S53 specifically includes the following steps: Step S531: Node construction. Construct visual nodes, where each node is aligned with a spatial location of a visual feature; construct text nodes, where each node corresponds to a word in the text sequence of the New Year picture; construct cultural context nodes, where each node corresponds to a cultural context dimension. Step S532: Edge construction, construct bidirectional edges between each pair of visual nodes, text nodes, and cultural context nodes; Step S533: Graph composition, all visual nodes, text nodes, cultural context nodes, and bidirectional edges constitute a heterogeneous graph; Step S534: Graph attention network processing. The heterogeneous graph is processed by a heterogeneous graph attention network to obtain multimodal fusion features.

[0015] Example 7: This example, based on the above examples, applies the above scheme to the style classification modeling of Yangliuqing New Year paintings and Taohuawu New Year paintings: Modeling background Yangliuqing New Year paintings and Taohuawu New Year paintings represent the typical styles of New Year paintings in the north and south, respectively. Traditional image classification methods are unable to capture the subtle differences between the two in terms of composition, color, brushstrokes and cultural connotations. Implementation steps Data collection: Collect 500 high-definition New Year pictures of Yangliuqing and Taohuawu, with metadata such as title, inscription, era, and place of origin attached; Visual feature extraction: Use ResNet-50 to extract composition features; Use DEPnet to extract the delicate brushstrokes of Yangliuqing New Year pictures and the woodblock textures of Taohuawu New Year pictures; Extract color histograms. Yangliuqing New Year pictures emphasize red, yellow, and green, while Taohuawu New Year pictures emphasize blue, cyan, and light colors; Through the style perception attention module, strengthen the "meticulous and heavy-color" style features of Yangliuqing and the "ink-wash and elegant" style features of Taohuawu; Text feature extraction: Identify auspicious words in New Year pictures (such as "Fu" [happiness], "Shou" [longevity], "年年有余" [more than enough every year]); Use CultureLLM to analyze their symbolic meanings (such as "fish" symbolizes "abundance"); Cultural context feature extraction: Construct a knowledge graph to associate "Yangliuqing New Year pictures → Tianjin → Influence of the Qing court" and "Taohuawu New Year pictures → Suzhou → Jiangnan folk customs"; Extract usage scenario features (such as Yangliuqing New Year pictures are mostly used as Spring Festival door gods, while Taohuawu are used for festival decorations); Cross-modal fusion and classification: Through the cultural semantic alignment matrix, align the "image area of the door god" with the text concept of "warding off evil spirits"; Use the heterogeneous graph attention network to fuse visual, text, and context features; Output classification labels: Yangliuqing school or Taohuawu school; Classification results The model achieves a genre classification accuracy of 94.2% on the test set, significantly higher than the baseline model that only uses visual features (78.5%).

[0016] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0017] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0018] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for modeling traditional Chinese New Year paintings based on multimodal feature fusion, characterized in that: Specifically, the following steps are included: Step S1: New Year picture data collection, which involves collecting a dataset of New Year pictures using big data analytics. Step S2: Visual feature extraction. Visual directional features are extracted from the New Year picture and multi-scale fusion optimization is performed to obtain visual features. Step S3: New Year picture text feature extraction. The text in the New Year picture is detected, recognized and its features are extracted to obtain the text features; Step S4: Cultural context feature extraction, extracting cultural context features related to the historical background, regional culture and usage scenarios of the New Year picture; Step S5: Processing based on heterogeneous graph cross-modal attention mechanism, visual features, text features and cultural context features are processed by heterogeneous graph cross-modal attention mechanism to obtain multimodal fusion features; Step S6: New Year picture classification modeling, activate and classify the multimodal fusion features, and assign classification labels to different New Year picture categories to obtain the classification model of New Year pictures.

2. The method for modeling traditional Chinese New Year paintings based on multimodal feature fusion according to claim 1, characterized in that: Step S5 specifically includes the following steps: Step S51: Intramodal feature self-enhancement, performing self-attention optimization on visual and text features; Step S52: Visual-text fine-grained alignment. The optimized visual features and text features interact with each other through an attention mechanism. Based on the dimension of cultural context features, a fine correspondence is established between pixel-level visual features and word-level text features. The semantic association between the image region and text concept of the New Year picture is captured, and the aligned visual features and aligned text features are obtained. Step S53: Heterogeneous graph attention network processing: Construct a heterogeneous graph from the multimodal features consisting of aligned visual features, aligned text features, and cultural context features, and process it using a heterogeneous graph attention network to obtain multimodal fusion features.

3. The method for modeling traditional Chinese New Year paintings based on multimodal feature fusion according to claim 2, characterized in that: Step S52 specifically includes the following steps: Step S521: Construction of the cultural semantic alignment matrix. This involves constructing a cultural semantic alignment matrix to introduce cultural knowledge to guide attention allocation. ； In the formula, Represents cultural similarity, Represents the optimized visual and textual features. Represents the preset weight matrix. Represents the ReLU activation function. The cultural semantic alignment matrix represents the cultural similarity, which is calculated as follows: ； In the formula, The function representing the mapping from visual features to cultural contextual features. The mapping function represents the mapping from text features to cultural context features, where i and j represent the i-th position of the optimized visual feature and the j-th position of the optimized text feature, respectively. Step S522: Constructing the spatial semantic bias matrix. Based on the specific cultural meanings of different regions in the New Year picture, construct a region-based spatial semantic bias matrix for the New Year picture: ； In the formula, The sine code representing the image position (i,j) of the New Year picture. The classification vector representing the 9-square grid area of ​​the New Year picture. Represents spatial location weight. Represents the regional classification weight. Represents the spatial semantic bias matrix; Step S523: Cross-attention calculation: Construct a query vector based on the optimized visual features, construct key and value vectors based on the optimized text features, and calculate the cross-attention score. ； In the formula, These represent the query vector and the key vector, respectively. Represents the activation function. Represents the cross-attention score; Step S524: Cross-modal feature fusion. Align the optimized visual features and text features according to the cross-attention scores to obtain aligned visual features and aligned text features.

Images