Multi-modal graph learning method oriented to oracle classification

Abstract

The invention relates to the technical field of oracle classification, and provides an oracle classification-oriented multi-modal graph learning method, which comprises the following steps of: 1, generating an LLM-based oracle pseudo-semantic text; generating structural description and cultural background interpretation of oracle fonts, and analyzing semantic information in the oracle single character rubbing image from two aspects; step 2, embedding dual-channel unified semantic multi-modal information features; on the basis of pre-trained ViT and BERT, in combination with multi-head attention calculation of an Oracle-QF module, image and text features are mapped to a shared 2560-D semantic space, and joint representation is formed through feature splicing; step 3, multi-modal feature fusion based on a heterogeneous graph neural network; a heterogeneous graph containing image nodes and text nodes is constructed, and cross-modal and same-modal feature fusion is realized through a multi-stage message passing mechanism; and step 4, dynamic feature splicing and classification. According to the method, the oracle can be better classified.

Description

Technical Field

[0001] This invention relates to the field of oracle bone script classification technology, and more specifically, to a multimodal graph learning method for oracle bone script classification. Background Technology

[0002] Oracle bone script, as the earliest existing mature Chinese character system, carries historical, cultural, and social information from the Shang and Zhou dynasties. Its study is of great value to linguistics, archaeology, and the exploration of the origins of Chinese civilization. However, the identification of oracle bone script has long faced multiple technical bottlenecks: on the one hand, due to differences in the material of oracle bones, the evolution of writing tools, and natural weathering, the characters of oracle bone script have undergone complex deformations and lack unified standards. At the same time, there is a scarcity of rubbings (the number of samples for a single category is generally less than one hundred). On the other hand, of the approximately 160,000 oracle bone fragments that exist, only 40% of the characters have been deciphered. The remaining undeciphered characters lack comparison with transmitted documents and rely heavily on annotations, resulting in low efficiency of traditional manual deciphering and susceptibility to subjective judgment.

[0003] With breakthroughs in artificial intelligence technology, especially the successful application of deep learning in image recognition, a new paradigm has been provided for oracle bone script research. Models based on convolutional neural networks (CNNs) and attention mechanisms have achieved breakthroughs in local feature modeling, but have not yet effectively solved the problem of semantic association modeling in small sample scenarios. In recent years, LLMs (such as Qwen-VL) have demonstrated powerful cross-modal reasoning capabilities, generating structural descriptions and cultural background explanations of oracle bone script characters, extracting semantic prior knowledge from oracle bone image data, supplementing the limitations of single-modal information, and alleviating the problem of insufficient feature extraction in small sample data. At the same time, graph neural network research has been gradually applied to the field of computer vision, breaking the traditional methods of feature extraction and feature aggregation, paying more attention to the structural information between features, and being able to model complex relationships between multimodal nodes (images, text), propagating semantic information through graph structures, and improving the feature generalization ability under small sample conditions.

[0004] However, current methods for recognizing and classifying oracle bone inscriptions still face the following problems:

[0005] (1) The bone image dataset lacks high-quality semantic annotations (such as stroke structure and cultural background). Traditional methods rely on expert knowledge for manual annotation, which is costly and has limited coverage, thus limiting the training of multimodal models.

[0006] (2) Images (visual features) and text (semantic description) are heterogeneous modalities, with semantic gaps and dimensional differences (such as ViT's 768-D vs. BERT's 768-D). Direct fusion is prone to information interference.

[0007] (3) Existing multimodal fusion methods (such as simple splicing or multi-head attention) are difficult to explicitly model the topological relationships and semantic associations of oracle bone script characters, resulting in insufficient cross-modal interaction.

[0008] (4) Oracle data is scarce and the categories are unevenly distributed, which can easily lead to model overfitting and affect the stability of practical applications. Summary of the Invention

[0009] The present invention provides a multimodal graph learning method for oracle bone script classification, which can overcome some or all of the defects of the prior art.

[0010] The multimodal graph learning method for oracle bone script classification according to the present invention includes the following steps:

[0011] Step 1: Oracle pseudo-semantic text generation based on LLM;

[0012] The Qwen-VL-Plus model is driven by Prompt Engineering to generate structural descriptions and cultural background interpretations of oracle bone script characters; the semantic information in rubbings of individual oracle bone characters is analyzed from two aspects: micro-information—structural analysis and macro-information—background semantic analysis.

[0013] Step 2: Embedding dual-channel unified semantic multimodal information features;

[0014] Based on the pre-trained Vision Transformer and BERT, and combined with the multi-head attention computation of the Oracle-QF module, image and text features are mapped to a shared 2560-D semantic space, and a joint representation is formed by feature concatenation, providing input for subsequent heterogeneous graph modeling;

[0015] Step 3: Multimodal feature fusion based on heterogeneous graph neural networks;

[0016] Construct a heterogeneous graph containing image nodes and text nodes, and achieve cross-modal and same-modal feature fusion through a multi-level message passing mechanism;

[0017] Step 4: Dynamic feature splicing and classification.

[0018] Preferably, in step 1, the Qwen-VLPlus model is used to call the API interface and the QWENAPI interface to analyze the semantic information in the rubbings of oracle bone characters from two aspects; considering the graphic characteristics of oracle bone characters, a Prompt guided by pictographic priors is designed:

[0019]

[0020] Among them, φstruct represents structural analysis hints, which are used to describe the character structural features of oracle bone script, including stroke shape, spatial layout and topological structure; φscmantic represents background semantic analysis hints, which are used to explain the historical connotation of the character, including possible ancient usage scenarios and possible semantic changes in the evolution of the character.

[0021] As a preferred option, step 2 specifically involves:

[0022] 2.1) Image feature extraction;

[0023] First, for oracle bone inscription rubbings, the ViT-B model is used to extract initial visual features. The input image is divided into 9×9 pixel blocks, and after passing through an embedding layer and a Transformer backbone, patch features are generated.

[0024] Subsequently, the Oracle-QF module introduced learnable query vectors. Image features strongly correlated with the text description are selected using a multi-head cross-attention mechanism; the query vector interacts with the patch features output by ViT, and the calculation formula is as follows:

[0025] A img =MultiHeadAttn(Q img ,K img V img (2)

[0026] Where K img =W k V, V img =W v V represents the key and value after linear projection, W... k W v Let V be an independent weight matrix, and V be the image patch features output by ViT-B. This represents the semantically strongly correlated image features output by the 32-head attention mechanism;

[0027] Finally, through global average pooling and linear mapping, semantically strongly correlated image features are compressed to a 2560-D space:

[0028]

[0029] in, This is the projection matrix, ensuring that the dimensions of the features are consistent with those of the text modality; Image features; b img Represents the offset vector;

[0030] 2.2) Text feature extraction;

[0031] For text features, the Oracle-QF module uses the BERT-wwm-chinese pre-trained model to process the oracle bone script description text and generate token-level features. L is the text length; similarly, the query vector can be learned. Interacting with BERT features, the semantic information related to image modality is enhanced through a multi-head attention mechanism. The specific calculation is as follows:

[0032]

[0033] Among them, K txt =W k 'T、V txt =W v 'T' represents the key and value of the text feature, W k ′、W v ′ is the independent weight matrix, and T is the initial text-level feature output by the BERT pre-trained model; For attention output; text features are mapped to a 2560-D space through a linear layer:

[0034]

[0035] in, Ensure that text features and image features share the same semantic space; For text features, b txt This is the offset vector.

[0036] As a preferred option, step 3 specifically involves:

[0037] 3.1) Heterogeneous graph structure modeling and message passing;

[0038] The strategy for constructing heterogeneous graphs is as follows:

[0039] Homomodal neighborhood generation: Image features from Oracle-QF output and text features Calculate pairwise similarity matrices and And retain the top k=5 nearest neighbors for each node;

[0040] Cross-modal mapping: Explicitly models intermodal relationships through a one-to-one correspondence, forming an edge index E. cross ={(i,i)|i=1,2,...,N};

[0041] Same-modal edges:

[0042] image-to-image: Based on cosine similarity, it connects oracle bone script image nodes with similar stroke layouts;

[0043] text-to-text: Based on the semantic similarity of text descriptions, it connects text nodes related to cultural background;

[0044] Cross-modal edge:

[0045] image-to-text: A one-to-one mapping that connects each image node to its corresponding text node;

[0046] text-to-image: Inverse mapping, which guides image features to text features;

[0047] 3.2) Heterogeneous graph message passing mechanism;

[0048] HeteroConv is used to define differentiated graph convolution operations for different edge types, as shown in the following formula:

[0049]

[0050] Among them, h i (l+1) Let h be the new feature vector updated at layer l+1 after the central node i has received and processed the information from all its neighboring nodes. j (l) Let R be the feature vector of a neighbor node j of node i in the l-th layer of a heterogeneous graph neural network, and let N be the set of edge types. r (i) Let i be a neighboring node of node i under relation r. For attention weights, W r (l) Let σ be the learnable parameter matrix, and σ be the ReLU activation function.

[0051] ① Same-modal propagation;

[0052] Between image nodes: Features of similar images are aggregated using GATConv, with the following formula:

[0053]

[0054] in, Calculated using learnable attention coefficients, emphasizing the similarity of local stroke layouts; img indicates that the node is an image node. h represents the updated feature vector of the i-th image node after computation by the GATConv layer. img,j N represents the original feature vector of neighbor node j. img (i) represents the set of neighboring nodes of the i-th image node;

[0055] Between text nodes: Similarly, GATConv is used to aggregate semantically related text features, with the following formula:

[0056]

[0057] This represents the updated feature vector of the i-th text node after computation by the GATConv layer. N represents the attention coefficient between the central text node i and its neighboring text nodes j. The higher the attention coefficient value, the greater the weight of the features of neighbor j when updating node i. txt (i) represents the set of neighboring nodes of the i-th text node, h txt,j The original feature vector of the neighboring text node j represents the semantic information carried by the neighboring node j.

[0058] ② Cross-modal propagation

[0059] Image node to text node: Passing information using SAGEConv:

[0060] Enhance the ability of text nodes to perceive image features; W represents the updated feature vector of the i-th text node after receiving information from the image node. img→txt N represents a learnable weight matrix used to process information passed from image nodes to text nodes. img→txt (i) represents the set of image neighbor nodes used to update the i-th text node;

[0061] Text node to image node: Backpropagation, the formula is:

[0062]

[0063] Enhancing image features by leveraging semantic information from textual descriptions.

[0064] As a preferred option, step 4 specifically involves:

[0065] 4.1) Feature integration;

[0066] Image features output by heterogeneous image convolution Original image features generated by Oracle-QF By combining the elements, a joint representation can be formed:

[0067]

[0068] Where ⊕ represents the vector concatenation operation; z img This represents the final joint image representation; LayerNorm represents feature fusion and normalization.

[0069] 4.2) Multimodal fusion under gating mechanism;

[0070] Introducing a text gating mechanism to dynamically adjust the importance of image features:

[0071] Text gate signal g txt generate:

[0072] g txt =Sigmoid(Linear(t) txt (12)

[0073] in, Text characteristics for Oracle-QF output.

[0074] Image feature enhancement:

[0075] f img =v img ·(1+g txt (13)

[0076] Through the gating signal g txt Selectively enhance image features that are strongly correlated with text descriptions;

[0077] Final assembly and classification: Convolve the heterogeneous graphs to output z. img With the gating enhanced feature f img Concatenate the data and input it into the classifier:

[0078]

[0079] y pred =Softmax(Linear(z) final (15)

[0080] z final This represents the final feature vector after complete fusion and integration before being fed into the final classifier; y pred This indicates the final predicted output.

[0081] The beneficial effects of this invention are as follows:

[0082] I. This invention is the first to introduce a multimodal large language model (LLM) into the field of oracle bone script recognition, proposing an automated text generation method. By driving the Qwen-VL-Plus model through structured prompt engineering, it generates structural descriptions and cultural background explanations of oracle bone script characters, overcoming the bottleneck of traditional reliance on manual annotation. This method, through a two-stage instruction design, forces the model to output separate descriptions of visual structure (e.g., stroke layout) and semantic connotation (e.g., historical usage), significantly improving the diversity and professionalism of text features. The generated pseudo-text not only provides semantically enhanced input for subsequent feature extraction but can also be transferred to cultural heritage tasks such as bronze inscriptions and bamboo and silk manuscripts, providing a low-cost, high-efficiency data augmentation path for ancient script research.

[0083] II. To address the semantic gap between oracle bone inscription images and text, this invention designs the Oracle-QF module, replicating the Q-Former concept of BLIP-2. It features lightweight modifications to the ViT and BERT architectures, enabling dynamic filtering of image and text features and the construction of a unified semantic space. This module dynamically captures the correlation between complex strokes and cultural keywords in oracle bone inscriptions through a multi-head attention mechanism, mapping 768-D ViT / BERT features to a 2560-D space, and ensuring semantic consistency through a normalization strategy. Its core innovations lie in: utilizing learnable query vectors to achieve fine-grained alignment of cross-modal features, significantly outperforming traditional splicing methods; and enhancing the model's perception of key stroke regions and cultural background through a 32-head attention mechanism, providing a more accurate interaction path for cross-modal fusion.

[0084] III. This invention constructs a heterogeneous graph containing image nodes, text nodes, and various edge types. It explicitly models cross-modal interactions through a multi-level message passing mechanism, overcoming the limitation of traditional multimodal fusion methods that ignore complex intermodal relationships. The heterogeneous graph defines same-modal edges (e.g., image-to-image) and cross-modal edges (e.g., image-to-text), employing differentiated convolution operations: GATConv strengthens common patterns of similar glyphs through attention weights, while SAGEConv achieves bidirectional guidance of image and text information through mean pooling. Furthermore, a text gating mechanism is introduced to dynamically adjust image feature weights, ensuring accurate enhancement of key regions. This strategy not only solves the problem of ambiguous image-text interactions in traditional methods but also significantly improves classification performance through dual optimization at the structural and feature levels, providing a reusable technical paradigm for cross-modal modeling.

[0085] IV. To address the scarcity and class imbalance of oracle bone script data, this invention proposes a graph-based sample augmentation strategy and constructs a modular technical framework to support the digitization of cultural heritage. Cosine similarity is used to dynamically generate congruent neighborhoods, leveraging neighbor node information to alleviate the generalization bottleneck of rare character forms. Regularization strategies are introduced in the classifier design to reduce overfitting risk. The modular framework supports flexible replacement of core components (such as ResNet image encoders or RoBERT text encoders) to adapt to different cultural heritage tasks. Furthermore, by expanding edge types (such as "image-historical background"), a cross-modal knowledge graph can be constructed, supporting cross-modal retrieval (such as searching for related oracle bone script images by inputting "sacrifice"). This design not only provides a robust solution for oracle bone script recognition but also offers a general technical path for tasks involving bronze inscriptions and bamboo slips, promoting the application of artificial intelligence in cultural heritage protection. Attached Figure Description

[0086] Figure 1 This is a flowchart illustrating a multimodal graph learning method for oracle bone script classification in this embodiment.

[0087] Figure 2 This is a diagram illustrating the framework for generating pseudo-semantic text in oracle bone script in the embodiment.

[0088] Figure 3 This is an example of random oracle bone script characters in the embodiment;

[0089] Figure 4 This is a schematic diagram of image feature extraction in the embodiment;

[0090] Figure 5 This is a schematic diagram of the Oracle-QF module in the embodiment. Detailed Implementation

[0091] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0092] Example

[0093] like Figure 1 As shown, this embodiment provides a multimodal graph learning method for oracle bone script classification, which includes the following steps:

[0094] Step 1: Oracle pseudo-semantic text generation based on LLM;

[0095] The oracle bone rubbing dataset used in this embodiment was collected from the "Global Oracle Bone Multimodal Dataset" released by the "Digital Oracle Bone Co-creation Center". This dataset contains rubbings and copies of 10,000 oracle bones. This embodiment extracted the key fields.

[0096] To address the issue of poor feature extraction performance of traditional deep learning due to the long-tail distribution of the oracle bone script rubbings dataset in this embodiment, this embodiment designs a pseudo-semantic text generation framework for oracle bone script rubbings based on LLM, such as... Figure 2 As shown. The structured prompt engineering (Prompt Engineering) drives the Qwen-VL-Plus model to generate structural descriptions and cultural background explanations of oracle bone script characters. The Qwen-VL-Plus model is used to call API interfaces and QWENAPI interfaces to analyze the semantic information in oracle bone script rubbings from two aspects: micro-information—structural analysis (φstruct) and macro-information—contextual semantic analysis (φscmantic). Considering the characteristics of oracle bone script images, a prompt guided by pictographic priors is designed.

[0097]

[0098] Here, φstruct represents structural analysis hints, used to describe the character structural features of oracle bone script, including stroke shape, spatial layout, and topological structure; φscmantic represents background semantic analysis hints, used to explain the historical connotations of the characters, including possible ancient usage scenarios and possible semantic changes in the evolution of the characters. This design of the hint template transforms the oracle bone script research paradigm into machine-understandable instructions, ensuring that the generated "image-text pair" content meets the requirements of this embodiment.

[0099] An example of the generation result from a random rubbing of oracle bone inscriptions is as follows: Figure 3 As shown in Table 1.

[0100]

[0101]

[0102] Step 2: Embedding dual-channel unified semantic multimodal information features;

[0103] This embodiment proposes a unified semantic multimodal feature embedding method for oracle bone classification tasks. Its core lies in designing a dedicated query vector module (Oracle-QF) to achieve deep alignment of image and text features through a dynamic cross-modal attention mechanism. Based on pre-trained Vision Transformer (ViT) and BERT, this method combines multi-head attention computation from the Oracle-QF module to map image and text features to a shared 2560-D semantic space. Furthermore, it forms a joint representation through feature concatenation, providing input for subsequent heterogeneous graph modeling.

[0104] Specifically:

[0105] 2.1) Image feature extraction;

[0106] First, for the oracle bone rubbing images, the ViT-B model is used to extract initial visual features, such as... Figure 4 As shown, the input image is divided into 9×9 pixel blocks, and after passing through the embedding layer and the Transformer backbone, patch features are generated.

[0107] Subsequently, the Oracle-QF module introduced learnable query vectors. Image features strongly correlated with the text description are selected using a multi-head cross-attention mechanism; the query vector interacts with the patch features output by ViT, such as... Figure 5 As shown, the calculation formula is:

[0108] A img =MultiHeadAttn(Q img ,K img V img (2)

[0109] Where K img =W k V, V img =W v V represents the key and value after linear projection, W k W v Let V be an independent weight matrix, and V be the image patch features output by ViT-B. This represents the semantically strongly correlated image features output by the 32-head attention mechanism;

[0110] Finally, through global average pooling and linear mapping, semantically strongly correlated image features are compressed to a 2560-D space:

[0111]

[0112] in, This is the projection matrix, ensuring that the dimensions of the features are consistent with those of the text modality. Image features; b img This represents the offset vector.

[0113] 2.2) Text feature extraction;

[0114] For text features, the Oracle-QF module uses the BERT-wwm-chinese pre-trained model to process the oracle bone script description text and generate token-level features. L is the text length; similarly, the query vector can be learned. Interacting with BERT features, the semantic information related to image modality is enhanced through a multi-head attention mechanism. The specific calculation is as follows:

[0115] A txt =MultiHeadAttn(Q txt ,K txt V txt (4)

[0116] Among them, K txt =W k 'T、V txt =W v 'T' represents the key and value of the text feature, W k ′、W v ′ is the independent weight matrix, and T is the initial text-level feature output by the BERT pre-trained model; For attention output; text features are mapped to a 2560-D space through a linear layer:

[0117]

[0118] in, Ensure that text features and image features share the same semantic space; For text features, b txt This is the offset vector.

[0119] The core innovation of the Oracle-QF module lies in its cross-modal alignment capability: through a 32-head attention mechanism, the model can simultaneously focus on the local strokes of oracle bone script characters (such as "horizontal stroke with a dot") and the cultural background in the text description (such as "sacrificial scene"), thereby dynamically selecting the features most relevant to the task. For example, when the text description emphasizes "bronze inscriptions," Oracle-QF will prioritize activating texture regions in the image related to metallic texture; while when the image presents radial strokes (such as the character "rain"), the query vector on the text side will strengthen the response to keywords such as "rain" and "weather" in the description. This bidirectional interaction mechanism significantly improves the semantic consistency of image and text features.

[0120] Step 3: Multimodal feature fusion based on heterogeneous graph neural networks;

[0121] After completing the unified semantic feature embedding based on Oracle-QF, this embodiment constructs a heterogeneous graph containing image nodes and text nodes, and realizes cross-modal and same-modal feature fusion through a multi-level message passing mechanism.

[0122] Specifically:

[0123] 3.1) Heterogeneous graph structure modeling and message passing;

[0124] The strategy for constructing heterogeneous graphs is as follows:

[0125] Homomodal neighborhood generation: Image features from Oracle-QF output and text features Calculate pairwise similarity matrices and And retain the top k=5 nearest neighbors for each node;

[0126] Cross-modal mapping: Explicitly modeling intermodal relationships through one-to-one correspondence (e.g., ), forming edge index E cross ={(i,i)|i=1,2,...,N};

[0127] Same-modal edges:

[0128] image-to-image: Based on cosine similarity, it connects oracle bone script image nodes with similar stroke layouts;

[0129] text-to-text: Based on the semantic similarity of text descriptions, it connects text nodes related to cultural background;

[0130] Cross-modal edge:

[0131] image-to-text: A one-to-one mapping that connects each image node to its corresponding text node;

[0132] text-to-image: Inverse mapping, which guides image features to text features;

[0133] 3.2) Heterogeneous graph message passing mechanism;

[0134] HeteroConv is used to define differentiated graph convolution operations for different edge types, as shown in the following formula:

[0135]

[0136] Among them, h i (l+1) Let h be the new feature vector updated at layer l+1 after the central node i has received and processed the information from all its neighboring nodes. j (l) Let R be the feature vector of a neighbor node j of node i in the l-th layer of a heterogeneous graph neural network, and let N be the set of edge types. r (i) Let i be a neighboring node of node i under relation r. For attention weights, W r (l) Let σ be the learnable parameter matrix, and σ be the ReLU activation function.

[0137] ① Same-modal propagation;

[0138] Between image nodes: Features of similar images are aggregated using GATConv (multi-head attention mechanism), with the following formula:

[0139]

[0140] in, Calculated using learnable attention coefficients, emphasizing the similarity of local stroke layouts; img indicates that the node is an image node. h represents the updated feature vector of the i-th image node after computation by the GATConv layer. img,j N represents the original feature vector of neighbor node j. img (i ) This represents the set of neighboring nodes of the i-th image node;

[0141] Between text nodes: Similarly, GATConv is used to aggregate semantically related text features (such as text nodes describing "sacrifice" and "bronze ware"), using the following formula:

[0142]

[0143] This represents the updated feature vector of the i-th text node after computation by the GATConv layer. N represents the attention coefficient between the central text node i and its neighboring text nodes j. The higher the attention coefficient value, the greater the weight of the features of neighbor j when updating node i. txt (i) represents the set of neighboring nodes of the i-th text node, h txt,j The original feature vector of the neighboring text node j represents the semantic information carried by the neighboring node j.

[0144] ② Cross-modal propagation

[0145] Image node to text node: Information is passed using SAGEConv (mean pooling):

[0146]

[0147] Enhance the ability of text nodes to perceive image features; W represents the updated feature vector of the i-th text node after receiving information from the image node. img→txt N represents a learnable weight matrix used to process information passed from image nodes to text nodes. img→txt (i) represents the set of image neighbor nodes used to update the i-th text node;

[0148] Text node to image node: Backpropagation, the formula is:

[0149]

[0150] The semantic information of the text description can be used to guide the enhancement of image features (such as the description of "horizontal stroke with dots" to enhance the features of the corresponding stroke area in the image).

[0151] Step 4: Dynamic feature assembly and classification;

[0152] Specifically:

[0153] 4.1) Feature integration;

[0154] Image features output by heterogeneous image convolution Original image features generated by Oracle-QF By combining the elements, a joint representation can be formed:

[0155]

[0156] Where ⊕ represents the vector concatenation operation; the dimension after concatenation is 512 + 2560 = 3072 - D. img The final joint image representation is represented by LayerNorm; LayerNorm represents feature fusion and normalization.

[0157] 4.2) Multimodal fusion under gating mechanism;

[0158] Introducing a text gating mechanism to dynamically adjust the importance of image features:

[0159] Text gating signal generation:

[0160] g txt =Sigmoid(Linear(t) txt (12)

[0161] in, Text characteristics for Oracle-QF output.

[0162] Image feature enhancement:

[0163] f img =v img ·(1+g txt (13)

[0164] Through the gating signal g txt Selectively enhance image features that are strongly correlated with the text description (e.g., emphasize the image region corresponding to the description of "sunrise");

[0165] Final assembly and classification: Convolve the heterogeneous graphs to output z. img With the gating enhanced feature f img Concatenate the data and input it into the classifier:

[0166]

[0167] y pred =Softmax(Linear(z) final )) (15)z final This represents the final feature vector after complete fusion and integration before being fed into the final classifier; y pred This indicates the final predicted output.

[0168] Technology maturity and transferability

[0169] (1) Feasibility of LLM in Generating Pseudo-Text: Qwen-VL-Plus, an open-source multimodal large model from Alibaba Cloud, has been widely used in image-text generation tasks. Its ability to pre-train on large-scale image-text corpora enables it to directly generate structural descriptions and cultural background explanations of oracle bone script images, adapting to the oracle bone script domain without additional fine-tuning. Experiments show that the generated text descriptions, guided by structured instructions (such as separating "stroke layout" from "cultural connotation"), can provide high-quality semantic information (such as "horizontal stroke with dots" and "sacrificial scene"), providing reliable input for subsequent feature extraction.

[0170] (2) Technical Basis of BLIP-2 and Heterogeneous Graph Neural Networks: The BLIP-2 model referenced in this embodiment, as a classic architecture for image-text alignment, has been verified to have excellent performance in multimodal tasks (such as image-text retrieval). Heterogeneous Graph Neural Networks (HeteroGNN) are an extension of Graph Neural Networks (GNN), supporting the modeling of multiple types of nodes (such as images and text) and edges (same-modal and cross-modal), and have been maturely applied in recommendation systems, knowledge graphs, and other fields. This embodiment further enhances the model's ability to capture the topological relationships and semantic associations of oracle bone inscriptions by combining HeteroConv with multi-level convolutional operations of GATConv and SAGEConv.

[0171] Data adaptability and robustness

[0172] (1) Adaptability in small sample scenarios: Oracle bone script datasets typically face the problems of limited sample size (thousands of images) and class imbalance (some glyphs appear with extremely low frequency). This solution improves robustness through the following strategies:

[0173] ① Pseudo-text generation: LLM is used to extend semantic annotation (such as the description of "bronze inscriptions") to alleviate the bottleneck of high manual annotation costs;

[0174] ② Neighborhood enhancement: When constructing heterogeneous graphs, homomodal neighborhoods (k=5 nearest neighbors) are dynamically generated based on cosine similarity, and the generalization ability of small samples is enhanced through information propagation from neighboring nodes;

[0175] ③Regularization design: Dropout(0.5), BatchNorm, and Label Smoothing are introduced into the classifier to reduce the risk of overfitting.

[0176] (2) Stability of multimodal feature alignment: The Oracle-QF module compresses the 768-D features of ViT (image) and BERT (text) into a unified 2560-D space through a 32-head attention mechanism, and ensures the consistency of feature distribution through L2 normalization. Experiments show that the module achieves a fusion accuracy of 92.3% on the oracle bone dataset, significantly better than the traditional splicing method (85.7%).

[0177] Rationality of the model structure

[0178] (1) Scientificity of heterogeneous graph modeling: The edge type design of the heterogeneous graph fully considers the characteristics of the oracle bone recognition task:

[0179] ① Intra-modal edges: Through the attention weights of GATConv, capture the commonalities of the stroke layouts of similar glyphs (such as the radial strokes of "rain" and "cloud");

[0180] ② Cross-modal edges: Through the mean pooling mechanism of SAGEConv, realize the guidance of text descriptions on image features (such as the description of "sacrifice" enhancing the feature weights of corresponding symbols);

[0181] ③ Gating mechanism: The text gating signal generates dynamic weights through the Sigmoid function to ensure that the enhancement of image features is strongly correlated with text semantics.

[0182] (2) Verification of model interpretability: Through attention heatmaps and t-SNE visualization, it is found that the model shows significant attention to the key strokes in oracle bone rubbings (such as the circular structure of the character "sun") and cultural keywords in text descriptions (such as "bronze inscriptions"). This feature not only verifies the decision-making basis of the model but also provides an interpretable auxiliary analysis tool for oracle bone experts.

[0183] This embodiment proposes an oracle bone image pseudo-semantic text generation method based on a large language model (LLM). By calling the Qwen-VL-Plus multimodal large model, a structured semantic description is generated for each oracle bone single-character rubbing image. This method aims to solve the problem of scarce manually annotated semantic information in the oracle bone image dataset. By automatically generating "pseudo-semantic" texts (i.e., descriptions that simulate expert knowledge using the historical oracle bone data trained by the large model), an "image-text pair" dataset of oracle bones is constructed to provide high-quality text input for subsequent multimodal feature extraction and heterogeneous graph modeling. This embodiment has the following contributions:

[0184] (1) It solves the problems of data scarcity and long-tailed distribution of datasets. Traditional oracle bone image interpretation and annotation rely on expert knowledge, which is costly. By automatically generating structured descriptions through multimodal large models, the need for manual annotation is significantly reduced, and large-scale dataset construction is supported.

[0185] (2) The semantic information of the Oracle Bone Script dataset was enhanced, and the generated text description contained visual structure and cultural background information, providing high-quality input for subsequent text feature extraction.

[0186] (3) It is portable and extensible, and the method can be extended to semantic enhancement tasks of other ancient scripts. At the same time, the accuracy and professionalism of the generated text can be further improved by adjusting the instruction format (such as adding Few-Shot examples).

[0187] This embodiment draws inspiration from the bimodal feature embedding method of the BLIP-2 model. By decoupling the encoding paths of Oracle images and text, it constructs a dedicated Oracle Query Vector Module (Oracle-QF) to map different modalities (text and images) in the Oracle dataset to the same vector space, solving the feature heterogeneity problem and constructing a multimodal feature representation under a unified semantic space. The core idea of ​​this method is to design a dedicated query vector module (Oracle-QF) and achieve deep alignment of image and text features through a dynamic cross-modal attention mechanism. Based on pre-trained VisionTransformer (ViT) and BERT, this method combines the multi-head attention computation of the Oracle-QF module to map image and text features to a shared 2560-D semantic space, and forms a joint representation through feature concatenation, providing input for subsequent heterogeneous graph modeling.

[0188] This embodiment proposes a multimodal feature fusion method based on heterogeneous graph neural networks, aiming to solve the semantic gap between image and text modalities in oracle bone script recognition tasks. Its core modeling idea is to construct a heterogeneous graph containing image and text nodes, explicitly modeling local neighborhood relationships within the same modality and semantic associations across modalities, and combining this with a dynamic gating mechanism to achieve synergistic enhancement of image and text features. Specifically, the heterogeneous graph defines three edge types—same-modal edges (image-to-image, text-to-text) are used to capture local topological relationships of similar glyphs or cultural backgrounds, while cross-modal edges (image-to-text, text-to-image) achieve image-text semantic alignment through one-to-one mapping. Based on this, differentiated graph convolution operations (the attention mechanism of GATConv and the mean pooling of SAGEConv) are used for multi-level message passing and aggregation. This method overcomes the limitation of traditional multimodal fusion methods that ignore complex interactions between modalities through heterogeneous graph modeling, and its technical advantages are reflected in:

[0189] (1) Structure perception ability: The topological modeling of heterogeneous graphs not only preserves the component combination rules of oracle bone script, but also significantly improves classification performance by explicitly associating cultural background (such as "bronze inscriptions") across modal edges;

[0190] (2) Dynamic fusion efficiency: The gating mechanism achieves adaptive alignment of image and text features through learnable weights;

[0191] (3) High adaptability: By enhancing the generalization ability of small samples through neighborhood information, a technical path is provided for data-scarce scenarios such as oracle bone inscriptions. This design provides a high-quality joint representation foundation for subsequent classification tasks, and at the same time provides a transferable framework for the digital preservation of ancient scripts.

[0192] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A multimodal graph learning method for oracle bone script classification, characterized by: Includes the following steps: Step 1: Oracle pseudo-semantic text generation based on LLM; The Qwen-VL-Plus model is driven by Prompt Engineering to generate structural descriptions and cultural background interpretations of oracle bone script characters; the semantic information in rubbings of individual oracle bone characters is analyzed from two aspects: micro-information—structural analysis and macro-information—background semantic analysis. Step 2: Embedding dual-channel unified semantic multimodal information features; Based on the pre-trained Vision Transformer and BERT, and combined with the multi-head attention computation of the Oracle-QF module, image and text features are mapped to a shared 2560-D semantic space, and a joint representation is formed by feature concatenation, providing input for subsequent heterogeneous graph modeling; Step 3: Multimodal feature fusion based on heterogeneous graph neural networks; Construct a heterogeneous graph containing image nodes and text nodes, and achieve cross-modal and same-modal feature fusion through a multi-level message passing mechanism; Step 4: Dynamic feature splicing and classification.

2. The multimodal graph learning method for oracle bone script classification according to claim 1, characterized in that: In step 1, the Qwen-VLPlus model is used to call the API interface and the QWENAPI interface to parse the semantic information in the rubbings of oracle bone characters from two aspects; considering the graphic characteristics of oracle bone characters, a prompt guided by pictographic priors is designed: Among them, φstruct represents structural analysis hints, which are used to describe the character structural features of oracle bone script, including stroke shape, spatial layout and topological structure; φscmantic represents background semantic analysis hints, which are used to explain the historical connotation of the character, including possible ancient usage scenarios and possible semantic changes in the evolution of the character.

3. The multimodal graph learning method for oracle bone script classification according to claim 2, characterized in that: Step 2 specifically involves: 2.1) Image feature extraction; First, for oracle bone rubbings, the ViT-B model is used to extract initial visual features. The input image is divided into 9×9 pixel blocks, and after passing through an embedding layer and a Transformer backbone, patch features are generated. Subsequently, the Oracle-QF module introduced learnable query vectors. Image features strongly correlated with the text description are selected using a multi-head cross-attention mechanism; the query vector interacts with the patch features output by ViT, and the calculation formula is as follows: And img =MultiHeadAttn(Q img ,K img ,V img ), (2) Where K img =W k V, V img =W v V represents the key and value after linear projection, W... k W v Let V be an independent weight matrix, and V be the image patch features output by ViT-B. This represents the semantically strongly correlated image features output by the 32-head attention mechanism; Finally, through global average pooling and linear mapping, semantically strongly correlated image features are compressed to a 2560-D space: in, This is the projection matrix, ensuring that the dimensions of the features are consistent with those of the text modality; Image features; b img Represents the offset vector; 2.2) Text feature extraction; For text features, the Oracle-QF module uses the BERT-wwm-chinese pre-trained model to process the oracle bone script description text and generate token-level features. L is the text length; similarly, the query vector can be learned. Interacting with BERT features, the semantic information related to image modality is enhanced through a multi-head attention mechanism. The specific calculation is as follows: And txt =MultiHeadAttn(Q txt ,K txt ,V txt ), (4) Among them, K txt =W k 'T、V txt =W v 'T' represents the key and value of the text feature, W k ′、W v ′ is the independent weight matrix, and T is the initial text-level feature output by the BERT pre-trained model; For attention output; text features are mapped to a 2560-D space through a linear layer: in, Ensure that text features and image features share the same semantic space; For text features, b txt This is the offset vector.

4. The multimodal graph learning method for oracle bone script classification according to claim 3, characterized in that: Step 3 specifically involves: 3.1) Heterogeneous graph structure modeling and message passing; The strategy for constructing heterogeneous graphs is as follows: Homomodal neighborhood generation: Image features from Oracle-QF output and text features Calculate pairwise similarity matrices and And retain the top k=5 nearest neighbors for each node; Cross-modal mapping: Explicitly models intermodal relationships through a one-to-one correspondence, forming an edge index E. cross ={(i,i)|i=1,2,...,N}; Same-modal edges: image-to-image: Based on cosine similarity, it connects oracle bone script image nodes with similar stroke layouts; text-to-text: Based on the semantic similarity of text descriptions, it connects text nodes related to cultural background; Cross-modal edge: image-to-text: A one-to-one mapping that connects each image node to its corresponding text node; text-to-image: Inverse mapping, which guides image features to text features; 3.2) Heterogeneous graph message passing mechanism; HeteroConv is used to define differentiated graph convolution operations for different edge types, as shown in the following formula: in, This represents the new feature vector updated by the central node i at layer l+1 after receiving and processing information from all its neighboring nodes. Let R be the feature vector of a neighbor node j of node i in the l-th layer of a heterogeneous graph neural network, and let R be the set of edge types. Let i be a neighboring node of node i under relation r. For attention weights, Let σ be the learnable parameter matrix, and σ be the ReLU activation function. ① Same-modal propagation; Between image nodes: Features of similar images are aggregated using GATConv, with the following formula: in, Calculated using learnable attention coefficients, emphasizing the similarity of local stroke layouts; img indicates that the node is an image node. h represents the updated feature vector of the i-th image node after computation by the GATConv layer. img,j N represents the original feature vector of neighbor node j. img (i) represents the set of neighboring nodes of the i-th image node; Between text nodes: Similarly, GATConv is used to aggregate semantically related text features, with the following formula: This represents the updated feature vector of the i-th text node after computation by the GATConv layer. N represents the attention coefficient between the central text node i and its neighboring text nodes j. The higher the attention coefficient value, the greater the weight of the features of neighbor j when updating node i. txt (i ) Let h represent the set of neighboring nodes of the i-th text node. txt,j The original feature vector of the neighboring text node j represents the semantic information carried by the neighboring node j. ② Cross-modal propagation Image node to text node: Passing information using SAGEConv: Enhance the ability of text nodes to perceive image features; W represents the updated feature vector of the i-th text node after receiving information from the image node. img→txt N represents a learnable weight matrix used to process information passed from image nodes to text nodes. img→txt (i) represents the set of image neighbor nodes used to update the i-th text node; Text node to image node: Backpropagation, the formula is: Enhancing image features by leveraging semantic information from textual descriptions.

5. The multimodal graph learning method for oracle bone script classification according to claim 4, characterized in that: Step 4 specifically involves: 4.1) Feature integration; Image features output by heterogeneous image convolution Original image features generated by Oracle-QF By combining the elements, a joint representation can be formed: in, This represents a vector concatenation operation; z img This represents the final joint image representation; LayerNorm represents feature fusion and normalization. 4.2) Multimodal fusion under gating mechanism; Introducing a text gating mechanism to dynamically adjust the importance of image features: Text gate signal g txt generate: g txt =Sigmoid(Linear(t txt )), (12) in, Text characteristics for Oracle-QF output. Image feature enhancement: f img =v img ·(1+g txt ) (13) Through the gating signal g txt Selectively enhance image features that are strongly correlated with text descriptions; Final assembly and classification: Convolve the heterogeneous graphs to output z. img With the gating enhanced feature f img Concatenate the data and input it into the classifier: and pred =Softmax(Linear(z final )) (15) z final This represents the final feature vector after complete fusion and integration before being fed into the final classifier; y pred This indicates the final predicted output.

Images