Method and device for recognizing oracle bone inscriptions based on hierarchical structure and multi-modal fusion, and medium

Abstract

This invention relates to a method, apparatus, and medium for recognizing long-tailed oracle bone script based on hierarchical structure and multimodal fusion. The method includes: constructing an oracle bone script image dataset; generating a superclass-subclass tree structure of oracle bone script using a self-supervised model combined with the Ward linking criterion; generating visual descriptions of each oracle bone script category using a large language model; constructing a pure visual hierarchical recognition model, constraining the superclass-subclass prediction logic through a hierarchical consistency loss function, and filtering irrelevant fine-grained categories using a winner-takes-all strategy; constructing a multimodal recognition model, freezing the parameters of the pure visual hierarchical recognition model, obtaining the superclass prediction result and retrieving the corresponding visual description from the multimodal knowledge base, inputting the input image, task instructions, and retrieval results introduced based on a random corpus injection strategy into the multimodal recognition model, and outputting the subclass classification result; obtaining the image to be recognized, inputting it into the pure visual hierarchical recognition model to obtain the superclass prediction result, and then inputting it along with the task instructions into the multimodal recognition model to obtain the subclass prediction result. Compared with the prior art, this invention has the advantages of effectively solving the problems of extreme long-tailed distribution, feature space collapse, and misjudgment of visually similar characters in oracle bone script recognition.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence, computer vision and natural language processing, and in particular to a method, device and medium for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion. Background Technology

[0002] Oracle bone script is the earliest known systematic writing system in China, the source of Chinese characters and the gene of Chinese civilization. It records various aspects of social life in the late Shang Dynasty, including sacrifices, military affairs, and agriculture, and is a valuable primary source for studying early forms of civilization. Since its discovery in 1899, despite over a century of research, a large portion of the characters remain "unidentified" or "questionable." These characters often involve key personal names, place names, or institutional terms, and are crucial for unraveling the mysteries of Shang Dynasty history. With the development of artificial intelligence technology, using computer vision to assist in oracle bone script recognition has become a core tool in the field of digital humanities, and is of great significance for the inheritance and development of this niche and esoteric field.

[0003] Existing technologies primarily rely on deep learning models such as convolutional neural networks or visual Transformers for automatic recognition of oracle bone script, supplemented by general strategies such as resampling, reweighting, or transfer learning based on modern Chinese characters to alleviate the data imbalance problem. For example, Chinese patent CN117975490A discloses an oracle bone script recognition method based on multiple image sources, starting with the oracle bone script system and using intelligent recognition technology to achieve intelligent character detection and automatic segmentation based on original rubbings. However, these technologies still have significant drawbacks in practical applications: First, oracle bone script data exhibits a typical extreme long-tail distribution. When existing models deal with the "tail" category, which has only a very small number of samples, the feature space is prone to collapse, causing the model to tend to misclassify scarce samples as visually similar high-frequency "head" categories. Second, existing recognition schemes mostly adopt a "flat-level classification" strategy, treating each character as an independent label, completely ignoring the inherent component hierarchy and character formation system (such as radicals and pictographic components) of oracle bone script as an ideographic script. This prevents the model from using hierarchical logic to narrow the search space, resulting in insufficient discriminative power when facing fine-grained categories with similar character shapes. In addition, since oracle bone script images mostly originate from damaged and cracked rubbings, existing models are prone to overfitting background noise rather than character skeletons when samples are scarce. Finally, existing multimodal large-scale model schemes are mostly loose end-to-end predictions, lacking deep integration of ancient script expertise. When dealing with vertical domain problems, they are prone to illusion phenomena, making it difficult to guarantee the academic reliability of the recognition results. Summary of the Invention

[0004] The purpose of this invention is to address the problems of low recognition rate, poor generalization ability, and insufficient semantic utilization in existing technologies when facing the extremely long-tailed distribution of oracle bone script. It provides a method, device, and medium for recognizing long-tailed oracle bone script based on hierarchical structure and multimodal fusion. By constructing a superclass-subclass hierarchical structure that conforms to cognitive psychology, and combining the fine extraction of pure visual features with the deep alignment of multimodal semantic knowledge, it achieves high-precision recognition of difficult and low-frequency ancient characters.

[0005] The objective of this invention can be achieved through the following technical solutions: A long-tailed distribution oracle bone script recognition method based on hierarchical structure and multimodal fusion includes the following steps: S1. Data Construction: Obtain oracle bone images containing head and tail categories, perform data augmentation processing, and construct an oracle bone image dataset; S2. Hierarchical structure generation: Using the oracle bone image dataset as input, the image features are extracted using a self-supervised model, the geometric centroids of class prototypes are calculated to construct a prototype set, and hierarchical agglomerative clustering is performed using the Ward link criterion to generate a superclass-subclass tree structure of oracle bone text. S3. Semantic Library Construction: Utilize large language models to generate visual descriptions of each oracle bone script category and construct a multimodal knowledge base; S4. Training of Pure Visual Hierarchical Recognition Model: A pure visual hierarchical recognition model is constructed and trained based on the Oracle Bone Script image dataset and the corresponding superclass-subclass tree structure. The superclass-subclass prediction logic is constrained by the hierarchical consistency loss function. The coarse classification head of the hierarchical network outputs the superclass prediction result, and the fine classification head further outputs the subclass prediction result. During the training process, a winner-takes-all strategy is adopted, using the mask of the output result of the coarse classification head to filter irrelevant fine-grained categories. S5. Multimodal Fusion Training: Construct a multimodal recognition model, perform LoRA fine-tuning on the basis of the pre-trained multimodal large model, freeze the parameters of the pure visual hierarchical recognition model trained in S4, obtain the superclass prediction results corresponding to the input image, retrieve the visual description corresponding to the superclass prediction results in the multimodal knowledge base, input the input image, task instructions and retrieval results introduced based on the random corpus injection strategy into the multimodal recognition model, and output the subclass classification results using the multimodal fine classification head; S6. Inference stage: Acquire the image to be recognized, input it into the trained pure visual hierarchical recognition model to obtain the superclass prediction result, input the image to be recognized and the task instructions into the trained multimodal recognition model to obtain the subclass prediction result.

[0006] The data augmentation process specifically includes: Geometric transformation: Apply slight random rotation, random scaling, and random translation within a preset range to the original oracle bone script image, while avoiding horizontal flipping operations; Noise injection: Gaussian noise is introduced into the geometrically transformed oracle bone script image.

[0007] The method for calculating the geometric centroid of the aforementioned class prototype is as follows: ; in, These are real oracle bone script image samples. This is a feature extraction function used to map an input image into feature vectors in a feature space. For the c-th class of samples, Let be the number of samples in the c-th class sample set Sc. Let be the geometric centroid of class c in the feature space (i.e., the mean of the feature vectors of all samples in this class); Select the real sample closest to the centroid as the prototype: ; in, For the sample set of class c, the centroid of the class is... The closest real sample in Euclidean distance. Let L2 be the norm, used to measure the Euclidean distance between eigenvectors. Let be the prototype feature vector of class c; Build a prototype set based on the selected prototypes. ,in, This represents the total number of categories.

[0008] The process of generating a superclass-subclass tree structure for oracle bone inscriptions using hierarchical agglomerative clustering based on the Ward linking criterion specifically includes: For prototype set Calculate the cosine distance between any two prototypes in the equation. : ; in, Let pi be the k-th dimension feature component. The dimension of the prototype feature vector. The combination form of the prototypes to be fused is determined based on the cosine distance; Based on the determined combination form, the Ward link criterion is used to minimize the variance within clusters for clustering, generating a binary tree dendrogram, where clusters are merged at each step. and When choosing the pair that minimizes the increase in the sum of squared errors within the total cluster, the ESS increment formula is: ; in, , Let U and V be the center vectors, i.e., the centroids, of clusters U and V in the feature space, respectively. This represents the increment of the sum of squared errors within the cluster caused by merging cluster U and cluster V; The binary tree diagram is horizontally cut at a specific height to generate a mapping table, which then assigns the fine-grained results to a preset number of superclasses.

[0009] The pure vision-based hierarchical recognition model uses ConvNeXt V2-Base as the backbone network to extract feature vectors from the input image. Coarse classification head pairs feature vectors Perform the following processing: ; in, This is the output logits of the coarse classification header. , As weight, For batch normalization operations, For activation function, The predicted probability is obtained after Softmax normalization. ; Inject the output logits of the coarse classification head into the feature vector. In the process, the fusion vector is obtained. : ; in, It is a learnable scalar parameter. Indicates splicing; The fine-classification head processes the fused vector as follows: ; in, , As weight, For the output logits of the fine classification header, The predicted probability is obtained after Softmax normalization. .

[0010] The specific steps of using the coarse classification head output result mask to filter irrelevant fine-grained categories are as follows: Predicted probabilities based on coarse classification head output Find the superclass with the highest probability ; Construct a mask vector based on the superclass with the highest probability: ; in, For superclass The set of all corresponding subclass tags, For the mask vector M, the th The mask value at position n is used to indicate the nth position. Does each category belong to a superclass? The range of subclasses; The output logits of the fine-grained classification head are masked based on the mask vector to obtain the final Logits: .

[0011] The loss of the pure visual hierarchical recognition model is: in, The total loss of the pure visual hierarchical recognition model is... It is the standard cross-entropy loss. It is a loss of hierarchical consistency. For the superclass truth value, For the truth value of the subclass, For superclass prediction results, The prediction result for the subclass; wherein, the hierarchical consistency loss is defined as: ; in, Let be the superclass prediction result vector. The Middle The predicted probability values ​​of each superclass. For the subclass prediction result vector The predicted probability value of the j-th subclass. Let be the set of all subclass labels corresponding to the k-th superclass. The total number of superclasses.

[0012] The multimodal recognition model uses Qwen3-VL as its base and consists of a ViT visual encoder, a position-aware adapter, and a Qwen LLM. The hidden state of the penultimate layer of the Qwen LLM is extracted as the output feature, and a fixed-length vector is obtained by average pooling. The fixed-length vector is then processed by a multimodal fine-classification head to output the prediction result.

[0013] A long-tailed distributed oracle bone script recognition device based on hierarchical structure and multimodal fusion includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0014] A storage medium having a program stored thereon, which, when executed, implements the method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) Fundamentally alleviate the long tail effect: This invention greatly reduces the risk of tail samples being overwhelmed by head categories by decomposing large-scale, extremely imbalanced fine-grained classification tasks into balanced coarse-grained classification and local fine-grained classification.

[0016] (2) Integrating cognitive psychology and paleography: This invention simulates the cognitive process of human experts who “first observe the general appearance and then investigate the details”. The superclasses discovered by clustering often correspond to character families with similar structural logic, which is consistent with the evolutionary rules of oracle bone script.

[0017] (3) Multimodal deep empowerment: Unlike simple multimodal splicing, this invention uses descriptions generated by a large model as expert priors to help the model distinguish between visually similar variant characters. The random injection strategy ensures that the model remains robust even when the rubbing quality is poor and the text description may be inaccurate.

[0018] (4) High accuracy and strong generalization: The recognition accuracy of this invention is better than that of existing related methods in multiple public datasets (such as OBC306, HUST-OBC, etc.). Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram illustrating the implementation architecture of the pure visual hierarchical recognition model and the multimodal recognition model of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0021] This embodiment provides a long-tailed distribution oracle bone script recognition method based on hierarchical structure and multimodal fusion, such as... Figure 1 As shown, it includes the following steps: S1. Data Construction: Obtain oracle bone images containing head and tail categories, perform data augmentation processing, and construct an oracle bone image dataset.

[0022] S11 integrates multiple open-source Oracle datasets such as OBC306 and HUST-OBS to construct a large-scale comprehensive dataset containing over 300,000 images. ,in, The original input consists of single images of oracle bone script characters. These are fine-grained category labels, corresponding to specific oracle bone script character IDs. In this embodiment, .

[0023] S2. Considering the characteristics of oracle bone script rubbings (monochrome, lines, noise), this invention designs a specific enhancement pipeline to prevent model overfitting: (1) Geometric transformation: Due to the varied writing directions of oracle bone script, a slight random rotation is applied ( Random scaling (0.8-1.2 times) and random translation. Avoid horizontal flipping, as the orientation of oracle bone script sometimes has semantic meaning.

[0024] (2) Noise Injection: Gaussian noise is introduced to simulate the damage of the rubbing. Let the original image be... Enhanced image for: in Uniform sampling is performed. During the training phase, images are randomly cropped and resized. (For ConvNeXt) or (For Qwen-VL).

[0025] S2. Hierarchical Structure Generation: Using the oracle bone image dataset as input, the self-supervised model DINOv2 is used to extract image features, calculate the geometric centroid of class prototypes to construct a prototype set, and perform hierarchical agglomerative clustering through the Ward link criterion to generate a superclass-subclass tree structure of oracle bone text.

[0026] The key to this step is to flatten the surface. The categories are reorganized into a hierarchical structure with semantic relationships. This clustering process is not based on manually defined radicals, but on data-driven visual semantic similarity. Specifically, the cosine distance between class prototypes is calculated, and the Ward algorithm is used to minimize the intra-cluster variance, thereby ensuring that the generated superclasses have high compactness and separability in the visual feature space.

[0027] S21. DINOv2-Large (ViT-L / 14) is used as the feature extractor. DINOv2 is a self-supervised learning model that learns highly robust visual features through multi-scale cropping and teacher-student network distillation. For each image... Input DINOv2 and extract the output as a feature vector. .

[0028] S22. Due to the extreme imbalance of the data, directly clustering all samples would be dominated by the head class. Therefore, a class prototype is calculated for each fine-grained class. The geometric centroid of the class prototype is calculated as follows: ; in, These are real oracle bone script image samples. This is a feature extraction function used to map an input image into feature vectors in a feature space. For the c-th class of samples, Let be the number of samples in the c-th class sample set Sc. Let be the geometric centroid of class c in the feature space (i.e., the mean of the feature vectors of all samples in this class).

[0029] While the geometric centroid represents the distribution center, it is an abstract vector and may not correspond to any actual glyph (it lies outside the manifold). To ensure semantic authenticity, this embodiment selects the closest real sample to the centroid as the prototype: ; in, For the sample set of class c, the centroid of the class is... The closest real sample in Euclidean distance. Let L2 be the norm, used to measure the Euclidean distance between eigenvectors. Let be the prototype feature vector of class c; Build a prototype set based on the selected prototypes. ,in, This represents the total number of categories.

[0030] S23. In order to construct a tree structure, the prototype set... Perform clustering.

[0031] For prototype set Calculate the cosine distance between any two prototypes in the equation. Focus on the direction of the feature vectors (semantic similarity) rather than their magnitude: ; in, Let pi be the k-th dimension feature component. is the dimension of the prototype feature vector.

[0032] The combination form of the prototypes to be fused is determined based on the cosine distance.

[0033] Based on the determined combination form, the Ward link criterion is used to minimize the variance within clusters for clustering, generating a binary tree dendrogram, where clusters are merged at each step. and When choosing the pair that minimizes the increase in the sum of squared errors within the total cluster, the ESS increment formula is: ; in, , Let U and V be the center vectors, i.e., the centroids, of clusters U and V in the feature space, respectively. This represents the sum of squared errors within the cluster caused by merging cluster U and cluster V.

[0034] The Ward method typically generates clusters that are relatively uniform in size and compact in shape, making them ideal for constructing superclasses for classification tasks. The resulting dendrogram is a binary tree. To obtain clusters that are easier to classify... In this embodiment, a superclass is used to perform horizontal cutting at a specific height in the tree. The value is set to around 200. This generates a mapping table. The fine-grained words are categorized into K superclasses.

[0035] S3. Semantic Library Construction: Utilize a large language model to generate visual descriptions of each oracle bone script category and construct a multimodal knowledge base.

[0036] First, a Prompt design is performed. Using Gemini-2.5-Flash, representative images of the subclasses within each superclass are selected to generate descriptions, which are then compiled into a corpus Tcorpus(k) for that superclass.

[0037] S4. Training of Pure Visual Hierarchical Recognition Model: A pure visual hierarchical recognition model is constructed and trained based on the Oracle image dataset and the corresponding superclass-subclass tree structure. The superclass-subclass prediction logic is constrained by the hierarchical consistency loss function. The coarse classification head of the hierarchical network outputs the superclass prediction result, and the fine classification head further outputs the subclass prediction result. During the training process, a winner-takes-all strategy is adopted, and irrelevant fine-grained categories are filtered out using the mask of the output result of the coarse classification head.

[0038] The processing procedure of a purely visual hierarchical recognition model is as follows: Figure 2 As shown in (a), the specific steps include: S41. Select ConvNeXt V2-Base as the backbone of the pure vision-level recognition model, and perform feature extraction on the input image to obtain the feature vector. : ConvNeXt V2 combines the efficiency of convolutional neural networks with the global awareness of Transformers, and introduces a fully convolutional mask autoencoder pre-training technique, giving it a strong ability to capture the structural features of line drawings like oracle bone script. It also introduces a global response normalization layer, significantly enhancing feature competition between channels and preventing feature map collapse. Pre-training: Initialized using weights pre-trained on ImageNet-1K using a fully convolutional mask autoencoder (FCMAE). FCMAE forces the network to learn the low-level structure and texture of images through masked image reconstruction tasks, which is crucial for recognizing incomplete oracle bone script.

[0039] S42, Coarse classification head pairs feature vectors Perform the following processing: ; in, This is the output logits of the coarse classification header. , As weight, For batch normalization operations, For activation function, The predicted probability is obtained after Softmax normalization. .

[0040] In order for the fine classifier to "know" the current macro category, the output logits of the coarse classifier head are injected into the feature vector. In the process, the fusion vector is obtained. : ; in, It is a learnable scalar parameter. Indicates splicing.

[0041] S43, Fine-classification head-to-fusing vector Perform the following processing: ; in, , As weight, For the output logits of the fine classification header, The predicted probability is obtained after Softmax normalization. .

[0042] The loss of the pure visual hierarchical recognition model is: in, The total loss of the pure visual hierarchical recognition model is... It is the standard cross-entropy loss. It is a loss of hierarchical consistency. For the superclass truth value, For the truth value of the subclass, For superclass prediction results, This refers to the prediction results for subclasses. In probability theory, the probability of a parent event occurring should be equal to the sum of the probabilities of all its mutually exclusive child events occurring. Therefore, the hierarchical consistency loss is defined as: ; in, Let be the superclass prediction result vector. The Middle The predicted probability values ​​of each superclass. For the subclass prediction result vector The predicted probability value of the j-th subclass. Let be the set of all subclass labels corresponding to the k-th superclass. The total number of superclasses.

[0043] This constraint not only ensures logical consistency but, more importantly, enables regularization of gradient backpropagation. When the model is uncertain about a particular tail subclass sample, the consistency loss guides it to refer to the features of the parent class, thus avoiding blind guessing.

[0044] Furthermore, this step employs a hierarchical masking strategy, also known as "winner-takes-all," which uses a mask to filter irrelevant fine-grained categories from the output of the coarse classification head. In long-tail classification, the biggest source of error is "cross-superclass confusion." Once the parent class is determined, the search space for subclasses will shrink by tens of times, and the error rate will decrease exponentially. The specific process is as follows: Predicted probabilities based on coarse classification head output Find the superclass with the highest probability ; Construct a mask vector based on the superclass with the highest probability: ; in, For superclass The set of all corresponding subclass tags, For the mask vector M, the th The mask value at position n is used to indicate the nth position. Does each category belong to a superclass? The range of subclasses; The output logits of the fine-grained classification head are masked based on the mask vector to obtain the final Logits: .

[0045] S5. Multimodal Fusion Training: Construct a multimodal recognition model, perform LoRA fine-tuning on the pre-trained multimodal large model, freeze the parameters of the pure visual hierarchical recognition model trained in S4, obtain the superclass prediction results corresponding to the input image, retrieve the visual descriptions corresponding to the superclass prediction results in the multimodal knowledge base, input the input image, task instructions, and retrieval results introduced based on the random corpus injection strategy into the multimodal recognition model, and output the subclass classification results using the multimodal fine classification head.

[0046] The processing procedure of the multimodal recognition model is as follows: Figure 2 As shown in (b) above, the specific steps include: S51. The multimodal recognition model uses Qwen3-VL as its base and consists of a ViT visual encoder, a position-aware adapter, and a Qwen LLM. Input: Image +Text prompt .

[0047] Feature extraction: Instead of using the output of the last layer, the hidden state of the penultimate layer of Qwen LLM is extracted as the output feature. This layer contains visual-text alignment features fused by multiple self-attention mechanisms.

[0048] Pooling: Average pooling is applied to the sequence dimension to obtain a fixed-length vector. The fixed-length vector is processed by a multimodal fine-classification head to output the prediction result.

[0049] S52. During training, the model is randomly prompted with a Bernoulli distribution to determine whether to input textual prompts. This mechanism forces the model to learn "dual-path" representations: it can utilize semantic information when aided by expert knowledge, and it can rely on pure visual features when there are no textual prompts. When the textual description matches the image features, it uses the text to enhance confidence, which greatly improves robustness and avoids modality dependency and modality collapse.

[0050] Specifically, the coarse predictions from a pure visual model are used as the index, and randomness is introduced to generate the Prompt: Step A: Predict coarse categories using a frozen pure visual hierarchical recognition model. .

[0051] Step B: Retrieve from the multimodal knowledge base .

[0052] Step C: With probability Injected text: Case 1 (Injection): Input = Image + "Database Corpus + Task Instructions"; Case 2 (without injection): Input = image + "task instructions".

[0053] This mechanism simulates a training environment with "noise cues." The model is forced to learn to leverage textual confidence when the textual description matches the image features, which greatly improves robustness.

[0054] S6. Inference stage: Acquire the image to be recognized, input it into the trained pure visual hierarchical recognition model to obtain the superclass prediction result, input the image to be recognized and the task instructions into the trained multimodal recognition model to obtain the subclass prediction result.

[0055] The above is an introduction to the method embodiments. The following describes the solution of the present invention further through device embodiments.

[0056] A long-tailed distributed oracle bone script recognition device based on hierarchical structure and multimodal fusion, comprising: Data Construction Module: Acquire oracle bone inscription images containing head and tail categories, perform data augmentation processing, and construct an oracle bone inscription image dataset; Hierarchical structure generation module: Taking the oracle bone image dataset as input, it uses a self-supervised model to extract image features, calculates the geometric centroid of class prototypes to construct a prototype set, and performs hierarchical agglomerative clustering through the Ward link criterion to generate a superclass-subclass tree structure of oracle bone text. Semantic library construction module: Utilizes a large language model to generate visual descriptions of each oracle bone script category, and constructs a multimodal knowledge base; Pure visual hierarchical recognition model training module: Construct a pure visual hierarchical recognition model, trained based on the Oracle image dataset and the corresponding superclass-subclass tree structure, and constrain the superclass-subclass prediction logic through a hierarchical consistency loss function. The coarse classification head of the hierarchical network outputs the superclass prediction result, and the fine classification head further outputs the subclass prediction result. During the training process, a winner-take-all strategy is adopted, using the mask of the output result of the coarse classification head to filter irrelevant fine-grained categories. Multimodal fusion training module: Constructs a multimodal recognition model, performs LoRA fine-tuning on the pre-trained multimodal large model, freezes the parameters of the S4-trained pure visual hierarchical recognition model, obtains the superclass prediction results corresponding to the input image, retrieves the visual description corresponding to the superclass prediction results in the multimodal knowledge base, inputs the input image, task instructions and retrieval results introduced based on the random corpus injection strategy into the multimodal recognition model, and outputs the subclass classification results using the multimodal fine classification head; Inference module: It acquires the image to be recognized, inputs it into the trained pure visual hierarchical recognition model, obtains the superclass prediction result, and inputs the image to be recognized and the task instructions into the trained multimodal recognition model to obtain the subclass prediction result.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0058] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A long-tailed distribution oracle bone script recognition method based on hierarchical structure and multimodal fusion, characterized in that, Includes the following steps: S1. Data Construction: Obtain oracle bone script images containing head and tail categories, perform data augmentation processing, and construct an oracle bone script image dataset; S2. Hierarchical structure generation: Using the oracle bone image dataset as input, the image features are extracted using a self-supervised model, the geometric centroids of class prototypes are calculated to construct a prototype set, and hierarchical agglomerative clustering is performed using the Ward link criterion to generate a superclass-subclass tree structure of oracle bone text. S3. Semantic Library Construction: Utilize large language models to generate visual descriptions of various oracle bone script categories and construct a multimodal knowledge base; S4. Training of Pure Visual Hierarchical Recognition Model: A pure visual hierarchical recognition model is constructed and trained based on the Oracle Bone Script image dataset and the corresponding superclass-subclass tree structure. The superclass-subclass prediction logic is constrained by the hierarchical consistency loss function. The coarse classification head of the hierarchical network outputs the superclass prediction result, and the fine classification head further outputs the subclass prediction result. During the training process, a winner-takes-all strategy is adopted, using the mask of the output result of the coarse classification head to filter irrelevant fine-grained categories. S5. Multimodal Fusion Training: Construct a multimodal recognition model, perform LoRA fine-tuning on the basis of the pre-trained multimodal large model, freeze the parameters of the pure visual hierarchical recognition model trained in S4, obtain the superclass prediction results corresponding to the input image, retrieve the visual description corresponding to the superclass prediction results in the multimodal knowledge base, input the input image, task instructions and retrieval results introduced based on the random corpus injection strategy into the multimodal recognition model, and output the subclass classification results using the multimodal fine classification head; S6. Inference stage: Acquire the image to be recognized, input it into the trained pure visual hierarchical recognition model to obtain the superclass prediction result, input the image to be recognized and the task instructions into the trained multimodal recognition model to obtain the subclass prediction result.

2. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The data augmentation process specifically includes: Geometric transformation: Apply slight random rotation, random scaling, and random translation within a preset range to the original oracle bone script image, while avoiding horizontal flipping operations; Noise injection: Gaussian noise is introduced into the geometrically transformed oracle bone script image.

3. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The method for calculating the geometric centroid of the class prototype is as follows: ； in, These are real oracle bone script image samples. This is a feature extraction function used to map an input image into feature vectors in a feature space. For the c-th class of samples, Let be the number of samples in the c-th class sample set Sc. Let be the geometric centroid of class c in the feature space (i.e., the mean of the feature vectors of all samples in this class); Select the real sample closest to the centroid as the prototype: ； in, For the sample set of class c, the centroid of the class is... The closest real sample in Euclidean distance. Let L2 be the norm, used to measure the Euclidean distance between eigenvectors. Let be the prototype feature vector of class c; Build a prototype set based on the selected prototypes. ,in, This represents the total number of categories.

4. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The process of generating a superclass-subclass tree structure for oracle bone inscriptions using hierarchical agglomerative clustering based on the Ward linking criterion specifically includes: For prototype set Calculate the cosine distance between any two prototypes in the equation. : ； in, Let pi be the k-th dimension feature component. The dimension of the prototype feature vector. The combination form of the prototypes to be fused is determined based on the cosine distance; Based on the determined combination form, the Ward link criterion is used to minimize the variance within clusters for clustering, generating a binary tree dendrogram, where clusters are merged at each step. and When choosing the pair that minimizes the increase in the sum of squared errors within the total cluster, the ESS increment formula is: ； in, , Let U and V be the center vectors, i.e., the centroids, of clusters U and V in the feature space, respectively. This represents the increment of the sum of squared errors within the cluster caused by merging cluster U and cluster V; The binary tree diagram is horizontally cut at a specific height to generate a mapping table, which then assigns the fine-grained results to a preset number of superclasses.

5. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The pure vision-based hierarchical recognition model uses ConvNeXt V2-Base as the backbone network to extract feature vectors from the input image. Coarse classification head pairs feature vectors Perform the following processing: ； in, This is the output logits of the coarse classification header. , As weight, For batch normalization operations, For activation function, The predicted probability is obtained after Softmax normalization. ; Inject the output logits of the coarse classification head into the feature vector. In the process, the fusion vector is obtained. : ； in, It is a learnable scalar parameter. Indicates splicing; The fine-classification head processes the fusion vector as follows: ； in, , As weight, For the output logits of the fine classification header, The predicted probability is obtained after Softmax normalization. .

6. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 5, characterized in that, The specific steps of using the coarse classification head output result mask to filter irrelevant fine-grained categories are as follows: Predicted probabilities based on coarse classification head output Find the superclass with the highest probability ; Construct a mask vector based on the superclass with the highest probability: ； in, For superclass The set of all corresponding subclass tags, For the mask vector M, the th The mask value at position n is used to indicate the nth position. Does each category belong to a superclass? The range of subclasses; The output logits of the fine-grained classification head are masked based on the mask vector to obtain the final Logits: .

7. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The loss of the pure visual hierarchical recognition model is: in, The total loss of the pure visual hierarchical recognition model is... It is the standard cross-entropy loss. It is a loss of hierarchical consistency. For the superclass truth value, For the truth value of the subclass, For superclass prediction results, The prediction result for the subclass; wherein, the hierarchical consistency loss is defined as: ； in, Let be the superclass prediction result vector. The Middle The predicted probability values ​​of each superclass. For the subclass prediction result vector The predicted probability value of the j-th subclass. Let be the set of all subclass labels corresponding to the k-th superclass. The total number of superclasses.

8. The method for long-tailed distribution oracle bone script recognition based on hierarchical structure and multimodal fusion according to claim 1, characterized in that, The multimodal recognition model uses Qwen3-VL as its base and consists of a ViT visual encoder, a position-aware adapter, and a Qwen LLM. The hidden state of the penultimate layer of the Qwen LLM is extracted as the output feature, and a fixed-length vector is obtained by average pooling. The fixed-length vector is then processed by a multimodal fine-classification head to output the prediction result.

9. A long-tailed distributed oracle bone script recognition device based on hierarchical structure and multimodal fusion, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images