A method for zero-sample entity relation extraction from oracle bone inscriptions in a multimodal large model

Abstract

This invention discloses a method for zero-shot entity relation extraction from oracle bone script using a multimodal large-scale model. First, multimodal oracle bone script data is collected and preprocessed. Then, an oracle bone script knowledge system is defined, and the multimodal data is labeled according to entity and relation categories. A multimodal labeled dataset is then constructed, divided into training and test sets. Next, a zero-shot entity relation extraction model is built, inputting the multimodal oracle bone script data from the training set into the model and outputting multimodal triples. Finally, a modality fusion model is constructed, inputting the multimodal triples into the model for multimodal information fusion, ultimately obtaining structured triples. This invention achieves zero-shot oracle bone script entity relation extraction by fusing text, image, and HTML multimodal information, combined with large-scale model prompting learning and efficient parameter fine-tuning techniques. This reduces dependence on labeled data and improves the accuracy, comprehensiveness, and generalization ability of the extraction results.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, ancient script information processing, entity relation extraction, and large language model fine-tuning. Specifically, it refers to a method for zero-sample entity relation extraction of oracle bone script using a multimodal large model, applicable to scenarios such as knowledge mining, knowledge graph construction, and intelligent question answering in the field of oracle bone script. Background Technology

[0002] Oracle bone script is the oldest known Chinese writing system and a crucial resource for studying the history and culture of the Shang and Zhou dynasties. Effectively integrating and deeply exploring these highly specialized materials and documents is of great significance for promoting the systematic and digital research of oracle bone script. For a long time, the collation and research of oracle bone script has mainly relied on expert interpretation and traditional textual analysis. However, this approach is inefficient, highly subjective, and unable to meet the current demands for large-scale knowledge extraction from oracle bone script data.

[0003] Entity relation extraction technology can identify entities and their semantic relationships from unstructured text and represent them in a structured form of (subject-relation-object) triples. Applying this technology to oracle bone research can effectively support downstream tasks such as knowledge graph construction and question-answering systems, improve information organization and research efficiency, and is a key technology for realizing intelligent oracle bone research.

[0004] While entity relation extraction technology has made significant progress in general domains, it still faces numerous challenges in the oracle bone script context. On the one hand, the oracle bone script domain lacks high-quality annotated corpora, and its domain knowledge system and relation classifications continue to expand with new research, making traditional methods relying on annotated data unsuitable. Therefore, a zero-shot entity relation extraction method is needed, enabling the model to identify new relation types based on existing knowledge and relation descriptions even in the absence of annotated examples. On the other hand, the same Chinese character exhibits different glyphic features at different times, and much key information is contained in oracle bone fragment images, character evolution diagrams, and semi-structural layouts of web pages. Relying solely on text is insufficient to fully capture this information; therefore, introducing multimodal information plays a crucial role in improving the comprehensiveness and accuracy of oracle bone script entity relation extraction. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, the present invention aims to provide a multimodal large-model oracle bone script zero-shot entity relation extraction method. By integrating multimodal information from text, images, and HTML, and combining large-model prompting learning with efficient parameter fine-tuning techniques, the method achieves zero-shot oracle bone script entity relation extraction, reduces dependence on labeled data, and improves the accuracy, comprehensiveness, and generalization ability of the extraction results.

[0006] This invention provides a method for zero-sample entity relation extraction from a multimodal large-model oracle bone script, comprising the following steps: S1. Collect multimodal data of oracle bone inscriptions and perform preprocessing; The Oracle multimodal data includes text data, HTML data, and image data; S2. Define the Oracle Bone Script knowledge system, which includes entity categories, relation categories, and constraint rules. Entity categories are conceptual types formed by semantically classifying identifiable objects in the Oracle Bone Script knowledge domain, used to indicate the object category to which an entity belongs. Relationship categories are abstract classifications of semantic associations between entity instances, used to indicate the logical connections between entities. Constraint rules include entity constraint rules and relation constraint rules. Entity constraint rules are used to restrict the attributes or existence conditions of a certain entity in an entity category, and relation constraint rules are used to restrict the subject and object types of the triplet structured relation connected by the relation. The types of subject and object refer to the entity category. The Oracle multimodal data is labeled according to entity category and relation category to obtain multimodal labeled data. A multimodal labeled dataset is constructed and divided into training set and test set. S3. Construct a zero-shot entity relation extraction model; the zero-shot entity relation extraction model includes an adjusted DeepSeek-R1 module, an adjusted MiniCPM-V4.5 module, and a prompt word module; the adjusted DeepSeek-R1 module is obtained by adjusting and optimizing the DeepSeek-R1 module using the LoRA parameter efficient fine-tuning method; the adjusted MiniCPM-V4.5 module is obtained by adjusting and optimizing the MiniCPM-V4.5 module using the LoRA parameter efficient fine-tuning method; the adjusted DeepSeek-R1 module is used to process text annotation data and HTML annotation data, and the adjusted MiniCPM-V4.5 module is used to process image annotation data; the prompt word module is constructed based on the Oracle knowledge system and the thought chain reasoning mechanism; The multimodal labeled data is input into the zero-sample entity relation extraction model to obtain multimodal triples, which include text modal triples, HTML modal triples and image modal triples; S4. Construct a modal fusion model, input the multimodal triples into the modal fusion model to complete the fusion of multimodal information, and finally obtain structured triples; the modal fusion model includes a large inference model with a fusion module, and embeds the constraint rules, modal voting mechanism and semantic similarity scoring mechanism in the large inference model; In S3, the expression for the prompt word module is:

[0007] In the formula, The prompt word template generated for the prompt word module, which has an instruction-like nature. This is the core generation function of the prompt word module, representing a mapping relationship; This is the entity category set of the prompt word module, used to define the entity types that need to be recognized in the prompt words. This is the set of relation categories for the prompt word module, used to define the relation types to be extracted from the prompt words. The constraint rules for the prompt word module are used to define the format, logic, and other constraints that the prompt words must follow; the prompt word module clarifies the entity and relation recognition requirements and constraint rules, guiding the zero-sample entity relation extraction model to output multimodal triples.

[0008] Furthermore, in S1, the preprocessing is divided into text data preprocessing, image data preprocessing, and HTML data preprocessing. Specifically, the text data is deduplicated and redundant is removed; the image data is cropped and formatted to retain key visual features; and redundant tags and irrelevant content are removed from the HTML data, and structured layout information is extracted.

[0009] Furthermore, in S3, the LoRA parameter efficient fine-tuning method uses the following parameters to adjust and optimize the DeepSeek-R1 module: batch size of 2, number of training epochs of 4, and initial learning rate of 2.00 × 10⁻⁶. -4 The optimizer is AdamW; the LoRA parameter efficient fine-tuning method was used to adjust and optimize the MiniCPM-V4.5 module. The parameters used were: batch size of 2, number of training epochs of 3, and initial learning rate of 1.00 × 10⁻⁶. -4 The optimizer is AdamW.

[0010] Furthermore, in S3, the specific steps for obtaining the multimodal triplet are as follows: Sa1. Input the text-annotated data into the fine-tuned DeepSeek-R1 module, and then output the text modal triples guided by the prompt word module. After the text-annotated data is input into the fine-tuned DeepSeek-R1 module, the zero-shot entity relation extraction model first performs semantic parsing based on the prompt word template to identify candidate entities in the text; then it performs relation reasoning by combining the entity category set and the relation category set; then it filters out illegal entity relations according to constraint rules; finally, it generates text modal triples in the form of "subject-relation-object". Sa2. The HTML-annotated data is converted into a text format that can be recognized by the zero-shot entity relation extraction model. Then it is input into the adjusted DeepSeek-R1 model, which uses HTML structural features to help identify entity associations and outputs HTML modal triples. The HTML-annotated data is first processed by the HTML parsing module to extract the DOM hierarchy, tag attributes, and text node information. Then, the structured HTML content is converted into a text representation that the model can recognize. It is then input into the adjusted DeepSeek-R1 module for entity recognition and relation reasoning. Finally, it is combined with constraint rules to filter out abnormal relations and generate HTML modal triples. Sa3. Input the image annotation data into the adjusted MiniCPM-V4.5 model, extract visual features, combine the prompt word module to identify entities and relationships, and output image modal triples; After the image annotation data is input into the adjusted MiniCPM-V4.5 model, the zero-shot entity relation extraction model first extracts visual features such as oracle bone script characters, rubbing information, and page structure; then it combines the prompt word module to identify entity information in the image; next, it infers entity relations through visual semantic association; finally, it filters valid relations according to constraint rules and outputs image modal triples.

[0011] Furthermore, in S4, the specific steps for obtaining the structured triplet are as follows: Sb1. Text modal triples, HTML modal triples, and image modal triples are uniformly input into the inference-based large model fusion module in the modality fusion model; the inference-based large model embeds constraint rules by combining forward prompts and backward logical filtering; the inference-based large model calls the constraint rules as judgment conditions to perform logical verification on the generated multimodal triples, and performs backward filtering to remove multimodal triples that do not meet the entity constraint rules or relation constraint rules; Sb2. Aggregate the multimodal triples with the same subject and the same relationship to form a corresponding candidate object set to handle conflicting or redundant information; use the modal voting mechanism of the reasoning-type large model to count the total number of times a specific object in the candidate object set is extracted in each modality as the basic confidence score; at the same time, use the semantic similarity scoring mechanism to measure the semantic closeness between candidate objects. Sb3. Construct a comprehensive scoring function that combines modal voting confidence score and semantic similarity. Perform comprehensive logical adjudication on the candidate object set and select the item with the highest comprehensive score as the final output result. The final output is a structured triple with a unified format, logical consistency and elimination of redundancy.

[0012] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) This invention makes full use of the semantic understanding and knowledge reasoning capabilities of the reasoning-type large model. By introducing zero-sample entity relationship extraction technology, it realizes the automatic extraction of entity relationships in oracle bone data, improves the automation level and structural expression capability of oracle bone information processing, and provides reliable data support and technical foundation for the subsequent construction of oracle bone knowledge graph.

[0013] (2) This invention integrates text, image and HTML data into oracle bone entity relationship extraction for the first time, making full use of the complementary advantages of each modality, fully mining the entity relationship information contained in oracle bone data, avoiding the limitations of single modality extraction, and improving the comprehensiveness of the extraction results.

[0014] (3) This invention reduces invalid extraction by constructing constraint rules to standardize the types and association logic of entities and relationships; by introducing a thought chain reasoning mechanism to guide the model to reason step by step, it improves the extraction accuracy in complex semantic scenarios and further ensures the reliability of the final result by screening for conflicts and redundancies in the multimodal fusion stage.

[0015] (4) The model of this invention can quickly adapt to the characteristics of oracle bone script such as "one character with multiple forms", complex data structure and dynamic updating of interpretation system through the transfer learning ability and parameter fine-tuning optimization of the reasoning-type large model. It shows good generalization performance on unseen relation types, and provides high-quality structured data support for downstream tasks such as oracle bone script knowledge graph construction and intelligent question answering, and promotes the development of oracle bone script research towards intelligence and digitalization. Attached Figure Description

[0016] Figure 1 This is a flowchart of the multimodal large-model oracle bone script zero-sample entity relation extraction method of the present invention; Figure 2 This is an example diagram illustrating the "one character, multiple forms" feature of oracle bone script in this invention, showing the variant forms of the same character head in different oracle bone fragments or periods; Figure 3 The structure diagram of the Oracle Bone Script knowledge system defined in this invention clearly presents the logical association between 8 types of entities and 7 types of relationships; Figure 4 The flowchart for obtaining the multimodal triplet in this invention; Figure 5 This is a flowchart of the multimodal information fusion process of the present invention, showing the complete process of multimodal triple collection, conflict and redundancy screening and final result output. Detailed Implementation

[0017] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, these descriptions do not constitute a limitation of the present invention and are merely illustrative. Through these descriptions, the advantages of the present invention will become clearer. All modifications that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention. Other parts not described in detail in the embodiments are prior art.

[0018] Example 1: This invention proposes a method for zero-sample entity relation extraction from multimodal large-scale oracle bone inscriptions, such as... Figure 1 As shown, it includes the following steps: S1. Collect multimodal data of oracle bone inscriptions from "Yinqi Wenyuan" and "Hanzi Holographic Resource Application System" (the collected multimodal data of oracle bone inscriptions is obtained from at least one publicly available data source related to oracle bone inscriptions, including but not limited to oracle bone inscription research databases, digital literature resource platforms, and academic resource platforms.) (The multimodal data of oracle bone inscriptions includes text data, HTML data, and image data; "Yinqi Wenyuan" provides oracle bone inscription bibliographic information, oracle bone fragment images, and character shape description data; "Hanzi Holographic Resource Application System" is used to supplement information on the evolution of Chinese character shapes and the correlation data between character shapes), and perform preprocessing; The preprocessing is divided into text data preprocessing, image data preprocessing, and HTML data preprocessing. Specifically, the text data is deduplicated and redundant is removed; the image data is cropped and formatted to retain key visual features; redundant tags and irrelevant content are removed from the HTML data, and structured layout information is extracted (the preprocessing steps are as follows: Text data: The text data is collected by parsing the DOM nodes of the target webpage and selecting text elements with preset class names; The collected text data is manually integrated to obtain semantically complete descriptive text. Image data: The image data are long screenshots of web pages. In order to maintain the continuity and integrity of the web page context information, this study did not crop the images to a fixed size, but retained the original resolution to ensure that the text layout, chart structure and other content are not damaged. The format is uniformly PNG. HTML data: The BeautifulSoup library is used to parse the HTML code, removing irrelevant tags such as script, style, and navigation bar, while retaining the core structural information of the body section. S2. Define the Oracle Bone Script knowledge system, which includes entity categories, relation categories, and constraint rules. The entity category is a conceptual type formed by semantically classifying identifiable objects in the Oracle Bone Script knowledge domain, used to indicate the object category to which the entity belongs. The entity categories are divided into 8 types, specifically: Head Character (Word): Modern Chinese characters corresponding to oracle bone inscriptions; e.g., core oracle characters such as "fire", "water"; the same head character has different glyph forms in different oracle bones or periods, such as Figure 2 shown.

[0019] Oracle Bone (OracleBone): The physical carrier carrying oracle bone inscriptions, usually represented by a number or a name; e.g., specific numbers or names of oracle bones such as "1991HD-H3-281", "JB5".

[0020] Interpretation (Meaning): Modern semantic interpretation corresponding to a head character; e.g., modern semantic interpretations such as "fire, shaped like flame burning and rising; the original meaning refers to the flame produced when an object burns.", "rain, shaped like rain falling from the sky, the horizontal stroke above represents the sky, and the vertical strokes and dots below represent rainwater and raindrops".

[0021] Location (Place): The current or collection location of an oracle bone; e.g., current collection locations of oracle bones such as "National Museum of China", "Anyang Yinxu Museum".

[0022] Image Link (ImgLink): A URL link pointing to image resources of oracle bone glyphs or rubbings; e.g., a URL pointing to image resources such as "https: / / www.jgwlbq.org.cn / img / word / yu.png".

[0023] Era (Age): The historical period to which the oracle bone inscription belongs; e.g., historical periods such as "the period of King Wu Ding".

[0024] Appearance Information (Appearance): Physical description of an oracle bone, such as material (tortoise shell) and specification and size; e.g., physical descriptions such as "tortoise shell", "residual length 20.4 cm, width 20.6 cm".

[0025] Excavation Time and Place (BornTP): Record of the specific time and place where the oracle bone was unearthed; e.g., unearthed information such as "East of Huayuanzhuang, Anyang, 1991".

[0026] The above-mentioned relationship categories are abstract classification of semantic associations between entity instances, used to indicate the logical connection existing between entities; the relationship categories are divided into 7 categories, specifically: Currently collected at: Connect an oracle bone to its storage location; for example (1991HD-H3-281, currently collected at, Anyang Workstation of the Institute of Archaeology, Chinese Academy of Social Sciences).

[0027] Interpreted as: Connect a head character to its corresponding interpretation; for example (one, interpreted as, the oracle bone glyph uses one horizontal stroke to represent the quantity "one", which is used for its original meaning in divination inscriptions).

[0028] Image links: Links connecting the initial character to its corresponding image resource; for example, (1. Oracle bone image link, (https: / / static.jgw.qq.com / server / 88bb9db9-87cd-4c50-b323-7815221f6b60.svg).

[0029] Engraved at: Connects the character head to the actual oracle bone fragment it is located on; for example (I, Engraved at, 1991HD-H3-281).

[0030] Era: Connect the oracle bone fragment to the era to which it belongs; for example (1991HD-H3-281, era to which it belongs, the reign of King Wu Ding).

[0031] Having appearance: Information connecting the oracle bone fragments to their appearance; for example (1991HD-H3-281, having appearance, tortoise shell).

[0032] "Excavated from": Information connecting the oracle bone fragments with the time and place of their excavation; for example (1991HD-H3-281, excavated in October 1991, east of Huayuanzhuang, Yinxu).

[0033] like Figure 3 As shown, Figure 3 It presents the logic of the association between entity categories and relationship categories.

[0034] The constraint rules include entity constraint rules and relation constraint rules. Entity constraint rules are used to restrict the attributes or existence conditions of a certain entity in an entity category. For example, entity constraint rules include that the character must be attached to the oracle bone fragment, and the location of the excavation must include both time and place. Relation constraint rules are used to restrict the types of the subject and object in the triple structured relation connected by the relation (the subject is the head entity in the triple structured relation, i.e., the initiator of the relation; the object is the tail entity in the triple structured relation, i.e., the pointer of the relation; the triple is composed of subject-relation-object). For example, the relation "engraved in" only associates the character and the oracle bone fragment, and the relation "currently hidden in" only associates the oracle bone fragment and the location. The types of the subject and object refer to the entity category. The Oracle multimodal data was labeled according to entity category and relation category to obtain multimodal labeled data. A multimodal labeled dataset was constructed, and the multimodal labeled dataset was divided into training set and test set in a 7:3 ratio. The dataset was manually labeled and contains a total of 3,585 samples, which were divided into a training set of 2,508 samples and a test set of 1,077 samples in a 7:3 ratio. In order to more comprehensively evaluate the model’s ability to extract zero-sample relationships, two additional relationship categories, “interpretation” and “era”, were introduced into the test set to test the zero-sample relationship extraction capability of the model of this invention. S3. Construct a zero-shot entity relation extraction model; the zero-shot entity relation extraction model includes an adjusted DeepSeek-R1 module, an adjusted MiniCPM-V4.5 module, and a prompt word module; the adjusted DeepSeek-R1 module is obtained by adjusting and optimizing the DeepSeek-R1 module using the LoRA parameter efficient fine-tuning method; the adjusted MiniCPM-V4.5 module is obtained by adjusting and optimizing the MiniCPM-V4.5 module using the LoRA parameter efficient fine-tuning method; the adjusted DeepSeek-R1 module is used to process text annotation data and HTML annotation data, and the adjusted MiniCPM-V4.5 module is used to process image annotation data; the prompt word module is constructed based on the Oracle knowledge system and the thought chain reasoning mechanism; The LoRA parameter fine-tuning method was used to adjust and optimize the DeepSeek-R1 module. The parameters used were: batch size of 2, number of training epochs of 4, and initial learning rate of 2.00 × 10⁻⁶. -4 The optimizer is AdamW; the LoRA parameter efficient fine-tuning method was used to adjust and optimize the MiniCPM-V4.5 module. The parameters used were: batch size of 2, number of training epochs of 3, and initial learning rate of 1.00 × 10⁻⁶. -4 The optimizer is AdamW; The DeepSeek-R1 module has 14 bytes of parameters and is used to process text and HTML labeled data. During optimization, the LoRA parameter fine-tuning method was employed, with the LoRA rank set to 8 and the scaling factor (alpha) set to 16. The batch size during training was set to 2, and the number of training epochs was set to 4. The DeepSeek-R1 module training used the AdamW optimizer for parameter updates, with an initial learning rate set to 2.0 × 10⁻⁶. -4 This is to improve training stability and generalization ability while ensuring the model's convergence speed. The MiniCPM-V4.5 module has 9 parameters and is used for image annotation data processing. During the tuning and optimization process, the LoRA parameter efficient fine-tuning method is also employed, with the LoRA rank set to 8, the batch size during training set to 2, and the number of training epochs set to 3. The MiniCPM-V4.5 module uses the AdamW optimizer during training, with an initial learning rate set to 1.0 × 10⁻⁶. -4 To adapt to the convergence characteristics of image modal models in multimodal training scenarios; The expression for the prompt word module is:

[0035] In the formula, The prompt word template generated for the prompt word module, which has an instruction-like nature. This is the core generation function of the prompt word module, representing a mapping relationship; This is the entity category set of the prompt word module, used to define the entity types that need to be recognized in the prompt words. This is the set of relation categories for the prompt word module, used to define the relation types to be extracted from the prompt words. The constraint rules for the prompt word module are used to define the format, logic, and other constraints that the prompt words must follow; the prompt word module clarifies the entity and relation recognition requirements and constraint rules, guiding the zero-sample entity relation extraction model to output multimodal triples.

[0036] The constraint rules embedded in the prompt word module ensure that the zero-sample entity relationship extraction model strictly follows the rules during the extraction process. For example, when the zero-sample entity relationship extraction model identifies the "hidden in" relationship, it will only establish an association between OracleBone and Place entities to avoid erroneous associations of other entity types.

[0037] The thought chain reasoning mechanism in the prompt word module extracts entity and relation information according to the following thought chain pattern: ① Cultural Relics Origin Tracing Type: Time and Place of Excavation → Event of Excavation → Oracle Bone Fragments → Current Location → Era ②Textual Carrier Type: Character Head → Engraving Action → Oracle Bone Fragment → Appearance Information → Era To Which It Belongs ③ Cultural semantic type: prefix → meaning → era → place of excavation The multimodal labeled data is input into the zero-sample entity relation extraction model to obtain multimodal triples, such as... Figure 4 As shown; the multimodal triples include text modal triples, HTML modal triples, and image modal triples; The specific steps to obtain the multimodal triplet are as follows: Sa1. Input the text-annotated data into the fine-tuned DeepSeek-R1 module, and then output the text modal triples guided by the prompt word module. After the text-annotated data is input into the fine-tuned DeepSeek-R1 module, the zero-shot entity relation extraction model first performs semantic parsing based on the prompt word template to identify candidate entities in the text; then it performs relation reasoning by combining the entity category set and the relation category set; then it filters out illegal entity relations according to constraint rules; finally, it generates text modal triples in the form of "subject-relation-object". Sa2. The HTML-annotated data is converted into a text format that can be recognized by the zero-shot entity relation extraction model. Then it is input into the adjusted DeepSeek-R1 model, which uses HTML structural features to help identify entity associations and outputs HTML modal triples. The HTML-annotated data is first processed by the HTML parsing module to extract the DOM hierarchy, tag attributes, and text node information. Then, the structured HTML content is converted into a text representation that the model can recognize. It is then input into the adjusted DeepSeek-R1 module for entity recognition and relation reasoning. Finally, it is combined with constraint rules to filter out abnormal relations and generate HTML modal triples. Sa3. Input the image annotation data into the adjusted MiniCPM-V4.5 model, extract visual features, combine the prompt word module to identify entities and relationships, and output image modal triples; After the image annotation data is input into the adjusted MiniCPM-V4.5 model, the zero-shot entity relation extraction model first extracts visual features such as oracle bone script shapes, rubbing layout, and page structure; then it combines the prompt word module to identify entity information in the image; next, it infers entity relations through visual semantic association; finally, it filters valid relations according to constraint rules and outputs image modal triples. Image modal triples refer to triples extracted by the model from visual perception regarding glyphs, carrier features, and image links, which compensate for the lack of non-textual information.

[0038] Example of implementing the prompt word module and mind chain output multimodal triples: Text modal cue module: "Task: Extract entity relation triples from the following oracle bone text. Entity categories: Word, Oracle Bone, Meaning, Place, Image Link, Age, Appearance, BornTP. Relationship categories: Currently in, Meaning, Image Link, Engraved in, Age, Appearance, Unearthed in. Constraints: 1. Word must be attached to Oracle Bone; 2. Unearthed in must include time and location; 3. The 'Engraved in' relation only relates to Word and Oracle Bone; 4. Currently in..." The context is as follows: The relationship is only linked to the oracle bone fragment and its location. The thought process is as follows: 1. First, identify all entities in the text and label their categories; 2. Analyze the possible relationships between entities and eliminate those that do not conform to the constraints; 3. Output the text modal triples in the form of (subject-relationship-object). Text content: The oracle bone fragment '1991HD-H3-101+168+1549' was unearthed in October 1991 at the Huayuanzhuang East Site in Yinxu, and is now housed at the Anyang Workstation of the Institute of Archaeology, Chinese Academy of Social Sciences. It is made of tortoise shell, with approximate dimensions of 19 cm in length and 16.1 cm in width, and is presumed to belong to the period of King Wu Ding of the Shang Dynasty. Output requirements: Output only the text modal triples, one line per fragment. Image modal cue template: "Task: Extract entity relation triples from the following oracle bone images. Entity categories: Word, OracleBone, Meaning, Place, Image Link, Age, Appearance, BornTP. Relationship categories: Currently held, Meaning, Image Link, Engraved, Age, Has Appearance, Unearthed." 1. The character head must be attached to the oracle bone fragment; 2. The location of excavation must include both time and place; 3. The 'engraved at' relationship only relates to the character head and the oracle bone fragment; 4. The 'currently hidden in' relationship only relates to the oracle bone fragment and its location. Thinking process: 1. Identify the oracle bone fragment or character head entities in the image and label their appearance features; 2. Analyze possible relationships based on the image context; 3. Output the image modal triples in the form of (subject-relationship-object). Image description: The image is a picture of an oracle bone fragment / oracle bone script page. Output requirements: Output only the image modal triples, one line per line. S4. Construct a modal fusion model, input the multimodal triples into the modal fusion model to complete the fusion of multimodal information, and finally obtain structured triples; the modal fusion model includes a large inference model with a fusion module, and embeds the constraint rules, modal voting mechanism and semantic similarity scoring mechanism in the large inference model.

[0039] like Figure 5As shown, the specific implementation steps for obtaining the structured triplet are as follows: Sb1. The text modal triples, HTML modal triples, and image modal triples are uniformly input into the inference-based large model fusion module in the modality fusion model. In this embodiment, the inference-based large model fusion module uses the DeepSeek-R1-Distill-Qwen-7B distillation model as the core decision-maker. The inference-based large model embeds the constraint rules using a combination of forward prompts and backward logic filtering. The inference-based large model calls the constraint rules as hard logic discrimination conditions to perform backward logic verification on all generated initial multimodal triples. Specifically, the inference-based large model checks one by one whether each multimodal triple violates entity constraint rules or relation constraint rules (e.g., the same character is not attached to a specific oracle bone fragment, or the entity unearthed only contains time but lacks location, or the subject of the "currently hidden in" relation is not an oracle bone fragment or the object is not a location). For any multimodal triple that does not meet any constraint condition, the fusion module performs physical elimination, deletes invalid multimodal triples, and prevents them from entering the subsequent conflict candidate pool.

[0040] Sb2. After verification and filtering, multimodal triples with the same subject and the same relationship are aggregated and grouped into the same conflict candidate set to handle conflict or redundant information; in this embodiment, the corresponding candidate object set is represented as follows:

[0041] In the formula, The set of candidate objects (i.e., the aggregation function) is formed by the aggregation of conflicting or redundant multimodal triples; the subject represents the same initiating entity (i.e., the head entity) in the multimodal triples; the relation represents the specific semantic relationship corresponding to the subject. Given a subject and relation, this represents the set of all candidate objects (i.e., tail entities) independently extracted from the text modality, HTML modality, and image modality, where... The total number of candidate objects. It is an integer greater than or equal to 1.

[0042] Sb3. Using the aforementioned reasoning-based large model for scoring and adjudication, the comprehensive scoring function expression used in the specific fusion decision is as follows:

[0043] In the formula, The final confidence score for the candidate object; is the modal voting score, and its specific calculation method is: the inference large model calls a counter to count the total frequency of each candidate object in the candidate object set being independently extracted in different modals (said text modality, said HTML modality and said image modality); the higher the extraction frequency, the candidate object obtains a higher base score; is the semantic similarity score, and its specific calculation method is: using the deep semantic understanding network of said DeepSeek-R1-Distill-Qwen-7B model to calculate the semantic proximity between different vocabulary expressions within said candidate object set, and output a similarity score between 0 and 1, which is used as a weight gain to enhance the comprehensive weight of entities with similar expressions but different presentations in different modals.

[0044] calculating the final score of each candidate object in said candidate object set through said comprehensive scoring function, and the inference large model automatically selects said comprehensive scoring function the candidate object with the highest calculation result is used as the only fusion output result, and other low-score items with literal conflicts and semantic redundancy are automatically discarded by said fusion module, so that while eliminating inter-modal conflicts, the final structured triples with unified format and logical self-consistency are output.

[0045] Implementation example of multi-modal fusion: Triple collection stage: For the same oracle bone inscription header "horse", different modal models output triples with expression differences: Text modality triples: (horse, interpreted as, the glyph is shaped like a horse, with a long horse face, long mane behind the neck, and a loose horse tail, original meaning is horse), (horse, carved on, 1991HD-H3-1295 oracle bone fragment), (1991HD-H3-1295, unearthed from, Huayuanzhuang East Site, Yin Ruins in October 1991), (1991HD-H3-1295, currently collected by, Anyang Workstation, Institute of Archaeology, Chinese Academy of Social Sciences), (1991HD-H3-1295, belonging to, Shang Dynasty); HTML modality triples: (horse, interpreted as, the glyph is shaped like a horse, with a long horse face, long mane behind the neck, and a loose horse tail, original meaning is horse), (horse, carved on, 1991HD-H3-1295 oracle bone fragment), (horse, has oracle bone inscription glyph image, https: / / static.jgw.qq.com / server / 7141afdb-d124-4546-92ba-b262bc49c831.svg), (1991HD-H3-1295, belonging to, King Wuding period of Shang Dynasty); Image modal triplet: (1991HD-H3-1295, with shape, tortoise shell, length 26.6 cm, width 20.0 cm), (1991HD-H3-1295, period of King Wu Ding of the Shang Dynasty). Fusion screening process: Invalid filtering: After backward logic verification by calling the constraint rules of the inference-type large model, the subject and object types and attributes of all the above multimodal triples do not violate the constraint rules and there is no abnormal data, so they are all retained.

[0046] Candidate set construction: The system aggregates data with the same subject and relationship. For era information with different expressions, the system extracts different object expressions for the subject "1991HD-H3-1295" and the relationship "era," constructing a conflict candidate object set: For completely redundant identical expressions (such as "interpretation of the subject "horse" and "engraved on" of the object), they are also assigned to their respective candidate sets.

[0047] Fusion Decision Computation: Applying a Comprehensive Scoring Function Quantitative judgment: For the set of conflicts across eras, the basic frequency of the large model statistics: "The period of King Wu Ding of Shang" appeared twice across modes ( =2), "Shang Dynasty" appears once ( =1); At the same time, the large model calculates semantic closeness: it determines that "Shang Dynasty" and "the period of King Wu Ding of Shang" have a strong semantic correlation and assigns corresponding... Weighted gain; the final comprehensive score for the "Shang Dynasty during the reign of King Wu Ding" is calculated by superimposing the results using a formula. The highest; similarly, for redundant objects in the sets "interpreted as" and "engraved on", due to their cross-modal nature... With a score of 2, it was successfully selected as the highest-scoring item after calculation, and redundancy was automatically eliminated.

[0048] Finally output the structured triples: (Ma, defined as, a character whose glyph resembles the shape of a horse, with a long horse face, a long mane behind the neck, and a loose tail, with the original meaning of horse), (Ma, engraved on, 1991HD-H3-1295 oracle bone fragment), (1991HD-H3-1295, excavated from, Huayuanzhuang East Site, Yin Ruins in October 1991), (1991HD-H3-1295, currently collected at, Anyang Workstation of Institute of Archaeology, Chinese Academy of Social Sciences), (Ma, has oracle bone character glyph image, https: / / static.jgw.qq.com / server / 7141afdb-d124-4546-92ba-b262bc49c831.svg), (1991HD-H3-1295, has shape, tortoise shell, 26.6 cm in length and 20.0 cm in width), (1991HD-H3-1295, belongs to the era of, King Wuding period of Shang Dynasty).

[0049] Finally, the precision, recall and F1-score indicators are calculated, and the zero-shot generalization ability of the zero-shot entity-relation extraction model is evaluated through the test set The calculation formula of said precision is: P=TP / (TP+FP)

[0050] wherein P is precision, TP is the number of correctly predicted structured triples, and FP is the number of incorrectly predicted structured triples; The calculation formula of said recall is: R=TP / (TP+FN) wherein R is recall, and FN is the number of real structured triples not recognized by the model; The calculation formula of said F1-score is: F1=2PR / (P+R) wherein F1-score is the harmonic mean of precision and recall. If either precision or recall is relatively low, the F1-score will be lowered, and the F1-score is a comprehensive indicator for evaluating the whole work.

[0051] Experimental Verification The performance indicators of the zero-shot entity-relation extraction model, namely precision, recall and F1-score, are verified through comparative experiments.

[0052] Comparative experiment: the zero-shot entity-relation extraction model of the present invention is compared with existing mainstream models, and the results are shown in Table 1 below: Table 1 Comparison table of precision, recall and F1-score between the model of the present invention and existing mainstream models

[0053] As shown in Table 1, the zero-sample entity relation extraction model of the present invention is significantly better than the single-modal model and the untuned basic model in terms of F1 score, demonstrating the advantages of multimodal fusion.

[0054] Ablation Experiment: To verify the impact of each modality on the performance of the zero-sample entity relation extraction model of this invention, an ablation experiment was designed, and the results are shown in Table 2 below: Table 2 Ablation Experiment

[0055] As shown in Table 2, different modal combinations have a significant impact on model performance. When only text is used as input, the model has high precision but low recall, indicating that the model can effectively identify relationships with obvious semantic features, but it is insufficient for identifying triples with complex structures or hidden distributions.

[0056] Introducing the HTML modality significantly improved the model's recall. HTML provides additional information such as webpage structure and tag hierarchy, helping the model to understand the input content more comprehensively. However, the addition of structural information also increases data complexity and may introduce some noise, leading to a slight decrease in precision.

[0057] The addition of the image modality further improved the model's performance, achieving the highest F1 score (0.66). Image information can supplement visual features such as glyph differences and layout, compensating for key information that text and HTML cannot cover. Although the accuracy was slightly lower than that of text and bimodal models, the overall performance was the highest, indicating that the image modality provided crucial complementary information, effectively enhancing the model's robustness and recognition coverage.

[0058] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0059] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for zero-sample entity relation extraction from multimodal large-scale oracle bone inscriptions, characterized in that, Includes the following steps: S1. Collect oracle bone script multimodal data and perform preprocessing; the oracle bone script multimodal data includes text data, HTML data and image data; S2. Define the Oracle Bone Script knowledge system, which includes entity categories, relation categories, and constraint rules. Entity categories are conceptual types formed by semantically classifying identifiable objects in the Oracle Bone Script knowledge domain, used to indicate the object category to which an entity belongs. Relationship categories are abstract classifications of semantic associations between entity instances, used to indicate the logical connections between entities. Constraint rules include entity constraint rules and relation constraint rules. Entity constraint rules are used to restrict the attributes or existence conditions of a certain entity in an entity category, and relation constraint rules are used to restrict the subject and object types of the triplet structured relation connected by the relation. The types of subject and object refer to the entity category. The Oracle multimodal data is labeled according to entity category and relation category to obtain multimodal labeled data. A multimodal labeled dataset is constructed and divided into training set and test set. S3. Construct a zero-shot entity relation extraction model; the zero-shot entity relation extraction model includes an adjusted DeepSeek-R1 module, an adjusted MiniCPM-V4.5 module, and a prompt word module; the adjusted DeepSeek-R1 module is obtained by adjusting and optimizing the DeepSeek-R1 module using the LoRA parameter efficient fine-tuning method; the adjusted MiniCPM-V4.5 module is obtained by adjusting and optimizing the MiniCPM-V4.5 module using the LoRA parameter efficient fine-tuning method; the adjusted DeepSeek-R1 module is used to process text annotation data and HTML annotation data, and the adjusted MiniCPM-V4.5 module is used to process image annotation data; the prompt word module is constructed based on the Oracle knowledge system and the thought chain reasoning mechanism; The multimodal labeled data is input into the zero-sample entity relation extraction model to obtain multimodal triples, which include text modal triples, HTML modal triples and image modal triples; S4. Construct a modal fusion model, input the multimodal triples into the modal fusion model to complete the fusion of multimodal information, and finally obtain structured triples; the modal fusion model includes a large inference model with a fusion module, and embeds the constraint rules, modal voting mechanism and semantic similarity scoring mechanism in the large inference model; In S3, the expression for the prompt word module is: ， In the formula, The prompt word template generated for the prompt word module, which has an instruction-like nature. This is the core generation function of the prompt word module, representing a mapping relationship; This is the entity category set of the prompt word module, used to define the entity types that need to be recognized in the prompt words. This is the set of relation categories for the prompt word module, used to define the relation types to be extracted from the prompt words. The constraint rules for the prompt word module are used to define the format and logical constraints that the prompt words must follow; the prompt word module clarifies the entity and relation recognition requirements and constraint rules, and guides the zero-shot entity relation extraction model to output multimodal triples; In S3, the specific steps for obtaining the multimodal triplet are as follows: Sa1. Input the text-annotated data into the adjusted DeepSeek-R1 module, and then output the text modal triples guided by the prompt word module. After the text-annotated data is input into the adjusted DeepSeek-R1 module, the zero-shot entity relation extraction model first performs semantic parsing based on the prompt word template to identify candidate entities in the text; then it performs relation reasoning by combining the entity category set and the relation category set; then it filters out illegal entity relations according to constraint rules; finally, it generates text modal triples in the form of "subject-relation-object". Sa2. The HTML-annotated data is converted into a text format that can be recognized by the zero-sample entity relation extraction model. Then it is input into the adjusted DeepSeek-R1 module, which uses HTML structural features to help identify entity associations and outputs HTML modal triples. The HTML-annotated data is first processed by the HTML parsing module to extract the DOM hierarchy, tag attributes, and text node information. Then, the structured HTML content is converted into a text representation that the model can recognize. It is then input into the adjusted DeepSeek-R1 module for entity recognition and relation reasoning. Finally, it is combined with constraint rules to filter out abnormal relations and generate HTML modal triples. Sa3. Input the image annotation data into the adjusted MiniCPM-V4.5 module to extract visual features, combine the prompt word module to identify entities and relationships, and output image modal triples; After the image annotation data is input into the adjusted MiniCPM-V4.5 module, the zero-sample entity relationship extraction model first extracts visual features such as oracle bone script shapes, rubbing information and page structure. Then, the entity information in the image is identified by combining the prompt word module; then, the entity relationship is inferred through visual semantic association; finally, the valid relationship is selected according to the constraint rules, and the image modal triplet is output.

2. The method for extracting zero-sample entity relations from multimodal large-model oracle bone inscriptions according to claim 1, characterized in that, In S1, the preprocessing is divided into text data preprocessing, image data preprocessing, and HTML data preprocessing. Specifically, the text data is deduplicated and redundant is removed; the image data is cropped, formatted, and key visual features are preserved. Remove redundant tags and irrelevant content from HTML data to extract structured layout information.

3. The method for extracting zero-sample entity relationships from multimodal large-model oracle bone inscriptions according to claim 1, characterized in that, In S3, the LoRA parameter efficient fine-tuning method uses the following parameters to adjust and optimize the DeepSeek-R1 module: batch size of 2, number of training epochs of 4, and initial learning rate of 2.00 × 10⁻⁶. -4 The optimizer is AdamW; the LoRA parameter efficient fine-tuning method was used to adjust and optimize the MiniCPM-V4.5 module. The parameters used were: batch size of 2, number of training epochs of 3, and initial learning rate of 1.00 × 10⁻⁶. -4 The optimizer is AdamW.

4. The method for extracting zero-sample entity relationships from multimodal large-model oracle bone inscriptions according to claim 1, characterized in that, In S4, the specific steps to obtain the structured triplet are as follows: Sb1. Text modal triples, HTML modal triples, and image modal triples are uniformly input into the inference-based large model fusion module in the modality fusion model; the inference-based large model embeds constraint rules by combining forward prompts and backward logical filtering; the inference-based large model calls the constraint rules as judgment conditions to perform logical verification on the generated multimodal triples, and performs backward filtering to remove multimodal triples that do not meet the entity constraint rules or relation constraint rules; Sb2. Aggregate the multimodal triples with the same subject and the same relationship to form a corresponding candidate object set to handle conflicting or redundant information; use the modal voting mechanism of the reasoning-type large model to count the total number of times a specific object in the candidate object set is extracted in each modality as the basic confidence score. At the same time, a semantic similarity scoring mechanism is used to measure the semantic closeness between candidate objects; Sb3. Construct a comprehensive scoring function that combines the basic confidence score of modal voting with the semantic similarity score. Perform comprehensive logical adjudication on the candidate object set and select the item with the highest comprehensive score as the final output result. The final output is a structured triple with a unified format, logical consistency and elimination of redundancy.

Images