A low-resource thanka multi-modal knowledge graph construction method and device

Abstract

The application relates to the technical field of knowledge graphs, in particular to a low-resource Thang multi-modal knowledge graph construction method and device, which acquires Thang field image and text data; extracts core terms from authoritative works, defines entities and attributes thereof, and establishes inter-class relationships; annotates the text data; performs fine-grained classification on the images; extracts triplets composed of entity categories, attributes and inter-class relationships from the text data by using a pre-constructed joint extraction model; the joint extraction model adopts ERNIE as an encoding layer and adopts a cascaded pointer network as a triplet extraction layer; and a Thang knowledge graph is established according to the triplets. The technical scheme can deeply integrate Thang field image and text data, realizes inferable association of image visual information and text semantic information, and the joint extraction model can still realize high-precision and fine-grained extraction of Thang professional entities and relationships under the condition of insufficient annotated data.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, specifically to a method and device for constructing a low-resource Thangka multimodal knowledge graph. Background Technology

[0002] Knowledge graphs, as a powerful tool for organizing and representing knowledge, have expanded from traditional text and structured data processing to multimodal data such as images, videos, and audio, giving rise to multimodal knowledge graphs.

[0003] Currently, research on multimodal knowledge graphs has formed a typical technical path in general fields. However, when existing technical paradigms are applied to specific fields such as Thangka, which have extremely high professionalism and unique cultural connotations, their inherent limitations become increasingly apparent. First, there is insufficient domain adaptability. Thangka art involves a large number of professional terms, and models trained on general corpora are extremely unfamiliar with these domain-specific concepts. In the "low-resource" environment where labeled data is scarce in the Thangka field, existing models struggle to accurately understand and extract the subtle and complex knowledge, resulting in insufficient accuracy and coarse granularity in the constructed knowledge graph. A deeper problem lies in the superficiality of multimodal fusion. Existing technologies often remain at the "image pasting" stage, simply associating a Thangka image with a text entity, without achieving deep semantic interaction.

[0004] Therefore, existing Thangka images and texts remain largely separate, failing to achieve true knowledge complementarity. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a low-resource Thangka multimodal knowledge graph construction method and device to solve the problem that Thangka images and text are still largely separate in the prior art, failing to achieve true knowledge complementarity.

[0006] According to a first aspect of the present invention, a method for constructing a low-resource Thangka multimodal knowledge graph is provided, characterized in that it includes:

[0007] Acquire original Thangka images and text data from the Thangka field, and preprocess the original Thangka images; The core terminology of Thangka is extracted from authoritative works, the entity categories and their attributes in the core terminology are defined, and inter-class relationships are established through related elements; based on entity categories, attributes and inter-class relationships, a Thangka knowledge graph ontology structure is constructed. Entity categories and inter-class relationships are used as annotation criteria to annotate the preprocessed text data according to preset annotation rules; The preprocessed Thangka images are classified in a fine-grained manner according to the defined entity categories. Using a pre-built joint extraction model, triples consisting of entity category, attribute, and inter-class relationship are extracted from labeled text data; the joint extraction model uses ERNIE as the encoding layer and a cascaded pointer network as the triple extraction layer. Based on the triples, and in accordance with the ontology structure of the Thangka knowledge graph, a Thangka knowledge graph corresponding to the original Thangka images and text data is established.

[0008] Preferably, the original Thangka image is preprocessed, including: The original Thangka image is subjected to grayscale conversion and noise reduction processing; The text data in the original Thangka image is recognized by text recognition and converted into editable digital text data.

[0009] Preferably, the preprocessed text data is annotated, including: The preprocessed text data is divided into several fields according to chapters and paragraphs; For each field, extract the words corresponding to the entity category from each sentence and treat the words as entities; prioritize establishing relationships between entities in the same sentence, and then establish relationships between entities and entities in other sentences of the same field.

[0010] Preferably, a pre-built joint extraction model is used to extract triples consisting of entity category, attribute, and inter-class relationship from the labeled text data, including: The encoding layer of the joint extraction model encodes the text data to obtain an initial encoded character sequence; The initial encoded character sequence is normalized to obtain the first encoded character sequence. On the first encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the head entity through the pointer network, and obtains the start position and end position of the head entity by threshold determination, thus obtaining the head entity encoding information. The header entity encoding information is fused with the initial encoded character sequence to obtain the second encoded character sequence; On the second encoded character sequence, the triplet extraction layer calculates the probability of each character being the start position and end position of the tail entity through a pointer network, and obtains the start position and end position of the tail entity by threshold determination, thus obtaining the tail entity encoding information. On the second encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the relation through a pointer network, and obtains the relation start position and end position by threshold determination, thus obtaining the relation encoding information; Decode the header entity encoding information, tail entity encoding information, and relation encoding information to obtain a triple containing the header entity, tail entity, and relation.

[0011] Preferably, the encoding layer of the joint extraction model encodes the text data, including: The encoding layer of the joint extraction model uses ERNIE to mask the core terms in the Thangka field in the text data; The masked text data is encoded to obtain an initial encoded character sequence containing only semantic features.

[0012] Preferably, fine-grained classification is performed on the preprocessed Thangka images, including: The preprocessed Thangka images are divided into corresponding entity categories, and images containing only the Buddha's body are selected from each category.

[0013] Preferably, a Thangka knowledge graph is established corresponding to the original Thangka images and text data, including: The categorized Thangka images are defined as attributes of corresponding entities, and the Thangka images are associated with the entities; or, The categorized Thangka images are defined as independent entities in the Thangka knowledge graph, and semantic relationships are established with other entities.

[0014] Preferably, after constructing the Thangka knowledge graph, it also includes: Convert the textual form of each entity into a vector, calculate the semantic similarity between the vectors, and if there are two vectors with a semantic similarity greater than a threshold, then merge the two entities.

[0015] Preferably, after constructing the Thangka knowledge graph, it also includes: The Thangka knowledge graph was converted to CSV format and imported into the Neo4j database for storage using UTF-8 encoding.

[0016] According to a second aspect of the present invention, a low-resource Thangka multimodal knowledge graph construction device is provided, comprising: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0017] The technical solution provided by this invention may include the following beneficial effects: It is understood that the technical solution presented in this invention acquires image and text data in the Thangka field; extracts core terms from authoritative works, defines entities and their attributes, and establishes inter-class relationships; annotates the text data; performs fine-grained classification of images; and uses a pre-constructed joint extraction model to extract triples composed of entity categories, attributes, and inter-class relationships from the text data. The joint extraction model uses ERNIE as the encoding layer and a cascaded pointer network as the triple extraction layer. Based on the triples, a Thangka knowledge graph is established. This technical solution can deeply integrate image and text data in the Thangka field, realizing a reasonable association between image visual information and text semantic information. Furthermore, the joint extraction model can still achieve high-precision, fine-grained extraction of Thangka-specific entities and relationships even under conditions of scarce labeled data.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 This is a schematic diagram illustrating the steps of a low-resource Thangka multimodal knowledge graph construction method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating the construction process of a multimodal knowledge graph according to an exemplary embodiment; Figure 3 This is a schematic diagram of a joint extraction model according to an exemplary embodiment; Figure 4 This is an example MLM diagram illustrating the BERT model and the ERNIR model according to an exemplary embodiment; Figure 5 This is a schematic diagram of knowledge fusion according to an exemplary embodiment. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0022] In one embodiment, Figure 1 This is a schematic diagram illustrating the steps of a low-resource Thangka multimodal knowledge graph construction method according to an exemplary embodiment. See also... Figure 1 This paper provides a low-resource method for constructing a multimodal knowledge graph for Thangka paintings, including: Step S11: Obtain original Thangka images and text data from the Thangka field, and preprocess the original Thangka images.

[0023] Step S12: Extract core terms in the Thangka field from authoritative works, define the entity categories and their attributes in the core terms, and establish inter-class relationships through associated elements; based on entity categories, attributes and inter-class relationships, establish the ontology structure of the Thangka knowledge graph.

[0024] Step S13: Using entity categories and inter-class relationships as annotation constraints, annotate the preprocessed text data according to preset annotation criteria.

[0025] Step S14: Perform fine-grained classification on the preprocessed Thangka images according to the defined entity categories; Step S15: Using a pre-built joint extraction model, extract triples consisting of entity category, attribute and inter-class relationship from the labeled text data; the joint extraction model uses ERNIE as the encoding layer and a cascaded pointer network as the triple extraction layer.

[0026] Step S16: Based on the triplet, and in accordance with the ontology structure of the Thangka knowledge graph, establish a Thangka knowledge graph corresponding to the original Thangka image and text data.

[0027] It is understood that the technical solution shown in this embodiment acquires image and text data in the Thangka field; extracts core terms from authoritative works, defines entities and their attributes, and establishes inter-class relationships; annotates the text data; performs fine-grained classification of images; and uses a pre-built joint extraction model to extract triples composed of entity categories, attributes, and inter-class relationships from the text data. The joint extraction model uses ERNIE as the encoding layer and a cascaded pointer network as the triple extraction layer. Based on the triples, a Thangka knowledge graph is established. This technical solution can deeply integrate Thangka field image and text data, realizing the reasonable association between image visual information and text semantic information. Furthermore, the joint extraction model can still achieve high-precision, fine-grained extraction of Thangka professional entities and relationships even under conditions of scarce labeled data.

[0028] In practice, Figure 2 It is a flowchart for constructing a multimodal knowledge graph, combined with Figure 2 The method shown in this embodiment will be described.

[0029] First, when performing step S11 to acquire data, multimodal data can be acquired, including structured data: such as standardized records in a database; semi-structured data: such as data in JSON, XML and other formats; and unstructured data: such as plain text, images, videos, etc.

[0030] The acquisition of image and text data is centered on authoritative resources in the field of Thangka painting: on the one hand, images were collected offline by scanning Thangka paintings in the collection of the XXX Art Museum and hand-painted works by Master XX, an inheritor of intangible cultural heritage, to construct an image dataset containing more than 200 images; on the other hand, textual knowledge was systematically extracted from authoritative works and literature in the field of Thangka painting. This integration of two authoritative data sources laid a high-quality and highly reliable data foundation for the subsequent construction of a multimodal knowledge graph.

[0031] The data is then preprocessed, including preprocessing the original Thangka image, which includes: performing grayscale conversion and noise reduction on the original Thangka image; and performing text recognition on the text data Thangka image in the original Thangka image to convert it into text data in an editable digital format.

[0032] Original Thangka images are mostly obtained through computer scanning of museum collections and hand-painted works by intangible cultural heritage inheritors. These methods are prone to introducing color interference and speckle noise due to scanning equipment and transmission processes. By performing grayscale conversion through a program, color redundancy can be reduced, focusing on the core visual features of the image (such as expression and contours). Simultaneous noise reduction eliminates irrelevant pixels, ensuring the purity of image pixel information and laying the foundation for subsequent fine-grained classification of Thangka images.

[0033] In terms of implementation, the TK-SEnet model can be used to classify the original images, eliminate noise interference introduced by scanning or transmission, and ensure the cleanliness of the input data.

[0034] Some images contain text information such as scriptures, Sanskrit names, and Tibetan names. This text needs to be extracted using OCR (Optical Character Recognition) technology, converting it from image format to an editable and searchable digital text format (e.g., .txt). Subsequent clipping can then be used to align the text and image, ensuring the accuracy and reliability of subsequent feature extraction tasks.

[0035] In addition, the input data needs to be standardized, cleaned and integrated, including: unifying the format and specifications of various types of data to ensure the consistency of data representation; filtering and extracting effective information from the raw data and removing irrelevant and redundant content; extracting core features for different modal data such as text and images and completing cross-modal feature fusion to lay the foundation for subsequent processing.

[0036] Next, the multimodal knowledge extraction stage is carried out. First, the ontology is constructed by executing step S12, extracting the core terms in the Thangka field from authoritative works, defining the entity categories and their attributes in the core terms, and establishing inter-class relationships through associated elements; based on the entity categories, attributes and inter-class relationships, the Thangka knowledge graph ontology structure is constructed.

[0037] In practice, the first step is to identify the sources of knowledge and core terms. Based on authoritative works such as "Explanation of Tibetan Buddhist Sacred Images", we systematically sort out the core terms specific to the Thangka field and strictly follow the Tibetan Buddhist tradition to classify Buddhist images and construct a standardized basic concept system.

[0038] Next, entity categories, attributes, and inter-class relationships are defined. Based on the core terminology system outlined above, the exclusive attributes of each type of entity are clarified. Semantic relationships between entity classes are established using related elements such as objects, mounts, and handprints as links, and finally, multiple core entities and multiple core relationships are defined.

[0039] Finally, create instances and semantic relationships: the abstract ontology model (including entity categories, attributes, and relationships) can be transformed into concrete instances using the dedicated ontology building software Protégé, while establishing semantic relationships between instances to form the ontology structure of the Thangka knowledge graph.

[0040] After the entity categories, attributes, and relationships are determined in the ontology construction, step S13 is executed to annotate the text data, and step S14 is executed to perform fine-grained classification of the Thangka images.

[0041] In step S13, the preprocessed text data is labeled, including: dividing the preprocessed text data into several fields according to chapters and paragraphs; for each field, extracting the words corresponding to the entity category from each sentence, and treating the words as entities; prioritizing the establishment of relationships between entities in the same sentence, and then establishing relationships between entities and entities in other sentences of the same nearest field.

[0042] First, the text data is divided into independent fields to ensure that the semantics of each field are relatively complete and to avoid information confusion across chapters and topics.

[0043] During the annotation phase, entities defined in the ontology construction phase (such as Bodhisattvas, objects, hand gestures, etc.) serve as constraints. Words matching the entity category (such as "Manjushri Bodhisattva") are selected from individual sentences in each field, and these words are explicitly labeled as entities. Relationship establishment follows the principle of "proximity association and semantic priority": semantic associations between entities within the same sentence are analyzed first to establish direct relationships (e.g., "Name A - Handheld - Object A"). If there are no entities in the current sentence or no direct associations between entities, entities in adjacent sentences within the same field are associated with the nearest entity, ensuring that entity relationships do not deviate from the context (e.g., if the previous sentence is labeled "Name B" and the next sentence is labeled "Seat B," then the relationship "Name B - Sitting - Seat B" is established). The entire process strictly adheres to the pre-defined annotation constraints, ensuring both the consistency and accuracy of the annotated data and providing high-quality training samples for the joint extraction model under the prompting learning framework, helping the model accurately capture the association patterns between entities and relationships in the Thangka domain.

[0044] In step S14, the preprocessed Thangka images are classified in a fine-grained manner, including: classifying the preprocessed Thangka images into corresponding entity categories, and selecting images containing only the Buddha's body from each category of Thangka images.

[0045] In practice, when performing fine-grained classification of Thangka images, the entity classification standards defined during the ontology construction phase are referenced to ensure complete consistency between image classification and text entity classification names. Subsequently, the classification rules are embedded into the TK-SEnet model. Leveraging the model's optimized feature recognition capabilities, accurate differentiation is achieved for Thangka images with similar expressions and ritual implements. After classification, images containing only the pure Buddha figure (without superfluous background or decorative interference) are selected from each category for targeted image segmentation, preserving the core visual features of the entities. This provides high-quality data support for the precise association of "entity-image attributes" in subsequent multimodal knowledge graphs.

[0046] Then, a joint extraction model was constructed using cue learning, pointer networks, and ERNIE to extract triples. The joint extraction model uses ERNIE as the encoding layer, and extracts the head entity and relation-specific tail entity sequentially through a cascaded pointer network, finally decoding to generate triples.

[0047] In a preferred embodiment, see Figure 3 Using a pre-built joint extraction model, triples consisting of entity category, attribute, and inter-class relationship are extracted from labeled text data, including: Step S21: The encoding layer of the joint extraction model encodes the text data to obtain the initial encoded character sequence.

[0048] First, the input Thangka text sentence is fed into the encoding layer, which encodes and generates an initial encoded character sequence g0 rich in semantics, providing a basic semantic representation for subsequent entity and relation recognition.

[0049] Step S22: Normalize the initial encoded character sequence to obtain the first encoded character sequence g1.

[0050] Step S23: On the first encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the head entity through the pointer network, and obtains the start position and end position of the head entity by threshold determination, thus obtaining the head entity encoding information.

[0051] This step performs head entity localization calculation after the initial encoding is completed. Based on g1, the probability of each character being the start and end position of the head entity is calculated, and this is done using the sigmoid activation function δ combined with trainable weights ω. ) and bias a( Complete the probability output, set a threshold (probabilities greater than the threshold are marked as 1, otherwise as 0), and lock the head entity boundary. The calculation formula is as follows:

[0052]

[0053] in, The probability of the starting position of the head entity. ω( is the probability of the head entity ending position; ) represents trainable weights; a( ) represents the trainable bias; δ represents the sigmoid activation function; g1 represents the sentence encoding.

[0054] Step S24: Merge the header entity encoding information with the initial encoded character sequence to obtain the second encoded character sequence.

[0055] This step merges the header entity encoding information with the initial encoded character sequence g0 to generate a new context representation containing the header entity information, which is the second encoded character sequence g2.

[0056] Step S25: On the second encoded character sequence, the triplet extraction layer calculates the probability of each character being the start position and end position of the tail entity through the pointer network, and obtains the start position and end position of the tail entity by threshold determination, thus obtaining the tail entity encoding information.

[0057] Step S26: On the second encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the relation through the pointer network, and obtains the start position and end position of the relation by threshold determination, thus obtaining the relation encoding information.

[0058] Steps S25 and S26 perform tail entity and relation matching calculations. The calculation method is the same as in step S23.

[0059] Step S27: Decode the header entity encoding information, tail entity encoding information, and relation encoding information to obtain a triple containing the header entity, tail entity, and relation.

[0060] In a preferred embodiment, the encoding layer of the joint extraction model encodes the text data, including: the encoding layer of the joint extraction model uses ERNIE to mask the core terms of the Thangka field in the text data; and encodes the masked text data to obtain an initial encoded character sequence containing only semantic features.

[0061] For the principles of ERNIE, please refer to Figure 4 Compared to BERT's single-character masking, ERNIE uses word masking, which better captures Chinese lexical semantics and improves the accuracy of identifying technical terms.

[0062] Preferably, to address the scarcity of labeled data in the Thangka field, a customized prompt template is designed to transform the original knowledge extraction task into a model generation task, guiding the model to directly output the target knowledge answer, thereby effectively reducing the dependence on large-scale labeled data and adapting to the needs of low-resource scenarios.

[0063] Alternatively, generative pre-training methods such as BART / T5 can be used, employing a sequence-to-sequence generative framework, to transform the joint entity and relation extraction task into a text generation task. This effectively addresses the low-resource challenge in the Thangka field and, through its end-to-end unified architecture, achieves high-quality triple extraction, providing a reliable text structuring foundation for the construction of multimodal knowledge graphs and offering another technical path for achieving "joint extraction."

[0064] After the multimodal knowledge extraction is completed, the multimodal Thangka knowledge graph (TK-MMKG) structure is designed.

[0065] The TK-MMKG structural design method extends the directed graph. (Where: E is the entity set; R is the relation set; N is the attribute set; S is the set of text and attribute values; W) R W is the set of relation triples; N(As a set of attribute triples), and further subdivided into two implementation paradigms, T-MMKG and K-MMKG, to provide a unified and scalable representation foundation for the systematic organization of multimodal knowledge.

[0066] The implementation paradigm of T-MMKG is as follows: the classified Thangka images are defined as attributes of the corresponding entities, and the Thangka images are associated with the entities.

[0067] T-MMKG defines an image as a graph attribute value, representing it as a directed graph. , and S KG It is a set of attribute values ​​of a knowledge graph, S MM It is a collection of multimodal data.

[0068] The implementation paradigm of K-MMKG is as follows: the classified Thangka images are defined as independent entities in the Thangka knowledge graph, and semantic relationships are established with other entities.

[0069] K-MMKG defines images as graph entities, representing them as directed graphs. ,in It is a set of relation triples. It is a collection of entities in a knowledge graph. It is a collection of multimodal data.

[0070] Taking the XX Buddha statue as an example, the entity "Thangka" is associated with the image "XX Buddha statue.jpg" through the hasImage attribute. This image is further treated as an independent entity and is associated with sub-image nodes (such as "treasure A.jpg" and "treasure B.jpg") through visual semantic relationships, forming a hierarchical cross-modal knowledge network.

[0071] In a preferred embodiment, after constructing the Thangka knowledge graph, the method further includes: converting the textual form of each entity into vectors, calculating the semantic similarity between each vector, and merging the two entities if there are two vectors with a semantic similarity greater than a threshold.

[0072] This step involves multimodal knowledge fusion to achieve entity alignment and disambiguation. Taking Manjushri Bodhisattva as an example, see [link to example]. Figure 5 By converting names into vectors and calculating their semantic similarity, it determines whether "Manjushri" is an abbreviation or synonym of "Manjushri Bodhisattva," thus deciding whether to merge them in the knowledge base. This is a key technology used for data cleaning and unification when building knowledge graphs.

[0073] When calculating semantic similarity, the word frequency of each word in the entity relation set can be calculated and a word frequency vector can be generated. Finally, the cosine similarity of similar entities can be calculated. The calculation formula is as follows:

[0074] Where A and B are the word frequency vectors of different entities, A i and B i This represents the word frequency components of A and B. Entity fusion is performed based on a preset similarity threshold: when the calculated value approaches the upper threshold of 1, the entities are considered to be the same object, and fusion and name unification are performed; when the calculated value approaches the lower threshold of 0, the entities are considered to be unrelated, and their independent state is maintained.

[0075] In a preferred embodiment, after constructing the Thangka knowledge graph, the method further includes: converting the Thangka knowledge graph into CSV format and importing it into the Neo4j database for storage using UTF-8 encoding.

[0076] In terms of storage, Neo4j graph database is selected as the core storage carrier, relying on its flexible graph structure data model to adapt to the association characteristics of multimodal knowledge; firstly, the entity and relation data after knowledge fusion are deduplicated, then the clean data is converted into CSV format, and finally imported into Neo4j database in batches with UTF-8 encoding to complete the persistent storage of structured knowledge.

[0077] This method combines the advantages of prompting learning and joint extraction models, achieving higher accuracy and finer-grained knowledge extraction under low-resource conditions. This significantly improves the automation and semantic depth of Thangka knowledge in constructing multimodal knowledge maps. The following experimental comparisons illustrate this: This paper compares the proposed method with traditional methods, including pipelined models: BERT+BILSTM+CRF, BERT+MLP; joint extraction models: GP-linker, ChatGPT, Wenxin Yiyan, Zhipu AI; and image classification comparison models: AlexNet, VGG 19, SEnet.

[0078] The comparative experimental data of the joint extraction model in this embodiment with the methods described above are shown in Table 1. The F1 score for the entity extraction task reaches 0.8682, and the F1 score for the relation extraction task reaches 0.8774, significantly outperforming the various pipelined models and joint extraction models compared. It can be seen that the method proposed in this embodiment demonstrates comprehensive advantages in low-resource Thangka knowledge extraction tasks, enabling more accurate identification of specialized entities and their complex relationships.

[0079] Table 1

[0080] According to a second aspect of the present invention, a low-resource Thangka multimodal knowledge graph construction device is provided, comprising: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0081] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0082] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0083] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0084] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0085] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0086] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0087] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0088] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0089] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for constructing a low-resource Thangka multimodal knowledge graph, characterized in that, include: Acquire original Thangka images and text data from the Thangka field, and preprocess the original Thangka images; The core terminology of Thangka is extracted from authoritative works, the entity categories and their attributes in the core terminology are defined, and inter-class relationships are established through related elements; based on entity categories, attributes and inter-class relationships, a Thangka knowledge graph ontology structure is constructed. Entity categories and inter-class relationships are used as annotation criteria to annotate the preprocessed text data according to preset annotation rules; The preprocessed Thangka images are classified in a fine-grained manner according to the defined entity categories. Using a pre-built joint extraction model, triples consisting of entity category, attribute, and inter-class relationship are extracted from labeled text data; the joint extraction model uses ERNIE as the encoding layer and a cascaded pointer network as the triple extraction layer. Based on the triples, and in accordance with the ontology structure of the Thangka knowledge graph, a Thangka knowledge graph corresponding to the original Thangka images and text data is established.

2. The method according to claim 1, characterized in that, Preprocessing the original Thangka image includes: The original Thangka image is subjected to grayscale conversion and noise reduction processing; The text data in the original Thangka image is recognized by text recognition and converted into editable digital text data.

3. The method according to claim 1, characterized in that, The preprocessed text data is annotated, including: The preprocessed text data is divided into several fields according to chapters and paragraphs; For each field, extract the words corresponding to the entity category from each sentence and treat the words as entities; prioritize establishing relationships between entities in the same sentence, and then establish relationships between entities and entities in other sentences of the same field.

4. The method according to claim 1, characterized in that, Using a pre-built joint extraction model, triples consisting of entity category, attribute, and inter-class relationship are extracted from labeled text data, including: The encoding layer of the joint extraction model encodes the text data to obtain an initial encoded character sequence; The initial encoded character sequence is normalized to obtain the first encoded character sequence. On the first encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the head entity through the pointer network, and obtains the start position and end position of the head entity by threshold determination, thus obtaining the head entity encoding information. The header entity encoding information is fused with the initial encoded character sequence to obtain the second encoded character sequence; On the second encoded character sequence, the triplet extraction layer calculates the probability of each character being the start position and end position of the tail entity through a pointer network, and obtains the start position and end position of the tail entity by threshold determination, thus obtaining the tail entity encoding information. On the second encoded character sequence, the triplet extraction layer calculates the probability of each character as the start position and end position of the relation through a pointer network, and obtains the relation start position and end position by threshold determination, thus obtaining the relation encoding information; Decode the header entity encoding information, tail entity encoding information, and relation encoding information to obtain a triple containing the header entity, tail entity, and relation.

5. The method according to claim 4, characterized in that, The encoding layer of the joint extraction model encodes the text data, including: The encoding layer of the joint extraction model uses ERNIE to mask the core terms in the Thangka field in the text data; The masked text data is encoded to obtain an initial encoded character sequence containing only semantic features.

6. The method according to claim 1, characterized in that, Fine-grained classification of the preprocessed Thangka images includes: The preprocessed Thangka images are divided into corresponding entity categories, and images containing only the Buddha's body are selected from each category.

7. The method according to claim 6, characterized in that, Establish a Thangka knowledge graph corresponding to the original Thangka images and text data, including: The categorized Thangka images are defined as attributes of corresponding entities, and the Thangka images are associated with the entities; or, The categorized Thangka images are defined as independent entities in the Thangka knowledge graph, and semantic relationships are established with other entities.

8. The method according to claim 1, characterized in that, After constructing the Thangka knowledge graph, it also includes: Convert the textual form of each entity into a vector, calculate the semantic similarity between the vectors, and if there are two vectors with a semantic similarity greater than a threshold, then merge the two entities.

9. The method according to claim 1, characterized in that, After constructing the Thangka knowledge graph, it also includes: The Thangka knowledge graph was converted to CSV format and imported into the Neo4j database for storage using UTF-8 encoding.

10. A low-resource Thangka multimodal knowledge graph construction device, characterized in that, include: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 9.

Images