Multimodal cultural relic knowledge graph construction method, device and system and storage medium

Abstract

The application discloses a multi-modal cultural relic knowledge graph construction method, device and system and a storage medium. The method comprises the following steps: acquiring multi-modal data in the cultural relic field, wherein the multi-modal data comprises structured and unstructured cultural relic image-text data; processing the unstructured cultural relic data into the form of an image-text pair, and converting the structured cultural relic data into the form of multi-modal triples; constructing an ontology in the cultural relic field and forming a multi-modal cultural relic ontology model; performing entity extraction and relation extraction on the unstructured cultural relic data by using a multi-modal knowledge extraction network based on target guidance; and fusing the triples formed through extraction and the multi-modal triples obtained by converting the structured cultural relic data to form a complete multi-modal cultural relic knowledge graph. The application can solve the problems of insufficient cultural relic knowledge expression ability of a single mode and low knowledge extraction accuracy. Meanwhile, the multi-modal knowledge graph constructed can enrich downstream tasks in multiple purposes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer knowledge graph, and in particular to a multi-modal cultural relic knowledge graph construction method, device and system and a storage medium. BACKGROUND

[0002] The knowledge graph technology stores the factual knowledge of the real existing entity and its relationship, and constructs a structured semantic knowledge base, so as to realize the conversion from information to knowledge, effectively realize the organization and management of information resources, and further realize the fusion and sharing of knowledge. In recent years, there are many knowledge graph applications in different fields. For example, the cultural relic knowledge graph can use the triples formed by the cultural relic entity and the relationship between the entities to construct a cultural relic association knowledge base, use the powerful semantic expression and knowledge interconnection ability of the knowledge graph to "connect" the isolated information points of the cultural relics, interpret the cultural connotation behind the cultural relics, maximize the interconnection between the cultural relics, expand the depth and breadth of cultural relic knowledge, and realize the intelligent fusion and sharing of cultural relic information.

[0003] Chinese patent application CN109657068A discloses a cultural relic knowledge graph generation and visualization method for a smart museum. The method acquires cultural relic data, and uses a knowledge extraction tool to extract entity, attribute, relationship and other information. However, this method only collects the text materials of cultural relics as the content to be extracted, but using only the single text modality of cultural relics is not enough to express the rich visual information contained in cultural relics, and there is a problem of low accuracy of graph construction. At the same time, the single modality cultural relic knowledge graph cannot meet more complex downstream tasks such as cultural relic intelligent identification, cultural relic history background mining, and cultural relic visualization. Therefore, there is an urgent need in the existing cultural relic knowledge graph field to construct a multi-modal cultural relic graph by using the information of multiple modalities of cultural relics to solve the above problems. Since the multi-modal information contained in cultural relics is mainly image with obvious subject and easy to standardize structure, it is a feasible choice to construct a multi-modal cultural relic knowledge graph facing image and text. In addition, due to the heterogeneity of information between different modalities, special inter-modal fusion strategies need to be formulated for the neural network in multi-modal knowledge graph extraction, fusion and construction, so as to reduce the labor cost of constructing cultural relic graph, improve the accuracy of knowledge graph construction, and make the finally constructed multi-modal knowledge graph meet the characteristics of rich knowledge within the modalities and complementary knowledge between the modalities. SUMMARY

[0004] In order to solve the above problems, the application provides a multi-modal cultural relic knowledge graph construction method for text and image, wherein multi-modal cultural relic data of structured and unstructured is collected, and a multi-modal cultural relic ontology is constructed in combination with field experts to guide knowledge processing, knowledge fusion, knowledge extraction and other stages in the automatic construction process of the knowledge graph, and multi-modal knowledge graphs of different sources are fused and completed.

[0005] In order to achieve the above object, the application adopts the following technical solutions:

[0006] According to a first aspect of the application, a multi-modal cultural relic knowledge graph construction method is provided, which comprises: acquiring multi-modal data in the cultural relic field, the multi-modal data in the cultural relic field including structured multi-modal data in the cultural relic field and unstructured multi-modal data in the cultural relic field, processing the unstructured multi-modal data in the cultural relic field into the form of image-text pairs, and converting the structured cultural relic data into the form of multi-modal triples; constructing a multi-modal ontology in the cultural relic field and forming a multi-modal cultural relic ontology model; using the multi-modal cultural relic ontology model and a multi-modal knowledge extraction network based on target guidance to perform entity extraction, attribute extraction and relation extraction on the unstructured cultural relic data; fusing the multi-modal triples formed through entity extraction, attribute extraction and relation extraction with the multi-modal triples obtained by converting the structured cultural relic data, and forming a complete multi-modal cultural relic knowledge graph.

[0007] Further, the acquisition of multi-modal data in the cultural relic field, the multi-modal data in the cultural relic field including structured multi-modal data in the cultural relic field and unstructured multi-modal data in the cultural relic field, the processing of the unstructured multi-modal data in the cultural relic field into the form of image-text pairs, and the conversion of the structured cultural relic data into the form of multi-modal triples specifically comprises: based on existing data sources, extracting cultural relic images and corresponding text descriptions of cultural relics, and extracting cultural relic attribute values and corresponding cultural relic attribute labels through structured data segments contained in the data sources, the structured data segments including tables and labels; removing content irrelevant to semantics in the original text content; using a saliency network to extract the main part of the image for each picture.

[0008] Further, the construction of the multi-modal artifact domain ontology and the formation of a multi-modal artifact ontology model specifically include: based on the main extraction targets in the artifact entity, artifact relationship, artifact attribute, and artifact legend, the relationship between artifact images, and the relationship between artifact text regions and image regions, an original multi-modal artifact ontology library is constructed; the existing structured artifact knowledge base and open unstructured artifact knowledge base are collected to extract artifact entity information, artifact relationship information, and artifact attribute information; artifact entity information, artifact relationship information, and artifact attribute information with high confidence are selected as candidate supplementary knowledge, and are fused with the original multi-modal artifact ontology library to form a multi-modal artifact ontology library.

[0009] Further, the multi-modal artifact ontology model and the target-guided multi-modal knowledge extraction network are used to perform entity extraction, attribute extraction, and relationship extraction on unstructured artifact data, specifically including: the cleaned multi-modal artifact text data is encoded by a pre-trained language model, and each character is encoded as a character vector for representation; artifact pictures matching the text are detected by a target detection network to identify artifact subjects and feature point bounding boxes, and are uniformly scaled to the same image size; all nominal phrases are obtained from the text data by a syntax parser, and the feature representation corresponding to the nominal phrases is obtained according to the encoding result; a graph is constructed between the artifact text and the text, the text and the image, and the image and the image, wherein the nodes of the graph are text target phrases or visual regions, and the edges are the relationships between the two nodes, corresponding to the pre-defined relationships in the multi-modal artifact ontology library; the obtained graph structure is input into a double-flow Transformer network for intra-modal and inter-modal feature interaction and fusion, and the fused text sequence and image region sequence features are decoded by a CRF conditional random field to obtain output multi-modal triples, wherein all triples comply with the knowledge expression form in the artifact ontology library.

[0010] Further, the cleaned multi-modal artifact text data is encoded by a pre-trained language model, and each character is encoded as a character vector for representation, and artifact pictures matching the text are detected by a target detection network to identify artifact subjects and feature point bounding boxes, and are uniformly scaled to the same image size, specifically satisfying the following definitions:

[0011]

[0012]

[0013] wherein LanguageModel is a pre-trained language model, x i is the output character vector, ObjectDetection represents a target detection model, each piece of original text Corresponding to K original pictures, all pictures detect N targets, and the characteristics are O i .

[0014] Further, the obtained graph structure is input into a double-flow Transformer network for intra-modal and inter-modal feature interaction and fusion. The fused text sequence and image region sequence features are decoded by a CRF conditional random field to obtain an output multi-modal triple, specifically including: decoding all text node states decoding all image region node states wherein l represents the lth layer of the Transformer fusion block, and the feature representation of the text context is represented as:

[0015]

[0016] wherein MultiHead(·) represents a multi-head self-attention function,

[0017] The context feature representation of the image region node is represented as:

[0018]

[0019] After a certain number of intra-modal fusion layers, inter-modal fusion layers are used to fuse inter-modal features, and the specific representation is:

[0020]

[0021]

[0022] wherein, represents a set composed of adjacent nodes of , represents the feature representation of the lth layer after inter-modal cross fusion, represents the feature representation of the text modal context, and a i,j represents a cross-modal fusion coefficient, represents the context feature representation of the image region node, and represents a dot product operation, and sigmoid(·) represents an activation function, represents a parameter matrix to be learned; The fused representation of the text and the image is decoded by a CRF conditional random field to obtain a joint probability distribution of each label, and the specific representation is:

[0023]

[0024]

[0025] wherein y represents a real label sequence, y i-1 represents the real value of the i-1th label in the sequence, and y i ​the true value of the i-th label in the sequence, y' represents all possible labels that can occur, y' i-1 the possible value of the i-1-th label in the sequence, y' i the possible value of the i-1-th label in the sequence, representing a potential function, the training goal is to make the conditional probability distribution of the input closer to the true value, the model solves the entity recognition overlap problem by establishing different CRF layers for each recognized entity, and finally constructs a graph representation as follows: where ε represents an entity of any modality, representing the relationship between entities, representing the attribute, representing a relationship triple, representing an attribute triple, wherein the image entity and the text entity have the same representation status.

[0026] Further, the multi-modal triples formed by entity extraction, attribute extraction and relationship extraction are fused with the multi-modal triples converted from the structured cultural relic data between graphs to form a complete multi-modal cultural relic knowledge graph, specifically including: the multi-modal knowledge graph extracted is split into a subgraph composed of a single specific cultural relic; the subgraph composed of a single cultural relic is compared with the input layer of the structured cultural relic data graph, when a conflict value is encountered, the structured data is fused as a higher credibility part, and a similarity recognition algorithm is used to compare the pictures to remove the same pictures.

[0027] According to a second aspect of the present application, a multi-modal cultural relic knowledge graph construction device is provided, the device comprising a processor configured to: acquire multi-modal data in the cultural relic field, the multi-modal data in the cultural relic field including structured multi-modal data in the cultural relic field and unstructured multi-modal data in the cultural relic field, process the unstructured multi-modal data in the cultural relic field into an image-text pair form, and convert the structured cultural relic data into a multi-modal triple form; construct a multi-modal cultural relic ontology model by constructing a multi-modal cultural relic ontology; perform entity extraction, attribute extraction and relationship extraction on the unstructured cultural relic data by using the multi-modal cultural relic ontology model and a multi-modal knowledge extraction network based on target guidance; and fuse the multi-modal triples formed by entity extraction, attribute extraction and relationship extraction with the multi-modal triples converted from the structured cultural relic data between graphs to form a complete multi-modal cultural relic knowledge graph.

[0028] According to a third aspect of the present application, a multi-modal cultural relic knowledge graph construction system is provided, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method as described above.

[0029] According to a fourth aspect of the present application, there is provided a readable storage medium storing one or more programs executable by one or more processors to implement the method as described above.

[0030] The present application has the following advantages:

[0031] 1. The application discloses a multi-modal cultural relic knowledge graph construction method, which enriches the representation form of the cultural relic graph by utilizing the rich multi-modal information on the Internet, so that the constructed multi-modal cultural relic knowledge graph can better meet multi-purpose downstream tasks such as visualization and cultural relic intelligent retrieval.

[0032] 2. The application proposes a target-guided multi-modal knowledge extraction network to meet the demand for automatic knowledge extraction in the knowledge graph construction process, and can utilize multi-modal features such as cultural relic images to enhance the efficiency of knowledge extraction. BRIEF DESCRIPTION OF DRAWINGS

[0033] The application will be further described below in conjunction with the drawings and embodiments:

[0034] Figure 1 A flowchart of a multi-modal cultural relic knowledge graph construction method according to the application is shown in the figure;

[0035] Figure 2 A data flowchart of a multi-modal cultural relic knowledge graph construction method according to the application in the construction process is shown in the figure;

[0036] Figure 3 A multi-modal knowledge extraction network architecture diagram based on target guidance according to the application is shown in the figure. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0038] If not specifically indicated, the technical means used in the examples is the conventional means known to those skilled in the art.

[0039] The multi-modal cultural relic knowledge graph construction method provided by the embodiments of the application can include the following steps, as shown in Figure 1 and Figure 2

[0040] ​Step S101: Obtain multi-modal data in the field of cultural relics, the multi-modal data in the field of cultural relics including structured multi-modal data in the field of cultural relics and unstructured multi-modal data in the field of cultural relics, process the unstructured multi-modal data in the field of cultural relics into the form of image-text pairs, and convert the structured cultural relic data into the form of multi-modal triples. It should be noted that the multi-modal data in the field of cultural relics at least includes image data and text data.

[0041] Step S102: Construct a multi-modal ontology of cultural relics and form a multi-modal cultural relic ontology model.

[0042] Step S103: Perform entity extraction, attribute extraction and relation extraction on unstructured cultural relic data by using the multi-modal cultural relic ontology model and a multi-modal knowledge extraction network based on target guidance.

[0043] Step S104: Integrate the multi-modal triples formed by entity extraction, attribute extraction and relation extraction with the multi-modal triples converted from the structured cultural relic data to form a complete multi-modal cultural relic knowledge graph.

[0044] The four steps will be described in detail below to fully illustrate the feasibility and progressiveness of the present application.

[0045] In step S101, first, original image-text pairs and corresponding cultural relic attribute relations and other data are extracted from archaeological records and Internet data, and then the original text content is preprocessed by removing punctuation marks and other irrelevant contents in the text. The text preprocessing can be performed by using an NLP preprocessing tool such as Stanford parser. For the original image information, the main part of the image is extracted by using a saliency network, and the saliency network can use the adaptive threshold segmentation algorithm provided by OpenCV. The original image-text pairs and corresponding cultural relic attribute relations and other data extracted from the archaeological records and Internet data satisfy the following definitions:

[0046] Es::=<Et,Eo,Rs>

[0047] Rs::={Ra,Rr}

[0048] Where Es is a triple representation in the cultural relic knowledge graph, Et is a text representation of all cultural relic entities, Eo is an image representation of the cultural relic entity, Rs is a relationship and attribute value in the cultural relic graph, Ra is an attribute edge in the cultural relic graph, and Rr is a relationship edge in the cultural relic graph.

[0049] In step S102, first, the original multi-modal cultural relic ontology knowledge base is constructed by the cultural relic attribute and the representative legend compiled by cultural relic experts manually, and a certain number of target annotations are made on the cultural relic legend, including cultural relic main target frame, cultural relic attribute feature area, etc.; then the existing structured knowledge base is used to extract cultural relic entity, relationship, attribute and other information; and the objects with high confidence in the existing knowledge base are selected as candidate supplementary knowledge, and are fused with the multi-modal ontology library to form a multi-modal cultural relic corpus with a certain scale.

[0050] In step S103, first, the cleaned multi-modal cultural relic text data is encoded by a pre-trained language model, and each character is encoded as a character vector for representation; second, the cultural relic pictures matched with the text are detected by a target detection network such as FastRCNN to identify the cultural relic main body and feature point bounding box, and are uniformly scaled to the same image size; and the text data is parsed by a syntax parser to obtain all noun phrases, and the feature representation corresponding to the noun phrases is obtained according to the encoding result of the character vector; then a graph is constructed between the cultural relic text and the text, the text and the image, and the image and the image, wherein the nodes of the graph are the text target phrases or visual regions, and the edges are the relationships between the two nodes, and the construction result is Figure 3 As shown in the bottom, the hollow circle represents the text node, and the solid circle represents the image region node; and the obtained graph structure is input into a double-flow Transformer network for intra-modal and inter-modal feature fusion, and the fused features are decoded by a CRF conditional random field to obtain output multi-modal triples, as shown in the process of Figure 3 .

[0051] In step S103, the cleaned multi-modal cultural relic text data is encoded by a pre-trained language model, and each character is encoded as a character vector for representation, and the cultural relic pictures matched with the text are detected by a target detection network to identify the cultural relic main body and feature point bounding box, and are uniformly scaled to the same image size, which satisfies the following definitions:

[0052]

[0053]

[0054] wherein LanguageModel is a pre-trained language model, x i is the output character vector, ObjectDetection represents a target detection model, each original text may correspond to K original pictures, and N targets are detected in all pictures, and the features are O i .

[0055] In step S103, the obtained graph structure is input into a two-stream Transformer network to perform intra-modal and inter-modal feature fusion, and the fused features are decoded using a CRF (Conditional Random Field) to obtain the output multimodal triples, as shown below:

[0056] State of all text nodes For the node status of all image regions Where l represents the Transformer fusion block of the l-th layer, the features fused with the text context can be represented as:

[0057]

[0058] Where MultiHead(·) represents the multi-head self-attention function, similarly, the contextual features of nodes in an image region can be represented as:

[0059]

[0060] After a certain number of intramodal fusion layers, an intermodal fusion layer is used to fuse the intermodal features, specifically as follows:

[0061]

[0062]

[0063] in, Representative by The set of adjacent nodes, This represents the feature representation of the l-th layer after intermodal cross-fusion. The feature representation representing the text modal context, α i,j Represents the fusion coefficient across modalities. The contextual feature representation of a node in an image region is represented by ⊙, which represents the dot product operation, and sigmoid(·) represents the activation function. This represents the parameter matrix to be learned;

[0064] Finally, the fused representation of text and image is decoded using a CRF (Conditional Random Field) to obtain the joint probability distribution of each label, which can be specifically represented as:

[0065]

[0066] Where y represents the true label sequence, y i-1 y represents the true value of the (i-1)th label in the sequence. i Let y' represent the true value of the i-th label in the sequence, and let y' represent all possible labels. i-1 Let y′ represent the possible values ​​of the (i-1)th label in the sequence.i a possible value of the i-1th tag in the representative sequence, represent a potential function, the goal of training is to make the conditional probability distribution of input closer to the true value, the model solves the entity recognition overlap problem by establishing different CRF layers for each recognized entity, and finally constructs a graph representation as follows: where epsilon represents an arbitrary modal entity, represent the relationship between entities, represent the attribute, represent the relationship triple, represent the attribute triple, and the image entity and the text entity have the same representation status.

[0067] In step S104, first, the extracted multi-modal knowledge graph is divided into a subgraph composed of a single specific cultural relic; then the subgraph composed of a single cultural relic is compared with the structured cultural relic data graph of the input layer, when a conflict value is encountered, the structured data is fused as the part with higher credibility, and the similarity recognition algorithm is used to compare the pictures to remove the same pictures.

[0068] In summary, the present application proposes a multi-modal knowledge graph construction method to solve the technical problems of weak representation ability of single modal construction of cultural relic graph, low accuracy of automatic extraction, etc. Starting from the introduction of cultural relic ontology, considering the use method of multi-modal knowledge in each stage of knowledge graph construction, special fusion strategy is formulated for neural network in knowledge graph extraction, fusion and construction, to achieve the purpose of efficient and accurate construction of multi-modal cultural relic knowledge graph.

[0069] The embodiment of the present application provides a multi-modal cultural relic knowledge graph construction device, the device comprises a processor, the processor is configured to: acquire multi-modal data in the cultural relic field, the multi-modal data in the cultural relic field includes structured multi-modal data in the cultural relic field and unstructured multi-modal data in the cultural relic field, process the unstructured multi-modal data in the cultural relic field into the form of image-text pair, and convert the structured cultural relic data into the form of multi-modal triple; construct a multi-modal cultural relic ontology and form a multi-modal cultural relic ontology model; adopt the multi-modal cultural relic ontology model and a multi-modal knowledge extraction network based on target guidance to perform entity extraction, attribute extraction and relationship extraction on the unstructured cultural relic data; perform graph fusion between the multi-modal triple formed through entity extraction, attribute extraction and relationship extraction and the multi-modal triple obtained by converting the structured cultural relic data, and form a complete multi-modal cultural relic knowledge graph.

[0070] The processor can be a processing device including one or more general-purpose processing devices, such as microprocessors, central processing units (CPUs), graphics processing units (GPUs), and the like. More particularly, the processor can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor running other instruction sets, or processors running a combination of instruction sets. The processor can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), and the like. The processor can be communicatively coupled to the memory and configured to execute computer-executable instructions stored thereon to perform the multi-modal cultural relic knowledge graph construction method of the above embodiments.

[0071] The embodiments of the present application provide a multi-modal cultural relic knowledge graph construction system, the system comprising: a memory for storing a computer program; a processor for executing the computer program to implement the method as described above.

[0072] The embodiments of the present application provide a readable storage medium, the readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the method as described above.

[0073] The above-described embodiments are merely preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A method for constructing a multimodal cultural relic knowledge graph, characterized in that: The method comprises: acquiring multi-modal data in the field of cultural relics, the multi-modal data in the field of cultural relics including structured multi-modal data in the field of cultural relics and unstructured multi-modal data in the field of cultural relics, processing the unstructured multi-modal data in the field of cultural relics into the form of image-text pairs, and converting the structured cultural relic data into the form of multi-modal triples; constructing a multi-modal cultural relic ontology and forming a multi-modal cultural relic ontology model; performing entity extraction, attribute extraction and relation extraction on unstructured cultural relic data by using the multi-modal cultural relic ontology model and a multi-modal knowledge extraction network based on target guidance; fusing the multi-modal triples formed through entity extraction, attribute extraction and relation extraction with the multi-modal triples converted from the structured cultural relic data to form a complete multi-modal cultural relic knowledge graph; the entity extraction, attribute extraction and relation extraction on unstructured cultural relic data by using the multi-modal cultural relic ontology model and the multi-modal knowledge extraction network based on target guidance specifically comprises: encoding the cleaned multi-modal cultural relic text data through a pre-trained language model to represent each character as a character vector; detecting the cultural relic pictures matched with the text through a target detection network to identify the cultural relic main body and feature point bounding boxes and uniformly scale them to the same image size; obtaining all nominal phrases from the text data through a syntax parser and taking out the feature representation corresponding to the nominal phrases according to the encoding result; constructing a graph between cultural relic text and text, text and image, and image and image, wherein the nodes of the graph are text target phrases or visual areas, and the edges are the relations between the two nodes, which correspond to the pre-defined relations in the multi-modal cultural relic ontology library; inputting the obtained graph structure into a double-stream Transformer network to perform intra-modal and inter-modal feature interaction and fusion, and decoding the fused text sequence and image region sequence features through a CRF conditional random field to obtain output multi-modal triples, wherein all the triples comply with the knowledge expression form in the cultural relic ontology library. 2.The method of claim 1, wherein: The acquisition of multi-modal data in the field of cultural relics, the multi-modal data in the field of cultural relics including structured multi-modal data in the field of cultural relics and unstructured multi-modal data in the field of cultural relics, processing the unstructured multi-modal data in the field of cultural relics into the form of image-text pairs, and converting the structured cultural relic data into the form of multi-modal triples specifically comprises: based on the existing data source, extracting cultural relic images and cultural relic corresponding text descriptions, and extracting cultural relic attribute values and corresponding cultural relic attribute labels through structured data segments contained in the data source, the structured data segments including tables and labels; removing the text content irrelevant to semantics from the original text content; using a saliency network to extract the main part of the image for each picture. 3.The method of claim 1, wherein: The construction of the multi-modal ontology in the field of cultural relics and the formation of the multi-modal cultural relic ontology model specifically comprise: based on the main extraction targets in the cultural relic entities, cultural relic relations, cultural relic attributes and cultural relic legends, the relations between cultural relic images, and the relations between cultural relic text areas and image areas, constructed by experts in the field of cultural relics, an original multi-modal cultural relic ontology library is constructed; Collect the existing structured cultural relic knowledge base and the open unstructured cultural relic knowledge base through information extraction to extract cultural relic entity information, cultural relic relationship information and cultural relic attribute information; Select cultural relic entity information, cultural relic relationship information and cultural relic attribute information with high confidence as candidate supplementary knowledge, and fuse the candidate supplementary knowledge with the original multi-modal cultural relic ontology to form a multi-modal cultural relic ontology. 4.The method of claim 1, wherein: The cleaned multi-modal cultural relic text data is encoded through a pre-trained language model, each character is encoded as a character vector for representation, and cultural relic pictures matched with the text are detected through a target detection network to identify cultural relic main bodies and feature point bounding boxes, and are uniformly scaled to the same image size, which specifically satisfies the following definitions: ； ； wherein LanguageModel is a pre-trained language model, is the output word vector, ObjectDetection represents the object detection model, each piece of original text corresponds to Zhang original picture, all pictures are detected objects, characterized in that . 5.The method of claim 1, wherein: The obtained graph structure is input into a dual-flow Transformer network for intra-modal and inter-modal feature interaction and fusion, and the fused text sequence and image region sequence features are decoded through a CRF conditional random field to obtain output multi-modal triples, which specifically include: For all text node states For all image region node states where denotes the layer of the Transformer fusion block, which fuses the feature representations of the text context as: ； wherein represents a multi-head self-attention function; The context feature representation of the image region node is: ； After a certain number of intra-modal fusion layers, inter-modal fusion layers are used to fuse inter-modal features, which are specifically represented as: ； wherein, represents a set of neighboring nodes of represents the first layer after inter-modal cross-fusion, represents a feature representation of a text modality context, represents a fusion coefficient across inter-modal, represents a context feature representation of an image region node, represents a dot product operation, ) represents an activation function, , represents a parameter matrix to be learned;​ The fusion representation of the text and the image is decoded through a CRF conditional random field to obtain the joint probability distribution of each label, which is specifically represented as: ； wherein, represents the real label sequence, represents the real value of the th label in the sequence, represents the real value of the th label in the sequence, represents all possible labels, represents the possible value of the th label in the sequence, represents the possible value of the th label in the sequence, represents the potential function, the training target is to make the conditional probability distribution of the input closer to the real value, the model solves the entity recognition overlap problem by establishing different CRF layers for each recognized entity, and finally constructs a graph representation as follows: wherein, represents an arbitrary modal entity, represents the relationship between entities, represents the attribute, represents a relationship triple, represents an attribute triple, wherein the image entity and the text entity have the same representation status. 6.The method of claim 1, wherein: The multi-modal triples formed through entity extraction, attribute extraction and relationship extraction are fused with the multi-modal triples obtained by converting the structured cultural relic data to form a complete multi-modal cultural relic knowledge graph, which specifically includes: The extracted multi-modal knowledge graph is split into subgraphs composed of single specific cultural relics; The subgraph composed of a single cultural relic is compared with the structured cultural relic data graph of the input layer, when a conflict value is encountered, the structured data is fused as the higher confidence part, and the similarity recognition algorithm is used to compare the pictures to remove the same pictures.

7. A multi-modal cultural relic knowledge graph construction apparatus, characterized in that: The device includes a processor configured to: Obtain multi-modal data in the cultural relic field, which includes structured multi-modal data in the cultural relic field and unstructured multi-modal data in the cultural relic field, process the unstructured multi-modal data in the cultural relic field into an image-text pair form, and convert the structured cultural relic data into a multi-modal triple form; Construct a multi-modal cultural relic ontology and form a multi-modal cultural relic ontology model; Use the multi-modal cultural relic ontology model and the target-guided multi-modal knowledge extraction network to perform entity extraction, attribute extraction and relationship extraction on unstructured cultural relic data; Fuse the multi-modal triples formed through entity extraction, attribute extraction and relationship extraction with the multi-modal triples obtained by converting the structured cultural relic data to form a complete multi-modal cultural relic knowledge graph; The use of the multi-modal cultural relic ontology model and the target-guided multi-modal knowledge extraction network to perform entity extraction, attribute extraction and relationship extraction on unstructured cultural relic data specifically includes: The cleaned multi-modal cultural relic text data is encoded through a pre-trained language model, each character is encoded as a character vector for representation; The relic picture matched with the text is detected by a target detection network to identify a relic main body and a feature point bounding box, and is uniformly scaled to the same image size; All nominal phrases are obtained by a syntax parser on the text data, and a feature representation corresponding to the nominal phrases is taken according to an encoding result; A graph is constructed between the relic text and the text, the text and the image, and the image and the image, wherein a node of the graph is a text target phrase or a visual area, and an edge is a relationship between two nodes, corresponding to a pre-defined relationship in a multi-modal relic ontology library; The obtained graph structure is input into a double-flow Transformer network for intra-modal and inter-modal feature interaction and fusion, and a fused text sequence and image area sequence feature are decoded by a CRF conditional random field to obtain an output multi-modal triple, wherein all the triples comply with a knowledge expression form in the relic ontology library. 8.A multi-modal cultural relic knowledge graph construction system, characterized in that: The system comprises: a memory for storing a computer program; a processor for executing the computer program to implement the method of any one of claims 1 to 6.

9. A readable storage medium characterized by: The readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 6. The readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 6.

Images