Multi-modality-based minority non-abandoned pattern knowledge graph construction method and multi-modality-based minority non-abandoned pattern knowledge graph construction system

Abstract

The invention discloses a multi-modality-based minority non-abandoned pattern knowledge graph construction method and system, and relates to the technical field of cultural heritage digital protection. Deep association of multi-modality knowledge is realized through a double-path entity relationship extraction mechanism, context semantics are coded by a text path by utilizing a pre-training language model, and a multi-modality knowledge graph is constructed; the method comprises the following steps: accurately extracting entities such as a pattern and an inheritor, a semantic relationship and a visual path, analyzing a pattern topological structure through a graph convolutional network, converting visual features such as lines and contours into structured relationship data, calculating cosine similarity of a text and a visual feature vector through comparative learning, establishing cross-modal mapping of the visual features and culture description, and obtaining a visual feature model; according to the mechanism, the knowledge graph simultaneously contains semantic logic and visual feature association, and construction of a complete knowledge chain from a pattern form to cultural connotation is realized.

Description

Technical Field

[0001] This invention relates to the field of digital preservation technology for cultural heritage, specifically to a method and system for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodality. Background Technology

[0002] Intangible cultural heritage patterns of ethnic minorities are cultural crystallizations formed by various ethnic groups in their long-term production and life. They encompass rich connotations such as totem worship, religious beliefs, and folk customs. These patterns are not only decorative elements on clothing, architecture, and artifacts, but also important symbols of ethnic identity and cultural inheritance. With the development of digital technology, the systematic protection and utilization of intangible cultural heritage patterns has become an important issue in cultural inheritance. Knowledge graphs, as an efficient way of organizing and displaying knowledge, provide the possibility of integrating scattered pattern information and exploring its deep cultural connections.

[0003] Existing technologies have significant limitations in constructing intangible cultural heritage knowledge graphs. In multimodal data processing, most methods only analyze single-type data, lacking the ability to collaboratively process text, images, audio, and video. This leads to the fragmentation of semantic connections between different modalities. For example, while patent document CN117851609A mentions multi-source heterogeneous data fusion, it focuses on solving entity disambiguation problems through semantic similarity models, failing to achieve in-depth mapping of cross-modal features. This makes it difficult to accurately associate the visual topological structure of pattern images with the cultural connotations of textual descriptions. Furthermore, patent document CN1173... While 12578B utilizes convolutional neural networks and Transformers to process multimodal information, it focuses more on predicting the overall inheritance path of intangible cultural heritage. It lacks sufficient mining of multimodal attributes of specific elements such as patterns. In terms of entity relationship extraction, existing methods rely on manual annotation or simple rule matching. For example, CN117851609A analyzes entity relationships through text classification and context relationship models, but does not combine graph convolutional networks to parse the topological structure relationships of pattern images. It is difficult to handle the entity associations implicit in the visual modality, and the authenticity of entity information is difficult to verify, which affects the application value of knowledge graphs.

[0004] In summary, existing technologies are insufficient to meet the multi-dimensional and in-depth protection and utilization needs of ethnic minority intangible cultural heritage patterns. How to break through the limitations of a single modality, achieve effective integration of multi-source data, and improve the accuracy of entity relationship extraction and the credibility of knowledge graphs have become urgent problems to be solved in constructing high-quality knowledge graphs of ethnic minority intangible cultural heritage patterns. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for constructing a knowledge graph of ethnic minority intangible cultural heritage patterns based on multimodality. It can achieve deep association of multimodal knowledge through a dual-path entity relationship extraction mechanism. The text path uses a pre-trained language model to encode contextual semantics and accurately extract entities and semantic relationships such as patterns and inheritors. The visual path uses a graph convolutional network to parse the topological structure of patterns and transforms visual features such as lines and contours into structured relational data. Then, through comparative learning, the cosine similarity between text and visual feature vectors is calculated to establish a cross-modal mapping between visual features and cultural descriptions. This mechanism enables the knowledge graph to simultaneously contain semantic logic and visual feature associations, realizing the construction of a complete knowledge chain from pattern form to cultural connotation.

[0006] To solve the above-mentioned technical problems, this invention provides the following technical solution: On one hand, a method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodality, the specific steps of which are as follows:

[0007] S100. Collect original multimodal data of intangible cultural heritage patterns of ethnic minorities, including text descriptions, pattern images, associated audio and video, and preprocess the multimodal data to generate structured preprocessed data;

[0008] S200. Based on the structured preprocessed data, perform dual-path entity relationship extraction:

[0009] Text path and semantic relationship: Extract entities and semantic relationships of patterns, cultural background, inheritors, and regions through a pre-trained language model;

[0010] Visual path and attribute relationship: Graph convolutional networks are used to analyze the topological relationships of pattern images, and cross-modal attribute mapping between text descriptions and pattern images is achieved through contrastive learning;

[0011] S300: Construct an ontology model with patterns as the core, define entity types and attribute relationships, inject the entities, semantic relationships and cross-modal attribute mapping results extracted in S200 into the ontology model, establish a triplet structure and store it in the knowledge graph.

[0012] S400: Employing embedded vector technology, entities, semantic relationships, and attribute relationships in the graph are mapped into low-dimensional vectors, generating pattern semantic vectors, visual feature vectors, and cultural association vectors, which are then stored in a vector database.

[0013] S500 utilizes blockchain technology to build a trust channel and performs three-level trust authentication on entities in the vector database.

[0014] Furthermore, in step S100, the collection of multimodal raw data of intangible cultural heritage patterns of ethnic minorities specifically includes:

[0015] Textual data: Descriptions of pattern names, eras, regions, and folk meanings from intangible cultural heritage databases, as well as survey reports provided by local authorities;

[0016] Image data: Pattern images from scanned copies of ancient books, photos of intangible cultural heritage objects, and digital design drawings by inheritors;

[0017] Audio data: Related audio from interview recordings of inheritors and recordings of folk rituals;

[0018] Video data: Related videos from recordings of the pattern-making process and oral histories by inheritors.

[0019] Furthermore, in S100, the preprocessing of the raw multimodal data is as follows:

[0020] When processing text data, the text descriptions in minority languages ​​are translated into the common language. A combination of manual annotation and machine-assisted annotation is used to annotate the entities in the text, such as pattern names, cultural background keywords, names of inheritors, and regional names, to form text data with entity labels.

[0021] When processing image data, after improving the clarity of blurry pattern images, the pattern subject and background in the image are segmented, and the feature information of the pattern's outline, lines, and colors is extracted to generate structured image feature data.

[0022] When processing audio and video data, the speech content in the audio is converted into text, and the video is analyzed frame by frame to extract key frames of pattern features, production process and cultural scene. The converted text and key frames are preprocessed in the same way as text data and image data.

[0023] Text data and image feature data with entity labels are integrated to generate structured preprocessed data.

[0024] Furthermore, in step S200, the specific process of performing dual-path entity relationship extraction is as follows:

[0025] For text path and semantic relation extraction: Input the text data with entity labels generated by S100 into the pre-trained language model. Through the contextual semantic encoding capability of the pre-trained language model, identify and extract the following content:

[0026] Entity types: patterns, cultural background, inheritors, regions;

[0027] Semantic relations: pattern-belongs to-region, inheritor-inherits-pattern, pattern-has-cultural background. Output text entity relation set, including entity tags and semantic relation triples;

[0028] For visual path and attribute relationship extraction: The structured image feature data generated by S100 is input into a graph convolutional network. A graph model is constructed with the topological structure of the pattern as nodes. Nodes are defined as basic elements in the pattern, and edges are defined as spatial positional relationships and structural associations between elements. Through the neighborhood information aggregation capability of the graph convolutional network, the topological structure relationship of the pattern is analyzed. At the same time, based on the visual features of the pattern, visual attribute relationships of pattern-having-symmetric attributes and pattern-containing-geometric elements are extracted, and a set of visual entity relationships is output, which includes topological structure relationship triples and visual attribute relationship triples.

[0029] Perform cross-modal association between the output text entity relation set and the input visual entity relation set:

[0030] Feature encoding: converting text entities into text feature vectors and visual entities into visual feature vectors;

[0031] Similarity calculation: Calculate the cosine similarity between the text feature vector and the visual feature vector;

[0032] Attribute mapping: Associating text entities with similarity reaching a threshold with visual attributes, ultimately generating a dual-path fusion entity relationship set, including text semantic relationships, visual topological / attribute relationships, and cross-modal association relationships.

[0033] Furthermore, in S300, the specific process of constructing the ontology model and knowledge graph storage centered on the pattern is as follows:

[0034] When constructing the ontology model, patterns are used as the core nodes to clearly define entity types, including four major categories: patterns, cultural background, inheritors, and regions. Specific attributes are defined for each type of entity.

[0035] Pattern attributes: pattern name, pattern medium, pattern era, and pattern symbolism description;

[0036] Cultural background attributes: period of cultural origin, ethnic group to which the culture belongs, core cultural connotations, and related folk customs;

[0037] Inheritor attributes: Name, gender, date of birth, level of inherited skills, and years of inheritance;

[0038] Regional attributes: region name, administrative division, geographical features, and ethnic distribution;

[0039] At the same time, the attribute relationships between entities are defined, including the relationship between patterns and cultural background, the relationship between patterns and inheritors, the relationship between patterns and regional origins, and the relationship between inheritors and regional affiliation, forming a complete ontology framework;

[0040] When injecting entities and relationships, the entities in the dual-path fusion entity relationship set generated by S200 are included in the corresponding entity types of the ontology model, and semantic relationships, visual topology / attribute relationships and cross-modal association relationships are matched and associated with the attribute relationships defined in the ontology model.

[0041] Based on the injected entities and relationships, construct triples in the form of head entity-relationship-tail entity;

[0042] All triple structures are stored in a knowledge graph database. Using the node-edge storage mode of the graph database, entities are presented as nodes, entity attributes are presented as node attribute values, and relationships are presented as edges connecting nodes, thus completing the construction of a knowledge graph of intangible cultural heritage patterns of ethnic minorities.

[0043] Furthermore, in step S400, the process of mapping entities, semantic relationships, and attribute relationships in the graph to low-dimensional vectors is as follows:

[0044] For low-dimensional entity vectors, entities in the knowledge graph are generated by sampling neighbor nodes and aggregating neighbor features to produce low-dimensional vector representations of entities. The dimension of the pattern entity vector is set to 256 dimensions, and the dimension of the cultural background, inheritor, and regional entity vectors is set to 128 dimensions.

[0045] For low-dimensional relation vectors, semantic and attribute relations are represented as translation vectors from the head entity vector to the tail entity vector, i.e. , For the head entity vector, For the tail entity vector, The relation vectors are defined such that the dimensions of the low-dimensional relation vectors are consistent with those of the low-dimensional entity vectors.

[0046] Furthermore, in step S400, the generation process of the pattern semantic vector, visual feature vector, and cultural association vector is as follows:

[0047] When generating the pattern semantic vector, the textual description features of the pattern are fused with the low-dimensional vector of the pattern entity. The fusion formula is as follows: , , representing a weighting coefficient with a value ranging from 0.3 to 0.7, is used to adjust the contribution of the text description feature vector of the pattern and the pattern entity vector in the fusion process. This represents the generated pattern semantic vector. It is a textual description feature of the pattern. It is a pattern entity vector;

[0048] When generating visual feature vectors, the structured feature data of the pattern image is fused with the low-dimensional vector of the pattern entity, and the fusion method is the same as that of the pattern semantic vector.

[0049] When generating cultural association vectors, the cosine similarity between the low-dimensional vector of pattern entities and the vector of cultural background entities is calculated, and the low-dimensional vector of the relationship between the two is combined to generate a vector representing the strength of cultural association.

[0050] All generated vectors are categorized and stored in a vector database according to entity type and vector type.

[0051] Furthermore, in S500, three-level trust authentication is performed on knowledge graph entities:

[0052] Level 1 Authentication: Verifies the identity and geographical authenticity of the inheritor by calling the cultural heritage data source API;

[0053] Level 2 authentication: Attaching a blockchain digital fingerprint to an entity that has passed Level 1 authentication;

[0054] Level 3 Authentication: Reliability score is dynamically calculated based on data consistency and verification records. Generate a traceable authentication graph, where A represents data source consistency authentication, with 1 for successful Level 1 authentication and 0 for failure. The consistency score for multimodal data is obtained by calculating the similarity between text and image vectors. To verify the number of times, As weight, and ,when When the value is ≥0.8, the entity is included in the traceable authentication graph.

[0055] On the other hand, a knowledge graph construction system for ethnic minority intangible cultural heritage patterns based on multimodality is proposed. The system consists of: a data acquisition and preprocessing module, a dual-path entity relationship extraction module, an ontology construction and knowledge storage module, a vector mapping and storage module, and a blockchain trust authentication module.

[0056] The data acquisition and preprocessing module is used to acquire original multimodal data of ethnic minority intangible cultural heritage patterns, including text descriptions, pattern images, associated audio and video, and to preprocess these multimodal data to generate structured preprocessed data.

[0057] The dual-path entity relationship extraction module performs dual-path entity relationship extraction based on structured preprocessed data, and realizes cross-modal attribute mapping between text description and pattern image through contrastive learning.

[0058] The ontology construction and knowledge storage module is used to construct an ontology model with patterns as the core, define entity types and attribute relationships, inject the entities, semantic relationships and cross-modal attribute mapping results extracted by the dual-path entity relationship extraction module into the ontology model, establish a triplet structure and store it in the knowledge graph.

[0059] The vector mapping and storage module uses embedded vector technology to map entities, semantic relationships and attribute relationships in the knowledge graph into low-dimensional vectors, generate pattern semantic vectors, visual feature vectors and cultural association vectors, and store these vectors in the vector database.

[0060] The blockchain trust authentication module uses blockchain technology to build a trust channel and performs three-level trust authentication on entities in the vector database.

[0061] Compared with existing technologies, this method and system for constructing a knowledge graph of ethnic minority intangible cultural heritage patterns based on multimodality has the following beneficial effects:

[0062] I. This invention achieves deep association of multimodal knowledge through a dual-path entity relationship extraction mechanism. The text path uses a pre-trained language model to encode contextual semantics, accurately extracting entities and semantic relationships such as patterns and inheritors. The visual path uses a graph convolutional network to parse the topological structure of patterns, transforming visual features such as lines and contours into structured relational data. Then, through contrastive learning, the cosine similarity between text and visual feature vectors is calculated to establish a cross-modal mapping between visual features and cultural descriptions. This mechanism enables the knowledge graph to simultaneously contain semantic logic and visual feature associations, realizing the construction of a complete knowledge chain from pattern form to cultural connotation.

[0063] Second, this invention relies on a three-tiered blockchain trust authentication system to ensure the credibility of knowledge. The first-level authentication calls the cultural heritage API to verify the identity and geographical information of the inheritor, ensuring the authenticity of the basic data. The second-level authentication generates a unique blockchain digital fingerprint for each entity, recording the modification trajectory throughout its entire lifecycle. The third-level authentication quantifies the credibility of an entity by dynamically scoring the consistency of data sources, the similarity of multimodal data, and the number of verifications. This mechanism utilizes the immutability of blockchain and dynamic scoring to make the origin and inheritance relationship of each pattern entity in the knowledge graph traceable, providing a reliable knowledge benchmark for cultural research.

[0064] Other advantages, objects and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be learned from the practice of the invention. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0066] Figure 1A flowchart illustrating the operation of a method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodality;

[0067] Figure 2 A module composition diagram for a multimodal knowledge graph construction system for ethnic minority intangible cultural heritage patterns. Detailed Implementation

[0068] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0069] Example 1

[0070] This embodiment provides a method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal data. By collecting multimodal data such as text, images, audio, and video, and performing dual-path entity relation extraction after preprocessing, an ontology model with the pattern as the core is constructed and entities and relations are injected to form a knowledge graph. Then, the entities and relations are mapped into low-dimensional vectors for storage. Finally, three-level trust authentication is completed through blockchain technology, realizing multi-dimensional and in-depth digital integration and trustworthy management of intangible cultural heritage patterns of ethnic minorities, and providing comprehensive knowledge support for the protection, inheritance and utilization of intangible cultural heritage patterns.

[0071] First, the data collection and preprocessing stage (S100) is entered. Descriptions of the names, eras, regions, and folk meanings of silver ornament patterns are obtained from the intangible cultural heritage database. Scanned copies of ancient silver ornament books, photos of intangible cultural heritage objects, and digital design drawings by inheritors are collected. Interview recordings with silver ornament inheritors are recorded, and videos of the silver ornament forging process and oral histories by inheritors are filmed. The survey report is translated into Chinese. Entities are labeled using a combination of manual and machine-aided labeling, forming text data with entity labels. Blurred rubbings are sharpened, separating the main body of the pattern from the background. The outline, lines, and color features of the pattern are extracted to generate structured image feature data. The interview recordings with inheritors are converted to text using speech-to-text technology. The forging video is analyzed frame by frame to extract keyframes of pattern engraving. The converted text is labeled with entities using the above text processing methods, and keyframes are used to extract features using image processing methods. The text data with entity labels and image feature data are integrated into structured preprocessed data, providing a unified input for subsequent steps.

[0072] Then, the dual-path entity relation extraction stage (S200) is entered. Entities and their relations are extracted from the preprocessed multimodal data, including semantic relations of text paths and attribute relations of visual paths, and cross-modal associations are achieved. For text path and semantic relation extraction: text data with entity labels is input into a pre-trained language model. Utilizing its contextual semantic encoding capabilities, entity types such as patterns, cultural backgrounds, inheritors, and regions are identified, and semantic relations such as pattern-belongs-region, inheritor-inheritance-pattern, and pattern-possesses-cultural background are extracted. The output is a text entity relation set containing entity labels and semantic relation triples. The pre-trained language model has powerful semantic understanding capabilities in natural language processing and can accurately capture the semantic relations between entities in the text. For visual path and attribute relation extraction: structured images are processed... Feature data is input into a graph convolutional network (GCN). The network constructs a graph model using basic elements of the pattern's topology as nodes and spatial relationships and structural connections between elements as edges. The GCN's neighborhood information aggregation capability is used to analyze the topological relationships. Simultaneously, visual attribute relationships such as "pattern has symmetry" and "pattern contains geometric elements" are extracted, outputting a set of visual entity relationships. Graph convolutional networks excel at processing graph-structured data and can effectively uncover complex relationships between elements in pattern images. Text entities and visual entities are converted into text feature vectors and visual feature vectors, respectively. The cosine similarity between the two is calculated, and text entities with similarity reaching a threshold are associated with visual attributes, generating a dual-path fusion entity relationship set. Cosine similarity effectively measures the similarity between two vectors, thus achieving cross-modal mapping of text and image information and compensating for the deficiencies of single-modal information.

[0073] Secondly, the process moves to the vector mapping and storage stage (S400). By constructing an ontology model, the extracted entities and relationships are integrated to form a structured knowledge graph. With patterns as the core node, entity types are clearly categorized into four main types: patterns, cultural backgrounds, inheritors, and regions. Each type of entity is given unique attributes; for example, patterns have attributes such as name, carrier, era, and symbolic description, while cultural backgrounds have attributes such as origin period, ethnicity, core connotations, and related folk activities. Simultaneously, attribute relationships between entities are defined, such as the relationship between patterns and cultural backgrounds, and the relationship between patterns and inheritors. This forms a complete ontology framework. The ontology model provides a standardized structure for knowledge organization, ensuring consistency and systematicity. Entities from the dual-path fusion entity relationship set are incorporated into the corresponding types of the ontology model. Relationships are matched and associated with attribute relationships in the ontology model, constructing triples in the form of head entity-relationship-tail entity, and storing them in the knowledge graph database. In the database, entities are presented as nodes, attributes as node attribute values, and relationships as edges. This storage method intuitively displays the relationships between entities, facilitating knowledge retrieval and mining.

[0074] Next, the vector mapping and storage stage (S400) is entered, where entities and relations in the knowledge graph are converted into low-dimensional vectors for efficient computer processing and computation. The generation of low-dimensional entity vectors is achieved by sampling neighbor nodes and aggregating neighbor features. The pattern entity vector is set to 256 dimensions, while the cultural background, inheritor, and regional entity vectors are set to 128 dimensions. Higher-dimensional pattern entity vectors can more meticulously depict the complex features of the patterns, while lower-dimensional vectors for other entities reduce computation while preserving basic features. The low-dimensional relation vector is represented as a translation vector from the head entity vector to the tail entity vector, i.e. , For the head entity vector, For the tail entity vector, The relation vector has the same dimension as the corresponding entity vector. This representation effectively captures the relational features between entities. The pattern semantic vector is generated by fusing the textual description features of the pattern with the low-dimensional vector of the pattern entity, as shown in the formula. , , representing a weighting coefficient with a value ranging from 0.3 to 0.7, is used to adjust the contribution of the text description feature vector of the pattern and the pattern entity vector in the fusion process. This represents the generated pattern semantic vector. It is a textual description feature of the pattern. The pattern entity vectors are generated by combining the semantic information of the text description with the features of the entity itself. The visual feature vectors are generated in the same way as the semantic vectors, integrating the structured feature data of the pattern image with the low-dimensional vectors of the pattern entities. This combines the visual information of the image with the features of the entities. The cultural association vectors are generated by calculating the cosine similarity between the pattern entities and the cultural background entity vectors and combining the relationship between the two to generate a low-dimensional vector. This can reflect the strength of the association between the pattern and the cultural background. All generated vectors are classified and stored in the vector database according to the entity type and vector type, providing data support for subsequent trust authentication and knowledge computing.

[0075] Finally, the blockchain trust authentication phase (S500) is initiated, utilizing blockchain technology to build a trust channel and perform three-level trust authentication on entities in the vector database.

[0076] Level 1 Authentication: Call the heritage data source API to verify the identity of the intangible cultural heritage inheritor (authentication passed, A=1) and the authenticity of the place of origin (authentication passed).

[0077] Level 2 authentication: Attaching a blockchain digital fingerprint to an entity that has passed Level 1 authentication. The immutability of the blockchain ensures the integrity and consistency of the entity's information in subsequent processes.

[0078] Level 3 certification: A credibility score is dynamically calculated based on data consistency and verification records, using the following formula: ( For data source consistency authentication, a value of 1 indicates success and 0 indicates failure. The consistency score for multimodal data is obtained from the similarity between text and image vectors; Number of verifications; , , (where the weights are 1), when When entities are included in a traceable authentication graph, the scoring takes into account multiple dimensions of factors, which can comprehensively assess the credibility of the entity. The traceability of the blockchain also enhances the credibility of the authentication results.

[0079] In summary, this embodiment constructs a comprehensive, accurate, and reliable knowledge graph of ethnic minority intangible cultural heritage patterns through steps such as multimodal data collection and preprocessing, dual-path entity relationship extraction, ontology model construction and knowledge storage, vector mapping and storage, and blockchain trust authentication. This method integrates multiple types of data, including text, images, audio, and video, and achieves effective cross-modal information association. It organizes knowledge in a structured manner through ontology models and knowledge graphs, improves knowledge processing efficiency through vector mapping, and ensures the credibility of knowledge using blockchain technology. This provides strong technical support and a complete solution for the digital protection, inheritance, and utilization of ethnic minority intangible cultural heritage patterns.

[0080] Example 2

[0081] like Figure 1 As shown in the figure, this embodiment provides a method for constructing a knowledge graph of intangible cultural heritage patterns based on multimodality. The specific process of constructing the knowledge graph of intangible cultural heritage patterns is as follows:

[0082] (1) Multimodal data acquisition and preprocessing

[0083] Collect pattern-related data: text (pattern name, era, folk meaning), images (scanned ancient books, physical photos, design drawings), audio (interviews, ritual recordings), and video (production process, oral history).

[0084] Text preprocessing: translating minority languages, manually and machine-labeling entities (pattern names, cultural backgrounds, inheritors, regions), and generating tagged text data;

[0085] Image preprocessing: improving the clarity of blurred images, segmenting the main body and background of patterns, extracting contours, lines, and color features, and generating structured image data;

[0086] Audio and video preprocessing: speech to text, keyframe extraction from video (pattern features / production process / cultural scenes), and content conversion by text / image process;

[0087] Output a structured preprocessed dataset;

[0088] (2) Dual-path entity relation extraction

[0089] Text path: Input labeled text data into a pre-trained language model;

[0090] Extracting entities: patterns, cultural background, inheritors, and regions;

[0091] Extract semantic relationships: pattern - belongs to - region, inheritor - inherits - pattern, pattern - has - cultural background;

[0092] Output a set of text entity relations (entity + semantic triples);

[0093] Visual path: Input structured image data to a graph convolutional network (GCN);

[0094] Constructing a graph model: Nodes = basic pattern elements, edges = spatial / structural relationships between elements;

[0095] Analyze topological relationships and extract visual attribute relationships (pattern - has - symmetry attribute, pattern - contains - geometric element);

[0096] Output a set of visual entity relationships (topology structure + visual attribute triples).

[0097] Cross-modal association: Convert text entities into text vectors, visual entities into visual vectors, calculate the cosine similarity between text and visual vectors, and associate text entities and visual attributes whose similarity reaches a threshold.

[0098] Output a dual-path fused entity relationship set (textual semantics + visual topology / attributes + cross-modal associations);

[0099] (3) Ontology model construction and knowledge storage

[0100] Define core entity types: pattern, cultural background, inheritor, and region (including exclusive attributes such as the pattern's era and the period of its cultural origin).

[0101] Define the attribute relationships between entities: Carrier (pattern - cultural background), Inherited (pattern - inheritor), Origin (pattern - region), Belonging (inheritor - region).

[0102] Injecting the merged entity relationship set into the ontology: entity classification, relationship matching, and construction of head entity-relation-tail entity triples;

[0103] Stored in a graph database: Entity = Node (attribute value), Relationship = Edge;

[0104] (4) Vector mapping and storage

[0105] Generate low-dimensional entity vectors: aggregate neighbor features, pattern entities (256-dimensional), and other entities (128-dimensional);

[0106] Generate low-dimensional vectors of relations: Calculate relation translation vectors (dimension same as entities);

[0107] Generate multimodal fusion vectors: pattern semantic vector (fusion of text description features and pattern entity vectors), and visual feature vector (fusion of image features and pattern entity vectors).

[0108] Cultural association vector: Generated based on the similarity and relationship vectors between pattern and cultural background entity vectors;

[0109] Store all vectors in a vector database (categorized by entity type and vector type);

[0110] (5) Blockchain Trust Authentication

[0111] Level 1 Authentication: Verifies the identity and geographical authenticity of the inheritor by calling an authoritative API;

[0112] Level 2 authentication: Attaching a blockchain digital fingerprint to the entity;

[0113] Level 3 Authentication: Dynamically calculate credibility scores (based on data source consistency, multimodal data consistency, and number of verifications), and generate traceable authentication graphs for entities with scores ≥ 0.8.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodality, characterized in that, The specific steps of this method are as follows: S100. Collect original multimodal data of intangible cultural heritage patterns of ethnic minorities, including text descriptions, pattern images, associated audio and video, and preprocess the multimodal data to generate structured preprocessed data; S200. Based on the structured preprocessed data, perform dual-path entity relationship extraction: Text path and semantic relationship: Extract entities and semantic relationships of patterns, cultural background, inheritors, and regions through a pre-trained language model; Visual path and attribute relationship: Graph convolutional networks are used to analyze the topological relationships of pattern images, and cross-modal attribute mapping between text descriptions and pattern images is achieved through contrastive learning; S300: Construct an ontology model with patterns as the core, define entity types and attribute relationships, inject the entities, semantic relationships and cross-modal attribute mapping results extracted in S200 into the ontology model, establish a triplet structure and store it in the knowledge graph. S400: Employing embedded vector technology, entities, semantic relationships, and attribute relationships in the graph are mapped into low-dimensional vectors, generating pattern semantic vectors, visual feature vectors, and cultural association vectors, which are then stored in a vector database. S500 utilizes blockchain technology to build a trust channel and performs three-level trust authentication on entities in the vector database.

2. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 1, characterized in that, In step S100, the collection of multimodal raw data on intangible cultural heritage patterns of ethnic minorities specifically includes: Textual data: Descriptions of pattern names, eras, regions, and folk meanings from intangible cultural heritage databases, as well as survey reports provided by local authorities; Image data: Pattern images from scanned copies of ancient books, photos of intangible cultural heritage objects, and digital design drawings by inheritors; Audio data: Related audio from interview recordings of inheritors and recordings of folk rituals; Video data: Related videos from recordings of the pattern-making process and oral histories by inheritors.

3. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 2, characterized in that, In step S100, the preprocessing of the original multimodal data is as follows: When processing text data, the text descriptions in minority languages ​​are translated into the common language. A combination of manual annotation and machine-assisted annotation is used to annotate the entities in the text, such as pattern names, cultural background keywords, names of inheritors, and regional names, to form text data with entity labels. When processing image data, after improving the clarity of blurry pattern images, the pattern subject and background in the image are segmented, and the feature information of the pattern's outline, lines, and colors is extracted to generate structured image feature data. When processing audio and video data, the speech content in the audio is converted into text, and the video is analyzed frame by frame to extract key frames of pattern features, production process and cultural scene. The converted text and key frames are preprocessed in the same way as text data and image data. Text data and image feature data with entity labels are integrated to generate structured preprocessed data.

4. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodality, as described in claim 2, is characterized in that... In step S200, the specific process of performing dual-path entity relationship extraction is as follows: For text path and semantic relation extraction: Input the text data with entity labels generated by S100 into the pre-trained language model. Through the contextual semantic encoding capability of the pre-trained language model, identify and extract the following content: Entity types: patterns, cultural background, inheritors, regions; Semantic relations: pattern-belongs to-region, inheritor-inherits-pattern, pattern-has-cultural background. Output text entity relation set, including entity tags and semantic relation triples; For visual path and attribute relationship extraction: The structured image feature data generated by S100 is input into a graph convolutional network. A graph model is constructed with the topological structure of the pattern as nodes. Nodes are defined as basic elements in the pattern, and edges are defined as spatial positional relationships and structural associations between elements. Through the neighborhood information aggregation capability of the graph convolutional network, the topological structure relationship of the pattern is analyzed. At the same time, based on the visual features of the pattern, visual attribute relationships of pattern-having-symmetric attributes and pattern-containing-geometric elements are extracted, and a set of visual entity relationships is output, which includes topological structure relationship triples and visual attribute relationship triples. Perform cross-modal association between the output text entity relation set and the input visual entity relation set: Feature encoding: converting text entities into text feature vectors and visual entities into visual feature vectors; Similarity calculation: Calculate the cosine similarity between the text feature vector and the visual feature vector; Attribute mapping: Associating text entities with similarity reaching a threshold with visual attributes, ultimately generating a dual-path fusion entity relationship set, including text semantic relationships, visual topological / attribute relationships, and cross-modal association relationships.

5. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 1, characterized in that, In S300, the specific process of constructing an ontology model and knowledge graph storage centered on patterns is as follows: When constructing the ontology model, patterns are used as the core nodes to clearly define entity types, including four major categories: patterns, cultural background, inheritors, and regions. Specific attributes are defined for each type of entity. Pattern attributes: pattern name, pattern medium, pattern era, and pattern symbolism description; Cultural background attributes: period of cultural origin, ethnic group to which the culture belongs, core cultural connotations, and related folk customs; Inheritor attributes: Name, gender, date of birth, level of inherited skills, and years of inheritance; Regional attributes: region name, administrative division, geographical features, and ethnic distribution; At the same time, the attribute relationships between entities are defined, including the relationship between patterns and cultural background, the relationship between patterns and inheritors, the relationship between patterns and regional origins, and the relationship between inheritors and regional affiliation, forming a complete ontology framework; When injecting entities and relationships, the entities in the dual-path fusion entity relationship set generated by S200 are included in the corresponding entity types of the ontology model, and semantic relationships, visual topology / attribute relationships and cross-modal association relationships are matched and associated with the attribute relationships defined in the ontology model. Based on the injected entities and relationships, construct triples in the form of head entity-relationship-tail entity; All triple structures are stored in a knowledge graph database. Using the node-edge storage mode of the graph database, entities are presented as nodes, entity attributes are presented as node attribute values, and relationships are presented as edges connecting nodes, thus completing the construction of a knowledge graph of intangible cultural heritage patterns of ethnic minorities.

6. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 1, characterized in that, In step S400, the process of mapping entities, semantic relationships, and attribute relationships in the graph to low-dimensional vectors is as follows: For low-dimensional entity vectors, entities in the knowledge graph are generated by sampling neighbor nodes and aggregating neighbor features to produce low-dimensional vector representations of entities. The dimension of the pattern entity vector is set to 256 dimensions, and the dimension of the cultural background, inheritor, and regional entity vectors is set to 128 dimensions. For low-dimensional relation vectors, semantic and attribute relations are represented as translation vectors from the head entity vector to the tail entity vector, i.e. , For the head entity vector, For the tail entity vector, The relation vectors are defined such that the dimensions of the low-dimensional relation vectors are consistent with those of the low-dimensional entity vectors.

7. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 6, characterized in that, In step S400, the generation process of the pattern semantic vector, visual feature vector, and cultural association vector is as follows: When generating the pattern semantic vector, the textual description features of the pattern are fused with the low-dimensional vector of the pattern entity. The fusion formula is as follows: , , representing a weighting coefficient with a value ranging from 0.3 to 0.7, is used to adjust the contribution of the text description feature vector of the pattern and the pattern entity vector in the fusion process. This represents the generated pattern semantic vector. It is a textual description feature of the pattern. It is a pattern entity vector; When generating visual feature vectors, the structured feature data of the pattern image is fused with the low-dimensional vector of the pattern entity, and the fusion method is the same as that of the pattern semantic vector. When generating cultural association vectors, the cosine similarity between the low-dimensional vector of pattern entities and the vector of cultural background entities is calculated, and the low-dimensional vector of the relationship between the two is combined to generate a vector representing the strength of cultural association. All generated vectors are categorized and stored in a vector database according to entity type and vector type.

8. The method for constructing a knowledge graph of intangible cultural heritage patterns of ethnic minorities based on multimodal methods according to claim 1, characterized in that, In S500, three-level trust authentication is performed on knowledge graph entities: Level 1 Authentication: Verifies the identity and geographical authenticity of the inheritor by calling the cultural heritage data source API; Level 2 authentication: Attaching a blockchain digital fingerprint to an entity that has passed Level 1 authentication; Level 3 Authentication: Reliability score is dynamically calculated based on data consistency and verification records. Generate a traceable authentication graph, where A represents data source consistency authentication, with 1 for successful Level 1 authentication and 0 for failure. The consistency score for multimodal data is obtained by calculating the similarity between text and image vectors. For the number of verifications, As weight, and ,when When the value is ≥0.8, the entity is included in the traceable authentication graph.

9. A multimodal knowledge graph construction system for ethnic minority intangible cultural heritage patterns, applicable to the multimodal knowledge graph construction method for ethnic minority intangible cultural heritage patterns as described in any one of claims 1-8, characterized in that, The system consists of: a data acquisition and preprocessing module, a dual-path entity relationship extraction module, an ontology construction and knowledge storage module, a vector mapping and storage module, and a blockchain trust authentication module. The data acquisition and preprocessing module is used to acquire original multimodal data of ethnic minority intangible cultural heritage patterns, including text descriptions, pattern images, associated audio and video, and to preprocess these multimodal data to generate structured preprocessed data. The dual-path entity relationship extraction module performs dual-path entity relationship extraction based on structured preprocessed data, and realizes cross-modal attribute mapping between text description and pattern image through contrastive learning. The ontology construction and knowledge storage module is used to construct an ontology model with patterns as the core, define entity types and attribute relationships, inject the entities, semantic relationships and cross-modal attribute mapping results extracted by the dual-path entity relationship extraction module into the ontology model, establish a triplet structure and store it in the knowledge graph. The vector mapping and storage module uses embedded vector technology to map entities, semantic relationships and attribute relationships in the knowledge graph into low-dimensional vectors, generate pattern semantic vectors, visual feature vectors and cultural association vectors, and store these vectors in the vector database. The blockchain trust authentication module uses blockchain technology to build a trust channel and performs three-level trust authentication on entities in the vector database.

Images