A method and system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data

Abstract

This invention discloses a method and system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, belonging to the field of knowledge graph construction. The method includes: collecting intangible cultural heritage data of different modalities, performing preliminary cleaning and annotation on the collected data, establishing preliminary associations between different modalities of data, and obtaining an initial version of the multimodal dataset; processing the initial version of the multimodal dataset through a multimodal data cleaning and annotation model to obtain a final multimodal dataset, an annotation consistency matrix, and a cross-modal association matrix; constructing a unified domain knowledge model to perform structured modeling of intangible cultural heritage-related concepts, entities, and their relationships, forming a semantic expression and knowledge organization form that supports multimodal data; and constructing a structured multimodal knowledge graph. This invention can effectively integrate multi-source heterogeneous intangible cultural heritage data, improving the efficiency of acquiring and utilizing intangible cultural heritage knowledge.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction, and in particular to a method and system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data. Background Technology

[0002] With the rapid development of information and digital technologies, the protection and transmission of intangible cultural heritage has received increasing attention. However, existing methods of intangible cultural heritage protection mainly rely on manual recording and organization, which is inefficient and makes it difficult to fully utilize multi-source heterogeneous data. Traditional methods of digitizing intangible cultural heritage often focus on the collection and display of single data modalities, such as text records, image preservation, or video recording, but these methods are inadequate when faced with increasingly complex intangible cultural heritage information. As a comprehensive cultural phenomenon, intangible cultural heritage has rich and diverse connotations, encompassing various data forms such as text, images, audio, video, and 3D models, making it difficult to fully express its value and significance through a single data modality.

[0003] Therefore, how to effectively integrate multi-source, heterogeneous intangible cultural heritage (ICH) data, construct a structured knowledge base, and achieve intelligent management and application of ICH has become an urgent problem to be solved. Existing technologies often face low storage and query efficiency when processing large-scale, multimodal data, hindering the application of knowledge graphs in the large-scale, real-time ICH field. Furthermore, existing technologies have weak processing capabilities for unstructured data such as text, images, and videos, resulting in the neglect of knowledge contained in large amounts of unstructured data and limiting the effectiveness of knowledge graph construction. Summary of the Invention

[0004] One of the objectives of this invention is to provide a method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, so as to solve the problem that the knowledge contained in unstructured data is ignored in the prior art.

[0005] This invention is achieved through the following technical solution: a method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, comprising the following steps: S100, collecting intangible cultural heritage data of different modalities, and performing preliminary cleaning and labeling on the collected data to establish preliminary associations between different modalities, thereby obtaining a preliminary multimodal dataset; S200, processing the preliminary multimodal dataset using a multimodal data cleaning and labeling model to obtain a final multimodal dataset, a labeling consistency matrix, and a cross-modal association matrix; S300, constructing a unified domain knowledge model to perform structured modeling of intangible cultural heritage-related concepts, entities, and their relationships, forming a semantic expression and knowledge organization form that supports multimodal data; S400, constructing a structured multimodal knowledge graph based on the data in the knowledge organization form output in S300.

[0006] Furthermore, intangible cultural heritage data in different modalities includes: textual data related to intangible cultural heritage, including texts, documents, historical documents, research papers, project introductions, and expert interviews describing the background, development, and transmission of intangible cultural heritage projects; image data related to intangible cultural heritage, including complete photographs of intangible cultural heritage projects, breakdown diagrams of process steps, photographs of inheritors and their skill demonstrations, and on-site images of folk activities; and audio and video data related to intangible cultural heritage, including audio and video recordings of the production process of intangible cultural heritage skills, and audio and video recordings of oral transmission of intangible cultural heritage.

[0007] Furthermore, intangible cultural heritage data in different modalities may also include: structured data from official institutions, such as UNESCO or national intangible cultural heritage lists, including the name, category, region, and information of the inheritors of the items. In addition, information on the inheritors' archives, such as their personal background, the process of skill transmission, and social impact, should also be collected.

[0008] Furthermore, preliminary cleaning and labeling include: removing duplicate paragraphs from the collected text data and standardizing terminology; performing preliminary association of data of different modalities based on filenames, timestamps, metadata, etc., and labeling the core entities and their interrelationships in text, image, audio, and video data.

[0009] Furthermore, the multimodal data cleaning and labeling model ensures efficient and accurate data cleaning across different modalities through multiple constraints and losses. The multimodal data cleaning and labeling model includes: S210, denoising, removing noise from the data and using denoising loss to avoid interference from noisy samples, thus improving the overall data cleaning effect; S220, cross-modal alignment, aligning features from different modalities to ensure similar representations of samples with the same semantics across different modalities, optimizing the relationship between modalities; S230, label consistency, ensuring consistent labels for samples of the same class across different modalities, avoiding classification errors caused by inconsistent labeling; and S240, association matrix constraint, using sparsity constraints to ensure the model only establishes associations between highly similar sample pairs, avoiding overfitting to irrelevant samples, and controlling the relative importance of each objective in the overall optimization by adjusting the weight coefficients of different functions.

[0010] Furthermore, the objective function of the multimodal data cleaning and labeling model can be expressed by the following formula:

[0011] ,in, The final multimodal dataset includes, The cleaned text data matrix, The cleaned image data matrix, The cleaned audio data matrix, This is the initial version of the multimodal dataset, including: Initial text data matrix, Initial image data matrix, Initial audio data matrix, To control the weighting coefficients of the denoising intensity, A data denoising function. The weighting coefficients are used to control the degree of modal alignment optimization. For cross-modal correlation optimization function, To control the weighting coefficients for annotation consistency, For annotation consistency constraint functions, The weighting coefficients are used to control the sparsity of the correlation matrix. The constraint function for the incidence matrix. For the annotation consistency matrix, This is the cross-modal correlation matrix.

[0012] Furthermore, the data denoising function ensures that the model can correctly extract effective features from the data. By penalizing inconsistent or erroneous features in the model, the data denoising function reduces the impact of noise, thereby improving the quality of feature representation. The cross-modal association optimization function optimizes the alignment of cross-modal features. By making features from different modalities as close as possible in a shared embedding space, it improves the model's performance in cross-modal tasks. The label consistency constraint function ensures that the model maintains the consistency of sample labels during the learning process, ensuring that samples of the same class in different modalities have consistent label information. The association matrix constraint function sparsifies the association matrix so that there is a non-zero association only between cross-modal features with sufficient similarity. The association matrix constraint function, through L1 norm constraints, makes most of the elements in the association matrix close to zero, thereby reducing the connection between irrelevant or low-similarity samples and maintaining the efficiency of the model.

[0013] Furthermore, the data denoising function can be expressed by the following formula:

[0014] ,in, For the initial set of data matrices, This is a collection of cleaned data matrices. The Frobenius norm, also known as the Euclidean norm of a matrix, is used to measure the difference between two matrices. In the above formula... This represents the sum of squared differences between the original data and the denoised data. It is used to measure the effect of denoising on the data matrix. The better the denoising effect, the smaller this difference and the lower the loss value. The nuclear norm is used to constrain the rank of a matrix, encouraging matrices to have a low-rank structure. In the above formula... This is used to force the denoised matrix to maintain a low rank and remove redundant information. This means that if the rank of the cleaned data matrix set is low, it will be more compact and contain fewer redundant terms, which helps to clean up invalid or duplicate data. is a hyperparameter for denoising intensity, used to control the denoising strength; it balances the weights of the Frobenius norm and nuclear norm terms in the objective function. When the value is large, the denoising effect is stronger, and the low-rank constraint of the matrix is ​​more stringent. In other words, at this time... It will be closer to a low-rank matrix, thus removing more noise.

[0015] Furthermore, the cross-modal correlation optimization function can be expressed by the following equation:

[0016] in, Positive sample pairs (i.e., sample pairs that are actually related) are usually predetermined in some way (such as file name, timestamp, tag, etc.), which means that the two samples should be actually related across modalities (e.g., text description and video clip of the same event). Negative sample pairs (i.e., sample pairs with no actual correlation) are obtained through random sampling. Their purpose is to help the model learn the differences between different modalities and prevent the model from mistakenly classifying unrelated samples as related ones. Let i be the feature vector of sample i. Let j be the feature vector of sample j. Let be the cosine similarity between the feature vectors of sample i and sample j. It is used to measure the similarity between the two samples in the feature space. The value range is [-1, 1]. The closer the value is to 1, the more similar the two samples are. The closer the value is to -1, the less similar the two samples are. For similarity threshold,

[0017] Furthermore, the annotation consistency constraint function can be expressed by the following formula:

[0018] ,in, The labeled sample set represents a set of samples with known labels, which have been annotated by experts. Typically, these samples are few in number, but their labels are very accurate, and the model can use these sample labels to guide training. Let be the label consistency probability of sample i, that is, the probability that the model predicts the label for sample i. During the optimization process, the model will learn how to adjust... To make it closer to the real label And by propagating the consistency of labels, it ensures that similar samples have similar labels; The true label for sample i is derived from expert annotations. To control the hyperparameters of tag propagation strength, if Larger numbers indicate a stronger impact of label propagation, leading to stricter constraints on label consistency among similar samples; if The smaller the sample size, the weaker the impact of label consistency constraints, and the more likely the model will rely on already labeled samples. The Laplacian matrix, constructed based on the similarity between samples, is typically used in graph structures to represent the relationships between nodes. Specifically, the Laplacian matrix describes the similarity structure between samples and controls the consistency of labels among similar samples. The trace operation of a matrix calculates... The sum of the diagonal elements represents a global consistency in tag propagation. This is the transpose of the annotation consistency matrix. This is the matrix transpose symbol.

[0019] Furthermore, the constraint function of the incidence matrix can be expressed by the following formula: , ,in, This is the cross-modal correlation matrix. Let L1 be the L1 norm of the cross-modal incidence matrix, which is the sum of the absolute values ​​of all elements in the matrix. The purpose of the L1 norm is to promote the sparsity of the cross-modal incidence matrix, making most of the elements close to zero. This is achieved by minimizing... The model is encouraged to retain only a few important (high similarity) associations, while other low similarity associations are suppressed; st is an abbreviation for subject to, indicating that the function is restricted to the cross-modal feature similarity between sample i and sample j.

[0020] Furthermore, constructing a unified domain knowledge model includes the following sub-steps: S310, defining core ontology classes based on the annotation consistency matrix, statistically analyzing the categories in the annotation consistency matrix, calculating the total confidence of each category, and defining the category as a core ontology class if the total confidence of a certain category is significantly higher than that of other categories; S320, defining relationships based on cross-modal associations and knowledge rules based on cross-modal association matrices, inferring the relationships between different entities through the information of cross-modal data pairs in the cross-modal association matrix, and achieving alignment between different modalities; S330, extracting textual knowledge based on the data in the data matrix set.

[0021] Furthermore, text knowledge extraction includes entity extraction and relation extraction. Entity extraction uses high-confidence labels from the consistency matrix as supervision signals and is implemented through a BiLSTM+CRF model. Relation extraction uses enhanced dependency parsing, combined with the cosine similarity of text and images, to extract and strengthen the confidence of relation types from the text.

[0022] Furthermore, extracting textual knowledge also includes: using images and audio as knowledge supplements; based on the alignment of images and text, using information from the cross-modal association matrix to find the text corresponding to each image, thereby generating annotations for the images; then using an object detection model to identify specific regions in the images and matching them with text entities based on similarity, thereby generating relation triples.

[0023] Furthermore, considering that different modalities may sometimes conflict in their descriptions of the same entity, this embodiment can also resolve the conflict by embedding entities from different modalities into a unified space to enhance cross-modal consistency and thus resolve the conflict problem.

[0024] Furthermore, the conflict loss function can be expressed by the following formula:

[0025] ,in, Let be the conflict loss function. A set of cross-modal data pairs for the same entity. Data pairs of the same entity in m modes. For n-modal data pairs of the same entity, For the input data of the m-mode, For n-modal input data, For m-mode embedding mapping function, For n-modal embedding mapping functions.

[0026] Another aspect of the present invention provides a system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data as described above.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] 1. This invention can effectively integrate multi-source heterogeneous intangible cultural heritage data, make full use of the rich information contained in multimodal data, improve the efficiency of acquiring and utilizing intangible cultural heritage knowledge, and improve the efficiency of processing large-scale multimodal data through optimized storage and query mechanisms, supporting the application of knowledge graphs in large-scale, real-time intangible cultural heritage fields.

[0029] 2. This invention employs efficient processing methods for unstructured data such as text, images, and videos, fully mining the knowledge contained in unstructured data, improving the construction effect of knowledge graphs, and proposing effective methods to resolve conflicts and contradictions between different modalities of data, ensuring the consistency and accuracy of knowledge graphs.

[0030] 3. This invention achieves deep fusion and correlation modeling of multimodal data, fully explores the intrinsic connections between different modalities, improves the expressive power of knowledge graphs, and the final constructed multimodal knowledge graph can support applications such as querying, reasoning and visualization of intangible cultural heritage, which is conducive to the effective protection and inheritance of intangible cultural heritage. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of the method provided in Embodiment 1 of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0034] Example 1

[0035] The inheritance and protection of intangible cultural heritage (hereinafter referred to as "ICH") is a manifestation of a nation's soft power at this stage. With the development of artificial intelligence technology, exploring the combination of artificial intelligence and ICH data to achieve intelligent management of ICH data has become an urgent research topic. Existing technologies struggle to effectively integrate multi-source, heterogeneous ICH data, failing to fully utilize the rich information contained in multimodal data, resulting in low efficiency in acquiring and utilizing ICH knowledge. Furthermore, storage and retrieval efficiency is low when processing large-scale multimodal data, hindering the application of multimodal knowledge graphs in large-scale, real-time ICH fields. Moreover, existing knowledge graph construction methods are weak in processing unstructured data such as text, images, and videos, neglecting much of the knowledge contained in unstructured data, affecting the construction effect of the knowledge graph. In constructing ICH knowledge graphs, there is a lack of effective methods to resolve conflicts and contradictions between different modalities, affecting the consistency and accuracy of the knowledge graph.

[0036] To address these issues and promote the integration and application of artificial intelligence in intangible cultural heritage, this embodiment provides a method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data. Figure 1 The flowchart of the method in this embodiment is shown. As can be seen from the flowchart, this embodiment includes the following steps:

[0037] Step 1: First, acquire diverse intangible cultural heritage data, preprocess the data to eliminate noise and establish preliminary associations, laying the foundation for the subsequent construction of a multimodal knowledge graph of intangible cultural heritage.

[0038] Diverse intangible cultural heritage data includes: textual data, collecting textual information related to intangible cultural heritage, covering documents, historical records, research papers, project introductions, expert interviews, etc. The focus should be on collecting texts describing the background, development, and transmission of intangible cultural heritage projects.

[0039] Image data includes high-resolution images related to intangible cultural heritage, such as complete photos of intangible cultural heritage projects, breakdown diagrams of the process steps, photos of inheritors and their demonstrations, and images of folk activities. These images help to visually showcase the form and details of intangible cultural heritage projects.

[0040] Audio / video data collection involves gathering audio and video files related to intangible cultural heritage (ICH) projects, including recordings and videos of ICH craft production processes, and audio recordings of oral transmission of ICH. This data supports the capture of dynamic information and voice interaction related to ICH projects.

[0041] Structured data collection involves gathering metadata about intangible cultural heritage projects from official institutions (such as UNESCO or national intangible cultural heritage lists), including the project's name, category, region, and information about the inheritors. Additionally, inheritor profiles should be collected, including personal background, the process of skill transmission, and social impact.

[0042] The above data will be integrated into a multimodal intangible cultural heritage database to facilitate subsequent data processing operations.

[0043] Then, data cleaning and labeling were performed on the data in the multimodal intangible cultural heritage database. Specifically, this included:

[0044] The collected text data is cleaned, including removing duplicate paragraphs and standardizing terminology (e.g., different names for the same skill should be unified into one term) to ensure data consistency and accuracy.

[0045] Preliminary associations are made between data of different modalities based on filenames, timestamps, and metadata. For example, filenames of images and timestamps of audio / video recordings can be used to align different audiovisual materials of the same event or technique.

[0046] Intangible cultural heritage inheritors, experts in the field of intangible cultural heritage, or relevant researchers can also be invited to participate in the data annotation work, focusing on annotating the core entities (such as skills, tools, and people) in the text, images, and audio and their interrelationships (such as master-apprentice relationships, process steps, and regional connections).

[0047] The multimodal dataset, including text, images, and audio data, which has been cleaned and labeled, is integrated into the initial dataset.

[0048] Step 2: Process the initial dataset using a multimodal data cleaning and annotation model to obtain the final dataset and cross-modal association matrix. The core objective of the multimodal data cleaning and annotation model is to optimize the quality of the multimodal data and ensure annotation consistency by using modules such as denoising, modality alignment, annotation consistency, and association matrix constraints. By jointly optimizing these objective functions, the final cleaned data can better serve the analysis and annotation tasks of intangible cultural heritage data.

[0049] Specifically, the objective function of the multimodal data cleaning and labeling model in this embodiment can be expressed by the following formula:

[0050] ,

[0051] in, For the initial text data matrix, For the initial image data matrix, For the initial audio data matrix, To control the weighting coefficients of the denoising intensity, A data denoising function. The weighting coefficients are used to control the degree of modal alignment optimization. For cross-modal correlation optimization function, To control the weighting coefficients for annotation consistency, For annotation consistency constraint functions, The weighting coefficients are used to control the sparsity of the correlation matrix. The constraint function for the incidence matrix. This is the cleaned text data matrix. This is the cleaned image data matrix. The cleaned audio data matrix, For the annotation consistency matrix, This is the cross-modal correlation matrix.

[0052] It should be noted that this objective function is a comprehensive loss function used to improve the performance of cross-modal data cleaning through multi-faceted optimization. This objective function ensures that the model can efficiently clean cross-modal data while maintaining a balance across various aspects through the synergistic effect of multiple factors such as denoising, modality alignment, label consistency, and association matrix constraints. Specifically, the purpose of this mathematical expression is to minimize the objective function, where... For the initial text data matrix, Represent a A matrix, where each row represents the word vector of a document. Indicates the number of samples. The dimension of text data is usually obtained through feature extraction methods such as TF-IDF or BERT embedding. For the initial image data matrix, Represent a The matrix, similar to the text data matrix, has each row representing a feature vector of an image. The dimension of each image feature vector can be extracted using a deep learning model. For the initial audio data matrix, Similarly, each row represents a feature of an audio segment. This refers to the dimension of audio features. Common features include MFCC (Mel-frequency cepstral coefficients) and mean pooling. The cross-modal correlation matrix... , This is a binary matrix used to represent the relationships between different modes, where This indicates that there is an actual correlation between samples i and j across different modalities. (Annotation consistency matrix) , This is The matrix represents the labeled probability of each sample, where k is the number of classes, and each row represents the probability that a sample belongs to one of the k classes. Finally, for ease of representation, the initial data matrix can be... , , Integrate into an initial data matrix At the same time, the cleaned data matrix , , It is also integrated into a cleaned data matrix set. A collection used to represent a matrix of text, image, or audio data.

[0053] The objective function in this embodiment is designed to simultaneously optimize multiple aspects of the model through multiple constraints and losses, ensuring efficient and accurate data cleaning across different modalities. Specifically, the objective function includes the following main optimization objectives: 1. Denoising: First, the model needs to remove noise from the data, which helps improve the quality and robustness of feature representations. Through denoising loss, the model can avoid interference from noisy samples, improving the overall data cleaning effect. 2. Cross-modal alignment: Second, a key task in multimodal data cleaning is to align features from different modalities, so that samples with the same semantics have similar representations across different modalities. Through cross-modal alignment loss, the model can optimize the relationships between modalities. 3. Label consistency: Label consistency ensures that samples of the same class in different modalities maintain consistent labels, thereby avoiding classification errors caused by inconsistent labeling. 4. Association matrix constraints: Finally, the sparsity constraint of the association matrix prompts the model to establish associations only between highly similar sample pairs, thereby avoiding overfitting of the model to irrelevant samples. The relative importance of each objective in the overall optimization is then controlled by adjusting the weight coefficients of different functions.

[0054] Specifically, in this embodiment, the data denoising function in the objective function aims to remove noise, ensuring that the model can correctly extract effective features from the data. This noise may be caused by irrelevant samples, incorrect labeling, or other factors; processing by this denoising function helps improve the robustness of the model. The data denoising function reduces the impact of noise by penalizing inconsistencies or erroneous features in the model, thereby improving the quality of feature representation.

[0055] This function can be represented by the following formula:

[0056] ,

[0057] in, For the initial set of data matrices, This is a collection of cleaned data matrices. The Frobenius norm, also known as the Euclidean norm of a matrix, is used to measure the difference between two matrices. In the above formula... This represents the sum of squared differences between the original data and the denoised data. It is used to measure the effect of denoising on the data matrix. The better the denoising effect, the smaller this difference and the lower the loss value. The nuclear norm is used to constrain the rank of a matrix, encouraging matrices to have a low-rank structure. In the above formula... This is used to force the denoised matrix to maintain a low rank and remove redundant information. This means that if the rank of the cleaned data matrix set is low, it will be more compact and contain fewer redundant terms, which helps to clean up invalid or duplicate data. is a hyperparameter for denoising intensity, used to control the denoising strength; it balances the weights of the Frobenius norm and nuclear norm terms in the objective function. When the value is large, the denoising effect is stronger, and the low-rank constraint of the matrix is ​​more stringent. In other words, at this time... It will be closer to a low-rank matrix, thus removing more noise. If The smaller the matrix, the weaker the denoising effect; the matrix may retain more of the original features, thus preserving some redundant information.

[0058] It should be noted that the goal of the above equation is to optimize the cleaned data matrix set by minimizing the data denoising loss, thereby removing redundant information and noise. By combining the Frobenius norm and the nuclear norm, the equation allows the model to retain the effective features of the data while removing irrelevant and redundant parts. Hyperparameters are used to control the intensity of denoising. Through this optimization process, a cleaner, lower-rank multimodal dataset can be obtained, which is beneficial for subsequent cross-modal association and annotation consistency optimization.

[0059] The cross-modal association optimization function in the objective function is used to optimize the alignment of cross-modal features. Its goal is to make features from different modalities (e.g., images and text) as close as possible within a shared embedding space, thereby improving the model's performance on cross-modal tasks. The modal alignment loss function measures the similarity or alignment between different modalities, forcing features with the same semantics to be closer together, while features with different semantics are kept as far apart as possible.

[0060] This function can be represented by the following formula:

[0061]

[0062] in, Positive sample pairs (i.e., sample pairs that are actually related) are usually predetermined in some way (such as file name, timestamp, tag, etc.), which means that the two samples should be actually related across modalities (e.g., text description and video clip of the same event). Negative sample pairs (i.e., sample pairs with no actual correlation) are obtained through random sampling. Their purpose is to help the model learn the differences between different modalities and prevent the model from mistakenly classifying unrelated samples as related ones. Let i be the feature vector of sample i. Let j be the feature vector of sample j. Let be the cosine similarity between the feature vectors of sample i and sample j. It is used to measure the similarity between the two samples in the feature space. The value range is [-1, 1]. The closer the value is to 1, the more similar the two samples are. The closer the value is to -1, the less similar the two samples are. The cosine similarity threshold is a threshold used to control whether two samples are considered similar. It is usually a small positive value, such as 0.5, used to distinguish between positive and negative sample pairs. If the cosine similarity is less than 1, then the two samples are considered related; if the cosine similarity is less than 1, then the two samples are considered related. If they are not related, then they are considered to be unrelated. These are the values ​​in the association matrix, indicating whether there is an actual association between sample i and sample j. If... If the result is positive, it means that sample i and sample j are related (e.g., they describe the same event or object); if they are unrelated, then... .

[0063] It should be noted that the goal of the above equation is to enhance the model's cross-modal correlation capability by optimizing the similarity between cross-modal samples. The equation includes two parts: optimization of positive sample pairs and optimization of negative sample pairs. Specifically, the optimization of positive sample pairs is achieved by optimizing the similarity between each pair... This means that for each predefined pair of related samples, the cosine similarity of their feature vectors in the feature space should be close to 1, i.e., they should be as similar as possible. By minimizing the difference between the cosine similarity and the correlation matrix value, the cosine similarity is expected to be close to 1; if the similarity is less than 1, the function will increase, prompting the model to adjust the feature vectors to achieve a greater similarity.

[0064] The optimization of negative sample pairs is achieved by optimizing each pair. That is, for each pair of negative samples that are not actually related, the cosine similarity of their feature vectors should be as close to 0 or lower as possible, indicating that they should be as far apart as possible in the feature space. This is used to penalize negative sample pairs whose similarity is greater than a threshold. That is, if the cosine similarity of two unrelated samples is higher than the threshold, a loss will be incurred, prompting the model to adjust its feature representation so that negative sample pairs are more separated in the feature space.

[0065] In other words, the above formula maximizes the cosine similarity of positive sample pairs, making them closer in the feature space. For negative sample pairs, it minimizes their cosine similarity to avoid them being mistakenly identified as related. This function design enables the model to learn more accurate cross-modal feature representations, allowing data from different modalities to effectively correspond and align in a shared feature space.

[0066] The label consistency constraint function in the objective function is used to ensure that the model maintains consistent labeling of samples during the learning process. In other words, the model should ensure that samples of the same class across different modalities have consistent labeling information. The label consistency function penalizes inconsistencies in labeling information, ensuring that similar samples (such as images and text) across different modalities have the same or similar label representations, thereby improving the accuracy of labeling across different modalities.

[0067] This function can be represented by the following formula:

[0068] ,

[0069] in, The labeled sample set represents a set of samples with known labels, which have been annotated by experts. Typically, these samples are few in number, but their labels are very accurate, and the model can use these sample labels to guide training. Let be the label consistency probability of sample i, that is, the probability that the model predicts the label for sample i. During the optimization process, the model will learn how to adjust... To make it closer to the real label And by propagating the consistency of labels, it ensures that similar samples have similar labels; The true label for sample i is derived from expert annotations. To control the hyperparameters of tag propagation strength, if Larger numbers indicate a stronger impact of label propagation, leading to stricter constraints on label consistency among similar samples; if The smaller the sample size, the weaker the impact of label consistency constraints, and the more likely the model will rely on already labeled samples. The Laplacian matrix, constructed based on the similarity between samples, is typically used in graph structures to represent the relationships between nodes. Specifically, the Laplacian matrix describes the similarity structure between samples and controls the consistency of labels among similar samples. The trace operation of a matrix calculates... The sum of the diagonal elements represents a global consistency in tag propagation. This is the transpose of the annotation consistency matrix. This is the matrix transpose symbol.

[0070] It should be noted that the goal of the above equation is to optimize the model by introducing label consistency constraints, enabling the model to reasonably propagate label information, allowing the labels of unlabeled samples to be reasonably inferred, while keeping the labels of labeled samples unchanged. In the above equation, the first term... This represents the label consistency constraint for labeled samples; for each labeled sample i, its label consistency probability... It should be as close as possible to the actual labels provided by experts. By minimizing and The difference between them ensures that the model's predictions match the actual labels. (Second term) This is a constraint term in label propagation, used to ensure label consistency among similar samples. This term uses a Laplace matrix to represent the similarity structure between samples, forming the label consistency matrix. This includes the label consistency probability for each sample. By forcing similar samples to have similar labels in the feature space, the optimization process will cause the labels of similar samples to develop in a consistent direction. In other words, this formula guides the model to learn the label relationships across samples through label consistency constraints, ensuring that similar samples have similar labels, while also guaranteeing the accuracy of the labels of labeled samples. Through such constraints, the model can not only make full use of the limited number of labeled samples, but also infer the labels of unlabeled samples through label propagation.

[0071] The correlation matrix constraint function in the objective function sparsifies the correlation matrix, ensuring that non-zero correlations only exist between cross-modal features with sufficient similarity. This constraint function, through L1 norm constraints, makes most elements in the correlation matrix close to zero, thereby reducing connections between irrelevant or low-similarity samples and maintaining the model's efficiency.

[0072] This function can be represented by the following formula:

[0073] ,

[0074] ,

[0075] in, This is the cross-modal correlation matrix. Let L1 be the L1 norm of the cross-modal incidence matrix, which is the sum of the absolute values ​​of all elements in the matrix. The purpose of the L1 norm is to promote the sparsity of the cross-modal incidence matrix, making most of the elements close to zero. This is achieved by minimizing... The model is encouraged to retain only a few important (high similarity) associations, while other low similarity associations are suppressed; st is an abbreviation for subject to, indicating that the function is restricted to the cross-modal feature similarity between sample i and sample j.

[0076] It should be noted that the above formula promotes sparsity of the cross-modal correlation matrix and uses a threshold. This ensures that associations are retained only between highly similar samples, thus optimizing the cross-modal learning model. It helps improve learning efficiency and reduces computation by focusing only on relevant sample pairs, avoiding invalid associations. Furthermore, by adjusting... This can control the model's sensitivity to similarity, thereby affecting the model's performance in cross-modal alignment.

[0077] Step 3: Construct a unified domain knowledge model based on the dataset and cross-modal association matrix, and perform structured modeling of intangible cultural heritage-related concepts, entities and their relationships to form a knowledge organization form that supports semantic expression of multimodal data and is machine-understandable.

[0078] The resulting data matrix was processed using a multimodal data cleaning and labeling model on the initial dataset. Annotation consistency matrix and cross-modal correlation matrix It should be noted that the data matrix set It contains cleared multimodal data; annotation consistency matrix The data contains the sample's category label and its corresponding confidence score; the cross-modal correlation matrix... The middle part records the correspondence between samples of different modalities.

[0079] In constructing a unified domain knowledge model, the core ontology class is first defined based on the annotation consistency matrix. The categories in the annotation consistency matrix are statistically analyzed, and the total confidence score for each category is calculated. If the total confidence score of a certain category is significantly higher than that of other categories, then that category is defined as the core ontology class. This constructs a hierarchical structure between categories and clarifies the parent-child relationships between different categories (for example, the category "Shu embroidery" can be defined as: Traditional textile craftsmanship → Embroidery → Shu embroidery).

[0080] Then, based on cross-modal association and knowledge rules derived from the cross-modal association matrix, relationship definitions are implemented. By utilizing information from cross-modal data pairs within the cross-modal association matrix, relationships between different entities are inferred, achieving alignment between different modalities. For example, Furthermore, since text i is "Shu embroidery production process" and image j is "Shu embroidery production steps diagram", the relationship is inferred to be a correlation. By matching the predefined relationship pattern in the knowledge graph, the rule relationship in the knowledge graph is established, thereby ensuring the reasoning ability of the knowledge graph for intangible cultural heritage data.

[0081] Finally, textual knowledge is extracted based on the data in the data matrix set. Textual knowledge extraction includes entity extraction and relation extraction. In this embodiment, entity extraction uses high-confidence labels (from the annotation consistency matrix) as supervision signals, implemented through a BiLSTM+CRF model. Relation extraction, through enhanced dependency resolution, combines the cosine similarity of text and images to extract and strengthen the confidence of relation types from the text.

[0082] Simultaneously, images and audio are used as supplementary knowledge. Based on the alignment of images and text, information from the cross-modal association matrix is ​​used to find the corresponding text for each image, thereby generating annotations for the images (for example, when the image describes "Shu embroidery," the text description "the production process of Shu embroidery" is automatically assigned to the image entity). Then, object detection models such as Mask R-CNN are used to identify specific regions in the images and match them with text entities based on similarity, thereby generating relational triples. Furthermore, Automatic Speech Recognition (ASR) is performed on the audio data, and entities are extracted from the generated text. Voiceprint recognition is used to determine whether the speaker in the audio matches the identity of the intangible cultural heritage inheritor, thereby verifying the authenticity and relevance of the recording.

[0083] Considering that different modalities may sometimes conflict in their descriptions of the same entity, this embodiment can also resolve the conflict by embedding entities from different modalities into a unified space to enhance cross-modal consistency and thus resolve the conflict problem.

[0084] Specifically, in this embodiment, a conflict loss function can be used to force the representations of the same entity to be similar across different modalities. The conflict loss function can be expressed by the following formula:

[0085] ,

[0086] in, Let be the conflict loss function. A set of cross-modal data pairs for the same entity. Data pairs of the same entity in m modes. For n-modal data pairs of the same entity, For the input data of the m-mode, For n-modal input data, For m-mode embedding mapping function, For n-modal embedding mapping functions.

[0087] It should be noted that the embedding mapping function is a mapping function that projects raw data (such as text, images, etc.) onto a unified embedding space. It can map the features of text and images to a shared vector space through neural networks. In this space, the different modal representations of the same entity should be as close as possible. This helps multimodal understanding, enabling the model to understand and reason about various forms of data (such as understanding the relationship between text and images). This is the square of the Euclidean distance, representing the difference between modality n and the entity embedding corresponding to modality n. The smaller this value, the closer the representations of the same entity are in different modalities. And... It contains all cross-modal data pairs of the same entity; in other words, in It includes all different modal pairs (e.g., text descriptions, images, and audio) corresponding to the same entity (such as a "non-heritage project"). In summary, the core of this conflict loss function is to embed the different modal representations (such as text and images) of the same entity into a shared vector space, and to minimize the Euclidean distance between them to make the representations as close as possible. For the same entity, the representations of different modalities should cluster in the same embedding space, rather than be scattered. By minimizing this conflict loss function, the relationships between multimodal data are enhanced, thereby helping the model better understand and reason about the complex knowledge in the field of intangible cultural heritage.

[0088] Step 4: The knowledge organization form output in Step 3 includes a complete ontology of the intangible cultural heritage domain (including classes, relations, constraints, etc.) and a set of cross-modal knowledge triples (entities and relations of text, images, and audio). These contents from the output knowledge organization form are used to construct a structured multimodal knowledge graph, enabling applications such as querying, reasoning, and visualization of intangible cultural heritage.

[0089] Example 2

[0090] This embodiment discloses a system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data. The system includes a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data as described in Embodiment 1.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, characterized in that, The multimodal knowledge graph construction method includes: S100. Collect intangible cultural heritage data in different modalities, perform preliminary cleaning and labeling of the collected data, establish preliminary associations between different modalities of data, and obtain the first version of the multimodal dataset; S200. The initial multimodal dataset is processed using a multimodal data cleaning and annotation model to obtain the final multimodal dataset, annotation consistency matrix, and cross-modal association matrix. S300. Construct a unified domain knowledge model to structurally model the concepts, entities and relationships related to intangible cultural heritage, forming a semantic expression and knowledge organization form that supports multimodal data. S400: Based on the data in the knowledge organization form output by S300, construct a structured multimodal knowledge graph; The multimodal data cleaning and labeling model ensures efficient and accurate data cleaning across different modalities through multiple constraints and loss mechanisms. Multimodal data cleaning and labeling models include: S210. Noise removal process: Remove noise from the data. Through noise removal loss, avoid interference from noisy samples and improve the overall data cleaning effect. S220, Cross-modal alignment, aligns features from different modalities so that samples with the same semantics have similar representations across different modalities, optimizing the relationship between modalities; S230. Label consistency: Ensure that samples of the same class in different modalities have consistent labels to avoid classification errors caused by inconsistent labeling. S240, the correlation matrix constraint, through sparsity constraint, forces the model to only establish the correlation between highly similar sample pairs, avoids the model from overfitting unrelated samples, and controls the relative importance of each objective in the overall optimization by adjusting the weight coefficients of different functions; The objective function of the multimodal data cleaning and labeling model is expressed by the following formula: ， in, The final multimodal dataset includes, The cleaned text data matrix, The cleaned image data matrix, The cleaned audio data matrix, This is the initial version of the multimodal dataset, including: Initial text data matrix, Initial image data matrix, Initial audio data matrix, To control the weighting coefficients of the denoising intensity, A data denoising function. The weighting coefficients are used to control the degree of modal alignment optimization. For cross-modal correlation optimization function, To control the weighting coefficients for annotation consistency, For annotation consistency constraint functions, The weighting coefficients are used to control the sparsity of the correlation matrix. The constraint function for the incidence matrix. For the annotation consistency matrix, This is the cross-modal correlation matrix; The data denoising function is used to ensure that the model can correctly extract effective features from the data. The data denoising function reduces the impact of noise by penalizing inconsistent or erroneous features in the model, thereby improving the quality of feature representation. The cross-modal association optimization function is used to optimize the alignment of cross-modal features. By making features from different modalities as close as possible in a shared embedding space, it improves the model's performance in cross-modal tasks. The label consistency constraint function is used to ensure that the model maintains the label consistency of samples during the learning process, and to ensure that the same class of samples in different modalities have consistent label information; The correlation matrix constraint function sparsifies the correlation matrix so that there is a non-zero correlation only between cross-modal features with sufficient similarity. The correlation matrix constraint function uses L1 norm constraint to make most of the elements in the correlation matrix close to zero, thereby reducing the connection between unrelated or low-similarity samples and maintaining the efficiency of the model. Building a unified domain knowledge model includes the following sub-steps: S310. Define the core ontology classes based on the annotation consistency matrix, perform statistical analysis on the categories in the annotation consistency matrix, and calculate the total confidence score for each category. If the overall confidence level of a certain category is significantly higher than that of other categories, then that category is defined as the core ontology class; S320. Based on the cross-modal association matrix, cross-modal association and knowledge rules are used to define relationships. By using the information of cross-modal data pairs in the cross-modal association matrix, the relationships between different entities are inferred, and alignment between different modalities is achieved. S330. Extract textual knowledge based on the data in the data matrix set.

2. The method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data according to claim 1, characterized in that, The intangible cultural heritage data in different modalities include: Textual data related to intangible cultural heritage includes: texts, documents, historical documents, research papers, project introductions, and expert interviews describing the background, development, and transmission of intangible cultural heritage projects. Image data related to intangible cultural heritage includes: complete photos of intangible cultural heritage projects, breakdown diagrams of the process steps, photos of inheritors and their skills demonstrations, and on-site images of folk activities; Audio and video data related to intangible cultural heritage include: audio and video recordings of the production process of intangible cultural heritage skills, and audio and video recordings of oral transmission of intangible cultural heritage.

3. The method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data according to claim 1, characterized in that, The preliminary cleaning and labeling includes: Remove duplicate paragraphs from the collected text data and standardize terminology; Preliminary associations are made between data of different modalities based on filenames, timestamps, and metadata. Label the core entities and their relationships in text, image, audio, and video data.

4. The method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data according to claim 1, characterized in that, The extracted text knowledge includes entity extraction and relation extraction, wherein... Entity extraction is achieved using high-confidence labels from the consistency matrix as supervision signals, and is implemented through a BiLSTM+CRF model. Relation extraction extracts and strengthens the confidence of relation types from text by combining enhanced dependency resolution with cosine similarity between text and images.

5. The method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data according to claim 4, characterized in that, The extracted text knowledge also includes: By using images and audio as supplementary knowledge, and based on the alignment of images and text, information from the cross-modal association matrix is ​​used to find the text corresponding to each image, thereby generating annotations for the images; Then, an object detection model is used to identify specific regions in the image and match them with text entities based on their similarity, thereby generating relation triples.

6. A system for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, characterized in that, The big data-based multimodal knowledge graph construction system for intangible cultural heritage includes: processor; The memory stores a computer program that, when executed by a processor, implements the method for constructing a multimodal knowledge graph of intangible cultural heritage based on big data, as described in any one of claims 1 to 5.

Images