An automatic construction method of a cross-modal knowledge graph for cultural relics
By constructing a cross-modal knowledge graph and utilizing text and image features, the problem that a single-modal graph cannot fully represent cultural relic knowledge is solved, thus realizing the structured management and effective utilization of cultural relic information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2026-03-31
AI Technical Summary
Existing single-modal textual atlases cannot fully represent the complexity of cultural relic knowledge and lack effective utilization of multimodal data on cultural relics.
By extracting textual and image data features from cultural relics, a cross-modal knowledge graph is constructed. Deep learning technology is used to establish the relationships between cultural relic entities. Named entity recognition is performed using BERT, BiLSTM, and CRF models. Feature representation and mapping are combined with deep convolutional networks and attention mechanisms to construct a unified cross-modal representation.
It has enabled structured management and organization of cultural relic information, improved the integrity and comprehensibility of cultural relic knowledge graphs, and provided a more effective way to utilize Internet cultural relic data.
Smart Images
Figure CN115129811B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of cross-modal learning and knowledge graphs, and relates to a method for constructing cross-modal relationships between image data and cultural relic text entities. Background Technology
[0002] Knowledge graphs aim to describe concepts, entities, and their relationships in the objective world in a structured form, providing a better ability to organize, manage, and understand the massive amounts of information on the internet. They represent an effective way to organize and utilize massive amounts of data. Cross-modal learning aims to effectively utilize the correlation between different modalities of content to optimize system performance, using deep learning techniques to alleviate the heterogeneity gap problem. Single-modal text graphs cannot fully represent the complexity of modern knowledge, and there is currently limited research on incorporating multimodal data from cultural relics into knowledge graphs for improvement. Summary of the Invention
[0003] To address the shortcomings and deficiencies in existing technologies, this invention provides an automatic method for constructing cross-modal knowledge graphs for cultural relics. This method extracts information from textual data of cultural relics heritage on the Internet, extracts corresponding image data features of cultural relics, models cross-modal relationships between the textual modality of cultural relic entities and the visual features of the image modality, and mines the associations between different modalities of the same cultural relic, thereby organizing cultural relic information on the Internet in a structured way.
[0004] To achieve the above objectives, the present invention adopts the following technical solution:
[0005] An automatic method for constructing cross-modal knowledge graphs for cultural relics includes the following steps:
[0006] Step 1, extract the text entities from the cultural relics:
[0007] The digital resources of cultural relics are divided into categories and attributes of cultural relic entities through fine-grained classification. After annotation, a text information dataset of cultural relics is obtained. The text information dataset of cultural relics is then input into the named entity recognition model to obtain the text entities of cultural relics.
[0008] Step 2: After constructing the cultural relic text knowledge graph ontology and filling it with data, extract the structured vector representations of the cultural relic entities:
[0009] Based on the categories and attributes of cultural relics entities obtained in step 1, as well as the relationships between entities, a cultural relics text knowledge graph ontology is constructed. The cultural relics text knowledge graph ontology is then filled with the cultural relics text entities obtained in step 1 to obtain the cultural relics text knowledge graph. The structured vector representations of cultural relics entities are then extracted from the cultural relics text knowledge graph.
[0010] Step 3, Representation of artifact image feature vectors:
[0011] Obtain an image dataset of cultural relics, use a deep neural network to extract features from each image of cultural relics, construct an image feature vector representation, project these image feature vector representations onto the structured information space of cultural relic entities, and obtain a cultural relic image feature vector representation with the same dimension as the structured vector representation of cultural relic entities in step 2. Use an attention-based method to construct an aggregated cultural relic image feature vector representation for cultural relic text entities.
[0012] Step 4, construct cross-modal relationships:
[0013] By mapping the textual entities of cultural relics in step 1, the structured vector representations of cultural relics entities in step 2, and the feature vector representations of cultural relics images obtained in step 3 to the public space, a unified cross-modal representation is obtained.
[0014] The present invention also includes the following technical features:
[0015] Specifically, in step 1, the named entity recognition model includes: a BERT pre-trained model, a BiLSTM model, and a CRF layer connected in sequence;
[0016] The process of inputting the text information dataset of cultural relics into a named entity recognition model to obtain the text entities of cultural relics includes the following steps: The text information dataset is input into a BERT pre-trained model to represent each character of the text information of cultural relics using a vector, and the position of each character in the vector space is obtained by learning from the surrounding characters in the text, and word vectors are output; then the word vectors are input into a BiLSTM model, and contextual features are automatically extracted through the contextual information of the text description, and the probability score matrix of the corresponding labels of the word vectors is output; finally, the CRF layer labels the probability score matrix output by BiLSTM to obtain the text entities of cultural relics.
[0017] Specifically, in step 2, the cultural relic text knowledge graph ontology is constructed using the Protégé knowledge graph ontology construction tool.
[0018] Specifically, step 3 includes:
[0019] Step 3.1: Based on the cultural relic entities in the constructed cultural relic text knowledge graph, obtain the corresponding image dataset of cultural relic from the digital cultural relic resources;
[0020] Step 3.2: The image of the cultural relic is fed as input into a deep convolutional network to construct an image feature vector representation; the deep convolutional network consists of a deep neural network AlexNet with five convolutional layers, two fully connected layers and one softmax layer;
[0021] Step 3.3: After obtaining the image feature vector of each cultural relic, the image feature vector in the high-dimensional vector space is projected to the low-dimensional vector space using a projection matrix, so that the feature vector of the image and the feature vector of the text entity are in the same dimension.
[0022] Step 3.4: Obtain attention weights through the feature vector representation of the cultural relic image and its corresponding feature vector representation of the text entity based on entity structure, and construct an aggregated feature vector representation of the cultural relic image for the text entity based on the obtained attention weights.
[0023] Specifically, step 4 includes:
[0024] Step 4.1: Combine the text entity representation of cultural relics from Step 1 with the image feature vector representation of cultural relics obtained in Step 3 to obtain a bimodal representation of the knowledge graph entity by concatenating the text feature vector and the image feature vector.
[0025] Step 4.2: Define scoring functions for the text entity of cultural relics, the structured vector representation of cultural relics entities, and the feature vector representation of cultural relics images and their combinations: expand the scoring function using the bimodal representation in Step 4.1 to obtain the cross-modal overall scoring function;
[0026] Step 4.3: The cross-modal overall scoring function obtained in step 4.2 is optimized using a loss function to minimize the global loss, thereby obtaining a unified cross-modal representation and realizing the fusion of multimodal information of cultural relics entities.
[0027] Compared with the prior art, the beneficial technical effects of this invention are:
[0028] This invention acquires text data of cultural relics from the Internet, extracts named entities of the relics, constructs a text knowledge graph, extracts the visual features of the entities based on the image information corresponding to the entities, supplements the knowledge of cultural relics by utilizing the correlation of different modal content, and finally models the relationship between the text entities in the knowledge graph and the extracted visual features to obtain a cross-modal knowledge graph of cultural heritage, enabling real-time structured storage and management of cultural heritage data. Attached Figure Description
[0029] Figure 1 This is a flowchart of the present invention.
[0030] Figure 2 This is a schematic diagram of the named entity recognition model of the present invention.
[0031] Figure 3 This is a flowchart illustrating the feature vector representation of cultural relic images according to the present invention.
[0032] Figure 4 This is a flowchart of the cross-modal relationship fusion process of the present invention. Detailed Implementation
[0033] This invention provides an automatic construction method for cross-modal knowledge graphs for cultural relics, comprising: (1) extracting information from unstructured data of digital resources of cultural relics and extracting text entities of cultural relics; (2) constructing a text knowledge graph ontology of cultural relics and filling it with data and then extracting the structured vector representation of cultural relics entities; (3) obtaining the feature vector representation of cultural relics images; (4) constructing cross-modal relationships, mapping the text entities of cultural relics, the structured vector representation of cultural relics entities and the feature vector representation of cultural relics images to a public space, obtaining a unified cross-modal representation, and constructing a cross-modal cultural heritage knowledge graph.
[0034] The steps for obtaining text entity data of cultural relics are as follows: using BERT and the training model to represent each character of the sentence as a vector, learning the position of each character in the vector space based on the characters around it in the text, using the BiLSTM model to effectively utilize contextual information and automatically extract contextual features, and finally labeling the sequence output by the BiLSTM with the CRF layer to obtain named entities.
[0035] The knowledge graph ontology construction first involves refining the fine-grained classification of cultural relics based on common cultural relic category attribute classification methods and knowledge extracted from text information, extracting the relationships between cultural relics and other entities and their corresponding concepts, and then using knowledge graph ontology construction tools to construct a cultural relic text knowledge graph ontology.
[0036] Feature extraction from images of cultural relics requires the use of deep neural networks to extract discriminative features from the images and construct an image feature representation for each image. These image feature representations are then projected from the image space to the entity space. Furthermore, many cultural relics exist in various scenarios with multiple images from different perspectives. To reduce the potential impact of noise or information loss on image representations, an attention-based approach is used to construct an aggregation-based image representation for each entity.
[0037] The purpose of constructing cross-modal relationships is to map the structured representation of entities in the cultural relic text knowledge graph, the text representation represented by the name of the cultural relic, and the feature representation of the image of the cultural relic obtained from the deep convolutional neural network to the public space, so as to obtain a unified multimodal representation and build a bridge between the cultural relic text knowledge graph and the image data of the cultural relic.
[0038] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.
[0039] Example 1:
[0040] This embodiment provides a method for automatically constructing cross-modal knowledge graphs for cultural relics, such as... Figure 1 The flowchart of the method of the present invention includes the following steps:
[0041] Step 1, extract the text entities from the cultural relics:
[0042] The digital resources of cultural relics (obtained from the Internet) are divided into categories and attributes of cultural relic entities through fine-grained segmentation. After annotation, a text information dataset of cultural relics is obtained. The text information dataset of cultural relics is then input into a named entity recognition model to obtain the text entities of cultural relics. The named entity recognition model consists of a BERT pre-trained model, a BiLSTM model, and a CRF layer connected in sequence.
[0043] Specifically, the process of inputting the text information dataset of cultural relics into a named entity recognition model to obtain the text entities of the cultural relics involves the following steps: The text information dataset is input into a BERT pre-trained model to represent each character of the text information of the cultural relics using a vector, and the position of each character in the vector space is obtained by learning from the surrounding characters in the text, outputting word vectors; then, the word vectors are input into a BiLSTM model, which automatically extracts contextual features based on the contextual information of the text and outputs a probability score matrix corresponding to the word vector labels; finally, a CRF layer annotates the probability score matrix output by the BiLSTM to obtain the text entities of the cultural relics. The structure of the named entity recognition model is as follows: Figure 2 As shown;
[0044] Step 1 above specifically includes the following steps:
[0045] Step 1.1: The digital cultural relic resources are obtained from the internet. The textual information dataset of the cultural relics obtained from these digital resources includes: referencing the classification methods for cultural relic attributes in the national cultural relic census and museum digital resources, fine-grained classification of the digital cultural relic resources on the internet is performed. In addition to public information such as category, dynasty, collection, and excavation information, attributes such as material, size, and texture, as well as specific attributes of a certain category of cultural relic, such as elements, patterns, and author, are also labeled for the cultural relics. The publicly available digital cultural relic resources on the internet are obtained, and the data is preprocessed. Following the BIO annotation format, the YEDDA tool is used to label the cultural relic entries in the data with various types of entities and detailed attributes of the cultural relics as tags, establishing the required labeled dataset for the network.
[0046] Step 1.2: Input the labeled cultural relic text information dataset into the BERT pre-trained model. BERT will obtain a segmented text sequence through word segmentation. Then, a special marker [CLS] is added to the beginning of the sequence to distinguish whether there is a contextual relationship between two input sentences. Sentences are separated by the marker [SEP]. The word vector embedding for each word is obtained: TokenEmbedding + Segment Embedding + Position Embedding. The word vectors are then input into a bidirectional Transformer for feature extraction, finally obtaining word vectors containing semantic features.
[0047] Step 1.3: BiLSTM receives each word vector from the BERT model output as input for each time step, obtaining the hidden state sequence of the forward LSTM layer. (Representing the hidden state sequences of the forward and reverse LSTM layers) (Indicates backward); the forward and backward hidden state sequences of each word vector are concatenated according to their positions to obtain the complete hidden state sequence, denoted as h. t =(h1,h2,h3…h n )∈R n×m Here, n represents the number of word vectors, and m represents the dimension of the word vectors. Then, the linear output layer in the BiLSTM model maps the complete hidden state sequence to an s-dimensional sequence (s being the number of label categories in the labeled dataset). Let the sequence of sentence features mapped after mapping, containing multiple word vectors, be matrix L = (l1, l2, l3…l…). n )∈R n×s ,l i ∈ R s Each dimension l of the matrix i,j Each corresponds to its word vector x i and corresponding category label y j The score values, i.e., the probability score matrix for each input;
[0048] Step 1.4: Use the probability score matrix output by BiLSTM as the input of the CRF layer. The CRF layer learns a transition score matrix (which represents the combination between all labeled states), processes the dependency between adjacent labels through random initialization, and obtains a more accurate probability score matrix for each input after training. The elements in this matrix are the scores of each word vector for each category label. The category label with the highest score is the category to which the cultural relic text entity belongs. Output the cultural relic text entity and its category.
[0049] While BiLSTM excels at handling long-range text information, it struggles with dependencies between adjacent labels. The CRF layer, however, learns a transition score matrix that represents all combinations of labeled states, with each element representing a score (or denormalized probability) for that combination. This transition score matrix can be randomly initialized and trained to obtain more rule-compliant combinations, thus avoiding non-compliant labels and outputting the predicted category of the cultural relic entity, completing the named entity recognition process.
[0050] Step 2: After constructing the cultural relic text knowledge graph ontology and filling it with data, extract the structured vector representations of the cultural relic entities:
[0051] Based on the categories and attributes of the cultural relics entities obtained in step 1, as well as the relationships between the entities, a cultural relics text knowledge graph ontology is constructed using the knowledge graph ontology construction tool (Protégé tool). The cultural relics text knowledge graph ontology is then filled with the cultural relics text entities obtained in step 1 to obtain the cultural relics text knowledge graph. The structured vector representations of the cultural relics entities are then extracted from the cultural relics text knowledge graph.
[0052] The TransE method is used to project all triples in the cultural relic text knowledge graph onto the structured information space of the cultural relic entities, resulting in structured vector representations of the cultural relic entities. Structured vector representation of relations The triples in the knowledge graph of cultural relics texts include the head entity h, the relation, and the relationship. The tail entity t;
[0053] Step 3, Representation of artifact image feature vectors:
[0054] A dataset of images of cultural relics is obtained. Features are extracted from each image using a deep neural network to construct image feature vector representations. These image feature vector representations are then projected onto the structured information space of the cultural relic entities, resulting in image feature vector representations of the same dimension as the structured vector representations of the cultural relic entities obtained in step 2. An attention-based method is then used to construct aggregated image feature vector representations for the textual entities of the cultural relics. Figure 3 As shown;
[0055] Specifically, step 3 includes:
[0056] Step 3.1: Based on the cultural relic entities in the constructed cultural relic text knowledge graph, obtain the corresponding image datasets of cultural relics from the museum's publicly available digital resource database and encyclopedia entries;
[0057] Step 3.2: Input the image of the cultural relic into a deep convolutional network to construct an image feature vector representation;
[0058] This deep convolutional network, AlexNet, is a deep neural network consisting of five convolutional layers, two fully connected layers, and one softmax layer. Before AlexNet extracts image features, each artifact image is cropped to 224×224 pixels, and the output is set to 4096 dimensions in the second fully connected layer (also known as fc7). The i-th image (img) i After processing by AlexNet, a 4096-dimensional image space feature vector f(img) is obtained. i );
[0059] Step 3.3: After obtaining the feature vector of each artifact image, a projection matrix is used to project the image feature vector in the high-dimensional vector space to the low-dimensional vector space, so that the feature vector of the image and the feature vector of the text entity are in the same dimension; the low-dimensional vector representation p of the i-th image in the entity space. i Defined as:
[0060] p i =M·f(img i )
[0061] Where M is d i ×d s The projection matrix, d i d represents the dimension of image features. s Represents the dimension of the physical space; f(img i ) represents the i-th image feature vector in the image space;
[0062] Step 3.4: Attention is generated through the image feature representation of the cultural relic and its corresponding entity-structure-based vector representation; for the i-th image feature representation of the k-th entity... The attention weight represented by this feature is as follows:
[0063]
[0064] in, The structure-based feature representation of the k-th entity is represented by the attention weights, which are derived from the low-dimensional image features of the i-th image of entity k. With structure-based features The product of these is used as the exponent of the exponential function exp, and the resulting value is divided by the sum of the feature exponential function values of all images of the entity.
[0065] Based on the obtained attention weights For multiple images of the k-th entity By calculating the weighted average, the aggregated image representation of the k-th entity can be obtained.
[0066]
[0067] This allows us to obtain an aggregated representation of multiple images for each cultural relic entity;
[0068] Step 4, construct cross-modal relationships:
[0069] The structured vector representations of the cultural relic text entities in step 1, the cultural relic entities in step 2, and the cultural relic image feature vector representations obtained in step 3 are mapped to a common space to obtain a unified cross-modal representation; a bridge is constructed between the cultural relic text knowledge graph and the cultural relic image data; the flowchart is shown below. Figure 4 As shown;
[0070] The cross-modal knowledge graph of cultural relics to be constructed is represented as follows: Where ε is the entity set. It is a set of relationships between entities. It is a set of triples of cultural relic knowledge. For each head and tail entity h,t∈ε, three representations (embeddings) are defined: structured representation Text representation and image representation Where N, M, and P are the corresponding dimensions.
[0071] The definition of the triplet scoring function in the TransE method is extended to integrate the structure and multimodal representation of cultural relic entities in knowledge graphs;
[0072] Step 4.1: Combine the textual entity representation of the cultural relic from Step 1 with the image feature vector representation of the cultural relic obtained in Step 3 to obtain a bimodal representation of the knowledge graph entity, which is a concatenation of the textual feature vector and the image feature vector; Head entity h m and tail entity t m The bimodal representation is as follows:
[0073]
[0074]
[0075] The above This represents the composition operator, used to combine text and image representations into a single bimodal representation. Here, vector concatenation is used to combine the text representation of a given entity e. w and image representation e i When pieced together, they form a bimodal representation e m ;
[0076] Step 4.2 defines a specific scoring function for the text entity of the cultural relic, the structured vector representation of the cultural relic entity, and the feature vector representation of the cultural relic image, as well as their combinations:
[0077] Input the structured vector representation of triples in the knowledge graph into the scoring function: E S =||h s +r s -t s ||, hs is the structured vector representation of the head entity and its relation. The tail entity t;
[0078] The scoring function is expanded using the bimodal representation from step 4.1, transferring the structured representation-based scoring function from the equation defined by TransE to the multimodal scenario.
[0079] E M1 =||h m +r s -t m ||
[0080] Based on the above formula, another scoring function is extended, which considers structured representation in addition to bimodal representation, as shown below:
[0081] E M2 =||(h m +h s )+r s -(t m +t s )||
[0082] To ensure that structured representations and bimodal representations are learned in the same space, the following scoring function is defined:
[0083] E SM =||h s +r s -t m ||
[0084] E MS =||h m +r s -t s ||
[0085] The cross-modal overall scoring function for a triplet is defined as:
[0086] E(h,r,t)=E S +E M1 +E M2 +E SM +E MS
[0087] Step 4.3: Optimize the scoring function obtained in Step 4.2 using a loss function:
[0088] For head and tail entities, the objective is to minimize the ranking loss between the scoring functions of positive and negative triples; the corresponding loss function is ultimately defined as:
[0089]
[0090]
[0091] Here, γ is a marginal parameter that controls the difference in scoring functions between positive triples and negative triples;
[0092] The ultimate goal is to minimize the global loss:
[0093] The structured and bimodal representations of head entities, tail entities, and relationships are fed into a feedforward neural network architecture. Fully connected layers map these inputs into a common space for training, thereby achieving the fusion of multimodal information of cultural relic entities.
Claims
1. A method for automatically constructing a cross-modal knowledge graph for cultural relics, characterized in that, The method comprises the following steps: Step 1, extracting cultural relic text entity: The cultural relic digital resource is divided into fine granularity to obtain the category of cultural relic entity and the attribute of cultural relic entity, and the cultural relic text information dataset is obtained after labeling, the cultural relic text information dataset is input into the named entity recognition model to obtain the cultural relic text entity; Step 2, constructing cultural relic text knowledge graph ontology and filling data to extract structured vector representation of cultural relic entity: According to the category of cultural relic entity and the attribute of cultural relic entity obtained in step 1, and the association relationship between entities, the cultural relic text knowledge graph ontology is constructed, and the cultural relic text knowledge graph is obtained by filling the cultural relic text knowledge graph ontology with the cultural relic text entity obtained in step 1, and the structured vector representation of the cultural relic entity is extracted from the cultural relic text knowledge graph; Step 3, cultural relic image feature vector representation: An image dataset of cultural relics is obtained, and the features in each cultural relic image are extracted by using a deep neural network to construct an image feature vector representation, the image feature vector representation is projected into the structured information space of the cultural relic entity, the cultural relic image feature vector representation with the same dimension as the structured vector representation of the cultural relic entity in step 2 is obtained, and an attention mechanism-based method is used to construct an aggregated cultural relic image feature vector representation for the cultural relic text entity; Step 4, constructing cross-modal relationship: The cultural relic text entity in step 1, the structured vector representation of the cultural relic entity in step 2 and the cultural relic image feature vector representation obtained in step 3 are mapped to a common space to obtain a unified cross-modal representation; In the step 1, the named entity recognition model comprises a BERT pre-training model, a BiLSTM model and a CRF layer connected in sequence; The cultural relic text information dataset is input into the named entity recognition model to obtain the cultural relic text entity, which comprises the following steps: the cultural relic text information dataset is input into the BERT pre-training model to represent each character of the cultural relic text information with a vector, and the position of the character in the vector space is obtained according to the learning of the characters around the character in the text, and a word vector is output; then the word vector is input into the BiLSTM model, the context features are automatically extracted through the context information of the text introduction, and then a possibility score matrix of the label corresponding to the word vector is output, finally the CRF layer labels the possibility score matrix output by the BiLSTM to obtain the cultural relic text entity; The step 4 comprises: Step 4.1, combining the cultural relic text entity in step 1 and the cultural relic image feature vector representation obtained in step 3 to obtain a double-modal representation of the text feature vector and the image feature vector spliced knowledge graph entity; Step 4.2, defining a scoring function for the cultural relic text entity, the structured vector representation of the cultural relic entity and the cultural relic image feature vector representation and its combination: expanding the scoring function in step 4.1 to obtain a cross-modal overall scoring function; Step 4.3, using a loss function to optimize the cross-modal overall scoring function obtained in step 4.2 to minimize the global loss, obtaining a unified cross-modal representation, and realizing the fusion of the multi-modal information of the cultural relic entity.
2. The artifact-oriented cross-modal knowledge graph automatic construction method of claim 1, wherein, The step 2 Chinese cultural relic text knowledge graph ontology is constructed by using a knowledge graph ontology construction tool, Protégé. 3.The method of claim 1, wherein, The step 3 includes: Step 3.1, according to the cultural relic entity of the constructed cultural relic text knowledge graph, image data sets of corresponding cultural relics are obtained from cultural relic digital resources; Step 3.2, cultural relic images are taken as input to a deep convolutional network to construct image feature vector representations; the deep convolutional network includes a deep neural network AlexNet with five convolutional layers, two fully connected layers and a softmax layer; Step 3.3, after obtaining the image feature vector of each cultural relic, a projection matrix is used to project the image feature vector in a high-dimensional vector space to a low-dimensional vector space, so that the feature vector of the image and the feature vector of the text entity are in the same dimension; Step 3.4, attention weights are obtained through the cultural relic image feature vector representation and the corresponding entity structure-based text entity feature vector representation, and an aggregated cultural relic image feature vector representation is constructed for the cultural relic text entity according to the obtained attention weights.