Celadon cross-modal knowledge graph construction method based on visual language model

By using the image multi-feature mapping method of VLP model in the matching of celadon images and text descriptions, the problem of insufficient matching accuracy and diversity in cross-modal knowledge graph construction is solved, and more efficient celadon details recognition and Knowledge graph construction.

CN118981541APending Publication Date: 2024-11-19ZHEJIANG UNIV OF TECH

Patent Information

Application Number
CN202410974053.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the construction of cross-modal knowledge graphs, especially when celadon images match text descriptions, there are problems such as manual annotation dependence, noise influence, instability and inconsistent semantic scales, resulting in insufficient matching accuracy and diversity.

Method used

Using the image multi-eigenmap cross-modal entity alignment method (IMFM-CMEA) based on the VLP model, by extracting the contour, texture and color local features of the celadon image, using a gated multivariate fusion device and a multi-layer fully connected network, the Features are mapped to a unified space, combined with the InfoNCE loss function optimization model, cross-modal entity alignment is achieved.

Benefits of technology

The cross-modal recognition accuracy of celadon details is significantly improved, the accuracy and diversity of the knowledge graph are enhanced, and the matching inaccuracy and noise problems in the prior art are alleviated.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981541A_ABST
    Figure CN118981541A_ABST
Patent Text Reader

Abstract

The invention discloses a celadon cross-modal knowledge graph construction method based on a visual language model. The method comprises the following steps: extracting local features in contour, texture and color aspects from a celadon image; then, a multi-element fusion device with gating is introduced to dynamically fuse a plurality of image features; further through a multi-layer full-connection network, learning to map the fusion feature to a proper intermediate representation space so as to guide a text encoder to generate a text feature which is more matched with the image feature; and finally, training and optimizing the model by means of an InfoNCE loss function. According to the method, on the premise that visual language model parameters are not changed, the effective intermediate representation of the image features is fully mined to complete reconstruction of the text features, the cross-modal recognition accuracy of the celadon detail features is improved, and effective support is provided for construction of the celadon cross-modal knowledge graph.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of visual processing, natural language processing and knowledge graph construction. The method involves the analysis and processing of visual data, including the extraction and analysis of celadon image features, as well as the semantic analysis and information extraction of text data. By integrating celadon image features and text features, the systematic representation and display of celadon shape, color, texture and other information can be achieved. Background Art

[0002] The core task of building a cross-modal knowledge graph is to achieve effective association between entities in the text knowledge graph and the corresponding images. Existing research mainly focuses on two directions: from image to text and from text to image.

[0003] The idea of ​​going from image to text usually first identifies and determines the visual elements in the image, and then annotates these elements with entities or concepts in the atlas. For example, combining the YOLOv3 (You Only Look Once version 3) technology with the text features in the atlas can achieve matching between characters in the oracle bone rubbing image and modern Chinese characters. However, this method relies on manual annotation and expert verification of image data, which has limitations. Another method is to collect images from the Internet through a fixed vocabulary, and use the visual knowledge extracted from the image to enhance the semantic understanding ability of pre-trained detectors and classifiers. The representative technology is NEIL (Never Ending Image Learner), which greatly reduces the cost of manual annotation. However, the images collected by this method may contain noise, which can easily have an adverse effect on the performance of the model. There is also a subtitle-guided visual saliency method, which maps the words or phrases in the subtitles to the relevant areas of the static image and generates the corresponding saliency map to obtain the visual objects and corresponding text labels. However, saliency methods are sensitive to input and it is difficult to produce stable and reliable results. In addition, image and text features are connected based on the attention mechanism in a shared multimodal subspace, but it is easy to cause inaccurate matching due to the inconsistent semantic scales of images and texts.

[0004] The idea of ​​going from text to image is to match the corresponding images for the entities or concepts in the text knowledge graph. Using online encyclopedias is a cost-effective way to find high-quality images that meet the conditions, but there are cases where there are no corresponding images. In addition, many images are only indirectly related to entities or concepts and cannot accurately reflect the true characteristics of the entities. Using web search engines can easily collect images related to the specified name, but may bring some irrelevant noise images. In order to effectively reduce the noise problem caused by entity labels with multiple meanings, the specific type information of the entity can be combined to clarify the query intent. To ensure the diversity of the selected images, a retrieval model that can filter out similar images can be constructed. Using online encyclopedias and web search engines can complement each other and jointly improve the accuracy and diversity of the graph.

[0005] Existing vision-language models usually adopt CLIP (Contrastive Vision-Language Pretraining) with contrastive learning strategy, and design a large-scale VLP (Vision-Language Pretraining) model with hundreds of millions of training data. Images and texts are extracted by CLIP encoders respectively, and then mapped into a shared feature space. The similarity of positive sample pairs and the difference of negative sample pairs are optimized by calculating cosine similarity. By creating a noisy dataset of 1.8 billion image-text pairs, a VLP model ALIGN (ALarge-scale Image and Noisy-text embedding) with stronger generalization ability was trained. In order to make these VLP models better adapt to specific downstream tasks, it is usually necessary to fine-tune the model using proprietary data. The pre-training-prompt method is a technology that effectively utilizes the knowledge of the VLP model. It can find the optimal prompt to adapt to downstream tasks without adjusting the parameters of the pre-trained model. Since all parameters of the pre-trained model do not need to be updated, the computational cost is greatly reduced.

[0006] By using learnable vectors to optimize the context words in the prompt, the CoOp (Context Optimization) model can be more flexible to adapt to specific downstream tasks. VPT (Visual Prompt Tuning) is proposed, which is the first attempt to use the prompt mechanism for the image encoder, that is, introducing a small number of trainable parameters in the image input space while keeping the model backbone unchanged. A small neural network UPT (Unified Prompt Tuning) is designed to simultaneously optimize the learnable prompts of text and vision, achieving better results than single modal prompts. MaPLe (Multi-modal Prompt Learning) promotes the tight coupling between visual-language prompts by learning in multiple transformer layers of image and text encoders.

[0007] Pic2Word (Mapping Pictures to Words) uses image features as cues and maps them to the text modality, thereby narrowing the similarity between image and text features and improving the corresponding matching ability between the two. Summary of the invention

[0008] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a method for constructing a cross-modal knowledge graph of celadon based on a visual language model. It mainly proposes a cross-modal entity alignment method of image multi-feature mapping based on the VLP model, referred to as IMFM-CMEA (cro-ss-modal entity alignment method based on imag-e multi-featuremapping). This method realizes the alignment of cross-modal entities by mapping the features of different modalities into a unified feature space, which is particularly suitable for matching celadon images with text descriptions. Furthermore, this method is also applied to the construction of a cross-modal knowledge graph of celadon. The public VLP model initializes the image and text encoders therein, and keeps the encoder parameters unchanged.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] A method for constructing a cross-modal knowledge graph of celadon based on a visual language model extracts local features of contour, texture and color from celadon images; then a gated multi-fuser is introduced to dynamically fuse multiple image features; further, through a multi-layer fully connected network, learning is used to map the fused features to a suitable intermediate representation space to guide the text encoder to generate text features that better match the image features; finally, the model is trained and optimized with the help of the InfoNCE loss function.

[0011] Further, the method comprises the following steps:

[0012] Step 1: Construction of the visual language model. Based on the model, the image multi-feature mapping cross-modal entity alignment method realizes cross-modal entity alignment by mapping the features of different modalities into a unified feature space, which is especially suitable for matching celadon images with text descriptions. The public VLP model initializes the image and text encoders, and keeps the encoder parameters unchanged.

[0013] Step 2: First, the ontology modeling of the pattern layer is carried out. On the basis of fully analyzing the knowledge in the porcelain field, the concept classes, hierarchical structures and attributes of the porcelain ontology are defined. The refined pattern layer can support the construction of the data layer. First, the data required for the data layer are collected from multiple sources, and then they are mapped into an initial set of triples according to the rules and constructed into a text knowledge graph. Furthermore, the IMFM-CMEA method is used to perform cross-modal alignment on the image data contained therein. Prior to this, it is necessary to create a ChinaWare dataset containing images and text titles for IMFM-CMEA training. Finally, the image and text data are mapped into a series of aligned cross-modal triple knowledge and stored in the text knowledge graph.

[0014] Furthermore, in step 1, the steps of constructing the visual language model are as follows:

[0015] Step 1.1: Extract local multi-features from images: In IMFM-CMEA, i.e., image multi-feature extraction and cross-modal entity alignment, image multi-feature extraction plays a vital role. By capturing the unique image features of celadon in terms of shape, pattern, and color, it can provide key visual information for the subsequent reconstruction of text features. This local multi-feature selection strategy can significantly improve the effective association and alignment of entities between different modalities.

[0016] For a given image I, local features in contour, texture and color are extracted based on digital image processing technology. In order to highlight the most prominent part of the image, U2-Net (two-level nested u-structure) is first used to segment the image to depict the main contour of the porcelain shape and preliminarily remove invalid pixels.

[0017] Then, the image is preprocessed to improve the quality of the image. First, the image is smoothed by Gaussian filtering to reduce random noise and small fluctuations in the image. Then, the contrast of the image is enhanced to make the difference between different parts of the image more obvious and the texture and structure more prominent. Furthermore, the Canny operator is used to identify the key structural contours of the subject in the image. The Canny operator is an edge detection algorithm with high accuracy and excellent noise resistance. At the same time, the local binary pattern (LBP) with strong grayscale and rotation invariance is used to capture the texture information of the subject. Then, the color features are extracted by Gaussian mixture models (Gaussian Mixture Models, GMMs). The model is composed of multiple Gaussian distributions superimposed and trained by the expectation maximization algorithm. It can adapt to complex data distribution forms. Before color feature extraction, the image is first converted to the HSI (Hue-Saturation-Intensity) color space, because the HSI space is closer to the human perception of color and is a more intuitive color representation than RGB (Red-Green-Blue).

[0018] The image encoder converts all image features μ into corresponding shape feature vectors μ S , pattern feature vector μ P and color feature vector μ C ;

[0019] By capturing the local features of celadon images in multiple dimensions, the entity association and alignment effects between different modalities are improved, providing a solid foundation for building a multimodal knowledge graph of celadon.

[0020] Step 1.2: Image multi-feature fusion: A gated multi-fuser is used to adaptively assign weights to the input image features and local multi-features to generate reliable fusion features. The image multi-feature fusion strategy draws on the gating-based feature fusion strategy. During the training process, this strategy can automatically assign weights to each feature without manual setting. The gated multi-fuser first learns the appropriate fusion weights for each feature to integrate multiple features μ,μ of a given image. S ,μ P ,μ C , the formula is:

[0021]

[0022] z i =σ i (LN i (μ ′ ))

[0023] In the formula, μ′ represents the result of the preliminary integration of multiple features of the image; LN represents the linear layer; represents the splicing calculation; z i represents the fusion weight corresponding to each image feature, i∈[0,3],z i ∈[0,1]; σ represents the Sigmoid activation function;

[0024] Finally, when performing feature fusion, the proportion of each image feature is determined by the fusion weight z i Determine that the formula is:

[0025]

[0026] In the formula, μ F represents the fusion result of multiple features of the image; M represents the number of features to be fused, M∈[0,3]; θ represents the tanh activation function; μ i Represents the features to be fused μ,μ S ,μ P ,μ C ;

[0027] Step 1.3: Text feature mapping reconstruction: The input fusion feature is converted into a mapping feature through a multi-layer fully connected mapper, and then it is fused with the encoded text feature and input into the text encoder to make the reconstructed text feature closer to the matching image feature. The image feature is used to guide the reconstruction of the text feature ν, making it closer to the image feature μ. First, a multi-layer fully connected mapper is designed to transform the fusion feature μ into a mapping feature. F Transformed into mapping feature λ, the process formula is:

[0028]

[0029] In the formula, λ represents the mapping feature; φ represents the ReLU (rectified linear unit) activation function, where there is no ReLU in the output layer; η represents the dropout regularization technology, where there is no dropout in the output layer; l represents the number of fully connected propagation layers; m represents the number of hidden neurons in the middle layer; o represents the output dimension of the mapping feature;

[0030] Next, the mapping feature λ is fused with the encoded text T and then input into the text encoder. The calculation formula is:

[0031]

[0032] In the formula, v represents the reconstructed text feature; f(·) represents the text encoder; ψ represents the Tokenizer text encoding; T represents the text; s represents the scaling range of the mapping feature λ;

[0033] Step 1.4: Cross-modal contrastive learning: Construct the InfoNCE (information noise contrastive estimation) loss function for IMFM-CMEA training, and obtain the similarity score between the image and the text through similarity calculation; refer to the cross-modal contrastive learning idea in CLIP, and use the InfoNCE loss function to optimize the learning results of the model, that is, during the training process, establish the connection between the image feature μ and the text feature ν, and measure the similarity between them by calculating the cosine similarity between the cross-modal features; first, calculate the text-to-image contrast loss L T→I , the formula is:

[0034]

[0035] Where N is the number of image-text pairs; v i ,μ i represents a positive sample pair; v i ,μ j represents a negative sample pair; τ∈R represents a learnable temperature coefficient;

[0036] Next, calculate the image-to-text contrast loss L I→T :

[0037]

[0038] Finally, we get the final contrast loss of the model.

[0039]

[0040] Furthermore, the process of step 2 is as follows:

[0041] Step 2.1: Modeling of the pattern layer ontology: A top-down seven-step method is used to construct the knowledge ontology in the field of porcelain. Porcelain knowledge is divided into five categories: basic, modeling, craftsmanship, function and culture, and the key concepts, hierarchical structures and related attributes are clearly defined. Porcelain is the top-level structure of the ontology, and each piece of porcelain is regarded as an independent entity. Basic, modeling and craftsmanship are the three core concept categories of porcelain. The kiln and collection subcategories belong to the basic concept category and are presented in the form of text descriptions. The modeling concept category is subdivided into subcategories such as shape and components. The craft concept category has a subcategory of glaze color. Shape and components describe the physical appearance characteristics of porcelain. Patterns depict the patterns on the surface of porcelain. Colors represent the color composition on the body after glaze firing. Glaze color is a description of the color classification of porcelain. These are important visual features of porcelain. Relational attributes are designed according to the defined concept categories to express The connection between each concept and the porcelain entity is expressed in the format of has[concept class]Of / In; in the design of data attributes, a unique identifier (id) is used to accurately identify different entities or relationships; Chinese names (name) or English names (enName) are used to intuitively distinguish different entities or relationships; dataUrl is used to record the storage location of the data; label is used to record the unstructured text description of the entity; longitude and latitude are used to record the geographic coordinate information of the kiln and collection; a complete porcelain ontology can provide a rule framework for the subsequent extraction of celadon-related entities and attributes. First, relevant knowledge content is collected and sorted out, and then entities corresponding to the ontology concept categories are extracted and integrated into the ontology. Then specific data attributes are filled in, and the mutual connection between entities is clarified based on the relationship attributes defined in the ontology.

[0042] Step 2.2: Construction of the initial triple set in the data layer: The graph data layer is constructed using a bottom-up approach. In order to enrich the content of the graph, some structured data is extracted from public Internet data, and the initial triple set is formed according to the rules defined by the porcelain ontology;

[0043] Step 2.3: Construction of ChinaWare dataset: To provide IMFM-CMEA with more comprehensive knowledge of porcelain, the ChinaWare dataset was created based on the search engine method. First, representative entity information such as shapes, patterns, and glaze colors were selected based on the pre-defined porcelain ontology, and text titles were constructed based on this. Then, image data matching the titles were collected in the image search engine. After screening and cleaning, these image data and text titles together constituted the ChinaWare dataset.

[0044] Step 2.4: Cross-modal entity alignment and graph visualization: IMFM-CMEA trained with the ChinaWare dataset can be applied to the cross-modal entity alignment of celadon. First, IMFM-CMEA is used to score the similarity of image-text pairs. Then, based on the scoring results, the top-ranked image-text pairs with scores higher than the predetermined threshold are selected and converted into cross-modal triples. Finally, these triples are integrated and stored in Neo4j.

[0045] The visual language model part of the present invention is mainly composed of four modules:

[0046] Image local multi-feature extraction module: In IMFM-CMEA, image multi-feature extraction plays a vital role. By capturing the unique image features of celadon in terms of shape, pattern and color, it can provide key visual information for the subsequent reconstruction of text features. This local multi-feature selection strategy can significantly improve the effective association and alignment of entities between different modalities.

[0047] Specifically, for a given image, this paper extracts its local features in terms of contour, texture and color based on digital image processing technology. In order to highlight the most prominent part of the image, U2-Net (two-level nested u-structure) is first used to segment the image to depict the main contour of the porcelain shape and preliminarily remove invalid pixels. Then, image preprocessing is performed to improve the quality of the image. First, Gaussian filtering is used to smooth the image to reduce random noise and small fluctuations in the image; then, the contrast of the image is enhanced to make the difference between different parts of the image more obvious and the texture and structure more prominent.

[0048] Furthermore, the Canny operator is used to identify the key structural contours of the subject in the image. The Canny operator is an edge detection algorithm with high accuracy and excellent noise resistance. At the same time, the local binary pattern (LBP) with strong grayscale and rotation invariance is used to capture the texture information of the subject. Then, the color features are extracted through the Gaussian Mixture Models (GMMs). The model is composed of multiple Gaussian distributions superimposed and trained by the expectation maximization algorithm, which can adapt to complex data distribution forms. Before color feature extraction, the image is first converted to the HSI (Hue-Saturation-Intensity) color space, because the HSI space is closer to the human perception of color and is a more intuitive color representation than RGB (Red-Green-Blue).

[0049] The visualization result of the local features extracted after the input image is the image of the shape, pattern and color. Finally, the image encoder converts all image features μ into the corresponding shape feature vector μ S , pattern feature vector μ P and color feature vector μ C .

[0050] This method improves the entity association and alignment effects between different modalities by capturing the local features of celadon images in multiple dimensions, providing a solid foundation for building a multimodal knowledge graph of celadon.

[0051] Image multi-feature fusion module: A gated multi-fuser is used to adaptively assign weights to the input image features and local multi-features to generate reliable fusion features. The image multi-feature fusion strategy draws on the gating-based feature fusion strategy. During the training process, this strategy can automatically assign weights to each feature without manual setting. Specifically, the gated multi-fuser first learns the appropriate fusion weights for each feature to integrate multiple features μ,μ of a given image. S ,μ P ,μ C , the calculation formula is as follows:

[0052]

[0053] z i =σ i (LN i (μ′)

[0054] In the formula, μ′ represents the result of the preliminary integration of multiple features of the image; LN represents the linear layer; represents the splicing calculation; z i represents the fusion weight corresponding to each image feature, i∈[0,3],z i ∈[0,1]; σ represents the Sigmoid activation function.

[0055] Finally, when performing feature fusion, the proportion of each image feature is determined by the fusion weight z i The calculation formula is as follows:

[0056]

[0057] In the formula, μ F represents the fusion result of multiple features of the image; M represents the number of features to be fused, M∈[0,3]; θ represents the tanh activation function; μ i Represents the features to be fused μ,μ S ,μ P ,μ C .

[0058] Text feature mapping reconstruction module: The input fusion feature is converted into a mapping feature through a multi-layer fully connected mapper, which is then fused with the encoded text feature and input into the text encoder to make the reconstructed text feature closer to the matching image feature. Using image features to guide the reconstruction of text features v can make it closer to image features μ. First, a multi-layer fully connected mapper is designed to transform the fusion feature μ into a mapping feature. F Converted into mapping feature λ, the conversion process formula is as follows:

[0059] λ=φ(η(LN(l,m,o,μ F )))

[0060] In the formula, λ represents the mapping feature; φ represents the ReLU (rectified linear unit) activation function, in which there is no ReLU in the output layer; η represents the dropout regularization technology, in which there is no dropout in the output layer; l represents the number of fully connected propagation layers; m represents the number of hidden neurons in the middle layer; o represents the output dimension of the mapping feature.

[0061] Next, the mapping feature λ is fused with the encoded text T and then input into the text encoder. The calculation formula is as follows:

[0062]

[0063] Where ν represents the reconstructed text feature; f(·) represents the text encoder; ψ represents the Tokenizer text encoding; T represents the text; and s represents the scaling range of the mapping feature λ.

[0064] Cross-modal contrastive learning module: Construct the InfoNCE (information noise contrastive estimation) loss function for IMFM-CMEA training, and obtain the similarity score between the image and the text through similarity calculation. Drawing on the cross-modal contrastive learning idea in CLIP, the InfoNCE loss function is used to optimize the learning results of the model. That is, during the training process, the connection between the image feature μ and the text feature v is established, and the similarity between them is measured by calculating the cosine similarity between the cross-modal features. First, the formula is as follows to calculate the contrast loss L from text to image: T→I :

[0065]

[0066] Where N is the number of image-text pairs; v i ,μ i represents a positive sample pair; v i ,μ j represents a negative sample pair; τ∈R represents a learnable temperature coefficient.

[0067] Next, calculate the image-to-text contrast loss L I→T :

[0068]

[0069] Finally, we get the final contrast loss of the model.

[0070]

[0071] The complete construction framework of the cross-modal knowledge graph of celadon needs to be constructed in a top-down manner. First, the ontology modeling of the pattern layer is carried out. On the basis of fully analyzing the knowledge in the field of porcelain, the concept classes, hierarchical structures and attributes of the porcelain ontology are defined. The refined pattern layer can support the construction of the data layer. First, the data required for the data layer are collected from multiple sources, and then they are mapped into an initial set of triples according to the rules and constructed into a text knowledge graph. Furthermore, the IMFM-CMEA method is used to perform cross-modal alignment on the image data contained. Prior to this, a dataset containing images and text titles needs to be created for the training of IMFM-CMEA. Finally, the image and text data are mapped into a series of aligned cross-modal triple knowledge and stored in the text knowledge graph.

[0072] When constructing the knowledge ontology in the field of porcelain using the top-down seven-step method. Porcelain knowledge is divided into five categories: basic, modeling, craftsmanship, function and culture, and the key concepts, hierarchical structures and related attributes are clearly defined. Porcelain is the top-level structure of the ontology, and each piece of porcelain is regarded as an independent entity. Basic, modeling and craftsmanship are the three core concept classes of porcelain. Subcategories such as kilns and collections belong to the basic concept class and are presented in the form of text descriptions. The modeling concept class is subdivided into subcategories such as shape and parts. There is a subcategory of glaze in the craft concept class. Shape and parts describe the physical appearance characteristics of porcelain, patterns depict the patterns on the surface of porcelain, colors represent the color composition on the body after glaze firing, and glaze color is a description of the color classification of porcelain. These are all important visual features of porcelain. Design relationship attributes based on the defined concept class to express the connection between each concept and the porcelain entity, using the format of has [concept class] Of / In.

[0073] In terms of data attribute design, unique identifiers (id) are used to accurately identify different entities or relationships; Chinese names (name) or English names (enName) are used to intuitively distinguish different entities or relationships; dataUrl is used to record the storage location of data; label is used to record the unstructured text description of the entity; longitude and latitude are used to record the geographic coordinate information of kilns and collections. A complete porcelain ontology can provide a rule framework for the subsequent extraction of celadon-related entities and attributes.

[0074] First, relevant knowledge content is collected and sorted out, and then entities corresponding to the ontology concept categories are extracted and integrated into the ontology. Then specific data attributes are filled in, and the relationship between entities is clarified based on the relationship attributes defined in the ontology.

[0075] The atlas data layer is constructed using a bottom-up approach. In order to enrich the content of the atlas, data is extracted from professional books or authoritative industry websites. Based on the rules defined by the porcelain ontology, an initial set of triples is formed.

[0076] In order to give IMFM-CMEA more comprehensive knowledge about porcelain, it is necessary to construct a dataset for visual language model training. Specifically, first, based on the pre-defined porcelain ontology, representative entity information such as shape, pattern and glaze color is selected, and a text title is constructed based on this. Then, image data matching the title is collected in the image search engine. Through screening and cleaning, these image data and text titles together constitute the dataset.

[0077] IMFM-CMEA trained with the dataset can be applied to the cross-modal entity alignment of celadon. First, IMFM-CMEA is used to score the similarity of image-text data pairs. Then, according to the scoring results, the top-ranked image-text pairs with scores higher than the predetermined threshold are selected and converted into cross-modal triples. Finally, these triples are integrated and stored in Neo4j. By integrating the knowledge graph in the form of image data and text, not only the visual display effect of the graph is enriched, but also the accuracy and quality of the knowledge graph construction are further improved through intelligent entity alignment technology.

[0078] The beneficial effect of the present invention is that a visual language model is proposed to calculate the similarity scores of image and text features, and a method for constructing a cross-modal knowledge graph of celadon is proposed based on the model. The present invention effectively alleviates the problem of cross-modal entity alignment in the celadon field and significantly improves the cross-modal recognition accuracy of celadon detail features. The overall process demonstrates a method for constructing a cross-modal knowledge graph of celadon based on a visual language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] DRAWINGS In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0080] Figure 1 It is a framework diagram of step 2 of the present invention for constructing a cross-modal knowledge graph of celadon;

[0081] Figure 2 It is a diagram of the porcelain body of step 2.1 of the present invention. DETAILED DESCRIPTION

[0082] In order to explain the technical problems, technical solutions, and related implementation steps to be solved by the present invention in more detail and clarity, the implementation details of the present invention will be described in conjunction with the accompanying drawings so that technicians in the relevant fields can understand and implement the invention.

[0083] Reference Figure 1 and Figure 2 , a method for constructing a celadon knowledge graph based on a visual language model, wherein the visual language model extracts local features of contour, texture and color from the celadon image; then a gated multivariate fuser is introduced to dynamically fuse multiple image features; then, through a multi-layer fully connected network, learning is performed to map the fused features to a suitable intermediate representation space to guide the text encoder to generate text features that better match the image features; finally, the model is trained and optimized with the help of the InfoNCE loss function. And a celadon knowledge graph is constructed according to a cross-modal entity alignment method of image multi-feature mapping based on the model, the method comprises the following steps:

[0084] Step 1: Construction of the visual language model. Based on the model, the image multi-feature mapping cross-modal entity alignment method realizes cross-modal entity alignment by mapping the features of different modalities into a unified feature space. It is especially suitable for matching celadon images with text descriptions. The public VLP model initializes the image and text encoders and keeps the encoder parameters unchanged. The steps for building the visual language model are as follows.

[0085] Step 1.1: Extract local multi-features of the image: In image multi-feature extraction and cross-modal entity alignment, image multi-feature extraction plays a vital role. By capturing the unique image features of celadon in terms of shape, pattern and color, it can provide key visual information for the subsequent reconstruction of text features. This local multi-feature selection strategy can significantly improve the effective association and alignment of entities between different modalities;

[0086] Specifically, for a given image I, this paper extracts its local features in contour, texture and color based on digital image processing technology. In order to highlight the most prominent part of the image, U2-Net is first used to segment the image to depict the main contour of the porcelain shape and preliminarily remove invalid pixels;

[0087] Then, image preprocessing is performed to improve the image quality. First, Gaussian filtering is used to smooth the image to reduce random noise and small fluctuations in the image. Then, the contrast of the image is enhanced to make the differences between different parts of the image more obvious and the texture and structure more prominent. Furthermore, the Canny operator is used to identify the key structural contours of the subject in the image. The Canny operator is an edge detection algorithm with high accuracy and excellent noise resistance. At the same time, the local binary pattern with strong grayscale and rotation invariance is used to capture the texture information of the subject. Then, color features are extracted through the Gaussian mixture model. The model is composed of multiple Gaussian distributions superimposed and trained through the expectation maximization algorithm. It can adapt to complex data distribution forms. Before color feature extraction, the image is first converted to the HSI color space because the HSI space is closer to the human perception of color and is a more intuitive color representation than RGB.

[0088] The image encoder converts all image features μ into corresponding shape feature vectors μ S , pattern feature vector μ P and color feature vector μ C .

[0089] This method improves the entity association and alignment effects between different modalities by capturing the local features of celadon images in multiple dimensions, providing a solid foundation for building a multimodal knowledge graph of celadon.

[0090] Step 1.2: Image multi-feature fusion: A gated multi-fuser is used to adaptively assign weights to the input image features and local multi-features to generate reliable fusion features. The image multi-feature fusion strategy draws on the gating-based feature fusion strategy. During the training process, this strategy can automatically assign weights to each feature without manual setting. Specifically, the gated multi-fuser first learns the appropriate fusion weights for each feature to integrate multiple features μ,μ of a given image. S ,μ P ,μ C , the calculation formula is as follows:

[0091]

[0092] z i =σ i (LN i (μ′)

[0093] In the formula, μ′ represents the result of the preliminary integration of multiple features of the image; LN represents the linear layer; represents the splicing calculation; z i represents the fusion weight corresponding to each image feature, i∈[0,3],z i ∈[0,1]; σ represents the Sigmoid activation function;

[0094] Finally, when performing feature fusion, the proportion of each image feature is determined by the fusion weight z i The calculation formula is as follows:

[0095]

[0096] In the formula, μ F represents the fusion result of multiple features of the image; M represents the number of features to be fused, M∈[0,3]; θ represents the tanh activation function; μ i Represents the features to be fused μ,μ S ,μ P ,μ C ;

[0097] Step 1.3: Text feature mapping reconstruction: The input fusion feature is converted into a mapping feature through a multi-layer fully connected mapper, and then it is fused with the encoded text feature and input into the text encoder to make the reconstructed text feature closer to the matching image feature. Using image features to guide the reconstruction of text features v can make it closer to image features μ. First, a multi-layer fully connected mapper is designed to transform the fusion feature μ into a mapping feature. F Converted into mapping feature λ, the conversion process formula is as follows:

[0098]

[0099] In the formula, λ represents the mapping feature; φ represents the ReLU activation function, where there is no ReLU in the output layer; η represents the dropout regularization technology, where there is no dropout in the output layer; l represents the number of fully connected propagation layers; m represents the number of hidden neurons in the middle layer; o represents the output dimension of the mapping feature;

[0100] Next, the mapping feature λ is fused with the encoded text T and then input into the text encoder. The calculation formula is as follows:

[0101]

[0102] In the formula, ν represents the reconstructed text feature; f(·) represents the text encoder; ψ represents the Tokenizer text encoding; T represents the text; s represents the scaling range of the mapping feature λ;

[0103] Step 1.4: Cross-modal contrastive learning: Construct the InfoNCE loss function for IMFM-CMEA training, and obtain the similarity score between the image and the text through similarity calculation. Drawing on the cross-modal contrastive learning idea in CLIP, the InfoNCE loss function is used to optimize the learning results of the model. That is, during the training process, the connection between the image feature μ and the text feature v is established, and the similarity between them is measured by calculating the cosine similarity between the cross-modal features. First, the formula is as follows to calculate the contrast loss L from text to image: T→I :

[0104]

[0105] Where N is the number of image-text pairs; v i ,μ i represents a positive sample pair; v i ,μ j represents a negative sample pair; τ∈R represents a learnable temperature coefficient;

[0106] Next, calculate the image-to-text contrast loss L I→T :

[0107]

[0108] Finally, we get the final contrast loss of the model.

[0109]

[0110] Step 2: Construct a cross-modal knowledge graph of Celadon. Figure 1 The complete construction framework of the cross-modal knowledge graph of celadon is presented in the paper. It adopts a top-down construction method. First, the ontology modeling of the pattern layer is carried out. On the basis of fully analyzing the knowledge in the field of porcelain, the concept classes, hierarchical structures and attributes of the porcelain ontology are defined. The refined pattern layer can support the construction of the data layer. First, the data required for the data layer is collected from multiple sources. Then, according to the rules, they are mapped into an initial set of triples and constructed into a text knowledge graph. Furthermore, the IMFM-CMEA method is used to perform cross-modal alignment on the image data contained therein. Prior to this, it is necessary to create a ChinaWare dataset containing images and text titles for IMFM-CMEA training. Finally, the image and text data are mapped into a series of aligned cross-modal triple knowledge and stored in the text knowledge graph;

[0111] The process of step 2 is as follows:

[0112] Step 2.1: Modeling of the ontology at the model layer: A top-down seven-step approach is used to construct the knowledge ontology in the field of porcelain. Porcelain knowledge is divided into five categories: basic, modeling, craftsmanship, function, and culture, and the key concepts, hierarchical structures, and related attributes are clearly defined. Figure 2 It shows the parts of the porcelain ontology that are closely related to the image modality. Porcelain is the top-level structure of the ontology, and each piece of porcelain is regarded as an independent entity. Basic, modeling and craftsmanship are the three core concept categories of porcelain. Subcategories such as kilns and collections belong to the basic concept category and are presented in the form of text descriptions. The modeling concept category is subdivided into subcategories such as shapes and components. There is a subcategory called glaze color in the craft concept class; shape and parts describe the physical appearance characteristics of porcelain, patterns depict the patterns on the surface of porcelain, colors represent the color composition on the body after glaze firing, and glaze color is a description of the color classification of porcelain, which are all important visual features of porcelain; design relationship attributes based on the defined concept class to express the connection between each concept and the porcelain entity, using the format of has[concept class]Of / In. In the design of data attributes, use unique identifiers (id) to accurately identify different entities or relationships; use Chinese names (name) or English names (enName) to intuitively distinguish different entities or relationships; use dataUrl to record the storage location of data, such as the link address of an image; use label to record the unstructured text description of the entity; use longitude and latitude to record the geographic coordinate information of kilns and collections; a complete porcelain ontology can provide a rule framework for the subsequent extraction of celadon-related entities and attributes, first collect and sort out relevant knowledge content, then extract entities corresponding to the ontology concept categories, and integrate them into the ontology (such as the shape entity "jar" and the age entity "Southern Song"). Then fill in specific data attributes (such as id and name), and clarify the connections between entities based on the relationship attributes defined in the ontology (for example, the shape entity "bottle" is connected to two pieces of porcelain);

[0113] Step 2.2: Construction of the initial triplet set in the data layer: The atlas data layer is constructed in a bottom-up manner. In order to enrich the content of the atlas, some structured data is extracted from the public data on the Internet. According to the rules defined by the porcelain ontology, an initial triplet set of 2,840 items is formed, such as [porcelain, pattern, lotus petal pattern] and [porcelain, glaze color, green glaze]

[0114] wait;

[0115] Step 2.3: Construction of ChinaWare Dataset: To endow IMFM-CMEA with more comprehensive porcelain knowledge, the ChinaWare dataset was created based on the search engine method. Specifically, first, according to the predefined porcelain ontology, representative entity information such as shapes, patterns, and glaze colors was selected, and based on this, text titles were constructed, such as "Jade Spring Vase with Celadon Glaze". Then, image data matching the titles was collected in the image search engine. Through screening and cleaning, these image data and text titles together constituted the ChinaWare dataset;

[0116] Step 2.4: Cross-modal Entity Alignment and Atlas Visualization: IMFM-CMEA trained with the ChinaWare dataset can be applied to the cross-modal entity alignment work of celadon. First, IMFM-CMEA is used to score the similarity of text-image data pairs, and then according to the scoring results, the text-image pairs ranked靠前 and with scores higher than a predetermined threshold are selected and transformed into knowledge in the form of cross-modal triples, such as [porcelain image, glaze color, celadon glaze], [porcelain image, shape, Jade Spring Vase], and [porcelain image, pattern, string pattern], etc. Finally, these triple knowledge is integrated and stored in Neo4j;

[0117] By fusing the knowledge graph in the form of image data and text, not only the visual display effect of the graph is enriched, but also through the intelligent entity alignment technology, the accuracy and quality of the knowledge graph construction are further improved. The above entire process demonstrates a method for constructing a cross-modal knowledge graph of celadon based on a vision-language model proposed by the present invention.

[0118] The content described in the embodiments of this specification is only a list of implementation forms of the inventive concept and is only for illustrative purposes. The protection scope of the present invention should not be regarded as limited to the specific forms stated in this embodiment. The protection scope of the present invention also extends to equivalent technical means that can be想到 by those of ordinary skill in the art based on the inventive concept of the present invention.

Claims

1. A method for constructing a cross-modal knowledge graph of celadon based on a visual language model. Features: Local features of contour, texture and color are extracted from the celadon image; then a gated multi-fuser is introduced to dynamically fuse multiple image features; further, through a multi-layer fully connected network, learning is performed to map the fused features to a suitable intermediate representation space to guide the text encoder to generate text features that better match the image features; finally, the model is trained and optimized with the help of the InfoNCE loss function.

2. The method for constructing a celadon cross-modal knowledge graph based on a visual language model as claimed in claim 1, Features: The method comprises the following steps: Step 1: Construction of the visual language model. Based on the model, the image multi-feature mapping cross-modal entity alignment method realizes cross-modal entity alignment by mapping the features of different modalities into a unified feature space, which is especially suitable for matching celadon images with text descriptions. The public VLP model initializes the image and text encoders, and keeps the encoder parameters unchanged. Step 2: First, the ontology modeling of the pattern layer is carried out. On the basis of fully analyzing the knowledge in the porcelain field, the concept classes, hierarchical structures and attributes of the porcelain ontology are defined. The refined pattern layer can support the construction of the data layer. First, the data required for the data layer are collected from multiple sources, and then they are mapped into an initial set of triples according to the rules and constructed into a text knowledge graph. Furthermore, the IMFM-CMEA method is used to perform cross-modal alignment on the image data contained therein. Prior to this, it is necessary to create a ChinaWare dataset containing images and text titles for IMFM-CMEA training. Finally, the image and text data are mapped into a series of aligned cross-modal triple knowledge and stored in the text knowledge graph.

3. The method for constructing a celadon cross-modal knowledge graph based on a visual language model as claimed in claim 2, Features: In step 1, the steps for constructing the visual language model are as follows: Step 1.1: Extract local multi-features from images: In IMFM-CMEA, i.e., image multi-feature extraction and cross-modal entity alignment, image multi-feature extraction plays a vital role. By capturing the unique image features of celadon in terms of shape, pattern, and color, it can provide key visual information for the subsequent reconstruction of text features. This local multi-feature selection strategy can significantly improve the effective association and alignment of entities between different modalities. For a given image I, local features in contour, texture and color are extracted based on digital image processing technology. In order to highlight the most prominent part of the image, U2-Net is first used to segment the image to depict the main contour of the porcelain shape and initially remove invalid pixels. Then, image preprocessing is performed to improve the image quality. First, Gaussian filtering is used to smooth the image to reduce random noise and small fluctuations in the image. Then, the contrast of the image is enhanced to make the difference between different parts of the image more obvious, and the texture and structure more prominent. Furthermore, the Canny operator is used to identify the key structural contours of the subject in the image. The Canny operator is an edge detection algorithm with high accuracy and excellent noise resistance. At the same time, the local binary pattern with strong grayscale and rotation invariance is used to capture the texture information of the subject. Then, the color features are extracted through the Gaussian mixture model. The model is composed of multiple Gaussian distributions superimposed and trained by the expectation maximization algorithm. It can adapt to complex data distribution forms. Before color feature extraction, the image is first converted to the HSI color space. The image encoder converts all image features μ into corresponding shape feature vectors μ S , pattern feature vector μ P and color feature vector μ C ; By capturing the local features of celadon images in multiple dimensions, the entity association and alignment effects between different modalities are improved, providing a solid foundation for building a multimodal knowledge graph of celadon. Step 1.2: Image multi-feature fusion: A gated multi-fuser is used to adaptively assign weights to the input image features and local multi-features to generate reliable fusion features. The image multi-feature fusion strategy draws on the gating-based feature fusion strategy. During the training process, this strategy can automatically assign weights to each feature without manual setting. The gated multi-fuser first learns the appropriate fusion weights for each feature to integrate multiple features μ, μ of a given image. s , μ P , μ C , the formula is: z i =s i (LN i (m′)) In the formula, μ′ represents the result of the preliminary integration of multiple features of the image; LN means linear layer; represents the splicing calculation; z i represents the fusion weight corresponding to each image feature, i∈[0,3], z i ∈[0, 1]; σ represents the Sigmoid activation function; Finally, when performing feature fusion, the proportion of each image feature is determined by the fusion weight z i Determine that the formula is: In the formula, μ F represents the fusion result of multiple features of the image; M represents the number of features to be fused, M∈[0,3]; θ represents the tanh activation function; μ i Represents the features to be fused μ, μ s , μ P , μ C ; Step 1.3: Text feature mapping reconstruction: The input fusion feature is converted into a mapping feature through a multi-layer fully connected mapper, and then it is fused with the encoded text feature and input into the text encoder to make the reconstructed text feature closer to the matching image feature. The image feature is used to guide the reconstruction of the text feature v to make it closer to the image feature μ. First, a multi-layer fully connected mapper is designed to transform the fusion feature μ into a mapping feature. F Transformed into mapping feature λ, the process formula is: λ=φ(η(LN(l,m,o,μ F ))) In the formula, λ represents the mapping feature; φ represents the ReLU activation function, where there is no ReLU in the output layer; η represents the dropout regularization technique, where there is no dropout in the output layer; l represents the number of fully connected propagation layers; m represents the number of hidden neurons in the middle layer; o represents the output dimension of the mapping feature; Next, the mapping feature λ is fused with the encoded text T and then input into the text encoder. The calculation formula is: In the formula, v represents the reconstructed text feature; f(·) represents the text encoder; ψ represents the Tokenizer text encoding; T represents the text; s represents the scaling range of the mapping feature λ; Step 1.4: Cross-modal contrastive learning: Construct the InfoNCE loss function for IMFM-CMEA training, and obtain the similarity score between image and text through similarity calculation; refer to the cross-modal contrastive learning idea in CLIP, and use the InfoNCE loss function to optimize the learning results of the model, that is, during the training process, establish the connection between image features μ and text features v, and measure the similarity between them by calculating the cosine similarity between cross-modal features; first, calculate the text-to-image contrast loss L T→I , the formula is: Where N is the number of image-text pairs; v i , μ i represents a positive sample pair; v i , μ j represents a negative sample pair; τ∈R represents a learnable temperature coefficient; Next, calculate the image-to-text contrast loss L I→T : Finally, we get the final contrast loss of the model.

4. The method for constructing a celadon cross-modal knowledge graph based on a visual language model as claimed in claim 2 or 3, Features: The process of step 2 is as follows: Step 2.1: Modeling of the pattern layer ontology: A top-down seven-step method is used to construct the knowledge ontology in the field of porcelain. The porcelain knowledge is divided into five categories: basic, modeling, craftsmanship, function and culture, and the key concepts, hierarchical structures and related attributes are clearly defined. Porcelain is the top-level structure of the ontology, and each piece of porcelain is regarded as an independent entity. Basic, modeling and craftsmanship are the three core concept categories of porcelain. The kiln and collection subcategories belong to the basic concept category and are presented in the form of text descriptions. The modeling concept category is subdivided into subcategories such as shape and components. The craft concept category has a subcategory of glaze color. Shape and components describe the physical appearance characteristics of porcelain. Patterns depict the patterns on the surface of porcelain. Colors represent the color composition on the body after glaze firing. Glaze color is a description of the color classification of porcelain. These are all important visual features of porcelain. Relationship attributes are designed according to the defined concept categories. To express the relationship between each concept and the porcelain entity, the format of has[concept class]Of / In is used; in terms of data attribute design, a unique identifier (id) is used to accurately identify different entities or relationships; Chinese name name or English name enName is used to intuitively distinguish different entities or relationships; dataUrl is used to record the storage location of the data; label is used to record the unstructured text description of the entity; longitude and latitude are used to record the geographic coordinate information of the kiln and collection; a complete porcelain ontology can provide a rule framework for the subsequent extraction of celadon-related entities and attributes. First, relevant knowledge content is collected and sorted out, and then entities corresponding to the ontology concept categories are extracted and integrated into the ontology. Then specific data attributes are filled in, and the mutual relationship between entities is clarified based on the relationship attributes defined in the ontology. Step 2.2: Construction of the initial triple set in the data layer: The graph data layer is constructed using a bottom-up approach. In order to enrich the content of the graph, some structured data is extracted from public Internet data, and the initial triple set is formed according to the rules defined by the porcelain ontology; Step 2.3: Construction of ChinaWare dataset: To provide IMFM-CMEA with more comprehensive knowledge of porcelain, the ChinaWare dataset was created based on the search engine method. First, representative entity information such as shapes, patterns, and glaze colors were selected based on the pre-defined porcelain ontology, and text titles were constructed based on this. Then, image data matching the titles were collected in the image search engine. After screening and cleaning, these image data and text titles together constituted the ChinaWare dataset. Step 2.4: Cross-modal entity alignment and graph visualization: IMFM-CMEA trained with the ChinaWare dataset can be applied to the cross-modal entity alignment of celadon. First, IMFM-CMEA is used to score the similarity of image-text pairs. Then, based on the scoring results, the top-ranked image-text pairs with scores higher than the predetermined threshold are selected and converted into cross-modal triples. Finally, these triples are integrated and stored in Neo4j.

Citation Information

Patent Citations

  • Multi-modal knowledge graph construction method

    CN112200317A

Cited By

  • Image drawing system and method based on big data and artificial intelligence generation system technology

    CN119251343A

  • Oracle bone text image recognition method, device and equipment based on multi-modal large model and medium

    CN119251852A

  • LED display screen color adjusting system and method based on ambient light perception

    CN119626149A

  • LED Display Color Adjustment System and Method Based on Ambient Light Sensing

    CN119626149B

  • Scientific and technological text picture and text matching algorithm

    CN119782502A