Multi-modal domain knowledge graph construction method and system

Abstract

The invention discloses a multi-modal domain knowledge graph construction method and system, and belongs to the technical field of knowledge graphs. The method comprises the following steps: firstly, respectively processing text data and image data: dividing the text data into semi-structured text data and unstructured text data, directly extracting a triple from the semi-structured text data, and generating a triple text instance from the unstructured text data by using a UIE model; a description text is added for image data according to existing visual concepts, visual and semantic features are extracted and fused through a ResNet50-CA-RoBERTa model, visual concept categories are output, triple image examples are filled, and triple text examples and triple image examples are visually stored. According to the method, the problem of multi-modal data isomerism is solved, knowledge standardized representation and intelligent application can be supported, and the potential of the knowledge graph in multi-modal data organization is exerted.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph technology, specifically relating to a method and system for constructing a multimodal domain knowledge graph. Background Technology

[0002] With the rapid development of the internet and big data technologies, knowledge is exhibiting multimodal characteristics, with data resources in different modalities such as text and images constantly increasing. However, these different modalities are generally characterized by dispersion and heterogeneity, lacking a unified modeling and organization mechanism, thus limiting the systematic representation and efficient utilization of knowledge. Knowledge graphs, as an effective means of knowledge association, possess powerful information integration and knowledge organization capabilities, and have become a research hotspot in recent years.

[0003] Because the data structures and representations of different modal data differ significantly, current research on multimodal learning mainly focuses on two aspects: the semantic gap and the heterogeneous gap. The semantic gap is mainly reflected in the differences between humans and computers in their descriptions and understandings of the same entities, while the heterogeneous gap is mainly reflected in the differences in the feature representations of different modal data.

[0004] Wang Chunshan et al. proposed the crop disease identification model bimodalNet, which utilizes the correlation and complementarity of text and image data to achieve joint representation and learning of crop disease features; Liu et al. proposed a novel multimodal named entity recognition hybrid algorithm DGHC, which effectively improves the accuracy and robustness of named entity recognition by combining bimodal information from images and text; Chen et al. used a unified machine reading comprehension MRC framework and multi-query technology to extract multimodal entities, achieving effective recognition and fusion alignment of cross-modal information.

[0005] In the application of multimodal knowledge graphs, Yang Shuo et al. designed a ComplEx-based method to learn multimodal knowledge graph embeddings, enhancing the question-answering model for grape knowledge and significantly improving the question-answering effect; Xu Duo et al. introduced agricultural spatiotemporal multimodal knowledge graphs to systematize and abstract agricultural knowledge, and used graph structures to achieve functions such as precise fertilization decisions; GAO et al. proposed an image recognition technology that integrates Transformer technology and knowledge graph methods, effectively improving the accuracy of pest and disease feature recognition and increasing the processing speed and recognition efficiency of cotton pest and disease detection; Zhu et al. proposed a method based on knowledge graphs and deep learning, using an image recognition system mounted on a Raspberry Pi, to effectively detect and identify pests and diseases in fruits such as lychee and longan.

[0006] However, current applications of knowledge graphs are mainly focused on the semantic processing and association organization of textual knowledge. The fusion and application of multimodal knowledge such as images and videos are still in the exploratory stage. Moreover, most existing research revolves around multimodal knowledge extraction and fusion techniques, lacking a complete theoretical and technical roadmap from the ontology layer downwards. These shortcomings make it difficult for current knowledge graphs to achieve unified modeling and comprehensive utilization of multimodal knowledge.

[0007] Therefore, there is an urgent need to propose a knowledge graph-based method for constructing multimodal data knowledge graphs that extracts, fuses, and models multimodal data such as text and images, so as to realize the standardized representation, systematic management, and visualization application of domain knowledge, thereby fully leveraging the potential of knowledge graphs in multimodal data organization and intelligent applications. Summary of the Invention

[0008] To address the aforementioned problems in the prior art, the present invention is achieved through the following technical solution: A first aspect of the present invention provides a method for constructing a multimodal domain knowledge graph, comprising the following steps: Step 1: Extract entity information that conforms to the triple structure from the text data to populate triple text instances; Step 2: Use the ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model to identify the visual concept categories of the image data, and fill in the triplet image examples that conform to the triplet structure according to the visual concept categories; Step 3: Store the text instances and image instances of the triples to form a multimodal domain knowledge graph.

[0009] Text data includes semi-structured text data and unstructured text data.

[0010] Step 1 extracts entity information that conforms to the triple pattern from the text data to populate triple text instances, and also includes: For semi-structured text data, entity information that conforms to the triple structure is directly extracted to obtain triple text instances. For unstructured text data, the UIE information extraction model is used to extract entity information that conforms to the triple structure, resulting in triple text instances.

[0011] The ternary structure includes: subject, relation, and object.

[0012] A triplet text instance includes: the triplet text instance body (text entity), the triplet text instance relation (object property), and the triplet text instance object (text entity); a triplet image instance includes: the triplet image instance body (text entity), the triplet image instance relation (image instance), and the triplet image instance object (image).

[0013] The visual entity recognition model based on ResNet50-CA-RoBERTa multimodal fusion is used to identify visual concept categories in image data, including the following steps: (1) By decomposing the concrete visual features corresponding to the main body (text entity) of the triplet text instance, the defined visual concept is obtained, and the corresponding image description text is added to the image data based on the defined visual concept; (2) Visual features of the image data are extracted based on the ResNet50-CA model to obtain the visual features of the image data; (3) Extract text features from the image description text based on the RoBERTa model to obtain the semantic features of the image description text; (4) Based on the Transformer Encoder structure, the visual features of the image data and the semantic features of the image description text are fused into multimodal features to obtain the fused multimodal features; (5) Visual concept categories are obtained based on the fused multimodal features.

[0014] Step 3 stores the triplet text instance and triplet image instance in Neo4j.

[0015] A second aspect of the present invention provides a multimodal domain knowledge graph construction system, comprising: The triplet text instance filling module extracts entity information that conforms to the triplet structure from text data and fills in triplet text instances. Triple Image Example Filling Module: The ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model is used to identify the visual concept categories of image data, and triple image examples that conform to the triple structure are filled according to the visual concept categories. Knowledge graph construction module: Stores triple text instances and triple image instances to form a multimodal domain knowledge graph.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention enhances the visual feature expression by adding descriptive text corresponding to the image, thereby improving the accuracy of complex image recognition. It also consolidates the diversity of images from multiple sources by using the similarity of descriptive text, correcting the differences in visual expression caused by various factors. At the same time, it supplements image details with textual descriptions to improve the model's ability to recognize similar symptoms and to uncover deeper, finer-grained features under image-text fusion.

[0017] 2. This invention constructs a ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model. By embedding a coordinate attention mechanism (CA) in ResNet50 to extract image spatial features and introducing RoBERTa text features to enhance visual expression, it achieves accurate recognition of complex images and similar pictures. It effectively improves the visual feature capture capability and the accuracy of multimodal information fusion, providing high-quality image data support for knowledge graph filling.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multimodal domain knowledge graph construction method provided by an embodiment of the present invention; Figure 2 This is a flowchart of visual concept category recognition for wheat disease and pest image data provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model provided in this embodiment of the invention; Figure 4 This is a diagram of a multimodal domain knowledge graph construction system provided by an embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the solution according to the present invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0021] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0023] like Figure 1 As shown, this is a method for constructing a multimodal domain knowledge graph provided in Embodiment 1 of the present invention. The method includes the following steps: Step 1: Extract entity information that conforms to the triple structure from the wheat disease and pest text data to populate triple text instances.

[0024] The wheat disease and pest text data includes semi-structured and unstructured text data.

[0025] The triple structure includes: subject, relation, and object; a triple text instance includes: triple text instance subject (text entity), triple text instance relation (object attribute), and triple text instance object (text entity); among them, the triple text instance subject (text entity) serves as the semantic starting point of the triple, clearly defining the core text entity in the description of wheat disease and pest text data; the triple text instance relation (object attribute) serves as the semantic bridge between the subject and object, clarifying the relationship type between the subject and object, while ensuring that the triple text instance conforms to the domain knowledge logic; the triple text instance object (text entity) serves as supplementary information to the subject, enriching the core text content in the description of wheat disease and pest text data.

[0026] The semi-structured text data comes from authoritative websites, including: CNKI, the Crop Diseases, Pests and Weeds Image and Text Database, and the China Crop Germplasm Information Network. For semi-structured text data, entity information that conforms to the triple structure is directly extracted to obtain triple text instances.

[0027] Unstructured text data consists of news, articles, and patents, originating from popular science websites and patent websites. For unstructured text data, the UIE information extraction model is used to extract entity information that conforms to the triple structure, resulting in triple text instances.

[0028] The UIE information extraction model's input layer is based on a prompt mechanism, concatenating unstructured text data and a Structured Pattern Instruction (SSI) to form a complete input sequence. The model's encoding layer uses ERNIE 3.0 as the encoder. The lower layer of the encoder is a general representation module, using Transformer-XL as the core network architecture. It leverages a memory loop mechanism to process longer text sequences, thereby strengthening the learning of contextual relationships and capturing basic features of natural language. The upper layer of the encoder is a task-specific representation module, also based on Transformer-XL, capable of learning higher-level semantic features according to different task requirements to adapt to and encode relevant features for specific tasks. The decoding layer receives the hidden representation generated by the encoder as input and combines it with the sequence information generated in the previous decoding step to generate a SEL expression. The SEL expression enters the output layer, transforming it into specific triple text instances, completing the information extraction from the unstructured text data.

[0029] Step 2: Utilize the ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model to identify visual concept categories in wheat disease and pest image data. Based on the visual concept category, fill in image examples with triplet structures. Specifically: like Figure 2 The diagram shows the flowchart for visual concept category recognition of wheat disease and pest image data. The visual entity recognition model using ResNet50-CA-RoBERTa multimodal fusion identifies the visual concept categories of wheat disease and pest image data, including the following steps: (1) By decomposing the concrete visual features corresponding to the main body (text entity) of the triplet text instance, the defined visual concept is obtained, and the corresponding image description text is added to the image data based on the defined visual concept.

[0030] Among them, the concrete visual features corresponding to the main body (text entity) of the triplet text instance refer to the specific attribute features that are directly matched with the semantics of the text entity and can be directly observed visually, including shape, color, position, and structure.

[0031] The defined visual concept is a visual feature description constructed based on the triple text instance subject (text entity). The defined visual concept is obtained by decomposing the concrete visual features corresponding to the triple text instance subject (text entity). Based on the defined visual concept, the text semantics in the triple text instance subject (text entity) are transformed into visual feature description text of image data that can be directly observed, that is, image description text.

[0032] (2) such as Figure 3The diagram shows the structure of the ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model. Visual features are extracted from image data based on the ResNet50-CA model to obtain the visual features of the image data.

[0033] The image input format in the model's input layer is (C, H, W), where C is the number of channels, H is the height, and W is the width. The input is uniformly 3 channels. Image; image feature extraction involves the first convolutional layer (Convlayer), a max pooling layer, and a residual block + CA processing layer (…). The model consists of five modules: an average pooling layer, an average pooling layer, a flatten layer, and finally, an output layer that outputs the visual features (Img feature) of the image data.

[0034] In the first convolutional layer, after the input image undergoes a convolution transformation with a kernel size of 7×7, the feature map size becomes (64, 112, 112). After batch normalization, the ReLU activation function is used for nonlinear mapping, as shown in formula (1): (1) Among them, the input raw image data First feature map , represents the final output of the convolutional layer; R represents the real number field; This is an activation function used to perform non-linear mapping on the batch-normalized feature data; For batch normalization; This is a convolution operation.

[0035] The pooling kernel is set in the maxpool layer. The downsampling process is performed, and the maximum value of the pooling window is selected as the output to obtain a feature map with size (64, 56, 56), as shown in formula (2): (2) Among them, the second feature map , representing the final output of the max pooling layer; This is a max pooling operation.

[0036] This is the core of the model. This module consists of multiple bottleneck structures, each of which includes three convolutional layers and one skip connection. Equation (3) represents a bottleneck structure: (3) in, This represents the feature map output after the bottleneck structure is processed by taking the feature map Fx as input. Fx represents the function representation of the bottleneck structure; Fx represents the feature input map of the bottleneck structure.

[0037] After passing through 16 bottleneck structures, the feature map size is increased and downsampled to (B, 2048, 7, 7), and then the coordinate attention weights are adjusted by the CA module, as shown in formula (4): (4) F3 refers to the deep feature map of the image obtained after processing 16 bottleneck structures stacked together and adjusting the attention weights using the CA module. ; ~ This represents the 1st to 16th bottleneck structures; This is the function representation of the coordinate attention mechanism module.

[0038] The CA module reweights the feature map by adding coordinate information through the calculation of attention weights in the horizontal and vertical directions. Before calculating the attention weights, the feature map needs to be decomposed and the positional information in different directions needs to be encoded. The CA module uses average pooling kernels of size (H,1) and (1,W) to encode each channel along the horizontal and vertical directions, as shown in formulas (5) and (6): (5) (6) in, This represents the original pixel value of the c-th channel, h-th row, and i-th column of the feature map; This represents the original pixel value of the c-th channel, j-th row, and w-th column of the feature map; Encode the feature map channel index c at the feature map height coordinate h; The feature map channel index c is encoded at the position of the feature map width coordinate w; i represents the horizontal coordinate variable, used to traverse all positions in the horizontal direction of the feature map, with a value range of 0 to W-1; j represents the vertical coordinate variable, used to traverse all positions in the vertical direction of the feature map, with a value range of 0 to H-1; H is the height of the feature map, and W is the width of the feature map.

[0039] After completing the position encoding, the CA module generates attention weights based on the encoded coordinate information. Specifically, it concatenates the feature maps from two directions along the spatial dimension, then performs a 1×1 convolution operation to generate an intermediate feature map f, and finally splits the intermediate feature map into independent tensors in the horizontal direction according to the dimension. and vertical independent tensors After the convolution transformation, a horizontal attention weight tensor with the same number of channels as the input is generated. and vertical attention weight tensor As shown in formulas (7), (8) and (9): (7) (8) (9) in, For activation function, This represents a 1×1 convolution operation. This represents the vertical position encoding feature map. This represents the horizontal position encoding feature map.

[0040] After expanding the obtained weights, the horizontal attention weights are then applied. and vertical attention weight Compared with the original feature map of the input Multiply to generate attention-weighted feature maps As shown in formula (10): (10) Image data after The processing completes the acquisition, representation, and learning of attention-weighted image features. After obtaining the deep feature map F3 of the image, the model downsamples it to (2048, 1, 1) through an average pooling layer, and then flattens the multidimensional tensor into a one-dimensional vector (2048) through a Flatten layer, outputting the visual feature (Imgfeature) of the image data, as shown in formula (11): (11) in, For visual features of image data, For flattening operation function, This is the average pooling operation function.

[0041] ResNet50 effectively addresses network degradation through its residual block structure. By stacking more convolutional layers, it extracts complex features from images, enabling automatic feature extraction and effective representation. Building upon ResNet50, a Coordinate Attention (CA) mechanism is added. This mechanism combines the advantages of spatial and channel attention, allowing for convenient embedding into different locations within the convolutional network without disrupting its overall architecture. To enhance the CA module's effectiveness while reducing costs, this invention adds a CA module before the average pooling layer, strengthening the model's ability to capture complex spatial features.

[0042] (3) Based on the RoBERTa model, text features are extracted from the image description text to obtain the semantic features of the image description text.

[0043] RoBERTa is based on the Transformer architecture, and its core structure mainly includes an output representation layer and an encoder layer (Transformer layer). In the input representation layer, the image description text is divided into tokens. When the RoBERTa model processes the description text, it adds an extra [CLS] token at the beginning of the segmented text token sequence. This token does not correspond to any actual text content and serves as an aggregation carrier for the global semantics of the text.

[0044] Each token needs to go through three embedding layers: token embedding, segment embedding, and positional embedding. These layers embed semantic information, segment information, and positional information respectively, obtaining character vectors, sentence vectors, and position vectors corresponding to the information. The three vectors are then added together to generate the token embedding vector. , which serves as the input to the Transformer encoder.

[0045] The RoBERTa encoder layer consists of multiple identical Transformer encoders stacked together. This embodiment preferably uses the Achinese-roberta-wwm-ext model from the RoBERT model, which comprises 12 identical Transformer encoders. Each Transformer encoder consists of a multi-head attention mechanism layer, residual connections and layer normalization modules, and a feedforward neural network. Since the self-attention mechanism is insensitive to the order of the input sequence, the Transformer encoder requires additional positional encoding to add positional information. This positional encoding is then combined with the input embedding vector. After being stacked, the input is fed into the multi-head attention mechanism layer. For each independent attention head, a query vector for the i-th attention head is generated through a linear transformation. The key vector of the i-th attention head and the value vector of the i-th attention head The sub-attention output of the i-th attention head is calculated using the formula shown in (12): (12) in, =d / h, where d is the model dimension and h is the number of heads. It is the dimension of a single attention head. Let T be the activation function, and let T denote the matrix transpose.

[0046] The computation results of each attention head are concatenated, and the concatenated vector is linearly transformed by a fully connected layer before being input into the residual connection and layer normalization. The residual connection and layer normalization modules are located after the multi-head attention mechanism layer and the feedforward neural network. The residual connection learns the difference between the input and the output by adding the input to the processed output, and the layer normalization accelerates the network convergence by normalizing the data distribution. After that, the vector obtained after the above processing is input into the feedforward neural network, and the features are mapped to a high-dimensional space through linear transformation to enhance the representation learning of complex features. At the same time, an activation function is added between the two linear transformations to enhance the ability to capture nonlinear relationships, as shown in formula (13): (13) in, This represents the output of the feedforward neural network; is the activation function used to perform nonlinear mapping on the batch-normalized feature data; X represents the input vector of the feedforward neural network; This is the weight matrix of the first linear layer in the feedforward neural network; This is the bias vector of the first linear layer in the feedforward neural network; This is the weight matrix of the second linear layer in the feedforward neural network; This is the bias vector of the second linear layer in the feedforward neural network.

[0047] After being processed by 12 Transformer encoders stacked together, the model extracts vectors labeled with [CLS] from the last hidden layer. After linear transformation and activation by the tanh function, the output is a sentence-level semantic representation (Pooler Output) containing the semantic information of the entire sentence. As the semantic features of the image describing the text, they participate in subsequent feature fusion, as shown in formula (14): (14) This represents the activation function, used to map the result of a linear transformation to the interval [-1, 1]. The feature vector representing the [CLS] tag; This is the weight matrix of the third linear layer in the feedforward neural network; This is the bias vector of the third linear layer in the feedforward neural network.

[0048] (4) Based on the Transformer Encoder structure, the visual features of the image data and the semantic features of the image description text are fused into multimodal features to obtain the fused multimodal features.

[0049] While possessing powerful feature extraction capabilities, the Transformer Encoder can effectively fuse information from different modalities. Its internal self-attention mechanism can calculate the relationship between features from different modalities and achieve accurate classification after multimodal feature fusion by allocating attention weights.

[0050] Visual features in acquiring image data Semantic features of images describing text Then, the features of the two modalities are concatenated along the same spatial dimension and input into the Transformer encoder for processing to obtain the fused multimodal feature, as shown in formula (15): (15) in, This represents the fused multimodal features. Indicates encoder processing operation, This indicates a feature concatenation operation, where dim represents the dimension.

[0051] (5) Visual concept categories are obtained based on the fused multimodal features.

[0052] To achieve accurate classification based on fused features, a fully connected classifier (FC layer) is created to classify the fused multimodal features. The input linear layer captures complex features in high dimensionality, and a ReLU activation function is added to capture non-linear features. Finally, a fully connected layer is connected to map the feature dimension to the number of categories. The SoftMax function is used to obtain the expected probability distribution of the image-description text joint data under different categories, and the argmax function outputs the category with the highest probability as the visual concept category. As shown in formula (16): (16) in, Indicates categories of visual concepts; This represents the fused multimodal features; This represents the maximum index value function, used to select the index corresponding to the category with the highest probability value; This represents the activation function, used to normalize the class scores output by the fully connected classifier; This indicates the connection to the classifier.

[0053] Since visual concepts are visual feature descriptions constructed based on triplet text instance subjects (text entities), by decomposing their corresponding concrete visual features, the textual semantics in the triplet text instance subjects (text entities) are transformed into visual feature description texts that can be directly observed. Therefore, the textual semantics of visual concept categories and triplet text instance subjects (text entities) are completely equivalent. That is, the corresponding triplet image instance subject (text entity) can be determined through the visual concept category. At the same time, based on the visual concept category, the image of the corresponding visual concept category can be determined, that is, the triplet image instance object (image). The triplet image instance subject (text entity) is associated with the triplet image instance object (image) through the triplet image instance relation (image instance), thereby realizing the filling of triplet image instance subject (text entity), triplet image instance relation (image instance), and triplet image instance object (image).

[0054] Step 3: Store the triplet text instances and triplet image instances in Neo4j to form a multimodal domain knowledge graph. Simultaneously, store the image data in a MySQL database, linking the two via URLs. Specifically, during storage, preprocessing and formatting the triplet text instances and triplet image instances are performed using pandas in Python. Then, py2neo in Python performs batch writing to Neo4j, and pandas is used to visualize the modal domain knowledge graph.

[0055] By storing image data in a MySQL database, the performance of the knowledge graph is avoided by preventing image data from consuming Neo4j resources and affecting its operation. At the same time, MySQL's mature management capabilities for structured data facilitate the maintenance of image metadata. The URL serves as the core link, which not only achieves seamless integration between the multimodal knowledge graph and the image data, ensuring that the corresponding image example can be quickly located and called when querying text knowledge, but also reduces the coupling between the two databases, allowing the iteration of the multimodal knowledge graph and the maintenance of image data to be carried out independently. Ultimately, a multimodal domain knowledge storage system that balances knowledge association depth, query efficiency, maintenance flexibility, and storage security is formed.

[0056] like Figure 4 As shown, this is a multimodal domain knowledge graph construction system provided in Embodiment 2 of the present invention, comprising: The triplet text instance filling module extracts entity information that conforms to the triplet structure from text data and fills in triplet text instances.

[0057] Triple Image Example Filling Module: This module uses a ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model to identify the visual concept categories of image data, and fills in triple image examples that conform to the triple structure based on the visual concept categories.

[0058] Knowledge graph construction module: Stores triple text instances and triple image instances to form a multimodal domain knowledge graph.

[0059] This invention discloses a method and system for constructing a multimodal domain knowledge graph. For semi-structured text data, entity information conforming to a triplet structure is directly extracted. For unstructured text data, a UIE information extraction model is used to obtain triplet text instances. For image data, a ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model is used to add descriptive text to images based on defined visual concepts. ResNet50-CA extracts visual features, and RoBERTa extracts semantic features from the descriptive text. After feature fusion via a Transformer Encoder, a fully connected classifier and SoftMax output the visual concept category, which is then used to fill in triplet image instances. The two types of triples are stored in Neo4j to form a multimodal domain knowledge graph. Image data is stored in a MySQL database and linked via URLs.

[0060] This invention utilizes multimodal fusion of textual and visual features to correct differences in visual image representation and supplement image details, effectively improving the recognition accuracy of complex and similar images and providing high-quality image data support for knowledge graphs. In the storage architecture, Neo4j leverages its efficient query capabilities, while MySQL enables structured management of image metadata. URL links reduce the coupling between the two, preventing image data from consuming Neo4j resources and impacting performance, while ensuring independent maintenance and seamless integration of multimodal data. From feature extraction and triple construction of textual and image data to the storage and association of multimodal knowledge, this invention comprehensively solves the problems of scattered and heterogeneous multimodal data and the lack of a unified modeling mechanism, meeting the needs of systematic management and visualization applications of domain knowledge.

[0061] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal domain knowledge graph, characterized in that, Includes the following steps: Step 1: Extract entity information that conforms to the triple structure from the text data to populate triple text instances; Step 2: Use the ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model to identify the visual concept categories of the image data, and fill in the triplet image examples that conform to the triplet structure according to the visual concept categories; Step 3: Store the text instances and image instances of the triples to form a multimodal domain knowledge graph.

2. The method for constructing a multimodal domain knowledge graph according to claim 1, characterized in that, The text data includes semi-structured text data and unstructured text data.

3. The method for constructing a multimodal domain knowledge graph according to claim 2, characterized in that, Step 1, which extracts entity information that conforms to the triple pattern from the text data to populate triple text instances, also includes: For semi-structured text data, entity information that conforms to the triple structure is directly extracted to obtain triple text instances. For unstructured text data, the UIE information extraction model is used to extract entity information that conforms to the triple structure, resulting in triple text instances.

4. The method for constructing a multimodal domain knowledge graph according to claim 1, characterized in that, The triplet structure includes: subject, relation, and object.

5. The method for constructing a multimodal domain knowledge graph according to claim 1, characterized in that, The triple text instance includes: triple text instance body (text entity), triple text instance relation (object attribute), and triple text instance object (text entity); the triple image instance includes: triple image instance body (text entity), triple image instance relation (image instance), and triple image instance object (image).

6. The method for constructing a multimodal domain knowledge graph according to claim 5, characterized in that, The visual entity recognition model based on ResNet50-CA-RoBERTa multimodal fusion is used to identify visual concept categories in image data, including the following steps: (1) By decomposing the concrete visual features corresponding to the main body (text entity) of the triplet text instance, the defined visual concept is obtained, and the corresponding image description text is added to the image data based on the defined visual concept; (2) Visual features of the image data are extracted based on the ResNet50-CA model to obtain the visual features of the image data; (3) Extract text features from the image description text based on the RoBERTa model to obtain the semantic features of the image description text; (4) Based on the Transformer Encoder structure, the visual features of the image data and the semantic features of the image description text are fused into multimodal features to obtain the fused multimodal features; (5) Visual concept categories are obtained based on the fused multimodal features.

7. The method for constructing a multimodal domain knowledge graph according to claim 1, characterized in that, Step 3 involves storing the triplet text instance and the triplet image instance in Neo4j.

8. A method system for constructing a multimodal domain knowledge graph, characterized in that, The method for constructing a multimodal domain knowledge graph according to any one of claims 1 to 7 includes: The triplet text instance filling module extracts entity information that conforms to the triplet structure from text data and fills in triplet text instances. Triple Image Example Filling Module: The ResNet50-CA-RoBERTa multimodal fusion visual entity recognition model is used to identify the visual concept categories of image data, and triple image examples that conform to the triple structure are filled according to the visual concept categories. Knowledge graph construction module: Stores triple text instances and triple image instances to form a multimodal domain knowledge graph.

Images