Multi-modal knowledge graph construction method and device

Abstract

Embodiments of the invention provide a multi-modal knowledge graph construction method and apparatus. The method comprises the steps of obtaining an image and a text having an association relationship; performing feature extraction on the image and the text to obtain image features and text features; inputting the image features and the text features into a pre-constructed knowledge graph model, and performing multi-modal fusion processing on the image features and the text features by using the knowledge graph model based on a prompt vector obtained by training to obtain multi-modal fusion features; and based on the multi-modal fusion features, identifying a plurality of entities and association relationships among the entities, and determining a triple of the multi-modal knowledge graph. According to the method, the construction process of the multi-modal knowledge graph is optimized through a method of combining continuous learning, multi-modal fusion and prompt learning, the method is suitable for a dynamically changing knowledge graph scene, and the performance of a knowledge graph construction task can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method and device for constructing a multimodal knowledge graph. Background Art

[0002] A multimodal knowledge graph is a graph that integrates multiple sources of data such as text and images, and is used to enrich information expression and knowledge mining. This type of graph has important applications in scenarios such as intelligent question answering, recommendation systems, information retrieval, and intelligent assistants. The current method for constructing a multimodal knowledge graph has poor multimodal information fusion effects, and is prone to knowledge forgetting during the dynamic knowledge update process, which affects the generalization ability of the model. It also has high computational overhead and low model training efficiency. Summary of the invention

[0003] In view of this, the purpose of the embodiments of the present application is to propose a method and device for constructing a multimodal knowledge graph to solve the problem of constructing a multimodal knowledge graph.

[0004] Based on the above purpose, the present application embodiment provides a method for constructing a multimodal knowledge graph, including:

[0005] Get images and texts with associated relationships;

[0006] Extracting features from the image and text to obtain image features and text features;

[0007] Inputting the image features and text features into a pre-built knowledge graph model, and using the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the hint vector obtained through training, to obtain multimodal fusion features;

[0008] Based on the multimodal fusion features, multiple entities and the relationships between entities are identified, and triples of the multimodal knowledge graph are determined.

[0009] Optionally, feature extraction is performed on the image and the file to obtain image features and text features, including:

[0010] Extracting features from the image using a preset multimodal model to obtain global features, mesoscopic features, and local features of the image;

[0011] The text is vectorized using a preset language representation model to obtain text features.

[0012] Optionally, the prompt vector includes a first prompt vector and a second prompt vector; and the prompt vector obtained by training is used to perform multimodal fusion processing on the image feature and the text feature to obtain a multimodal fusion feature, including:

[0013] The first prompt vector is concatenated with the global image feature, the mesoscopic image feature and the local image feature respectively to obtain the global image prompt feature, the mesoscopic image prompt feature and the local image prompt feature;

[0014] concatenating the second prompt vector with the text feature to obtain a text prompt feature;

[0015] Using a preset multi-layer perceptron to perform dimension alignment processing on the image global prompt features, image meso prompt features, image local prompt features, and text prompt features, to obtain dimensionally aligned image global prompt features, image meso prompt features, image local prompt features, and text prompt features;

[0016] The dimensionally aligned image global prompt features are concatenated with the text prompt features to obtain the image global text features, the dimensionally aligned image mesoscopic prompt features are concatenated with the text prompt features to obtain the image mesoscopic text features, and the dimensionally aligned image local prompt features are concatenated with the text prompt features to obtain the image local text features;

[0017] The preset multi-head attention module is used to perform feature fusion processing on the global text features, the meso-text features and the local text features of the image, respectively, to obtain the fused global text features, the meso-text features and the local text features of the image;

[0018] The fused image local text features, image meso-text features and image local text features are added together to obtain a multimodal fusion feature.

[0019] Optionally, the method further includes:

[0020] The prompt vector is adjusted, and the knowledge graph model is lightly fine-tuned based on the adjusted prompt vector to obtain an updated knowledge graph model.

[0021] Optionally, adjusting the prompt vector includes:

[0022] A first hint vector and a second hint vector are randomly generated and obey a normal distribution, and the first hint vector and the second hint vector have the same length.

[0023] Optionally, the method further includes:

[0024] Using a preset influence function, the influence values ​​of all samples are calculated, and multiple samples with high influence values ​​are selected as replay samples; wherein the samples are constructed based on image samples and text samples with associated relationships;

[0025] The knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model.

[0026] Optionally, the influence function is:

[0027]

[0028] Among them, Influence i is the influence value of the i-th sample, s i is the gradient vector of the ith sample, g i is the gradient of the replay sample loss function of the i-th sample to all the parameters of the model that need to be updated, v i is the predicted probability of the i-th sample, H -1 is the inverse of the Hessian matrix of all model parameters that need to be updated.

[0029] Optionally, the training of the knowledge graph model based on all samples and the replay samples to obtain an updated knowledge graph model includes:

[0030] Based on all samples and replay samples, the knowledge graph model is optimized using the gradient descent method to update the parameters of the knowledge graph model. The updating method is:

[0031]

[0032] Where η is the learning rate, is the gradient of the total loss function of the knowledge graph model to the set of model parameters, θ new is the set of updated model parameters, θ old is a set of model parameters before updating, wherein the set of model parameters includes all parameters of the model that need to be updated, and the total loss function includes a conditional random field loss function for identifying entities, a relationship extraction loss function for extracting associations between entities, and a replay sample loss function.

[0033] Optionally, based on the multimodal fusion features, multiple entities and the relationship between entities are identified to determine the triples of the multimodal knowledge graph, including:

[0034] Performing a linear transformation on the multimodal fusion features to obtain a score matrix for entity recognition; calculating the path score and the total score of the entity label sequence based on the score matrix using a conditional random field layer, and determining the recognized entity according to the calculation results;

[0035] The multimodal fusion features are linearly transformed to obtain the scores of the multimodal fusion features mapped to each relationship category; the scores are converted into probability distributions through normalization processing to obtain the predicted probability of each relationship category.

[0036] The present application also provides a multimodal knowledge graph construction device, including:

[0037] An acquisition module, used for acquiring images and texts with associated relationships;

[0038] An extraction module, used for extracting features from the image and text to obtain image features and text features;

[0039] A fusion module, used to input the image features and text features into a pre-built knowledge graph model, and use the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the hint vector obtained through training to obtain multimodal fusion features;

[0040] The recognition module is used to identify multiple entities and the relationship between entities based on the multimodal fusion features, and determine the triples of the multimodal knowledge graph.

[0041] From the above, it can be seen that the multimodal knowledge graph construction method and device provided in the embodiment of the present application obtains images and texts with associated relationships, extracts features from the images and texts, obtains image features and text features, inputs the image features and text features into a pre-constructed knowledge graph model, and uses the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the prompt vector obtained through training to obtain multimodal fusion features. Based on the multimodal fusion features, multiple entities and the associations between entities are identified, and the triples of the multimodal knowledge graph are determined. The present application optimizes the construction process of the multimodal knowledge graph through a method that combines continuous learning, multimodal fusion and prompt learning. It is suitable for dynamically changing knowledge graph scenarios and can effectively improve the performance of knowledge graph construction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a method flow of an embodiment of the present application;

[0044] Figure 2 This is a block diagram of the device structure of an embodiment of the present application;

[0045] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0047] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0048] In related technologies, when traditional models are used to construct multimodal knowledge graphs, due to the lack of continuous learning ability for constantly updated knowledge, new knowledge is learned but old knowledge is forgotten, that is, catastrophic forgetting is prone to occur, which affects the generalization ability of the model; the fusion efficiency of multimodal information is not high, and the model is trained by full fine-tuning, which has high computational overhead and low training efficiency.

[0049] In view of this, this application adopts a continuous learning approach to solve the problem of knowledge forgetting, adopts multimodal information fusion based on prompt learning to improve the efficiency of information fusion, and updates the model in a lightweight fine-tuning manner, which can greatly reduce computing overhead and improve training efficiency. That is, through a method that combines continuous learning, multimodal fusion and prompt learning, the construction process of multimodal knowledge graphs is optimized, which is particularly suitable for dynamically changing knowledge graph scenarios and can effectively improve the performance of knowledge graph construction tasks.

[0050] The technical solution of the present application is further described in detail below through specific embodiments.

[0051] like Figure 1 As shown, the embodiment of the present application provides a method for constructing a multimodal knowledge graph, including:

[0052] S101: Acquire images and texts with associated relationships;

[0053] In this embodiment, to construct a knowledge graph, images and texts with associated relationships are obtained, and a pre-trained knowledge graph model is used to process the images and texts to obtain triples related to the images and texts, and a multi-modal knowledge graph is constructed based on the triples. Among them, the images and texts with associated relationships are, for example, product pictures and product text descriptions, news pictures and text report contents, recipe pictures and recipe text introductions, etc., and specific application scenarios are not limited.

[0054] S102: Extract features from the images and texts to obtain image features and text features;

[0055] In this embodiment, after obtaining the images and texts, it is necessary to preprocess the images and texts first. Among them, the preprocessing of the text includes: tokenizing the text, removing stop words such as "de" (的) and "le" (了) and special characters, and then standardizing the text form through conversion to obtain standardized text. For example, it is uniformly converted to lowercase. The preprocessing of the image includes: adjusting the pixels of the image to a unified size and performing pixel value normalization processing to facilitate mapping the pixel space to the feature space subsequently.

[0056] In some ways, to enhance the robustness and adaptability of the knowledge graph model, during the training stage of the knowledge graph model, based on the existing image samples, methods such as random cropping (retaining a certain proportion of the original Figure 1 area), horizontal flipping according to a certain flipping probability, adjusting brightness, contrast, saturation, etc. are used to generate new image samples, and then the generated image samples are processed into corresponding image tensor representations.

[0057] In some embodiments, based on the preprocessed images and texts, features are extracted to obtain image features and text features. The methods include:

[0058] Using a preset multi-modal model to extract features from the image to obtain global image features, mesoscopic image features, and local image features;

[0059] Using a preset language representation model to perform vector conversion processing on the text to obtain text features.

[0060] In this embodiment, for the preprocessed image, a multi-modal model is used to extract features, and the global feature, mesoscopic structural feature, and local detail feature of the image are respectively extracted to obtain global image features, mesoscopic image features, and local image features.

[0061] Among them, the multimodal model adopts the CLIP (Contrastive Language-Image Pre-Training) model. In order to be suitable for the multi-level feature extraction processing of the CLIP model, the preprocessed image is adjusted to the main image, auxiliary image and RCNN image of different scales. The CLIP model is used to extract features from multi-scale images, which can simultaneously focus on the global semantic information, mesoscopic structural features and local detail features of the image, thereby improving the comprehensiveness and robustness of feature extraction.

[0062] Specifically: the size of the preprocessed image is adjusted to a uniform fixed size, for example, first uniformly adjusted to 256×256, then centered cropped to obtain an image of size 224×224, and finally pixel value normalization is performed to obtain the main image, realizing the standard image transformation sequence. The main image effectively retains the main visual information of the image and is suitable for extracting global features. The main image is input into the visual encoder of the CLIP model, and the visual encoder encodes the main image to generate an image sequence feature matrix, and the feature vector corresponding to the first element in the image sequence feature matrix is ​​extracted as the global feature of the image.

[0063] The size of the preprocessed image is scaled to a medium size (which can be adjusted by setting parameters, such as 128×128) to obtain an auxiliary image. The medium-sized auxiliary image helps capture the mesoscopic structural features in the image. The auxiliary image is input into the visual encoder of the CLIP model, which encodes the auxiliary image and generates a corresponding image sequence feature matrix. The feature vector corresponding to the first element of the image sequence feature matrix is ​​extracted as the image mesoscopic feature.

[0064] The size of the preprocessed image is scaled to a small size (which can be adjusted by setting parameters, such as 64×64) to obtain an RCNN image. The small-sized RCNN image can better focus on the local detail features of the image. The RCNN image is input into the visual encoder of the CLIP model. The visual encoder encodes the RCNN image and generates the corresponding image sequence feature matrix. The feature vector corresponding to the first element of the image sequence feature matrix is ​​extracted as the local feature of the image.

[0065] For the preprocessed text, the BERT model is used to convert the text vector. Specifically, the preprocessed text is converted into a word unit ID matrix, a type ID matrix, and an attention mask matrix using a word segmenter. The three matrices represent the word unit information, word unit type information, and attention information of the text respectively; the three matrices are input into the BERT model, and the BERT model is used to encode the text information to generate a text sequence feature matrix, in which each word unit position corresponds to a feature vector, and the feature vector of the [CLS] position is extracted from the text sequence feature matrix as the text feature.

[0066] S103: inputting the image features and the text features into a pre-trained knowledge graph model, and using the knowledge graph model to perform multimodal fusion processing on the image features and the text features based on the hint vector obtained through training, to obtain multimodal fusion features;

[0067] In this embodiment, after extracting image features and text features from images and texts, the image features and text features are input into the knowledge graph model, and the knowledge graph model is used to perform multimodal fusion processing of image features and text features based on the hint vector obtained through training to obtain multimodal fusion features.

[0068] The prompt vector includes a first prompt vector from vision to text and a second prompt vector from text to vision, and the method for obtaining a multimodal fusion feature through multimodal fusion processing includes:

[0069] The first prompt vector is concatenated with the global feature of the image, the mesoscopic feature of the image and the local feature of the image respectively to obtain the global prompt feature of the image, the mesoscopic prompt feature of the image and the local prompt feature of the image;

[0070] The second prompt vector is concatenated with the text feature to obtain the text prompt feature;

[0071] Using a preset multi-layer perceptron to perform dimension alignment processing on the image global prompt features, image mesoscopic prompt features, image local prompt features, and text prompt features, to obtain dimensionally aligned image global prompt features, image mesoscopic prompt features, image local prompt features, and text prompt features;

[0072] The dimensionally aligned image global prompt features are concatenated with the text prompt features to obtain the image global text features, the dimensionally aligned image mesoscopic prompt features are concatenated with the text prompt features to obtain the image mesoscopic text features, and the dimensionally aligned image local prompt features are concatenated with the text prompt features to obtain the image local text features;

[0073] The preset multi-head attention module is used to perform feature fusion processing on the global text features, the meso-text features and the local text features of the image, respectively, to obtain the fused global text features, the meso-text features and the local text features of the image;

[0074] The fused image local text features, image meso-text features and image local text features are added together to obtain multimodal fusion features.

[0075] This embodiment provides a method for realizing multimodal feature fusion based on prompt learning. The model parameters of the trained knowledge graph model include a first prompt vector and a second prompt vector. The first prompt vector is spliced ​​with the global features of the image, the mesoscopic features of the image, and the local features of the image, and the second prompt vector is spliced ​​with the text features. Since the dimensions of the global features of the image, the mesoscopic features of the image, the local features of the image, and the text features are different, it is necessary to use a multi-layer perceptron to align the dimensions of the spliced ​​global prompt features, the mesoscopic prompt features of the image, the local prompt features of the image, and the text prompt features. After dimensional alignment, the global text features of the image, the mesoscopic text features of the image, and the local text features of the image are fused by a multi-head attention mechanism, and then the local text features of the image, the mesoscopic text features of the image, and the local text features of the image are added after the feature fusion to obtain multimodal fusion features. Through the learnable prompt vector and multi-level feature fusion, the effective alignment and integration of text features and image features can be achieved, the prompt vector is used to act as a bridge between different modalities, and the multi-head attention mechanism is used to capture the complex associations between features, which can improve the depth and effect of cross-modal fusion.

[0076] In some implementations, the method further includes: adjusting the prompt vector, and performing lightweight fine-tuning on the knowledge graph model based on the adjusted prompt vector to obtain an updated knowledge graph model. That is, the trained knowledge graph model needs to be updated regularly or when predetermined conditions are met. The prompt vector is used as a model parameter, and the prompt vector can be dynamically adjusted to perform lightweight fine-tuning on the knowledge graph model based on the adjusted prompt vector to achieve model update. In this way, only the prompt vector needs to be updated while other parameters of the model remain unchanged, which can effectively solve the imbalance problem of multimodal information in the fusion process. Compared with full quantitative fine-tuning, it can greatly reduce computational overhead and improve model training efficiency.

[0077] In some embodiments, the method for adjusting the hint vector includes: randomly generating a first hint vector and a second hint vector that obey a normal distribution, and the first hint vector and the second hint vector have the same length. It can be expressed as:

[0078]

[0079] Among them, P V2t is the first hint vector, P t2v is the second prompt vector, N(0,σ 2 ) means the mean is 0 and the variance is σ 2 is a normal distribution, σ is the standard deviation of the setting.

[0080] In some embodiments, the first hint vector P v2t Respectively with the image global feature V m_cls , image mesoscopic features V a_cls, local image features V r_cls The global image feature V′ is obtained by splicing m_cls , the image visual hint feature V′ a_cls And the local image feature V′ r_cls , expressed as:

[0081] V′ m_cls =[V m_cls ;P v2t ]

[0082] V′ a_cls =[V a_cls ;P v2t ]

[0083] V′ r_cls =[V r_cls ;P v2t ] (2)

[0084] The second hint vector P t2v With the text feature T cls Concatenate them to get the text prompt feature T′ cls , expressed as:

[0085] T′ cls =[T cls ;P t2v ] (3)

[0086] In some methods, the spliced ​​global image prompt features, the image mesoscopic prompt features, and the image local prompt features, and the spliced ​​text prompt features are respectively input into a preset multi-layer perceptron, and the multi-layer perceptron outputs dimensionally aligned image global prompt features, image mesoscopic prompt features, image local prompt features, and text prompt features. The multi-layer perceptron can be expressed as:

[0087] MLP(x)=W2·ReLU(W1·x+b1)+b2 (4)

[0088] Among them, W1 and W2 are weight matrices, b1 and b2 are biases, and x is the input of the multi-layer perceptron, which are the global prompt features of the spliced ​​image, the mesoscopic prompt features of the image, the local prompt features of the image, and the text prompt features. The output of the multi-layer perceptron is the dimensionally aligned global prompt feature V″ m_cls , the image visual hint feature V″ a_cls , local image hint feature V″ r_cls and text prompt feature T″ cls .

[0089] In some embodiments, the dimensionally aligned image global hint feature V″ m_cls With text prompt feature T″ clsConcatenate them to get the global text feature F of the image combined_main , align the dimensions of the image mesopic cue feature V″ a_cls With text prompt feature T″ cls Concatenate them to get the image meso-text feature F combined_aux , align the dimensions of the image local cue feature V″ r_cls With text prompt feature T″ cls Concatenate them to get the local text feature F of the image combined_rcnn , expressed as:

[0090] F combined_main =[T″ cls ; V″ m_cls ]

[0091] F combined_aux =[T″ cls ; V″ a_cls ]

[0092] F combined_rcnn =[T″ cls ; V″ r_cls ] (5)

[0093] The image global text feature F combined_main , image-level text features F combined_aux , image local text feature F combined_rcnn Input the multi-head attention module, use the multi-head attention module to fuse the input features, and obtain the local text features of the image after feature fusion F final_main , image-level text features F final_aux and image local text feature F final_rcnn , and then the local text feature F final_main , image-level text features F final_aux and image local text feature F final_rcnn Add together to get the final fused multimodal fusion feature F final , expressed as:

[0094] F final =F final_main +F final_aux +F final_rcnn (6)

[0095] In some ways, the multi-head attention module converts the input features into a query vector Q, a key vector K, and a value vector V, expressed as:

[0096] Q=F combined W Q

[0097] K=F combined W K

[0098] V=F combined W V (7)

[0099] Among them, F combined is the input feature, which can be the global text feature F of the image combined_main , image-level text features F combined_aux , image local text feature F combined_rcnn One of the Q , W K , W V are weight matrices, which are optimized by back propagation during model training.

[0100] After that, the dot product of the query vector and the key vector is calculated to get the attention score, which is normalized by the softmax function to get the attention weight Attention_wgights, and the value vector is weighted and summed using the attention weight to get the weighted output, which is expressed as:

[0101]

[0102] Attention_output=Attention_weights×V (9)

[0103] Among them, d k is the dimension of the key vector.

[0104] In the feature summation stage, the weighted output of the attention is added to the features of the original input multi-head attention module to form a residual connection, which is expressed as:

[0105] Residual Output=Attention output+F(Input Feature)(10)

[0106] After the residual connection, the layer normalization process is performed to help the gradient propagation in the training process to be more stable. After the layer normalization process, the final output of the multi-head attention module is obtained, that is, the local text feature F of the image after feature fusion. final_main , image-level text features F fin a l_aux Or image local text feature F final_rcn n.

[0107] S104: Based on the multimodal fusion features, identify multiple entities and the relationships between entities, and determine the triples of the multimodal knowledge graph.

[0108] In this embodiment, after processing the multimodal fusion features, the knowledge graph model identifies entities based on the multimodal fusion features and extracts relationships between entities. According to the identified entities and the relationships between entities, entity-relationship-entity triples are generated, and a multimodal knowledge graph is constructed based on the triples.

[0109] Specifically, entity recognition is performed based on multimodal fusion features, including: performing linear transformation on the multimodal fusion features to obtain a score matrix for entity recognition, the score matrix includes the emission score of each entity label, and the emission score is used to represent the possibility of each entity category at each sequence position; the path score and the total score of the entity label sequence are calculated through a conditional random field (CRF) layer to measure the possibility of different entity label combinations, and the entity with the highest possibility is the recognized entity; wherein the path score is a score based on the real entity label sequence, representing the cumulative score of the model along the correct entity label sequence, and the total score is the sum of the scores of all possible entity label sequences, which is used to measure the overall score distribution of the entity label space.

[0110] Among them, the entity label sequence is a sequence of entity category labels corresponding to each word. For example, the entity label sequence of the sentence "Zhang San works in Beijing." may be [B-PER, O, B-LOC, O, O], which means that "Zhang San" is a person's name, and the corresponding label is B-PER, "Beijing" is a place name, and the corresponding label is B-LOC, and other words are non-entities, and the corresponding label is O. The length of the entity label sequence is 5, that is, it includes 5 labels. The score matrix refers to the score matrix corresponding to different entity labels for each word. Each row of the score matrix corresponds to a word, and each column corresponds to an entity label. The path score refers to the cumulative score of the real entity label sequence. Assuming that the real entity label sequence is [B-PER, O, B-LOC, O, O], the path score is the sum of the emission scores and transfer scores of these entity labels. The total score refers to the sum of the scores of all possible entity label sequences, which is the sum of the scores of all possible entity label sequences (such as [B-PER, O, B-LOC, O, O], [B-PER, B-LOC, O, O, O], etc.).

[0111] In some methods, a multi-layer perceptron is used to map the multimodal fusion features into a score matrix, which is expressed as:

[0112] e t [y t ]=MLP(F final ) t [y t ] (12)

[0113] Among them, y tis the tth label in the entity label sequence Y.

[0114] The loss function of the conditional random field layer is defined as the negative log likelihood, which means that the model can more accurately identify entities by maximizing the ratio of the true path score to the total score. The loss function is expressed as:

[0115]

[0116] Among them, Y is the real entity label sequence, θ is the parameter of the conditional random field layer, P(Y|F final ; θ) represents a given feature F final The conditional probability of entity tag sequence Y when Y′ is the set of all possible entity tag sequences, and entity tag sequence Y is one of the set Y′, y′ t is the tth label in the set of all possible entity label sequences Y′, y′ t+1 is the t+1th label in the set of all possible entity label sequences Y′, A is the transfer matrix, A[y t ,y t+1 ] indicates label y t Move to label y t+1 where T is the length of the entity tag sequence Y. The entity that can be identified by the conditional random field layer contains information such as type, location, and attribute value.

[0117] Entity relationship extraction is performed based on multimodal fusion features, including: performing linear transformation on the multimodal fusion features to obtain the scores of the multimodal fusion features mapped to each relationship category; converting the scores into probability distributions through normalization to represent the predicted probability of each relationship category.

[0118] In some methods, the loss function of relation extraction is defined as a cross entropy loss function, which is used to measure the difference between the relation category predicted by the model and the true relation category. This loss function takes the logarithm of the predicted probability of the true category and finds the negative value to reflect the error of the model in the relation classification task. By minimizing the cross entropy loss, the model can learn the distinguishing features of the relation category more accurately. Among them, the cross entropy loss function can be expressed as:

[0119]

[0120] Among them, N is the number of samples of the relation extraction task, M is the number of relation categories, and y ij is the actual label. If the jth class of the i-th sample is a true relationship, then y ij =1, otherwise y ij =0; The probability that the i-th sample belongs to the j-th class is predicted by the model.

[0121] In some embodiments, in order to solve the catastrophic forgetting problem of the knowledge graph, the multimodal knowledge graph construction method further includes:

[0122] Using a preset influence function, the influence values ​​of all samples are calculated, and multiple samples with high influence values ​​are selected as replay samples; wherein the samples are constructed based on image samples and text samples with associated relationships;

[0123] The knowledge graph model is trained based on all samples and replay samples to obtain an updated knowledge graph model.

[0124] In this embodiment, in order to solve the problem of catastrophic forgetting in continuous learning, a sample replay strategy based on influence function is adopted. For all samples of the training knowledge graph model, the influence value of each sample is calculated using the influence function, and all samples are sorted in order from high to low influence values. Multiple samples with high influence values ​​are selected from the sorted samples as replay samples, for example, a predetermined number of samples in the front are selected as replay samples; all samples and replay samples are used to update the knowledge graph model. The updated knowledge graph model can not only maintain the ability to learn new knowledge, but also effectively maintain the memory of historical knowledge, and can improve the accuracy and completeness of entity and relationship extraction, thereby ensuring the accuracy and completeness of the knowledge graph and improving the performance of the knowledge graph model.

[0125] In some implementations, the loss function of the replay sample is:

[0126]

[0127] Where R is the number of replay samples, Y i is the entity label sequence of the i-th replay sample, F final is the multimodal fusion feature of the i-th replay sample, T i is the length of the entity label sequence corresponding to the i-th replay sample, is the emission score vector of the t-th label in the entity label sequence corresponding to the i-th replay sample, The i-th replay sample corresponds to the t-th label in the entity label sequence, For label Transfer to label The score, Y i ′ is the set of all possible entity tag sequences, y t ′ is the tth label in the set of all possible entity label sequences. is the emission fraction term, indicating that the i-th replay sample corresponds to the t-th label in the entity label sequence score.

[0128] In some embodiments, the sample generation method for training the knowledge graph model includes: obtaining a number of image samples and text samples with an associated relationship, extracting features from the image samples and text samples to obtain image sample features and text sample features, and constructing training samples based on the image sample features, text sample features and the associated relationship between the two. The training samples can be expressed as:

[0129] sample={'text':T cls ,'image global ':V m_vls ,'image middle ':V a_cls ,'image local ':V r_cls ,'label':h}(16)

[0130] Among them, h is a binary label used to indicate the semantic matching of the text image pair. When h takes the value of 1, it means that the image and text semantics match each other. When h takes the value of 0, it means that the image and text semantics do not match each other.

[0131] In some implementations, an influence function based on random matrix approximation is constructed, and the conjugate gradient descent method is used to efficiently identify and replay key historical samples, which can effectively alleviate the forgetting problem of the model during the dynamic knowledge update process. The influence function is expressed as:

[0132]

[0133] Among them, Influence i is the influence value of the i-th sample; s i is the gradient vector of the i-th sample; g i is the gradient of the loss function of the ith sample (the loss function shown in Formula 15) to all parameters of the model that need to be updated; v i is the predicted probability vector of the i-th sample, specifically the predicted probability of the true label; H -1 It is the inverse of the Hessian matrix of all the parameters that need to be updated in the model, and is used to measure the second-order derivative information of all the parameters that need to be updated in the model with respect to the loss function.

[0134] In some implementations, after selecting important replay samples, the knowledge graph model is optimized based on all samples and replay samples using a gradient descent method to update the parameters of the model. The updating method is:

[0135]

[0136] Where η is the learning rate, is the gradient of the total loss function of the model to the set of model parameters, θ newis the set of updated model parameters, θ old It is the set of model parameters before updating. The set of model parameters of the knowledge graph model includes the weights and biases of the multi-layer perceptron, the weights of the multi-head attention module, the transfer matrix of the conditional random field, the prompt vector and other model parameters that need to be updated.

[0137] The total loss function includes three parts: conditional random field loss function, relation extraction loss function and replay sample loss function, which can be expressed as:

[0138] L total =L CRF +L RE +L replay (19)

[0139] The gradient of the total loss function with respect to the set of model parameters Including the gradient of the conditional random field loss function Gradient of the relation extraction loss function and the gradient of the loss function of the replayed sample It is expressed as:

[0140]

[0141] Where α is the weight of the replay loss, which is used to balance the importance of the current loss and the replay loss.

[0142] The multimodal knowledge graph construction method provided in the embodiment of the present application adopts a prompt learning strategy. In the training and updating of the knowledge graph model, the prompt vector is dynamically adjusted to interact with multimodal information, which can effectively solve the imbalance problem of multimodal information in the fusion process; when fine-tuning the model, only a small number of prompt vectors need to be updated instead of all the parameters of the model, which greatly reduces the computational overhead and improves resource utilization efficiency. A sample replay strategy based on the influence function is adopted. By calculating the influence value and selecting the most representative historical samples for replay, the model can effectively maintain the memory of historical knowledge while maintaining its ability to learn new knowledge. During the model training process, the loss of the current task is combined with the loss of the replay sample for optimization to achieve balanced learning of new and old knowledge. The knowledge graph model after training, updating and optimization can efficiently and accurately identify the relationship between entities and entities, determine the triples including entities and relationships, and thus construct a complete and accurate knowledge graph.

[0143] The technical effect of the method of the present application is explained below in combination with experimental data.

[0144] As shown in Table 1, the method of the present application is compared with the baseline model (MKGformer) and the model without a sample replay mechanism in terms of the accuracy of recognizing human entities (PER), location entities (LOC), organization entities (ORG), other types of entities (MISC), and the mean (AVG) of each entity type. For the baseline model, the recognition accuracy of entities other than human entities is not high, indicating that the model has a serious problem of knowledge forgetting. For the model without a sample replay mechanism, the recognition accuracy of organization entities and other types of entities is low, while the method of the present application has good recognition performance for various entities, especially the recognition performance of organization entities and other types of entities is significantly improved, and the performance of location entities is also improved, indicating that the use of a replay mechanism helps the model better retain and utilize previously learned knowledge.

[0145] Table 1 Comparison of entity recognition accuracy

[0146]

[0147] As shown in Table 2, in terms of resource usage, after the application adopts the prompt learning mechanism, the memory usage is greatly reduced due to the reduction of gradient storage, the reduction of optimizer states, and the simplification of the calculation graph.

[0148] Table 2 Comparison of resource usage

[0149]

[0150] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0151] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] like Figure 2 As shown, the embodiment of the present application provides a multimodal knowledge graph construction device, including:

[0153] An acquisition module, used for acquiring images and texts with associated relationships;

[0154] An extraction module is used to extract features from images and texts to obtain image features and text features;

[0155] A fusion module is used to input image features and text features into a pre-built knowledge graph model, and use the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the hint vector obtained through training to obtain multimodal fusion features;

[0156] The recognition module is used to identify multiple entities and the relationships between entities based on multimodal fusion features, and determine the triples of the multimodal knowledge graph.

[0157] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0158] The apparatus of the above-mentioned embodiment is used to implement the corresponding method in the above-mentioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0159] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0160] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0161] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0162] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0163] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0164] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0165] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0166] The electronic device of the above embodiment is used to implement the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0167] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0168] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0169] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0170] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0171] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present disclosure.

Claims

1. A method for constructing a multimodal knowledge graph, characterized in that: include: Get images and texts with associated relationships; Extracting features from the image and text to obtain image features and text features; Inputting the image features and text features into a pre-built knowledge graph model, and using the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the hint vector obtained through training, to obtain multimodal fusion features; Based on the multimodal fusion features, multiple entities and the relationships between entities are identified, and triples of the multimodal knowledge graph are determined.

2. The method according to claim 1, characterized in that Perform feature extraction on the image and the file to obtain image features and text features, including: Extracting features from the image using a preset multimodal model to obtain global features, mesoscopic features, and local features of the image; The text is vectorized using a preset language representation model to obtain text features.

3. The method according to claim 2, characterized in that The prompt vector includes a first prompt vector and a second prompt vector; the prompt vector obtained based on the training performs multimodal fusion processing on the image feature and the text feature to obtain a multimodal fusion feature, including: The first prompt vector is concatenated with the global image feature, the mesoscopic image feature and the local image feature respectively to obtain the global image prompt feature, the mesoscopic image prompt feature and the local image prompt feature; concatenating the second prompt vector with the text feature to obtain a text prompt feature; Using a preset multi-layer perceptron to perform dimension alignment processing on the image global prompt features, image meso prompt features, image local prompt features, and text prompt features, to obtain dimensionally aligned image global prompt features, image meso prompt features, image local prompt features, and text prompt features; The dimensionally aligned image global prompt features are concatenated with the text prompt features to obtain the image global text features, the dimensionally aligned image mesoscopic prompt features are concatenated with the text prompt features to obtain the image mesoscopic text features, and the dimensionally aligned image local prompt features are concatenated with the text prompt features to obtain the image local text features; The preset multi-head attention module is used to perform feature fusion processing on the global text features, the meso-text features and the local text features of the image, respectively, to obtain the fused global text features, the meso-text features and the local text features of the image; The fused image local text features, image meso-text features and image local text features are added together to obtain a multimodal fusion feature.

4. The method according to claim 1, characterized in that: Also includes: The prompt vector is adjusted, and the knowledge graph model is lightly fine-tuned based on the adjusted prompt vector to obtain an updated knowledge graph model.

5. The method according to claim 4, characterized in that Adjusting the prompt vector comprises: A first hint vector and a second hint vector are randomly generated and obey a normal distribution, and the first hint vector and the second hint vector have the same length.

6. The method according to claim 1 or 4, characterized in that: Also includes: Using a preset influence function, the influence values ​​of all samples are calculated, and multiple samples with high influence values ​​are selected as replay samples; wherein the samples are constructed based on image samples and text samples with associated relationships; The knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model.

7. The method according to claim 6, characterized in that The influence function is: Among them, Influence i is the influence value of the i-th sample, s i is the gradient vector of the ith sample, g i is the gradient of the replay sample loss function of the i-th sample to all the parameters of the model that need to be updated, v i is the predicted probability of the i-th sample, H -1 is the inverse of the Hessian matrix of all model parameters that need to be updated.

8. The method according to claim 7, characterized in that The step of training the knowledge graph model based on all samples and the replay samples to obtain an updated knowledge graph model includes: Based on all samples and replay samples, the knowledge graph model is optimized using the gradient descent method to update the parameters of the knowledge graph model. The updating method is: Where η is the learning rate, is the gradient of the total loss function of the knowledge graph model to the set of model parameters, θ new is the set of updated model parameters, θ old is a set of model parameters before updating, wherein the set of model parameters includes all parameters of the model that need to be updated, and the total loss function includes a conditional random field loss function for identifying entities, a relationship extraction loss function for extracting associations between entities, and a replay sample loss function.

9. The method according to claim 1, characterized in that: Based on the multimodal fusion features, multiple entities and the relationships between entities are identified, and triples of the multimodal knowledge graph are determined, including: Performing a linear transformation on the multimodal fusion features to obtain a score matrix for entity recognition; calculating the path score and the total score of the entity label sequence based on the score matrix using a conditional random field layer, and determining the recognized entity according to the calculation results; The multimodal fusion features are linearly transformed to obtain the scores of the multimodal fusion features mapped to each relationship category; the scores are converted into probability distributions through normalization processing to obtain the predicted probability of each relationship category.

10. A multimodal knowledge graph construction device, characterized in that: include: An acquisition module, used for acquiring images and texts with associated relationships; An extraction module, used for extracting features from the image and text to obtain image features and text features; A fusion module, used to input the image features and text features into a pre-built knowledge graph model, and use the knowledge graph model to perform multimodal fusion processing on the image features and text features based on the hint vector obtained through training to obtain multimodal fusion features; The recognition module is used to identify multiple entities and the relationship between entities based on the multimodal fusion features, and determine the triples of the multimodal knowledge graph.

Images