A multi-modal knowledge graph construction method and device

Abstract

Embodiments of the present application provide a multi-modal knowledge graph construction method and device, comprising: acquiring images and texts having a correlation relationship; performing feature extraction on the images and texts to obtain image features and text features; inputting the image features and text features into a pre-constructed knowledge graph model, and performing multi-modal fusion processing on the image features and text features based on a prompt vector obtained through training by using the knowledge graph model to obtain multi-modal fusion features; and identifying a plurality of entities and a correlation relationship between the entities based on the multi-modal fusion features to determine triplets of a multi-modal knowledge graph. The method combining continuous learning, multi-modal fusion and prompt learning optimizes the construction process of the multi-modal knowledge graph, is suitable for a dynamically changing knowledge graph scenario, and can effectively improve the performance of a knowledge graph construction task.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of artificial intelligence, and particularly relates to a multi-modal knowledge graph construction method and device. BACKGROUND

[0002] The multi-modal knowledge graph is a graph that fuses multi-source data such as text and image, and is used for enriching information expression and knowledge mining. Such a graph has important applications in intelligent question answering, recommendation system, information retrieval, intelligent assistant and the like. The current multi-modal knowledge graph construction method has poor multi-modal information fusion effect, and is prone to knowledge forgetting in the dynamic knowledge updating process, which affects the generalization ability of the model, and has large computational overhead and low training efficiency of the model. SUMMARY

[0003] Therefore, the embodiment of the present application aims to provide a multi-modal knowledge graph construction method and device to solve the problem of multi-modal knowledge graph construction.

[0004] To achieve the above object, the embodiment of the present application provides a multi-modal knowledge graph construction method, which comprises the following steps:

[0005] acquiring an image and a text having an association relationship;

[0006] performing feature extraction on the image and the text to obtain image features and text features;

[0007] inputting the image features and the text features into a pre-constructed knowledge graph model, performing multi-modal fusion processing on the image features and the text features based on a prompt vector obtained through training by using the knowledge graph model, and obtaining multi-modal fusion features;

[0008] identifying a plurality of entities and the association relationship between the entities based on the multi-modal fusion features, and determining the triplets of the multi-modal knowledge graph.

[0009] Optionally, the feature extraction on the image and the text to obtain the image features and the text features comprises:

[0010] performing feature extraction on the image by using a pre-set multi-modal model to obtain image global features, image middle features and image local features;

[0011] performing vector conversion processing on the text by using a pre-set language representation model to obtain text features.

[0012] Optionally, the prompt vector comprises a first prompt vector and a second prompt vector; and the multi-modal fusion processing on the image features and the text features based on the prompt vector obtained through training to obtain the multi-modal fusion features comprises:

[0013] The first prompt vector is spliced with the image global feature, the image mid-level feature and the image local feature respectively to obtain an image global prompt feature, an image mid-level prompt feature and an image local prompt feature.

[0014] The second prompt vector is spliced with the text feature to obtain a text prompt feature.

[0015] The image global prompt feature, the image mid-level prompt feature and the image local prompt feature and the text prompt feature are subjected to dimension alignment processing by using a preset multi-layer perception to obtain dimension-aligned image global prompt feature, image mid-level prompt feature, image local prompt feature and text prompt feature.

[0016] The dimension-aligned image global prompt feature is spliced with the text prompt feature to obtain an image global text feature, the dimension-aligned image mid-level prompt feature is spliced with the text prompt feature to obtain an image mid-level text feature, and the dimension-aligned image local prompt feature is spliced with the text prompt feature to obtain an image local text feature.

[0017] The image global text feature, the image mid-level text feature and the image local text feature are subjected to feature fusion processing by using a preset multi-head attention module to obtain fused image global text feature, image mid-level text feature and image local text feature.

[0018] The fused image local text feature, image mid-level text feature and image local text feature are added to obtain a multi-modal fusion feature.

[0019] Optionally, the method further comprises:

[0020] The prompt vector is adjusted, and the knowledge graph model is fine-tuned based on the adjusted prompt vector to obtain an updated knowledge graph model.

[0021] Optionally, adjusting the prompt vector comprises:

[0022] A first prompt vector and a second prompt vector conforming to a normal distribution are randomly generated, and the first prompt vector and the second prompt vector have the same length.

[0023] Optionally, the method further comprises:

[0024] A preset influence function is used to calculate the influence values of all samples, and a plurality of samples with high influence values are selected as replay samples from the samples; wherein the samples are constructed based on image samples and text samples having a correlation relationship.

[0025] The knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model.

[0026] Optionally, the influence function is:

[0027]

[0028] wherein Influence i is the influence value of the i-th sample, s i is the gradient vector of the i-th sample, g i is the gradient of the replay sample loss function on all parameters of the model that need to be updated, v i is the prediction probability of the i-th sample, H -1 is the inverse of the Hessian matrix of all parameters of the model that need to be updated.

[0029] Optionally, the knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model, comprising:

[0030] The knowledge graph model is optimized based on all samples and the replay samples by using a gradient descent method to update the parameters of the knowledge graph model, and the updating method is:

[0031]

[0032] wherein η is a learning rate, is the gradient of the total loss function of the knowledge graph model on the set of model parameters, θ new is the set of updated model parameters, θ old is the set of model parameters before updating, and the set of model parameters includes all parameters of the model that need to be updated, and the total loss function includes a conditional random field loss function for identifying entities, a relationship extraction loss function for extracting the correlation between entities, and a replay sample loss function.

[0033] Optionally, based on the multi-modal fusion feature, a plurality of entities and the correlation between the entities are identified, and a triple of the multi-modal knowledge graph is determined, comprising:

[0034] The multi-modal fusion feature is linearly transformed to obtain a score matrix for entity identification; a conditional random field layer is used to calculate path scores and total scores of an entity label sequence based on the score matrix, and the identified entity is determined according to the calculation result.

[0035] The multi-modal fusion feature is linearly transformed to obtain a score of the multi-modal fusion feature mapping to each relationship category; the score is converted into a probability distribution through normalization processing to obtain a prediction probability of each relationship category.

[0036] Embodiments of the present application also provide a multi-modal knowledge graph construction device, comprising:

[0037] an acquisition module configured to acquire images and texts having a correlation relationship;

[0038] an extraction module configured to perform feature extraction on the images and texts to obtain image features and text features;

[0039] a fusion module configured to input the image features and the text features into a pre-constructed knowledge graph model, and perform multi-modal fusion processing on the image features and the text features based on a prompt vector obtained through training by using the knowledge graph model, to obtain multi-modal fusion features;

[0040] a recognition module configured to recognize a plurality of entities and a correlation relationship between the entities based on the multi-modal fusion features, and determine triplets of a multi-modal knowledge graph.

[0041] As can be seen from the above, the multi-modal knowledge graph construction method and device provided by the embodiments of the present application can acquire images and texts having a correlation relationship, perform feature extraction on the images and texts to obtain image features and text features, input the image features and the text features into a pre-constructed knowledge graph model, perform multi-modal fusion processing on the image features and the text features based on a prompt vector obtained through training by using the knowledge graph model, to obtain multi-modal fusion features, recognize a plurality of entities and a correlation relationship between the entities based on the multi-modal fusion features, and determine triplets of a multi-modal knowledge graph. The present application optimizes the construction process of the multi-modal knowledge graph by combining the methods of continuous learning, multi-modal fusion and prompt learning, is suitable for a dynamically changing knowledge graph scenario, and can effectively improve the performance of the knowledge graph construction task. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 a method flowchart of the embodiments of the present application;

[0044] Figure 2 a device structure block diagram of the embodiments of the present application;

[0045] Figure 3 an electronic device structure block diagram of the embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the following will be further described in detail with specific embodiments and with reference to the drawings.

[0047] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall have the common meaning understood by one of ordinary skill in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in the embodiments of the present application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0048] In the related art, when a multi-modal knowledge graph is constructed using a traditional model, due to the lack of continuous learning ability, there is a situation of learning new knowledge but forgetting old knowledge, i.e., easy to appear catastrophic forgetting, which affects the generalization ability of the model; the fusion efficiency of multi-modal information is not high, and the model is trained in a full-amount fine-tuning manner, which has large computational overhead and low training efficiency.

[0049] Therefore, the present application adopts a continuous learning method to solve the problem of knowledge forgetting, adopts multi-modal information fusion based on prompt learning to improve the information fusion efficiency, and adopts a lightweight fine-tuning method to update the model, which can greatly reduce the computational overhead and improve the training efficiency. That is, by combining the methods of continuous learning, multi-modal fusion, and prompt learning, the construction process of the multi-modal knowledge graph is optimized, which is particularly suitable for dynamic knowledge graph scenarios and can effectively improve the performance of the knowledge graph construction task.

[0050] The technical solutions of the present application will be further described in detail below through specific embodiments.

[0051] As shown in Figure 1 The embodiments of the present application provide a multi-modal knowledge graph construction method, which comprises:

[0052] S101: acquiring an image and text having an association relationship;

[0053] In this embodiment, to construct a knowledge graph, obtain images and texts with a correlation relationship, process the images and texts by using a pre-trained knowledge graph model, obtain triples related to the images and texts, and construct a multi-modal knowledge graph based on the triples. The images and texts with the correlation relationship are, for example, product pictures and product textual descriptions, news pictures and textual report contents, recipe pictures and recipe textual introductions, and the specific application scenarios are not limited.

[0054] S102: Perform feature extraction on the images and texts to obtain image features and text features.

[0055] In this embodiment, after obtaining the images and texts, the images and texts need to be preprocessed. The preprocessing of the texts includes: performing word segmentation on the texts, removing stop words such as “de” and “le” and special characters, and then standardizing the text form by conversion to obtain standardized texts, for example, converting to lowercase. The preprocessing of the images includes: adjusting the pixels of the images to a uniform size, performing pixel value normalization processing, and facilitating subsequent mapping of the pixel space to the feature space.

[0056] In some ways, to enhance the robustness and adaptability of the knowledge graph model, during the training of the knowledge graph model, on the basis of existing image samples, new image samples can be generated by random cropping (retaining a fixed proportion of the original image), horizontal flipping according to a certain flipping probability, adjusting brightness, contrast, and saturation, and the generated image samples are processed into corresponding image tensor representations. Figure 1

[0057] In some embodiments, based on the preprocessed images and texts, image features and text features are obtained by performing feature extraction, and the method includes:

[0058] The image is processed by using a preset multi-modal model to extract features, to obtain image global features, image medium features, and image local features.

[0059] The text is processed by using a preset language representation model to convert vectors, to obtain text features.

[0060] In this embodiment, for the preprocessed images, the multi-modal model is used to extract features, to extract image global features, medium structure features, and local detail features, and obtain image global features, image medium features, and image local features.

[0061] ​The multi-modal model adopts a CLIP (Contrastive Language-Image Pre-Training) model. In order to adapt to multi-level feature extraction processing of the CLIP model, the preprocessed image is adjusted into a main image, an auxiliary image and an RCNN image of different sizes, the CLIP model is used to extract features of the images of different sizes, the global semantic information, the medium structural features and the local detailed features of the image can be simultaneously focused on, and the comprehensiveness and robustness of feature extraction are improved.

[0062] Specifically, the size of the preprocessed image is adjusted to a uniform fixed size, for example, first adjusted to 256x256, then centered and cropped to obtain an image of 224x224, and finally pixel value normalization is performed to obtain the main image, realizing a standard image transformation sequence. The main image effectively retains the main visual information of the image and is suitable for extracting global features. The main image is input into the visual encoder of the CLIP model, the visual encoder encodes the main image to generate an image sequence feature matrix, and the feature vector corresponding to the first element in the image sequence feature matrix is extracted as the global feature of the image.

[0063] The size of the preprocessed image is scaled to a medium size (which can be adjusted by setting parameters, for example, 128x128) to obtain an auxiliary image. The medium-sized auxiliary image is helpful to capture the medium structural features in the image. The auxiliary image is input into the visual encoder of the CLIP model, the visual encoder encodes the auxiliary image to generate a corresponding image sequence feature matrix, and the feature vector corresponding to the first element in the image sequence feature matrix is extracted as the medium feature of the image.

[0064] The size of the preprocessed image is scaled to a small size (which can be adjusted by setting parameters, for example, 64x64) to obtain an RCNN image. The small-sized RCNN image can better focus on the local detailed features of the image. The RCNN image is input into the visual encoder of the CLIP model, the visual encoder encodes the RCNN image to generate a corresponding image sequence feature matrix, and the feature vector corresponding to the first element in the image sequence feature matrix is extracted as the local feature of the image.

[0065] For the preprocessed text, the BERT model is used for text vector conversion. Specifically, the preprocessed text is converted into a token ID matrix, a type ID matrix and an attention mask matrix by using a tokenizer, the three matrices respectively represent the token information, the token type information and the attention information of the text; the three matrices are input into the BERT model, the BERT model is used to encode the text information to generate a text sequence feature matrix, wherein the position of each token corresponds to a feature vector, the feature vector of the [CLS] position in the text sequence feature matrix is extracted as the text feature.

[0066] S103: input the image features and the text features into the pre-trained knowledge graph model, and perform multi-modal fusion processing on the image features and the text features based on the prompt vectors obtained through training by using the knowledge graph model, to obtain multi-modal fusion features;

[0067] In this embodiment, after the image features and the text features are extracted from the image and the text, the image features and the text features are input into the knowledge graph model, and multi-modal fusion processing of the image modal and the text modal is performed on the image features and the text features based on the prompt vectors obtained through training by using the knowledge graph model, to obtain multi-modal fusion features.

[0068] The prompt vectors include a first prompt vector from vision to text and a second prompt vector from text to vision, and the method for obtaining the multi-modal fusion features through multi-modal fusion processing includes:

[0069] The first prompt vector is spliced with the image global feature, the image mid-level feature and the image local feature respectively, to obtain image global prompt features, image mid-level prompt features and image local prompt features;

[0070] The second prompt vector is spliced with the text features, to obtain text prompt features;

[0071] The image global prompt features, the image mid-level prompt features, the image local prompt features and the text prompt features are subjected to dimension alignment processing by using a preset multi-layer perception, to obtain dimension-aligned image global prompt features, image mid-level prompt features, image local prompt features and text prompt features;

[0072] The dimension-aligned image global prompt features are spliced with the text prompt features, to obtain image global text features, the dimension-aligned image mid-level prompt features are spliced with the text prompt features, to obtain image mid-level text features, and the dimension-aligned image local prompt features are spliced with the text prompt features, to obtain image local text features;

[0073] The image global text features, the image mid-level text features and the image local text features are subjected to feature fusion processing by using a preset multi-head attention module respectively, to obtain fused image global text features, image mid-level text features and image local text features;

[0074] The fused image local text features, the image mid-level text features and the image local text features are added, to obtain multi-modal fusion features.

[0075] The embodiment provides a method for realizing multi-modal feature fusion based on prompt learning. Model parameters of a trained knowledge graph model include a first prompt vector and a second prompt vector, the first prompt vector is spliced with image global features, image mid-level features and image local features, and the second prompt vector is spliced with text features. Since the dimensions of the image global features, the image mid-level features and the image local features are different from that of the text features, a multi-layer perception machine is used to perform dimension alignment processing on the spliced image global prompt features, the image mid-level prompt features, the image local prompt features and the text prompt features. After the dimension alignment, a multi-head attention mechanism is used to perform feature fusion processing on the image global text features, the image mid-level text features and the image local text features respectively. Then, the image local text features, the image mid-level text features and the image local text features after the feature fusion are added, and multi-modal fusion features are obtained. Through the learnable prompt vectors and the multi-level feature fusion, effective alignment and integration of the text features and the image features can be realized. The prompt vectors serve as a bridge between different modalities, and the multi-head attention mechanism captures complex correlations between features, so that the depth and effect of cross-modal fusion can be improved.

[0076] In some embodiments, the method further comprises adjusting the prompt vectors, and performing light-weight fine-tuning on the knowledge graph model based on the adjusted prompt vectors to obtain an updated knowledge graph model. That is, the trained knowledge graph model needs to be updated regularly or when a predetermined condition is met. As the prompt vectors are model parameters, the model can be updated by dynamically adjusting the prompt vectors, performing light-weight fine-tuning on the knowledge graph model based on the adjusted prompt vectors. In this way, only the prompt vectors need to be updated while other parameters of the model remain unchanged, which can effectively solve the imbalance problem of multi-modal information in the fusion process, greatly reduce the computational overhead compared to full-quantization fine-tuning, and improve the model training efficiency.

[0077] In some embodiments, the method of adjusting the prompt vectors comprises: randomly generating a first prompt vector and a second prompt vector subject to a normal distribution, and the first prompt vector and the second prompt vector have the same length. This can be expressed as:

[0078]

[0079] wherein P V2t is the first prompt vector, P t2v is the second prompt vector, N(0, σ 2 ) represents a normal distribution with a mean of 0 and a variance of σ 2 , and σ is a set standard deviation.

[0080] In some embodiments, the first prompt vector P v2t is spliced with image global features V m_cls , image mid-level features V a_cls, image local feature V r_cls , image global feature V′ m_cls , image mid-level feature V′ a_cls , and image local feature V′ r_cls , denoted as:

[0081] V′ m_cls = [V m_cls ; P v2t ]

[0082] V′ a_cls = [V a_cls ; P v2t ]

[0083] V′ r_cls = [V r_cls ; P v2t ] (2)

[0084] The second prompt vector P t2v is concatenated with the text feature T cls to obtain a text prompt feature T′ cls , denoted as:

[0085] T′ cls = [T cls ; P t2v ] (3)

[0086] In some ways, the concatenated image global prompt feature, image mid-level prompt feature, and image local prompt feature are respectively input into a preset multi-layer perception, and the multi-layer perception outputs dimension-aligned image global prompt feature, image mid-level prompt feature, image local prompt feature, and text prompt feature. Wherein, the multi-layer perception can be represented as:

[0087] MLP(x) = W2·ReLU(W1·x + b1) + b2 (4)

[0088] Wherein, W1, W2 are weight matrices, b1, b2 are biases, x is the input of the multi-layer perception, and respectively are the concatenated image global prompt feature, image mid-level prompt feature, image local prompt feature, and text prompt feature. The output of the multi-layer perception is dimension-aligned image global prompt feature V″ m_cls , image mid-level prompt feature V″ a_cls , image local prompt feature V″ r_cls , and text prompt feature T″ cls .

[0089] In some embodiments, the dimension-aligned image global prompt feature V″ m_cls and the text prompt feature T″ clsThe image global text feature F is obtained by splicing combined_main The image global text feature F is obtained by splicing a_cls The image global text feature F is obtained by splicing cls The image global text feature F is obtained by splicing combined_aux The image global text feature F is obtained by splicing r_cls The image global text feature F is obtained by splicing cls The image global text feature F is obtained by splicing combined_rcnn , and is expressed as:

[0090] F combined_main =[T″ cls ;V″ m_cls ]

[0091] F combined_aux =[T″ cls ;V″ a_cls ]

[0092] F combined_rcnn =[T″ cls ;V″ r_cls ] (5)

[0093] The image global text feature F is obtained by splicing combined_main The image global text feature F is obtained by splicing combined_aux The image global text feature F is obtained by splicing combined_rcnn The image global text feature F is obtained by splicing final_main The image global text feature F is obtained by splicing final_aux The image global text feature F is obtained by splicing final_rcnn The image global text feature F is obtained by splicing final_main The image global text feature F is obtained by splicing final_aux The image global text feature F is obtained by splicing final_rcnn The image global text feature F is obtained by splicing final , and is expressed as:

[0094] F final =F final_main +F final_aux +F final_rcnn (6)

[0095] In some ways, the multi-head attention module converts the input feature into a query vector Q, a key vector K and a value vector V, and is expressed as:

[0096] Q=F combined W Q

[0097] K=F combined W K

[0098] V = F combined W V (7)

[0099] wherein, F combined is an input feature, which can be one of image global text feature F combined_main , image mid-level text feature F combined_aux , or image local text feature F combined_rcnn ; W Q , W K , and W V are weight matrices, which are obtained by back propagation optimization during model training.

[0100] Then, the dot product of the query vector and the key vector is calculated to obtain the attention score, which is normalized by the softmax function to obtain the attention weight Attention_weights, and the value vector is weighted and summed using the attention weight to obtain the weighted output, denoted as:

[0101]

[0102] Attention_output = Attention_weights x V (9)

[0103] wherein, d k is the dimension of the key vector.

[0104] In the feature addition stage, the attention weighted output and the features of the original input multi-head attention module are added to form a residual connection, denoted as:

[0105] Residual Output = Attention output + F (Input Feature) (10)

[0106] After the residual connection, layer normalization processing is performed to help the gradient propagation in the training process more stable, and after the layer normalization processing, the final output of the multi-head attention module is obtained, i.e., the image local text feature F final_main , image mid-level text feature F fin a l_aux , or image local text feature F final_rcn n after feature fusion.

[0107] S104: Based on the multi-modal fusion feature, the multiple entities and the correlation between the entities are recognized, and the triples of the multi-modal knowledge graph are determined.

[0108] In this embodiment, after obtaining the multi-modal fusion feature, the knowledge graph model identifies entities and extracts relationships between entities based on the multi-modal fusion feature, generates a triple of entity-relation-entity according to the identified entities and the correlation between entities, and constructs a multi-modal knowledge graph based on the triple.

[0109] Specifically, the entity recognition based on the multi-modal fusion feature includes: performing linear transformation on the multi-modal fusion feature to obtain a score matrix for entity recognition, the score matrix including emission scores of each entity label, and the emission scores being used to represent the possibility of each entity category at each sequence position; and calculating path scores and total scores of an entity label sequence through a conditional random field (CRF) layer, so as to measure the possibility of different entity label combinations, and the entity with high possibility being the identified entity; wherein the path score is a score based on the real entity label sequence, representing the cumulative score of the model along the correct entity label sequence, and the total score is the sum of scores of all possible entity label sequences, and is used to measure the overall score distribution of the entity label space.

[0110] wherein the entity label sequence is a sequence composed of entity category labels corresponding to each word. For example, the entity label sequence of the sentence "Zhang San works in Beijing." can be [B-PER, O, B-LOC, O, O], indicating that "Zhang San" is a person name, the corresponding label is B-PER, "Beijing" is a place name, the corresponding label is B-LOC, and the other words are non-entities, the corresponding label is O, and the length of the entity label sequence is 5, that is, it includes 5 labels. The score matrix refers to the score matrix of different entity labels corresponding to each word, each row of the score matrix corresponds to a word, and each column corresponds to an entity label. The path score refers to the cumulative score of the real entity label sequence, assuming that the real entity label sequence is [B-PER, O, B-LOC, O, O], the path score is the sum of the emission scores and the transition scores of these entity labels. The total score refers to the sum of the scores of all possible entity label sequences, which is the sum of the scores of all possible entity label sequences (such as [B-PER, O, B-LOC, O, O], [B-PER, B-LOC, O, O, O], etc.).

[0111] In some manners, the multi-modal fusion feature is mapped to the score matrix by using a multi-layer perception, and is represented as:

[0112] e t [y t ]=MLP(F final ) t [y t ] (12)

[0113] wherein y tis the t-th label in the entity tag sequence Y.

[0114] The loss function of the conditional random field layer is defined as the negative log-likelihood, that is, by maximizing the ratio of the true path score to the total score, the model can more accurately identify entities. The loss function is represented as:

[0115]

[0116] where Y is the true entity tag sequence, θ is the parameter of the conditional random field layer, P(Y|F final ; θ) represents the conditional probability of the entity tag sequence Y given the feature F final , Y' is the set of all possible entity tag sequences, Y is one of the set Y', y t is the t-th label in the set Y', y t+1 is the t+1-th label in the set Y', A is the transition matrix, A[y t , y t+1 ] represents the score of the label y t transitioning to the label y t+1 , and T is the length of the entity tag sequence Y. The entity that can be identified by the conditional random field layer contains information such as type, location, and attribute value.

[0117] Based on the multi-modal fusion feature, the entity relationship extraction includes: performing linear transformation on the multi-modal fusion feature to obtain scores of the multi-modal fusion feature mapping to each relationship category; and converting the scores into probability distributions through normalization processing to represent the prediction probability of each relationship category.

[0118] In some ways, the loss function of relationship extraction is defined as a cross-entropy loss function, which is used to measure the difference between the predicted relationship category of the model and the true relationship category. The loss function takes the logarithm of the predicted probability of the true category and takes the negative value to reflect the error of the model in the relationship classification task. By minimizing the cross-entropy loss, the model can more accurately learn the distinguishing features of the relationship category. Wherein, the cross-entropy loss function can be represented as:

[0119]

[0120] where N is the number of samples of the relationship extraction task, M is the number of relationship categories, y ij is the actual label, if the j-th category of the i-th sample is the true relationship, then y ij = 1, otherwise y ij = 0; is the probability that the i-th sample belongs to the j-th category predicted by the model.

[0121] In some embodiments, to solve the catastrophic forgetting problem of the knowledge graph, the multi-modal knowledge graph construction method further comprises the following steps:

[0122] The influence value of all samples is calculated by using the preset influence function, and a plurality of samples with high influence value are selected as the replay samples; wherein the samples are constructed based on the image samples and the text samples having the correlation relationship;

[0123] The knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model.

[0124] In this embodiment, to solve the catastrophic forgetting problem in continuous learning, a sample replay strategy based on an influence function is adopted. For all samples for training the knowledge graph model, the influence value of each sample is calculated by using the influence function, all samples are sorted in descending order of the influence value, and a plurality of samples with high influence value are selected as replay samples from the sorted samples, for example, a predetermined number of samples at the front are selected as replay samples. The knowledge graph model is updated by using all samples and the replay samples, the updated knowledge graph model can not only maintain the learning ability of new knowledge, but also effectively maintain the memory of historical knowledge, and can improve the accuracy and completeness of entity and relationship extraction, thereby ensuring the accuracy and completeness of the knowledge graph and improving the performance of the knowledge graph model.

[0125] In some embodiments, the loss function of the replay sample is:

[0126]

[0127] wherein R is the number of replay samples, Y i is the entity label sequence of the i-th replay sample, F final is the multi-modal fusion feature of the i-th replay sample, T i is the length of the entity label sequence corresponding to the i-th replay sample, is the emission score vector of the i-th replay sample corresponding to the t-th label in the entity label sequence, is the i-th replay sample corresponding to the t-th label in the entity label sequence, is the label transition to the label , Y i ' is a set of all possible entity label sequences, y t ' is the t-th label of the set of all possible entity label sequences. is the emission score item, which represents the score of the i-th replay sample corresponding to the t-th label in the entity label sequence.

[0128] In some embodiments, the sample generation method for training the knowledge graph model comprises: obtaining a plurality of image samples and text samples having a correlation relationship, performing feature extraction on the image samples and the text samples to obtain image sample features and text sample features, and constructing training samples according to the image sample features, the text sample features, and the correlation relationship between the image sample features and the text sample features. The training sample can be expressed as:

[0129] sample = {‘text’: T cls ,‘image global ’: V m_vls ,‘image middle ’: V a_cls ,‘image local ’: V r_cls ,‘label’: h}(16)

[0130] wherein h is a binary label, used to indicate the semantic matching of the text-image pair, when h takes the value of 1, it represents that the image and the text are semantically matched, and when h takes the value of 0, it represents that the image and the text are not semantically matched.

[0131] In some embodiments, an influence function based on random matrix approximation is constructed, and key historical samples are efficiently identified and replayed by using the conjugate gradient descent method, which can effectively alleviate the forgetting problem of the model in the dynamic knowledge updating process. The influence function is expressed as:

[0132]

[0133] wherein Influence i is the influence value of the i-th sample; s i is the gradient vector of the i-th sample; g i is the gradient of the loss function (the loss function shown in formula 15) of the i-th sample with respect to all parameters of the model that need to be updated; v i is the prediction probability vector of the i-th sample, specifically the prediction probability of the true label; H -1 is the inverse of the Hessian matrix of all parameters of the model that need to be updated, used to measure the second-order derivative information of all parameters of the model that need to be updated with respect to the loss function.

[0134] In some embodiments, after selecting important replay samples, the knowledge graph model is optimized based on all samples and replay samples by using the gradient descent method to update the parameters of the model, and the updating method is:

[0135]

[0136] wherein η is the learning rate, is the gradient of the total loss function of the model with respect to the set of parameters of the model, and θ newθ old θ

[0137] The total loss function includes a conditional random field loss function, a relation extraction loss function and a replay sample loss function, and is expressed as:

[0138] L total = L CRF + L RE + L replay (19)

[0139] The gradient of the total loss function with respect to the model parameter set The gradient of the conditional random field loss function The gradient of the relation extraction loss function And the gradient of the replay sample loss function is expressed as:

[0140]

[0141] In the formula, a is the weight of the replay loss, used to balance the importance of the current loss and the replay loss.

[0142] The multi-modal knowledge graph construction method provided by the embodiments of the present application adopts a prompt learning strategy, and in the training and updating of the knowledge graph model, the multi-modal information is interacted by dynamically adjusting the prompt vector, which can effectively solve the imbalance problem of multi-modal information in the fusion process; when the model is fine-tuned, only a small amount of prompt vectors need to be updated instead of all the parameters of the model, which greatly reduces the computational overhead and improves the resource utilization efficiency. The sample replay strategy based on the influence function is adopted, the influence value is calculated, and the most representative historical samples are selected for replay, which can effectively maintain the memory of historical knowledge while maintaining the learning ability of the model to new knowledge. In the model training process, the loss of the current task and the loss of the replay sample are combined for optimization, realizing the balanced learning of new and old knowledge. The knowledge graph model optimized by training, updating and optimization can efficiently and accurately identify entities and the association relationship between entities, determine the triplets including entities and relationships, and thus construct a complete and accurate knowledge graph.

[0143] The technical effects of the method of the present application are illustrated below in combination with experimental data.

[0144] As shown in Table 1, the method of the present application is compared with the baseline model (MKGformer) and the model without the sample replay mechanism in terms of the accuracy of identifying human entities (PER), location type entities (LOC), organization type entities (ORG), other type entities (MISC), and the average (AVG) of each entity type. For the baseline model, the identification accuracy of other types of entities except human entities is not high, indicating that the model has a serious knowledge forgetting problem. For the model without the sample replay mechanism, the identification accuracy of organization type entities and other type entities is low, while the identification performance of the method of the present application for each type of entity is good, especially the identification performance of organization type entities and other type entities is significantly improved, and the performance of location type entities is also improved, indicating that the use of the replay mechanism helps the model to better retain and utilize previously learned knowledge.

[0145] Table 1 Comparison of entity identification accuracy

[0146]

[0147] As shown in Table 2, in terms of resource occupation, after adopting the hint learning mechanism, the memory occupation is greatly reduced due to the reduction of gradient storage, the reduction of optimizer state, and the simplification of the computation graph.

[0148] Table 2 Comparison of resource occupation

[0149]

[0150] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices will interact with each other to complete the method.

[0151] It should be noted that the above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0152] As shown in Figure 2 the embodiments of the present application provide a multi-modal knowledge graph construction device, comprising:

[0153] An acquisition module is configured to acquire images and texts having a correlation relationship;

[0154] An extraction module is configured to perform feature extraction on the images and the texts to obtain image features and text features.

[0155] A fusion module is configured to input the image features and the text features into a pre-constructed knowledge graph model, and perform multi-modal fusion processing on the image features and the text features based on a prompt vector obtained through training by using the knowledge graph model, to obtain multi-modal fusion features.

[0156] An identification module is configured to identify a plurality of entities and a correlation relationship between the entities based on the multi-modal fusion features, and determine a triple of a multi-modal knowledge graph.

[0157] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the embodiments of the present application.

[0158] The apparatus of the above embodiments is used to implement the corresponding method in the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again.

[0159] Figure 3 A more specific hardware structure of an electronic device is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0160] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present specification.

[0161] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1020 and called and executed by the processor 1010.

[0162] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0163] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0164] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0165] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain components necessary for implementing the embodiments of the present specification, and does not have to contain all the components shown in the figure.

[0166] The electronic device of the above embodiment is used to realize the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described here.

[0167] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0168] Those skilled in the art should understand that the above discussion of any embodiment is only intended to be exemplary in nature and is not intended to suggest any limitation as to the scope of the disclosure including the claims. Alternate embodiments of both known and / or novel configurations can implement the same or similar techniques in a manner consistent with the disclosure. In other words, any of the embodiments of the present disclosure (including all features and / or embodiments thereof) can be altered or modified and can be implemented in a variety of ways without departing from the spirit or scope of the present disclosure.

[0169] In addition, to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. Furthermore, devices can be shown in block diagram form in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details regarding the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (i.e., these details should be well within the understanding of one of skill in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the disclosure, it should be apparent to those skilled in the art that the embodiments of the present application can be practiced without, or with variation of, these specific details. The description is thus to be considered as illustrative and not restrictive, and the scope of the disclosure to be indicated by the appended claims.

[0170] Although the present disclosure has been described in connection with certain specific embodiments thereof, many modifications, changes, variations and substitutions will be apparent to those skilled in the art once they are given the benefit of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0171] It is therefore intended that the embodiments of the present application be covered by all of the following claims as they can now or in the future be amended to expressly include all such alternatives, modifications and variations.

Claims

1. A method for constructing a multi-modal knowledge graph, characterized in that, The method comprises the following steps: obtaining images and texts with an association relationship; performing feature extraction on the images and texts to obtain image features and text features; wherein the image features include image global features, image medium features, and image local features; inputting the image features and text features into a pre-constructed knowledge graph model, and performing multi-modal fusion processing on the image features and text features based on first and second prompt vectors obtained through training by using the knowledge graph model to obtain multi-modal fusion features, including: concatenating the first prompt vector with the image global features, image medium features, and image local features respectively to obtain image global prompt features, image medium prompt features, and image local prompt features; concatenating the second prompt vector with the text features to obtain text prompt features; performing dimension alignment processing on the image global prompt features, image medium prompt features, and image local prompt features, and text prompt features by using a pre-set multi-layer perception to obtain dimension-aligned image global prompt features, image medium prompt features, image local prompt features, and text prompt features; concatenating the dimension-aligned image global prompt features with the text prompt features to obtain image global text features, concatenating the dimension-aligned image medium prompt features with the text prompt features to obtain image medium text features, and concatenating the dimension-aligned image local prompt features with the text prompt features to obtain image local text features; performing feature fusion processing on the image global text features, image medium text features, and image local text features respectively by using a pre-set multi-head attention module to obtain fused image global text features, image medium text features, and image local text features; and adding the fused image local text features, image medium text features, and image local text features to obtain multi-modal fusion features; based on the multi-modal fusion features, identifying a plurality of entities and the association relationship between the entities to determine the triples of the multi-modal knowledge graph.

2. The method of claim 1, wherein, performing feature extraction on the images and files to obtain image features and text features, including: performing feature extraction on the images by using a pre-set multi-modal model to obtain image global features, image medium features, and image local features; performing vector conversion processing on the text by using a pre-set language representation model to obtain text features.

3. The method of claim 1, wherein, Further comprising: adjusting the first and second prompt vectors, and performing lightweight fine-tuning on the knowledge graph model based on the adjusted first and second prompt vectors to obtain an updated knowledge graph model.

4. The method of claim 3, wherein, adjusting the first and second prompt vectors, including: randomly generating first and second prompt vectors subject to a normal distribution, and the lengths of the first and second prompt vectors are the same.

5. The method according to claim 1 or 3, characterized in that, Further comprising: calculating the influence values of all samples by using a pre-set influence function, and selecting a plurality of samples with high influence values as replay samples; wherein the samples are constructed based on image samples and text samples with an association relationship; training the knowledge graph model based on all samples and the replay samples to obtain an updated knowledge graph model.

6. The method of claim 5, wherein, The influence function is: where Influence i is the influence value of the i-th sample, s i is the gradient vector of the i-th sample, g i is the gradient of the replay sample loss function with respect to all the parameters of the model that need to be updated for the i-th sample, v i is the predicted probability of the i-th sample, H -1 is the inverse of the Hessian matrix of all the parameters of the model that need to be updated.

7. The method of claim 6, wherein, The knowledge graph model is trained based on all samples and the replay samples to obtain an updated knowledge graph model, including: Based on all samples and the replay samples, the gradient descent method is used to optimize the knowledge graph model, update the parameters of the knowledge graph model, and the update method is: wherein η is a learning rate, is a gradient of the total loss function of the knowledge graph model with respect to the set of model parameters, θ new is the set of updated model parameters, θ old is the set of model parameters before updating, the set of model parameters including all parameters of the model that need to be updated, and the total loss function including a conditional random field loss function for identifying entities, a relation extraction loss function for extracting relations between entities, and a replay sample loss function.

8. The method of claim 1, wherein, Based on the multi-modal fusion feature, the correlation between multiple entities is identified, and the triplets of the multi-modal knowledge graph are determined, including: The multi-modal fusion feature is linearly transformed to obtain a score matrix for entity recognition; the conditional random field layer calculates the path score and total score of the entity label sequence based on the score matrix, and determines the recognized entity according to the calculation result; The multi-modal fusion feature is linearly transformed to obtain a score for mapping the multi-modal fusion feature to each relationship category; the score is converted into a probability distribution through normalization processing to obtain the prediction probability of each relationship category. 9.A multi-modal knowledge graph construction apparatus, characterized in that, Comprising: An acquisition module is configured to acquire images and texts having a correlation relationship; An extraction module is configured to extract features from the images and texts to obtain image features and text features; wherein the image features include image global features, image mid-level features, and image local features; A fusion module is configured to input the image features and text features into a pre-constructed knowledge graph model, and perform multi-modal fusion processing on the image features and text features based on first and second prompt vectors obtained through training by using the knowledge graph model, to obtain multi-modal fusion features, including: The first prompt vector is concatenated with the image global features, image mid-level features, and image local features respectively to obtain image global prompt features, image mid-level prompt features, and image local prompt features; the second prompt vector is concatenated with the text features to obtain text prompt features; a preset multi-layer perception machine is used to perform dimension alignment processing on the image global prompt features, image mid-level prompt features, image local prompt features, and text prompt features to obtain dimension-aligned image global prompt features, image mid-level prompt features, image local prompt features, and text prompt features; the dimension-aligned image global prompt features are concatenated with the text prompt features to obtain image global text features, the dimension-aligned image mid-level prompt features are concatenated with the text prompt features to obtain image mid-level text features, and the dimension-aligned image local prompt features are concatenated with the text prompt features to obtain image local text features; a preset multi-head attention module is used to perform feature fusion processing on the image global text features, image mid-level text features, and image local text features respectively to obtain fused image global text features, image mid-level text features, and image local text features; and the fused image local text features, image mid-level text features, and image local text features are added to obtain multi-modal fusion features; An identification module is configured to identify multiple entities and the correlation between the entities based on the multi-modal fusion features, and determine triplets of a multi-modal knowledge graph.

Images