Intelligent media asset multi-modal knowledge graph management method based on large model

Abstract

The invention provides an intelligent media resource multi-modal knowledge graph management method based on a large model. The method comprises the following steps: S1, collecting and preprocessing media resource multi-modal data; step S2, adopting a multi-modal entity relationship extraction method to extract a triple from the preprocessed media asset multi-modal data; step S3, according to the triple obtained by extraction, constructing a media asset multi-modal knowledge graph; and S4, performing iterative management on the media asset multi-modal knowledge graph through a large model. Aiming at the problems of large order of magnitude, various modes and inconvenience in manual management of the media asset information, the invention provides an intelligent solution method based on a large model, and the management efficiency of the media asset information is effectively improved.

Description

Technical Field

[0001] The present invention relates to a media asset multimodal knowledge graph management method, and in particular to an intelligent media asset multimodal knowledge graph management method based on a large model. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the continuous development of network technology, the channels, communicators, content, and audiences of information dissemination have the characteristics of personalization, selectivity, diverse forms of expression, rapid information release, diversified interactive methods, and high mass communication efficiency compared with the traditional model without network applications, thus generating new media. Content assets generated by new media, referred to as media assets, include various similar media materials such as pictures, texts, and audio. With the growing growth of the new media industry, media asset data presents the characteristics of large data volume, high and different dimensions, and high overhead and storage costs, which brings challenges to the management of media content. At present, most media asset management methods in the market are single-mode and multi-modal internal relationship management. It is still difficult to manage multi-modal data with large scale and high dimensions. The difficulties are manifested in two aspects: first, the media asset data has diverse modalities and complex structures, and the data volume is large, which poses great challenges to the efficiency of storage and processing; second, there are differences in high-level semantics and underlying features in the data, resulting in a "semantic gap", and there is also a "heterogeneous gap" due to the different structures between different modal data. It is a challenging and practical task to give full play to the value of media asset data, which has received more and more attention.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a large-model-based intelligent media multimodal knowledge graph management method in view of the deficiencies in the prior art.

[0006] In order to solve the above technical problems, the present invention discloses a method for managing multimodal knowledge graphs of intelligent media assets based on a large model, comprising the following steps:

[0007] Step S1, collecting and preprocessing media asset multimodal data;

[0008] Step S2, using a multimodal entity relationship extraction method to extract triples from the preprocessed media asset multimodal data;

[0009] Step S3, constructing a media asset multimodal knowledge graph based on the extracted triples;

[0010] Step S4, iteratively managing the media asset multimodal knowledge graph through a large model.

[0011] Furthermore, the preprocessing described in step S1 specifically includes:

[0012] Step S1-1, continuously collecting multimodal data of new media websites; the multimodal data includes text, images, audio and video; the continuous collection means tracking the updated data of the new media websites and collecting newly released data;

[0013] Step S1-2, preprocessing the collected data, and finally obtaining the preprocessed media multimodal data including text and images.

[0014] Furthermore, the preprocessing of the collected data in step S1-2 specifically includes:

[0015] Step S1-2-1, extracting information or parsing documents from data in different formats;

[0016] Step S1-2-2, for data with duplication, outliers and different specifications, data merging, data cleaning and data standardization are used to process them respectively;

[0017] Delete the information that does not contain images and the percentage of stop words in the text exceeds the preset value;

[0018] Discard images with resolutions lower than the preset value;

[0019] Remove duplicate information with the same content.

[0020] Furthermore, the step S2 of extracting triples from the preprocessed media asset multimodal data specifically includes:

[0021] Step S2-1, encode the preprocessed data; wherein, for text encoding, the BERT network is used to obtain the text modality feature vector, and for image encoding, the ResNet network is used to obtain the top three sub-objects [I1, I2, I3] with the highest confidence and the corresponding sub-object label information, and the sub-object label information is vectorized through the Word2vec model to obtain the sub-object label vector [w1, w2, w3];

[0022] Step S2-2, label the characters in the text to achieve entity extraction;

[0023] Step S2-3, through the MLM mask model, use the entities extracted in step S2-2 for pre-training, predict the relationship between entities, and finally obtain triples for constructing a knowledge graph, wherein the triples include entities and the relationship between entities.

[0024] Furthermore, the entity extraction described in step S2-2 is implemented in the following specific steps:

[0025] Step S2-2-1, using an image adapter to fuse text and image information, and obtaining a character semantic vector with enhanced visual information based on the text modality feature vector;

[0026] Step S2-2-2, using contrast layer learning to optimize the distribution distance between text entities and images based on the correlation between images and texts;

[0027] Step S2-2-3, predict the entity tag order through the CRF model, label each character in the text with an entity tag, and finally realize entity extraction.

[0028] Furthermore, the construction of the multimodal knowledge graph described in step S3 specifically includes:

[0029] Step S3-1, define the structure of the media asset multimodal knowledge graph, as follows:

[0030] The media asset multimodal data is regarded as the specific attribute value of the entity or concept, which is expressed as: G = {E, R, A, V, TR, TA}, and TA = E × A × (VKG ∪ VMM) is regarded as the set of attribute triples;

[0031] Among them, V in VKG represents attribute value, KG represents knowledge graph, and VKG represents the set of attribute values ​​of knowledge graph; V in VMM represents attribute value, MM represents media asset multimodal data, VMM represents the set of media asset multimodal data as attribute value, and ∪ represents the set of two attribute values;

[0032] In any triple (s, p, o), s represents an entity, o represents its corresponding media asset multimodal data, and when o is an image, a preset label is set for the relationship p;

[0033] Step S3-2, constructing a media asset multimodal knowledge graph according to the media asset multimodal knowledge graph structure defined in step S3-1 and the triples obtained in step S2, as follows:

[0034] Take entities as nodes, the relationships between entities as edges between nodes, and then connect the nodes to build a multimodal knowledge graph of media resources.

[0035] Furthermore, the iterative management of the media asset multimodal knowledge graph described in step S4 specifically includes:

[0036] Step S4-1, using a large model to iteratively update the media asset multimodal knowledge graph;

[0037] Step S4-2: Iteratively train the large model using the media asset multimodal knowledge graph.

[0038] Furthermore, the text and image information fusion is performed by using an image adapter as described in step S2-2-1, that is, by injecting the image adapter into each layer of the Transformer module of the BERT network, and the specific process includes:

[0039] A linear layer is used to project the sub-object label vector to the same dimension as the text modality feature vector, expressed as follows:

[0040] v i =W2(tanh(W1w i +b1))+b2

[0041] Among them, W1, W2, b1, b2 are learnable weights and bias terms, w i is the i-th sub-object label vector obtained by vectorizing the sub-object label information, v i The sub-object label vector matching the i-th dimension;

[0042] Bilinear attention is used to calculate the correlation between the sub-object label vector and the entity, which is expressed as follows:

[0043] s ij =Softmax(c i W attn v j )

[0044] Among them, c i and v j are the text modality feature vector of the i-th character and the j-th sub-object label vector, respectively, W attn is the weight matrix of bilinear attention, which is used to calculate the semantic relationship between entities and sub-objects, s ij is the correlation score between the BERT semantic vector of the i-th character and the j-th sub-object label vector;

[0045] Calculate the weighted sub-object eigenvector as follows:

[0046]

[0047] Among them, z i represents the weighted image sub-object feature vector with relevance to the i-th character, m is the number of image sub-objects, and the character semantic vector enhanced by visual information is expressed as:

[0048] h i =c i +z i

[0049] Among them, h i Character semantic vectors enhanced for visual information.

[0050] Furthermore, the contrast layer learning described in step S2-2-2, that is, the cross-modal alignment judgment, is used to obtain the semantic relationship between the sub-object label vector and the entity category, specifically including:

[0051] After the entity categories are vectorized by the Word2vec model, the similarity scores are obtained by calculating the cosine similarity, as follows:

[0052] e n =word2vec(tag)

[0053] score in = cosine(w i , e n )

[0054] Among them, tag represents the text entity, w i and e n are the corresponding sub-object label vector and entity category vector, score in is the similarity score matrix, representing the similarity score between the i-th sub-object label vector and the n-th entity category;

[0055] Set a different threshold for each entity category. If the semantic similarity score between the current category entity and the image sub-object is higher than the threshold, the entity image pair is considered a positive example, otherwise it is considered a negative example.

[0056] The contrastive learning method is used to optimize the distribution distance between entities and images in the vector space, as follows:

[0057]

[0058] Among them, loss CLL is the loss function of contrastive learning, h i and I j They are respectively the i-th character semantic vector and the j-th sub-object image vector enhanced by visual information.

[0059] Furthermore, the entity tag order is predicted by the CRF model described in step S2-2-3, that is, the transition probability between entity tags is modeled using the CRF model, as follows:

[0060]

[0061] loss crf =-∑p(y|h)

[0062] Among them, p(y|h) represents the transition probability of the predefined label y to the character semantic vector h, S i Represents y i-1 ,y i The transition probability between i represents the i-th predefined label, h represents the text representation or character semantic vector after visual information enhancement, and loss crf Represents the loss function of the CRF model.

[0063] Beneficial effects:

[0064] The present invention provides a large-model-based intelligent media multimodal knowledge graph management method, which constructs a knowledge graph by extracting entities and relationships of multimodal information, iterates the knowledge base and personalizes the information required by users according to the characteristics of different user personalities, thereby improving the efficiency of using media information. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0066] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0067] Figure 2 It is a schematic diagram of continuous collection and preprocessing of multimodal data of media assets in the present invention.

[0068] Figure 3 It is a schematic diagram of the multimodal entity relationship extraction architecture in the present invention.

[0069] Figure 4 It is a schematic diagram of constructing a multimodal knowledge graph in the present invention.

[0070] Figure 5 It is a schematic diagram of dynamic iterative management of a large model in the present invention.

[0071] Figure 6 It is a schematic diagram of displaying multimodal knowledge graph information in the present invention. DETAILED DESCRIPTION

[0072] The present invention proposes a method for managing multimodal knowledge graphs of intelligent media assets based on a large model, which integrates knowledge graphs and large models into the field of media asset management, and has innovative and practical significance. Knowledge graph is an advanced way of organizing knowledge, which mines information value through triples consisting of entities and relationships. Its essence is a graph data structure that represents the semantic relationship network of things based on the knowledge graph theory. The large model is a large model with hundreds of millions of parameters generated through long-term training of massive corpus, which is an important direction and key technology for the development of various fields of artificial intelligence in the future. While the present invention uses the large model to enrich the content of the knowledge graph, it can also track and build personalized knowledge according to different user settings, thereby achieving intelligent iteration of the knowledge graph.

[0073] like Figure 1 As shown, it is a schematic diagram of the overall structure of the present invention, which includes the following steps:

[0074] Step S1, continuous collection and preprocessing of media asset multimodal data;

[0075] Step S2, extracting triples through multimodal entity relationship extraction;

[0076] Step S3, constructing a multimodal knowledge graph;

[0077] Step S4, dynamically iteratively managing the multimodal knowledge graph through the large model;

[0078] like Figure 2 As shown, it is a schematic diagram of continuous collection and preprocessing of multimodal data of media assets in the present invention. Further, in a large model-based intelligent media asset multimodal knowledge graph management method, step S1 includes:

[0079] First, we collect data from new media websites, mainly including various websites: news websites, forums, blogs, social media, etc. The collection program is mainly written in Python language and Scrapy framework (https: / / www.cnblogs.com / -hz01 / p / 18237917). Multimodal data formats include plain text, web documents, pdf, word, ppt, excel documents, pictures and other information.

[0080] Next, the collected data is preprocessed. The preprocessing is divided into two steps: the first step is to unify information in different formats, including web document information extraction and document parsing of PDF, Word, PPT, and Excel documents, and convert them into text and image data. In the second step, due to the existence of duplication, outliers, inconsistent specifications (html tags in text data, or blurred images), etc., the traditional methods of data merging, data cleaning, and data standardization are used to improve data utilization efficiency. In addition, the following cleaning rules are included: delete microblogs that do not contain images and have a text stop word ratio of more than 80%; discard images with a resolution lower than 224×224 pixels; and remove duplicate microblogs with the same content under the same public opinion hot search.

[0081] like Figure 3 Shown is a schematic diagram of the multimodal entity relationship extraction architecture in the present invention.

[0082] The multimodal entity relationship extraction refers to extracting multimodal entity relationships based on fine-grained semantic matching, encoding text and images, achieving fine-grained element matching between images and text, and then identifying the relationship between entities, and finally generating triples for constructing a knowledge graph.

[0083] The triples, entities and relationships constitute triples, namely head entities, relationships and tail entities. The basis of knowledge graph construction is triples, which essentially connects these triples to form a network structure.

[0084] Wherein, step S2 comprises:

[0085] First, the preprocessed multimodal data is encoded. BERT is used for text encoding, and ResNet-50 is used for image encoding. Furthermore, BERT (Bidirectional Encoder Representations From Transformers) is a bidirectional encoder representation based on Transformer. Transformer (https: / / zhuanlan.zhihu.com / p / 338817680) is a deep learning architecture based on the self-attention mechanism, which can effectively handle long-distance dependencies in text sequences. BERT learns rich semantic information and contextual relationships through two stages of pre-training and fine-tuning. ResNet can achieve more efficient image feature extraction. Resnet can effectively solve the problems of gradient disappearance and representation bottlenecks by introducing residual structures and jump connections, thereby optimizing deep networks. The present invention uses a pre-trained Resnet-50 model as the basic network for image encoding. Resnet-50 has 50 convolutional layers, including 1 input layer, 16 residual blocks, 1 average pooling layer, and 1 fully connected layer. These residual blocks contain skip connections between shallow and deep layers, allowing gradients to propagate directly to previous layers. This connection method helps improve network performance while reducing training difficulty.

[0086] Next, multimodal fine-grained semantic alignment is performed. In this step, an image adapter is first used to accurately fuse text-visual information, which can help each entity representation fuse the most helpful visual information based on relevance. Then contrast layer learning is used to promote fine-grained cross-modal alignment, and optimize the distribution distance of text entities and images in the vector space based on the relevance of images and text, so that semantically related entities and images are closer in the vector space, while semantically unrelated entities and images are farther apart in the vector space. Finally, the order of entity labels is predicted through the CRF model (https: / / www.51cto.com / article / 707140.html). This step can reduce the occurrence of unreasonable label sequence combinations.

[0087] Furthermore, the image adapter integrates visual information into text representation by injecting the image adapter into each layer of the Transformer module of BERT. Specifically, the sub-object label vector is first projected to the same dimension as the BERT semantic vector through a linear layer:

[0088] v i =W2(tanh(W1w i +b1))+b2

[0089] Where W1, W2, b1, b2 are learnable weights and bias terms, v i is the sub-object label vector matched in the i-th dimension. The correlation between sub-objects and entities is calculated through bilinear attention:

[0090] s ij =Softmax(c i W attn v j )

[0091] where c i , v j are the BERT semantic vector of the i-th character and the j-th sub-object label vector, W attn is the weight matrix of bilinear attention, responsible for calculating the semantic relationship between entities and sub-objects, s ij is the correlation score between the BERT semantic vector of the i-th character and the i-th sub-object label vector. After obtaining the correlation score, the weighted sub-object feature vector can be calculated:

[0092]

[0093] z i represents the relevance weighted visual sub-object feature vector with respect to the i-th character, m is the number of visual sub-objects, which is set to 3 in the present invention. The text feature vector with visual information enhancement is expressed as:

[0094] h i =c i +z i

[0095] h i Character semantic vectors enhanced for visual information. The correlation between entities and visual sub-objects is calculated through bilinear attention, and the correlation-weighted visual sub-object information is obtained and integrated into the entity representation, ensuring that each entity finds semantically relevant visual information and reducing the introduction of noisy visual information, thereby promoting the construction of multimodal semantic space and improving the final result.

[0096] Furthermore, in order to promote fine-grained cross-modal alignment, contrast layer learning adopts a unique cross-modal alignment judgment to obtain the semantic relationship between visual objects and text entities. Specifically, after the visual sub-objects and entity categories are vectorized through the Word2vec model (https: / / zhuanlan.zhihu.com / p / 616515135), the cosine similarity is calculated to obtain the similarity score:

[0097] e n =word2vec(tag)

[0098] scorenj = cosine(w i , e n )

[0099] where w i , e n Refers to the corresponding sub-object label vector and entity category vector respectively, score nj It refers to the similarity score matrix, which represents the similarity score between the i-th visual sub-object and the n-th entity category. Next, different thresholds are set for each entity category. Only when the semantic similarity score between the entity and the visual sub-object of this category is higher than the threshold, the visual sub-object and this type of entity are considered to be semantically related, which is helpful for entity recognition, and the entity-image pair is considered to be a positive example; when the semantic similarity score is lower than the threshold, it is considered that this type of entity and the visual sub-object are semantically irrelevant, which is of limited or no help for entity recognition, and the entity-image pair is considered to be a negative example. The proposed semantic relevance judgment method can explicitly and accurately align the fine-grained knowledge between different modalities, improve the accuracy of multimodal knowledge fusion, filter the noise brought by irrelevant visual modal information, and help build a multimodal shared semantic space. After fine-grained cross-modal alignment, the distribution distance of entities and sub-object images in the vector space is optimized. Specifically, it is hoped that the distribution distance of semantically related entities and sub-object images in the vector space will be closer, while the distribution distance of semantically unrelated entities and sub-object images in the vector space will be farther. This paper proposes a contrastive learning method to optimize the distribution distance between entities and images in vector space. The formula is as follows:

[0100]

[0101] Among them, h i and I j are the i-th character semantic vector and the j-th sub-object image vector enhanced by visual information. The proposed contrastive learning method will shorten the distance between semantically related entities and sub-object images in the vector space, while increasing the distance between semantically unrelated entities and sub-object images, helping entities to more accurately understand semantically related visual knowledge and accurately integrate visual information.

[0102] Furthermore, the CRF model models the transition probability between labels. The specific formula is as follows:

[0103]

[0104] loss crf =-∑p(y|h)

[0105] Y represents the predefined label set, loss represents the final loss function, and S represents the transition probability between different labels.

[0106] Finally, the MLM mask model (https: / / blog.csdn.net / jun778895 / article / details / 140825854) is used to pre-train the relationship between entities, thereby predicting the relationship between entities and finally obtaining the triples for constructing the knowledge graph. Further MLM mask model. The masked language model technology mainly draws on the strategy of the BERT model and uses unmasked information to infer the representation of masked content. This technology takes into account the learning task to ensure that the multimodal pre-trained model not only relies on a single modality (text or image) information mask, but also integrates multimodal information to infer the masked content. The masked language model technology can effectively learn the connection between vision and language and improve the model's ability to understand multimodal data.

[0107] like Figure 4 As shown, it is a schematic diagram of the construction of the multimodal knowledge graph in the present invention. The multimodal knowledge graph is constructed according to entities and relationships, and the media asset entities are regarded as nodes in the graph, and the relationships are used as edges between nodes, and then the nodes are connected. According to the fused multimodal semantic information and knowledge tracking results, the media asset multimodal knowledge graph is constructed, and finally the graph database is used for knowledge storage and query.

[0108] Specifically, step S3 includes:

[0109] To construct a multimodal knowledge graph, first clarify the definition of the multimodal knowledge graph in the present invention. The traditional knowledge graph (KG) can be defined as a directed graph G = {E, R, A, V, TR, TA}, where E, R, A, V are sets of entities, relationships, attributes and attribute values, and TR = E × R × E and TA = E × A × V represent relationship triples (TR) and attribute triples (TA), respectively. For the triple (s, p, o) ∈ TR, it means that there is a relationship p ∈ R between the entity s ∈ E and the entity o ∈ E. For the triple (s, p, o) ∈ TA, it means that the entity s ∈ E has an attribute p ∈ A, and its attribute value is o ∈ V.

[0110] In the present invention, the Multi-Modal Knowledge Graph (MMKG) takes multimodal data (image) as a specific attribute value of an entity or concept. It is consistent with the traditional knowledge graph and is expressed as G = {E, R, A, V, TR, TA}, except that TA = E × A × (VKG ∪ VMM) is a set of attribute triples. In VKG, V represents the attribute value, KG represents the knowledge graph, VKG represents the set of attribute values ​​of the traditional knowledge graph, V represents the attribute value in VMM, MM represents multimodal data, VMM represents the set of multimodal data as attribute values, and ∪ represents the set of two attribute values. Since multimodal data is regarded as attribute values, in a triple (s, p, o), s represents an entity, o represents its corresponding multimodal data, and when o is an image, the relationship p is "hasImage".

[0111] like Figure 5 The figure shows a schematic diagram of dynamic iteration management of a large model in the present invention.

[0112] The large model is a deep learning model that contains a large number of training parameters and complex network structures. It can process massive data and assist in completing complex tasks, including image recognition, computer vision, speech recognition, natural language processing, etc. At present, open source large models have become artificial intelligence infrastructure. The present invention uses ChatGLM3 to discover new knowledge and enrich the conditional knowledge graph.

[0113] The dynamic iterative management of the multimodal knowledge graph refers to performing knowledge reasoning based on the constructed media multimodal knowledge graph and recommending relevant media information based on different user profiles.

[0114] Specifically, step S4 includes:

[0115] The dynamic iteration management of large models includes two parts. First, the large model can be used for the iteration of the knowledge graph to enrich the knowledge of the knowledge graph. Second, the knowledge graph itself can be used as the embedded knowledge of the large model to pre-train the large model. The present invention uses ChatGLM3 to realize the dynamic iteration management of large models. On October 27, 2023, Zhipu AI launched the self-developed third-generation base large model ChatGLM3 and its related series of products at the 2023 China Computer Conference. This is a major breakthrough after the launch of the dialogue models ChatGLM and ChatGLM2 with a trillion bases. ChatGLM3 adopts an original multi-stage enhanced pre-training method, ranking first among domestic models of the same size. ChatGLM3 has functions such as multimodal understanding, code generation and execution, network search enhancement, and significant improvements in performance and deployment.

[0116] Specifically, the large model iterates the knowledge graph by finding examples related to the input sample from the artificially predefined samples through multimodal similarity detection, integrating them into the multimodal prompt template to assist the large model in generating high-quality auxiliary knowledge, and then splicing the original text with the auxiliary knowledge generated by the large language model to perform multimodal entity relationship extraction in step S2 to form new graph knowledge.

[0117] Furthermore, the manually predefined samples are manually labeled for a part of the data samples. When labeling, it is necessary to judge the categories of the two entities in the sentence, and at the same time provide relevant auxiliary knowledge as the basis for making such judgments. The auxiliary knowledge should include images, text content and relevant prior knowledge. Multimodal similarity detection is used to ensure that appropriate context examples are selected, because the learning ability of the large model is greatly affected by the context examples. The multimodal prompt template is a series of standard questions and answers, which serve as the context of the large model, including text, images, questions, and answers. Put the above multimodal prompt template and input samples into the large model to get the answer of the large model.

[0118] Specifically, the knowledge graph is used as a large model pre-training corpus: the present invention converts the knowledge graph into synthetic natural language sentences to enhance the existing pre-training corpus, so that it can be integrated into the pre-training language model without changing the architecture, mainly converting the knowledge graph into a knowledge-enhanced language model (KELM) (AGARWAL O, GE H, SHAKERI S, et al. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training[J].arXivpreprint arXiv:2010.12688, 2020.) corpus.

[0119] Example:

[0120] This embodiment provides a method for managing a multimodal knowledge graph of intelligent media assets based on a large model. Taking the perspective of a public opinion analyst as an example, the practical application of this method is demonstrated. The steps are as follows:

[0121] Step S1, continuous collection and preprocessing of media asset multimodal data;

[0122] Step S2, extracting triples through multimodal entity relationship extraction;

[0123] Step S3, constructing a multimodal knowledge graph;

[0124] Step S4, dynamically iteratively managing the multimodal knowledge graph through the large model;

[0125] Step S1, continuous collection and preprocessing of media asset multimodal data, including:

[0126] First, data on new media websites are collected, which mainly include five types of websites: government agencies, news websites, forums, blogs, and social media.

[0127] Next, the collected data is preprocessed. The preprocessing is divided into two steps: the first step is to unify information in different formats, including web document information extraction and document parsing of PDF, Word, PPT, and Excel documents, and convert them into text and image data. In the second step, due to the existence of duplication, outliers, inconsistent specifications (html tags in text data, or blurred images), etc., the traditional methods of data merging, data cleaning, and data standardization are used to improve data utilization efficiency. In addition, the following cleaning rules are included: delete microblogs that do not contain images and have a text stop word ratio of more than 80%; discard images with a resolution lower than 224×224 pixels; and remove duplicate microblogs with the same content under the same public opinion hot search.

[0128] Step S2, extracting triples through multimodal entity relationship extraction, including:

[0129] First, the preprocessed multimodal data is encoded. BERT is used for text encoding and ResNet is used for image encoding. BERT learns rich semantic information and contextual relationships through two stages of pre-training and fine-tuning. ResNet can achieve more efficient image feature extraction.

[0130] Next, multimodal fine-grained semantic alignment is performed. In this step, an image adapter is first used to accurately fuse text-visual information, which can help each entity representation fuse the most helpful visual information based on relevance. Then contrast layer learning is used to promote fine-grained cross-modal alignment and optimize the distribution distance of text entities and images in the vector space based on the relevance of images and text, so that semantically related entities and images are closer in the vector space, while semantically unrelated entities and images are farther apart in the vector space. Finally, the order of entity labels is predicted through the CRF model. This step can reduce the occurrence of unreasonable label sequence combinations.

[0131] Finally, the MLM mask model is used to pre-train the relationship between entities, so as to predict the relationship between entities and finally obtain the triples for constructing the knowledge graph. Further MLM mask model. The masked language model technology mainly draws on the strategy of the BERT model and uses unmasked information to infer the representation of masked content. This technology takes into account the learning task to ensure that the multimodal pre-trained model not only relies on a single modality (text or image) information mask, but also integrates multimodal information to infer the masked content. The masked language model technology can effectively learn the connection between vision and language and improve the model's ability to understand multimodal data.

[0132] Step S3, constructing a multimodal knowledge graph, including:

[0133] Multi-Modal Knowledge Graph (MMKG) is used to treat multimodal data (images) as specific attribute values ​​of entities or concepts. It is consistent with the traditional knowledge graph and is expressed as G = {E, R, A, V, TR, TA}. The difference is that TA = E × A × (VKG ∪ VMM) is a set of attribute triples. In VKG, V represents the attribute value, KG represents the knowledge graph, VKG represents the set of attribute values ​​of the traditional knowledge graph, V represents the attribute value in VMM, MM represents multimodal data, VMM represents the set of multimodal data as attribute values, and ∪ represents the set of two attribute values. Since multimodal data is regarded as attribute values, in a triple (s, p, o), s represents an entity, o represents its corresponding multimodal data, and when o is an image, the relationship p is "hasImage".

[0134] Step S4, dynamically iteratively managing the multimodal knowledge graph through the large model, including:

[0135] First, the large model can be used for the iteration of the knowledge graph and enrich the knowledge of the knowledge graph. Secondly, the knowledge graph itself can be used as the embedded knowledge of the large model to pre-train the large model. The present invention uses ChatGLM3 to realize the dynamic iteration management of the large model. On October 27, 2023, Zhipu AI launched the self-developed third-generation base large model ChatGLM3 and its related series of products at the 2023 China Computer Conference. This is a major breakthrough after the launch of the dialogue models ChatGLM and ChatGLM2 with a trillion bases. ChatGLM3 adopts an original multi-stage enhanced pre-training method, ranking first among domestic models of the same size. ChatGLM3 has functions such as multimodal understanding, code generation and execution, network search enhancement, and significant improvements in performance and deployment.

[0136] Specifically, the large model iterates the knowledge graph by finding examples related to the input sample from the artificially predefined samples through multimodal similarity detection, integrating them into the multimodal prompt template to assist the large model in generating high-quality auxiliary knowledge, and then splicing the original text with the auxiliary knowledge generated by the large language model to perform multimodal entity relationship extraction in step S2 to form new graph knowledge.

[0137] Furthermore, the manually predefined samples are manually labeled for a part of the data samples. When labeling, it is necessary to judge the categories of the two entities in the sentence, and at the same time provide relevant auxiliary knowledge as the basis for making such judgments. The auxiliary knowledge should include images, text content and relevant prior knowledge. Multimodal similarity detection is used to ensure that appropriate context examples are selected, because the learning ability of the large model is greatly affected by the context examples. The multimodal prompt template is a series of standard questions and answers, which serve as the context of the large model, including text, images, questions, and answers. Put the above multimodal prompt template and input samples into the large model to get the answer of the large model.

[0138] Specifically, the knowledge graph is used as a large model pre-training corpus: the present invention converts the knowledge graph into synthetic natural language sentences to enhance the existing pre-training corpus, so that it can be integrated into the pre-training language model without changing the architecture, mainly converting the knowledge graph into a knowledge-enhanced language model (KELM) (AGARWAL O, GE H, SHAKERI S, et al. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training[J].arXivpreprint arXiv:2010.12688, 2020.) corpus.

[0139] Public opinion analysts obtain multimodal knowledge graph information through the multimodal retrieval interface. The multimodal knowledge graph information is displayed in Figure 6 (This is for illustration only, the content is blurred). Based on the search text content, the large model retrieval returns the corresponding information of each modality. The content that is not suitable for public disclosure has been blurred in the figure. In the search results in the figure, the left side is text content, the middle is a picture, and the right side is a video.

[0140] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a large-model-based intelligent media multimodal knowledge graph management method provided by the present invention and some or all of the steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0141] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0142] The present invention provides a method and idea for a multimodal knowledge graph management method for intelligent media assets based on a large model. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A method for managing multimodal knowledge graphs of intelligent media assets based on a large model, characterized in that: The steps include: Step S1, collecting and preprocessing media asset multimodal data; Step S2, using a multimodal entity relationship extraction method to extract triples from the preprocessed media asset multimodal data; Step S3, constructing a media asset multimodal knowledge graph based on the extracted triples; Step S4, iteratively managing the media asset multimodal knowledge graph through a large model.

2. According to claim 1, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The preprocessing described in step S1 specifically includes: Step S1-1, continuously collecting multimodal data of new media websites; the multimodal data includes text, images, audio and video; the continuous collection means tracking the updated data of the new media websites and collecting newly released data; Step S1-2, preprocessing the collected data, and finally obtaining the preprocessed media multimodal data including text and images.

3. According to claim 2, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The preprocessing of the collected data in step S1-2 specifically includes: Step S1-2-1, extracting information or parsing documents from data in different formats; Step S1-2-2, for data with duplication, outliers and different specifications, data merging, data cleaning and data standardization are used to process them respectively; Delete the information that does not contain images and the percentage of stop words in the text exceeds the preset value; Discard images with resolutions lower than the preset value; Remove duplicate information with the same content.

4. According to claim 3, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The step S2 of extracting triples from the preprocessed media asset multimodal data specifically includes: Step S2-1, encode the preprocessed data; wherein, for text encoding, the BERT network is used to obtain the text modality feature vector, and for image encoding, the ResNet network is used to obtain the top three sub-objects [I1, I2, I3] with the highest confidence and the corresponding sub-object label information, and the sub-object label information is vectorized through the Word2vec model to obtain the sub-object label vector [w1, w2, w3]; Step S2-2, label the characters in the text to achieve entity extraction; Step S2-3, through the MLM mask model, use the entities extracted in step S2-2 for pre-training, predict the relationship between entities, and finally obtain triples for constructing a knowledge graph, wherein the triples include entities and the relationship between entities.

5. According to claim 4, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The specific steps for implementing entity extraction described in step S2-2 are as follows: Step S2-2-1, using an image adapter to fuse text and image information, and obtaining a character semantic vector with enhanced visual information based on the text modality feature vector; Step S2-2-2, using contrast layer learning to optimize the distribution distance between text entities and images based on the correlation between images and texts; Step S2-2-3, predict the entity tag order through the CRF model, label each character in the text with an entity tag, and finally realize entity extraction.

6. According to claim 5, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The construction of a multimodal knowledge graph described in step S3 specifically includes: Step S3-1, define the structure of the media asset multimodal knowledge graph, as follows: The multimodal data of media assets are regarded as the specific attribute values ​​of entities or concepts, which are expressed as: G = {E, R, A, V, TR, TA}, and TA = E × A × (VKG ∪ VMM) is regarded as the set of attribute triples; Among them, V in VKG represents attribute value, KG represents knowledge graph, and VKG represents the set of attribute values ​​of knowledge graph; V in VMM represents attribute value, MM represents media asset multimodal data, VMM represents the set of media asset multimodal data as attribute value, and ∪ represents the set of two attribute values; In any triple (s, p, o), s represents an entity, o represents its corresponding media asset multimodal data, and when o is an image, a preset label is set for the relationship p; Step S3-2, constructing a media asset multimodal knowledge graph according to the media asset multimodal knowledge graph structure defined in step S3-1 and the triples obtained in step S2, as follows: Take entities as nodes, the relationships between entities as edges between nodes, and then connect the nodes to build a multimodal knowledge graph of media resources.

7. The method for managing multimodal knowledge graphs of intelligent media assets based on a large model according to claim 6, characterized in that: The iterative management of the media asset multimodal knowledge graph described in step S4 specifically includes: Step S4-1, using a large model to iteratively update the media asset multimodal knowledge graph; Step S4-2: Iteratively train the large model using the media asset multimodal knowledge graph.

8. According to claim 7, a large model-based intelligent media asset multimodal knowledge graph management method is characterized in that: The text and image information fusion is performed by using an image adapter as described in step S2-2-1, that is, by injecting the image adapter into each layer of the Transformer module of the BERT network. The specific process includes: A linear layer is used to project the sub-object label vector to the same dimension as the text modality feature vector, expressed as follows: v i =W2(tanh(W1w i +b1))+b2 Among them, W1, W2, b1, b2 are learnable weights and bias terms, w i is the i-th sub-object label vector obtained by vectorizing the sub-object label information, v i The sub-object label vector matching the i-th dimension; Bilinear attention is used to calculate the correlation between the sub-object label vector and the entity, which is expressed as follows: s ij =Softmax(c i W attn v j ) Among them, c i and v j are the text modality feature vector of the i-th character and the j-th sub-object label vector, respectively, W attn is the weight matrix of bilinear attention, which is used to calculate the semantic relationship between entities and sub-objects, s ij is the correlation score between the BERT semantic vector of the i-th character and the j-th sub-object label vector; Calculate the weighted sub-object eigenvector as follows: Among them, z i represents the weighted image sub-object feature vector with relevance to the i-th character, m is the number of image sub-objects, and the character semantic vector enhanced by visual information is expressed as: h i =c i +z i Among them, h i Character semantic vectors enhanced for visual information.

9. The method for managing multimodal knowledge graphs of intelligent media assets based on a large model according to claim 8, characterized in that: The contrast layer learning described in step S2-2-2, i.e., the cross-modal alignment judgment, is used to obtain the semantic relationship between the sub-object label vector and the entity category, specifically including: After the entity categories are vectorized by the Word2vec model, the similarity scores are obtained by calculating the cosine similarity, as follows: e n =word2vec(tag) score in =cosine(w i ,e n ) Among them, tag represents the text entity, w i and e n are the corresponding sub-object label vector and entity category vector, score in is the similarity score matrix, representing the similarity score between the i-th sub-object label vector and the n-th entity category; Set a different threshold for each entity category. If the semantic similarity score between the current category entity and the image sub-object is higher than the threshold, the entity image pair is considered a positive example, otherwise it is considered a negative example. The contrastive learning method is used to optimize the distribution distance between entities and images in the vector space, as follows: Among them, loss CLL is the loss function of contrastive learning, h i and I j They are respectively the i-th character semantic vector and the j-th sub-object image vector enhanced by visual information.

10. The method for managing multimodal knowledge graphs of intelligent media assets based on a large model according to claim 9, characterized in that: The prediction of entity tag order by the CRF model described in step S2-2-3 is to use the CRF model to model the transition probability between entity tags, as follows: loss crf =-∑p(y|h) Among them, p(y|h) represents the transition probability of the predefined label y to the character semantic vector h, S i Represents y i-1 ,y i The transition probability between i represents the i-th predefined label, h represents the text representation or character semantic vector after visual information enhancement, and loss crf Represents the loss function of the CRF model.

Images