Multi-modal data processing method and device, terminal equipment and storage medium

By acquiring and comparing the entity knowledge of multimodal data information, determining the entity knowledge with the least impact, the problem of low accuracy of multimodal knowledge graphs is solved, and the completeness and application effect of the knowledge graph are improved.

CN118939806APending Publication Date: 2024-11-12TCL TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202310541914.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The completeness and accuracy of the multimodal knowledge graph are low, which affects its effectiveness in multimodal relationship prediction and reasoning, knowledge questions and answers, etc.

Method used

By obtaining entity knowledge of multimodal data information, performing feature extraction and comparison, the entity knowledge with the least impact on the knowledge graph is determined as the correct knowledge, thereby improving the accuracy of knowledge insertion into the knowledge graph.

Benefits of technology

It improves the accuracy of knowledge insertion into knowledge graphs, enhances the completeness and accuracy of multimodal knowledge graphs, and improves its performance in application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939806A_ABST
    Figure CN118939806A_ABST
Patent Text Reader

Abstract

The invention provides a multi-modal data processing method and device, terminal equipment and a storage medium. The method comprises the steps that first knowledge graph information and multi-modal data information are acquired; performing feature extraction on the multi-modal data information to obtain multi-modal feature information; and complementing the first knowledge graph information by using the multi-modal feature information to obtain target knowledge graph information. According to the method, each piece of entity knowledge of the multi-modal data information is obtained, and after the fact that each piece of entity knowledge is respectively inserted into each branch of the knowledge graph is determined, the difference between each new branch and the initial branch is compared, so that the accuracy of inserting the knowledge into the knowledge graph is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal data processing method, apparatus, terminal device and storage medium. Background Art

[0002] At present, multimodal knowledge graphs can integrate various types of data such as text, images, videos, and voice, and help artificial intelligence understanding and reasoning tasks through the integration of multi-source information.

[0003] The realization of multimodal knowledge graph is an inevitable step for artificial intelligence to develop from perceptual intelligence to cognitive intelligence. Therefore, how to build and maintain multimodal knowledge graphs has become a research hotspot. At present, there are some multimodal knowledge graphs, such as: IMGpedia (a Linked Dataset with Content-basedAnalysis of Wikimedia Images), MMKG (Multi-Modal Knowledge Graphs), Yago (YetAnother Great Ontology), Richpedia (A Large-Scale, Comprehensive Multi-Modal Knowledge Graph,), zhishi.me (Weaving Chinese Linking Open Data), PKUPie (PekingUniversityPie), CN-DBpedia (Databasemedia). The construction of these multimodal knowledge graphs usually goes through several main steps: domain concept discovery, cross-media entity recognition, cross-media relationship extraction, and cross-media knowledge fusion. These graphs have been applied in application fields such as multimodal relationship prediction and reasoning, multimodal knowledge question and answer, multimodal knowledge graph recommendation, and zero-shot image classification, and have achieved good results.

[0004] However, the development of multimodal knowledge graphs is still in its early stages, and the completeness and accuracy of multimodal knowledge graphs need to be improved urgently. Summary of the invention

[0005] The present application provides a multimodal data processing method, which obtains each entity knowledge of the multimodal data information, determines the impact of each entity knowledge on the knowledge graph after being inserted into the knowledge graph, and determines that the entity knowledge with the least impact on the knowledge graph is the correct knowledge, thereby improving the accuracy of knowledge insertion into the knowledge graph.

[0006] In a first aspect, the present application provides a multimodal data processing method, the method comprising:

[0007] Obtaining first knowledge graph information and multimodal data information;

[0008] Performing feature extraction on the multimodal data information to obtain multimodal feature information;

[0009] The multimodal feature information is used to complete the first knowledge graph information to obtain the target knowledge graph information.

[0010] In a second aspect, the present application further provides a multimodal data processing device, the device comprising:

[0011] An acquisition module, used to acquire first knowledge graph information and multimodal data information;

[0012] An extraction module, used for performing feature extraction on the multimodal data information to obtain multimodal feature information;

[0013] A completion module is used to use the multimodal feature information to complete the first knowledge graph information to obtain target knowledge graph information.

[0014] In a third aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any one of the multimodal data processing methods.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in any one of the multimodal data processing methods.

[0016] The multimodal data processing method provided in the present application obtains each entity knowledge of the multimodal data information, and then determines that each entity knowledge is inserted into each branch of the knowledge graph respectively, and then compares the difference between each new branch and the initial branch, so as to determine that the entity knowledge with the smallest difference in impact on the knowledge graph is the correct knowledge, thereby improving the accuracy of knowledge insertion into the knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1is a schematic diagram of a scenario of a multimodal data processing system provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an embodiment of a multimodal data processing method in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of an embodiment of a knowledge graph of the multimodal data processing method in the embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of an embodiment of inserting knowledge into a knowledge graph in a multimodal data processing method in an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of an embodiment of inserting knowledge into a knowledge graph in a multimodal data processing method in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of an embodiment of a knowledge graph comparison of a multimodal data processing method in an embodiment of the present application;

[0024] Figure 7 is a functional module schematic diagram of a multimodal data processing device in an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0027] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0028] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. At the same time, it is to be understood that in the specific implementation of this application, when user information, user data and other related data are involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data shall comply with relevant laws, regulations and standards of relevant countries and regions.

[0029] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.

[0030] The present application provides a multimodal data processing method, apparatus, device and storage medium, which are described in detail below.

[0031] See also Figure 1 , Figure 1 This is a schematic diagram of a scenario of a multimodal data processing system provided in an embodiment of the present application. The multimodal data processing system may include a terminal device 100 and a storage device 200. The storage device 200 may transmit data to the terminal device 100. Figure 1 The terminal device 100 in the embodiment can obtain the transmission multimodal data information stored in the storage device 200 to execute the multimodal data processing method in the present application.

[0032] In the embodiment of the present application, the terminal device 100 includes but is not limited to a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a tablet computer, a wireless terminal device, an embedded device, etc.

[0033] In an embodiment of the present application, communication between the terminal device 100 and the storage device 200 can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0034] It should be noted that Figure 1 The scenario diagram of the multimodal data processing system shown is merely an example. The multimodal data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the multimodal data processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0035] like Figure 2 As shown, Figure 2 The flowchart of an embodiment of the multimodal data processing method in the embodiment of the present application is as follows. The multimodal data processing method may include the following steps 201 to 203:

[0036] 201. Obtain first knowledge graph information and multimodal data information.

[0037] In an embodiment of the present application, multimodal data information may refer to data that needs to be supplemented to the knowledge graph. Specifically, the multimodal data information may be stored in a designated storage medium, such as a local memory, or a cloud, etc.; or, the multimodal data information may not be stored data, such as news, videos, audio and other information on a web page. It should be noted that if audio information is obtained when acquiring multimodal data information, the audio information can be converted into text information. In addition, the first knowledge graph information in the embodiment of the present application can be any kind of knowledge graph.

[0038] 202. Perform feature extraction on the multimodal data information to obtain multimodal feature information.

[0039] After obtaining the multimodal data information, feature extraction can be performed on the multimodal data information to obtain the knowledge that needs to be completed in the first knowledge graph information. The method for feature extraction can be through the traditional machine learning CRF method, the deep learning method BiLSTM, etc., which is not limited here. When feature extraction is performed in the above-described manner, multiple multimodal feature information will be obtained. For example, there is the following news: In the quarter-finals of the World Cup, the Netherlands played against Argentina. Depay said in an interview before the game that "I hope we can beat Argentina within 90 minutes. If we have to take a penalty kick, there is no problem." At the same time, the news also attached a close-up image of Depay playing football in the orange Dutch men's national team uniform. At this time, the news is multimodal data information. When knowledge extraction is performed based on the news, multiple entity knowledge can be extracted, including: entity "Depay", entity "Argentina", entity "Netherlands", and entity "close-up image". Since the entity "Argentina" and the entity "Netherlands" are the same type of entity knowledge, triples will be established for these extracted entity knowledge to obtain three entity knowledge, including:

[0040] Knowledge 1: Argentina (entity) - Player (relationship) - Depay (entity).

[0041] Knowledge 2: Depay (entity) - has (relationship) - close-up image of kicking the ball (entity).

[0042] Knowledge 3: Netherlands (entity) - Player (relationship) - Depay (entity).

[0043] These three entity knowledge are the multimodal feature information in the embodiments of the present application.

[0044] In addition, in the embodiment of the present application, the first knowledge graph information may be a graph including multiple subsets, each of which is connected by a connecting line, as shown in FIG. Figure 3 shown. Figure 3 A knowledge graph about the men's national football team is shown in FIG. It should be noted that the knowledge graph may also include other types of knowledge graphs, such as a cuisine knowledge graph, a car knowledge graph, etc., which are not specifically limited here. Figure 3 It can be obtained that each knowledge graph branch in the knowledge graph is a relationship branch. Since the current multimodal data information has not yet been inserted into the knowledge graph, each relationship branch in the current knowledge graph is an initial graph branch compared to the current multimodal data information.

[0045] According to the above steps, the multimodal feature information obtained based on multimodal data information includes specific entity knowledge. Figure 3 It can be seen that these entity knowledge actually correspond to different branches in the knowledge graph. For example, there is Argentina (entity) in knowledge 1, so it corresponds to Figure 3 The Argentine branch of the knowledge graph in . The Netherlands (entity) exists in knowledge 3, which also corresponds to Figure 3 Therefore, based on the knowledge to be supplemented, the corresponding initial graph branch in the knowledge graph can be directly determined.

[0046] In addition, in the specific implementation process, the first knowledge graph information can be backed up multiple times, and the knowledge to be supplemented can be inserted into the first knowledge graph information in each backup. The advantage of this method is that if the extracted knowledge is all wrong, since the first knowledge graph information is backed up, there is no need to restore the changed knowledge graph. When it is determined that the changed knowledge graph does not meet the requirements, the changed erroneous knowledge graph can be directly deleted.

[0047] Among them, updating the corresponding initial graph branches according to different entity knowledge may include first tensorizing the entity knowledge, and inserting the tensorized entity knowledge into each corresponding initial graph branch. Taking knowledge 1 in the above steps as an example, knowledge 1 includes three entities, Argentina (entity), player (relationship), and Depay (entity). The three entities in knowledge 1 are all text entities, so the three text entities can be converted into text tensors, and the converted text tensors can be incorporated into the corresponding initial graph branches. For example, in the knowledge graph listed in the above steps, since knowledge 1 involves Argentina, knowledge 1 will be incorporated into Figure 3 The Argentine player branch.

[0048] In addition, when inserting knowledge 1 into the Argentine player branch, knowledge 1 needs to be characterized first, for example, tensorized based on the entity keywords involved in knowledge 1. At the same time, it should be noted that each player branch is stored in the knowledge graph as a tensor matrix. Therefore, when inserting knowledge 1 into or updating the Argentine player branch, the tensor features of knowledge 1 can be directly spliced ​​into the branch tensor matrix corresponding to the Argentine player branch. For example: convert the entity keywords of knowledge 1 into text tensors, and then splice the text tensor of knowledge 1 into the tensor matrix of the Argentine player branch. Specifically, the method of tensor splicing can refer to any existing technology, and is not limited here.

[0049] It should be noted that in the embodiment of the present application, the entity knowledge obtained according to the multimodal data information, that is, the knowledge to be supplemented, can be multiple. Therefore, if we still use the knowledge 1, knowledge 2 and knowledge 3 listed in the above embodiment as an example, knowledge 1 is used to update Figure 3 The Argentine branch in the Knowledge 2 is used to update Figure 3 The Dutch branch in the game, Knowledge 3 is still used for updating Figure 3At this time, knowledge 3 and knowledge 2 are both used to update the Dutch branch. Therefore, when updating the same branch through different knowledge, as described in the above paragraph, the original knowledge graph can be backed up in the form of a backup. When the same branch needs to be updated through different knowledge, each different knowledge updates an initial knowledge graph to avoid two different knowledge updating a branch in a graph at the same time.

[0050] 203. Use multimodal feature information to complete the first knowledge graph information to obtain target knowledge graph information.

[0051] According to the above steps, after obtaining the multimodal data information, multiple multimodal feature information, that is, multiple knowledge, can be obtained. Moreover, based on different multimodal feature information, the multimodal data information can be merged with the corresponding entity branches in the first knowledge graph information to obtain a new first knowledge graph information. However, in actual situations, only one entity branch is a real branch of the multimodal data information. Therefore, an optimal branch can be selected based on each updated branch. For example: the similarity between each updated branch and the previous unupdated state can be calculated. The higher the similarity, the smaller the impact on the branch after merging the multimodal data information into the current branch, that is, when the multimodal data and the data already existing in the branch belong to the same type of data, the impact on the branch will be minimized. Therefore, a branch with the highest similarity can be selected as the branch that the multimodal data information needs to merge. For example: Assume that there are branches A, B, C, and so on in the first knowledge graph information. At this time, after merging the multimodal data information with branch A, branch B and branch C respectively, we get branch A', branch B' and branch C'. If the similarity between branch A and branch A' is greater than the similarity between branch B and branch B', and the similarity between branch C and branch C', the multimodal data information is merged into branch A to form a new first knowledge graph information.

[0052] The multimodal data processing method provided in the present application obtains each entity knowledge of the multimodal data information, and then determines that each entity knowledge is inserted into each branch of the knowledge graph respectively, and then compares the difference between each new branch and the initial branch, so as to determine that the entity knowledge with the smallest difference in impact on the knowledge graph is the correct knowledge, thereby improving the accuracy of knowledge insertion into the knowledge graph.

[0053] In order to better implement the embodiment of the present application, in one embodiment of the present application, the multimodal data information includes text data and image data, and feature extraction is performed on the multimodal data information to obtain multimodal feature information, including:

[0054] Keyword extraction is performed on text data to obtain multiple keywords; multiple keywords are combined to obtain multiple keyword combination features; image feature extraction is performed on image data to obtain target image features; and association relationships are established between each keyword combination feature and the target image feature to obtain multiple multimodal feature information, each of which includes a keyword combination feature and a target image feature.

[0055] In order to better supplement the knowledge graph, when extracting features from multimodal data information to obtain different multimodal feature information, it is necessary not only to extract features from the text part, but also to extract features from the image part to ensure that the knowledge extracted from the multimodal data information includes image features. The purpose of this method is that when updating the relationship branch in the knowledge graph, the image features can be integrated into the relationship branch, so that when the updated relationship branch is compared with the corresponding initial graph branch for difference, the comparison can be based on the image.

[0056] For ease of understanding, the examples are still given by the knowledge 1, knowledge 2 and knowledge 3 listed in the above embodiment. According to the above embodiment, knowledge 1 and knowledge 3 do not include the image data of Depay in the news, and only knowledge 2 includes the image data of Depay. In actual situations, the uniforms of the Dutch men's national team are orange, and the uniforms of the Argentine men's national team are blue and white striped images. Therefore, if the image features of Depay wearing an orange jersey can be extracted and incorporated into the Dutch branch at the same time as the other entity texts of knowledge 3, the features of Depay's orange jersey can match the orange jersey features of other Dutch men's players already in the Dutch branch. Depay wearing an orange jersey obviously does not match the features of the blue and white jersey in the Argentine branch, and the difference is greater. Therefore, when calculating the difference and considering the image features at the same time, the calculation result of the difference is more accurate. Therefore, after incorporating the image features into knowledge 1 and knowledge 3, knowledge 1 becomes: Argentina (entity)-player (relationship)-Depay (entity)-close-up image of kicking the ball (entity). Knowledge 3 becomes: Netherlands (entity) - player (relationship) - Depay (entity) - close-up image of kicking ball (entity). According to the above description, the process of incorporating Depay's image features into knowledge 1 and knowledge 3 is the process of establishing an association relationship, that is, the image features here are target image features, and the keywords Argentina (entity), player (relationship) and Depay (entity) in knowledge 1 are combined to form keyword combination features, and the same is true for knowledge 3. It should be noted that knowledge 1 and knowledge 3 in the embodiments of the present application do not constitute a limitation on keyword combination features.

[0057] Among them, extracting multiple keywords can be performed through a preset keyword extraction model. After extracting multiple keywords, the way of recombining keywords can refer to the way of triples in the above embodiment to obtain multiple keyword combinations, namely knowledge 1, knowledge 2 and knowledge 3 described in the above embodiment. Afterwards, when extracting image features, it can still be extracted through a preset image feature extraction model. For example, the graph convolutional network GCN (Graph Convolutional Network) is used for extraction. The specific GCN network is a prior art. The process of extracting image features through GCN is not described in detail in the embodiments of this application. It should be noted that the keyword extraction model and image feature extraction model provided in the embodiments of this application are both trained models. The specific training process can still refer to any prior art and will not be described here.

[0058] In order to better implement the embodiment of the present application, in one embodiment of the present application, the first knowledge graph information is supplemented using multimodal feature information to obtain target knowledge graph information, including:

[0059] Determine the labels to be supplemented for each multimodal feature information; obtain the entity labels of each initial graph branch in the first knowledge graph information; match each label to be supplemented with the entity label of each initial graph branch to determine the initial graph branch corresponding to each multimodal feature information in the first knowledge graph information; update each multimodal feature information to the corresponding initial graph branch to obtain the updated graph branch after each initial graph branch is updated; determine the first difference between each initial graph branch and the corresponding updated graph branch; according to each first difference, determine the target updated graph branch and the target multimodal feature information corresponding to the target updated graph branch.

[0060] In the above embodiment, when extracting keywords from knowledge 1, knowledge 2, and knowledge 3, the keywords that may be extracted may have label attributes. For example, words such as player, Argentina, and the Netherlands can correspond to branches in the graph. However, in actual situations, the extracted keywords may not have label attributes. In this case, it is necessary to add labels to the keywords to correspond to different initial graph branches. For example, an existing knowledge graph may be a cuisine classification, in which each branch includes one of the eight cuisines. At this time, if the extracted keyword is boiled pork slices, and boiled pork slices are Sichuan cuisine, a label of "Sichuan cuisine" can be added to the boiled pork slices. At the same time, if the extracted keyword is Buddha Jumps Over the Wall, since Buddha Jumps Over the Wall is Cantonese cuisine, a label of "Cantonese cuisine" can also be added to Buddha Jumps Over the Wall. Furthermore, when updating the corresponding branches, the knowledge including the keyword boiled pork slices, that is, the knowledge including the Sichuan cuisine label, can be added to the Sichuan cuisine branch; and the knowledge including the keyword Buddha Jumps Over the Wall, that is, the knowledge including the Cantonese cuisine label, can be added to the Cantonese cuisine branch.

[0061] Among them, the label of the keyword can be determined by a preset label determination model. After the keyword input value is input into the label determination model, the label determination model can output a label. In addition, when establishing a knowledge graph, labels can be set in advance for each branch of the knowledge graph manually. The labels set in the knowledge graph include the labels output by the label determination model. It should be noted that the label determination model is also a trained model, and the specific training method can also refer to the prior art, which will not be repeated here.

[0062] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining the first difference between each initial graph branch and the corresponding updated graph branch after the update includes:

[0063] For each initial atlas branch, feature extraction is performed on the initial atlas branch and the updated atlas branch after the initial atlas branch is updated to obtain a first branch feature of the initial atlas branch and a second branch feature of the updated atlas branch after the initial atlas branch is updated; based on the difference between the first branch feature and the second branch feature, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained.

[0064] The above embodiment provides a knowledge graph in which the features corresponding to each relationship branch are stored through a tensor matrix, and different knowledge or images are first converted into corresponding tensor features and spliced. However, in actual situations, when different knowledge is spliced ​​to different relationship branches, there is a situation where splicing is done first. Still using the example listed in the above embodiment to illustrate, if knowledge 3 is spliced ​​to the Netherlands branch, a new sub-branch can be established in the Netherlands branch first, and the image in the news can be placed in the sub-branch, and the corresponding keywords in knowledge 3 can be associated with the sub-branch, thus completing the update of the Netherlands branch.

[0065] If the branch is updated in this way, it is not possible to directly compare the difference between the updated branch and the branch before the update. Furthermore, the features of the initial graph branch, that is, the first branch features, can be determined before the branch is updated. Through the feature extraction model, the features of all images in an initial graph branch are extracted, and the same features are retained. For example, the same text features and the same image features, including the same keywords and the same blue and white striped jersey features. Afterwards, the updated branch features are extracted. It should be noted that the updated branch features can only extract the features of the newly established sub-branches. For example, if knowledge 1 is inserted into the Argentina branch, only the features corresponding to knowledge 1 can be extracted as the updated branch features, which can reduce the amount of calculation. At this time, feature extraction is performed on the sub-branch corresponding to knowledge 1, and relevant text features and image features, that is, the second branch features, can be extracted.

[0066] In addition, according to the above embodiment, when updating the relationship branch, the knowledge to be supplemented is converted into a tensor, and then the converted tensor is spliced ​​into the tensor matrix corresponding to the relationship branch. Therefore, when determining the difference between the initial graph branch and the updated relationship branch, the distance between the two tensor matrices of the updated relationship branch and the corresponding initial graph branch, such as the Euclidean distance, can be calculated as the first difference. It should be noted that determining the difference between the two tensor matrices can also include other methods, which are not specifically limited here. At the same time, the method of calculating the Euclidean distance can also refer to the prior art, and the specific details are not repeated here.

[0067] Based on this, for ease of understanding, we still use the above-mentioned knowledge 1, knowledge 2 and knowledge 3 as examples to illustrate that if knowledge 1 is used to update the initial Argentine branch, knowledge 2 is used to update the initial Dutch branch, and knowledge 3 is used to update the initial Dutch branch, we can get the updated Argentine branch, the updated first Dutch branch and the updated second Dutch branch. At this time, we determine the Euclidean distance between the updated Argentine branch and the initial Argentine branch, the Euclidean distance between the updated first Dutch branch and the initial Dutch branch, and the Euclidean distance between the updated second Dutch branch and the initial Dutch branch, so that we can get multiple first differences.

[0068] Afterwards, after obtaining the first difference, a minimum first difference can be determined. If it is a Euclidean distance, the minimum first difference is the minimum Euclidean distance. According to the above embodiment, knowledge 3 is the most effective entity knowledge because Depay is a member of the Dutch men's football team, and knowledge 3 is most relevant to Figure 3 The Dutch player branch in the knowledge graph matches. Therefore, the Euclidean distance corresponding to knowledge 3 must be the smallest. Therefore, knowledge 3 is the target multimodal feature information corresponding to the multiple knowledge to be supplemented. And the initial Dutch branch used for updating by knowledge 3 is also the target update graph branch in the multiple initial graph branches. In summary, it is based on knowledge 3 that multimodal data information is added to the knowledge graph. Specifically, Figure 4 , Figure 5 as well as Figure 6 As shown, Figure 4 shows a schematic diagram of inserting knowledge 1 into the Argentina branch, Figure 5 shows a schematic diagram of inserting knowledge 3 into the Dutch branch, Figure 6 It shows that the result of inserting knowledge 3 into the Netherlands branch takes precedence over the result of inserting knowledge 1 into the Argentina branch. Therefore, inserting knowledge 3 into the Netherlands branch can complete the first knowledge graph information and obtain the target knowledge graph information.

[0069] In order to better implement the embodiment of the present application, in one embodiment of the present application, according to the difference between the first branch feature and the second branch feature, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained, including:

[0070] Perform feature segmentation on the first branch feature to obtain a first text feature and a first image feature; perform feature segmentation on the second branch feature to obtain a second text feature and a second image feature; determine the text feature similarity between the first text feature in the first branch feature and the second text feature in the second branch feature; determine the image feature similarity between the first image feature in the first branch feature and the second image feature in the second branch feature; and obtain a first difference between an initial atlas branch and an updated atlas branch after the initial atlas branch is updated based on the text feature similarity and the image feature similarity.

[0071] The above embodiment provides a method of determining which specific knowledge is the correct knowledge by determining the difference between the updated branch features and the branch features before the update. In actual situations, since the knowledge in each knowledge to be supplemented may include both text knowledge and image knowledge. Therefore, after the knowledge is spliced ​​to the corresponding initial atlas branch, the similarity between the text features and the similarity between the image features will be involved. Therefore, when determining the difference between the updated relationship branch and the initial atlas branch before the update, it is necessary to compare the text features and image features of the two respectively, and the cross-comparison problem of image feature comparison and text feature comparison can also be avoided. Therefore, it is necessary to change the features corresponding to the initial atlas branch into the first text feature and the first image feature. And, the features corresponding to the updated relationship branch become the second text feature and the second image feature. It should be noted that in actual situations, the text features and image features are spliced ​​into a tensor matrix for storage, so when performing feature segmentation, the tensor matrix is ​​directly reversed according to the initial splicing method and divided into a text tensor matrix and an image tensor matrix.

[0072] After completing the feature segmentation, the first text feature and the second text feature can be subjected to matrix similarity calculation to obtain text feature similarity; the first image feature and the second image feature can be subjected to matrix similarity calculation to obtain image feature similarity. It should be noted that the similarity calculation can be performed by any existing technology, including determining the similarity between features by using the cosine similarity calculation method. The specific formula is as follows:

[0073]

[0074] Among them, A and B can both represent text feature or image feature matrices, and |||A||| and |||B||| both represent the range number of the matrix.

[0075] After the two similarities are determined, the average similarity of the two similarities can be calculated to obtain the final first difference. That is, the first difference = Avg (text feature similarity + image feature similarity). So far, each updated relationship branch and the initial graph branch before the update can obtain the corresponding first difference in this way. At this time, the similarity score is the first difference, and the first difference with the highest similarity score is the first difference with the smallest difference.

[0076] In order to better implement the embodiment of the present application, in one embodiment of the present application, according to the text feature similarity and the image feature similarity, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained, including:

[0077] Obtain a preset first weight corresponding to the text feature similarity and a preset second weight corresponding to the image feature similarity; perform weighted summation of the text feature similarity and the image feature similarity according to the preset first weight corresponding to the text feature similarity and the preset second weight corresponding to the image feature similarity to obtain a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated.

[0078] In the above embodiment, a method for determining the similarity score by calculating the arithmetic mean is provided. In order to meet different actual situations, a method for determining the similarity score by calculating the weighted average can also be provided in the embodiment of the present application. For example: the first difference = Avg (a × text feature similarity + b × image feature similarity), where a is the preset weight corresponding to the text feature similarity, b is the preset weight corresponding to the image feature similarity, and a+b=1.

[0079] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining the target update graph branch and the target multimodal feature information corresponding to the target update graph branch according to each first difference degree includes:

[0080] Determine the knowledge to be supplemented corresponding to the initial graph branch with the smallest first difference as the target multimodal feature information; fuse the target multimodal feature information to each initial graph branch respectively to obtain each fusion relationship branch; determine the second difference between each fusion relationship branch and the corresponding initial graph branch; determine the fusion relationship branch corresponding to the smallest second difference among the second differences as the target updated graph branch of the target multimodal feature information.

[0081] In order to ensure that the target multimodal feature information obtained is the most accurate, when determining the target multimodal feature information and the target update graph branch through the first difference degree, a knowledge to be supplemented corresponding to a multimodal data information can be preferentially selected, that is, a knowledge. If the above-mentioned examples are still used for explanation, according to the method of the above-mentioned embodiment, firstly through the first difference degree, first obtain knowledge 3 from knowledge 1, knowledge 2 and knowledge 3. The specific process is as shown in the above-mentioned embodiment, and will not be described in detail here. At this time, knowledge 3 corresponds to the initial target update graph branch and the target multimodal feature information in this embodiment.

[0082] After that, knowledge 3 is fused or spliced ​​into each branch in the knowledge graph. If combined with the above embodiment, when screening from each knowledge, knowledge 3 is only fused or spliced ​​into the Netherlands branch. In this embodiment, after obtaining knowledge 3, when fused into each branch, knowledge 3 is fused or spliced ​​into Figure 3 In each branch of Brazil, Argentina and the Netherlands, the updated branch and the corresponding branch before the update are calculated to verify whether the similarity of knowledge 3 added to the Netherlands branch is higher. Assuming that the similarity of fusion to the Brazilian branch is higher than the similarity of fusion to the Netherlands, it proves that there may be certain problems with the knowledge 3, such as the method of similarity calculation, or problems in keyword extraction or image feature extraction of multimodal data information, and problems need to be eliminated by manual determination. Of course, under normal circumstances, the similarity of knowledge 3 fused to the Netherlands branch must be higher than the similarity of knowledge 3 fused to Brazil.

[0083] In order to better implement the multimodal data processing method in the embodiment of the present application, in addition to the multimodal data processing method, the embodiment of the present application also provides a multimodal data processing device, such as Figure 7 As shown, the device 300 includes:

[0084] An acquisition module 301 acquires first knowledge graph information and multimodal data information;

[0085] An extraction module 302 is used to extract features from multimodal data information to obtain multimodal feature information;

[0086] The completion module 303 is used to complete the first knowledge graph information using multimodal feature information to obtain target knowledge graph information.

[0087] The multimodal data processing device provided in the present application determines each entity knowledge of the multimodal data information through the acquisition module 301 and the extraction module 302, and then inserts each entity knowledge into each branch of the knowledge graph through the completion module 303, and compares the differences between each new branch and the initial branch to determine that the entity knowledge with the least difference in impact on the knowledge graph is the correct knowledge, thereby improving the accuracy of knowledge insertion into the knowledge graph.

[0088] In some embodiments of the present application, the extraction module 302 is specifically used to:

[0089] Extract keywords from text data to obtain multiple keywords;

[0090] Combine multiple keywords to obtain multiple keyword combination features;

[0091] Perform image feature extraction on the image data to obtain target image features;

[0092] Establish association relationships between each keyword combination feature and the target image feature respectively, and obtain multiple multimodal feature information, each multimodal feature information includes a keyword combination feature and a target image feature.

[0093] In some embodiments of the present application, the completion module 303 is specifically used to:

[0094] Determine the labels to be supplemented for each multimodal feature information;

[0095] Obtain entity labels for each initial graph branch in the first knowledge graph information;

[0096] Match each to-be-supplemented label with the entity label of each initial graph branch to determine the initial graph branch corresponding to each multimodal feature information in the first knowledge graph information;

[0097] Updating each multimodal feature information to the corresponding initial atlas branch to obtain an updated atlas branch after each initial atlas branch is updated;

[0098] Determining a first difference between each initial graph branch and the corresponding updated graph branch after the update;

[0099] According to each first degree of difference, the first knowledge graph information is completed to obtain the target knowledge graph information.

[0100] In some embodiments of the present application, the completion module 303 is further used to:

[0101] For each initial atlas branch, feature extraction is performed on the initial atlas branch and the updated atlas branch after the initial atlas branch is updated, to obtain a first branch feature of the initial atlas branch and a second branch feature of the updated atlas branch after the initial atlas branch is updated;

[0102] According to the difference between the first branch feature and the second branch feature, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained.

[0103] In some embodiments of the present application, the completion module 303 is further used to:

[0104] Performing feature block division on the first branch feature to obtain a first text feature and a first image feature;

[0105] Performing feature block division on the second branch feature to obtain a second text feature and a second image feature;

[0106] Determine the text feature similarity between a first text feature in the first branch feature and a second text feature in the second branch feature;

[0107] Determine an image feature similarity between a first image feature in a first branch feature and a second image feature in a second branch feature;

[0108] According to the text feature similarity and the image feature similarity, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained.

[0109] In some embodiments of the present application, the completion module 303 is further used to:

[0110] Obtaining a preset first weight corresponding to the text feature similarity and a preset second weight corresponding to the image feature similarity;

[0111] According to the preset first weight corresponding to the text feature similarity and the preset second weight corresponding to the image feature similarity, the text feature similarity and the image feature similarity are weightedly summed to obtain the first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated.

[0112] In some embodiments of the present application, the completion module 303 is specifically used to:

[0113] Determine the multimodal feature information corresponding to the initial graph branch with the smallest first difference as the target multimodal feature information;

[0114] The target multimodal feature information is fused into each initial graph branch to obtain each fusion relationship branch;

[0115] Determining a second difference between each fusion relationship branch and the corresponding initial atlas branch;

[0116] Determine the fusion relationship branch corresponding to the minimum second difference degree among the second differences as the target update graph branch of the target multimodal feature information;

[0117] The multimodal data information is completed to the initial graph branch corresponding to the target updated graph branch to obtain the target knowledge graph information.

[0118] The present application also provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of any multimodal data processing method provided in the present application. The terminal device integrates any multimodal data processing method provided in the present application, such as Figure 8 As shown, it shows a schematic diagram of the structure of the terminal device involved in the embodiment of the present application, specifically:

[0119] The terminal device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Figure 8 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0120] The processor 401 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 executes various functions of the terminal device and processes data, thereby monitoring the terminal device as a whole. Optionally, the processor 401 may include one or more processing cores; the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0121] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0122] The terminal device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0123] The terminal device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0124] Although not shown, the terminal device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the terminal device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, such as:

[0125] Obtaining first knowledge graph information and multimodal data information;

[0126] Perform feature extraction on multimodal data information to obtain multimodal feature information;

[0127] The first knowledge graph information is completed using multimodal feature information to obtain the target knowledge graph information.

[0128] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0129] To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any multimodal data processing method provided in the embodiment of the present application. For example, the computer program is loaded by a processor to execute the following steps:

[0130] Obtaining first knowledge graph information and multimodal data information;

[0131] Perform feature extraction on multimodal data information to obtain multimodal feature information;

[0132] The first knowledge graph information is completed using multimodal feature information to obtain the target knowledge graph information.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0134] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0135] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0136] The above is a detailed introduction to a multimodal data processing method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal data processing method, characterized in that: The method comprises: Obtaining first knowledge graph information and multimodal data information; Performing feature extraction on the multimodal data information to obtain multimodal feature information; The multimodal feature information is used to complete the first knowledge graph information to obtain the target knowledge graph information.

2. The multimodal data processing method according to claim 1, characterized in that: The multimodal data information includes text data and image data, and the feature extraction of the multimodal data information to obtain multimodal feature information includes: Extracting keywords from the text data to obtain multiple keywords; Combining the multiple keywords to obtain multiple keyword combination features; Performing image feature extraction on the image data to obtain target image features; Establish association relationships between each of the keyword combination features and the target image features respectively to obtain a plurality of multimodal feature information, each of which includes a keyword combination feature and the target image feature.

3. The multimodal data processing method according to claim 1, characterized in that: The using the multimodal feature information to complete the first knowledge graph information to obtain target knowledge graph information includes: Determining a label to be supplemented for each of the multimodal feature information; Obtaining entity labels of each initial graph branch in the first knowledge graph information; Matching each of the to-be-supplemented labels with the entity labels of each of the initial graph branches, and determining the initial graph branch corresponding to each of the multimodal feature information in the first knowledge graph information; Updating each of the multimodal feature information to the corresponding initial atlas branch to obtain an updated atlas branch after each of the initial atlas branches is updated; Determining a first difference between each of the initial graph branches and the corresponding updated graph branch after the update; According to each of the first difference degrees, the first knowledge graph information is completed to obtain target knowledge graph information.

4. The multimodal data processing method according to claim 3, characterized in that: The determining of a first difference between each of the initial graph branches and the corresponding updated graph branch comprises: For each of the initial atlas branches, feature extraction is performed on the initial atlas branch and the updated atlas branch after the initial atlas branch is updated, to obtain a first branch feature of the initial atlas branch and a second branch feature of the updated atlas branch after the initial atlas branch is updated; According to the difference between the first branch feature and the second branch feature, a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated is obtained.

5. The multimodal data processing method according to claim 4, characterized in that: The obtaining, according to the difference between the first branch feature and the second branch feature, a first difference between the initial graph branch and an updated graph branch after the initial graph branch is updated, comprises: Performing feature block division on the first branch feature to obtain a first text feature and a first image feature; Performing feature block division on the second branch feature to obtain a second text feature and a second image feature; Determine a text feature similarity between a first text feature in the first branch feature and a second text feature in the second branch feature; Determine an image feature similarity between a first image feature in the first branch feature and a second image feature in the second branch feature; According to the text feature similarity and the image feature similarity, a first difference between the initial atlas branch and an updated atlas branch after the initial atlas branch is updated is obtained.

6. The multimodal data processing method according to claim 5, characterized in that: The obtaining, according to the text feature similarity and the image feature similarity, a first difference between the initial atlas branch and an updated atlas branch after the initial atlas branch is updated, comprises: Obtaining a preset first weight corresponding to the text feature similarity and a preset second weight corresponding to the image feature similarity; According to the preset first weight corresponding to the text feature similarity and the preset second weight corresponding to the image feature similarity, the text feature similarity and the image feature similarity are weightedly summed to obtain a first difference between the initial atlas branch and the updated atlas branch after the initial atlas branch is updated.

7. The multimodal data processing method according to claim 3, characterized in that: The step of completing the first knowledge graph information according to each of the first differences to obtain target knowledge graph information includes: Determine the multimodal feature information corresponding to the initial graph branch with the smallest first difference as the target multimodal feature information; The target multimodal feature information is respectively fused to each of the initial graph branches to obtain each fusion relationship branch; Determining a second difference between each of the fusion relationship branches and the corresponding initial atlas branch; Determine the fusion relationship branch corresponding to the minimum second difference degree among the second difference degrees as the target update graph branch of the target multimodal feature information; The multimodal data information is completed to the initial graph branch corresponding to the target updated graph branch to obtain the target knowledge graph information.

8. A multimodal data processing device, characterized in that: The device comprises: An acquisition module, used to acquire first knowledge graph information and multimodal data information; An extraction module, used for performing feature extraction on the multimodal data information to obtain multimodal feature information; A completion module is used to use the multimodal feature information to complete the first knowledge graph information to obtain target knowledge graph information.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the multimodal data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the multimodal data processing method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Rescue equipment configuration method and system of emergency rescue vehicle

    CN119578823A

  • Three-dimensional visualization method and device suitable for medical teaching, equipment and medium

    CN120070780A

  • Multi-source data fusion power knowledge processing method and device, medium and equipment

    CN121095008A