Multi-modal memory method and device based on knowledge graph, equipment and storage medium
By constructing a multimodal knowledge graph, integrating entities and their relationships from text and image data, the problem of ineffective fusion of multimodal data is solved, and the information processing and decision-making capabilities of intelligent agents are improved.
Patent Information
- Application Number
- CN202510530527.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-25
- Publication Date
- 2026-02-10
AI Technical Summary
In existing technologies, multimodal data has not been effectively fused, resulting in limited system performance and an inability to fully utilize the information in multimodal data.
By acquiring text and image data, extracting entities and their relationships, constructing a multimodal knowledge graph, and integrating data from different modalities to form a unified structured representation.
It enhances the intelligent agent's ability to understand scenarios and adapt to and make decisions in complex environments, and improves the understanding of user queries.
Smart Images
Figure CN121503601A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a multimodal memory method, apparatus, device, and storage medium based on knowledge graphs. Background Technology
[0002] With the rapid development of artificial intelligence technology, intelligent agents are playing an increasingly important role in handling complex tasks. These systems need to process and remember large amounts of information, including multimodal data such as text, images, and sound. However, most existing works generally only process single-modal data. Multimodal works often extract features from images and text separately using image encoders and text embedding modules, and then input the resulting image vectors and text vectors into a large model for label prediction. However, these works do not provide a unified representation of this multimodal data, resulting in ineffective fusion of multimodal data and limited system performance. Summary of the Invention
[0003] This disclosure provides at least one multimodal memory method, apparatus, device, and storage medium based on knowledge graphs to solve the aforementioned technical problem of limited system performance.
[0004] In a first aspect, embodiments of this disclosure provide a multimodal memory method based on knowledge graphs, including:
[0005] Acquire multimodal data, which includes text data and image data;
[0006] Extract text entities and text entity relationships from the text data, and extract image entities from the image data, and determine the image entity relationships of the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities;
[0007] Align the text entities and the image entities to obtain the image-text entity relationship; the image-text entity relationship is the correspondence between image entities and text entities that correspond to the same entity;
[0008] Based on the text entity relationships, the image entity relationships, and the image-text entity relationships, a multimodal knowledge graph of the memory module in the intelligent agent is constructed.
[0009] In one optional implementation, the extraction of text entities and text entity relationships from the text data, and the extraction of image entities from the image data, include:
[0010] The text data is processed by an entity relationship recognizer to obtain the text entities in the text data and the text entity relationships between the text entities;
[0011] The image data is processed by an image entity recognizer to obtain the image entities in the image data.
[0012] In one optional implementation, determining the image entity relationship of the image entities includes:
[0013] Determine a group of image entities located in the same image, and construct a first sub-entity relationship for any two image entities in the group of image entities;
[0014] Determine the similarity between any two image entities in the image entity, and construct a second sub-entity relationship between the image entities based on the similarity;
[0015] The image entity relationship is determined based on the first sub-entity relationship and the second sub-entity relationship.
[0016] In one optional implementation, constructing a first sub-entity relationship for any two image entities in the image entity group includes:
[0017] The entity relationship between any two image entities in the image entity group is defined as a first relationship; wherein, the first relationship is used to indicate that the two image entities are located in the same image;
[0018] Based on the first relationship and any two image entities, a first triplet is constructed to obtain the first sub-entity relationship.
[0019] In one optional implementation, constructing the second sub-entity relationship between the image entities based on the similarity includes:
[0020] Based on the similarity, the entity relationship between the pairs of image entities is determined to obtain a second relationship; the second relationship is used to indicate whether the pairs of image entities represent the same entity.
[0021] Based on the second relationship and the pairwise image entities, a second triplet is constructed to obtain the second sub-entity relationship.
[0022] In one optional implementation, determining the entity relationship between the pairwise image entities based on the similarity to obtain the second relationship includes:
[0023] If the similarity is determined to be within a first interval, the second relationship between the two pairs of image entities is determined to be a same relationship; wherein, the same relationship is used to indicate that the entities represented by the two pairs of image entities are the same;
[0024] If the similarity is determined to be within the second interval, the second relationship between the two pairs of image entities is determined to be a similarity relationship; wherein the similarity relationship is used to indicate that the entities represented by the two pairs of image entities are not the same, and the maximum boundary value of the second interval is less than or equal to the minimum boundary value of the first interval.
[0025] In one optional implementation, aligning the text entity and the image entity to obtain the image-text entity relationship includes:
[0026] The features of the text entities and the image entities are extracted using a multimodal model to obtain text entity features and image entity features;
[0027] Calculate the feature similarity between the text entity features and the image entity features;
[0028] If the feature similarity is greater than a preset threshold, the relationship between the text entity and the image entity is determined to be a text-image alignment relationship; wherein, the text-image alignment relationship is used to indicate that the text entity and the image entity correspond to the same entity;
[0029] The image-text entity relationship is constructed based on the image-text alignment relationship, the text entity, and the image entity.
[0030] In an optional implementation, after constructing a multimodal knowledge graph based on the text entity relationships, the image entity relationships, and the image-text entity relationships, the method further includes:
[0031] The text entity relationships, the image entity relationships, and the image-text entity relationships are converted into vectors to obtain the first relationship vector, the second relationship vector, and the third relationship vector, respectively.
[0032] According to the columnar storage format, the first relation vector, the second relation vector, the third relation vector, the text entity features of the text entity, and the image entity features of the image entity are stored in the specified directory of the graph database.
[0033] In one optional implementation, the method further includes:
[0034] After detecting a user's query request, the query request is converted into the query language of the graph database to obtain the target query language;
[0035] The target retrieval results are determined in the multimodal knowledge graph based on the target query language.
[0036] Based on the target retrieval results, the prompt words of the large language model are determined, and the prompt words and the query request are input into the large language model for processing to obtain the processing results.
[0037] Secondly, embodiments of this disclosure provide a multimodal memory method based on knowledge graphs, including:
[0038] An acquisition unit is used to acquire multimodal data, which includes text data and image data.
[0039] An extraction unit is used to extract text entities and text entity relationships between text entities from the text data, and to extract image entities from the image data and determine the image entity relationships between the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities.
[0040] An alignment unit is used to align the text entity and the image entity to obtain a text-image entity relationship; the text-image entity relationship is the correspondence between image entities and text entities that correspond to the same entity;
[0041] The construction unit is used to construct a multimodal knowledge graph of the memory module in the intelligent agent based on the text entity relationship, the image entity relationship and the image-text entity relationship.
[0042] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.
[0043] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.
[0044] This disclosure provides a multimodal memory method, apparatus, device, and storage medium based on a knowledge graph. In embodiments of this disclosure, firstly, multimodal data is acquired, including text data and image data; then, text entities and their relationships are extracted from the text data, and image entities are extracted from the image data, with the image entity relationships determined; wherein the image entity relationships indicate the degree of entity similarity between image entities; next, text entities and image entities are aligned to obtain image-text entity relationships; these relationships represent the correspondence between image entities and text entities that share the same entity; finally, based on the text entity relationships, image entity relationships, and image-text entity relationships, a multimodal knowledge graph for the memory module in the intelligent agent is constructed.
[0045] In the above embodiments, by integrating and processing data from different modalities, a memory mechanism based on a multimodal knowledge graph is constructed, providing the agent with a more comprehensive and in-depth scene understanding capability. This processing method can overcome the limitations of a single data type, fusing multimodal data into a unified, structured knowledge representation, thereby enhancing the agent's ability to understand user queries and its adaptability and decision-making capabilities in complex environments.
[0046] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0047] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.
[0048] Figure 1 A flowchart of a knowledge graph-based multimodal memory method provided in an embodiment of this disclosure is shown;
[0049] Figure 2 This diagram illustrates the structure of a multimodal knowledge graph provided in an embodiment of the present disclosure.
[0050] Figure 3 A flowchart of another knowledge graph-based multimodal memory method provided in an embodiment of this disclosure is shown;
[0051] Figure 4 A flowchart illustrating a method for querying a multimodal knowledge graph provided in an embodiment of this disclosure is shown.
[0052] Figure 5 This diagram illustrates a knowledge graph-based multimodal memory device provided in an embodiment of the present disclosure.
[0053] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0056] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0057] With the rapid development of artificial intelligence technology, intelligent agents are playing an increasingly important role in handling complex tasks. These systems need to process and remember large amounts of information, including multimodal data such as text, images, and sound. However, most existing works generally only process single-modal data. Multimodal works often extract features from images and text separately using image encoders and text embedding modules, and then input the resulting image vectors and text vectors into a large model for label prediction. However, these works do not provide a unified representation of this multimodal data, resulting in ineffective fusion of multimodal data and limited system performance.
[0058] Based on the above research, this disclosure provides a multimodal memory method, apparatus, device, and storage medium based on knowledge graphs. In embodiments of this disclosure, firstly, multimodal data is acquired, including text data and image data; then, text entities and text entity relationships between text entities are extracted from the text data, and image entities are extracted from the image data, and image entity relationships between image entities are determined; wherein, image entity relationships are used to indicate the degree of entity similarity between image entities; next, text entities and image entities are aligned to obtain image-text entity relationships; image-text entity relationships are the correspondence between image entities and text entities that correspond to the same entity; finally, based on text entity relationships, image entity relationships, and image-text entity relationships, a multimodal knowledge graph for the memory module in the intelligent agent is constructed.
[0059] In the above embodiments, by integrating and processing data from different modalities, a memory mechanism based on a multimodal knowledge graph is constructed, providing the agent with a more comprehensive and in-depth scene understanding capability. This processing method can overcome the limitations of a single data type, fusing multimodal data into a unified, structured knowledge representation, thereby enhancing the agent's ability to understand user queries and its adaptability and decision-making capabilities in complex environments.
[0060] To facilitate understanding of this embodiment, a detailed description of a knowledge graph-based multimodal memory method disclosed in this disclosure will be provided first. The execution entity of the knowledge graph-based multimodal memory method provided in this disclosure is generally an electronic device with a certain computing capability. In some possible implementations, this knowledge graph-based multimodal memory method can be implemented by a processor calling computer-readable instructions stored in memory.
[0061] See Figure 1 The diagram shows a flowchart of a knowledge graph-based multimodal memory method provided in this embodiment of the present disclosure. The method includes steps S101 to S104, wherein:
[0062] S101: Acquire multimodal data, which includes text data and image data.
[0063] Here, multimodal data refers to text data and image data input by the user to the agent. In this embodiment of the disclosure, if the agent's memory module does not contain a multimodal knowledge graph, and if it is detected that the user simultaneously inputs text data and image data to the agent, then the following steps S102 to S104 are triggered.
[0064] Here, text data can be obtained by converting voice data. For example, voice data input by the user to the agent can be acquired and converted into text data.
[0065] S102: Extract text entities from the text data and the text entity relationships between the text entities, and extract image entities from the image data, and determine the image entity relationships of the image entities; wherein the image entity relationships are used to indicate the positional relationships and / or similarity between image entities.
[0066] In this embodiment of the disclosure, both text entities and image entities can be understood as real-world objects or entities within text data. For example, a text entity or image entity can be the name of a building, a person, a region, or an object. The text entity is text-based data; for example, it could be "Beijing Forbidden City." The image entity can be image-based data.
[0067] Text entity relations refer to the association between any two text entities. For example, the association between the text entities "XX Person" and "YY Region" is: "XX Person's hometown is YY Region." Image entity relations indicate the entity category labels between image entities. For example, the entity category label indicates whether the image entities are located in the same image, or the similarity between any two image entities located in different images.
[0068] S103: Align the text entity and the image entity to obtain the image-text entity relationship; the image-text entity relationship is the correspondence between the image entity and the text entity that correspond to the same entity.
[0069] In this embodiment of the disclosure, by aligning text entities and image entities, text entities and image entities corresponding to the same entity can be identified, and the association relationship between the text entities and image entities corresponding to the same entity is marked as: text-image alignment, thereby obtaining the text-image entity relationship.
[0070] S104: Based on the text entity relationships, the image entity relationships, and the image-text entity relationships, construct a multimodal knowledge graph of the memory module in the intelligent agent.
[0071] A knowledge graph is a structured semantic knowledge base that stores entities and their relationships in the form of a graph. Entities are the basic nodes in a knowledge graph, representing a specific object or concept in the real world. These can be people, locations, or organizations, such as "Beijing," which is a location-type entity. Relationships are the edges in a knowledge graph, representing a connection or interaction between two entities. For example, in the statement "Beijing is the capital of China," there is a "capital" relationship between the entities "Beijing" and "China."
[0072] For the entity system, this disclosed technical solution constructs different relationship systems for different target scenarios. Regarding the relationship system, this disclosed technical solution identifies the following three types of relationships: relationships between text entities (i.e., text entity relationships); alignment relationships between text and image entities (i.e., text and image entity relationships); and relationships between image entities (i.e., image entity relationships). Then, a multimodal knowledge graph of the intelligent agent is constructed based on text entity relationships, text and image entity relationships, and image entity relationships.
[0073] In this embodiment of the disclosure, after constructing the multimodal indicator graph, if text data and / or image data input by the user to the agent are detected, the multimodal knowledge graph can be updated according to the text data processing flow and / or image data processing flow described above.
[0074] In this embodiment, the multimodal memory mechanism of an intelligent agent is represented using a multimodal knowledge graph. Constructing a multimodal knowledge graph is a complex process involving the extraction and fusion of information from various data sources. This disclosure considers both text and image modalities. The core objective of this process is to integrate cross-modal information into a unified graph structure, enabling a more comprehensive and richer representation of entities and relationships in the real world.
[0075] In this embodiment of the disclosure, step S102, which extracts text entities and text entity relationships from the text data and extracts image entities from the image data, specifically includes the following steps:
[0076] Step S11: Process the text data using an entity relationship recognizer to obtain the text entities in the text data and the text entity relationships between the text entities;
[0077] Step S12: Process the image data using an image entity recognizer to obtain the image entities in the image data.
[0078] After acquiring text and image data, entities and their relationships can be extracted from the text and image data. For extracting text entity relationships, this disclosed technical solution uses the pre-trained language large model GPT-4 as the entity relationship recognizer. Specifically, a piece of text data can be input into GPT-4, and GPT-4 will extract all text entities E from this text data according to a given format. t Relationship between text entities R t In the form of triples, that is in, These refer to the i-th and j-th text entities, respectively, where i is not equal to j; The term refers to the k-th text relation, representing the association between the i-th and j-th text entities.
[0079] For image entity recognition, the technical solution disclosed herein first collects a small amount of labeled data for the entity system of the current scene, including entity categories and the bounding box positions of entities in the image. This data is then used to train an image entity recognizer. Next, a new image is input into the image entity recognizer, which outputs the image entity E enclosed by a bounding box. v .
[0080] In this embodiment of the disclosure, step S102, which determines the image entity relationship of the image entities, specifically includes the following steps:
[0081] Step S21: Determine the image entity group located in the same image, and construct a first sub-entity relationship for any two image entities in the image entity group;
[0082] Step S22: Determine the similarity between any two image entities in the image entity, and construct a second sub-entity relationship between the image entities based on the similarity;
[0083] Step S23: Determine the image entity relationship based on the first sub-entity relationship and the second sub-entity relationship.
[0084] Next is the image relationship generation task. In this embodiment of the disclosure, corresponding entity relationships R can be generated between different image entities. v Regarding the relationships between image entities, we mainly focus on three types: "proximity," "identity," and "similarity," all of which are bidirectional relationships.
[0085] In practice, multiple image entities located in the same image can be identified and grouped into an image entity group, where one image corresponds to one image entity group. Next, a first sub-entity relationship, i.e., "adjacent," can be established between any two image entities in each image entity group.
[0086] For any two image entities located in different locations, the similarity between the two image entities can be calculated, and then the second sub-entity relationship between the two image entities can be determined based on the similarity. For example, the second sub-entity relationship can be: identical or similar.
[0087] After obtaining the first sub-entity relationship and the second sub-entity relationship, the image entity relationship can be determined based on the first sub-entity relationship and the second sub-entity relationship.
[0088] In this embodiment of the disclosure, step S21, which constructs a first sub-entity relationship between any two image entities in the image entity group, specifically includes the following steps:
[0089] First, the entity relationship between any two image entities in the image entity group is determined as the first relationship; wherein, the first relationship is used to indicate that the any two image entities are located in the same image;
[0090] Secondly, based on the first relationship and any two image entities, a first triplet is constructed to obtain the first sub-entity relationship.
[0091] In this embodiment of the disclosure, for image entities that appear in the same image, i.e., image entities, a "neighborhood" entity relationship (i.e., a first relationship) is generated for them, which will eventually form a triple (i.e., a first triple): And this triple is identified as the first child entity relation, where, r refers to the a-th image entity and the b-th image entity respectively. near ∈R v This represents the "proximity" relationship between the a-th image entity and the b-th image entity.
[0092] In this embodiment of the disclosure, step S22, which constructs a second sub-entity relationship between the image entities based on the similarity, specifically includes the following steps:
[0093] First, based on the similarity, the entity relationship between the pairs of image entities is determined to obtain a second relationship; the second relationship is used to indicate whether the pairs of image entities represent the same entity.
[0094] Secondly, based on the second relationship and the pairwise image entities, a second triplet is constructed to obtain the second sub-entity relationship.
[0095] In this embodiment of the disclosure, the features of image entities can be extracted using the CLIP model to obtain the vector representation E of the image entities. v Here, we can calculate the vectors of pairwise distinct image entities. and The similarity between them is calculated, where the similarity calculation function is cosine similarity:
[0096]
[0097] After calculating the similarity, the entity relationship between pairwise image entities can be determined based on the similarity, thus obtaining the second relationship. Then, based on the second relationship and the pairwise image entities, a second triplet can be constructed, thus obtaining the second sub-entity relationship. For example, the second triplet could be: Where r same∈R v represents the "same" relationship between image entities; alternatively, the second triple can be: where r similar ∈R v refers to the "similar" relationship between image entities.
[0098] In the embodiments of the present disclosure, the above steps determine the entity relationship between the pairwise image entities based on the similarity degree, and obtain the second relationship, which specifically includes the following steps:
[0099] When it is determined that the similarity degree is within the first interval, determine that the second relationship between the pairwise image entities is the same relationship; wherein, the same relationship is used to indicate that the entities represented by the pairwise image entities are the same;
[0100] When it is determined that the similarity degree is within the second interval, determine that the second relationship between the pairwise image entities is the similar relationship; wherein, the similar relationship is used to indicate that the entities represented by the pairwise image entities are not the same, and the maximum boundary value of the second interval is less than or equal to the minimum boundary value of the first interval.
[0101] In the embodiments of the present disclosure, when the similarity score is within the first interval of [M, 1] (0 < M < 1), determine that the second relationship between the two image entities is the "same" relationship. At this time, the second triple is: where r same ∈R v represents the "same" relationship between image entities; when the similarity degree is within the second interval of [N, M) (0 < N < M), determine that the second relationship between the two image entities is the "similar" relationship, and the corresponding second triple is: where r similar ∈R v refers to the "similar" relationship between image entities. In summary, the formula is expressed as follows, r v ∈R v is the relationship category label between images:
[0102]
[0103] In the embodiments of the present disclosure, the above step S103 aligns the text entity and the image entity to obtain the graphic-text entity relationship, which specifically includes the following steps:
[0104] Step S31: Extract the features of the text entity and the image entity through a multi-modal model to obtain the text entity features and the image entity features;
[0105] Step S32: Calculate the feature similarity between the text entity features and the image entity features;
[0106] Step S33: When the feature similarity is greater than a preset threshold, determine that the relationship between the text entity and the image entity is a text-image alignment relationship; wherein, the text-image alignment relationship is used to indicate that the text entity and the image entity correspond to the same entity;
[0107] Step S34: Construct the text-image entity relationship based on the text-image alignment relationship, the text entity, and the image entity.
[0108] In the embodiments of the present disclosure, first, the text entity E t and the image entity E v are input into the pre-trained CLIP large model, which can extract features for them respectively and generate corresponding vector representations E t (text entity feature) and E v (image entity feature). For the i-th text entity and the a-th image entity calculate the similarity score between their vectors using cosine similarity:
[0109]
[0110] When the similarity score score is greater than the preset threshold L (0 < L < 1), it is determined that the text entity and the image entity refer to entities in the same world, that is, there is a "text-image alignment" relationship between them, which is represented by a triple as: where, r align represents the "text-image alignment relationship", and this text-image alignment relationship is a two-way relationship.
[0111] Finally, the triple set formed by the text entity E t , the text entity relationship R t , the image entity E v , the image entity relationship R v and the text-image entity relationship r align can be jointly organized into a unified graph structure, that is, a multi-modal knowledge graph G. For example, a multi-modal knowledge graph as shown in Figure 2 can be obtained.
[0112] In this multi-modal knowledge graph, it contains nodes and edges. Among them, the nodes represent entities (that is, text entities and image entities), and the edges represent the relationships between entities. Figure 2 There are two types of nodes in , one is a text type node, and the other is an image type node. There is a "text-image alignment" relationship between the aligned text entity and the image entity, and there are corresponding text entity relationships and image entity relationships between text entities and between image entities. As Figure 2 The knowledge graph shown can not only store cross-modal information, but also support complex query and reasoning tasks, providing rich background knowledge and context for various applications.
[0113] In this embodiment of the disclosure, after constructing a multimodal knowledge graph based on the text entity relationships, the image entity relationships, and the image-text entity relationships, the method further includes the following steps:
[0114] Step S105: Convert the text entity relationship, the image entity relationship, and the image-text entity relationship into vectors to obtain the first relationship vector, the second relationship vector, and the third relationship vector, respectively;
[0115] Step S106: Store the first relation vector, the second relation vector, the third relation vector, the text entity features of the text entity, and the image entity features of the image entity in a designated directory of the graph database according to the columnar storage format.
[0116] After constructing a multimodal knowledge graph, it is necessary to store it. In traditional storage methods, the agent's memory module typically uses a vector database because it can efficiently retrieve text information related to the input query, thus supporting generation tasks. For the structured information of a knowledge graph, a graph database is a common choice. Graph databases can store complex networks of entities and relationships, support rich graph query languages, and facilitate complex graph queries and analyses.
[0117] To leverage the advantages of both vector databases and graph databases, this disclosed technical solution combines vector databases and graph databases, representing entities and relationships using vectors and storing them in the graph database. This mainly includes two steps: vector construction and storage.
[0118] In the vector construction phase, the technical solution disclosed herein will address all entities and relations (i.e., text entity E). t Text Entity Relations R t Image entity E v Image entity relationship R v Relationship between text and image entities r align Representation is done using vectors. For text entity E... t and image entity E v The corresponding characterization E has already been obtained in the CLIP step above. t (Text entity features) and E v (Image entity features). For text entity relations R... t Image entity relationship R v Relationship between text and image entities r alignStill selecting CLIP, generate the corresponding vector representation R for each. t R v ={r near r same r similar} and r align The first relation vector, the second relation vector, and the third relation vector are obtained respectively.
[0119] Subsequently, the constructed multimodal knowledge graph data can be persisted to the output directory and stored in Parquet format (i.e., columnar storage format). These files can be loaded into the memory and vector database during the query phase and used for retrieval. To more intuitively understand and analyze the constructed knowledge graph, the Parquet files are imported into the graph database Neo4j, allowing the knowledge graph to be viewed and analyzed through the visualization and management functions of the graph database.
[0120] In this embodiment of the disclosure, the method further includes the following steps:
[0121] Step S107: After detecting the user's query request, the query request is converted into the query language of the graph database to obtain the target query language;
[0122] Step S108: Determine the target retrieval result in the multimodal knowledge graph based on the target query language;
[0123] Step S109: Determine the prompt words of the large language model based on the target retrieval results, and input the prompt words and the query request into the large language model for processing to obtain the processing results.
[0124] In this embodiment of the disclosure, the retrieval of the multimodal memory mechanism is mainly divided into several steps: query parsing, retrieval, and result return.
[0125] In practical applications, when a user enters a query, the query must first be parsed. Since the technical solution disclosed herein stores the memory in the form of a multimodal knowledge graph in a graph database, it is necessary to first convert the user's query request into a graph query language similar to SQL. The technical solution disclosed herein chooses to use the Text2Cypher tool to convert the user query into Cypher query language, i.e., the target query language.
[0126] During the retrieval phase, the Cypher query language uses the MATCH statement to retrieve nodes and relationships in the graph database, uses pattern matching to locate data in the graph, retrieves the corresponding subgraph, and selects the top K search results (i.e., the target search results) that best match the query by setting the LIMIT statement.
[0127] For the top K retrieved results, the vectors are first mapped back to the original text or image data using the decoder of the large model. Then, the graph structure is converted into a serialized representation and returned. This representation is then used as a prompt and concatenated with the user query as input to the agent. By fusing the retrieved information with the current context, the agent can combine the retrieved information with the current state and target, better understand the user's query intent, enhance its ability to understand complex queries, provide more accurate responses, and further improve the agent's planning and decision-making capabilities.
[0128] In summary, multimodal memory mechanisms provide agents with powerful information processing and utilization capabilities, enabling them to better understand user queries and make better task plans and decisions in complex and ever-changing environments. By constructing and fusing multimodal knowledge graphs and designing effective update and retrieval mechanisms, the planning and decision-making capabilities of agents are significantly enhanced.
[0129] The following is combined with Figure 3 This paper introduces multimodal memory methods based on knowledge graphs, such as... Figure 3 As shown, the method includes the following steps:
[0130] S301: Acquire multimodal data;
[0131] S302: Recognition of text entities and relationships between text entities;
[0132] S303: Construct text triples based on text entities and text entity relationships;
[0133] Here, after constructing the entity system and relational hierarchy, a piece of text data can be input into GPT-4. GPT-4 will extract all text entities E from this text data according to a given format. t Relationship between text entities R t In the form of triples, that is
[0134] S304: Recognition of image entities;
[0135] Here, after constructing the entity system and relational framework, a new image data is input into the image entity recognizer, which outputs the image entity E enclosed by a bounding box. v and its entity category tags.
[0136] Before recognizing image entities, it is necessary to collect a small amount of labeled data for the entity system of the current scene, including entity categories and the bounding box positions of entities in the image. The image entity recognizer mentioned above is trained using this data.
[0137] S305: Determine the image entity relationship of the said image entity;
[0138] Here, multiple image entities located in the same image can be determined, and the multiple image entities are determined as an image entity group. Among them, one image corresponds to one image entity group. Next, a first sub-entity relationship, that is, "adjacent", can be constructed between any two image entities in each image entity group.
[0139] For any two image entities located in different images, the similarity between the two image entities can be calculated, and then a second sub-entity relationship between the image entities can be determined according to the similarity. For example, the second sub-entity relationship can be: the same or similar.
[0140] After obtaining the first sub-entity relationship and the second sub-entity relationship, the image entity relationship can be determined based on the first sub-entity relationship and the second sub-entity relationship.
[0141] S306: Form an image triple based on the image entity and the image entity relationship;
[0142] Here, the execution order of steps S302 to S303 and the execution order of steps S304 to S306 are not specifically limited. For example, steps S302 to S303 can be executed before steps S304 to S306, or steps S302 to S303 can be executed after steps S304 to S306 or simultaneously.
[0143] S307: Align the text and image entities to obtain the result of text and image entity alignment;
[0144] Here, the text entity E t and the image entity E v are input into the pre-trained CLIP large model. The CLIP large model can extract features for them respectively and generate corresponding vector representations E t (text entity feature) and E v (image entity feature). For the i-th text entity and the a-th image entity the similarity score between their vectors is calculated using cosine similarity:
[0145]
[0146] When the similarity score score is greater than the preset threshold L (0 < L < 1), it is considered that the text entity and the image entity refer to entities in the same world, that is, there is a "text and image alignment" relationship between them, which is represented by a triple as: where, r alignThis represents the "image and text alignment relationship", and this image and text alignment relationship is bidirectional.
[0147] S308: Construct a text-image triplet based on the text-image entity alignment result, text entities, and image entities;
[0148] S309: Construct a multimodal knowledge graph based on text triples, image triples, and image-text triples.
[0149] The following is combined with Figure 4 This section introduces knowledge graph-based query methods, such as... Figure 4 As shown, the method includes the following steps:
[0150] S401: Receive user query requests;
[0151] S402: Transform the query request into the query language of the graph database to obtain a query request in graph query language format;
[0152] S403: Determine the target retrieval results in the graph database that match the query request in the graph query language format;
[0153] S404: Determine the prompt words for the large language model based on the target retrieval results, and input the prompt words and query request into the large language model for processing to obtain the processing results;
[0154] S405: Provide feedback on the processing result to the user.
[0155] As described above, the technical solution disclosed herein utilizes a multimodal knowledge graph based on text and images to represent the agent's memory. It constructs triples through entity relation extraction, text-image entity alignment, and image relation generation, forming a complete and unified knowledge graph based on these triples. Furthermore, this technical solution considers three types of relationships: relationships between text entities, alignment relationships between text and image entities, and relationships between image entities. It generates "proximity," "identity," and "similarity" relationships between image entities based on whether they are in the same image and the similarity between image entities. This technical solution extracts entities from both text and images, inputs them into the CLIP model, outputs the text-image entity alignment results, and constructs triples containing the "text-image alignment" relationship to connect and map text and image entities.
[0156] Compared to existing technologies, the memory module of this disclosure provides a richer and more powerful knowledge representation and reasoning framework by combining symbolic knowledge and structured graph representation through knowledge graph representation methods, which helps solve complex problems in user queries. This disclosure provides rich background knowledge and contextual information for different modalities by uniformly representing images and text using a knowledge graph, enabling better alignment and fusion of images and text, thereby improving the agent's understanding of images and text. Through vector-based retrieval and graph-based query language, this disclosure can efficiently mine potential and implicit connections between entities in the knowledge graph, retrieving richer and deeper memory information for user queries.
[0157] In summary, the knowledge graph-based multimodal memory method proposed in this disclosure helps intelligent agents process and understand information from different modalities, thereby making more accurate decisions and inferences in complex environments. This capability is of significant value for applications requiring the processing of multimodal data (such as customer service, medical diagnosis, and security monitoring). Furthermore, through the memory mechanism, intelligent agents can better understand user intentions and preferences, providing more personalized and coherent services. This has a direct impact on improving user experience and satisfaction. In the long term, the research and application of multimodal memory mechanisms may spur new technologies and products, such as more advanced robots and more intelligent virtual assistants.
[0158] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0159] Based on the same inventive concept, this disclosure also provides a knowledge graph-based multimodal memory device corresponding to the knowledge graph-based multimodal memory method. Since the principle of the device in this disclosure for solving the problem is similar to the knowledge graph-based multimodal memory method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0160] Reference Figure 5 The diagram shown is a schematic of a knowledge graph-based multimodal memory device provided in an embodiment of this disclosure. The device includes: an acquisition unit 10, an extraction unit 20, an alignment unit 30, and a construction unit 40; wherein,
[0161] An acquisition unit is used to acquire multimodal data, which includes text data and image data.
[0162] An extraction unit is used to extract text entities and text entity relationships between text entities from the text data, and to extract image entities from the image data and determine the image entity relationships between the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities.
[0163] An alignment unit is used to align the text entity and the image entity to obtain a text-image entity relationship; the text-image entity relationship is the correspondence between image entities and text entities that correspond to the same entity;
[0164] The construction unit is used to construct a multimodal knowledge graph of the memory module in the intelligent agent based on the text entity relationship, the image entity relationship and the image-text entity relationship.
[0165] In one possible implementation, the extraction unit is further configured to: process the text data using an entity relationship recognizer to obtain text entities in the text data and text entity relationships between the text entities; and process the image data using an image entity recognizer to obtain image entities in the image data.
[0166] In one possible implementation, the extraction unit is further configured to: determine a group of image entities located in the same image, and construct a first sub-entity relationship for any two image entities in the group of image entities;
[0167] Determine the similarity between any two image entities in the image entity, and construct a second sub-entity relationship between the image entities based on the similarity; determine the image entity relationship based on the first sub-entity relationship and the second sub-entity relationship.
[0168] In one possible implementation, the extraction unit is further configured to: determine the entity relationship between any two image entities in the image entity group as a first relationship; wherein the first relationship is used to indicate that the any two image entities are located in the same image; and construct a first triplet based on the first relationship and the any two image entities to obtain the first sub-entity relationship.
[0169] In one possible implementation, the extraction unit is further configured to: determine the entity relationship between the pair of image entities based on the similarity to obtain a second relationship; the second relationship is used to indicate whether the pair of image entities represent the same entity; and construct a second triplet based on the second relationship and the pair of image entities to obtain the second sub-entity relationship.
[0170] In one possible implementation, the extraction unit is further configured to: determine that the second relationship between the pair of image entities is a same relationship when the similarity is determined to be in a first interval; wherein the same relationship is used to indicate that the entities represented by the pair of image entities are the same; and determine that the second relationship between the pair of image entities is similar when the similarity is determined to be in a second interval; wherein the similar relationship is used to indicate that the entities represented by the pair of image entities are not the same, and the maximum boundary value of the second interval is less than or equal to the minimum boundary value of the first interval.
[0171] In one possible implementation, the alignment unit is further configured to: extract features of the text entity and the image entity using a multimodal model to obtain text entity features and image entity features; calculate the feature similarity between the text entity features and the image entity features; if the feature similarity is greater than a preset threshold, determine that the relationship between the text entity and the image entity is a text-image alignment relationship; wherein the text-image alignment relationship is used to indicate that the text entity and the image entity correspond to the same entity; and construct the text-image entity relationship based on the text-image alignment relationship, the text entity, and the image entity.
[0172] In one possible implementation, the device is further configured to: after constructing a multimodal knowledge graph based on the text entity relations, the image entity relations, and the text-image entity relations, convert the text entity relations, the image entity relations, and the text-image entity relations into vectors to obtain a first relation vector, a second relation vector, and a third relation vector, respectively; and store the first relation vector, the second relation vector, the third relation vector, the text entity features of the text entities, and the image entity features of the image entities in a columnar storage format to a specified directory of the graph database.
[0173] In one possible implementation, the device is further configured to: after detecting a user's query request, convert the query request into a query language of a graph database to obtain a target query language; determine a target retrieval result in the multimodal knowledge graph based on the target query language; determine prompt words for a large language model based on the target retrieval result, and input the prompt words and the query request into the large language model for processing to obtain a processing result.
[0174] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0175] Corresponding to Figure 1 In addition to the knowledge graph-based multimodal memory method, this disclosure also provides an electronic device 600, such as... Figure 6The diagram shown is a structural schematic of an electronic device 600 provided in an embodiment of this disclosure, including:
[0176] The system includes a processor 61, a memory 62, and a bus 63. The memory 62 stores execution instructions and includes main memory 621 and external memory 622. The main memory 621, also called internal memory, temporarily stores the computational data in the processor 61, as well as data exchanged with external memory such as a hard disk. The processor 61 exchanges data with the external memory 622 through the main memory 621. When the electronic device 600 is running, the processor 61 communicates with the memory 62 through the bus 63, causing the processor 61 to execute the following instructions:
[0177] Acquire multimodal data, which includes text data and image data;
[0178] Extract text entities and text entity relationships from the text data, and extract image entities from the image data, and determine the image entity relationships of the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities;
[0179] Align the text entities and the image entities to obtain the image-text entity relationship; the image-text entity relationship is the correspondence between image entities and text entities that correspond to the same entity;
[0180] Based on the text entity relationships, the image entity relationships, and the image-text entity relationships, a multimodal knowledge graph of the memory module in the intelligent agent is constructed.
[0181] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the knowledge graph-based multimodal memory method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0182] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the knowledge graph-based multimodal memory method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0183] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0184] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0186] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0187] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0188] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.
Claims
1. A multimodal memory method based on knowledge graphs, characterized in that, include: Acquire multimodal data, which includes text data and image data; Extract text entities and text entity relationships from the text data, and extract image entities from the image data, and determine the image entity relationships of the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities; Align the text entities and the image entities to obtain the image-text entity relationship; the image-text entity relationship is the correspondence between image entities and text entities that correspond to the same entity; Based on the text entity relationships, the image entity relationships, and the image-text entity relationships, a multimodal knowledge graph of the memory module in the intelligent agent is constructed.
2. The method according to claim 1, characterized in that, The step of extracting text entities and text entity relationships from the text data, and extracting image entities from the image data, includes: The text data is processed by an entity relationship recognizer to obtain the text entities in the text data and the text entity relationships between the text entities; The image data is processed by an image entity recognizer to obtain the image entities in the image data.
3. The method according to claim 1, characterized in that, Determining the image entity relationships of the image entities includes: Determine a group of image entities located in the same image, and construct a first sub-entity relationship for any two image entities in the group of image entities; Determine the similarity between any two image entities in the image entity, and construct a second sub-entity relationship between the image entities based on the similarity; The image entity relationship is determined based on the first sub-entity relationship and the second sub-entity relationship.
4. The method according to claim 3, characterized in that, The step of constructing a first sub-entity relationship for any two image entities in the image entity group includes: The entity relationship between any two image entities in the image entity group is defined as a first relationship; wherein, the first relationship is used to indicate that the two image entities are located in the same image; Based on the first relationship and any two image entities, a first triplet is constructed to obtain the first sub-entity relationship.
5. The method according to claim 3, characterized in that, The construction of the second sub-entity relationship between the image entities based on the similarity includes: Based on the similarity, the entity relationship between the pairs of image entities is determined to obtain a second relationship; the second relationship is used to indicate whether the pairs of image entities represent the same entity. Based on the second relationship and the pairwise image entities, a second triplet is constructed to obtain the second sub-entity relationship.
6. The method according to claim 5, characterized in that, The step of determining the entity relationship between each pair of image entities based on the similarity to obtain the second relationship includes: If the similarity is determined to be within a first interval, the second relationship between the two pairs of image entities is determined to be a same relationship; wherein, the same relationship is used to indicate that the entities represented by the two pairs of image entities are the same; If the similarity is determined to be within the second interval, the second relationship between the two pairs of image entities is determined to be a similarity relationship; wherein the similarity relationship is used to indicate that the entities represented by the two pairs of image entities are not the same, and the maximum boundary value of the second interval is less than or equal to the minimum boundary value of the first interval.
7. The method according to claim 1, characterized in that, The step of aligning the text entities and the image entities to obtain the image-text entity relationship includes: The features of the text entities and the image entities are extracted using a multimodal model to obtain text entity features and image entity features; Calculate the feature similarity between the text entity features and the image entity features; If the feature similarity is greater than a preset threshold, the relationship between the text entity and the image entity is determined to be a text-image alignment relationship; wherein, the text-image alignment relationship is used to indicate that the text entity and the image entity correspond to the same entity; The image-text entity relationship is constructed based on the image-text alignment relationship, the text entity, and the image entity.
8. The method according to claim 1, characterized in that, After constructing a multimodal knowledge graph based on the text entity relationships, the image entity relationships, and the image-text entity relationships, the method further includes: The text entity relationships, the image entity relationships, and the image-text entity relationships are converted into vectors to obtain the first relationship vector, the second relationship vector, and the third relationship vector, respectively. According to the columnar storage format, the first relation vector, the second relation vector, the third relation vector, the text entity features of the text entity, and the image entity features of the image entity are stored in the specified directory of the graph database.
9. The method according to claim 8, characterized in that, The method further includes: After detecting a user's query request, the query request is converted into the query language of the graph database to obtain the target query language; The target retrieval results are determined in the multimodal knowledge graph based on the target query language. Based on the target retrieval results, the prompt words of the large language model are determined, and the prompt words and the query request are input into the large language model for processing to obtain the processing results.
10. A multimodal memory method based on knowledge graphs, characterized in that, include: An acquisition unit is used to acquire multimodal data, which includes text data and image data. An extraction unit is used to extract text entities and text entity relationships between text entities from the text data, and to extract image entities from the image data and determine the image entity relationships between the image entities; the image entity relationships are used to indicate the positional relationships and / or similarity between image entities. An alignment unit is used to align the text entity and the image entity to obtain a text-image entity relationship; the text-image entity relationship is the correspondence between image entities and text entities that correspond to the same entity; The construction unit is used to construct a multimodal knowledge graph of the memory module in the intelligent agent based on the text entity relationship, the image entity relationship and the image-text entity relationship.
11. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the knowledge graph-based multimodal memory method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the knowledge graph-based multimodal memory method as described in any one of claims 1 to 9.
13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the knowledge graph-based multimodal memory method as described in any one of claims 1 to 9.