Active reasoning method and device based on knowledge graph, equipment and medium

Abstract

The invention relates to an active reasoning method and device based on a knowledge graph, equipment and a medium, and belongs to the technical field of multi-modal large models, and the active reasoning method based on the knowledge graph comprises the steps: obtaining an image and a problem inputted into a multi-modal large model; keywords in the question are extracted, and associated entities related to the keywords and attributes of the associated entities are retrieved in a knowledge graph; classifying the associated entities according to the attributes of the associated entities; detecting and identifying characters and / or characters in the image according to a classification result; judging whether the identification result is related to the keyword or not, and if yes, supplementing the identification result into the question; otherwise, re-retrieving the associated entity in the knowledge graph; and generating an answer according to the supplemented question and the image. According to the method and the device, the answer with higher accuracy is generated, and the possibility of illusion is reduced.

Description

Technical Field

[0001] This invention belongs to the field of multimodal large model technology, and specifically relates to an active reasoning method, device, equipment and medium based on knowledge graph. Background Technology

[0002] Multimodal large models refer to artificial intelligence models capable of processing and understanding data from multiple different modalities. These models typically have deep learning architectures and can extract features from data of different modalities and fuse these features together to perform specific tasks, such as classification, recognition, and generation. Multimodal large models have a wide range of applications, such as image description and generation, where models can generate text describing the content of an image or generate an image based on a text description; visual question answering, where models can understand the content of an image and answer questions related to the image; video understanding, where models can analyze the visual content and audio in a video to perform tasks such as action recognition and event detection; and robot navigation and interaction, where models can help robots understand their visual environment and interact with humans in a natural way.

[0003] However, the limitations of existing multimodal large models are also obvious. When a large model cannot determine whether it is answering a question correctly, this situation is called an illusion. This illusion refers to the phenomenon that when generating text or answering questions, the model fabricates information that is not true, inaccurate, or inconsistent with the facts. The illusion may be because the model has learned pattern matching and probabilistic inference during training, rather than based on accurate memory of facts. Therefore, when the model is faced with unfamiliar or ambiguous input, it may generate some seemingly reasonable but actually fictitious content, resulting in low accuracy and a tendency to produce illusions. Summary of the Invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a knowledge graph-based active reasoning method, apparatus, device and medium to solve one or more of the above-mentioned problems existing in the prior art.

[0005] The objective of this invention is achieved as follows:

[0006] A first aspect of the present invention provides an active reasoning method based on a knowledge graph, comprising:

[0007] Obtain the image and problem of the input multimodal large model;

[0008] Extract keywords from the question, and retrieve related entities and attributes of the related entities from the knowledge graph;

[0009] The associated entities are classified according to their attributes;

[0010] Based on the classification results, the people and / or text in the image are detected and identified;

[0011] Determine whether the identification result is related to the keyword. If it is, add the identification result to the question; otherwise, re-search for related entities in the knowledge graph.

[0012] Generate an answer based on the supplemented question and image.

[0013] Furthermore, the extraction of keywords from the question includes: using natural language processing technology to segment and identify entities in the input question in order to extract the keywords.

[0014] Furthermore, the step of retrieving related entities and attributes in the knowledge graph that are associated with the keyword includes: searching for entities corresponding to the keyword in the knowledge graph; performing a search based on the entity relationships of the entities corresponding to the keyword, and setting entities in the knowledge graph whose correlation with the entities corresponding to the keyword is greater than a preset threshold as related entities; and outputting the related entities and their attributes.

[0015] Furthermore, classifying the associated entities based on their attributes includes: inputting the associated entities and their attributes into a pre-trained classifier, determining the category of the associated entities based on their attributes, and outputting the classification result.

[0016] Furthermore, the step of detecting and recognizing people and / or text in the image based on the classification result includes: if the classification result is a person category, using a target recognition method to detect and recognize people in the image, and outputting the person's name and corresponding confidence score; using a character recognition method to detect and recognize text in the image, and outputting the text content and corresponding confidence score.

[0017] Furthermore, the step of generating an answer based on the supplemented question and image includes: encoding the supplemented question using a text encoder to obtain a question encoding vector; encoding the image using an image encoder to obtain an image encoding vector; fusing the question encoding vector and the image encoding vector to obtain a comprehensive encoding vector; and generating an answer based on the comprehensive encoding vector using a pre-trained large language model.

[0018] Furthermore, it also includes: extracting entities and their relationships from the answer, and using knowledge graph retrieval to verify whether the extracted entities and their relationships are correct; if correct, the answer is output as the final answer; otherwise, the answer is input into a large language model for correction.

[0019] A second aspect of the present invention provides an active reasoning device based on a knowledge graph, comprising:

[0020] The acquisition module is used to acquire the image and question of the input multimodal large model;

[0021] The retrieval module is used to extract keywords from the question and retrieve related entities and attributes of the related entities in the knowledge graph.

[0022] A classification module is used to classify the associated entities according to their attributes;

[0023] The detection and recognition module is used to detect and recognize people and / or text in the image based on the classification results;

[0024] The judgment module is used to determine whether the recognition result is related to the keyword. If it is related, the recognition result is added to the question; otherwise, the related entity is retrieved again from the knowledge graph.

[0025] The generation module is used to generate answers based on the supplemented questions and images.

[0026] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the knowledge graph-based active reasoning method described in any embodiment.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the knowledge graph-based active reasoning method described in any embodiment.

[0028] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0029] The proactive reasoning method based on knowledge graphs provided by this invention introduces knowledge graphs into a multimodal large model to proactively reason about user-input questions. By retrieving related entities that are relevant to the keywords in the input question, and then detecting and recognizing the image based on the related entities, the recognition results are used as clues to supplement the question, thereby generating more accurate answers and reducing the possibility of hallucinations. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1A flowchart of the knowledge graph-based active reasoning method provided in Embodiment 1 of the present invention;

[0032] Figure 2 This is a schematic diagram of the knowledge graph-based active reasoning device provided in Embodiment 2 of the present invention;

[0033] Figure 3 This is a schematic diagram of the electronic device architecture provided in Embodiment 3 of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be noted that, unless otherwise specified, the implementation methods and features in the implementation methods in this disclosure can be combined, separated, interchanged, and / or rearranged. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1

[0036] A specific embodiment of the present invention, such as Figure 1 As shown, a knowledge graph-based active reasoning method is disclosed, including the following steps:

[0037] S1. Obtain the image and problem of the input multimodal large model;

[0038] For example, a user uploads an image and enters the question "Which elements in this image are related to 'Dream of the Red Chamber'?" in a multimodal large model.

[0039] S2. Extract keywords from the question and retrieve related entities and attributes of the related entities in the knowledge graph;

[0040] In this embodiment, step S2 specifically includes the following steps:

[0041] S201. Use natural language processing technology to segment the input question into words and recognize entities in order to extract the keywords;

[0042] S202. Search for the entity corresponding to the keyword in the knowledge graph;

[0043] S203. Based on the entity relationship of the entity corresponding to the keyword, perform a search and set the entity in the knowledge graph whose correlation with the entity corresponding to the keyword is greater than a preset threshold as the associated entity.

[0044] S204. Output the associated entity and its attributes.

[0045] Taking the question in step S1 as an example, the keyword "Dream of the Red Chamber" is extracted from the question. The entity of Dream of the Red Chamber is found in the knowledge graph. Then, the correlation between entities that have an entity relationship with Dream of the Red Chamber and the entity of Dream of the Red Chamber is calculated. If the correlation is greater than a preset threshold, the entity is set as an associated entity. For example, associated entities include Lin Daiyu, Jia Baoyu, Grand View Garden, the four major families, "The Burial of Flowers", etc. The associated entities and their respective attributes are output.

[0046] S3. Classify the associated entities according to their attributes;

[0047] In this embodiment, step S3 is specifically implemented as follows: inputting the associated entity and its attributes into a pre-trained classifier, determining the category of the associated entity based on its attributes, and outputting the classification result.

[0048] For example, by collecting a dataset containing a large number of entities and their attributes, labeling it, and then inputting it into a classifier for training, the classifier can determine whether the associated entity belongs to the category of people based on the attributes of the associated entity, and otherwise belongs to the category of text.

[0049] S4. Detect and identify people and / or text in the image based on the classification results;

[0050] In this embodiment, step S4 specifically includes:

[0051] If the classification result is a person category, use the object recognition method to detect and identify the person in the image, and output the person's name and the corresponding confidence score;

[0052] Use character recognition methods to detect and recognize text in images, and output the text content and corresponding confidence scores.

[0053] Specifically, after classifying the associated entities, if the classification result is a person, the coordinate region of the person in the image is detected by the target recognition method, and the specific person's name and corresponding confidence score are identified; if the classification result is not a person, the coordinate region of the text in the image is detected by the character recognition method, and the text content and corresponding confidence score are identified.

[0054] For example, step S4 identifies the person in the image as Lin Daiyu, played by Chen Xiaoxu in the TV series "Dream of the Red Chamber", and the text in the image is "Flowers fall and fly, filling the sky; red fades and fragrance disappears, who will pity them?", which comes from "The Burial of Flowers".

[0055] S5. Determine whether the identification result is related to the keyword. If it is related, add the identification result to the question; otherwise, re-search for related entities in the knowledge graph.

[0056] In this embodiment, after obtaining the recognition result in step S4, the degree of correlation between the recognition result and the keywords extracted from the input question is calculated. If the degree of correlation is greater than a certain preset threshold, the recognition result is considered to be related to the keywords. In order for the multimodal large model to generate more accurate answers based on more clues, the recognition result is added to the input question.

[0057] For example, the supplementary question could be, "What elements in this picture are related to 'Dream of the Red Chamber'? Including tasks, text, etc. Known clues include that the person in the picture is Lin Daiyu, played by Chen Xiaoxu in the TV series 'Dream of the Red Chamber', and the text in the picture is 'Flowers fall and fly, filling the sky; red fades and fragrance dies, who will pity them?', which is from 'The Burial of Flowers'."

[0058] S6. Generate answers based on the supplemented questions and images.

[0059] In this embodiment, step S6 specifically includes:

[0060] S601. Use a text encoder to encode the supplemented question to obtain the question encoding vector;

[0061] S602. Encode the image using an image encoder to obtain an image encoding vector;

[0062] S603. Merge the problem encoding vector and the image encoding vector to obtain a comprehensive encoding vector;

[0063] S604. Generate an answer using a pre-trained large language model based on the comprehensive encoding vector.

[0064] Compared with existing technologies, the knowledge graph-based active reasoning method provided in this embodiment introduces knowledge graphs into a multimodal large model to actively reason about user-input questions. By retrieving related entities that are relevant to the keywords in the input question, and then detecting and recognizing the image based on the related entities, the recognition results are used as clues to supplement the question, thereby generating more accurate answers and reducing the possibility of hallucinations.

[0065] In some embodiments, the knowledge graph-based active reasoning method further includes the following steps:

[0066] S7. Extract the entities and their relationships from the answer, and use knowledge graph retrieval to verify whether the extracted entities and their relationships are correct; if correct, output the final answer; otherwise, input the answer into the large language model for correction.

[0067] This step can verify the correctness of the relationships between entities in the generated answers using a knowledge graph, further reducing the possibility of hallucinations.

[0068] Example 2

[0069] This embodiment provides an active reasoning device based on a knowledge graph, such as... Figure 2 As shown, it includes:

[0070] The acquisition module is used to acquire the image and question of the input multimodal large model;

[0071] The retrieval module is used to extract keywords from the question and retrieve related entities and attributes of the related entities in the knowledge graph.

[0072] A classification module is used to classify the associated entities according to their attributes;

[0073] The detection and recognition module is used to detect and recognize people and / or text in the image based on the classification results;

[0074] The judgment module is used to determine whether the recognition result is related to the keyword. If it is related, the recognition result is added to the question; otherwise, the related entity is retrieved again from the knowledge graph.

[0075] The generation module is used to generate answers based on the supplemented questions and images.

[0076] Example 3

[0077] This embodiment provides an electronic device, such as... Figure 3 As shown, it includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the knowledge graph-based active reasoning method as described in any of the above embodiments.

[0078] Example 4

[0079] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the knowledge graph-based active reasoning method as described in any of the above embodiments.

[0080] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0081] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0082] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0083] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge graph-based active reasoning method, characterized in that, include: Obtain the image and problem of the input multimodal large model; Extract keywords from the question, and retrieve related entities and attributes of the related entities from the knowledge graph; The associated entities are classified according to their attributes; Based on the classification results, the people and / or text in the image are detected and identified; Determine whether the recognition result is related to the keyword; if so, add the recognition result to the question. Otherwise, retrieve the related entities again from the knowledge graph; Generate an answer based on the supplemented question and image.

2. The knowledge graph-based active reasoning method according to claim 1, characterized in that, The step of extracting keywords from the question includes: using natural language processing technology to segment and identify entities from the input question in order to extract the keywords.

3. The knowledge graph-based active reasoning method according to claim 1, characterized in that, The step of retrieving related entities and attributes in the knowledge graph that are associated with the keyword includes: Search for the entities corresponding to the keywords in the knowledge graph; Based on the entity relationship of the entity corresponding to the keyword, the entity in the knowledge graph with a correlation degree greater than a preset threshold is set as the associated entity; Output the associated entity and its attributes.

4. The knowledge graph-based active reasoning method according to claim 3, characterized in that, The step of classifying the associated entities based on their attributes includes: The associated entity and its attributes are input into a pre-trained classifier. The classifier determines the category of the associated entity based on its attributes and outputs the classification result.

5. The knowledge graph-based active reasoning method according to claim 1, characterized in that, The detection and recognition of people and / or text in the image based on the classification results includes: If the classification result is a person category, use the object recognition method to detect and identify the person in the image, and output the person's name and the corresponding confidence score; Use character recognition methods to detect and recognize text in images, and output the text content and corresponding confidence scores.

6. The knowledge graph-based active reasoning method according to claim 1, characterized in that, The step of generating an answer based on the supplemented question and image includes: The supplemented question is encoded using a text encoder to obtain a question encoding vector; The image is encoded using an image encoder to obtain an image encoding vector; By fusing the problem encoding vector and the image encoding vector, a comprehensive encoding vector is obtained; The answer is generated using a pre-trained large language model based on the comprehensive encoding vector.

7. The knowledge graph-based active reasoning method according to any one of claims 1-6, characterized in that, Also includes: Extract the entities and their relationships from the answers, and use knowledge graph retrieval to verify whether the extracted entities and their relationships are correct; If correct, the output is the final answer; otherwise, the answer is input into the large language model for correction.

8. A knowledge graph-based active reasoning device, characterized in that, The device includes: The acquisition module is used to acquire the image and question of the input multimodal large model; The retrieval module is used to extract keywords from the question and retrieve related entities and attributes of the related entities in the knowledge graph. A classification module is used to classify the associated entities according to their attributes; The detection and recognition module is used to detect and recognize people and / or text in the image based on the classification results; The judgment module is used to determine whether the recognition result is related to the keyword. If it is related, the recognition result is added to the question; otherwise, the related entity is retrieved again from the knowledge graph. The generation module is used to generate answers based on the supplemented questions and images.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the knowledge graph-based active reasoning method as described in any one of claims 1-7.

10. A storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the knowledge graph-based active reasoning method as described in any one of claims 1-7.

Images