Embodied scene question answering method, device and electronic equipment based on search enhancement generation

Abstract

The application discloses a somatization scene question and answer method and device based on retrieval enhancement generation and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: matching a target object meeting question information from a scene graph; generating scene description information of the target object according to visual information of the target object in the scene graph; and inputting the scene description information and the question information into a language model to obtain reply information for the question information output by the language model. According to the embodiment of the application, the target object related to the question is retrieved from the scene graph based on the retrieval enhancement technology, then the visual perception information of the target object in the scene graph is converted into natural language information, and the information is input as the context of the language model to enhance the understanding and answering ability of the language model to the question information. Even if the question and answer need complex reasoning, more accurate answers can be obtained, so that the accuracy of the question and answer is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a body-possessed scene question-answering method and device based on retrieval enhancement generation and electronic equipment. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, question-answering robots, as one of the important applications thereof, have shown great potential in various fields.

[0003] At present, most of the robot question-answering technologies are based on encyclopedic knowledge graphs. As for the body-possessed scene, most of them are concentrated on the planning and execution of actions and operations of robots following simple instructions in the scene. For question-answering that requires complex reasoning, especially in the application of body-possessed scenes, robots often have difficulty in outputting accurate answers. SUMMARY

[0004] The application aims to at least solve one of the technical problems existing in the prior art. To this end, the application provides a body-possessed scene question-answering method and device based on retrieval enhancement generation to improve the accuracy of question-answering.

[0005] In a first aspect, the application provides a body-possessed scene question-answering method based on retrieval enhancement generation, comprising:

[0006] matching a target object satisfying the question information from a scene graph;

[0007] generating scene description information of the target object according to visual information of the target object in the scene graph;

[0008] inputting the scene description information and the question information into a language model to obtain reply information for the question information output by the language model.

[0009] According to the body-possessed scene question-answering method based on retrieval enhancement generation of the application, a target object satisfying the question information is matched from a scene graph; scene description information of the target object is generated according to visual information of the target object in the scene graph; and the scene description information and the question information are input into a language model to obtain reply information for the question information output by the language model. The embodiment of the application based on retrieval enhancement technology retrieves a target object related to the question from a scene graph, then converts visual perception information of the target object in the scene graph into natural language information, and inputs these information as the context of the language model to enhance the understanding and answering ability of the language model for the question information. Even if the question-answering requires complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of question-answering.

[0010] According to one embodiment of the present application, the matching of the target object meeting the question information from the scene graph comprises:

[0011] Structural extraction is performed on the question information to obtain first triple information comprising entities in the question information and positional relationships between the entities;

[0012] The target object meeting the question information is matched from the scene graph according to the first triple.

[0013] In this embodiment, by extracting the question information into structured triple information, the entities discussed in the question and the positional relationships between the entities can be more clearly identified, the ambiguity in the information matching process is reduced, the response capability to hidden information in a complex scene is enhanced, the target object related to the question can be matched from the scene graph in a targeted manner, and the accuracy of the target object matching is improved.

[0014] According to one embodiment of the present application, the matching of the target object meeting the question information from the scene graph according to the first triple comprises:

[0015] The entities and the positional relationships between the entities in the first triple are parsed;

[0016] The target object meeting the question information is matched from the scene graph according to the entities and the positional relationships.

[0017] In this embodiment, by parsing the entities and the positional relationships between the entities in the first triple, the specific object related to the question information can be accurately identified from the scene graph.

[0018] According to one embodiment of the present application, the generation of the scene description information of the target object according to the visual information of the target object in the scene graph comprises:

[0019] The first triple information is supplemented according to the visual information of the target object in the scene graph to obtain second triple information;

[0020] The second triple information is converted into the scene description information.

[0021] In this embodiment, by combining the visual information of the target object in the scene graph with the first triple information, the original structured information is supplemented, and the descriptiveness of the information is enhanced, so that the converted scene description information is more detailed and specific, and a more accurate context can be provided for a language model, so that more accurate and more consistent reply information of the embodied scene can be generated in the question and answer system.

[0022] According to one embodiment of the present application, the conversion of the second triple information into the scene description information comprises:

[0023] The second triple information is filled into a preset natural language template to obtain the scene description information.

[0024] In this embodiment, the automatic conversion from structured data to natural language description is realized by filling the triple information into the preset natural language template. This way makes the generation of scene description information both fast and standardized, improves the consistency and readability of information expression, and also makes the generated scene description information easier to be understood and processed by the language model, so as to make the model generate more accurate answers.

[0025] According to one embodiment of the present application, before the scene description information and the question information are input into the language model, further comprising:

[0026] The scene description information is filtered according to the relevance of the question information and the scene description information.

[0027] In this embodiment, by the filtering step, the scene description information is filtered according to the relevance of the question information and the scene description information, reducing the illusion effect of information redundancy on the model, so that the information input into the language model is highly relevant, improving the accuracy of the language model answer.

[0028] According to one embodiment of the present application, before the scene description information and the question information are input into the language model, further comprising:

[0029] The prompt example is input into the language model to make the language model learn the output mode in the embodied scene; the prompt example includes a question, a scene description and an answer corresponding to the question.

[0030] In this embodiment, by providing specific scene and question examples, the understanding and generation ability of the model for specific tasks can be enhanced, so that the model can learn and imitate the output mode in the embodied scene, thereby improving the response accuracy and relevance of the model in similar situations.

[0031] In a second aspect, the present application provides a embodied scene question and answer method based on retrieval enhancement generation, comprising:

[0032] The matching module is configured to match a target object satisfying the question information from the scene graph;

[0033] The generation module is configured to generate scene description information of the target object according to visual information of the target object in the scene graph.

[0034] The input module is configured to input the scene description information and the question information into a language model to obtain reply information for the question information output by the language model.

[0035] According to the embodiment of the present application, the embodiment of the present application generates the embodied scene question answering device based on retrieval enhancement. The embodiment of the present application matches a target object meeting the question information from a scene graph, generates scene description information of the target object according to visual information of the target object in the scene graph, and inputs the scene description information and the question information into a language model to obtain reply information for the question information output by the language model. The embodiment of the present application generates the embodied scene question answering device based on retrieval enhancement. The embodiment of the present application retrieves a target object related to a question from a scene graph, converts visual perception information of the target object in the scene graph into natural language information, and inputs the information as a context of a language model to enhance understanding and answering ability of the language model for question information. Even if a question and answer requiring complex reasoning are encountered, a more accurate answer can be obtained, thereby improving accuracy of the question and answer.

[0036] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the embodied scene question answering method based on retrieval enhancement generated according to the first aspect or the second aspect.

[0037] In a fourth aspect, the present application provides a non-transitory computer readable storage medium, and a computer program is stored in the non-transitory computer readable storage medium, and the computer program is executable on a processor to implement the embodied scene question answering method based on retrieval enhancement generated according to the first aspect or the second aspect.

[0038] In a fifth aspect, the present application provides a chip, and the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is configured to run a program or an instruction to implement the embodied scene question answering method based on retrieval enhancement generated according to the first aspect or the second aspect.

[0039] In a sixth aspect, the present application provides a computer program product, and the computer program product includes a computer program, and the computer program is executable on a processor to implement the embodied scene question answering method based on retrieval enhancement generated according to the first aspect or the second aspect.

[0040] The one or more technical solutions in the embodiment of the present application at least have one of the following technical effects:

[0041] According to the embodiment of the present application, the embodied scene question answering method based on retrieval enhancement is to match the target object meeting the question information from the scene graph; generate the scene description information of the target object according to the visual information of the target object in the scene graph; input the scene description information and the question information into the language model to obtain the reply information for the question information output by the language model. The embodiment of the present application is to retrieve the target object related to the question from the scene graph based on the retrieval enhancement technology, then convert the visual perception information of the target object in the scene graph into natural language information, and input these information as the context of the language model to enhance the understanding and answering ability of the language model to the question information. Even if the question and answer needs complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the question and answer.

[0042] Further, in some embodiments, by extracting the question information into structured triple information, the entity discussed in the question and the positional relationship between the entities can be more clearly identified, so that the matching process reduces the ambiguity in the information matching process, enhances the response ability to hidden information in the complex scene, can be targeted to match the target object related to the question from the scene graph, and improves the accuracy of the target object matching.

[0043] Further, in some embodiments, by analyzing the entity and its positional relationship in the first triple, the specific object related to the question information can be accurately identified from the scene graph.

[0044] Further, in some embodiments, by combining the visual information of the target object in the scene graph with the first triple information, not only the original structured information is supplemented, but also the description of the information is enhanced, so that the converted scene description information is more detailed and specific, which can provide more accurate context for the language model, thereby generating more accurate and more embodied scene reply information in the question and answer system.

[0045] Further, in some embodiments, by filling the triple information into the preset natural language template, the automatic conversion from structured data to natural language description is realized. This way makes the generation of scene description information fast and standardized, improves the consistency and readability of information expression, and also makes the generated scene description information easier to be understood and processed by the language model, thereby making the model generate more accurate answers.

[0046] Further, in some embodiments, by the screening processing step, the scene description information is screened according to the relevance of the question information and the scene description information, the illusion influence of information redundancy on the model is reduced, and the information input into the language model is highly relevant, thereby improving the accuracy of the language model answer.

[0047] Further, in some embodiments, by providing specific scene and question examples, the model's understanding and generation capabilities for specific tasks can be enhanced, enabling the model to learn and imitate output patterns in embodied scenarios, thereby improving its response accuracy and relevance in similar situations.

[0048] Additional aspects and advantages of the application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and / or additional aspects and advantages of the application will become apparent and be more readily understood through consideration of the following description, the accompanying drawings, and the claims.

[0050] Figure 1 is a flowchart of an embodied scene question answering method based on retrieval enhancement generation provided by an embodiment of the application;

[0051] Figure 2 is a structural schematic diagram of an embodied scene question answering device based on retrieval enhancement generation provided by an embodiment of the application;

[0052] Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the application will be described clearly below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the application.

[0054] The terms "first", "second", etc. in the specification of the application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / ", generally represents a "or" relationship between the objects before and after it.

[0055] The embodied scene question answering method, device and electronic device based on retrieval enhancement generation provided by the embodiments of the application will be described in detail below in conjunction with the drawings and specific embodiments and their application scenarios.

[0056] The embodied scene question answering method based on retrieval enhancement generation can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0057] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or a tablet computer having a touch-sensitive surface (for example, a touchscreen display and / or a touchpad). It should also be understood that, in some embodiments, the terminal can not be a portable communication device, but a desktop computer having a touch-sensitive surface (for example, a touchscreen display and / or a touchpad).

[0058] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal can include one or more other physical user interface devices such as physical keyboards, mice, and joysticks.

[0059] The embodied scene question answering method based on retrieval enhancement generation provided by the embodiments of the present application can be executed by an electronic device or a functional module or functional entity capable of implementing the embodied scene question answering method based on retrieval enhancement generation in the electronic device. The electronic device mentioned in the embodiments of the present application includes, but is not limited to, a mobile phone, a tablet computer, a computer, a camera, a wearable device, and the like. The following will take the electronic device as an example to describe the embodied scene question answering method based on retrieval enhancement generation provided by the embodiments of the present application.

[0060] As shown in the embodiment of the present application, Figure 1 The embodied scene question answering method based on retrieval enhancement generation includes steps 110, 120 and 130.

[0061] Step 110: Matching a target object satisfying the question information from a scene graph.

[0062] Embodied AI refers to an intelligent system that can understand, reason and interact with the physical world. It emphasizes that the intelligent agent obtains knowledge and skills through the interaction between the body and the environment. Embodied AI systems usually have the ability of perception, cognition, decision-making and action, and can interact with the environment through sensors and actuators, and make corresponding decisions and actions according to the changes of the environment.

[0063] The embodied scenario refers to an environment in which an agent is located. The embodied scenario can be a virtual scenario or a real scenario. Regardless of whether it is a virtual scenario or a real scenario, the agent can be part of the embodied scenario. In the embodied scenario, the agent needs to perform tasks such as navigation, question answering, and object manipulation. A scene graph can be a representation of a virtual scenario or a real scenario in the embodied scenario. The scene graph captures objects in the scenario and their mutual relationships through visual information. In one example, a plurality of scene graphs are included in a scene knowledge base. One of the scene graphs is a virtual children's room scenario. There is a blue table in the room, and some colorful blocks are placed on the table. The walls of the room are white, and a picture is hung on the wall. The picture has a yellow duck. Each object in the scene is recorded in detail by the scene graph, including position, size, color, and other visual information.

[0064] In an embodiment of the present application, the question information can be a specific question raised by a user for the scene graph. The question information can include a query for a specific object or relationship in the scene graph. The target object is an object in the scene graph that is associated with the question information. For example, for the question information "What color are the blocks on the table?", the question information includes the object "table" and the object "block". The two objects "table" and "block" are associated with the question information and are objects that satisfy the question information. The "table" and "block" are matched from the scene graph, and if the scene graph includes the "table" and "block", the "table" and "block" are the target objects.

[0065] In some embodiments, the question information can be parsed to identify keywords, entities, attributes, and other information of the question information. Since the scene graph includes visual information of multiple objects such as position, size, color, etc., the visual information of each object in the scene graph can be determined through computer vision technology such as image recognition technology. The corresponding target object is matched from the visual information of the scene graph according to the keywords, entities, attributes, and other information of the question information.

[0066] In some embodiments, a graph structure corresponding to the scene graph can also be obtained. The graph structure contains information of all objects in the scene and their mutual relationships. The graph structure is the visual information of the scene graph. A query statement such as Cypher query language is constructed through the keywords, entities, attributes, and other information of the question information. Then, a node that matches the question information is retrieved from the graph structure according to the query statement. The node is the target object.

[0067] Step 120: generating scene description information of the target object according to visual information of the target object in the scene graph.

[0068] In the embodiments of the present application, the visual information of the target object in the scene graph can include the features of the target object that can be perceived and identified in the scene, which can be position, size, shape, color, etc., or the relationship with other objects, such as the relative position and spatial relationship of the target object with other objects, such as "on", "next to", etc.

[0069] The scene description information is a natural language expression of the visual information of the target object in the scene, and the scene description information can include detailed descriptions of the features of the target object, which can help understand the scene content and context associated with the target object.

[0070] In some embodiments, the visual information of the target object can be extracted from the scene graph by using computer vision technology, such as determining the color and shape of the target object by image recognition technology, and determining other objects on, under, left, right, front, back, inside, next to, etc. of the target object. After the visual information of the target object is extracted, the visual information can be converted into scene description information of the target object by natural language generation technology or other means.

[0071] In one example, taking the question information "What is on your left?" as an example, the target object includes "you", i.e. the intelligent agent, and the scene description information is:

[0072] "In this scene, you have 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom rack, 4 yellow bananas, 1 white bathtub, 1 bathroom, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains, 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow hanging picture, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue hanging picture, 1 hanging picture, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair, behind you have 1 blue water bottle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green paper basket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 bowl cabinet, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow paper basket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white chandelier, in front of you have 2 carpets, 6 yellow curtains, 2 white pillows."

[0073] Step 130, input the scene description information and question information into the language model to obtain the reply information for the question information output by the language model.

[0074] The language model is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The language model can handle various natural language tasks such as text classification, question answering, and dialogue.

[0075] In some embodiments, the model used is not limited and can be a self-developed language model or a large language model such as GPT (Generative Pretrained Transformer) for processing.

[0076] In this embodiment, question information and scene description information can be input into a language model. Upon receiving the question information and scene description information, the language model performs internal processing to understand the content of the question information and identify the content of the scene description information. The content of the scene description information is used as the context of the language model, enhancing its ability to understand and answer the question information, and ultimately analyzing the answer to the question. In one example, the question information is "What color are the blocks on the table?", and the scene description information describes that the table in the scene diagram includes multiple different objects and describes the colors of these objects. Based on the description in the scene description information, the language model outputs the answer information "The blocks on the table are blue."

[0077] According to the embodied scenario question-answering method based on retrieval enhancement of this application, target objects that satisfy the question information are matched from the scene graph; scene description information of the target objects is generated based on the visual information of the target objects in the scene graph; the scene description information and question information are input into a language model to obtain the response information output by the language model for the question information. This embodiment of the application, based on retrieval enhancement technology, retrieves target objects related to the question from the scene graph, then converts the visual perception information of the target objects in the scene graph into natural language information, and uses this information as context input to the language model to enhance the language model's ability to understand and answer question information. Even when encountering question-answering that requires complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of question answering.

[0078] In some embodiments, matching target objects that satisfy the problem information from the scene graph includes:

[0079] The problem information is structured and extracted to obtain the first triplet information, which includes the entities in the problem information and the locational relationships between the entities;

[0080] Match the target object that satisfies the problem information from the scene graph based on the first triplet.

[0081] In this embodiment, the entities in the problem information can be any objects that may appear in the scene diagram, such as people, televisions, refrigerators, sofas, etc. The orientation relationship is the spatial positional relationship between entities, such as above, below, left, right, front, back, inside, and beside.

[0082] In this embodiment, key information in the problem can be extracted using natural language processing techniques, such as named entity recognition and relation extraction, and then organized into triple information. Triple information can include three parts: entity, location relation, and entity. For example, a triple relation can be represented as: {entity (subject): block, relation: above, entity (object): table}.

[0083] In one example, the GPLinker model can be used to jointly extract entities and entity-direct-location triples from question information. Taking the question "What color are the blocks on the table?" as an example, the extracted information will be in the following format:

[0084] {

[0085] "query_info": {

[0086] "entity_info": [

[0087] {

[0088] "entity": "table",

[0089] "entity_type": "common_base:table"

[0090] },

[0091] {

[0092] "entity": "building blocks",

[0093] "entity_type": "common_base:buildingBlock"

[0094] }

[0095] ],

[0096] "spo_info": [

[0097] {

[0098] "predicate": "common_base:isOn",

[0099] "subject": "building blocks",

[0100] "subject_type": "common_base:buildingBlock",

[0101] "object": {

[0102] @value: "table"

[0103] },

[0104] "object_type": {

[0105] @value": "common_base:table"

[0106] }

[0107] } ]

[0109] }

[0110] }

[0111] wherein, entity_info is an entity extracted from the question information query and information linked to the common base of the scene knowledge base; and spo_info is a triple information extracted from the question information query, wherein, predicate is a predicate relationship, i.e. the above-mentioned positional relationship word between entities, subject is the subject word of the predicate, object is the object word of the predicate, and subject_type and object_type are the corresponding information of the subject and object in the common base. The extracted triple information is organized into a JSON format data structure, which is convenient for computer reading and processing.

[0112] In this embodiment, by extracting the question information into structured triple information, the entity discussed in the question and the positional relationship between the entities can be more clearly identified, so that the matching process reduces the ambiguity in the information matching process, enhances the response ability to hidden information in complex scenes, can be targeted to match the target object related to the question from the scene graph, and improves the accuracy of target object matching.

[0113] In some embodiments, the target object satisfying the question information is matched from the scene graph according to the first triple, comprising:

[0114] parsing the entity and the positional relationship between the entities in the first triple;

[0115] matching the target object satisfying the question information from the scene graph according to the entity and the positional relationship.

[0116] In this embodiment, the first triple information can be parsed to identify the entity and the positional relationship between the entities in the first triple information. Since the scene graph includes visual information of multiple objects, such as position, size, color, etc., the visual information of each object in the scene graph can be determined by computer vision technology, such as image recognition technology, and the corresponding target object is matched from the visual information in the scene graph according to the entity and the positional relationship between the entities in the first triple information.

[0117] In some embodiments, a graph structure corresponding to the scene graph can also be obtained, which contains information of all objects in the scene and their mutual relationships. The graph structure is the visual information of the scene graph, and a query statement is constructed through the entity and the positional relationship between entities in the first triple information, such as the Cypher query language, and then a node matching the question information is retrieved from the graph structure according to the query statement, which is the target object.

[0118] In this embodiment, by analyzing the entity and its positional relationship in the first triple, the specific object related to the question information can be accurately identified from the scene graph.

[0119] In some embodiments, the scene description information of the target object is generated according to the visual information of the target object in the scene graph, comprising:

[0120] The first triple information is supplemented according to the visual information of the target object in the scene graph to obtain second triple information;

[0121] The second triple information is converted into scene description information.

[0122] In this embodiment, the information represented by the first triple information is the content of the question information. In order to enhance the related information of the target object in the scene graph, so that the subsequent language model can obtain more rich information input, the first triple information can be supplemented.

[0123] Specifically, the visual information of the target object in the scene graph can be obtained, such as identifying the features of the target object, such as color, shape, size, etc., and the positional relationship between the target object and other target objects, etc. through image processing technology, or through the graph structure corresponding to the scene graph. For example, for the question information "What color is the block on the table?", the extracted first triple information is: {entity (subject): block, relationship: above, entity (object): table}, according to the visual information of "block" and "table" in the scene graph, it can be obtained that the table includes multiple blocks, these blocks are blue, purple, red, and in addition to the blocks, there are water cups, paper towels, etc. on the table, these objects in the visual information can also be extracted, and the first triple information is supplemented to obtain the second triple information. In addition to the content of the first triple information, the second triple information also includes the visual information of the corresponding target object in the first triple information.

[0124] In this embodiment, by combining the visual information of the target object in the scene graph with the first triple information, not only the original structured information is supplemented, but also the description of the information is enhanced, so that the converted scene description information is more detailed and specific, which can provide more accurate context for the language model, so as to generate more accurate and more consistent reply information in the embodied scene in the question and answer system.

[0125] In some embodiments, converting the second triple information into the scene description information comprises:

[0126] filling the second triple information into a preset natural language template to obtain the scene description information.

[0127] In this embodiment, one or more natural language templates can be predefined, and the natural language template includes a preset natural language sentence structure for converting structured data into fluent natural language description. The natural language template can also include placeholders for subsequent filling of specific data.

[0128] For example, the natural language template is: “[object] is located at [location].” The filled scene description is “the vase is located in the center of the table.”

[0129] In one example, taking the question information “what is on your left?” as an example, the target object includes “you”, i.e. the agent, and the scene description information is:

[0130] “in this scene, there are 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom rack, 4 yellow bananas, 1 white bathtub, 1 bathroom, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains on your right, and 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow hanging picture, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue hanging picture, 1 hanging picture, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair on your left, and 1 blue water bottle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green paper basket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 bowl cabinet, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow paper basket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white ceiling lamp in front of you.”

[0131] In this embodiment, the automatic conversion from structured data to natural language description is realized by filling the triple information into the preset natural language template, which makes the generation of scene description information fast and standardized, improves the consistency and readability of information expression, and makes the generated scene description information easier to be understood and processed by the language model, so as to make the model generate more accurate answers.

[0132] In some embodiments, before inputting the scene description information and the question information into the language model, further comprising:

[0133] The scene description information is filtered according to the relevance of the question information and the scene description information.

[0134] In this embodiment, the scene description information describes the visual information of the target object in the scene graph. Since the content of the visual information is more, for example, it can include the color, size, and positional relationship with other objects of the target object, if the requirement of the question information is to understand the color of the target object, if a large amount of description of the size of the target object and the positional relationship with other objects is contained in the scene description information, the part associated with the question information in the scene description information will be less, so that the language model will produce hallucination when referring to the scene description information due to information redundancy, and the answer information obtained is not accurate enough.

[0135] In this embodiment, the part of the scene description information that is semantically associated with the question information can be identified according to the content of the question information and the scene description information through semantic analysis, and the information with high relevance to the question information is filtered out, and the redundant or low-relevance information is removed.

[0136] In one example, taking the question information "What is on your left?" as an example, the scene description information describes the height, weight, hairstyle, sitting posture of "you", and other objects on the left, right, front, etc. of "you". It can be analyzed that the question information may want to know the other objects in the spatial position of "you", and the scene description information describes the description about "you" itself, such as height, weight, hairstyle, sitting posture, etc. which may be low-relevance information, so these information can be deleted. The scene description information obtained after filtering can be:

[0137] "In this scenario, you have 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom rack, 4 yellow bananas, 1 white bathtub, 1 bathroom, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains, 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow hanging picture, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue hanging picture, 1 hanging picture, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair, behind you have 1 blue water bottle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green paper basket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 bowl cabinet, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow paper basket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white chandelier, in front of you have 2 carpets, 6 yellow curtains, 2 white pillows."

[0138] In this embodiment, the scene description information is filtered according to the relevance of the question information and the scene description information through the filtering processing step, the illusion influence caused by information redundancy on the model is reduced, the information input into the language model is highly relevant, and the accuracy of the language model answer is improved.

[0139] In some embodiments, before inputting the scene description information and the question information into the language model, further comprising:

[0140] The prompt examples are input into the language model to make the language model learn the output mode in embodied scenarios; the prompt examples include questions, scene descriptions, and answers corresponding to the questions.

[0141] In this embodiment, the prompt examples are one or more sets of data containing questions, scenario descriptions, and corresponding answers. These examples can be used to train or adjust the language model, enabling the language model to learn and understand questions in embodied scenarios and generate appropriate answers. The language model learns to combine questions with scenario description information to generate accurate answers in embodied scenarios by learning these examples. Through learning prompt examples, the generalization ability of the language model is improved, enabling it to generate reasonable answers according to scenario description information when facing new, unseen questions.

[0142] In one example, the prompt example is as follows:

[0143] [input]

[0144] [Q] What color is the potted plant on the ground?

[0145] [Common knowledge] The explanation of the ground is: The ground has two pronunciations: dì, de. It can be used after a verb as a auxiliary word, or as a noun, indicating the earth (referring to the surface of the earth; the surface part close to the crust), the earth, etc. Pinyin: dì de, annotation: ㄉㄧˋ(ㄉㄜ), radical: earth, structure: left and right, total strokes: 6, external strokes: 3, stroke order number: 121525, Bopomofo: GPD, four-corner code: 4411, five-pin 86: FBN, five-pin 98: FBN Love is what color the author is Keno. The classification of potted plants (traditional art form) is tree planting, landscape bonsai. The explanation of potted plants is to express the landscape of potted plants. The book title of potted flowers is Potted Flowers. The varieties of potted vegetables are lettuce, green onions, and coriander. The color of the ground paint is various. The series of Potted Flowers Colorful Pictures is the Practical Gardening Series. The characteristics of small potted plants are landscape.

[0146] [Memory]

[0147] [scene information] 1 pot in front of you on the floor, its color is green, its right side has 2 black mirrors, 1 black refrigerator, 2 black spherical apples, 1 red cabinet, 1 black door, 1 black radio, 2 green plants, 1 black dining table, 6 black ceiling lamps, 2 white windows, 1 red hanging picture, 1 purple carpet, 1 black cup, 1 black remote control, 1 brown range hood, 1 brown plant, 1 brown rectangular coffee table, 4 black chairs, 1 black vase, 1 red paper basket, 1 living room, 1 black TV, 1 gray TV cabinet, 1 green cushion, 1 red cabinet, its back side has 1 black plant, 1 black ottoman, 1 brown cabinet, 1 black sofa, 1 blue-green plant, 1 pillow, 1 black towel, 1 black sink, 6 black ceiling lamps, 1 brown towel rack, 2 red curtains,

[0148] [output]

[0149] [R] The pot on the floor is green.

[0150] In this embodiment, the retrieval part of the language model adopts the form of multi-path recall, which can include encyclopedia knowledge graph, historical action information, scene description input, etc. Therefore, common sense knowledge can be recalled in the encyclopedia knowledge graph based on the entity in the question; memory is the historical action information made by the robot in this conversation; scene information is the scene description input provided by the scenario. Finally, the language model generates answers based on the above retrieval-enhanced information and its powerful reasoning and summarizing ability.

[0151] In this embodiment, by providing specific scene and question examples, the understanding and generation ability of the model for specific tasks can be enhanced, so that the language model can learn and imitate the output mode in embodied scenarios, thereby improving its response accuracy and relevance in similar situations.

[0152] The embodiment of the present application provides an embodied scene question answering method based on retrieval enhancement generation. The execution subject can be an embodied scene question answering device based on retrieval enhancement generation. In the embodiment of the present application, the embodied scene question answering method based on retrieval enhancement generation is executed by the embodied scene question answering device based on retrieval enhancement generation. The embodiment of the present application provides an embodied scene question answering device based on retrieval enhancement generation.

[0153] The embodiment of the present application further provides an embodied scene question answering device based on retrieval enhancement generation.

[0154] As shown in Figure 2 the embodied scene question answering device based on retrieval enhancement generation comprises:

[0155] The matching module 210 is configured to match a target object meeting the question information from the scene graph;

[0156] The generating module 220 is configured to generate scene description information of the target object according to visual information of the target object in the scene graph;

[0157] The input module 230 is configured to input the scene description information and the question information into a language model to obtain reply information for the question information output by the language model.

[0158] According to the embodiment of the present application, the embodied scene question answering device based on retrieval enhancement generation is matched with a target object meeting the question information from the scene graph; scene description information of the target object is generated according to visual information of the target object in the scene graph; the scene description information and the question information are input into a language model to obtain reply information for the question information output by the language model. The embodiment of the present application is based on retrieval enhancement technology, which retrieves a target object related to the question from the scene graph, then converts visual perception information of the target object in the scene graph into natural language information, and inputs the information as a context of the language model to enhance the understanding and answering ability of the language model for the question information. Even if the question and answer need complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the question and answer.

[0159] In some embodiments, the matching module 210 is further configured to:

[0160] structurally extract the question information to obtain first triple information including entities in the question information and positional relationships between the entities;

[0161] match the target object meeting the question information from the scene graph according to the first triple.

[0162] In some embodiments, the matching module 210 is further configured to:

[0163] analyze the entities and the positional relationships between the entities in the first triple;

[0164] match the target object meeting the question information from the scene graph according to the entities and the positional relationships.

[0165] In some embodiments, the generating module 220 is further configured to:

[0166] supplement the first triple information according to the visual information of the target object in the scene graph to obtain second triple information;

[0167] convert the second triple information into the scene description information.

[0168] In some embodiments, the generating module 220 is further configured to:

[0169] The second triplet information is filled into a preset natural language template to obtain scene description information.

[0170] In some embodiments, the generation module 220 is further configured to:

[0171] The scene description information is filtered according to the relevance between the question information and the scene description information.

[0172] In some embodiments, the input module 230 is further configured to:

[0173] The prompt examples are input into the language model to enable the language model to learn the output mode in the embodied scene; the prompt examples include questions, scene descriptions, and answers corresponding to the questions.

[0174] The embodied scene question and answer device based on retrieval enhancement generation in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiments of the present application are not limited in this regard.

[0175] The embodied scene question and answer device based on retrieval enhancement generation in the embodiments of the present application can be a device with an operating system. The operating system can be a Microsoft (Windows) operating system, an Android operating system, an IOS operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0176] In some embodiments, as Figure 3As shown, the embodiment of the present application further provides an electronic device 300, comprising a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, which, when executed by the processor 301, implements each process of the embodiment of the above-mentioned method for generating embodied scene question answering based on search enhancement, and achieves the same technical effects. To avoid repetition, details are not repeated here.

[0177] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0178] The embodiment of the present application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements each process of the embodiment of the above-mentioned method for generating embodied scene question answering based on search enhancement, and achieves the same technical effects. To avoid repetition, details are not repeated here.

[0179] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0180] The embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned method for generating embodied scene question answering based on search enhancement.

[0181] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0182] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement each process of the embodiment of the above-mentioned method for generating embodied scene question answering based on search enhancement, and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0183] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0184] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the terms "first", "second" or the like herein do not denote any order, quantity, combination or arrangement, but are used to identify one of the elements described. In addition, it should be noted that the scope of the embodiments of the present application is not limited by the order of the steps recited in the claims. The steps of the various embodiments of the present application can be performed in any order or simultaneously, as appropriate, unless otherwise indicated or unless it is inherent to the methodology.

[0185] From the above description of the embodiments, it is apparent that the above-mentioned method can be realized by means of software and necessary universal hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solution of the present application can be embodied in the form of computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, or network equipment, etc.) execute the method described in various embodiments of the present application.

[0186] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, which are only illustrative, not restrictive, and those of ordinary skill in the art can make many forms without departing from the scope of the present application.

[0187] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A method for embodied scenario question answering based on retrieval enhancement generation, characterized in that, The method comprises the following steps: matching a target object meeting the question information from a scene graph; comprising: structurally extracting the question information to obtain first triple information comprising entities in the question information and positional relationships between the entities; matching a target object meeting the question information from the scene graph according to the first triple; the target object is an object associated with the question information in the scene graph; generating scene description information of the target object according to visual information of the target object in the scene graph; comprising: supplementing the first triple information according to the visual information of the target object in the scene graph to obtain second triple information; converting the second triple information into the scene description information; the scene description information is a natural language expression of visual information of a target object in a scene; inputting the scene description information and the question information into a language model to obtain reply information for the question information output by the language model; before inputting the scene description information and the question information into the language model, the method further comprises: filtering the scene description information according to the relevance of the question information and the scene description information.

2. The method of claim 1, wherein, The matching a target object meeting the question information from the scene graph according to the first triple comprises: analyzing the entities and the positional relationships between the entities in the first triple; matching a target object meeting the question information from the scene graph according to the entities and the positional relationships.

3. The method of claim 1, wherein, The converting the second triple information into the scene description information comprises: filling the second triple information into a preset natural language template to obtain the scene description information.

4. The method of claim 1, wherein, Before inputting the scene description information and the question information into the language model, the method further comprises: inputting a prompt example into the language model to enable the language model to learn an output mode in a embodied scene; the prompt example comprises a question, a scene description, and an answer corresponding to the question.

5. A body-aware scene question answering apparatus based on retrieval augmentation generation, the apparatus comprising: a scene question answering module configured to: receive a scene question; and generate a body-aware scene answer to the scene question based on a body-aware scene generation model. The method comprises the following steps: a matching module for matching a target object meeting the question information from a scene graph; the matching module is further configured to structurally extract the question information to obtain first triple information comprising entities in the question information and positional relationships between the entities; match a target object meeting the question information from the scene graph according to the first triple; the target object is an object associated with the question information in the scene graph; a generating module for generating scene description information of the target object according to visual information of the target object in the scene graph; the generating module is further configured to supplement the first triple information according to the visual information of the target object in the scene graph to obtain second triple information; and convert the second triple information into the scene description information; the scene description information is a natural language expression of visual information of a target object in a scene; an input module for inputting the scene description information and the question information into a language model to obtain reply information for the question information output by the language model; Before inputting the scene description information and the question information into the language model, further comprising: performing screening processing on the scene description information according to the relevance between the question information and the scene description information.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1-4 when executing the program.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the method of any one of claims 1-4 when executed by the processor.

Images