Body scene question answering method and device based on retrieval enhancement generation and electronic equipment

Abstract

The invention discloses an own scene question answering method and device based on retrieval enhancement generation and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: matching a target object meeting problem information from a scene graph; generating scene description information of the target object according to the visual information of the target object in the scene graph; and inputting the scene description information and the question information into a language model to obtain reply information which is output by the language model and aims at the question information. According to the embodiment of the invention, the target object related to the question is retrieved from the scene graph based on the retrieval enhancement technology, then the visual perception information of the target object in the scene graph is converted into the natural language information, and the information is used as the context input of the language model; therefore, the understanding and answering ability of the language model to question information is enhanced, even if a question and answer needing complex reasoning is encountered, a more accurate answer can be obtained, and the accuracy of the question and answer is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to an embodied scenario question-answering method, device and electronic device based on retrieval-enhanced generation. Background Art

[0002] With the continuous development of artificial intelligence technology, question-answering robots, as one of its important applications, have shown great potential in various fields.

[0003] Most of the current robot question-answering technologies are based on encyclopedia knowledge graphs. In the case of spatial embodied scenarios, most of them focus on robots in the scenario obeying simple instructions and planning and executing actions. For questions and answers that require complex reasoning, especially in embodied scenarios, it is often difficult for robots to output accurate answers. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes an embodied scenario question-answering method, device and electronic device based on retrieval-enhanced generation to improve the accuracy of question-answering.

[0005] In a first aspect, the present application provides an embodied scenario question answering method based on retrieval-enhanced generation, comprising: Match the target object that satisfies the question information from the scene graph; Generate scene description information of the target object according to visual information of the target object in the scene graph; The scene description information and the question information are input into a language model to obtain response information for the question information output by the language model.

[0006] According to the embodied scene question-answering method based on retrieval enhancement generation of the present application, the target object that satisfies the question information is matched from the scene graph; the scene description information of the target object is generated according to the visual information of the target object in the scene graph; the scene description information and the question information are input into the language model to obtain the reply information for the question information output by the language model. The embodiment of the present application is based on retrieval enhancement technology by retrieving the target object related to the question from the scene graph, and then converting the visual perception information of the target object in the scene graph into natural language information, and using this information as the context input of the language model to enhance the language model's ability to understand and answer the question information. Even when encountering questions and answers that require complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the questions and answers.

[0007] According to an embodiment of the present application, matching a target object satisfying the question information from a scene graph includes: Performing structured extraction on the question information to obtain first triplet information including entities in the question information and positional relationships between entities; A target object that satisfies the question information is matched from the scene graph according to the first triple.

[0008] In this embodiment, by extracting the question information into structured triple information, the entities discussed in the question and the positional relationships between the entities can be more clearly identified, so that the matching process reduces ambiguity in the information matching process, enhances the ability to respond to hidden information in complex scenes, and can specifically match the target objects related to the question from the scene graph, thereby improving the accuracy of target object matching.

[0009] According to an embodiment of the present application, matching a target object satisfying the question information from the scene graph according to the first triplet includes: parsing the directional relationship between entities in the first triplet; A target object that satisfies the question information is matched from the scene graph according to the entity and the orientation relationship.

[0010] In this embodiment, by parsing the entities in the first triplet and their positional relationships, specific objects related to the question information can be accurately identified from the scene graph.

[0011] According to an embodiment of the present application, generating scene description information of the target object according to visual information of the target object in the scene graph includes: Supplementing the first triplet of information according to the visual information of the target object in the scene graph to obtain second triplet of information; The second triplet information is converted into the scene description information.

[0012] In this embodiment, by combining the visual information of the target object in the scene graph with the first triplet information, not only the original structured information is supplemented, but also the descriptiveness of the information is enhanced, so that the converted scene description information is more detailed and specific, and can provide a more accurate context for the language model, thereby generating more precise and more embodied scenario-compliant response information in the question-answering system.

[0013] According to an embodiment of the present application, converting the second triplet information into the scene description information includes: The second triplet information is filled into a preset natural language template to obtain the scene description information.

[0014] In this embodiment, by filling the triple information into the preset natural language template, the automatic conversion from structured data to natural language description is achieved. This method makes the generation of scene description information both fast and standardized, improves the consistency and readability of information expression, and also makes the generated scene description information easier to be understood and processed by the language model, thereby enabling the model to generate more accurate answers.

[0015] According to an embodiment of the present application, before inputting the scene description information and the question information into the language model, the method further includes: The scene description information is filtered according to the relevance between the question information and the scene description information.

[0016] In this embodiment, through the screening process, the scene description information is screened according to the correlation between the question information and the scene description information, thereby reducing the illusion effect of information redundancy on the model, making the information input into the language model highly relevant, and improving the accuracy of the language model's answer.

[0017] According to an embodiment of the present application, before inputting the scene description information and the question information into the language model, the method further includes: Inputting prompt examples into a language model so that the language model learns output patterns in embodied scenarios; the prompt examples include questions, scenario descriptions, and answers corresponding to the questions.

[0018] In this embodiment, by providing specific scenarios and question examples, the model's understanding and generation capabilities for specific tasks can be enhanced, enabling the model to learn and imitate output patterns in embodied scenarios, thereby improving the accuracy and relevance of its responses in similar situations.

[0019] In the second aspect, the present application provides an embodied scenario question answering method based on retrieval-enhanced generation, comprising: A matching module, used to match target objects that meet the problem information from the scene graph; A generating module, used for generating scene description information of the target object according to visual information of the target object in the scene graph; An input module is used to input the scene description information and the question information into a language model to obtain response information for the question information output by the language model.

[0020] According to the embodied scene question-answering device based on retrieval enhancement generation of the present application, by matching the target object that satisfies the question information from the scene graph; generating the scene description information of the target object based on the visual information of the target object in the scene graph; inputting the scene description information and the question information into the language model, the language model outputs the reply information for the question information. The embodiment of the present application is based on retrieval enhancement technology by retrieving the target object related to the question from the scene graph, and then converting the visual perception information of the target object in the scene graph into natural language information, and using this information as the context input of the language model to enhance the language model's ability to understand and answer the question information. Even when encountering questions and answers that require complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the questions and answers.

[0021] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the embodied scenario question and answer method based on retrieval-enhanced generation as described in the first aspect or the second aspect above is implemented.

[0022] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the embodied scenario question-answering method based on retrieval-enhanced generation as described in the first or second aspect above.

[0023] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the embodied scenario question and answer method based on retrieval-enhanced generation as described in the first or second aspect above.

[0024] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the embodied scenario question and answer method based on retrieval-enhanced generation as described in the first aspect or the second aspect above.

[0025] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects: According to the embodied scene question-answering method based on retrieval enhancement generation of the present application, the target object that satisfies the question information is matched from the scene graph; the scene description information of the target object is generated according to the visual information of the target object in the scene graph; the scene description information and the question information are input into the language model to obtain the reply information for the question information output by the language model. The embodiment of the present application is based on retrieval enhancement technology by retrieving the target object related to the question from the scene graph, and then converting the visual perception information of the target object in the scene graph into natural language information, and using this information as the context input of the language model to enhance the language model's ability to understand and answer the question information. Even when encountering questions and answers that require complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the questions and answers.

[0026] Furthermore, in some embodiments, by extracting question information as structured triple information, the entities discussed in the question and the positional relationships between the entities can be more clearly identified, so that the matching process reduces ambiguity in the information matching process, enhances the ability to respond to hidden information in complex scenes, and can specifically match target objects related to the question from the scene graph, thereby improving the accuracy of target object matching.

[0027] Furthermore, in some embodiments, by parsing the entities in the first triplet and their positional relationships, specific objects related to the question information can be accurately identified from the scene graph.

[0028] Furthermore, in some embodiments, by combining the visual information of the target object in the scene graph with the first triplet information, not only the original structured information is supplemented, but also the descriptiveness of the information is enhanced, so that the converted scene description information is more detailed and specific, and can provide a more accurate context for the language model, thereby generating more precise and more embodied scenario-oriented response information in the question-answering system.

[0029] Furthermore, in some embodiments, by filling triple information into a preset natural language template, automatic conversion from structured data to natural language description is achieved. This approach makes the generation of scene description information both fast and standardized, improves the consistency and readability of information expression, and also makes the generated scene description information easier to be understood and processed by the language model, thereby enabling the model to generate more accurate answers.

[0030] Furthermore, in some embodiments, through a screening process, the scene description information is screened according to the correlation between the question information and the scene description information, thereby reducing the illusion effect of information redundancy on the model, making the information input into the language model highly relevant, and improving the accuracy of the language model's answers.

[0031] Furthermore, in some embodiments, by providing specific scenarios and problem examples, the model's understanding and generation capabilities for specific tasks can be enhanced, enabling the model to learn and imitate output patterns in embodied scenarios, thereby improving the accuracy and relevance of its responses in similar situations.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 It is a flowchart of the embodied scenario question-answering method based on retrieval-enhanced generation provided in an embodiment of the present application; Figure 2 It is a structural diagram of an embodied scenario question-answering device based on retrieval-enhanced generation provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] In conjunction with the accompanying drawings, the embodied scenario question-and-answer method, device, and electronic device based on retrieval-enhanced generation provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios.

[0037] Among them, the embodied scenario question and answer method based on retrieval enhancement generation can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0038] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or a tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0039] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0040] The embodied scenario question and answer method based on retrieval enhancement generation provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in an electronic device that can implement the embodied scenario question and answer method based on retrieval enhancement generation. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The embodied scenario question and answer method based on retrieval enhancement generation provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0041] like Figure 1 As shown, the embodied scenario question answering method based on retrieval enhanced generation includes: step 110, step 120 and step 130.

[0042] Step 110: Match target objects that meet the question information from the scene graph.

[0043] Embodied AI refers to intelligent systems that can understand, reason and interact with the physical world. It emphasizes that intelligent agents acquire knowledge and skills through the interaction between the body and the environment. Embodied intelligent systems usually have the ability to perceive, recognize, make decisions and act. They can interact with the environment through sensors and actuators, and make corresponding decisions and actions based on changes in the environment.

[0044] The embodied scene refers to the environment in which the agent is located. The embodied scene can be a virtual scene or a real scene. Whether it is a virtual scene or a real scene, the agent can be part of the embodied scene. In the embodied scene, the agent needs to perform tasks such as navigation, question answering, and object manipulation. The scene graph can be a representation of the virtual scene or the real scene in the embodied scene. The scene graph captures the objects in the scene and their relationships through visual information. In one example, the scene knowledge base includes multiple scene graphs, one of which is a virtual children's room scene. There is a blue table in the room with some colorful building blocks on the table. The walls of the room are white, and there is a painting on the wall with a yellow duck in the painting. Each object in the scene is recorded in detail by the scene graph, including visual information such as position, size, and color.

[0045] In an embodiment of the present application, the question information may be a specific question raised by the user with respect to the scene graph, and the question information may include inquiries about specific objects or relationships in the scene graph. The target object is an object in the scene graph that is associated with the question information. For example, for the question information "What color are the blocks on the table?", it can be identified that the question information includes the object "table" and the object "blocks", then the two objects "table" and "blocks" are associated with the question information, and are objects that meet the question information. Matching is performed from the scene graph. If the scene graph includes "table" and "blocks", then "table" and "blocks" are target objects.

[0046] In some embodiments, the question information can be parsed to identify keywords, entities, attributes and other information of the question information. Since the scene graph includes visual information of multiple objects, such as position, size, color, etc., computer vision technology, such as image recognition technology, can be used to determine the visual information of each object in the scene graph, and the corresponding target object can be matched from the visual information in the scene graph based on the keywords, entities, attributes and other information of the question information.

[0047] In some embodiments, a graph structure corresponding to the scene graph may also be obtained, which contains information about all objects in the scene and their relationships. The graph structure is the visual information of the scene graph, and a query statement is constructed using the keywords, entities, attributes, and other information of the question information, such as the Cypher query language, and then a node matching the question information is retrieved from the graph structure according to the query statement, and the node is the target object.

[0048] Step 120: Generate scene description information of the target object according to the visual information of the target object in the scene graph.

[0049] In an embodiment of the present application, the visual information of the target object in the scene graph may include features of the target object that can be perceived and identified in the scene. These features may be position, size, shape, color, etc., or may be relationships with other objects, such as the relative position and spatial relationship between the target object and other objects, such as "above...", "next to...", etc.

[0050] The scene description information is a natural language representation of the visual information of the target object in the scene. The scene description information may include a detailed description of the characteristics of the target object, which can help understand the scene content and context associated with the target object.

[0051] In some embodiments, visual information of the target object can be extracted from the scene graph by using computer vision technology, such as determining the color and shape of the target object by using image recognition technology, and determining other objects in spatial positions such as above, below, to the left, to the right, in front, behind, inside, and beside the target object. After the visual information of the target object is extracted, the visual information can be converted into scene description information of the target object by using natural language generation technology or other methods.

[0052] In one example, taking the question information “What is there on your left?” as an example, the target object includes “you”, i.e., the agent, and the scene description information is: "In this scene, on your right are 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom shelf, 4 yellow bananas, 1 white bathtub, 1 toilet, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains. On your left are 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow painting, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue painting, 1 painting, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red 1 chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair, behind you there are 1 blue kettle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green wastebasket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 cupboard, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow wastebasket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white chandelier, in front of you there are 2 carpets, 6 yellow curtains, 2 white pillows. " Step 130: Input the scene description information and question information into the language model to obtain response information for the question information output by the language model.

[0053] A language model is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0054] In some embodiments, the model used is not limited and can be a self-developed language model or a large language model such as GPT (Generative Pretrained Transformer).

[0055] In an embodiment of the present application, the question information and the scene description information can be input into the language model. After receiving the question information and the scene description information, the language model will perform internal processing, understand the content of the question information, and identify the content of the scene description information. The content of the scene description information is used as the context of the language model to enhance the language model's ability to understand and answer the question information and analyze the answer to the question information. In an example, the question information is "What color are the building blocks on the table?" The scene description information describes that the scene graph includes multiple different objects on the table and describes the colors of these objects. According to the description in the scene description information, the answer information output by the language model is "The building blocks on the table are blue." According to the embodied scene question-answering method based on retrieval enhancement generation of the present application, the target object that satisfies the question information is matched from the scene graph; the scene description information of the target object is generated according to the visual information of the target object in the scene graph; the scene description information and the question information are input into the language model to obtain the reply information for the question information output by the language model. The embodiment of the present application is based on retrieval enhancement technology by retrieving the target object related to the question from the scene graph, and then converting the visual perception information of the target object in the scene graph into natural language information, and using this information as the context input of the language model to enhance the language model's ability to understand and answer the question information. Even when encountering questions and answers that require complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the questions and answers.

[0056] In some embodiments, matching a target object satisfying the question information from the scene graph includes: Performing structured extraction on the question information to obtain first triplet information including entities in the question information and positional relationships between the entities; According to the first triplet, a target object that satisfies the question information is matched from the scene graph.

[0057] In this embodiment, the entity in the question information can be any object that may appear in the scene graph, such as a person, a television, a refrigerator, a sofa, etc. The orientation relationship is the spatial position relationship between the entities, such as above, below, left, right, front, back, inside, and next to it.

[0058] In this embodiment, the key information in the question can be extracted through natural language processing technology, such as named entity recognition and relation extraction, and the key information can be organized into triple information. The triple information can include three parts: entity, orientation relationship and entity. For example, a triple relationship is represented as: {entity (subject): building block, relationship: above, entity (object): table}.

[0059] In one example, the GPLinker model can be used to jointly extract the entity and the entity's direct orientation relationship triples from the question information. Taking the question information "What color are the blocks on the table?" as an example, after the extraction is completed, the output information is in the following form: { "query_info": { "entity_info": [ { "entity": "table", "entity_type": "common_base:table" }, { "entity": "Building Blocks", "entity_type": "common_base:buildingBlock" } ], "spo_info": [ { "predicate": "common_base:isOn", "subject": "Building Blocks", "subject_type": "common_base:buildingBlock", "object": { "@value": "table" }, "object_type": { "@value": "common_base:table" } } ] } } Among them, entity_info is the entity extracted from the question information query, and the information linking the entity to the scene knowledge base common_base; spo_info is the triple information extracted from the question information query, among which predicate is the predicate relationship, that is, the directional relationship word between the entities mentioned above, subject is the subject word of the predicate, object is the object word of the predicate, and subject_type and object_type are the corresponding information of the subject and object in common_base. The extracted triple information is organized into a JSON format data structure, which is convenient for computer reading and processing.

[0060] In this embodiment, by extracting the question information into structured triple information, the entities discussed in the question and the positional relationships between the entities can be more clearly identified, so that the matching process reduces ambiguity in the information matching process, enhances the ability to respond to hidden information in complex scenes, and can specifically match the target objects related to the question from the scene graph, thereby improving the accuracy of target object matching.

[0061] In some embodiments, matching a target object satisfying the question information from the scene graph according to the first triplet includes: Parsing the directional relationship between entities in the first triplet; Match the target objects that meet the question information from the scene graph based on entity and orientation relationships.

[0062] In this embodiment, the first triplet information can be parsed to identify entities of the first triplet information and information such as the orientation relationship between entities. Since the scene graph includes visual information of multiple objects, such as position, size, color, etc., computer vision technology, such as image recognition technology, can be used to determine the visual information of each object in the scene graph, and the corresponding target object can be matched from the visual information in the scene graph based on information such as the orientation relationship between entities of the first triplet information.

[0063] In some embodiments, a graph structure corresponding to the scene graph may also be obtained, and the graph structure includes information about all objects in the scene and their relationships. The graph structure is the visual information of the scene graph, and a query statement is constructed through information such as the orientation relationship between entities in the first triple information, such as the Cypher query language, and then a node matching the question information is retrieved from the graph structure according to the query statement, and the node is the target object.

[0064] In this embodiment, by parsing the entities in the first triplet and their positional relationships, specific objects related to the question information can be accurately identified from the scene graph.

[0065] In some embodiments, generating scene description information of the target object according to visual information of the target object in the scene graph includes: Supplementing the first triplet information according to the visual information of the target object in the scene graph to obtain the second triplet information; The second triplet information is converted into scene description information.

[0066] In this embodiment, the information represented by the first triplet is the content of the question information. In order to enhance the relevant information of the target object in the scene graph so that the subsequent language model can obtain richer information input, the first triplet information can be supplemented.

[0067] Specifically, visual information of the target object in the scene graph can be obtained, such as by using image processing technology or by using the graph structure corresponding to the scene graph to identify the features of the target object, such as color, shape, size, etc., as well as the orientation relationship between the target object and other target objects. For example, for the question information "What color are the blocks on the table?", the first triplet information extracted is: {entity (subject): block, relationship: above, entity (object): table}. According to the visual information of "block" and "table" in the scene graph, it can be obtained that there are multiple blocks on the table, and these blocks are blue, purple, and red. In addition to the blocks, there are also water cups, paper towels, etc. on the table. These objects in the visual information can also be extracted to supplement the first triplet information to obtain the second triplet information. In addition to the content of the first triplet information, the second triplet information also includes the visual information of the target object corresponding to the first triplet information.

[0068] In this embodiment, by combining the visual information of the target object in the scene graph with the first triplet information, not only the original structured information is supplemented, but also the descriptiveness of the information is enhanced, so that the converted scene description information is more detailed and specific, and can provide a more accurate context for the language model, thereby generating more precise and more embodied scenario-compliant response information in the question-answering system.

[0069] In some embodiments, converting the second triplet information into scene description information includes: The second triplet information is filled into a preset natural language template to obtain scene description information.

[0070] In this embodiment, one or more natural language templates may be predefined, and the natural language templates include a preset natural language sentence structure for converting structured data into a fluent natural language description. The natural language templates may also include placeholders for subsequent filling of specific data.

[0071] For example, the natural language template is: "[object] is located at [position]." The filled scene description is "the vase is located in the center of the table." In one example, taking the question information “What is there on your left?” as an example, the target object includes “you”, i.e., the agent, and the scene description information is: "In this scene, on your right are 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom shelf, 4 yellow bananas, 1 white bathtub, 1 toilet, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains. On your left are 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow painting, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue painting, 1 painting, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red 1 chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair, behind you there are 1 blue kettle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green wastebasket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 cupboard, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow wastebasket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white chandelier, in front of you there are 2 carpets, 6 yellow curtains, 2 white pillows. " In this embodiment, by filling the triple information into the preset natural language template, the automatic conversion from structured data to natural language description is achieved. This method makes the generation of scene description information both fast and standardized, improves the consistency and readability of information expression, and also makes the generated scene description information easier to be understood and processed by the language model, thereby enabling the model to generate more accurate answers.

[0072] In some embodiments, before inputting the scene description information and the question information into the language model, the method further includes: The scene description information is filtered and processed according to the relevance between the question information and the scene description information.

[0073] In this embodiment, the scene description information describes the visual information of the target object in the scene graph. Since the visual information contains a lot of content, such as the color, size, and positional relationship of the target object with other objects, if the question information is to understand the color of the target object, then in the scene description information, if it contains a large amount of descriptions of the size of the target object, the positional relationship with other objects, etc., the part of the scene description information that is associated with the question information will be small, causing the language model to produce hallucinations due to information redundancy when referring to the scene description information, and the answer information obtained is not accurate enough.

[0074] In this embodiment, semantic analysis can be performed to identify the parts of the scene description information that are semantically related to the keywords of the question information based on the content of the question information and the scene description information, filter out information that is highly related to the question information, and remove redundant or low-relevance information.

[0075] In one example, taking the question information "What's on your left?" as an example, the scene description information describes "your" height, weight, hairstyle, sitting posture, and other objects on the left, right, and front of "you". It can be analyzed that the question information may want to know other objects in the spatial position of "you", while the scene description information describes the description of "you" itself, such as height, weight, hairstyle, sitting posture, etc., which may be low-relevance information, so this information can be deleted. The scene description information obtained after screening can be: "In this scene, on your right are 6 green plants, 1 black remote control, 1 white DVD, 5 white windows, 1 black bathroom shelf, 4 yellow bananas, 1 white bathtub, 1 toilet, 1 black TV, 1 carpet, 1 clip, 2 white air conditioners, 1 blue TV cabinet, 6 yellow curtains. On your left are 1 red toy, 2 yellow pears, 3 yellow doors, 1 yellow painting, 1 living room, 5 white windows, 1 yellow rectangular coffee table, 2 green toys, 1 yellow shoe cabinet, 2 white pillows, 1 blue painting, 1 painting, 1 blue storage box, 1 red ball, 6 green plants, 1 green storage box, 1 blue sofa, 1 blue cushion, 4 yellow bananas, 1 white bed, 1 red 1 chair, 1 carpet, 2 white air conditioners, 1 toy, 1 blue cup, 1 ball, 2 carpets, 2 yellow toys, 6 yellow curtains, 1 blue chair, behind you there are 1 blue kettle, 3 yellow spoons, 3 yellow doors, 1 yellow cup, 1 green wastebasket, 1 white toilet, 5 white windows, 1 pot, 1 black pot, 6 green plants, 1 cupboard, 1 black sink, 4 white chairs, 1 white clock, 1 blue book, 1 yellow rectangular dining table, 1 white wardrobe, 2 yellow books, 1 yellow wastebasket, 4 white plates, 1 red cup, 1 bedroom, 1 white refrigerator, 6 yellow curtains, 1 white chandelier, in front of you there are 2 carpets, 6 yellow curtains, 2 white pillows. " In this embodiment, through the screening process, the scene description information is screened according to the correlation between the question information and the scene description information, thereby reducing the illusion effect of information redundancy on the model, making the information input into the language model highly relevant, and improving the accuracy of the language model's answer.

[0076] In some embodiments, before inputting the scene description information and the question information into the language model, the method further includes: Prompt examples are input into the language model so that the language model learns the output pattern in the embodied scenario; prompt examples include questions, scenario descriptions, and answers corresponding to the questions.

[0077] In this embodiment, the prompt examples are one or more sets of data that include questions, scenario descriptions, and corresponding answers. These examples can be used to train or adjust a language model so that the language model can learn and understand the questions in the embodied scenario and generate appropriate answers. By learning these examples, the language model understands how to combine questions with scenario description information in the embodied scenario to generate accurate answers. By learning the prompt examples, the generalization ability of the language model is improved, enabling it to generate reasonable answers based on the scenario description information even when faced with new and unseen questions.

[0078] In one example, the prompt examples are as follows: [input] [Q]What color is the potted plant on the ground? [Common sense knowledge] The explanations of "地" are as follows: "地" has two pronunciations: dì and de. It can be used as an auxiliary word after a verb or as a noun, representing the earth (referring to the lowest part of the earth's surface; the part close to the earth's crust), the earth, etc. Pinyin: dì de, Pronunciation: ㄉㄧˋ(ㄉㄜ), Radical: 土, Structure: left-right, Total strokes: 6, Strokes outside the radical: 3, Stroke order number: 121525, Cangjie: GPD, Four corner numbers: 4411, Wubi86: FBN, Wubi98: FBN. The author of "What Color is Love" is Kenuoyi. The classification of potted plants (traditional art form) is tree planting and landscape bonsai. The explanation of potted plants is potted plants expressing landscape scenery. The title of the book "Potted Flowers" is Potted Flowers. The varieties of potted vegetables are lettuce, scallions, and coriander. The colors of floor paint are various. The series of "Colored Illustrations of Potted Flowers" is Practical Horticulture Series. The feature of small potted plants is landscape scenery.

[0079] [Memory] [Scene information] There is 1 potted plant in front of you, located on the floor. Its color is green. There are 2 black mirrors, 1 black refrigerator, 2 black spherical apples, 1 red cabinet, 1 black door, 1 black radio, 2 green plants, 1 black dining table, 6 black chandeliers, 2 white windows, 1 red hanging painting, 1 purple carpet, 1 black cup, 1 black remote control, 1 brown range hood, 1 brown plant, 1 brown rectangular coffee table, 4 black chairs, 1 black vase, 1 red waste paper basket, 1 living room, 1 black TV, 1 gray TV cabinet, 1 green mat, 1 red cabinet. Behind it, there are 1 black plant, 1 black sofa stool, 1 brown cabinet, 1 black sofa, 1 blue-green plant, 1 pillow, 1 black towel, 1 black sink, 6 black chandeliers, 1 brown towel rack, 2 red curtains. [output] [R]The potted plants on the ground are green.

[0080] In this embodiment, the retrieval part of the language model adopts a multi-path recall form, and these paths can include encyclopedic knowledge graphs, historical action information, scene description inputs, etc. Therefore, common sense knowledge can be common sense knowledge recalled from the encyclopedic knowledge graph based on the entities in the question; memory is the historical action information that the robot has done in the conversation; and scene information is the scene description input provided by the solution. Finally, the language model generates answers based on the above retrieval-enhanced information and combines its own powerful reasoning and summarization capabilities.

[0081] In this embodiment, by providing specific scenarios and question examples, the model's understanding and generation capabilities for specific tasks can be enhanced, enabling the language model to learn and imitate output patterns in embodied scenarios, thereby improving the accuracy and relevance of its responses in similar situations.

[0082] The embodied scene question-answering method based on retrieval enhancement generation provided in the embodiment of the present application can be executed by an embodied scene question-answering device based on retrieval enhancement generation. In the embodiment of the present application, the embodied scene question-answering device based on retrieval enhancement generation executes the embodied scene question-answering method based on retrieval enhancement generation as an example to illustrate the embodied scene question-answering device based on retrieval enhancement generation provided in the embodiment of the present application.

[0083] The embodiment of the present application also provides an embodied scenario question and answer device based on retrieval enhanced generation.

[0084] like Figure 2 As shown, the embodied scenario question-answering device based on retrieval enhancement generation includes: A matching module 210, used to match target objects satisfying the question information from the scene graph; A generating module 220, configured to generate scene description information of a target object according to visual information of the target object in a scene graph; The input module 230 is used to input the scene description information and the question information into the language model to obtain the response information for the question information output by the language model.

[0085] According to the embodied scene question-answering device based on retrieval enhancement generation of the present application, by matching the target object that satisfies the question information from the scene graph; generating the scene description information of the target object based on the visual information of the target object in the scene graph; inputting the scene description information and the question information into the language model, the language model outputs the reply information for the question information. The embodiment of the present application is based on retrieval enhancement technology by retrieving the target object related to the question from the scene graph, and then converting the visual perception information of the target object in the scene graph into natural language information, and using this information as the context input of the language model to enhance the language model's ability to understand and answer the question information. Even when encountering questions and answers that require complex reasoning, more accurate answers can be obtained, thereby improving the accuracy of the questions and answers.

[0086] In some embodiments, the matching module 210 is further configured to: Performing structured extraction on the question information to obtain first triplet information including entities in the question information and positional relationships between the entities; According to the first triplet, a target object that satisfies the question information is matched from the scene graph.

[0087] In some embodiments, the matching module 210 is further configured to: Parsing the directional relationship between entities in the first triplet; Match the target objects that meet the question information from the scene graph based on entity and orientation relationships.

[0088] In some embodiments, the generating module 220 is further configured to: Supplementing the first triplet information according to the visual information of the target object in the scene graph to obtain the second triplet information; The second triplet information is converted into scene description information.

[0089] In some embodiments, the generating module 220 is further configured to: The second triplet information is filled into a preset natural language template to obtain scene description information.

[0090] In some embodiments, the generating module 220 is further configured to: The scene description information is filtered and processed according to the relevance between the question information and the scene description information.

[0091] In some embodiments, the input module 230 is further configured to: Prompt examples are input into the language model so that the language model learns the output pattern in the embodied scenario; prompt examples include questions, scenario descriptions, and answers corresponding to the questions.

[0092] The embodied scene question-answering device based on retrieval enhancement generation in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0093] The embodied scenario question-answering device based on search enhancement generation in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0094] In some embodiments, Figure 3 As shown, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, each process of the embodiment of the embodied scenario question and answer method based on retrieval-enhanced generation is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0095] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0096] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned embodiment of the embodied scenario question and answer method based on retrieval enhancement generation are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0097] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0098] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned embodied scenario question and answer method based on retrieval enhancement generation.

[0099] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0100] An embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the embodied scenario question and answer method based on retrieval enhancement generation, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0101] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0102] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0104] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present application, all of which are within the protection of the present application.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. An embodied scenario question answering method based on retrieval-enhanced generation, characterized in that: include: Match the target object that satisfies the question information from the scene graph; Generate scene description information of the target object according to visual information of the target object in the scene graph; The scene description information and the question information are input into a language model to obtain response information for the question information output by the language model.

2. The method according to claim 1, characterized in that The matching of the target object satisfying the question information from the scene graph includes: Performing structured extraction on the question information to obtain first triplet information including entities in the question information and positional relationships between entities; A target object that satisfies the question information is matched from the scene graph according to the first triple.

3. The method according to claim 2, characterized in that The matching of the target object satisfying the question information from the scene graph according to the first triplet includes: parsing the directional relationship between entities in the first triplet; A target object that satisfies the question information is matched from the scene graph according to the entity and the orientation relationship.

4. The method according to claim 2, characterized in that: The step of generating scene description information of the target object according to visual information of the target object in the scene graph comprises: Supplementing the first triplet of information according to the visual information of the target object in the scene graph to obtain second triplet of information; The second triplet information is converted into the scene description information.

5. The method according to claim 4, characterized in that The converting the second triplet information into the scene description information includes: The second triplet information is filled into a preset natural language template to obtain the scene description information.

6. The method according to claim 1, characterized in that Before inputting the scene description information and the question information into the language model, the method further includes: The scene description information is filtered according to the relevance between the question information and the scene description information.

7. The method according to claim 1, characterized in that Before inputting the scene description information and the question information into the language model, the method further includes: Inputting prompt examples into a language model so that the language model learns output patterns in embodied scenarios; the prompt examples include questions, scenario descriptions, and answers corresponding to the questions.

8. An embodied scenario question-answering device based on retrieval-enhanced generation, characterized in that: include: A matching module, used to match target objects that meet the problem information from the scene graph; A generating module, used for generating scene description information of the target object according to visual information of the target object in the scene graph; An input module is used to input the scene description information and the question information into a language model to obtain response information for the question information output by the language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images