Question and answer method, device and equipment based on scene fuzzy perception and reasoning and medium

Abstract

The invention discloses a question answering method and device based on scene fuzzy perception and reasoning, equipment and a medium, and the method comprises the steps: obtaining a picture frame of a scene for fuzzy perception, and obtaining a key element which comprises the appearance attribute and spatial position of a target object; reasoning and association are carried out based on the reasoning association knowledge base, derived elements are obtained, and the derived elements comprise function attributes of the target object and objects associated with the target object; according to the key elements and the derived elements, dynamic scene cognition is obtained; questions related to the scene are obtained, and answers of the questions are generated by adopting a large language model based on dynamic scene cognition; and if the answer does not meet the question demand, determining an optimal sensing path according to the question and the dynamic scene cognition, updating the dynamic scene cognition and the answer based on the optimal sensing path until the updated answer meets the question demand, and outputting the current answer. According to the invention, more efficient and accurate scene perception can be realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning. Background Art

[0002] With the rapid development of intelligent robot technology, the perception and cognitive ability of robots in complex environments has gradually become a research focus. At present, robot perception technology mainly includes: first, the method based on artificially set rules, which mainly extracts environmental information for specific tasks; second, the deep learning method based on reinforcement learning, which optimizes the environmental adaptability of robots when performing specific tasks through reward mechanisms; third, the perception method based on large language models, which enables robots to have the generalization of tasks and environments through the powerful reasoning ability of large language models.

[0003] However, existing rules and deep learning methods are difficult to cope with open environments and changing tasks. Although large language models have certain environmental generalization and task generalization, they rely too much on the reasoning ability of the large model itself and lack the level of cognition of the environment. When performing tasks, the large model can easily produce perceptual paths that are irrelevant to the scene, resulting in low perception efficiency. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning, which enhances the robot's reasoning and association capabilities when performing perception tasks, achieves more efficient and accurate scene perception, and is suitable for complex tasks in open environments.

[0005] The embodiment of the present invention provides a question-answering method based on scene blur perception and reasoning, including:

[0006] Acquire several picture frames of the scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0007] Based on a preset reasoning and associative knowledge base, the key elements are inferred and associated to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0008] Obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames;

[0009] Obtaining questions related to the scenario, parsing the questions, and obtaining structured question expressions;

[0010] Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question;

[0011] Determine whether the answer meets the requirements of the question; if so, output the current answer; if not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the requirements of the question, and output the current answer.

[0012] As an improvement of the above solution, the method for constructing the preset reasoning association knowledge base includes:

[0013] Acquire general scenario knowledge;

[0014] Based on the general scene knowledge, a large language model is used for reasoning and association to respectively construct a reasoning knowledge base and an association knowledge base; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0015] The inference knowledge base and the association knowledge base are aggregated to obtain an inference association knowledge base.

[0016] As an improvement of the above solution, the step of acquiring a plurality of picture frames of a scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes:

[0017] Get several frames of images observed by the agent as it moves in the scene;

[0018] Using a multimodal visual language model to extract semantic information of the picture frame;

[0019] Based on the semantic information, key elements of the picture frame are obtained.

[0020] As an improvement of the above solution, obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames includes:

[0021] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0022] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0023] As an improvement of the above solution, the problem is parsed to obtain a structured problem expression, including:

[0024] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, which include: essentiality, basic needs, potential needs and deep meaning;

[0025] A structured question expression is generated according to the semantic features corresponding to the question.

[0026] As an improvement of the above solution, the step of judging whether the answer meets the question requirement includes:

[0027] Calculate the solution score of the answer according to the question and the answer;

[0028] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0029] If the solution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0030] As an improvement of the above solution, determining the optimal perception path of the question according to the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes:

[0031] According to the problem and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the problem; the optimal perception path includes data of time dimension, space dimension and semantic dimension;

[0032] Acquire a number of picture frames observed by the agent when moving according to the optimal perception path, and determine the key elements and derived elements in each picture frame to obtain a scene knowledge subgraph of each picture frame;

[0033] According to the key elements, spatiotemporally aggregating scene knowledge subgraphs of a plurality of picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0034] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0035] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0036] The embodiment of the present invention further provides a question-answering device based on scene blur perception and reasoning, comprising:

[0037] A fuzzy perception module is used to obtain a plurality of picture frames of a scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0038] An inference and association module, used to infer and associate the key elements based on a preset inference and association knowledge base to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0039] A dynamic scene recognition module, used for obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames;

[0040] A question analysis module is used to obtain questions related to the scenario, analyze the questions, and obtain a structured question expression;

[0041] An answer generation module, configured to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression;

[0042] The scene cognition update module is used to determine whether the answer meets the question requirements; if so, output the current answer; if not, determine the optimal perception path of the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements and outputs the current answer.

[0043] An embodiment of the present invention also provides a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor, and when the processor executes the computer program, the question-answering method based on scene blur perception and reasoning as described in any one of the above items is implemented.

[0044] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned question-answering methods based on scene blur perception and reasoning.

[0045] Compared with the prior art, the question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning provided by the embodiments of the present invention have the following beneficial effects: by performing fuzzy perception on the picture frame to obtain key elements, and then performing reasoning and association based on the reasoning association knowledge base to obtain derived elements, and then forming dynamic scene cognition, it can avoid fine recognition of the scene, improve perception efficiency, and reduce information redundancy; by constructing dynamic scene cognition and generating question answers based on dynamic scene cognition, it overcomes the poor generalization of methods such as artificially set rules and reinforcement learning, and the problem that large models rely on their own reasoning ability but lack environmental cognition, and improves environmental adaptability and task generalization. The embodiments of the present invention can enable the intelligent agent to identify objects and understand the environment more quickly and accurately, achieve more efficient and complete scene cognition construction, and then provide a problem-solving solution that better meets user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of a question-answering method based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0047] Figure 2 It is a flowchart of a question-answering method based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0048] Figure 3 is a structural schematic diagram of a question-answering device based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0049] Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The existing methods of artificially setting rules and deep learning methods represented by reinforcement learning are not adaptable to open environments and open tasks when performing perception tasks. Such methods are mainly used in a specific field, while in our daily lives, the tasks and environments we face are diverse and even unpredictable. Therefore, such rule- or data-driven methods have poor generalization and cannot work well in such an open and dynamic environment for a long time. As for the methods of using large models for scene perception, they usually rely too much on the reasoning ability of the large models themselves and lack the level of cognition of the environment. When performing tasks, the large models are prone to generate perception paths that are irrelevant to the scene in which they are located, resulting in low perception efficiency.

[0052] In addition, the current robot cognitive construction methods for the environment include methods based on large visual language model description and methods based on scene graph generation to establish cognition of the entire scene. The method based on large visual language model description is to obtain the text description of the picture at the current perspective through the prompt word method during the movement of the robot, and then record it; the method based on scene graph generation is to perform structured semantic body extraction of the scene during the movement of the robot, obtain the spatial or semantic relationship between the elements in the scene, and complete the cognitive representation of the scene. However, whether it is a method based on visual text description or a method based on scene graph generation, it is hoped that all elements in the scene can be perceived in a fine-grained manner to obtain accurate scene knowledge, but there are two problems. On the one hand, the scene knowledge cannot be enumerated, and on the other hand, the speed and efficiency are very low, and it is easy to be limited by the model recognition ability, and it is easy to misidentify objects and stack objects from different perspectives, resulting in information explosion. For example, the keyboard is considered to be a black box from a certain perspective, and the keyboard is considered to be a black mouse pad from another perspective. In this case, the scene cognition generated contains a lot of noise, resulting in multiple incorrect representations of an object in the scene knowledge. Also, if the robot tries to perceive all the detailed information in the scene in order to support subsequent tasks, the speed will be very slow and the efficiency will be low.

[0053] Therefore, an embodiment of the present invention provides a question-answering method based on scene fuzzy perception and reasoning, which has reasoning and association capabilities when constructing scene cognition.

[0054] See also Figure 1 , Figure 1 1 is a flow chart of a question-answering method based on scene blur perception and reasoning provided by an embodiment of the present invention. The question-answering method based on scene blur perception and reasoning includes:

[0055] S1: Acquire several picture frames of a scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0056] S2: Based on a preset reasoning and association knowledge base, the key elements are inferred and associated to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0057] S3: obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames;

[0058] S4: Obtaining questions related to the scenario, parsing the questions, and obtaining structured question expressions;

[0059] S5: Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question;

[0060] S6: Determine whether the answer meets the requirements of the question; if so, output the current answer; if not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the requirements of the question, and output the current answer.

[0061] Specifically, when constructing scene cognition, the key elements of the target object in the scene are first identified, that is, based on the picture frames collected by the intelligent agent's movement, the objects (i.e., the target objects) in the picture frames are identified to obtain the appearance attributes and spatial position of the objects. Then, the key elements of the object are inferred and associated to obtain the corresponding derived elements, including the functional use of the object inferred based on the appearance attributes, possible semantic names, and other objects associated with the object but not identified in the picture; in order to achieve efficient and accurate reasoning and association, a reasoning and association knowledge base is pre-designed to ensure that the reasoning and association process is consistent with human and scene experience. During the movement of the intelligent agent, the key elements and derived elements of the acquired picture frames are continuously extracted and fused in time and space to obtain a holistic dynamic scene cognition with rich reasoning and association space.

[0062] It should be noted that when the agent has no mobile task, a simple patrol process can be set to build the current dynamic scene cognition when the agent moves naturally, so that when a new task (i.e., a problem related to the scene) arrives, a solution can be generated based on the current dynamic scene cognition. On the other hand, during the movement of each task performed by the agent, the newly recognized scene cognition is updated to the current dynamic scene cognition based on the information in the real-time collected picture frames.

[0063] When a user asks a scenario-related question, the question is first parsed to obtain the essential content and requirements of the question, and then the constructed dynamic scenario cognition and large language model are used to generate a scenario-based answer. The generated answer is then evaluated to determine whether it can meet the requirements of the question. If the answer has met the requirements, the answer is returned to the user; if the answer does not meet the requirements, the optimal perception path for re-perception is generated based on the question and the current dynamic scenario cognition, forming a new perception task, so that the intelligent agent moves according to the optimal perception path, obtains new scenario cognition and aggregates it into the original dynamic scenario cognition to obtain the current dynamic scenario cognition. Finally, based on the current dynamic scenario cognition, the large language model is used to regenerate the answer to the question and evaluate whether it meets the requirements. When the requirements of the question are met, the current answer is returned to the user.

[0064] As one of the optional embodiments, the method for constructing the preset reasoning association knowledge base includes:

[0065] Acquire general scenario knowledge;

[0066] Based on the general scene knowledge, a large language model is used for reasoning and association to respectively construct a reasoning knowledge base and an association knowledge base; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0067] The inference knowledge base and the association knowledge base are aggregated to obtain an inference association knowledge base.

[0068] Specifically, see Figure 2 First, we acquire general scene knowledge and build an expert knowledge base related to the scene, which includes a large amount of common sense knowledge. For example, the workbench cognitive group includes desks, chairs, computers, etc., and the computer cognitive group includes computers, keyboards, etc. Then, by setting expert instructions, the large language model can output the extracted knowledge content according to the needs based on the general scene knowledge, thereby building a reasoning knowledge base and an associative knowledge base, and then further aggregate these two knowledge bases to obtain a reasoning associative knowledge base.

[0069] Among them, inference knowledge refers to the functional use attributes inferred from the basic attributes of objects. For example, books have the attribute of hardness, which can be inferred that books can not only be used for reading, but also can be used to support mobile phones and notebooks; clothes can not only keep warm, but their flexible characteristics can also protect the desktop from scratches and protect privacy. Associative knowledge is represented as a set of strong and weak associations, reflecting the association relationship between objects. For example, a computer monitor is strongly associated with a keyboard, mouse, and power adapter, and weakly associated with a USB flash drive, USB data cable, etc.; a bed is strongly associated with bedding and pillows; and a study area is strongly associated with books, schoolbags, etc.

[0070] As one of the optional embodiments, the acquiring of a plurality of picture frames of a scene and performing blur perception on the picture frames to obtain key elements of the picture frames includes:

[0071] Get several frames of images observed by the agent as it moves in the scene;

[0072] Using a multimodal visual language model to extract semantic information of the picture frame;

[0073] Based on the semantic information, key elements of the picture frame are obtained.

[0074] As one of the optional embodiments, obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames includes:

[0075] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0076] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0077] Specifically, see Figure 2 When the agent moves in the scene, it collects environmental information and obtains a video sequence. By fuzzy perceiving the objects in the picture frames of the video sequence, it extracts scene knowledge to construct scene cognition. In order to ensure time efficiency, fuzzy perception is reflected in perceiving only some objects in the scene and not paying attention to the real semantic labels of the objects, such as: "black cylindrical, small object, placed on the bed, container or blunt object, like a box". Specifically, a multimodal visual language model VLM is first trained, which has the ability to extract key elements of objects in the scene and their specific semantic information. The VLM is trained by preset pairs of pictures and text data. For the picture frame currently observed by the intelligent agent, the VLM is used to extract the semantic information in the picture frame to obtain the key elements in the picture frame. Then, the reasoning and association knowledge base constructed as above is used to use a large language model to perform reasoning and association on the basis of the key elements of the object to obtain the derived elements of the object, that is, the function, purpose, possible name and other information of the object are inferred, as well as the objects that may exist in the scene but are not perceived in the picture frame (such as a mouse hidden behind the display screen, which is associated with the display screen); then the scene knowledge subgraph corresponding to the picture frame is generated from the key elements and derived elements of the picture frame.

[0078] Furthermore, based on the consistency of key elements (such as shape, color, relative position, etc.) of the same observed object in different picture frames in the scene, the same observed object from different perspectives in the scene is aligned, and the spatiotemporal aggregation of the scene knowledge subgraphs is performed to finally obtain complete dynamic scene cognition.

[0079] For example, at time t0, object A is observed: "black, cylindrical object, serves as a container, located on the left side of the bed, like a box", and at time t1, object B is observed: "black, cylindrical object, serves as a speaker, located on the left side of the bed, like a speaker". By judging the similarity of the attributes of object A and object B, it can be found that the similarity between object A and object B is very high, and it is believed that A and B are the same object. Their semantic names (box and speaker) may be caused by model recognition errors. Then, these two objects are aggregated in the scene cognition structure to obtain scene knowledge: there is object A on the bed: "black, cylindrical object, serves as a speaker, serves as a container, located on the left side of the bed, like a box / speaker". Furthermore, if object C is also observed on the bed at time t1, scene cognition can be obtained after aggregation: there are objects A and C on the bed.

[0080] It should be noted that when constructing dynamic scene cognition, for large objects commonly seen in the scene, such as beds, tables, sofas and other large furniture, their real names are directly perceived without fuzzy processing; only small objects and objects that are not within the preset common object range are fuzzy perceived (that is, only their attributes are perceived without recognizing their real names), thereby reducing the spatiotemporal alignment space of the objects and greatly reducing the impact of model illusions. Fuzzy processing is performed on small objects, on the one hand due to insufficient resolution, and on the other hand due to the diversity of small objects. For example, if the real name of a speaker in the shape of a potted plant is directly labeled on such objects, it is easy to misidentify the object and cause objects from different perspectives to stack, resulting in information explosion.

[0081] Among them, the overall dynamic scene cognition that is finally formed is hierarchical, for example: V0: home scene {V1: in the bedroom there is {V2: on the bed there is {V3: schoolbag (attributes), V3: pillow (attributes), …}, …}, V1: in the living room there is …}.

[0082] As one of the optional embodiments, parsing the question to obtain a structured question expression includes:

[0083] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, which include: essentiality, basic needs, potential needs and deep meaning;

[0084] A structured question expression is generated according to the semantic features corresponding to the question.

[0085] Specifically, the large language model is pre-trained and fine-tuned to enable it to parse the intent of any scenario problem. When a user has a scenario problem to be solved, the large language model is used to parse the intent of the problem to obtain the four-dimensional semantic features of the essence, basic needs, potential needs and deep meaning of the scenario problem, so as to generate the corresponding structured problem expression. Then, the problem expression and the current dynamic scenario cognition are input into the large language model, and the large language model is used for semantic reasoning and calculation to obtain the answer to the question combined with the scenario.

[0086] As one of the optional embodiments, the determining whether the answer meets the question requirement includes:

[0087] Calculate the solution score of the answer according to the question and the answer;

[0088] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0089] If the solution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0090] Specifically, after getting the answer to the question, the degree of resolution is calculated: Score = resolution (Result, Question). If the resolution score is greater than a preset threshold, the question is considered to be resolved; if the resolution score is less than a preset threshold, the question is considered to be unresolved and a new answer needs to be generated. The large language model can be used to calculate the degree of resolution.

[0091] As one of the optional embodiments, determining the optimal perception path of the question according to the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes:

[0092] According to the problem and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the problem; the optimal perception path includes data of time dimension, space dimension and semantic dimension;

[0093] Acquire a number of picture frames observed by the agent when moving according to the optimal perception path, and determine the key elements and derived elements in each picture frame to obtain a scene knowledge subgraph of each picture frame;

[0094] According to the key elements, spatiotemporally aggregating scene knowledge subgraphs of a plurality of picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0095] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0096] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0097] Specifically, when the answer does not meet the requirements of the question and needs to be regenerated, the optimal perception path is first generated based on the question and the current dynamic scene cognition, so that the agent can collect environmental data according to the optimal perception path. Among them, the optimal perception path includes information on the time dimension, space dimension and semantic dimension. For example, for the question "What is the name of the book", when perceiving the scene before, the name of the book was not paid attention to, and it is difficult to get the answer through reasoning. Therefore, according to the scene cognition, a re-perception path is given: "V1: living room, V2: second layer of the coffee table, V3: books, pay attention to the name of the book; navigation coordinates (x, y); time coordinates (ti) (for video positioning)", and then guide the agent to observe again according to the path, and dynamically update the new scene knowledge obtained: "The name of the book "XX"" to the overall scene cognition.

[0098] Finally, the dynamic scene cognition is updated according to the optimal perception path, and the answer to the question is regenerated according to the updated dynamic scene cognition until the answer meets the requirements of the question and the answer is output to complete the task.

[0099] Compared with other methods, other methods store the scene knowledge extracted at each moment in the database, and generate results by knowledge retrieval each time the scene reasoning is performed, which is not conducive to combining the overall scene. For example: for the question "I am cold", other methods may only retrieve one solution through retrieval: "The quilt on the bed can keep you warm", while the method of the embodiment of the present invention can provide multiple solutions by integrating overall cognition: "The quilt on the bed, the blanket and pillow on the sofa, and the clothes on the hanger can keep you warm", which can provide more intelligent scene questions and answers.

[0100] The embodiment of the present invention divides scene cognition into key elements and derived elements. Only some key elements need to be perceived, and then derived elements are derived through reasoning and association mechanisms, thereby avoiding fine-grained perception of all objects in the scene and reducing information redundancy and model illusion.

[0101] By adopting a fuzzy perception strategy, there is no need to finely identify all elements in the scene, which significantly improves perception efficiency. Combined with the reasoning association mechanism, even if only part of the information is perceived, it will not cause omission of key scene information, ensuring that the intelligent agent can quickly and accurately understand the environment;

[0102] By taking time into account as important implicit information during the movement of the agent, scene cognition is made more in line with the actual situation of the dynamic environment;

[0103] Object alignment is performed by associating the attributes of the same object in different image frames rather than aligning objects by semantic names, which effectively reduces the impact of model hallucinations and improves the accuracy of object recognition;

[0104] The embodiments of the present invention have stronger environmental adaptability and task generalization, can build scene cognition more efficiently, have richer scene information, high scene perception efficiency and strong reasoning and association capabilities, which is conducive to providing more accurate problem-solving solutions.

[0105] Correspondingly, the present invention also provides a question-answering device based on scene blur perception and reasoning, which can implement all processes of the question-answering method based on scene blur perception and reasoning in the above embodiment.

[0106] See also Figure 3 , Figure 3 : is a schematic diagram of a question-answering device based on scene blur perception and reasoning provided by an embodiment of the present invention. The question-answering device based on scene blur perception and reasoning comprises:

[0107] The fuzzy perception module 301 is used to obtain a plurality of picture frames of a scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0108] The reasoning and association module 302 is used to perform reasoning and association on the key elements based on a preset reasoning and association knowledge base to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0109] A dynamic scene recognition module 303, configured to obtain dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames;

[0110] A question analysis module 304 is used to obtain questions related to the scenario, analyze the questions, and obtain a structured question expression;

[0111] An answer generation module 305 is used to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression;

[0112] The scene cognition update module 306 is used to determine whether the answer meets the requirements of the question; if so, output the current answer; if not, determine the optimal perception path of the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the requirements of the question and outputs the current answer.

[0113] Preferably, the method for constructing the preset reasoning association knowledge base includes:

[0114] Acquire general scenario knowledge;

[0115] Based on the general scene knowledge, a large language model is used for reasoning and association to respectively construct a reasoning knowledge base and an association knowledge base; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0116] The inference knowledge base and the association knowledge base are aggregated to obtain an inference association knowledge base.

[0117] Preferably, the acquiring of a plurality of picture frames of a scene and performing blur perception on the picture frames to obtain key elements of the picture frames includes:

[0118] Get several frames of images observed by the agent as it moves in the scene;

[0119] Using a multimodal visual language model to extract semantic information of the picture frame;

[0120] Based on the semantic information, key elements of the picture frame are obtained.

[0121] Preferably, obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames includes:

[0122] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0123] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0124] Preferably, the problem is parsed to obtain a structured problem expression, including:

[0125] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, which include: essentiality, basic needs, potential needs and deep meaning;

[0126] A structured question expression is generated according to the semantic features corresponding to the question.

[0127] Preferably, the determining whether the answer meets the question requirement includes:

[0128] Calculate the solution score of the answer according to the question and the answer;

[0129] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0130] If the solution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0131] Preferably, determining the optimal perception path for the question according to the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes:

[0132] According to the problem and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the problem; the optimal perception path includes data of time dimension, space dimension and semantic dimension;

[0133] Acquire a number of picture frames observed by the agent when moving according to the optimal perception path, and determine the key elements and derived elements in each picture frame to obtain a scene knowledge subgraph of each picture frame;

[0134] According to the key elements, spatiotemporally aggregating scene knowledge subgraphs of a plurality of picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0135] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0136] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0137] In the specific implementation, the working principle, control process and technical effect of the question-answering device based on scene blur perception and reasoning provided by the embodiment of the present invention are the same as those of the question-answering method based on scene blur perception and reasoning in the above-mentioned embodiment, and will not be repeated here.

[0138] See also Figure 4 , Figure 4 4 is a structural block diagram of a computer device provided in an embodiment of the present invention, and the computer device includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program, the steps in the above-mentioned question-answering method embodiment based on scene blur perception and reasoning are implemented. Alternatively, when the processor 401 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0139] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device.

[0140] The computer device may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device, and may include more or fewer components than shown in the diagram, or may combine certain components, or different components, for example, the computer device may also include an input / output device, a network access device, a bus, etc.

[0141] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0142] The memory 402 can be used to store the computer program and / or module, and the processor 401 realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory 402, and calling the data stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0143] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 401. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0144] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the question-answering method based on scene blur perception and reasoning described in any of the above embodiments.

[0145] The embodiments of the present invention provide a question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning, and its beneficial effects are: by fuzzy perception of picture frames to obtain key elements, and then reasoning and association based on the reasoning association knowledge base to obtain derived elements, and then form dynamic scene cognition, it can avoid fine recognition of the scene, improve perception efficiency, and reduce information redundancy; by constructing dynamic scene cognition and generating question answers based on dynamic scene cognition, it overcomes the poor generalization of methods such as artificially set rules and reinforcement learning, and the problem that large models rely on their own reasoning ability but lack environmental cognition, and improves environmental adaptability and task generalization. The embodiments of the present invention can enable intelligent agents to identify objects and understand the environment more quickly and accurately, achieve more efficient and complete scene cognition construction, and then provide problem-solving solutions that better meet user needs.

[0146] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A question-answering method based on scene fuzzy perception and reasoning, characterized in that: include: Acquire several picture frames of the scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; The key elements include appearance attributes and spatial position of the target object; Based on a preset reasoning and associative knowledge base, the key elements are inferred and associated to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object; Obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames; Obtaining questions related to the scenario, parsing the questions, and obtaining structured question expressions; Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question; Determine whether the answer meets the question requirements; if so, output the current answer; If not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements and outputs the current answer.

2. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 1, characterized in that: The method for constructing the preset reasoning association knowledge base includes: Acquire general scenario knowledge; Based on the general scene knowledge, a large language model is used for reasoning and association to respectively construct a reasoning knowledge base and an association knowledge base; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene; The inference knowledge base and the association knowledge base are aggregated to obtain an inference association knowledge base.

3. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 1, characterized in that: The acquiring of a plurality of picture frames of the scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes: Get several frames of images observed by the agent as it moves in the scene; Using a multimodal visual language model to extract semantic information of the picture frame; Based on the semantic information, key elements of the picture frame are obtained.

4. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 3, characterized in that: The obtaining of dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames includes: For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame; According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

5. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 1, characterized in that: The problem is parsed to obtain a structured problem expression, including: Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, which include: essentiality, basic needs, potential needs and deep meaning; A structured question expression is generated according to the semantic features corresponding to the question.

6. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 1, characterized in that: The determining whether the answer meets the question requirement includes: Calculate the solution score of the answer according to the question and the answer; If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements; If the solution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

7. The question-answering method based on scene fuzzy perception and reasoning as claimed in claim 1, characterized in that: The step of determining an optimal perception path for the question according to the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes: According to the problem and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the problem; the optimal perception path includes data of time dimension, space dimension and semantic dimension; Acquire a number of picture frames observed by the agent when moving according to the optimal perception path, and determine the key elements and derived elements in each picture frame to obtain a scene knowledge subgraph of each picture frame; According to the key elements, spatiotemporally aggregating scene knowledge subgraphs of a plurality of picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path; Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition; An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

8. A question-answering device based on scene fuzzy perception and reasoning, characterized in that: include: A fuzzy perception module is used to obtain a plurality of picture frames of a scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; The key elements include appearance attributes and spatial position of the target object; An inference and association module, used to infer and associate the key elements based on a preset inference and association knowledge base to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object; A dynamic scene recognition module, used for obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames; A question analysis module is used to obtain questions related to the scenario, analyze the questions, and obtain a structured question expression; An answer generation module, configured to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression; A scene recognition update module is used to determine whether the answer meets the question requirements; if so, output the current answer; If not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements and outputs the current answer.

9. A computer device, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor, and when the processor executes the computer program, the question-answering method based on scene blur perception and reasoning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the question-answering method based on scene blur perception and reasoning as described in any one of claims 1 to 7 is implemented.

Images