Question answering method, device, equipment and medium based on scene fuzzy perception and reasoning

Abstract

The present invention discloses a question-answering method, apparatus, device, and medium based on scene fuzzy perception and reasoning. The method comprises: acquiring a picture frame of a scene for fuzzy perception to obtain key elements, which include the appearance attributes and spatial position of a target object; performing reasoning and association based on a reasoning association knowledge base to obtain derived elements, which include the functional attributes of the target object and objects associated with the target object; obtaining dynamic scene cognition based on the key elements and derived elements; acquiring questions related to the scene, and generating answers to the questions using a large language model based on the dynamic scene cognition; if the answer does not meet the question requirements, determining an optimal perception path based on the question and dynamic scene cognition, and updating the dynamic scene cognition and answer based on the optimal perception path until the updated answer meets the question requirements, and outputting the current answer. The present invention can achieve more efficient and accurate scene perception.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a question-answering method, apparatus, device, and storage medium based on scene fuzzy perception and reasoning. Background Art

[0002] With the rapid development of intelligent robotics, the perception and cognitive capabilities of robots in complex environments have gradually become a research focus. Currently, robotic perception technologies mainly include: First, methods based on manually set rules, which mainly extract environmental information for specific tasks; second, deep learning methods based on reinforcement learning, which optimize the robot's environmental adaptability when performing specific tasks through reward mechanisms; and third, perception methods based on large language models, which leverage the powerful reasoning capabilities of large language models to enable robots to generalize across tasks and environments.

[0003] However, existing rules and deep learning methods are difficult to cope with open environments and changing tasks. Although large language models have certain environmental generalization and task generalization, they rely too much on the reasoning ability of the large models themselves and lack the level of cognition of the environment. When performing tasks, large models can easily produce perception paths that are irrelevant to the scene they are in, resulting in low perception efficiency. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning, which enhances the robot's reasoning and association capabilities when performing perception tasks, achieves more efficient and accurate scene perception, and is suitable for complex tasks in open environments.

[0005] An embodiment of the present invention provides a question-answering method based on scene blur perception and reasoning, including:

[0006] Acquire several picture frames of the scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0007] Based on a preset reasoning and association knowledge base, the key elements are inferred and associated to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0008] Obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames;

[0009] Obtaining questions related to the scenario, parsing the questions, and obtaining structured question expressions;

[0010] Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question;

[0011] Determine whether the answer meets the question requirements; if so, output the current answer; if not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements, and output the current answer.

[0012] As an improvement to the above solution, the method for constructing the preset reasoning association knowledge base includes:

[0013] Acquire general scenario knowledge;

[0014] Based on the general scene knowledge, a large language model is used for reasoning and association to construct a reasoning knowledge base and an association knowledge base respectively; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0015] The reasoning knowledge base and the associative knowledge base are aggregated to obtain a reasoning associative knowledge base.

[0016] As an improvement to the above solution, the method of acquiring several picture frames of a scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes:

[0017] Get several frames of images observed by the agent as it moves in the scene;

[0018] Using a multimodal visual language model to extract semantic information of the image frame;

[0019] Based on the semantic information, key elements of the picture frame are obtained.

[0020] As an improvement to the above solution, obtaining dynamic scene recognition based on the key elements and the derived elements of the plurality of picture frames includes:

[0021] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0022] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0023] As an improvement to the above solution, the problem is parsed to obtain a structured problem expression, including:

[0024] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, including: essentiality, basic needs, potential needs, and deep meaning;

[0025] A structured question expression is generated according to the semantic features corresponding to the question.

[0026] As an improvement to the above solution, the step of determining whether the answer satisfies the question requirement includes:

[0027] Calculating a resolution score of the answer based on the question and the answer;

[0028] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0029] If the resolution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0030] As an improvement to the above solution, determining the optimal perception path for the question based on the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path, includes:

[0031] Based on the question and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the question; the optimal perception path includes data in the time dimension, the space dimension, and the semantic dimension;

[0032] Acquire a plurality of image frames observed by the agent while moving along the optimal perception path, and determine key elements and derived elements in each image frame to obtain a scene knowledge subgraph for each image frame;

[0033] Based on the key elements, spatiotemporally aggregating scene knowledge subgraphs of several picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0034] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0035] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0036] An embodiment of the present invention further provides a question-answering device based on scene blur perception and reasoning, comprising:

[0037] A fuzzy perception module is used to obtain a plurality of picture frames of a scene and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include the appearance attributes and spatial position of the target object;

[0038] An inference and association module is used to infer and associate the key elements based on a preset inference and association knowledge base to obtain derived elements of the image frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0039] a dynamic scene recognition module, configured to obtain dynamic scene recognition based on the key elements and the derived elements of the plurality of image frames;

[0040] A question parsing module is used to obtain questions related to the scenario, parse the questions, and obtain a structured question expression;

[0041] An answer generation module, configured to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression;

[0042] The scene cognition update module is used to determine whether the answer meets the question requirements; if so, output the current answer; if not, determine the optimal perception path of the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements and outputs the current answer.

[0043] An embodiment of the present invention also provides a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor, and when the processor executes the computer program, it implements any of the above-mentioned question-answering methods based on scene blur perception and reasoning.

[0044] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned question-answering methods based on scene blur perception and reasoning.

[0045] Compared with the existing technology, the beneficial effects of the question-answering method, device, equipment and storage medium based on scene fuzzy perception and reasoning provided by the embodiments of the present invention are as follows: by obtaining key elements through fuzzy perception of image frames, and then performing reasoning and association based on the reasoning association knowledge base to obtain derived elements, thereby forming dynamic scene cognition, which can avoid fine recognition of the scene, improve perception efficiency, and reduce information redundancy; by constructing dynamic scene cognition and generating question answers based on dynamic scene cognition, it overcomes the poor generalization of methods such as artificially set rules and reinforcement learning, as well as the problem that large models rely on their own reasoning ability but lack environmental cognition, thereby improving environmental adaptability and task generalization. The embodiments of the present invention can enable intelligent agents to identify objects and understand the environment more quickly and accurately, achieve more efficient and complete scene cognition construction, and thus provide problem-solving solutions that better meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 1 is a flow chart of a question-answering method based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0047] Figure 2 1 is a flow chart of a question-answering method based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0048] Figure 3 is a structural diagram of a question-answering device based on scene fuzzy perception and reasoning provided by an embodiment of the present invention;

[0049] Figure 4 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] Existing methods that rely on artificial rules and deep learning methods, such as reinforcement learning, lack adaptability to open environments and tasks when performing perception tasks. These methods are primarily applied to specific domains, but in our daily lives, the tasks and environments we face are diverse and even unpredictable. Therefore, these rule- or data-driven methods have poor generalization and cannot adapt to and operate over long periods of time in such open, dynamic environments. Methods that use large models for scene perception often rely too heavily on their own reasoning capabilities and lack a deep understanding of the environment. When performing tasks, these models can easily generate perception paths that are unrelated to the scene, resulting in low perception efficiency.

[0052] Current methods for constructing robot cognition of the environment include those based on large visual language models and those based on scene graph generation, which aim to establish cognition of the entire scene. Methods based on large visual language models use prompt words to obtain and record textual descriptions of images from the current perspective during robot movement. Methods based on scene graph generation perform structured semantic extraction of the scene during robot movement, obtaining spatial or semantic relationships between each element in the scene to complete the cognitive representation of the scene. However, both methods based on visual text description and scene graph generation aim to achieve fine-grained perception of all elements in the scene to obtain accurate scene knowledge. However, this presents two problems: first, scene knowledge cannot be enumerated; second, speed and efficiency are very low, and the method is easily limited by the recognition capabilities of the model. This can lead to misidentification of objects and the overlapping of objects from different perspectives, resulting in information explosion. For example, a keyboard may be perceived as a black box from one perspective but as a black mouse pad from another. In this case, the resulting scene cognition contains a large amount of noise, resulting in multiple incorrect representations of the same object in the scene knowledge. Also, if the robot tries to perceive all the detailed information in the scene in order to support subsequent tasks, its speed will be extremely slow and its efficiency will be low.

[0053] Therefore, an embodiment of the present invention provides a question-answering method based on scene fuzzy perception and reasoning, which has reasoning and association capabilities when constructing scene cognition.

[0054] See also Figure 1 , Figure 1 1 is a flow chart of a question-answering method based on scene blur perception and reasoning provided by an embodiment of the present invention. The question-answering method based on scene blur perception and reasoning includes:

[0055] S1: Acquire several picture frames of a scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include appearance attributes and spatial positions of target objects;

[0056] S2: Based on a preset reasoning and association knowledge base, reasoning and associating the key elements to obtain derived elements of the image frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0057] S3: Obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of image frames;

[0058] S4: Obtain questions related to the scenario, parse the questions, and obtain structured question expressions;

[0059] S5: Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question;

[0060] S6: Determine whether the answer meets the question requirements; if so, output the current answer; if not, determine the optimal perception path for the question based on the question and the dynamic scene cognition, and update the dynamic scene cognition and the answer to the question based on the optimal perception path until the updated answer meets the question requirements, and output the current answer.

[0061] Specifically, when constructing scene cognition, the key elements of the target object in the scene are first identified. That is, based on the image frames collected by the agent's movement, the objects present in the image frames (i.e., the target objects) are identified, and the object's appearance attributes and spatial position are obtained. The key elements of the object are then inferred and associated to obtain corresponding derived elements, including the object's functional purpose and possible semantic name derived from appearance attributes, as well as other objects associated with the object but not identified in the image. To achieve efficient and accurate reasoning and association, a pre-designed reasoning and association knowledge base ensures that the reasoning and association process is consistent with human and scene experience. During the agent's movement, key elements and derived elements are continuously extracted and spatially integrated from the acquired image frames, resulting in a holistic dynamic scene cognition with a rich space for reasoning and association.

[0062] It should be noted that before the agent has a mobile task, a simple patrol process can be set up to build the current dynamic scene cognition while the agent moves naturally. This allows the agent to generate solutions based on the current dynamic scene cognition when a new task (i.e., a problem related to the scene) arises. Furthermore, during each movement of the agent to perform a task, the newly recognized scene cognition is updated to the current dynamic scene cognition based on the information in the real-time image frames.

[0063] When a user asks a scenario-related question, the question is first parsed to obtain its essential content and requirements. The system then leverages the constructed dynamic scenario cognition and large language model to generate a scenario-based answer. The generated answer is then evaluated to determine whether it meets the question requirements. If so, the answer is returned to the user. If not, the system generates an optimal perception path for re-perception based on the question and the current dynamic scenario cognition, forming a new perception task. The agent then moves along the optimal perception path, acquiring new scenario cognition and aggregating it into the original dynamic scenario cognition to obtain the current dynamic scenario cognition. Finally, based on the current dynamic scenario cognition, the large language model is used to regenerate the answer to the question and evaluate whether it meets the requirements. If so, the current answer is returned to the user.

[0064] As one of the optional embodiments, the method for constructing the preset reasoning association knowledge base includes:

[0065] Acquire general scenario knowledge;

[0066] Based on the general scene knowledge, a large language model is used for reasoning and association to construct a reasoning knowledge base and an association knowledge base respectively; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0067] The reasoning knowledge base and the associative knowledge base are aggregated to obtain a reasoning associative knowledge base.

[0068] Specifically, see Figure 2 First, we acquire general scenario knowledge and build a scenario-related expert knowledge base. This base includes a large amount of common-sense knowledge. For example, the workbench cognitive group includes desks, chairs, and computers, while the computer cognitive group includes computers and keyboards. Then, by setting expert instructions, the large language model can output the extracted knowledge content as needed based on the general scenario knowledge, thereby building an inference knowledge base and an association knowledge base. These two knowledge bases are then further aggregated to form the inference association knowledge base.

[0069] Inference knowledge refers to functional attributes derived from the basic properties of objects. For example, books have the property of being hard, so we can infer that they can be used not only for reading but also as a support for mobile phones and laptops. Clothes not only provide warmth, but their flexibility can also protect desktops from scratches and provide privacy. Associative knowledge is represented as a set of strong and weak associations, reflecting the relationships between objects. For example, a computer monitor is strongly associated with a keyboard, mouse, and power adapter, but weakly associated with USB flash drives and USB data cables. A bed is strongly associated with bedding and pillows. A study area is strongly associated with books and backpacks.

[0070] As one of the optional embodiments, acquiring a plurality of picture frames of a scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes:

[0071] Get several frames of images observed by the agent as it moves in the scene;

[0072] Using a multimodal visual language model to extract semantic information of the image frame;

[0073] Based on the semantic information, key elements of the picture frame are obtained.

[0074] As one of the optional embodiments, obtaining dynamic scene recognition based on the key elements and the derived elements of the plurality of picture frames includes:

[0075] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0076] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0077] Specifically, see Figure 2 When the agent moves in the scene, it collects environmental information and obtains a video sequence. By fuzzy perceiving the objects in the picture frames of the video sequence, scene knowledge is extracted to construct scene cognition. In order to ensure time efficiency, fuzzy perception is reflected in perceiving only some objects in the scene and not paying attention to the true semantic labels of the objects, such as: "black cylindrical, small object, placed on the bed, container or blunt object, like a box". Specifically, a multimodal visual language model (VLM) is first trained to extract the key elements of objects in the scene and their specific semantic information. The VLM is trained using preset pairs of images and text data. For the picture frame currently observed by the intelligent agent, the VLM is used to extract the semantic information in the picture frame to obtain the key elements in the picture frame. Then, the reasoning and association knowledge base constructed above is used to use a large language model to perform reasoning and association on the basis of the key elements of the object to obtain the derived elements of the object, that is, the function, purpose, possible name and other information of the object are inferred, as well as objects that may exist in the scene but are not perceived in the picture frame (such as a mouse hidden behind the display screen, which is associated based on the display screen); then the scene knowledge subgraph corresponding to the picture frame is generated from the key elements and derived elements of the picture frame.

[0078] Furthermore, based on the consistency of key elements (such as shape, color, and relative position) of the same observed object in different image frames in the scene, the same observed object from different perspectives in the scene is aligned, and the spatiotemporal aggregation of scene knowledge subgraphs is performed to finally obtain complete dynamic scene cognition.

[0079] For example, at time t0, object A is observed: "black, cylindrical object, serves as a container, located on the left side of the bed, resembling a box." At time t1, object B is observed: "black, cylindrical object, serves as a speaker, located on the left side of the bed, resembling a speaker." By performing a similarity test on the attributes of object A and object B, it can be determined that object A and object B are highly similar, and A and B are considered to be the same object. Their semantic names (box and speaker) may be due to model recognition errors. Therefore, these two objects are aggregated in the scene cognition structure, resulting in scene knowledge: object A is on the bed: "black, cylindrical object, serves as a speaker, serves as a container, located on the left side of the bed, resembling a box / speaker." Furthermore, if object C is also observed on the bed at time t1, the aggregation yields scene cognition: object A and object C are on the bed.

[0080] It should be noted that when constructing dynamic scene cognition, for large objects commonly found in the scene, such as beds, tables, sofas and other large furniture, their real names are directly perceived without fuzzy processing; only small objects and objects that are not within the preset range of common objects are fuzzy perceived (that is, only their attributes are perceived without recognizing their real names), thereby narrowing the spatiotemporal alignment space of the objects and greatly reducing the impact of model hallucinations. The fuzzy processing of small objects is due to insufficient resolution on the one hand and the diversity of small objects on the other hand. For example, if a speaker in the shape of a potted plant is directly labeled with the recognized real name, it is easy to misidentify the object and cause objects from different perspectives to overlap, resulting in information explosion.

[0081] Among them, the overall dynamic scene cognition that is finally formed is hierarchical, for example: V0: home scene {V1: in the bedroom there is {V2: on the bed there is {V3: schoolbag (attribute), V3: pillow (attribute), ...}, ...}, V1: in the living room there is...}.

[0082] As one of the optional embodiments, parsing the question to obtain a structured question expression includes:

[0083] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, including: essentiality, basic needs, potential needs, and deep meaning;

[0084] A structured question expression is generated according to the semantic features corresponding to the question.

[0085] Specifically, a large language model is pre-trained and fine-tuned to enable it to parse the intent of questions for any scenario. When a user has a scenario-based question to be solved, the large language model is used to parse the intent of the question, obtaining the four-dimensional semantic features of the scenario-based question: its essential nature, basic needs, potential needs, and underlying meaning, to generate a corresponding structured question expression. The question expression and the current dynamic scenario cognition are then input into the large language model, which performs semantic reasoning and calculations to obtain a scenario-based answer to the question.

[0086] As one of the optional embodiments, determining whether the answer meets the question requirement includes:

[0087] Calculating a resolution score of the answer based on the question and the answer;

[0088] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0089] If the resolution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0090] Specifically, after obtaining an answer to a question, the answer's resolution is calculated: Score = Resolution(Result, Question). If the resolution score is greater than a preset threshold, the question is considered resolved; if the resolution score is less than the preset threshold, the question is considered unresolved and a new answer needs to be generated. Large language models can be used to calculate resolution.

[0091] As one of the optional embodiments, determining the optimal perception path for the question based on the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes:

[0092] Based on the question and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the question; the optimal perception path includes data in the time dimension, the space dimension, and the semantic dimension;

[0093] Acquire a plurality of image frames observed by the agent while moving along the optimal perception path, and determine key elements and derived elements in each image frame to obtain a scene knowledge subgraph for each image frame;

[0094] Based on the key elements, spatiotemporally aggregating scene knowledge subgraphs of several picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0095] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0096] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0097] Specifically, when the answer does not meet the requirements of the question and needs to be regenerated, the optimal perception path is first generated based on the question and the current dynamic scene cognition, so that the agent can collect environmental data along the optimal perception path. Among them, the optimal perception path includes information in the time dimension, space dimension, and semantic dimension. For example, for the question "What is the name of the book?", when perceiving the scene before, the name of the book was not paid attention to, and it is difficult to get the answer through reasoning. Therefore, based on the scene cognition, a new perception path is given: "V1: living room, V2: second layer of the coffee table, V3: books, pay attention to the name of the book; navigation coordinates (x, y); time coordinates (ti) (for video positioning)", which then guides the agent to observe again along the path and dynamically update the new scene knowledge obtained: "The name of the book "XX"" to the overall scene cognition.

[0098] Finally, the dynamic scene cognition is updated according to the optimal perception path, and then the answer to the question is regenerated according to the updated dynamic scene cognition until the answer meets the requirements of the question and the answer is output to complete the task.

[0099] Compared to other methods, which store the scene knowledge extracted at each moment in a database and generate results through knowledge retrieval each time scenario reasoning is performed, this approach is not conducive to integrating the overall scenario. For example, for the question "I'm cold," other methods may only retrieve a single solution through retrieval: "The quilt on the bed can keep you warm." However, the method of this embodiment, by integrating holistic cognition, can provide multiple solutions: "The quilt on the bed, the blanket and pillow on the sofa, and the clothes on the hanger can keep you warm," providing more intelligent scenario-based question and answering.

[0100] The embodiment of the present invention divides scene cognition into key elements and derived elements. Only some key elements need to be perceived, and then derived elements are derived through reasoning and association mechanisms. This avoids fine-grained perception of all objects in the scene, reduces information redundancy and model illusions;

[0101] By adopting a fuzzy perception strategy, there is no need to finely identify all elements in the scene, which significantly improves perception efficiency. Combined with the reasoning and association mechanism, even if only partial information is perceived, key scene information will not be missed, ensuring that the intelligent agent can quickly and accurately understand the environment.

[0102] By taking time into account as important implicit information during the movement of the intelligent agent, scene cognition is made more consistent with the actual situation of the dynamic environment;

[0103] Object alignment is performed by associating the attributes of the same objects in different image frames, rather than aligning objects by semantic names, which effectively reduces the impact of model hallucinations and improves the accuracy of object recognition;

[0104] The embodiments of the present invention have stronger environmental adaptability and task generalization, can build scene cognition more efficiently, have richer scene information, high scene perception efficiency and strong reasoning and association capabilities, which is conducive to providing more accurate problem-solving solutions.

[0105] Correspondingly, the present invention also provides a question-answering device based on scene blur perception and reasoning, which can implement all processes of the question-answering method based on scene blur perception and reasoning in the above embodiment.

[0106] See also Figure 3 , Figure 3 : This is a schematic diagram of the structure of a question-answering device based on scene blur perception and reasoning provided by an embodiment of the present invention. The question-answering device based on scene blur perception and reasoning includes:

[0107] The fuzzy perception module 301 is used to obtain a plurality of picture frames of a scene and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; the key elements include the appearance attributes and spatial position of the target object;

[0108] The reasoning and association module 302 is configured to perform reasoning and association on the key elements based on a preset reasoning and association knowledge base to obtain derived elements of the image frame; the derived elements include functional attributes of the target object and objects associated with the target object;

[0109] A dynamic scene recognition module 303 is configured to obtain dynamic scene recognition based on the key elements and the derived elements of the plurality of image frames;

[0110] The question parsing module 304 is used to obtain questions related to the scenario, parse the questions, and obtain a structured question expression;

[0111] An answer generation module 305 is configured to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression;

[0112] The scene cognition update module 306 is used to determine whether the answer meets the requirements of the question; if so, the current answer is output; if not, the optimal perception path of the question is determined based on the question and the dynamic scene cognition, and the dynamic scene cognition and the answer to the question are updated based on the optimal perception path until the updated answer meets the requirements of the question and the current answer is output.

[0113] Preferably, the method for constructing the preset reasoning association knowledge base includes:

[0114] Acquire general scenario knowledge;

[0115] Based on the general scene knowledge, a large language model is used for reasoning and association to construct a reasoning knowledge base and an association knowledge base respectively; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene;

[0116] The reasoning knowledge base and the associative knowledge base are aggregated to obtain a reasoning associative knowledge base.

[0117] Preferably, the acquiring of a plurality of picture frames of a scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes:

[0118] Get several frames of images observed by the agent as it moves in the scene;

[0119] Using a multimodal visual language model to extract semantic information of the image frame;

[0120] Based on the semantic information, key elements of the picture frame are obtained.

[0121] Preferably, obtaining dynamic scene recognition based on the key elements and the derived elements of the plurality of picture frames includes:

[0122] For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame;

[0123] According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

[0124] Preferably, parsing the question to obtain a structured question expression includes:

[0125] Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, including: essentiality, basic needs, potential needs, and deep meaning;

[0126] A structured question expression is generated according to the semantic features corresponding to the question.

[0127] Preferably, determining whether the answer meets the question requirements includes:

[0128] Calculating a resolution score of the answer based on the question and the answer;

[0129] If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements;

[0130] If the resolution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

[0131] Preferably, determining the optimal perception path for the question based on the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path includes:

[0132] Based on the question and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the question; the optimal perception path includes data in the time dimension, the space dimension, and the semantic dimension;

[0133] Acquire a plurality of image frames observed by the agent while moving along the optimal perception path, and determine key elements and derived elements in each image frame to obtain a scene knowledge subgraph for each image frame;

[0134] Based on the key elements, spatiotemporally aggregating scene knowledge subgraphs of several picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path;

[0135] Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition;

[0136] An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

[0137] In the specific implementation, the working principle, control process and technical effect of the question-answering device based on scene blur perception and reasoning provided by the embodiment of the present invention are the same as those of the question-answering method based on scene blur perception and reasoning in the above embodiment, and will not be repeated here.

[0138] See also Figure 4 , Figure 4 4 is a block diagram of a computer device provided in an embodiment of the present invention. The computer device includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program, the steps of the aforementioned question-answering method embodiment based on scene blur perception and reasoning are implemented. Alternatively, when the processor 401 executes the computer program, the functions of the modules / units in the aforementioned apparatus embodiments are implemented.

[0139] Illustratively, the computer program may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0140] The computer device may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that the schematic diagram is merely an example of a computer device and does not limit the computer device. The computer device may include more or fewer components than shown, or may combine certain components or different components. For example, the computer device may also include input and output devices, network access devices, buses, and the like.

[0141] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 401 is the control center of the computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0142] The memory 402 can be used to store the computer programs and / or modules. The processor 401 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 402 and calling the data stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 402 can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0143] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 401, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0144] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the question-answering method based on scene blur perception and reasoning described in any of the above embodiments.

[0145] The embodiments of the present invention provide a question-answering method, apparatus, device, and storage medium based on scene fuzzy perception and reasoning, which has the following beneficial effects: by performing fuzzy perception on image frames to obtain key elements, and then performing reasoning and association based on a reasoning association knowledge base to obtain derived elements, thereby forming dynamic scene cognition, which can avoid fine recognition of the scene, improve perception efficiency, and reduce information redundancy; by constructing dynamic scene cognition and generating question answers based on dynamic scene cognition, it overcomes the poor generalization of methods such as manually set rules and reinforcement learning, as well as the problem that large models rely on their own reasoning capabilities but lack environmental cognition, thereby improving environmental adaptability and task generalization. The embodiments of the present invention can enable intelligent agents to identify objects and understand the environment more quickly and accurately, achieve more efficient and complete scene cognition construction, and thus provide problem-solving solutions that better meet user needs.

[0146] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A question-answering method based on scene fuzzy perception and reasoning, characterized by: include: Acquire several picture frames of the scene, and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; The key elements include the appearance attributes and spatial position of the target object; Based on a preset reasoning and association knowledge base, the key elements are inferred and associated to obtain derived elements of the picture frame; the derived elements include functional attributes of the target object and objects associated with the target object; Obtaining dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames; Obtaining questions related to the scenario, parsing the questions, and obtaining structured question expressions; Based on the dynamic scene recognition and the structured question expression, a large language model is used to generate an answer to the question; Determine whether the answer meets the question requirements; if so, output the current answer; If not, the optimal perception path for the question is determined based on the question and the dynamic scene cognition, and the dynamic scene cognition and the answer to the question are updated based on the optimal perception path until the updated answer meets the question requirements and the current answer is output.

2. The question-answering method based on scene fuzzy perception and reasoning according to claim 1, wherein: The method for constructing the preset reasoning association knowledge base includes: Acquire general scenario knowledge; Based on the general scene knowledge, a large language model is used for reasoning and association to construct a reasoning knowledge base and an association knowledge base respectively; the reasoning knowledge base includes the functional attributes of each object in the scene; the association knowledge base includes a set of strongly associated objects and a set of weakly associated objects in the scene; The reasoning knowledge base and the associative knowledge base are aggregated to obtain a reasoning associative knowledge base.

3. The question-answering method based on scene fuzzy perception and reasoning according to claim 1, wherein: The acquiring of a plurality of picture frames of the scene and performing fuzzy perception on the picture frames to obtain key elements of the picture frames includes: Get several frames of images observed by the agent as it moves in the scene; Using a multimodal visual language model to extract semantic information of the image frame; Based on the semantic information, key elements of the picture frame are obtained.

4. The question-answering method based on scene fuzzy perception and reasoning according to claim 3, wherein: The obtaining of dynamic scene recognition according to the key elements and the derived elements of the plurality of picture frames includes: For each of the picture frames, obtaining a scene knowledge subgraph of the picture frame according to the key elements and derived elements of the picture frame; According to the key elements in the picture frames, the target objects in different picture frames are aligned to perform spatiotemporal aggregation on the scene knowledge subgraphs of several picture frames to obtain hierarchical dynamic scene cognition.

5. The question-answering method based on scene fuzzy perception and reasoning according to claim 1, wherein: The problem is parsed to obtain a structured problem expression, including: Based on the large language model, the question is analyzed for intent to obtain the semantic features corresponding to the question, including: essentiality, basic needs, potential needs, and deep meaning; A structured question expression is generated according to the semantic features corresponding to the question.

6. The question-answering method based on scene fuzzy perception and reasoning according to claim 1, wherein: Determining whether the answer meets the question requirements includes: Calculating a resolution score of the answer based on the question and the answer; If the resolution score is less than a preset threshold, it is determined that the answer does not meet the question requirements; If the resolution score is not less than a preset threshold, it is determined that the answer meets the question requirements.

7. The question-answering method based on scene fuzzy perception and reasoning according to claim 1, wherein: Determining an optimal perception path for the question based on the question and the dynamic scene cognition, and updating the dynamic scene cognition and the answer to the question based on the optimal perception path, includes: Based on the question and the dynamic scene cognition, a large language model is used to generate an optimal perception path for the question; the optimal perception path includes data in the time dimension, the space dimension, and the semantic dimension; Acquire a plurality of image frames observed by the agent while moving along the optimal perception path, and determine key elements and derived elements in each image frame to obtain a scene knowledge subgraph for each image frame; Based on the key elements, spatiotemporally aggregating scene knowledge subgraphs of several picture frames corresponding to the optimal perception path to obtain scene cognition corresponding to the optimal perception path; Updating the scene cognition corresponding to the optimal perception path into the dynamic scene cognition to obtain updated dynamic scene cognition; An answer to the question is determined based on the question expression and the updated dynamic scene cognition.

8. A question-answering device based on scene fuzzy perception and reasoning, characterized in that: include: A fuzzy perception module is used to obtain a plurality of picture frames of a scene and perform fuzzy perception on the picture frames to obtain key elements of the picture frames; The key elements include the appearance attributes and spatial position of the target object; An inference and association module is used to infer and associate the key elements based on a preset inference and association knowledge base to obtain derived elements of the image frame; the derived elements include functional attributes of the target object and objects associated with the target object; a dynamic scene recognition module, configured to obtain dynamic scene recognition based on the key elements and the derived elements of the plurality of image frames; A question parsing module is used to obtain questions related to the scenario, parse the questions, and obtain a structured question expression; An answer generation module, configured to generate an answer to the question using a large language model based on the dynamic scene recognition and the structured question expression; A scenario recognition update module is used to determine whether the answer meets the question requirements; if so, output the current answer; If not, the optimal perception path for the question is determined based on the question and the dynamic scene cognition, and the dynamic scene cognition and the answer to the question are updated based on the optimal perception path until the updated answer meets the question requirements and the current answer is output.

9. A computer device, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor, and when the processor executes the computer program, it implements the question-answering method based on scene blur perception and reasoning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the question-answering method based on scene blur perception and reasoning as described in any one of claims 1 to 7 is implemented.

Images