Three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval

Abstract

The application relates to the technical field of visual perception, in particular to a three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval. The method comprises the following steps: acquiring a three-dimensional scene evidence library and initial inquiry information; the three-dimensional scene evidence library organizes object instances, full-quantity visual frames and spatial relationships in a scene into a multi-level queryable structure; according to the initial inquiry information, retrieval hypothesis information is generated; based on the retrieval hypothesis information, retrieval is carried out in the three-dimensional scene evidence library to generate an initial evidence package; a preset tool matched with the initial evidence package is called to carry out supplementary retrieval in the three-dimensional scene evidence library to determine a candidate evidence package; according to the task type and the candidate evidence package, reply information corresponding to the initial inquiry information is determined. The three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval can improve the acquisition ability and reasoning accuracy of a large language model in three-dimensional scene understanding.

Description

Technical Field

[0001] This application relates to the field of visual perception technology, and in particular to a multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval. Background Technology

[0002] With the development of large language models and visual language models, embodied intelligent multimodal understanding tasks for pre-scanned 3D scenes face the challenge of handling multidimensional inputs spanning 2D images, 3D geometry, temporal trajectories, and language. Existing mainstream solutions mainly fall into two categories: one is text symbol reasoning based on discrete 3D scene graphs. While this approach offers fast retrieval and clear structure, it is essentially a compressed representation of the original vision, easily losing fine-grained features such as color, texture, and object state. Furthermore, once a missed detection or mislabeling occurs, the lack of a path back to the original image limits reasoning ability. The second is visual language question answering based on independent keyframes. While this approach preserves image details, it ignores the spatiotemporal adjacency of video sequences, camera trajectories, and viewpoints. This not only easily introduces a large amount of redundant data but also lacks a mechanism for dynamic backtracking along the trajectory and finding complementary viewpoints when the initial frame does not cover key details.

[0003] Therefore, there is an urgent need to propose a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval, in order to solve the problems of existing methods, such as loss of fine-grained information, inability to backtrack for verification, inability to supplement viewpoints as needed, and lack of a unified reasoning framework for multiple tasks, caused by discarding original image evidence or lack of dynamic evidence scheduling mechanism. Summary of the Invention

[0004] This application provides a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval, which addresses the problems of existing methods such as loss of fine-grained information, inability to backtrack for verification, inability to supplement viewpoints as needed, and lack of a unified reasoning framework for multiple tasks caused by discarding original image evidence or lacking a dynamic evidence scheduling mechanism.

[0005] This application provides a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval, including the following steps: Acquire a 3D scene evidence library and initial inquiry information, wherein the 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type; Based on the initial query information, retrieval hypothesis information is generated; Based on the retrieval hypothesis information, a search is performed in the three-dimensional scene evidence database to generate an initial evidence package; A preset tool matching the initial evidence package is invoked to perform a supplementary search in the three-dimensional scene evidence library to determine candidate evidence packages; Based on the task type and the candidate evidence package, determine the response information corresponding to the initial inquiry information.

[0006] According to the multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval provided in this application, the three-dimensional scene evidence base includes an object layer, a view layer, and a relation layer, wherein... The object layer is used to store object instances within the 3D scene, as well as semantic attributes, geometric boundary features, and visible view indexes corresponding to the object instances; The view layer is used to store all the visual frames of the three-dimensional scene and the acquisition metadata corresponding to the full visual frames; The relationship layer is used to store and record the visibility relationship between the object layer and the view layer, the spatial relationship between object instances, and the temporal adjacency relationship, pose adjacency relationship, and visibility overlap relationship between the full set of visual frames.

[0007] According to the multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval provided in this application, the step of generating retrieval hypothesis information based on the initial query information includes: The initial query information is converted into structured query information; Based on the structured query information, the retrieval hypothesis information is generated, and the retrieval hypothesis information includes direct hypothesis, proxy hypothesis and context hypothesis according to priority; The direct assumption is used to match the target object instance or the spatial relationship that appears explicitly in the initial query information and is identifiable in the object layer of the three-dimensional scene evidence library; The proxy hypothesis is used to retrieve object instances with similar functions as candidates when the target object instance cannot be matched by the direct hypothesis; The contextual assumptions are used to infer the location of other evidence for retrieval based on regional clues or spatial relationships in the initial inquiry information.

[0008] According to the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application, the relation layer is further used to construct a trajectory-aware view map based on the temporal adjacency relationship, pose adjacency relationship and visibility overlap relationship. The trajectory-aware view map uses the full set of visual frames as nodes and the adjacency relationship as edges to characterize the adjacent features between visual frames. The step of searching the three-dimensional scene evidence database based on the search hypothesis information to generate an initial evidence package includes: Based on the retrieval hypothesis information, candidate views are determined in the view layer; The candidate view is evaluated according to preset indicators to determine the evaluation result. The preset indicators include at least one of the following: the visibility of the object instance in the candidate view, the degree of common coverage of the object instance and the anchor object instance in the candidate view, the imaging area of ​​the object instance, the viewing quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. Based on the evaluation results, the initial keyframe is determined; The initial evidence package is generated based on the initial keyframe and the object instance information associated with the initial keyframe.

[0009] According to the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application, the preset tool includes at least one of the following: A view expansion tool is used to acquire visual frames that are adjacent to or complementary to the current view in the 3D scene evidence library. A local cropping tool is used to extract a close-up image of a local area of ​​the target region from the visual frame; An object context reading tool is used to read the semantic attributes, geometric boundary features, and visible view index of a target object instance from the object layer. The hypothesis switching tool is used to switch to other search hypothesis information for retrieval when the initial evidence package does not meet the matching requirements; A view exploration tool is used to retrieve additional visual frames in the 3D scene evidence library that are not included in the initial evidence package.

[0010] According to the multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval provided in this application, the step of invoking a preset tool matching the initial evidence package to perform supplementary retrieval in the 3D scene evidence database to determine candidate evidence packages includes: Based on the initial evidence package, determine the preset tools to be invoked and the invocation order; According to the calling order, the preset tool to be called is invoked to obtain supplementary evidence from the three-dimensional scene evidence library; The candidate evidence package is generated based on the supplementary evidence and the initial evidence package.

[0011] According to the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application, the step of determining the response information corresponding to the initial query information based on the task type and the candidate evidence package includes any one of the following: When the task type is a question-answering task, the natural language answer generated based on the candidate evidence package and the supporting visual frame corresponding to the natural language answer are used as the response information. When the task type is a visual positioning task, the identification information and location information of the target object instance generated based on the candidate evidence package are used as the response information; When the task type is a navigation planning task, the target area and path information generated based on the candidate evidence package will be used as the response information; When the task type is an operation planning task, the action sequence and execution constraint information generated based on the candidate evidence package will be used as the response information.

[0012] This application provides a 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval, including the following modules: The acquisition module is used to acquire a 3D scene evidence library and initial inquiry information. The 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type. The generation module is used to generate retrieval hypothesis information based on the initial query information; The first retrieval module is used to search the three-dimensional scene evidence database based on the retrieval hypothesis information to generate an initial evidence package; The second retrieval module is used to call a preset tool that matches the initial evidence package to perform a supplementary retrieval in the three-dimensional scene evidence library in order to determine the candidate evidence package; The determination module is used to determine the response information corresponding to the initial query information based on the task type and the candidate evidence package.

[0013] According to the multi-task evidence reasoning device for three-dimensional scenes based on view memory and hypothesis retrieval provided in this application, the three-dimensional scene evidence base includes an object layer, a view layer, and a relation layer, wherein... The object layer is used to store object instances within the 3D scene, as well as semantic attributes, geometric boundary features, and visible view indexes corresponding to the object instances; The view layer is used to store all the visual frames of the three-dimensional scene and the acquisition metadata corresponding to the full visual frames; The relationship layer is used to store and record the visibility relationship between the object layer and the view layer, the spatial relationship between object instances, and the temporal adjacency relationship, pose adjacency relationship, and visibility overlap relationship between the full set of visual frames.

[0014] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the generation module is specifically used for: The initial query information is converted into structured query information; Based on the structured query information, the retrieval hypothesis information is generated, and the retrieval hypothesis information includes direct hypothesis, proxy hypothesis and context hypothesis according to priority; The direct assumption is used to match the target object instance or the spatial relationship that appears explicitly in the initial query information and is identifiable in the object layer of the three-dimensional scene evidence library; The proxy hypothesis is used to retrieve object instances with similar functions as candidates when the target object instance cannot be matched by the direct hypothesis; The contextual assumptions are used to infer the location of other evidence for retrieval based on regional clues or spatial relationships in the initial inquiry information.

[0015] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the relation layer is further used to construct a trajectory-aware view map based on the temporal adjacency relationship, pose adjacency relationship and visibility overlap relationship. The trajectory-aware view map uses the full set of visual frames as nodes and the adjacency relationship as edges to characterize the adjacent features between visual frames. The first retrieval module is specifically used for: Based on the retrieval hypothesis information, candidate views are determined in the view layer; The candidate view is evaluated according to preset indicators to determine the evaluation result. The preset indicators include at least one of the following: the visibility of the object instance in the candidate view, the degree of common coverage of the object instance and the anchor object instance in the candidate view, the imaging area of ​​the object instance, the viewing quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. Based on the evaluation results, the initial keyframe is determined; The initial evidence package is generated based on the initial keyframe and the object instance information associated with the initial keyframe.

[0016] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the preset tool includes at least one of the following: A view expansion tool is used to acquire visual frames that are adjacent to or complementary to the current view in the 3D scene evidence library. A local cropping tool is used to extract a close-up image of a local area of ​​the target region from the visual frame; An object context reading tool is used to read the semantic attributes, geometric boundary features, and visible view index of a target object instance from the object layer. The hypothesis switching tool is used to switch to other search hypothesis information for retrieval when the initial evidence package does not meet the matching requirements; A view exploration tool is used to retrieve additional visual frames in the 3D scene evidence library that are not included in the initial evidence package.

[0017] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the second retrieval module is specifically used for: Based on the initial evidence package, determine the preset tools to be invoked and the invocation order; According to the calling order, the preset tool to be called is invoked to obtain supplementary evidence from the three-dimensional scene evidence library; The candidate evidence package is generated based on the supplementary evidence and the initial evidence package.

[0018] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the determining module is specifically used for: When the task type is a question-answering task, the natural language answer generated based on the candidate evidence package and the supporting visual frame corresponding to the natural language answer are used as the response information. When the task type is a visual positioning task, the identification information and location information of the target object instance generated based on the candidate evidence package are used as the response information; When the task type is a navigation planning task, the target area and path information generated based on the candidate evidence package will be used as the response information; When the task type is an operation planning task, the action sequence and execution constraint information generated based on the candidate evidence package will be used as the response information.

[0019] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described above.

[0020] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described above.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described above.

[0022] This application provides a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval. By constructing a multi-level queryable 3D scene evidence base, it organically integrates object instances, all visual frames, and spatial relationships, preserving complete visual backtracking capabilities for subsequent reasoning. By generating retrieval hypothesis information, it transforms fuzzy natural language queries into structured retrieval strategies, effectively addressing uncertainties such as object label mismatch and imprecise descriptions. By generating an initial evidence package based on retrieval hypotheses, it achieves efficient and low-redundancy first-round evidence screening. Through supplementary retrieval, the system can dynamically acquire incremental evidence such as viewpoint expansion and local cropping as needed, avoiding information redundancy or loss caused by fixed input. Finally, it outputs structured response information according to different task types, realizing a unified framework for multiple tasks from question answering and localization to navigation and operation planning. Therefore, this invention significantly improves the ability of large language models to acquire fine-grained visual evidence and the accuracy of reasoning in 3D scene understanding, reduces information loss and reasoning upper bound limitations caused by scene graph compression, and enhances cross-task versatility and interpretability through task-conditional output. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is one of the flowcharts of the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application.

[0025] Figure 2 This is the second flowchart of the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application.

[0026] Figure 3 This is a schematic diagram of the structure of the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application.

[0027] Figure 4 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Figure 1 This is one of the flowcharts illustrating the multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval provided in this application, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the 3D scene evidence library and initial query information. The 3D scene evidence library organizes object instances, full visual frames, and spatial relationships in the scene into a multi-level queryable structure, and the initial query information has a corresponding task type.

[0030] The 3D scene evidence base organically organizes object instances (such as chairs and tables), full visual frames (i.e., all original RGB-D images), and spatial relationships (such as above or near) in the pre-scanned 3D scene into a multi-level queryable structure, which can provide a knowledge base for large language models that can both quickly index and trace back the original visual details.

[0031] The initial query information refers to the natural language instructions or questions entered by the user, such as "Where is the red cup on the table?" or "Plan a path from the door to the sofa."

[0032] In this context, "task type" refers to a label that categorizes the essential type of the question or instruction corresponding to the user's initial inquiry. This invention uniformly classifies 3D scene understanding tasks into four main types: question-and-answer tasks, visual positioning tasks, navigation planning tasks, and operation planning tasks.

[0033] Optionally, in some embodiments, the 3D scene evidence library includes an object layer, a view layer, and a relationship layer, wherein, The object layer is used to store object instances within the 3D scene, as well as the semantic attributes, geometric boundary features, and visible view indexes corresponding to the object instances; The view layer is used to store all the visual frames of the 3D scene, as well as the acquisition metadata corresponding to the full visual frames; The relationship layer is used to store the visibility relationships between the record object layer and the view layer, the spatial relationships between object instances, and the temporal adjacency relationships, pose adjacency relationships, and visibility overlap relationships between all visual frames.

[0034] In other words, in this embodiment of the disclosure, all RGB-D image frames from the pre-scanning process may be retained first, and then organized together with camera trajectory, object mapping, and 3D geometric results into a scene evidence library. This evidence library is not simply a collection of text descriptions, but a queryable, traceable, and scalable multi-layered evidence structure.

[0035] Specifically, the object layer uses 3D object instances as the basic unit, recording object IDs, category candidates, multi-label semantics, 3D bounding boxes, point clouds, color point clouds, detection confidence, 2D detection boxes, a list of visible frames, and optional visual or textual features. The view layer uses image frames as the basic unit, storing RGB image paths, depth map paths, camera poses, a set of visible objects, the imaging area of ​​the target object in that frame, occlusion status, and viewpoint quality. The relationship layer records object-view visibility, spatial relationships between objects, object co-occurrence relationships, temporal adjacency relationships between views, camera pose adjacency relationships, gaze direction similarity relationships, and overlapping or complementary relationships of visible objects.

[0036] Step 102: Generate retrieval hypothesis information based on the initial query information.

[0037] In this context, retrieval hypothesis information refers to transforming a fuzzy, open-ended natural language query (i.e., the initial query information) into a set of conjectures or retrieval strategies that allow the computer system to perform specific, structured search operations within a 3D scene evidence base. These hypotheses have clear priorities, guiding the system to begin searching along the most probable path and automatically switch to alternative paths if those fail. This improves the success rate and robustness of retrieval even under uncertainties such as mismatched scene object labels or inaccurate user descriptions.

[0038] Optionally, in some embodiments, retrieval hypothesis information is generated based on the initial query information, including: Convert the initial query information into structured query information; Based on the structured query information, retrieval hypothesis information is generated. The retrieval hypothesis information includes direct hypothesis, proxy hypothesis and context hypothesis according to priority. Among them, direct assumptions are used to match target object instances or spatial relationships that appear explicitly in the initial query information and are identifiable in the object layer of the 3D scene evidence library; Proxy assumptions are used to search for object instances with similar functions when the target object instance cannot be matched by direct assumptions. Contextual assumptions are used to infer the location of other evidence based on regional clues or spatial relationships in the initial inquiry information.

[0039] Structured query information refers to the set of data fields with clear semantic labels extracted from the user's initial unstructured query information through natural language processing technology after semantic parsing. This set typically includes, but is not limited to: target objects (e.g., cups), anchor objects (e.g., tables), attribute constraints (e.g., red), spatial predicates (e.g., on...), selection conditions (e.g., largest), task types (e.g., question-and-answer, location), and output format requirements, etc., without restriction. For example, for the initial query: Please find the red cup on the table, the generated structured query information can be represented as: {Target object: cup, Anchor object: table, Attribute constraint: red, Spatial predicate: above, Task type: visual positioning}.

[0040] Among them, the direct assumption is used to match target object instances or spatial relationships that appear explicitly in the initial query information and are identifiable in the object layer of the 3D scene evidence base. It matches the exact words in the user query with the standard labels in the evidence base, which is the most efficient and direct retrieval path.

[0041] For example, if the initial query is: find the black office chair, and there is already an object instance in the object layer of the evidence database labeled as office chair with the attribute containing black, then the system will prioritize the direct assumption and search for that type of label in the object layer.

[0042] The surrogate hypothesis is used to retrieve functionally similar object instances as candidates when a direct hypothesis cannot match the target object instance (e.g., the user's vocabulary is inconsistent with the labels in the evidence base, such as the user saying "sofa" while the label is "bench"; or the target object category itself is unclear). This strategy employs an alternative retrieval strategy, using functionally similar objects, semantically nearest objects, or objects that commonly serve as the carrier of the target in the physical world as retrieval candidates. The essence of the surrogate hypothesis is to use common sense knowledge or semantic similarity for generalized matching.

[0043] For example, if the initial query is: "Install this part onto the machine," but the term "part" is too broad, the system will initiate an agent hypothesis to search for objects with similar functions to the installation action (such as screws or clips), or to search for common functional parts near the machine as candidates. Another example: if a user queries "find a place to sit," and the "chair" tag is missing, the agent hypothesis could search for stools, sofas, or bed edges with similar functions.

[0044] The context hypothesis is used to infer the location of other evidence based on regional clues or spatial relationships in the initial inquiry information. When neither the direct hypothesis nor the proxy hypothesis provides sufficient evidence, it infers the location of possible evidence or related target objects based on regional clues, activity scenarios, relationships between adjacent objects, or general knowledge implicit in the initial inquiry information or derived through common sense reasoning. This hypothesis fully utilizes the integrity of the scene and the co-occurrence relationships between objects to conduct a broader heuristic search.

[0045] For example, if the initial query finds a remote control, and both the direct hypothesis (searching for remote control tags) and the proxy hypothesis (searching for synonyms such as controller or remote control) fail, the system initiates a contextual hypothesis. Based on the common knowledge that remote controls are often found alongside televisions, sofas, and coffee tables, the system expands the search scope to the area surrounding these anchor objects, or prioritizes searching for any small, handheld electronic devices in image frames near these objects.

[0046] In this embodiment, the initial query information input by the user is first structured using natural language understanding technology or by calling a large language model to obtain the target object, anchor object, attribute constraints, spatial predicates, selection conditions, task type, and output format requirements, thus obtaining structured query information. Based on this structured query information, retrieval hypothesis information is generated. Each hypothesis is represented as a structured record, containing at least the target word, anchor word, spatial predicate, candidate category, attribute constraints, retrieval mode, priority, and fallback strategy. The system performs retrieval in the object index, relation index, and view index sequentially according to priority. When a high-confidence hypothesis cannot cover sufficient target evidence, the system switches to a lower-priority hypothesis or expands the retrieval scope. When the object label is inconsistent with the user's expression, the system can use multi-label semantics, text features, or visual features for semantic matching. This design makes the mapping process of user question—scene object—visual evidence explicit, facilitating analysis, reproduction, and ablation.

[0047] Step 103: Based on the retrieval hypothesis information, search the 3D scene evidence database to generate an initial evidence package.

[0048] In this embodiment of the disclosure, the relationship layer is also used to construct a trajectory-aware view graph based on temporal adjacency, pose adjacency, and visibility overlap. The trajectory-aware view graph uses all visual frames as nodes and adjacency relationships as edges to characterize the adjacent features between visual frames.

[0049] The trajectory-aware view map is a structured data model constructed in the relation layer of this invention. This map uses all visual frames (i.e., each image) in the 3D scene evidence library as nodes and various adjacency relationships between views as edges, to characterize the adjacency features and correlations between different visual frames. Specifically, the edges include: temporal adjacency relationships (i.e., the sequential adjacency of consecutive frames in a video sequence), camera pose adjacency relationships (i.e., the proximity of the camera's spatial position and orientation), gaze direction similarity relationships (i.e., the approximation of the viewing angle), shared visible object quantity relationships (i.e., the number of object instances that appear together in two frames), and complementary visible object quantity relationships (i.e., the number of visible objects unique to each of the two frames).

[0050] The initial evidence package is a structured data collection that includes not only initial keyframes (i.e., image files or paths), but also object instance information associated with these keyframes (such as object ID, category, 3D bounding box, visibility statistics), the type of hypothesis used in this retrieval, the reason for selecting each keyframe (e.g., clear and unobstructed imaging), and an extensible entry point (indicating which evidence tools can be invoked for supplementary retrieval). The initial evidence package is designed to initiate the reasoning process with minimal information while providing a clear entry point and context for subsequent rounds of evidence acquisition.

[0051] Optionally, in some embodiments, a search is performed in a 3D scene evidence base based on retrieval hypothesis information to generate an initial evidence package, including: Candidate views are determined in the view layer based on retrieval hypothesis information; The candidate view is evaluated based on preset indicators to determine the evaluation result. The preset indicators include at least one of the following: the visibility of the object instance in the candidate view, the degree of common coverage of the object instance and the anchor object instance in the candidate view, the imaging area of ​​the object instance, the viewing quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. Based on the evaluation results, the initial keyframes are determined; An initial evidence package is generated based on the initial keyframe and the object instance information associated with the initial keyframe.

[0052] An anchor object instance refers to a reference object that has a spatial relationship with the target object or appears together with it, and is used to help locate or verify reasoning. When a user query involves spatial description (such as C to the left of A and above B), A or B is the anchor object.

[0053] Candidate views refer to a set of image frames that may contain target object instances, anchor point object instances, or satisfy spatial relationship constraints, obtained after preliminary screening in the view layer of the 3D scene evidence base based on retrieval hypothesis information (such as direct hypothesis, proxy hypothesis, or context hypothesis).

[0054] The preset metrics are a set of quantitative evaluation standards predefined in this invention for selecting the optimal initial keyframe from candidate views. The system calculates and scores each candidate view based on these metrics, and the evaluation result reflects the view's value in answering the current user's query. The preset metrics include, but are not limited to: the visibility of object instances in the candidate view, the degree of shared coverage between the object instance and anchor object instances in the candidate view, the imaging area of ​​the object instance, the viewpoint quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. The system can fuse multiple metrics into a comprehensive score using a weighted summation method.

[0055] Visibility refers to whether a target object instance is fully or partially presented in a candidate view, and the degree to which it is occluded. This metric is typically expressed as a binary value (1 for full visibility, 0 for invisibility) or a continuous value (calculated based on the proportion of visible surface). Views with high visibility provide more reliable information about the object's appearance and structure for subsequent inference.

[0056] The degree of common coverage refers to the extent to which a candidate view can clearly observe both target object instances and anchor object instances (such as cups on a table) when a user query involves both.

[0057] The imaging area refers to the pixel area occupied by a target object instance in the two-dimensional image plane of the candidate view. This metric is usually expressed as the number of pixels or as a proportion of the total image area. A larger imaging area means richer object details, which is beneficial for local cropping, attribute recognition (such as color, texture, and text), and fine-grained feature extraction.

[0058] Viewpoint quality refers to the viewer-friendliness reflected by the camera angle, distance, and degree of occlusion when observing a target object instance in a candidate view. This metric quantifies whether the view provides the optimal viewing position. For example, a frontal view is generally better than a side or oblique view; views with moderate distance, the object centered in the image, and no foreground occlusion have higher viewpoint quality.

[0059] The adjacency relationship refers to the degree of connection between the candidate view and the selected view (or the view already present in the initial evidence package) on the trajectory-aware view map. This metric measures how much continuity or novelty on the trajectory is brought by introducing the candidate view. Specifically, it can include temporal adjacency (whether it is temporally adjacent to the selected frame), pose adjacency (whether the camera positions are close), or complementarity (whether objects not present in the selected frame can be seen).

[0060] The evaluation result is a quantitative or ranking result obtained by the system after comprehensively calculating each candidate view based on preset indicators. This result can be a comprehensive score for each candidate view, which is determined as follows: Score(v)=α·Vis(v)+β·CoVis(v)+γ·Area(v)+δ·Quality(v)+η·Novelty(v)+λ·Trajectory(v); Where Vis represents target visibility, CoVis represents the degree of co-coverage between the target and the anchor point, Area represents the imaging area, Quality represents the view quality, Novelty represents the increment of new evidence relative to the selected view, and Trajectory represents the adjacency or complementarity between the view and existing evidence on the trajectory map.

[0061] The initial keyframes refer to a set of optimal image frames selected from the candidate view set based on the evaluation results to construct the initial evidence package. These keyframes represent the most valuable visual evidence obtainable in the first round of retrieval, without considering subsequent rounds of supplementary evidence.

[0062] In this embodiment, candidate views are first determined in the view layer based on retrieval hypothesis information (such as direct hypothesis, surrogate hypothesis, or context hypothesis). Specifically, the system obtains a list of visible frames of the corresponding object instance from the object layer based on the target object or anchor object information in the retrieval hypothesis, and uses all image frames in the list as candidate views. Subsequently, the system evaluates each candidate view according to preset indicators to determine the evaluation result. The preset indicators include: the visibility of the object instance in the candidate view (Vis), the degree of common coverage of the object instance and the anchor object instance in the candidate view (CoVis), the imaging area of ​​the object instance (Area), the viewing quality of the candidate view (Quality), and the adjacency relationship of the candidate view on the trajectory-aware view map (Trajectory). The system calculates the comprehensive score of each candidate view by weighted summation of the above indicators.

[0063] It should be noted that for queries that require simultaneous observation of multiple objects or spatial relationships, the system adopts a marginal gain selection strategy: first, it selects the view that can most clearly observe the main target, and then iteratively selects the view that contributes the most to the uncovered objects, relationships or local attributes, until the preset coverage threshold is met.

[0064] In this embodiment of the disclosure, after obtaining the evaluation results of each candidate view, initial keyframes can be determined based on the evaluation results. Specifically, the system sorts the candidate views from high to low according to their comprehensive scores and selects one or more views with the highest scores as initial keyframes; when using a marginal gain selection strategy, the system selects views sequentially according to the principle of maximizing iterative gain. If the target object is occluded or the viewing angle is insufficient, the system searches for neighboring or complementary viewpoints along the trajectory-aware view map as supplementary keyframes. After determining the initial keyframes, the system generates an initial evidence package based on the initial keyframes and the object instance information associated with the initial keyframes.

[0065] Step 104: Invoke the preset tool that matches the initial evidence package to perform a supplementary search in the 3D scene evidence library to determine the candidate evidence package.

[0066] Among them, the preset tools refer to a set of predefined, standardized atomic functional modules that can be dynamically invoked by the visual language agent. Each tool encapsulates specific query or operation capabilities for the 3D scene evidence library. The agent can actively select and invoke the corresponding tool to obtain supplementary evidence based on the current reasoning needs.

[0067] The candidate evidence package is the final set of evidence formed after the initial search and subsequent multiple rounds of supplementary searches. It is a cumulative and structured set of evidence generated by dynamically adding information such as new image frames, local cropped images, object context data, new search results after hypothesis switching, and novel frames obtained from view exploration, based on the initial evidence package through multiple calls to preset tools.

[0068] In some embodiments, visual feature vectors of existing keyframes are first extracted from the initial evidence package and fused cross-modally with the semantic embedding vector of the user's initial query information to generate a joint query vector. Subsequently, the agent invokes a pre-defined semantic similarity retrieval tool. This tool calculates the cosine similarity between the visual features of each image frame not included in the initial evidence package and the joint query vector in the view layer of the 3D scene evidence library, and sorts them from highest to lowest similarity. The agent dynamically determines a similarity threshold based on the information gap of the current task, and includes image frames higher than this threshold as supplementary evidence in the candidate evidence package.

[0069] In other embodiments, the acquisition timestamps and camera pose information of the initial keyframes contained in the initial evidence package are first parsed to determine the currently observed spatiotemporal coverage. Subsequently, the agent invokes a preset spatiotemporal exploration tool, which reads the time series and camera trajectory of all visual frames recorded in the 3D scene evidence library, automatically identifying unobserved frame sequences that are temporally continuous with existing keyframes but have significant differences in spatial pose (e.g., camera translation exceeding a set threshold or rotation angle greater than 30 degrees). Based on the scene coverage required for the task, the agent uniformly samples a preset number (e.g., 3 to 5 frames) of image frames from this sequence as exploratory supplementary evidence. For example, for a navigation planning task, the initial keyframes may only cover the starting area. The agent uses the spatiotemporal exploration tool to select one frame every 10 frames along the camera trajectory until reaching the end of the scene, thereby obtaining continuous visual evidence for the entire path and generating a complete candidate evidence package.

[0070] Step 105: Determine the response information corresponding to the initial query information based on the task type and candidate evidence package.

[0071] Optionally, in some embodiments, response information corresponding to the initial query information is determined based on the task type and candidate evidence package, including any of the following: In the case of a question-and-answer task, the natural language answer generated based on the candidate evidence package and the supporting visual frame corresponding to the natural language answer will be used as the response information. In the case of a visual positioning task, the identification information and location information of the target object instance generated based on the candidate evidence package will be used as the response information. In the case of a navigation planning task, the target area and path information generated based on the candidate evidence package will be used as the response information. When the task type is an operation planning task, the action sequence and execution constraint information generated based on the candidate evidence package will be used as the response information.

[0072] Question answering tasks refer to tasks where users expect to obtain an answer in natural language. In this task, the system needs to generate a concise and accurate natural language sentence or paragraph as the answer based on visual evidence (such as keyframes and cropped images) and object context information in the candidate evidence package, and provide one or more visual frames to support the answer. The answer can be a factual statement (e.g., yes / no, quantity, color) or a description of a scene state or object attribute.

[0073] In this context, supporting visual frames refer to the original image frames or partial cropped images that the system uses to corroborate the correctness of the natural language answers it generates in question-answering tasks.

[0074] For example, when asked what color the cup on the table is, the system generates a natural language answer of red, along with a supporting visual frame that clearly shows the red cup (e.g., frame 15).

[0075] Visual localization is a task where a user wants to locate one or more specific object instances in a 3D scene. In this task, the system needs to identify and select the target object instance that best matches the query description from the object layer based on a candidate evidence package, and output the object's unique identifier and its precise location in 3D space. Unlike question-answering tasks, the output of visual localization is structured, actionable data, rather than a natural language description.

[0076] The identification information of the target object instance is a unique identifier assigned by the system to each 3D object instance in the scene, which is usually a number or string (e.g., obj_15, cup_03).

[0077] Location information refers to the spatial occupancy description of the target object instance in the three-dimensional world coordinate system. In this invention, the location information is preferably in the form of a three-dimensional bounding box, that is, a set of coordinate values ​​representing the smallest hexahedron that can completely enclose the object.

[0078] For example, when querying to find the blue mug, the system outputs the visual location response as: target object number = cup_07, 3D bounding box coordinates.

[0079] Among them, navigation planning tasks refer to the type of task in which users expect the system to plan a feasible path from the current location (or a specified starting point) to the target location.

[0080] In this context, the target area refers to the semantic description or spatial extent of the final destination in a navigation planning task. It can be a room name (such as a bedroom), a furniture area (such as the area in front of a sofa), or a specific three-dimensional spatial block.

[0081] Route information is the core output of navigation planning tasks, typically comprising two sub-items: key landmarks and waypoint sequences. Key landmarks refer to highly identifiable objects or locations along the route. Operation planning tasks refer to tasks in which users expect the system to plan a series of executable physical actions to achieve interaction with or change the state of objects in the scene (such as grabbing, placing, opening, closing, etc.).

[0082] Among them, the action sequence refers to a series of action primitives output by the system in chronological order during the operation planning task.

[0083] Among them, execution constraint information refers to the preconditions or safety checks that need to be met before or during the execution of the action sequence in the operation planning task. These constraints include, but are not limited to: the current state of the target object (e.g., the door is closed), environmental conditions (e.g., there are no obstacles on the path), the robot's own state (e.g., the robotic arm has been reset to zero), and risk warnings during the execution process (e.g., handle with care).

[0084] For example, when querying to open the refrigerator door, the system outputs the following operation plan response: Target object = refrigerator_01, Action sequence = [Move to the refrigerator, raise your right hand, grasp the door handle, and pull outward], Execution constraint = Confirm that there are no obstructions in front of the refrigerator door.

[0085] In this embodiment, the task type is used as an explicit conditional input for the visual language agent, and corresponding toolsets, output schemas, and validation criteria are bound to different tasks. For question-answering tasks, the agent primarily invokes tools for viewpoint expansion, local clipping, and object context, outputting the answer, evidence frames, and supporting explanations. For visual positioning tasks, the agent further binds tools for object selection, spatial comparison, and 3D bounding box output, outputting the target object number and its 3D bounding box. For navigation planning tasks, the agent binds tools for landmark retrieval, regional relationships, and waypoint generation, outputting the target region, key landmarks, and waypoint sequences. For operation planning tasks, the agent binds tools for target reachability, status confirmation, and action precondition checks, outputting action primitive sequences and execution checks.

[0086] This application's embodiment of a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval constructs a multi-level queryable 3D scene evidence base, organically integrating object instances, all visual frames, and spatial relationships to preserve complete visual backtracking capabilities for subsequent reasoning. By generating retrieval hypothesis information, it transforms fuzzy natural language queries into structured retrieval strategies, effectively addressing uncertainties such as object label mismatch and imprecise descriptions. By generating an initial evidence package based on retrieval hypotheses, it achieves efficient and low-redundancy first-round evidence screening. Supplementary retrieval enables the system to dynamically acquire incremental evidence such as viewpoint expansion and local cropping as needed, avoiding information redundancy or loss caused by fixed input. Finally, it outputs structured response information according to different task types, realizing a unified framework for multiple tasks from question answering and localization to navigation and operation planning. Therefore, this invention significantly improves the ability of large language models to acquire fine-grained visual evidence and the accuracy of reasoning in 3D scene understanding, reduces information loss and reasoning upper bound limitations caused by scene graph compression, and enhances cross-task versatility and interpretability through task-conditional output.

[0087] Figure 2 This is the second flowchart of the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided in this application, as shown below. Figure 2 As shown, the method includes the following: Step 201: Obtain the 3D scene evidence library and initial inquiry information. The 3D scene evidence library organizes object instances, full visual frames, and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type.

[0088] Step 202: Generate retrieval hypothesis information based on the initial query information.

[0089] Step 203: Based on the retrieval hypothesis information, search the 3D scene evidence database to generate an initial evidence package.

[0090] For a detailed description of steps 201-203, please refer to the above embodiments, which will not be repeated here.

[0091] Step 204: Based on the initial evidence package, determine the preset tools to be invoked and the invocation order.

[0092] Optionally, in some embodiments, the preset tool includes at least one of the following: A view expansion tool is used to acquire visual frames that are adjacent to or complementary to the current view in a 3D scene evidence library. The local cropping tool is used to extract a close-up image of a target area from a visual frame; An object context reading tool used to read the semantic attributes, geometric boundary features, and visible view index of a target object instance from the object layer; The hypothesis switching tool is used to switch to other search hypothesis information for retrieval when the initial evidence package does not meet the matching requirements; A view exploration tool for retrieving additional visual frames from the 3D scene evidence library that were not included in the initial evidence package.

[0093] The view expansion tool is a pre-defined tool used to acquire visual frames adjacent to or complementary to the current view in the 3D scene evidence library. When the visual language agent determines that the existing image frames do not provide sufficient viewing angles for the target object (e.g., occlusion, excessively off-center view, or the need to observe back details), the agent invokes this tool. Upon receiving the request, the system searches the trajectory-aware view map, using the current view as a node, for other views that have temporal adjacency, camera pose adjacency, or similar viewing direction, and returns those image frames that can provide new viewing angles for the target object.

[0094] The local cropping tool is a preset tool used to extract a close-up image of a target region from an existing visual frame. This tool is invoked when the agent needs to perform fine-grained observation of a specific area in an image (e.g., identifying text, textures, status indicators, small parts, or local interaction traces on an object's surface). Based on the source image frame identifier provided by the agent and the coordinate parameters of the region to be cropped (such as a 2D detection box, the projected area of ​​an object instance, or a user-specified rectangle), the system accurately crops the corresponding region from the original high-resolution image and can perform resolution optimization before returning a close-up image focusing on local details.

[0095] The object context reading tool is used to directly read the structured metadata of target object instances from the object layer of the 3D scene evidence library. Unlike inference based on visual images, this tool directly accesses the data records of the object layer, and can quickly return the semantic attributes of the target object (such as category labels, multi-label semantics, detection confidence), geometric boundary features (such as 3D bounding box coordinates, point cloud data), and visible view index.

[0096] The hypothesis switching tool is used to proactively switch to other search hypotheses for re-searching when the currently active search hypothesis cannot meet the matching requirements. When the visual language agent infers based on the initial evidence package and finds that the obtained evidence is insufficient to draw reliable conclusions (e.g., the target object is not found, or the spatial relationship cannot be confirmed), the agent invokes this tool, and the system immediately lowers the priority of the current hypothesis and switches to the next untried hypothesis in a preset order (usually the direct hypothesis, the proxy hypothesis, and the context hypothesis).

[0097] Among them, view exploration is used to retrieve other visual frames in the 3D scene evidence library that are not included in the current initial evidence package.

[0098] In this embodiment of the disclosure, after determining the initial evidence package, the initial keyframes, object context, adopted hypothesis types, and current task objectives contained therein can be parsed. The agent performs intermediate inference based on the currently acquired evidence to identify information gaps or uncertainties, such as occlusion of the target object, insufficient local details, limited viewpoint, or low confidence of the current hypothesis. Subsequently, the agent dynamically selects the tools to be invoked and their invocation order from a preset toolset according to the task type and evidence gaps: if it is necessary to observe the same object from different angles, the viewpoint expansion tool is selected first; if it is necessary to zoom in on a local area, the local cropping tool is selected; if it is necessary to obtain the semantic or geometric attributes of the object, the object context reading tool is selected; if the current hypothesis retrieval result is not ideal, the hypothesis switching tool is selected to switch to other priority hypotheses; if it is necessary to explore a completely new area, the view exploration tool is selected. The agent organizes the selected tools and invocation order into a structured tool invocation plan and executes them sequentially.

[0099] Step 205: According to the calling order, call the preset tools to be called to obtain supplementary evidence from the 3D scene evidence library.

[0100] In this embodiment, the invocation can be based on the order of invocation, calling preset tools. For each tool to be invoked, the agent constructs corresponding invocation parameters according to the tool type—when invoking the view expansion tool, the target object instance and the desired observation direction or angle change must be specified; when invoking the local cropping tool, the source image frame and the coordinates of the area to be cropped must be specified; when invoking the object context reading tool, the target object instance number must be specified; when invoking the hypothesis switching tool, the target hypothesis type to switch to must be specified; when invoking the view exploration tool, the exploration strategy (such as the trajectory step size, novelty threshold, etc.) must be specified. After receiving the invocation request, the system's underlying layer performs the corresponding retrieval or image processing operation in the 3D scene evidence library, and uses the returned newly added image frame, local cropping image, object metadata, hypothesis retrieval results, or novel view as supplementary evidence.

[0101] Step 206: Generate a candidate evidence package based on the supplementary evidence and the initial evidence package.

[0102] In this embodiment of the disclosure, after determining the supplementary evidence and the initial evidence package, the returned new image frames, partial cropped images, object metadata, hypothesis retrieval results, or novel views can be injected as supplementary evidence into the agent's context. Each time the agent completes a tool call, it updates the current evidence set and intermediate conclusions based on the returned supplementary evidence, and then continues executing the next tool in the call order until all planned tool calls are completed, thereby gradually enriching the initial evidence package into a candidate evidence package containing multi-view, multi-detail, and multi-source information.

[0103] Step 207: Determine the response information corresponding to the initial inquiry information based on the task type and candidate evidence package.

[0104] For a detailed description of step 207, please refer to the above embodiments, which will not be repeated here.

[0105] This application's embodiment of a multi-task evidence reasoning method for 3D scenes based on view memory and hypothesis retrieval achieves accurate parsing of user intent and efficient first-round evidence retrieval by acquiring a multi-level queryable 3D scene evidence library containing object instances, full visual frames, and spatial relationships, and generating structured retrieval hypothesis information by combining initial query information. This results in an initial evidence package containing core visual cues. Based on this, the system adaptively determines the preset tools to be invoked and their invocation order according to the initial evidence package, and dynamically invokes the tools to retrieve supplementary evidence from the evidence library in this order. The supplementary evidence is then fused with the initial evidence package to generate a candidate evidence package, achieving on-demand and progressive evidence supplementation and effectively avoiding information redundancy or loss caused by fixed input. Finally, structured response information is output according to the task type and candidate evidence package, enabling unified processing of various 3D scene understanding tasks such as question answering, visual positioning, navigation planning, and operation planning. Therefore, this invention significantly improves the efficiency of evidence acquisition and reasoning accuracy of large language models in 3D scenes, reduces information loss caused by scene graph compression, and enhances the system's versatility and interpretability through task-conditional output. The following describes the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application. The 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval described below can be referred to in correspondence with the 3D scene multi-task evidence reasoning method based on view memory and hypothesis retrieval described above.

[0106] Figure 3 This is a schematic diagram of the structure of the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, as shown below. Figure 3 As shown, the device includes the following: This application provides a 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval, including the following modules: The acquisition module 301 is used to acquire a 3D scene evidence library and initial inquiry information. The 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type. The generation module 302 is used to generate retrieval hypothesis information based on the initial query information; The first retrieval module 303 is used to perform a retrieval in the three-dimensional scene evidence database based on retrieval hypothesis information to generate an initial evidence package; The second retrieval module 304 is used to call a preset tool that matches the initial evidence package to perform a supplementary retrieval in the three-dimensional scene evidence library to determine the candidate evidence package; The determination module 305 is used to determine the response information corresponding to the initial query information based on the task type and candidate evidence package.

[0107] According to the multi-task evidence reasoning device for three-dimensional scenes based on view memory and hypothesis retrieval provided in this application, the three-dimensional scene evidence base includes an object layer, a view layer, and a relation layer, wherein... The object layer is used to store object instances within the 3D scene, as well as the semantic attributes, geometric boundary features, and visible view indexes corresponding to the object instances; The view layer is used to store all the visual frames of the 3D scene, as well as the acquisition metadata corresponding to the full visual frames; The relationship layer is used to store the visibility relationships between the record object layer and the view layer, the spatial relationships between object instances, and the temporal adjacency relationships, pose adjacency relationships, and visibility overlap relationships between all visual frames.

[0108] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the generation module is specifically used for: Convert the initial query information into structured query information; Based on the structured query information, retrieval hypothesis information is generated. The retrieval hypothesis information includes direct hypothesis, proxy hypothesis and context hypothesis according to priority. Among them, direct assumptions are used to match target object instances or spatial relationships that appear explicitly in the initial query information and are identifiable in the object layer of the 3D scene evidence library; Proxy assumptions are used to search for object instances with similar functions when the target object instance cannot be matched by direct assumptions. Contextual assumptions are used to infer the location of other evidence based on regional clues or spatial relationships in the initial inquiry information.

[0109] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the relation layer is also used to construct a trajectory-aware view map based on temporal adjacency, pose adjacency and visibility overlap. The trajectory-aware view map uses all visual frames as nodes and adjacency as edges to characterize the adjacent features between visual frames. The first retrieval module 303 is specifically used for: Candidate views are determined in the view layer based on retrieval hypothesis information; The candidate view is evaluated based on preset indicators to determine the evaluation result. The preset indicators include at least one of the following: the visibility of the object instance in the candidate view, the degree of common coverage of the object instance and the anchor object instance in the candidate view, the imaging area of ​​the object instance, the viewing quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. Based on the evaluation results, the initial keyframes are determined; An initial evidence package is generated based on the initial keyframe and the object instance information associated with the initial keyframe.

[0110] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the preset tools include at least one of the following: A view expansion tool is used to acquire visual frames that are adjacent to or complementary to the current view in a 3D scene evidence library. The local cropping tool is used to extract a close-up image of a target area from a visual frame; An object context reading tool used to read the semantic attributes, geometric boundary features, and visible view index of a target object instance from the object layer; The hypothesis switching tool is used to switch to other search hypothesis information for retrieval when the initial evidence package does not meet the matching requirements; A view exploration tool for retrieving additional visual frames from the 3D scene evidence library that were not included in the initial evidence package.

[0111] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the second retrieval module 304 is specifically used for: Based on the initial evidence package, determine the preset tools to be invoked and the invocation order; According to the order of invocation, the preset tools to be invoked are invoked to obtain supplementary evidence from the 3D scene evidence library; Based on the supplementary evidence and the initial evidence package, a candidate evidence package is generated.

[0112] According to the 3D scene multi-task evidence reasoning device based on view memory and hypothesis retrieval provided in this application, the determining module is specifically used to perform any one of the following: In the case of a question-and-answer task, the natural language answer generated based on the candidate evidence package and the supporting visual frame corresponding to the natural language answer will be used as the response information. In the case of a visual positioning task, the identification information and location information of the target object instance generated based on the candidate evidence package will be used as the response information. In the case of a navigation planning task, the target area and path information generated based on the candidate evidence package will be used as the response information. When the task type is an operation planning task, the action sequence and execution constraint information generated based on the candidate evidence package will be used as the response information.

[0113] This application provides a multi-task evidence reasoning device for 3D scenes based on view memory and hypothesis retrieval. By constructing a multi-level queryable 3D scene evidence library, it organically integrates object instances, all visual frames, and spatial relationships, preserving complete visual backtracking capabilities for subsequent reasoning. By generating retrieval hypothesis information, it transforms fuzzy natural language queries into structured retrieval strategies, effectively addressing uncertainties such as object label mismatch and imprecise descriptions. By generating an initial evidence package based on retrieval hypotheses, it achieves efficient and low-redundancy first-round evidence screening. Through supplementary retrieval, the system can dynamically acquire incremental evidence such as viewpoint expansion and local cropping as needed, avoiding information redundancy or loss caused by fixed input. Finally, it outputs structured response information according to different task types, realizing a unified framework for multiple tasks from question answering and localization to navigation and operation planning. Therefore, this invention significantly improves the ability of large language models to acquire fine-grained visual evidence and the accuracy of reasoning in 3D scene understanding, reduces information loss and reasoning upper bound limitations caused by scene graph compression, and enhances cross-task versatility and interpretability through task-conditional output.

[0114] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions from the memory 430 to execute a multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval. This method includes: Acquire a 3D scene evidence library and initial query information. The 3D scene evidence library organizes object instances, all visual frames, and spatial relationships in the scene into a multi-level queryable structure, and the initial query information has a corresponding task type. Based on the initial query information, generate retrieval hypothesis information; Based on the retrieval hypothesis information, a search is performed in the 3D scene evidence database to generate an initial evidence package; Invoke the preset tool that matches the initial evidence package to perform supplementary searches in the 3D scene evidence library to determine candidate evidence packages; Based on the task type and candidate evidence package, determine the response information corresponding to the initial inquiry information.

[0115] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval provided by the above methods. The method includes: Acquire a 3D scene evidence library and initial query information. The 3D scene evidence library organizes object instances, all visual frames, and spatial relationships in the scene into a multi-level queryable structure, and the initial query information has a corresponding task type. Based on the initial query information, generate retrieval hypothesis information; Based on the retrieval hypothesis information, a search is performed in the three-dimensional scene evidence database to generate an initial evidence package; A preset tool matching the initial evidence package is invoked to perform a supplementary search in the three-dimensional scene evidence library to determine candidate evidence packages; Based on the task type and the candidate evidence package, determine the response information corresponding to the initial inquiry information.

[0117] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval provided by the methods described above, the method comprising: Acquire a 3D scene evidence library and initial inquiry information, wherein the 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type; Based on the initial query information, retrieval hypothesis information is generated; Based on the retrieval hypothesis information, a search is performed in the three-dimensional scene evidence database to generate an initial evidence package; A preset tool matching the initial evidence package is invoked to perform a supplementary search in the three-dimensional scene evidence library to determine candidate evidence packages; Based on the task type and the candidate evidence package, determine the response information corresponding to the initial inquiry information.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-task evidence reasoning method for three-dimensional scenes based on view memory and hypothesis retrieval, characterized in that, include: Acquire a 3D scene evidence library and initial inquiry information, wherein the 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type; Based on the initial query information, retrieval hypothesis information is generated; Based on the retrieval hypothesis information, a search is performed in the three-dimensional scene evidence database to generate an initial evidence package; A preset tool matching the initial evidence package is invoked to perform a supplementary search in the three-dimensional scene evidence library to determine candidate evidence packages; Based on the task type and the candidate evidence package, determine the response information corresponding to the initial inquiry information.

2. The method according to claim 1, characterized in that, The 3D scene evidence library comprises an object layer, a view layer, and a relationship layer, wherein... The object layer is used to store object instances within the 3D scene, as well as semantic attributes, geometric boundary features, and visible view indexes corresponding to the object instances; The view layer is used to store all the visual frames of the three-dimensional scene and the acquisition metadata corresponding to the full visual frames; The relationship layer is used to store and record the visibility relationship between the object layer and the view layer, the spatial relationship between object instances, and the temporal adjacency relationship, pose adjacency relationship, and visibility overlap relationship between the full set of visual frames.

3. The method according to claim 2, characterized in that, The step of generating retrieval hypothesis information based on the initial query information includes: The initial query information is converted into structured query information; Based on the structured query information, the retrieval hypothesis information is generated, and the retrieval hypothesis information includes direct hypothesis, proxy hypothesis and context hypothesis according to priority; The direct assumption is used to match the target object instance or the spatial relationship that appears explicitly in the initial query information and is identifiable in the object layer of the three-dimensional scene evidence library; The proxy hypothesis is used to retrieve object instances with similar functions as candidates when the target object instance cannot be matched by the direct hypothesis; The contextual assumptions are used to infer the location of other evidence for retrieval based on regional clues or spatial relationships in the initial inquiry information.

4. The method according to claim 2, characterized in that, The relation layer is also used to construct a trajectory-aware view graph based on the temporal adjacency relationship, pose adjacency relationship and visibility overlap relationship. The trajectory-aware view graph uses the full set of visual frames as nodes and the adjacency relationship as edges to represent the adjacency features between visual frames. The step of searching the three-dimensional scene evidence database based on the search hypothesis information to generate an initial evidence package includes: Based on the retrieval hypothesis information, candidate views are determined in the view layer; The candidate view is evaluated according to preset indicators to determine the evaluation result. The preset indicators include at least one of the following: the visibility of the object instance in the candidate view, the degree of common coverage of the object instance and the anchor object instance in the candidate view, the imaging area of ​​the object instance, the viewing quality of the candidate view, and the adjacency relationship of the candidate view on the trajectory-aware view map. Based on the evaluation results, the initial keyframe is determined; The initial evidence package is generated based on the initial keyframe and the object instance information associated with the initial keyframe.

5. The method according to claim 1, characterized in that, The preset tool includes at least one of the following: A view expansion tool is used to acquire visual frames that are adjacent to or complementary to the current view in the 3D scene evidence library. A local cropping tool is used to extract a close-up image of a local area of ​​the target region from the visual frame; An object context reading tool is used to read the semantic attributes, geometric boundary features, and visible view index of a target object instance from the object layer. The hypothesis switching tool is used to switch to other search hypothesis information for retrieval when the initial evidence package does not meet the matching requirements; A view exploration tool is used to retrieve additional visual frames in the 3D scene evidence library that are not included in the initial evidence package.

6. The method according to claim 5, characterized in that, The invocation of a preset tool matching the initial evidence package to perform supplementary searches in the 3D scene evidence library to determine candidate evidence packages includes: Based on the initial evidence package, determine the preset tools to be invoked and the invocation order; According to the calling order, the preset tool to be called is invoked to obtain supplementary evidence from the three-dimensional scene evidence library; The candidate evidence package is generated based on the supplementary evidence and the initial evidence package.

7. The method according to claim 1, characterized in that, The step of determining the response information corresponding to the initial query information based on the task type and the candidate evidence package includes any one of the following: When the task type is a question-answering task, the natural language answer generated based on the candidate evidence package and the supporting visual frame corresponding to the natural language answer are used as the response information. When the task type is a visual positioning task, the identification information and location information of the target object instance generated based on the candidate evidence package are used as the response information; When the task type is a navigation planning task, the target area and path information generated based on the candidate evidence package will be used as the response information; When the task type is an operation planning task, the action sequence and execution constraint information generated based on the candidate evidence package will be used as the response information.

8. A three-dimensional scene multi-task evidence reasoning device based on view memory and hypothesis retrieval, characterized in that, include: The acquisition module is used to acquire a 3D scene evidence library and initial inquiry information. The 3D scene evidence library organizes object instances, full visual frames and spatial relationships in the scene into a multi-level queryable structure, and the initial inquiry information has a corresponding task type. The generation module is used to generate retrieval hypothesis information based on the initial query information; The first retrieval module is used to search the three-dimensional scene evidence database based on the retrieval hypothesis information to generate an initial evidence package; The second retrieval module is used to call a preset tool that matches the initial evidence package to perform a supplementary retrieval in the three-dimensional scene evidence library in order to determine the candidate evidence package; The determination module is used to determine the response information corresponding to the initial query information based on the task type and the candidate evidence package.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional scene multi-task evidence reasoning method based on view memory and hypothesis retrieval as described in any one of claims 1 to 7.

Images