Evaluation method and device for embodied spatial intelligence understanding

Abstract

The application provides an evaluation method and device for embodied space intelligence understanding, and the method comprises the following steps: obtaining simulation task data, wherein the simulation task data comprises environment observation images and task instructions; inputting the simulation task data into a visual language model to generate three-dimensional space coordinates of a target object and an operation posture of an embodied agent according to the task instructions and the environment observation images; determining whether the object in the corresponding environment is the target object according to the three-dimensional space coordinates, and performing feasibility verification on the operation posture of the embodied agent by using a motion planner to obtain an embodied evaluation result. The application can truly and systematically reflect the actual performance of the visual language model in the embodied intelligent task, ensure to support the large-scale, automated and combinable embodied task simulation for space understanding, and realize the analysis and evaluation of the multi-modal model fine-grained and multi-dimensional space understanding ability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence evaluation technology, and in particular to an evaluation method and apparatus for embodied spatial intelligence understanding. Background Technology

[0002] With the rapid development of artificial intelligence technology, embodied intelligence, as a bridge connecting the digital and physical worlds, has become an important frontier in current research. The core of realizing embodied intelligence lies in endowing intelligent agents with powerful spatial intelligence, enabling them not only to perform basic scene perception but also to support high-level spatial reasoning and environmental interaction. Against this backdrop, Visual Language Models (VLMs), as a key implementation of spatial intelligent agents, are widely used to understand and process complex spatial information. To improve the performance of VLMs in spatial understanding, the industry has invested considerable effort in research and achieved significant progress. This has made the evaluation technology for VLM spatial intelligence a hot topic in the field. How to construct a scientific and comprehensive evaluation system to accurately measure the spatial understanding ability of models has become a key link in promoting the further development of embodied intelligence.

[0003] Currently, the main evaluation technologies for VLM spatial intelligence are the ROBOSATIAL benchmark, the Open6DOR benchmark, and the EmbodiedBench benchmark. The ROBOSATIAL benchmark focuses on understanding binary spatial relationships, subdivides the corresponding task categories, constructs evaluation tasks based on image datasets with existing 3D annotation information, and adopts an automated annotation strategy. The Open6DOR benchmark is positioned as an open instruction task, also focusing on spatial relationship understanding, but adds tasks for object rotation and orientation, semantic complexity, and the combination of spatial relationships and rotation and orientation. Its evaluation process relies on discriminative orientation selection. The EmbodiedBench benchmark is positioned as an open instruction task, with automated testing in a simulation environment. It decomposes the operation on the target object into the prediction of the target position and target orientation, and simplifies it by relying on prior knowledge visual recognition and action execution.

[0004] However, the construction of the aforementioned evaluation benchmarks is mostly based on heuristic methods, covering evaluation dimensions through empirical enumeration or focusing only on a specific dimension, such as spatial relationships. This lacks systematic task design, resulting in insufficiently comprehensive evaluation dimensions. Furthermore, the reliance on complex annotation processes easily introduces annotation errors. Additionally, to facilitate automated evaluation, these methods widely employ discriminative evaluation based on Visual Question Answering (VQA), relying on the construction of distractors to avoid bias. The model may arrive at the correct answer by eliminating distractors rather than based on its understanding of spatial information, causing the benchmarks to fail to reflect the model's true spatial understanding ability. Moreover, tasks in embodied scenarios have an "interaction-execution" attribute, which contradicts static testing based on VQA, creating a "test-deployment" gap that fails to reflect the model's performance in real-world scenarios and hinders iterative model development. Summary of the Invention

[0005] This invention provides an evaluation method and apparatus for embodied spatial intelligence understanding, which addresses the shortcomings of existing technologies in systematically reflecting the true spatial intelligence level of VLMs. It provides a true and systematic reflection of the actual performance of visual language models in embodied intelligence tasks, ensuring support for large-scale, automated, and composable embodied task simulations for spatial understanding, and enabling the analysis and evaluation of fine-grained, multi-dimensional spatial understanding capabilities of multimodal models.

[0006] This invention provides an evaluation method for embodied spatial intelligence understanding, comprising: acquiring simulation task data, including environmental observation images and task instructions; inputting the simulation task data into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied intelligent agent based on the task instructions and environmental observation images; determining whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates, and using a motion planner to perform a feasibility verification of the operational posture of the embodied intelligent agent to obtain the embodied intelligence evaluation result.

[0007] According to the evaluation method for embodied spatial intelligent understanding provided by the present invention, the method for acquiring simulation task data includes: randomly sampling actions and identifying targets; combining the randomly sampled task type with the instantiated task type to obtain a task representation; generating a corresponding function program based on the task representation; executing the function program on a 3D scene map obtained based on the randomly sampled task scenario to filter candidate target objects; instantiating the task based on the filtered candidate target objects and the task template of the corresponding task type obtained by random sampling to obtain task instructions; and acquiring an environmental observation image of the environmental state corresponding to the 3D scene map and combining it with the task instructions to obtain simulation task data.

[0008] According to the evaluation method for embodied spatial intelligent understanding provided by the present invention, after executing a function program on a corresponding three-dimensional scene graph to filter candidate target objects, the method includes: for the filtered candidate target objects, calling a motion planner to perform reachability verification on each candidate target object under the corresponding environmental state; when it is determined that the motion planner outputs a feasible motion trajectory for the corresponding candidate target object, the candidate target object is determined to have passed the verification and is retained; or, when it is determined that the motion planner fails to plan, the corresponding candidate target object is determined to be unreachable and is removed.

[0009] According to the evaluation method for embodied spatial intelligent understanding provided by the present invention, simulation task data is input into a visual language model to generate the three-dimensional spatial coordinates of a target object based on task instructions and environmental observation images. The method includes: inputting simulation task data into a visual language model to extract features from the environmental observation images to obtain visual features containing depth information and perspective relationships, and extracting features from the task instructions to obtain text features; using an attention mechanism to associate the visual and text features to locate the target object region in the corresponding environmental observation image that conforms to the semantics of the task instructions; and performing spatial relationship reasoning on the target object region based on depth information and perspective relationships to determine the relative position of the target object in three-dimensional space, mapping it to low-dimensional values, and outputting the three-dimensional spatial coordinates of the target object.

[0010] According to the present invention, an evaluation method for embodied spatial intelligence understanding includes an operational posture comprising a three-dimensional grasping point and a grasping direction. The method involves inputting simulation task data into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied agent based on task instructions and environmental observation images. This includes: inputting simulation task data into a visual language model to extract features from the environmental observation images to obtain visual features containing information about the object's geometric structure and spatial position; extracting features from the task instructions to obtain text features; associating the visual features with the text features corresponding to the task instructions through an attention mechanism to infer the three-dimensional grasping point position and grasping direction that conform to the grasping geometric constraints in the semantic space; and mapping the three-dimensional grasping point position and grasping direction to numerical coordinates to output the operational posture.

[0011] According to the evaluation method for embodied spatial intelligence understanding provided by the present invention, a motion planner is used to verify the feasibility of the operation posture of the embodied intelligent agent. The method includes: calling the motion planner and performing path search by combining the kinematic constraints of the embodied intelligent agent and the physical environment map corresponding to the environmental state; when the motion planner successfully finds a collision-free path that guides the corresponding embodied intelligent agent to the operation posture, the operation posture is determined to be correct; when the motion planner fails to find a feasible path, the operation posture is determined to be incorrect.

[0012] According to the evaluation method for embodied spatial intelligence understanding provided by the present invention, after verifying the feasibility of the operation posture of the embodied agent using a motion planner, the method includes: when it is determined that the motion planner fails to verify the operation posture, controlling the corresponding embodied agent to randomly select a distance to retreat from the target position, reach a new position and update the environmental observation; inputting the task instruction and the updated environmental observation into the visual language model to regenerate the operation posture of the embodied agent, and re-verifying the feasibility of the operation posture of the embodied agent using the motion planner, repeating the above steps iteratively until the feasibility verification is passed or the maximum number of attempts is reached.

[0013] This invention also provides an evaluation device for embodied spatial intelligence understanding, comprising: a data acquisition module for acquiring simulation task data, including environmental observation images and task instructions; a prediction module for inputting the simulation task data into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied intelligent agent based on the task instructions and environmental observation images; and an evaluation module for determining whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates, and for using a motion planner to perform feasibility verification on the operational posture of the embodied intelligent agent to obtain embodied intelligence evaluation results.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the evaluation method for embodied spatial intelligence understanding as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the evaluation method for embodied spatial intelligent understanding as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the evaluation method for embodied spatial intelligent understanding as described above.

[0017] The evaluation method and apparatus for embodied spatial intelligence understanding provided by this invention, by combining environmental observation images and task instructions, provides environmental context and task objectives for subsequent models, laying the foundation for visual language navigation and operation. Utilizing the powerful cross-modal understanding capabilities of the visual language model, it directly infers three-dimensional spatial coordinates and specific operational postures from natural language instructions and two-dimensional images, achieving direct quantitative testing of the visual language model's three-dimensional spatial perception and geometric reasoning capabilities. This simplifies the process from perception to execution, improves the system's generalization ability to handle complex instructions, and comprehensively reflects the true performance of the embodied intelligent agent in both cognitive and behavioral dimensions by comparing the positioning results with the target object and verifying the physical accessibility of the operational postures using a motion planner. This ensures that the evaluation results are not only semantically correct but also safe and executable in the physical world, achieving a full-link systematic evaluation from static cognitive correctness to dynamic execution feasibility. This realistically and systematically reflects the actual effectiveness of the visual language model in embodied intelligent tasks, ensuring support for large-scale, automated, and composable embodied task simulation for spatial understanding, and enabling the analysis and evaluation of fine-grained, multi-dimensional spatial understanding capabilities of multimodal models. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the evaluation method for embodied spatial intelligent understanding provided by the present invention. Figure 2 This is a schematic diagram of the interactive environment based on physical simulation provided by the present invention; Figure 3 This is a schematic diagram of the structure of the evaluation device for embodied spatial intelligent understanding provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Figure 1 This is a flowchart illustrating the evaluation method for embodied spatial intelligent understanding provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: S11, acquire simulation task data, which includes environmental observation images and task instructions; S12, input the simulation task data into the visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent according to the task instructions and environmental observation images; S13, based on the three-dimensional spatial coordinates, determine whether the object in the corresponding environment is the target object, and use the motion planner to verify the feasibility of the operation posture of the embodied intelligent agent, and obtain the embodied intelligence evaluation result.

[0022] The following is a detailed combination Figure 2 The present invention describes an evaluation method for embodied spatial intelligence understanding.

[0023] S11, acquire simulation task data, which includes environmental observation images and task instructions.

[0024] In this embodiment, acquiring simulation task data includes: randomly sampling actions and identifying targets; combining the randomly sampled task type with the instantiated task type to obtain a task representation; generating a corresponding function program based on the task representation, and executing the function program on the corresponding 3D scene map obtained based on the randomly sampled task scenario to filter candidate target objects; instantiating the task based on the filtered candidate target objects and the task template of the corresponding task type obtained by random sampling to obtain task instructions; acquiring an environmental observation image of the environmental state corresponding to the 3D scene map, and combining it with the task instructions to obtain simulation task data.

[0025] It should be noted that by performing dual random sampling on both the scene and task type, the limitations of fixed datasets are broken, and rich and varied task backgrounds are constructed, laying the foundation for generating large-scale and diverse training data. By transforming task representations into executable function programs and running them on the 3D scene graph, irrelevant objects are quickly eliminated using the topological logic of the scene, accurately locking candidate targets that conform to the task logic. This significantly improves the logical rationality and execution efficiency of task generation. Furthermore, task templates are used to instantiate the selected candidate objects to automatically generate grammatically correct, semantically clear, and context-specific natural language instructions, reducing the human cost of data annotation and ensuring the consistency between instructions and the scene environment. In addition, environmental observation images are bound to task instructions, so that the obtained simulation task data not only contains objective visual information but also subjective task intent, ensuring that the data can be directly used to evaluate the model, thereby effectively improving the model's understanding and execution capabilities in complex environments.

[0026] It's important to add that task types include spatial understanding dimensions, reference frames, and reference objects. Spatial understanding dimensions include positional relationships, distance, orientation, and attributes such as dimensions and volume. Positional relationships and orientation depend on a reference frame, which can be relative, intrinsic, or absolute. A reference frame depends on a reference object. For example, in the task "pick up a book to the left of the picture frame," the "picture frame" has a front and back, thus providing its own reference frame. If replaced with a "sphere" without a clear front or back attribute, the default reference frame is the "observer's" coordinate system. Correspondingly, task types are represented as triples C = (S, F, O), where S represents the spatial understanding dimension, F represents the reference frame, and O represents the reference object. This systematically summarizes the main spatial understanding dimensions and task context categories, enabling large-scale, automated, and composable task generation, covering more comprehensive task scenarios, and supporting fine-grained model diagnostics.

[0027] Furthermore, for the instantiation of reference objects, value filtering can be performed on the corresponding 3D scene graph to remove invisible objects, and the reference objects can be randomly initialized based on the value-filtered 3D scene graph. The reference frame and spatial understanding dimension can be determined based on random sampling, such as the reference frame being the closest. No further restrictions are made here.

[0028] In addition, task representation includes task type, action, and recognition. Actions include grasping and placing actions, and target recognition is equivalent to the target recognition requirements in the task instructions, as described below and will not be repeated here. Similarly, task representation can also be represented as a triple, T = (C, A, P), where C represents the task format, A represents the action, and P represents target recognition.

[0029] For example, for the task instruction "take the farthest book from you", the corresponding functional program is filterFarthest(filter(book, TABLE), viewer)), which first filters all the books in the scene corresponding to the desk, and then locates the book that is farthest from the observer, thus obtaining the candidate target object.

[0030] In addition, task scenarios can be designed according to actual design requirements, such as desktop tasks and bookshelf tasks, which are used for grabbing and placing tasks respectively. For example, taking a book from the desktop and placing it on the bookshelf. By randomly sampling the type, quantity, and layout of objects in the scenario, task scenarios of different complexities can be obtained. No further limitations are made here.

[0031] Furthermore, the task scenario is represented as a 3D scene graph, where each node represents an object, and each edge connects two objects, defining the spatial relationship between the two connected objects. Further, in the simulation scenario, by acquiring the true annotations of all objects, such as object dimensions, distances between objects, and relationships, dependence on external annotation tools and annotation errors are eliminated. The 3D scene graph of the task scenario is randomly sampled, and the Isaac Sim simulation tool is used for physical rendering to ensure its physical validity. This allows for evaluation using a physically simulated environment. Compared to real-world evaluation, this method has advantages in scalability and reproducibility, accelerating model iteration and facilitating fair comparisons. Simultaneously, the initial pose of the embodied intelligent agent's robotic arm is randomly sampled in the scene, and a wrist camera and a scene-wide camera are configured to provide environmental observation.

[0032] Since task representation only defines the abstract representation of the instruction task, in order to obtain a visual language model (VLM) and / or a natural language form that humans can directly understand, a set of task templates can be defined on the basis of task representation. That is, multiple task templates are labeled for each task type, such as 3-6, to achieve the diversity of task instructions in natural language expression. Different templates are different expressions of the same task. Thus, by systematically defining spatial understanding tasks, automated and large-scale task generation can be achieved, and a physical simulation-based embodied evaluation platform can be provided. Based on physical simulation, rich task scenarios centered on embodied spatial understanding can be rendered, supporting interactive active perception and execution, and realizing active perception oriented towards action execution.

[0033] Assuming spatial understanding is based on distance, the frame of reference is the observer, and the action is "pick," with C = (distance, viewer, *) and A = "Pick," then task templates can include "[A] a book among the books [R]you" and "from the books on the table, [A] the one that is [R]you," where * indicates that the task format is independent of a specific frame of reference, A represents the action, and R represents the distance inference type, such as Closest / Farthest. Given a value for R, such as R = "Closest," the two task templates above correspond to the same function: filterClosest(filter(book,Table), viewer). Furthermore, to maintain syntactic consistency, variable types can be rewritten to their corresponding values, such as rewriting Pick as take and Closest as closest to; no further restrictions are imposed here.

[0034] Based on the above systematic design, 148 spatial reasoning task types can be constructed, distributed across 65 task families, including 31 "pick" families and 34 "place" families.

[0035] In addition, after executing the function program on the corresponding 3D scene graph to filter candidate target objects, the process includes: for the filtered candidate target objects, calling the motion planner to perform reachability verification on each candidate target object under the corresponding environmental state; when it is determined that the motion planner outputs a feasible motion trajectory for the corresponding candidate target object, the candidate target object is deemed to have passed the verification and is retained; or, when it is determined that the motion planner fails to plan, the corresponding candidate target object is deemed to be unreachable and is removed.

[0036] It should be noted that by introducing a motion planner for reachability verification, we can consider the kinematic feasibility of the embodied agent under specific environmental geometric constraints while taking into account the semantic attributes of the object. This ensures that the generated task objectives are physically reachable. Furthermore, by eliminating target objects that fail to be planned, we can automatically filter out tasks that are logically valid but physically impossible to complete, thus avoiding the model learning incorrect associations or getting stuck in unsolvable exploration during the evaluation process.

[0037] S12 inputs the simulation task data into the visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent based on the task instructions and environmental observation images.

[0038] In this embodiment, simulation task data is input into a visual language model to generate the three-dimensional spatial coordinates of the target object based on the task instructions and environmental observation images. This includes: inputting simulation task data into the visual language model to extract features from the environmental observation images to obtain visual features containing depth information and perspective relationships, and extracting features from the task instructions to obtain text features; using an attention mechanism to associate the visual and text features to locate the target object region that conforms to the semantics of the task instructions in the corresponding environmental observation images; and performing spatial relationship reasoning on the target object region based on depth information and perspective relationships to determine the relative position of the target object in three-dimensional space, mapping it to low-dimensional values, and outputting the three-dimensional spatial coordinates of the target object.

[0039] It should be noted that by explicitly preserving depth and perspective relationships in visual features, the model not only acquires the appearance information of objects but also obtains key spatial structural information. This provides the necessary data foundation for deriving three-dimensional coordinates from two-dimensional images. Furthermore, by using an attention mechanism to focus textual features on key areas of visual features, the model effectively filters out background noise in the image that is irrelevant to the current task, accurately captures the target object region that highly matches the semantics of the instruction, and improves the model's anti-interference ability in complex scenes. Moreover, by combining geometric rules for spatial reasoning, the model maps the two-dimensional projection of visual perception to three-dimensional coordinates in the real world and outputs them in low-dimensional numerical form. This allows the output of the visual language model to be directly used by downstream control or planning modules, thus opening up the key link from perception to execution.

[0040] In addition, the operational posture includes the three-dimensional grasping point and grasping direction. Simulation task data is input into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied intelligent agent based on task instructions and environmental observation images. This includes: inputting simulation task data into the visual language model to extract features from the environmental observation images to obtain visual features containing information about the object's geometric structure and spatial position; extracting features from the task instructions to obtain text features; associating the visual features with the text features corresponding to the task instructions through an attention mechanism to infer the three-dimensional grasping point position and grasping direction that conform to the grasping geometric constraints in the semantic space; mapping the three-dimensional grasping point position and grasping direction to numerical coordinates and outputting the operational posture.

[0041] It should be noted that by explicitly extracting visual features containing geometric structural information, the model can perceive key physical properties such as the shape and surface normal vectors of objects, providing the necessary geometric basis for determining how to grasp them. This overcomes the shortcomings of traditional methods that only focus on the position of objects. Furthermore, by using an attention mechanism to combine the action intention in the text command with the geometric structure of the object, the model can directly infer the optimal grasping point and direction that avoids obstacles and conforms to mechanical stability within the semantic space. This avoids generating physically infeasible or incorrect postures that may cause the object to slip. Moreover, by transforming the inferred abstract grasping concept into specific numerical coordinates and direction vectors, it directly provides executable motion parameters for the robotic arm control, significantly reducing the computational complexity of the subsequent motion planning module and improving the overall efficiency and success rate of the embodied intelligent agent in performing tasks.

[0042] It should be added that the embodied agent achieves grasping and placement through the closing and opening of the gripper. Taking grasping as an example, the embodied agent needs to generate the grasping posture, which consists of the three-dimensional grasping point (position) (x, y, z) and the grasping orientation (r). x r y r z It consists of two parts: grasping point and target recognition, which use the same point prediction strategy to combine spatial reasoning oriented towards understanding and execution oriented, grasping direction (r) x r y r z It is also generated by the visual language model, and the placement can refer to the logic of the grabbing, which will not be repeated here.

[0043] In addition, visual language models, based on spatial understanding, solve embodied perception and execution tasks. Compared with traditional discriminative evaluation based on visual question answering, they adopt a generative evaluation paradigm, thereby minimizing dataset bias.

[0044] S13, based on the three-dimensional spatial coordinates, determine whether the object in the corresponding environment is the target object, and use the motion planner to verify the feasibility of the operation posture of the embodied intelligent agent, and obtain the embodied intelligence evaluation result.

[0045] In this embodiment, obtaining the embodied intelligence evaluation result includes: when the object in the corresponding environment is determined to be the target object based on the three-dimensional spatial coordinates, the target recognition is determined to be correct, and a first labeling result is obtained; the operation posture of the embodied intelligent agent is verified for feasibility using a motion planner, and in response to the verification passing, the action planning is determined to be correct, and a second labeling result is obtained; based on the first labeling result and the second labeling result, the embodied intelligence evaluation result is obtained.

[0046] It should be noted that, for example, if the task instruction is "take the farthest book from you", the visual language model will record the three-dimensional spatial coordinates of the target object as correct as long as they fall on the target book. This generative evaluation does not rely on the construction of interference terms, which can reduce the risk of introducing dataset bias, and is closely related to the subsequent actions of the embodied agent.

[0047] Furthermore, the feasibility of the operation posture of the embodied agent is verified by using a motion planner, including: calling the motion planner and performing path search by combining the kinematic constraints of the embodied agent with the physical environment map corresponding to the environmental state; when the motion planner successfully finds a collision-free path that guides the corresponding embodied agent to the operation posture, the operation posture is determined to be correct; when the motion planner fails to find a feasible path, the operation posture is determined to be incorrect.

[0048] It should be noted that by introducing kinematic constraints and physical environment maps, the verification process no longer relies solely on visual perception. Instead, it incorporates the physical reality of whether the embodied intelligent agent can reach the target location. This effectively avoids situations where visual predictions are accurate but cannot be executed due to joint limitations or environmental obstacles, such as occlusions or narrow spaces. Furthermore, the correctness of the posture is determined by using path search results, thereby constructing a safe closed loop from perception prediction to motion verification. This automatically filters out postures that conform to semantic logic but pose a collision risk in physical space, thus ensuring that the final executed operation commands are safe and feasible in the real physical world.

[0049] In an optional embodiment, after using a motion planner to verify the feasibility of the embodied agent's operational posture, the process includes: when it is determined that the motion planner fails to verify the operational posture, controlling the corresponding embodied agent to randomly select a distance to retreat from the target position, reach a new position, and update the environmental observations; inputting the task instructions and the updated environmental observations into the visual language model to regenerate the embodied agent's operational posture, and reusing the motion planner to verify the feasibility of the embodied agent's operational posture, repeating the above steps iteratively until the feasibility verification is passed or the maximum number of attempts is reached.

[0050] It should be noted that by controlling the agent to retract and update observations when planning fails, the limitations of static decision-making are broken. The relative spatial relationship between the embodied agent and the target object is actively changed, which may eliminate occlusion, avoid singularities of the robotic arm, or discover a better approach angle. By constructing an iterative loop of perception, prediction, verification, and adjustment, the system is endowed with robustness and fault tolerance in the face of complex dynamic environments. This ensures that when the first attempt is not feasible, alternative solutions can be actively explored, thereby significantly improving the final success rate of complex tasks.

[0051] For example, suppose the model predicts the target pose (x, y, z, r) including three-dimensional position and orientation in the Tth iteration. x r y r z The system attempts to achieve the target pose using a motion planner. If it fails, it randomly retreats a certain distance d near the target position to reach a new position (x+d, y+d, z+d), updates the environmental observations, and prepares to predict the target pose at time T+1.

[0052] In the actual experiment, the accuracy of the model was evaluated in both the target localization and action execution phases. The execution phase metric was defined as the proportion of predicted operational postures accepted by the motion planner. Due to the accuracy limitations of physical simulation and the planner, an accepted operational posture does not necessarily mean the task is successfully completed. For example, an object might bounce off the target area after being placed, but because the focus is on reasoning rather than execution, the object meeting the target posture is considered a successful execution. Mainstream open-source and closed-source multimodal base models were selected, such as Gemini 2.5-Pro, Qwen 3-VL, InternVL 3.0, and RoboBrain 2.0. The main experimental observations are as follows: Overall, all models performed better on the "pick" task than on the "place" task. Specifically, RoboBrain 2.0 achieved a target localization accuracy of 20.58% on the "pick" task, but its execution success rate was only 2%. This means that fine-tuning the model on a general spatial reasoning dataset can improve passive spatial reasoning tasks, but this improvement does not translate into subsequent spatial reasoning capabilities for task execution. This experiment suggests that future model fine-tuning should incorporate execution-oriented spatial reasoning data. Furthermore, Qwen3-VL-8B-Instruct outperformed the larger model Qwen3-VL-235B-A22B-Instruct in both the "pick" and "place" tasks, indicating that larger models do not necessarily have stronger spatial understanding capabilities (see Table 1). The success rate indicates that both localization and execution were correct.

[0053] Table 1 shows the positioning accuracy (%), execution acceptance rate (%), and task success rate (%) for each evaluation model.

[0054] Furthermore, while reflection can improve target localization performance in most cases, it is detrimental to action execution. One possible reason is that reflection capability relies on the model's basic spatial reasoning ability, while the action execution phase mainly depends on the model's understanding of 3D rotation, but existing multimodal models are relatively weak in this aspect.

[0055] Regarding observations on spatial reasoning dimensions, the performance of the target localization stage across different spatial reasoning dimensions was analyzed. Referring to Table 2, it can be seen that, regardless of whether it's the "pick" or "place" task, the model's performance in the distance perception dimension lags significantly behind other spatial reasoning dimensions, indicating that future work needs to further enhance the model's understanding of physical distance. Furthermore, the performance differences between Gemini 2.5-Pro ​​and RoboBrain 2.0 across different spatial reasoning dimensions are relatively small, likely because the fine-tuning data for these two models covers these four reasoning dimensions more evenly.

[0056] Table 2 shows the performance in the spatial dimension.

[0057] Finally, regarding the analysis of the conditions for successful action execution, the common characteristics of successful action execution are mainly analyzed, which are broken down into (a) the average number of attempts required for target localization, (b) the average number of attempts required for action execution, (c) the distance to the target object when successful execution, and (d) the distance to the target object in the previous step before successful execution. Referring to Table 3, it is found that models with a high success rate, such as InternVL3, require an average of approximately 2.56 (pick) and 2.07 (place) moves, that is, by repeatedly trying, they gradually approach the target object. This is reflected in the minimum distance to the target object when successful execution and in the step before successful execution. On the other hand, models with too few or too many average moves usually have weak action execution capabilities. That is, a single execution may result in a collision leading to task failure and termination, or even after exhausting the upper limit of the number of attempts, the action may still fail to execute successfully.

[0058] Table 3 shows the average number of attempts required to achieve successful positioning and execution, as well as the average distance between the target and the end effector at the time of successful execution and before successful execution.

[0059] In summary, this invention, by combining environmental observation images and task instructions, provides the environmental context and task objectives for subsequent models, laying the foundation for visual language navigation and operation. Utilizing the powerful cross-modal understanding capabilities of the visual language model, it directly infers three-dimensional spatial coordinates and specific operational postures from natural language instructions and two-dimensional images. This enables direct quantitative testing of the visual language model's three-dimensional spatial perception and geometric reasoning capabilities, simplifying the process from perception to execution, improving the system's generalization ability to handle complex instructions, and comprehensively reflecting the embodied agent's true performance in both cognitive and behavioral dimensions by comparing the positioning results with the target object and verifying the physical reachability of the operational posture using a motion planner. This ensures that the evaluation results are not only semantically correct but also physically executable, achieving a full-link systematic evaluation from static cognitive correctness to dynamic execution feasibility. This realistically and systematically reflects the actual effectiveness of the visual language model in embodied intelligent tasks, ensuring support for large-scale, automated, and composable embodied task simulations oriented towards spatial understanding, and enabling the analysis and evaluation of fine-grained, multi-dimensional spatial understanding capabilities of multimodal models.

[0060] The evaluation device for embodied spatial intelligent understanding provided by the present invention will be described below. The evaluation device for embodied spatial intelligent understanding described below can be referred to in correspondence with the evaluation method for embodied spatial intelligent understanding described above.

[0061] Figure 3 A schematic diagram of an evaluation device for embodied spatial intelligence understanding is shown. The device includes: Data acquisition module 31 acquires simulation task data, which includes environmental observation images and task instructions; The prediction module 32 inputs the simulation task data into the visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent based on the task instructions and environmental observation images. Evaluation module 33 determines whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates, and uses a motion planner to verify the feasibility of the operation posture of the embodied intelligent agent, thereby obtaining the embodied intelligence evaluation result.

[0062] It should be noted that the specific principles of the embodiments of the present invention are the same as those of the method embodiments described above. For details, please refer to the method embodiments above. More detailed explanations will not be repeated here.

[0063] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an evaluation method for embodied spatial intelligence understanding. This method includes: acquiring simulation task data, which includes environmental observation images and task instructions; inputting the simulation task data into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied intelligent agent based on the task instructions and environmental observation images; determining whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates; and using a motion planner to perform a feasibility check on the operational posture of the embodied intelligent agent to obtain the embodied intelligence evaluation result.

[0064] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the evaluation method for embodied spatial intelligence understanding provided by the above methods. The method includes: acquiring simulation task data, which includes environmental observation images and task instructions; inputting the simulation task data into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent based on the task instructions and environmental observation images; determining whether the object in the corresponding environment is the target object based on the three-dimensional spatial coordinates; and using a motion planner to perform a feasibility verification of the operating posture of the embodied intelligent agent to obtain the embodied intelligence evaluation result.

[0066] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the evaluation method for embodied spatial intelligence understanding provided by the methods described above. The method includes: acquiring simulation task data, which includes environmental observation images and task instructions; inputting the simulation task data into a visual language model to generate three-dimensional spatial coordinates of a target object and the operational posture of an embodied intelligent agent based on the task instructions and environmental observation images; determining whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates; and using a motion planner to perform a feasibility verification of the operational posture of the embodied intelligent agent to obtain an embodied intelligence evaluation result.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An evaluation method for embodied spatial intelligence understanding, characterized in that, include: Acquire simulation task data, which includes environmental observation images and task instructions; The simulation task data is input into the visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent according to the task instructions and the environmental observation images. Based on the three-dimensional spatial coordinates, it is determined whether the object in the corresponding environment is the target object, and the feasibility of the operation posture of the embodied intelligent agent is verified by the motion planner to obtain the embodied intelligence evaluation result.

2. The evaluation method for embodied spatial intelligent understanding according to claim 1, characterized in that, Acquire simulation task data, including: Random sampling actions and target recognition are combined with random sampling task types and instantiated task types to obtain a task representation; Based on the task representation, a corresponding function program is generated and executed on the corresponding 3D scene map obtained based on the random sampling task scenario to filter candidate target objects; Based on the candidate target objects obtained through screening, and combined with the task templates of the corresponding task types obtained through random sampling, the task is instantiated to obtain the task instructions; Obtain environmental observation images corresponding to the environmental state of the three-dimensional scene map, and combine them with the task instructions to obtain simulation task data.

3. The evaluation method for embodied spatial intelligent understanding according to claim 2, characterized in that, After executing the function program on the corresponding 3D scene graph to filter candidate target objects, the process includes: For each candidate target object obtained from the screening, the motion planner is invoked to perform reachability verification on each candidate target object under the corresponding environmental state. When the motion planner outputs a feasible motion trajectory for a corresponding candidate target object, it determines that the candidate target object passes the verification and retains the corresponding candidate target object; or, When the motion planner fails to plan, the corresponding candidate target object is determined to be unreachable and is removed.

4. The evaluation method for embodied spatial intelligent understanding according to claim 1, characterized in that, The simulation task data is input into a visual language model to generate the three-dimensional spatial coordinates of the target object based on the task instructions and the environmental observation images, including: The simulation task data is input into the visual language model to extract features from the environmental observation image, obtaining visual features containing depth information and perspective relationships, and to extract features from the task instructions, obtaining text features. Based on the visual features and the text features, an attention mechanism is used to associate them in order to locate the target object region that conforms to the semantics of the task instruction in the corresponding environmental observation image. Based on the depth information and the perspective relationship, spatial relationship reasoning is performed on the target object region to determine the relative position of the target object in three-dimensional space, which is then mapped to a low-dimensional value, and the three-dimensional spatial coordinates of the target object are output.

5. The evaluation method for embodied spatial intelligence understanding according to claim 1, characterized in that, The operational posture includes a three-dimensional grasping point and a grasping direction; the simulation task data is input into a visual language model to generate the three-dimensional spatial coordinates of the target object and the operational posture of the embodied intelligent agent based on the task instructions and the environmental observation image, including: The simulation task data is input into the visual language model to extract features from the environmental observation image, thereby obtaining visual features containing object geometric structure information and spatial positional relationships, and to extract features from the task instructions to obtain text features. By associating the visual features with the text features corresponding to the task instructions through an attention mechanism, the position and direction of the three-dimensional grasping point that conform to the grasping geometric constraints are inferred in the semantic space. The three-dimensional gripping point position and gripping direction are mapped to numerical coordinates, and the operation posture is output.

6. The evaluation method for embodied spatial intelligent understanding according to claim 1, characterized in that, The feasibility of the embodied intelligent agent's operational postures is verified using a motion planner, including: The motion planner is invoked to perform path search by combining the kinematic constraints of the embodied agent with the physical environment map corresponding to the environmental state. When the motion planner successfully finds a collision-free path that guides the corresponding embodied intelligent agent to the operation posture, the operation posture is determined to be correct. If the motion planner fails to find a feasible path, the operation posture is deemed incorrect.

7. The evaluation method for embodied spatial intelligence understanding according to claim 1, characterized in that, After verifying the feasibility of the embodied intelligent agent's operational posture using a motion planner, the process includes: When the motion planner fails to verify the operation posture, it controls the corresponding embodied intelligent agent to randomly select a distance to retreat from the target position, reach a new position, and update the environmental observation; The task instructions and updated environmental observations are input into the visual language model to regenerate the embodied agent's operating posture. The motion planner is then used to re-verify the feasibility of the embodied agent's operating posture. The above steps are repeated iteratively until the feasibility verification is passed or the maximum number of attempts is reached.

8. An evaluation device for embodied spatial intelligence understanding, characterized in that, include: The data acquisition module acquires simulation task data, which includes environmental observation images and task instructions. The prediction module inputs the simulation task data into the visual language model to generate the three-dimensional spatial coordinates of the target object and the operating posture of the embodied intelligent agent based on the task instructions and the environmental observation images. The evaluation module determines whether an object in the corresponding environment is the target object based on the three-dimensional spatial coordinates, and uses a motion planner to verify the feasibility of the operation posture of the embodied intelligent agent, thereby obtaining the embodied intelligence evaluation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the evaluation method for embodied spatial intelligent understanding as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the evaluation method for embodied spatial intelligent understanding as described in any one of claims 1 to 7.

Images