Three-dimensional space anaphora reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional space anaphora reasoning method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multi-modal image data containing color visual information and depth information; inputting the RGB-D image data and the natural language instruction into a pre-trained visual language large model, and outputting a text containing an explicit reasoning process and target point coordinates conforming to spatial constraints; wherein the visual language large model is obtained through combined training of two-stage supervised learning fine tuning of depth alignment and space understanding enhancement and reinforcement learning fine tuning based on a display reasoning process; the visual language large model comprises an independent depth encoder, and the depth encoder is used for processing depth information. Through the method provided by the invention, the comprehensive performance in the complex space anaphora task is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a three-dimensional space reference reasoning method, device, electronic device and storage medium. Background Art

[0002] In recent years, with the expanding application of robots and intelligent systems in daily life and industry, spatial intelligence for open-world environments has become a growing research hotspot. For robots to interact precisely in complex scenarios, they need to fully understand and reason about the three-dimensional space within their environment. Spatial referring is a core issue. Spatial referring requires that, given multimodal sensory information (such as RGB or RGB-D images) and text instructions with spatial constraints, the system be able to accurately predict target points that satisfy specific spatial relationships, thereby providing a basis for downstream functions (such as navigation, grasping, and placement).

[0003] However, existing technologies have significant shortcomings in single-step spatial perception, multi-step complex reasoning, and the fusion of depth and RGB modalities, making it difficult to achieve both high-precision recognition and reasoning of complex three-dimensional spatial relationships. Furthermore, the lack of support for explicit reasoning processes limits the model's adaptability in open-world environments.

[0004] How to enhance the three-dimensional spatial understanding and multi-step reasoning capabilities of multimodal large models for complex instructions, thereby effectively improving their three-dimensional spatial reference reasoning capabilities, is a technical problem that needs to be solved at present. Summary of the Invention

[0005] The present invention provides a three-dimensional spatial reference reasoning method, device, electronic device and storage medium to solve the defects in the prior art.

[0006] The present invention provides a three-dimensional spatial reference reasoning method, comprising the following steps: Obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; Input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0007] According to a three-dimensional spatial reference inference method provided by the present invention, the visual language large model further includes: an image encoder, a multimodal projection layer and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer; The step of inputting the RGB-D image data and the natural language instructions into a pre-trained visual language model and outputting text containing an explicit reasoning process and target point coordinates that conform to spatial constraints includes: Extracting image features of the RGB-D image data by the image encoder, and extracting depth features of the RGB-D image data by the depth encoder; Mapping the image features and the depth features to a feature space matching the large language model via the image projection layer and the depth projection layer respectively; Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

[0008] According to a three-dimensional spatial reference reasoning method provided by the present invention, the two-stage supervised learning fine-tuning process includes: Obtaining a training dataset with RGB-D annotations, and updating the depth projection layer parameters based on the training dataset to achieve depth alignment; Obtain a dataset comprising three-dimensional annotations and an explicit reasoning process and a general instruction following dataset, and jointly update the image encoder parameters, the depth encoder parameters, the projection layer parameters, and the large language model parameters based on the dataset comprising three-dimensional annotations and an explicit reasoning process and the general instruction following dataset to achieve enhanced spatial understanding.

[0009] According to a three-dimensional spatial reference reasoning method provided by the present invention, the reinforcement learning fine-tuning process includes: Obtaining historical RGB-D image data and historical natural language instructions, and inputting the historical RGB-D image data and the historical natural language instructions into a visual language model fine-tuned by supervised learning to generate multiple candidate inference paths; wherein each of the multiple candidate inference paths includes: text containing an explicit inference process and target point coordinates that meet spatial constraints; For each candidate reasoning path among the multiple candidate reasoning paths, evaluating it using a pre-designed reward function to obtain a reward value; Based on the reward value of each candidate reasoning path, the visual language model after supervised learning fine-tuning is optimized through the group relative strategy optimization GRPO algorithm to obtain the trained visual language model.

[0010] According to a three-dimensional space reference reasoning method provided by the present invention, the reward function includes: a result-type reward function and a process-type reward function; The step of evaluating each of the plurality of candidate reasoning paths using a pre-designed reward function to obtain a reward value includes: Based on each candidate reasoning path among the multiple candidate reasoning paths, a format reward and a coordinate prediction accuracy reward are obtained through the result-like reward function; wherein the format reward and the coordinate prediction accuracy reward are used to: evaluate the compliance of the output format and the proximity of the target point coordinates to the true target point; Based on each candidate reasoning path among the plurality of candidate reasoning paths, obtaining an explicit step format and a key step accuracy through the process-like reward function; wherein the explicit step format and the key step accuracy are used to: evaluate the correctness of the explicit reasoning process; The reward value is obtained based on the format reward and the coordinate prediction accuracy reward, as well as the explicit step format and the key step accuracy.

[0011] According to a three-dimensional spatial referential reasoning method provided by the present invention, based on the reward value of each candidate reasoning path, the visual language model after supervised learning fine-tuning is optimized by the group relative strategy optimization GRPO algorithm to obtain the trained visual language model, including: Normalizing the reward value of each candidate reasoning path to obtain a relative advantage value of each candidate reasoning path; The strategy is updated based on the relative advantage value of each candidate reasoning path, and a KL divergence regularization term is added during the strategy update process to limit the deviation between the new strategy and the reference strategy, thereby obtaining the trained visual language model.

[0012] The present invention also provides a three-dimensional spatial reference reasoning device, comprising the following modules: An acquisition module, configured to acquire RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; An inference module, configured to input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit inference process and target point coordinates that conform to spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0013] According to a three-dimensional spatial reference inference device provided by the present invention, the visual language large model further includes: an image encoder, a multimodal projection layer and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer; The reasoning module is specifically used to: Extracting image features of the RGB-D image data by the image encoder, and extracting depth features of the RGB-D image data by the depth encoder; Mapping the image features and the depth features to a feature space matching the large language model via the image projection layer and the depth projection layer respectively; Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any one of the three-dimensional space reference reasoning methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the three-dimensional space reference reasoning methods described above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the three-dimensional space reference reasoning methods described above.

[0017] The present invention provides a three-dimensional spatial reference reasoning method, device, electronic device and storage medium, which obtains RGB-D image data of the target scene and natural language instructions containing spatial constraints; wherein the RGB-D image data is: multimodal image data containing color visual information and depth information; the RGB-D image data and the natural language instructions are input into a pre-trained visual language model, and the output contains text containing an explicit reasoning process and target point coordinates that meet the spatial constraints; wherein the visual language model is: obtained by jointly training two-stage supervised learning fine-tuning enhanced by depth alignment and spatial understanding and reinforcement learning fine-tuning based on the display reasoning process; the visual language model includes an independent depth encoder, which is used to process the depth information. It can be seen that the present invention introduces an independent depth encoder in the single-step spatial understanding stage to enhance the three-dimensional perception accuracy, and on this basis adopts a reinforcement learning fine-tuning strategy based on the display reasoning process to achieve multi-step spatial reasoning, which greatly improves the overall performance in complex spatial reference tasks. As a result, it can significantly improve the model's position recognition and reasoning accuracy in open environments, effectively meeting the needs of various downstream robot applications such as navigation, grasping, and placement. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flowchart of the three-dimensional space reference reasoning method provided by the present invention.

[0020] Figure 2 This is a complete flow chart of the three-dimensional spatial reference reasoning method provided by the present invention.

[0021] Figure 3 It is a structural diagram of the three-dimensional space reference reasoning device provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] The following combination Figures 1-4 A three-dimensional space reference reasoning method, device, electronic device and storage medium of the present invention are described.

[0025] It should be noted that the existing technology still has deficiencies in the following aspects, which directly affect the effectiveness and scalability of spatial indication in practical applications: 1. Focus only on single-step spatial understanding and lack multi-step reasoning capabilities.

[0026] Currently, most research is limited to identifying single or simple spatial relationships within a scene, with less in-depth exploration of the multi-step reasoning involved in complex instructions. In other words, research often focuses on matching an object's spatial location information or basic orientation features with textual information, but often struggles to support multi-stage, combinatorial thinking. In open-world scenarios, spatial descriptions often include multiple constraints, such as "first locate the nearest dinner plate, then find the soy sauce dish next to it." Existing methods lack the ability to fully reason about these multi-level instructions, making it difficult for the system to accurately obtain the target coordinates.

[0027] 2. Insufficient or inappropriate use of depth information can easily lead to modal interference.

[0028] Spatial reference relies on accurate single-step spatial understanding. Although existing methods enhance this capability by fusing 3D modalities, some methods require expensive 3D reconstruction of multi-view images, resulting in high computational and data acquisition costs. Alternatively, some methods leverage RGB-D data, directly treating depth as an equivalent input to RGB and processing it using the same image encoder. However, this can cause mutual interference between the RGB and depth modalities, reducing the overall performance of the encoder. Furthermore, to mitigate this interference, additional training data is often required, which in practice further increases costs and limits the model's adaptability to complex scenes.

[0029] 3. The lack of an explicit reasoning process can easily lead to insufficient model generalization capabilities.

[0030] Most current large multimodal models rely on fully supervised fine-tuning (SFT) to learn implicit reasoning processes when performing spatial representation tasks. This can lead to models "memorizing" the answers to training examples during training, making it difficult to provide reliable predictions for unseen scenarios or more complex instructions. In broadcast or unseen scenarios, without explicit reasoning steps, model errors in reasoning are difficult to detect and correct, which in turn reduces their generalization performance and accuracy in real-world applications.

[0031] Therefore, existing technologies have significant shortcomings in single-step spatial perception, multi-step complex reasoning, and the fusion of depth and RGB modalities, making it difficult to achieve both high-precision recognition and reasoning of complex three-dimensional spatial relationships. Furthermore, the lack of support for explicit reasoning processes limits the model's adaptability in open-world environments. Therefore, the present invention provides a three-dimensional spatial reference reasoning method to address at least one of the aforementioned issues.

[0032] Figure 1 is a flow chart of the three-dimensional space reference reasoning method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: Acquire RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information.

[0033] It should be noted that the core concept of this embodiment is to address the difficulty of existing spatial reference reasoning in balancing accurate three-dimensional perception with complex multi-step reasoning. A visual language model (VLM) for three-dimensional scenes with an explicit reasoning mechanism, named RoboRefer, is proposed. By introducing an independent deep encoder in the single-step spatial understanding stage to enhance three-dimensional perception accuracy, and based on this, a reinforcement learning fine-tuning (RFT) strategy based on the explicit reasoning process is adopted to achieve multi-step spatial reasoning, the overall performance in complex spatial reference tasks is significantly improved. This can significantly improve the model's position recognition and reasoning accuracy in open environments, effectively meeting the needs of various downstream robotic applications such as navigation, grasping, and placement.

[0034] Step 200: Input the RGB-D image data and the natural language instruction into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0035] It should be noted that the large visual language model also includes: an image encoder, a multimodal projection layer and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer.

[0036] Step 200 specifically includes: Step 210: Extract image features of the RGB-D image data using the image encoder, and extract depth features of the RGB-D image data using the depth encoder.

[0037] Step 220: Map the image features and the depth features to a feature space that matches the large language model via the image projection layer and the depth projection layer, respectively.

[0038] Step 230: Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

[0039] In one embodiment, a visual language model (VLM) for 3D scenes with an explicit reasoning mechanism, named RoboRefer, is proposed. It can simultaneously achieve single-step accurate 3D spatial perception and multi-step complex reasoning. The model includes the following key modules: 1. Image encoder: used to extract features from color images (RGB) while retaining the representation capabilities of existing pre-trained image encoders.

[0040] 2. Depth encoder: Independent of the RGB encoder, it is used to process depth information and can explicitly capture three-dimensional information such as distance, orientation, and size changes in the scene, avoiding interference with RGB features.

[0041] 3. Multimodal projection layer (image projection layer, depth projection layer): Projects the features output by the image and depth encoder into a feature space that matches the large language model, allowing the large language model to achieve consistent semantic and visual alignment when receiving multimodal input.

[0042] 4. Large Language Model (LLM): As the core understanding and reasoning engine, it achieves a comprehensive understanding of scenes and instructions by interacting with multimodal features. Based on this language model, the model can output location predictions (coordinate points) or question-and-answer results, and can also generate explicit textual reasoning steps during the reasoning process.

[0043] After receiving a "command containing spatial constraints" and the scene's RGB-D input, the model first independently encodes the depth information before multimodally fusing it with text features. The subsequent output includes both an explicit step-by-step reasoning process and ultimately produces target point predictions that adhere to spatial relationship constraints. This satisfies the need for spatial representation in complex 3D scenes, enabling diverse downstream tasks such as navigation, grasping, and placement.

[0044] To achieve accurate 3D perception and multi-step reasoning, this embodiment employs a two-stage training strategy: fully supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RFT). These two stages work together to deeply integrate 3D information, explicit reasoning, and highly accurate prediction capabilities, while ensuring good generalization and robustness in open environments. The following details the training process of the large visual language model provided in this embodiment.

[0045] The two-stage supervised learning fine-tuning process includes: Step S210: Obtain a training dataset with RGB-D annotations, and update the depth projection layer parameters based on the training dataset to achieve depth alignment.

[0046] Step S220: Obtain a dataset containing three-dimensional annotations and explicit reasoning processes and a general instruction following dataset, and jointly update the image encoder parameters, the depth encoder parameters, the projection layer parameters, and the large language model parameters based on the dataset containing three-dimensional annotations and explicit reasoning processes and the general instruction following dataset to achieve enhanced spatial understanding.

[0047] Supervised fine-tuning (SFT) consists of the following two steps: 1. Depth alignment.

[0048] For the newly added deep encoder, it is necessary to first align the text space with the deep feature space. To this end, the present invention selects a training dataset with RGB-D annotations (such as RefSpatial) and projects the deep features into a semantic space that matches the language model by updating only the "depth projection layer".

[0049] During this process, the image encoder and language model parameters do not change, thus avoiding the interference of depth information on the existing pre-trained encoder.

[0050] 2. Enhanced spatial comprehension ability.

[0051] After completing the depth alignment, the model is further fine-tuned using the combined image and depth inputs, including: Leverage data with detailed 3D annotations and explicit multi-step reasoning processes (such as the multi-step spatial reference examples in RefSpatial) to enhance the model's understanding of spatial dimensionality.

[0052] At the same time, an additional instruction following dataset is introduced to enable the model to have more general instruction understanding capabilities.

[0053] Through these training steps, the model retains its original language understanding and visual question answering (VQA) capabilities while further acquiring accurate perception of three-dimensional attributes such as distance, direction, and position. It also has a preliminary implicit grasp of multi-step spatial reasoning (although there may still be issues with memory generalization). Therefore, this stage of SFT also lays the foundation for subsequent enhancements to explicit reasoning.

[0054] The reinforcement learning fine-tuning process includes: Step S230: Obtain historical RGB-D image data and historical natural language instructions, and input the historical RGB-D image data and the historical natural language instructions into the visual large language model fine-tuned by supervised learning to generate multiple candidate inference paths; wherein each candidate inference path includes: text containing an explicit inference process and target point coordinates that meet spatial constraints.

[0055] Step S240 : Evaluate each of the multiple candidate reasoning paths using a pre-designed reward function to obtain a reward value.

[0056] It should be noted that the reward function includes: result-based reward function and process-based reward function.

[0057] Step S240 specifically includes: Step S241: Based on each candidate reasoning path among the multiple candidate reasoning paths, a format reward and a coordinate prediction accuracy reward are obtained through the result-class reward function; wherein the format reward and the coordinate prediction accuracy reward are used to: evaluate the compliance of the output format and the proximity of the target point coordinates to the actual target point.

[0058] Step S242: Based on each candidate reasoning path in the multiple candidate reasoning paths, obtain the explicit step format and the key step accuracy through the process-class reward function; wherein the explicit step format and the key step accuracy are used to: evaluate the correctness of the explicit reasoning process.

[0059] Step S243: Obtain the reward value based on the format reward and the coordinate prediction accuracy reward, as well as the explicit step format and the key step accuracy.

[0060] Step S250: Based on the reward value of each candidate reasoning path, the visual language model after supervised learning fine-tuning is optimized by the group relative strategy optimization GRPO algorithm to obtain the trained visual language model.

[0061] Step S250 specifically includes: Step S251 : normalize the reward value of each candidate reasoning path to obtain the relative advantage value of each candidate reasoning path.

[0062] Step S252: performing strategy update based on the relative advantage value of each candidate reasoning path, and adding a KL divergence regularization term during the strategy update process to limit the deviation between the new strategy and the reference strategy, thereby obtaining the trained visual language large model.

[0063] After the SFT is completed, although the model has a certain multi-step reasoning ability, it is still prone to relying on the "memory" mode, making it difficult to accurately reason about novel or more complex spatial constraints. To this end, this application further designs reinforcement learning fine-tuning (RFT) based on reinforcement learning to optimize the explicit reasoning process. Its main process includes: 1. Multi-step output of the explicit reasoning process.

[0064] For complex spatial reference tasks, the complete reasoning process is divided into multiple key steps; each step can output structured text (such as "[perception type] [target object]: ..."), which contains judgment information about a specific target or location, such as Figure 1 's "thinking process".

[0065] This makes it easier to measure and reward "process-level correctness" during training, avoiding relying solely on the final prediction results.

[0066] 2. Design of reward function.

[0067] Outcome Reward: It includes the structured reasoning output format reward (ROF) and the coordinate prediction accuracy reward (RP), which are used to measure the format compliance of the final output and the closeness of the coordinates to the true target point.

[0068] Process Reward: It includes the Explicit Step Format (RPF) and the Accuracy of Key Steps (RAcc), which are used to measure the correctness of the reasoning process and achieve detailed guidance of the intermediate reasoning links.

[0069] By combining outcome rewards with process rewards, the model is encouraged to make accurate spatial judgments at each step and give the correct answer in the final prediction.

[0070] 3. Reinforcement learning update based on GRPO.

[0071] The model is optimized using Group Relative Policy Optimization (GRPO). Specifically, several inference processes and prediction results are sampled (i.e., multi-threaded exploration), the corresponding reward values ​​are calculated against the aforementioned reward function, and the relative advantages are used to update the policy.

[0072] At the same time, to ensure model stability and avoid deviating too far from the initial SFT strategy, a KL divergence regularization term is added to the loss function. This not only retains the good priors learned in the SFT stage, but also further enhances high-precision spatial reasoning through exploratory attempts.

[0073] Combining the above training steps, the resulting model can not only obtain accurate three-dimensional perception based on an independent depth encoder, but also use explicit process supervision and reinforcement learning to perform adaptive correction and iteration during the multi-step reasoning process, thereby having higher spatial reference accuracy and better generalization ability for complex scenes in open scenarios.

[0074] The above is a description of the steps of the three-dimensional spatial reference reasoning method provided by the present invention. From the description of the above steps, it can be seen that the three-dimensional spatial reference reasoning method provided by the present invention obtains RGB-D image data of the target scene and natural language instructions containing spatial constraints; wherein, the RGB-D image data is: multimodal image data containing color visual information and depth information; the RGB-D image data and the natural language instructions are input into a pre-trained visual language model, and the text containing the explicit reasoning process and the target point coordinates that meet the spatial constraints are output; wherein, the visual language model is: obtained by two-stage supervised learning fine-tuning enhanced by depth alignment and spatial understanding and reinforcement learning fine-tuning based on the display reasoning process; the visual language model includes an independent depth encoder, which is used to process the depth information. It can be seen that the present invention enhances the three-dimensional perception accuracy by introducing an independent depth encoder in the single-step spatial understanding stage, and on this basis adopts a reinforcement learning fine-tuning strategy based on the display reasoning process to achieve multi-step spatial reasoning, which greatly improves the overall performance in complex spatial reference tasks. As a result, the model's position recognition and reasoning accuracy in open environments can be significantly improved, effectively meeting the needs of various downstream robotics applications such as navigation, grasping, and placement.

[0075] The core technical solution of this invention mainly revolves around the fusion of vision and language in three-dimensional space. To address the problem that existing visual language models that only rely on two-dimensional image patterns have difficulty in obtaining accurate three-dimensional perception, it is proposed to add a dedicated deep encoder to the network structure and combine it with a targeted reinforcement fine-tuning (RFT) strategy to achieve a high-precision model for multi-step complex spatial reasoning. Figure 2 This is a complete flow chart of the three-dimensional space reference reasoning method provided by the present invention. Figure 2 , a complete description of the three-dimensional space reference reasoning method provided by the present invention is given.

[0076] VLM architecture: First, the present invention proposes a VLM architecture for three-dimensional scenes. Figure 2 , we can see that the model uses independent image encoders and depth encoders to extract image and depth features respectively in the input stage, and then connects with the large language model (LLM) through the projection layer (projectors) to achieve multiple functions such as visual question answering (VQA) or coordinate prediction.

[0077] The reason for setting up a separate depth encoder is that three-dimensional information is crucial for spatial understanding. If only pictures are used for VLM pre-training, there will be a lack of accurate capture of information such as depth, distance, or near-far relationships. Although some previous methods simply regard depth as a special image modality for joint encoding, this will lead to modal interference and require additional image data collaborative training to compensate for performance loss. In comparison, the independent depth encoder adopted by the present invention is as follows during training: when the joint fine-tuning of the picture and picture-depth begins, the image encoder will not be disturbed by the depth input, and the depth encoder can be updated independently, thereby maximizing the model's adaptability to general visual question answering (VQA) and fully enhancing the understanding of depth information.

[0078] SFT training: Based on this architecture, this paper further proposes a two-stage supervised fine-tuning (SFT) scheme to enhance the model's multi-step complex reasoning ability in three-dimensional space. Suppose the training dataset is ,in represents an RGB or RGB-D image observation, Representing natural language questions, Represents the corresponding answer. The model optimization objectives are as follows: in, represents the output distribution under the current model parameters, Represents the t-th output word.

[0079] The fully supervised fine-tuning SFT includes two steps: the depth alignment stage and the spatial understanding enhancement stage: Depth alignment stage: In this phase, only the mapping module of the deep features (i.e., the deep projector) is updated, with the goal of aligning the newly added deep modality with the language modality. During training, a spatial reference dataset with RGB-D annotations (such as RefSpatial) is used to achieve modality alignment by minimizing the difference between the predicted results and the true answers. In the specific implementation, the maximum learning rate is set to , the weight decay coefficient is 0, and the warm-up ratio is 0.03. The model with 2B parameters uses a batch size of 7 on each GPU, and the 8B model uses a batch size of 3 on each GPU for one training cycle.

[0080] Spatial understanding enhancement stage: In this stage, all model parameters are trained, including the RGB encoder, depth encoder, language model, and its projection module. The training data includes not only single-step reference tasks, but also multi-step spatial reasoning tasks with explicit reasoning processes, as well as multiple instruction following datasets (including LLaVA-1.5 dataset, LRV dataset, RefCOCO / + / g dataset). This stage aims to enhance the model's spatial understanding and explicit reasoning capabilities through composite supervision signals, providing a good initialization for the next step of reinforcement learning fine-tuning (RFT). The training parameters for this stage are: maximum learning rate The batch size per GPU for the 2B model is 6, and for the 8B model it is 2. Other hyperparameters remain the same as in the first stage.

[0081] RFT training: Although SFT incorporates a large amount of data, even including reasoning data from multi-step thought processes, it tends to memorize answers and is unable to cope with novel spatial constraint combinations. Therefore, this paper provides a spatial reference reasoning method based on reinforcement learning fine-tuning (RFT), aiming to improve the model's generalization ability and reasoning accuracy in understanding complex spatial relationships and accurately predicting points. This method, based on fully supervised fine-tuning (SFT), introduces a Group Relative Policy Optimization (GRPO) mechanism and combines it with multiple reward function designs to more effectively guide the model to generate structured, verifiable reasoning processes and high-precision prediction results.

[0082] Specifically, it includes the following steps: 1. Reinforcement learning strategy optimization framework.

[0083] First, use the initial strategy model trained with SFT As an initial strategy in the RFT phase In each training iteration, the system is trained based on the input state ,in represents the visual encoding of the input image (RGB or RGB-D), The encoding representing the text question, through the current strategy Sample N candidate actions (answers): This approach ensures diversity in the sampled results to encourage policy exploration and avoid early convergence.

[0084] 2. Reward function design.

[0085] To achieve a comprehensive evaluation of the reasoning results and process, this embodiment designs a multiple reward mechanism including result-based reward functions and process-based reward functions.

[0086] (1) Result-based reward function.

[0087] Result format bonus : If the model output meets the following format specifications, the reward is 1, otherwise it is 0: <think> ...reasoning process...< / think> <answer> ...Final Answer...< / answer> Point accuracy reward : If the L1 distance between the point prediction result output by the model and the marked point is less than 50 pixels, it is considered accurate and the reward is 1, otherwise it is 0: (2) Process-based reward function.

[0088] To overcome the drawbacks of traditional process rewards that rely on large language or multimodal models (such as LLM / VLM), this embodiment innovatively designs a rule-based process reward mechanism that directly utilizes the key step annotation information provided in the dataset, avoiding the uncertainty of external evaluators.

[0089] Process Format Rewards : If the intermediate reasoning steps output by the model conform to the following structured format, the reward is 1: [Type] [Target Object]: [Value] The "Type" is limited to "Position", "Orientation", or "Size"; the "Target Object" is a uniquely identifiable entity; and the form of the "Value" depends on the type: If Position: the format is [(x, y)], where x,y∈[0,1] If Orientation: The format is a three-dimensional unit vector (x, y, z) If Size: the value is a scalar in meters Examples include: [Position] [Second largest cup]: [(0.245, 0.147)] [Towards] [Cup Handle]: (1.000, 0.000, 0.000) Accuracy Bonus : Only the perception output in the key step is evaluated. If the predicted target matches the label and the error between the predicted value and the true value is within the allowable range, a reward is given. For example: For the “position” perception, if the L1 distance is less than 50 pixels, the reward is 1; For the "direction" class, if the cosine similarity is > 0.8, the reward is 1; For the "size" class, if the predicted value is within ±15% of the true value, the reward is 1.

[0090] The definition is as follows: 3. Reward normalization and strategy update.

[0091] The total reward for each set of sampled actions consists of four parts: The process class reward is multiplied by a coefficient of 0.25 to prevent it from accumulating too high in multi-step reasoning.

[0092] The rewards within each group are then normalized to calculate the relative advantage value: Finally, this relative advantage value is used to update the policy, increasing the probability of high-quality actions and suppressing the generation of low-quality actions. At the same time, to ensure the stability of the training process, a KL divergence regularization term is added to the policy update to limit the deviation between the new policy and the reference policy.

[0093] Notably, the SFT initialization phase provides the model with strong prior knowledge, enabling it to quickly adapt to the output format and effectively supporting accurate step-by-step spatial reasoning by leveraging the spatial understanding learned during the SFT phase. As shown in Figure 2, the RFT-trained model demonstrates good generalization in tasks such as four-step spatial representation, capable of gradually processing complex spatial relationships and generating accurate point predictions.

[0094] The three-dimensional spatial reference reasoning method provided by the present invention has the following beneficial effects: 1. Significantly improved three-dimensional spatial perception accuracy.

[0095] Advantages: This invention addresses the three-dimensional spatial perception needs of robotic interactions by employing an independent depth encoder, effectively improving the accuracy of identifying spatial attributes such as target position, orientation, and depth. Compared to existing approaches that simply treat depth data as RGB input or require costly multi-view 3D reconstruction, this invention significantly reduces intermodal interference and learning difficulty through a dedicated architecture where depth and image branches interact independently.

[0096] reason: 1) Dedicated depth encoder: avoids modality conflicts caused by mixing depth and RGB features, and enhances the accurate capture of spatial information; 2) Reduce tedious 3D reconstruction: Eliminate the need for large-scale acquisition and processing of multi-view data, thus avoiding the resulting high computational costs and error accumulation.

[0097] 2. Comprehensive ability to support multi-step spatial reasoning.

[0098] Advantages: This paper explicitly decomposes and performs multi-step reasoning by introducing a reinforcement fine-tuning (RFT) phase after supervised fine-tuning (SFT). This strategy breaks through the limitations of previous approaches that primarily focus on single-step spatial understanding and can better handle complex tasks involving multiple spatial constraints or sequential logic.

[0099] reason: 1) Explicit reasoning process annotation: The reasoning process is annotated and measured during the RFT phase, enabling the model to decipher complex "spatial reference" chains step by step; 2) Metric-sensitive reward function: This paper designs a metric-sensitive reward function for inference accuracy, guiding the model to continuously correct the predicted point position in multi-step inference, reducing error accumulation and improving overall accuracy.

[0100] 2. Stronger open world generalization and accuracy.

[0101] Advantages: Leveraging these improved 3D perception and multi-step reasoning capabilities, the proposed method achieves leading performance on both existing single-step spatial understanding test sets (such as CV-Bench and BLINK) and a purpose-built multi-step spatial task test set (RefSpatial-Bench). These results demonstrate that the proposed method can maintain high levels of prediction accuracy and generalization performance across multiple scenarios and tasks.

[0102] reason: 1) Combining the advantages of single-step and multi-step reasoning: It not only performs well in basic spatial object recognition tasks, but can also handle complex scenarios with multiple intersecting conditions and requiring reasoning and decomposition; 2) Multi-type data and real-world scenario evaluation: RefSpatial-Bench covers a variety of real-world images and requires up to five steps of inference depth, which can fully test the robustness of the model in long-term, multi-constrained scenarios.

[0103] 4. Adaptability and scalability to various downstream tasks.

[0104] Advantages: This invention not only serves as a general visual assistant but also provides high-level spatial planning instructions for specific robotic platforms (such as robotic arms and humanoid robots), including operations such as grasping, placing, and navigation path planning. The multi-module collaboration allows for flexible adaptation to diverse task requirements, offering high portability and expansion potential.

[0105] reason: 1) Universal spatial reference representation: Through clear spatial constraint semantics and a multi-step reasoning process, this application has the potential to be easily scalable to a variety of hardware systems; 2) Multi-strategy control interface: Seamlessly connect different robot control strategies (such as manipulator path planning and mobile platform navigation) to further improve the overall efficiency of the system.

[0106] 5. Balancing high performance and security Advantages: While significantly enhancing spatial perception, reasoning, and generalization capabilities, this invention also addresses safety considerations in complex operational scenarios. By explicitly tracking and evaluating the reasoning process, decisions can be fine-tuned or safety thresholds can be set when necessary, mitigating potential risks in real-world applications.

[0107] reason: 1) Reasoning explainability: Multi-step reasoning makes the decision-making process transparent, facilitating auditing or increasing security redundancy in security-related applications.

[0108] 2) Open development and security assessment: After the model is opened to relevant research groups, more usage feedback and security testing can be continuously obtained, improving the overall reliability of the system.

[0109] The three-dimensional space reference reasoning device provided by the present invention is described below. The three-dimensional space reference reasoning device described below and the three-dimensional space reference reasoning method described above can be referenced to each other.

[0110] Figure 3 is a schematic diagram of the structure of the three-dimensional space reference reasoning device provided by the present invention, such as Figure 3 As shown, the three-dimensional space reference reasoning device provided by the present invention includes: An acquisition module 301 is configured to acquire RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; The reasoning module 302 is configured to input the RGB-D image data and the natural language instruction into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0111] The three-dimensional spatial reference reasoning device provided by the present invention obtains RGB-D image data of the target scene and natural language instructions containing spatial constraints; wherein the RGB-D image data is: multimodal image data containing color visual information and depth information; the RGB-D image data and the natural language instructions are input into a pre-trained visual language model, and the text containing the explicit reasoning process and the target point coordinates that meet the spatial constraints are output; wherein the visual language model is: obtained by jointly training two-stage supervised learning fine-tuning enhanced by depth alignment and spatial understanding and reinforcement learning fine-tuning based on the display reasoning process; the visual language model includes an independent depth encoder, which is used to process the depth information. It can be seen that the present invention introduces an independent depth encoder in the single-step spatial understanding stage to enhance the three-dimensional perception accuracy, and on this basis adopts a reinforcement learning fine-tuning strategy based on the display reasoning process to achieve multi-step spatial reasoning, which greatly improves the comprehensive performance in complex spatial reference tasks. As a result, the model's position recognition and reasoning accuracy in open environments can be significantly improved, effectively meeting the needs of various downstream robot applications such as navigation, grasping, and placement.

[0112] Based on the above embodiment, in this embodiment, the visual language large model further includes: an image encoder, a multimodal projection layer and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer; The reasoning module 302 is specifically configured to: Extracting image features of the RGB-D image data by the image encoder, and extracting depth features of the RGB-D image data by the depth encoder; Mapping the image features and the depth features to a feature space matching the large language model via the image projection layer and the depth projection layer respectively; Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

[0113] Based on the above embodiment, in this embodiment, the two-stage supervised learning fine-tuning process includes: Obtaining a training dataset with RGB-D annotations, and updating the depth projection layer parameters based on the training dataset to achieve depth alignment; Obtain a dataset comprising three-dimensional annotations and an explicit reasoning process and a general instruction following dataset, and jointly update the image encoder parameters, the depth encoder parameters, the projection layer parameters, and the large language model parameters based on the dataset comprising three-dimensional annotations and an explicit reasoning process and the general instruction following dataset to achieve enhanced spatial understanding.

[0114] Based on the above embodiment, in this embodiment, the reinforcement learning fine-tuning process includes: Obtaining historical RGB-D image data and historical natural language instructions, and inputting the historical RGB-D image data and the historical natural language instructions into a visual language model fine-tuned by supervised learning to generate multiple candidate inference paths; wherein each of the multiple candidate inference paths includes: text containing an explicit inference process and target point coordinates that meet spatial constraints; For each candidate reasoning path among the multiple candidate reasoning paths, evaluating it using a pre-designed reward function to obtain a reward value; Based on the reward value of each candidate reasoning path, the visual language model after supervised learning fine-tuning is optimized through the group relative strategy optimization GRPO algorithm to obtain the trained visual language model.

[0115] Based on the above embodiment, in this embodiment, the reward function includes: a result-based reward function and a process-based reward function; The device further comprises an evaluation module, specifically configured to: Based on each candidate reasoning path among the multiple candidate reasoning paths, a format reward and a coordinate prediction accuracy reward are obtained through the result-like reward function; wherein the format reward and the coordinate prediction accuracy reward are used to: evaluate the compliance of the output format and the proximity of the target point coordinates to the true target point; Based on each candidate reasoning path among the plurality of candidate reasoning paths, obtaining an explicit step format and a key step accuracy through the process-like reward function; wherein the explicit step format and the key step accuracy are used to: evaluate the correctness of the explicit reasoning process; The reward value is obtained based on the format reward and the coordinate prediction accuracy reward, as well as the explicit step format and the key step accuracy.

[0116] Based on the above embodiment, in this embodiment, the device further includes an evaluation module, which is specifically configured to: Normalizing the reward value of each candidate reasoning path to obtain a relative advantage value of each candidate reasoning path; The strategy is updated based on the relative advantage value of each candidate reasoning path, and a KL divergence regularization term is added during the strategy update process to limit the deviation between the new strategy and the reference strategy, thereby obtaining the trained visual language model.

[0117] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may be a robot or other electronic device, and may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the three-dimensional spatial reference reasoning method, including: Obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; Input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0118] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] In another aspect, the present invention further provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the three-dimensional spatial reference reasoning method provided by the above methods, including: Obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; Input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0120] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional spatial reference reasoning method provided by the above methods, including: Obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; Input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0122] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A three-dimensional spatial reference reasoning method, characterized in that: include: Obtaining RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; Input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit reasoning process and target point coordinates that meet spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

2. The three-dimensional space reference reasoning method according to claim 1, characterized in that: The large visual language model further includes: an image encoder, a multimodal projection layer, and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer; The step of inputting the RGB-D image data and the natural language instructions into a pre-trained visual language model and outputting text containing an explicit reasoning process and target point coordinates that conform to spatial constraints includes: Extracting image features of the RGB-D image data by the image encoder, and extracting depth features of the RGB-D image data by the depth encoder; Mapping the image features and the depth features to a feature space matching the large language model via the image projection layer and the depth projection layer respectively; Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

3. The three-dimensional space reference reasoning method according to claim 2, characterized in that: The two-stage supervised learning fine-tuning process includes: Obtaining a training dataset with RGB-D annotations, and updating the depth projection layer parameters based on the training dataset to achieve depth alignment; Obtain a dataset comprising three-dimensional annotations and an explicit reasoning process and a general instruction following dataset, and jointly update the image encoder parameters, the depth encoder parameters, the projection layer parameters, and the large language model parameters based on the dataset comprising three-dimensional annotations and an explicit reasoning process and the general instruction following dataset to achieve enhanced spatial understanding.

4. The three-dimensional space reference reasoning method according to claim 3, characterized in that: The reinforcement learning fine-tuning process includes: Obtaining historical RGB-D image data and historical natural language instructions, and inputting the historical RGB-D image data and the historical natural language instructions into a visual language model fine-tuned by supervised learning to generate multiple candidate inference paths; wherein each of the multiple candidate inference paths includes: text containing an explicit inference process and target point coordinates that meet spatial constraints; For each candidate reasoning path among the multiple candidate reasoning paths, evaluating it using a pre-designed reward function to obtain a reward value; Based on the reward value of each candidate reasoning path, the visual language model after supervised learning fine-tuning is optimized through the group relative strategy optimization GRPO algorithm to obtain the trained visual language model.

5. The three-dimensional space reference reasoning method according to claim 4, characterized in that: The reward function includes: a result-based reward function and a process-based reward function; The step of evaluating each of the plurality of candidate reasoning paths using a pre-designed reward function to obtain a reward value includes: Based on each candidate reasoning path among the multiple candidate reasoning paths, a format reward and a coordinate prediction accuracy reward are obtained through the result-like reward function; wherein the format reward and the coordinate prediction accuracy reward are used to: evaluate the compliance of the output format and the closeness of the target point coordinates to the true target point; Based on each candidate reasoning path among the plurality of candidate reasoning paths, obtaining an explicit step format and a key step accuracy through the process-like reward function; wherein the explicit step format and the key step accuracy are used to: evaluate the correctness of the explicit reasoning process; The reward value is obtained based on the format reward and the coordinate prediction accuracy reward, as well as the explicit step format and the key step accuracy.

6. The three-dimensional space reference reasoning method according to claim 4, characterized in that: The reward value of each candidate reasoning path is used to optimize the visual language model after supervised learning fine-tuning by using the group relative strategy optimization GRPO algorithm to obtain the trained visual language model, including: Normalizing the reward value of each candidate reasoning path to obtain a relative advantage value of each candidate reasoning path; The strategy is updated based on the relative advantage value of each candidate reasoning path, and a KL divergence regularization term is added during the strategy update process to limit the deviation between the new strategy and the reference strategy, thereby obtaining the trained visual language model.

7. A three-dimensional spatial reference reasoning device, characterized in that: include: An acquisition module, configured to acquire RGB-D image data of a target scene and a natural language instruction containing spatial constraints; wherein the RGB-D image data is multimodal image data containing color visual information and depth information; An inference module, configured to input the RGB-D image data and the natural language instructions into a pre-trained visual language model, and output text containing an explicit inference process and target point coordinates that conform to spatial constraints; Among them, the visual language large model is obtained by jointly training two-stage supervised learning fine-tuning with depth alignment and spatial understanding enhancement and reinforcement learning fine-tuning based on the display reasoning process; the visual language large model includes an independent depth encoder, which is used to process the depth information.

8. The three-dimensional space reference reasoning device according to claim 7, characterized in that: The large visual language model further includes: an image encoder, a multimodal projection layer, and a large language model; wherein the multimodal projection layer includes: an image projection layer and a depth projection layer; The reasoning module is specifically used to: Extracting image features of the RGB-D image data by the image encoder, and extracting depth features of the RGB-D image data by the depth encoder; Mapping the image features and the depth features to a feature space matching the large language model via the image projection layer and the depth projection layer respectively; Based on the projected image features, the projected depth features, and the text features extracted according to the natural language instructions, the large language model outputs text containing an explicit reasoning process and target point coordinates that meet spatial constraints.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the three-dimensional space reference reasoning method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional space reference reasoning method according to any one of claims 1 to 6 is implemented.

Images