A robot operation decision-making method and system based on three-dimensional Gaussian state and semantic correction

Abstract

The application discloses a robot operation decision-making method and system based on three-dimensional Gaussian state and semantic rectification, first acquires multi-view observation data, robot state information and language task conditions, and constructs dynamic three-dimensional Gaussian state representation; voxelizes and aggregates the dynamic three-dimensional Gaussian state representation into voxel tokens, obtains a structured state representation through semantic guidance compression; then generates a basic action distribution or a candidate action based on the structured state representation, and predicts a short-time future dynamic three-dimensional Gaussian state to update the action decision state; finally, judges whether semantic rectification is needed, generates target semantic constraints when needed, and maps the target semantic constraints into three-dimensional semantic index constraints to rectify the candidate action or the action distribution, and finally generates a robot control instruction. The application can reduce the semantic confusion risk in a multi-target similar scene, and improve the stability and success rate of robot grasping and multi-step operation in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of robot embodied intelligence technology, specifically relating to a robot operation decision-making method and system based on three-dimensional Gaussian state and semantic correction. Background Technology

[0002] With the development of service robots, industrial robots, and embodied intelligence technologies, the demand for robots to perform tasks such as grasping, placing, opening and closing drawers, and stacking objects in unstructured environments is constantly increasing. Compared to the single grasping task in traditional fixed workstations, robot operations in real-world scenarios typically face challenges such as multiple targets, multi-view occlusion, continuous changes in object states, and constraints from natural language tasks. Therefore, robots not only need to identify target objects but also need to understand the target's position, orientation, local structure in three-dimensional space, and its relationship with surrounding objects, and generate stable and reliable grasping or manipulating actions accordingly.

[0003] Existing robot grasping algorithms typically rely on 2D images, depth maps, point clouds, or voxel features for perception and decision-making. While 2D vision methods offer high computational efficiency, they struggle to fully represent the 3D geometry and occlusion relationships of objects. Point cloud and voxel methods can provide some spatial information, but in scenarios involving sparse observation, complex textures, fine-grained part localization, and multi-view fusion, they still suffer from discontinuous feature representation, insufficient semantic information, or high computational overhead. In recent years, visual-language action models, linguistic conditional behavior cloning models, and neural field-based robot manipulation methods have been used to improve multi-task generalization capabilities. These methods enhance the robot's understanding of scene structure and task semantics through 3D features or neural field representations; however, they still primarily rely on current observations or static scene representations, making it difficult to fully characterize the short-term state evolution caused by robot actions.

[0004] 3D Gaussian splatting, due to its explicit 3D primitive representation and high rendering efficiency, is increasingly being introduced into robot scene representation and operation decision-making. Existing research on robot operation based on 3D Gaussian splatting utilizes dynamic 3D Gaussian representation for multi-task robot operations, mining the spatiotemporal dynamics of the scene through future scene reconstruction, providing richer 3D state information for action prediction. Compared to traditional point clouds or implicit neural radiation fields, 3D Gaussian representation can describe the scene with multiple Gaussian primitives possessing position, scale, rotation, opacity, and appearance features. It has advantages in multi-view observation fusion, local structure preservation, and real-time rendering, making it suitable for spatial state modeling before robot grasping.

[0005] However, existing robot grasping or manipulation methods based on 3D Gaussian representation still have several shortcomings. First, the number of 3D Gaussian sigma elements is usually large, and directly inputting all Gaussian features into the action decision model will result in a high computational burden and may introduce a large amount of background information irrelevant to the current task. Second, although some methods can express scene geometry and short-term dynamics, they still do not fully utilize language task conditions, target categories, local parts, and attribute modifiers, making it difficult to accurately locate the target object under open-vocabulary semantic constraints such as "red square," "left drawer handle," and "target in green plate." Third, when there are multiple similar targets, partial occlusion of objects, or domain deviation between the target name and visual appearance, the basic action model is prone to semantic confusion, leading to grasping position deviation, selection of incorrect targets, or execution of actions on non-target areas.

[0006] On the other hand, open-vocabulary detection, open-vocabulary segmentation, and visual-language localization models can output candidate bounding boxes or semantic masks based on natural language phrases, providing a new technical path for robots to understand open-category targets. Existing research on language Gaussian fields has shown that embedding language features into a 3D Gaussian representation can support open-vocabulary queries in 3D space. However, existing open-vocabulary visual models mostly operate in the 2D image domain. If their output cannot be reliably mapped to the robot's executable 3D state space, it is difficult to directly use them for grasping action correction. At the same time, open-vocabulary models have high computational costs, and if they are used indiscriminately in every frame or every action decision, it will affect the real-time performance and stability of online control.

[0007] Therefore, this invention proposes a Gaussian grasping method that simultaneously considers 3D geometric representation, dynamic state prediction, language task understanding, and open-vocabulary semantic correction. Especially in scenarios with multiple targets, multi-step operations, and complex semantic constraints, this method can construct a dynamic 3D Gaussian state representation based on multi-view observations and retain task-related 3D state information through voxel tokenization and semantically guided compression. It can also improve the stability of action selection by combining short-term future state prediction of action conditions. Furthermore, when action uncertainty is high or there is a risk of semantic confusion, it can selectively invoke the open-vocabulary semantic enhancement module to generate target semantic constraints, and map these target semantic constraints to 3D semantic index constraints through a 3D semantic index mapping module. Then, the action correction and backtracking module corrects the candidate grasping actions or action distribution, reducing the risk of semantic confusion in multi-target similar scenarios and improving the robot's grasping success rate and multi-step operation stability in complex real-world scenarios. Summary of the Invention

[0008] To address the challenge of existing robot grasping or manipulation methods simultaneously considering 3D geometric representation, dynamic state prediction, language task understanding, and open-vocabulary semantic correction, this invention provides a robot operation decision-making method and system based on 3D Gaussian state and semantic correction. This invention uses dynamic 3D Gaussian state representation to structure multi-view observations, reduces the size of state input through voxel tokenization and semantically guided compression, and generates target semantic constraints when action decisions are uncertain or pose a risk of semantic confusion by invoking an open-vocabulary semantic enhancement module. Then, a 3D semantic index mapping module maps these target semantic constraints to 3D semantic index constraints, and an action correction and backtracking module corrects candidate actions or action distributions based on these 3D semantic index constraints.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a robot operation decision-making method based on three-dimensional Gaussian state and semantic correction, the method comprising: (1) Acquire robot operation task data containing multi-view observation data, robot state information and language task conditions and perform preprocessing; (2) A dynamic three-dimensional Gaussian state representation is constructed based on multi-view observation data, and the dynamic three-dimensional Gaussian state representation is organized into a structured state representation by combining language task conditions through voxel tokenization and semantically guided compression; (3) Generate basic action distribution or candidate action set based on structured state representation, predict future dynamic three-dimensional Gaussian state or Gaussian state increment based on current dynamic three-dimensional Gaussian state representation, candidate actions and language task conditions, and evaluate candidate actions based on prediction results; (4) Obtain the preliminary action decision state based on the evaluation results, calculate the action uncertainty, and determine whether semantic correction is required. If so, generate target semantic constraints based on language task conditions and map them to three-dimensional semantic index constraints. When the three-dimensional semantic index constraints are valid, correct the basic action distribution or candidate actions based on the three-dimensional semantic index constraints to obtain the final action to be executed. When semantic correction is not required or the three-dimensional semantic index constraints are invalid, obtain the final action to be executed based on the preliminary action decision state.

[0010] (5) Generate robot control instructions based on the final execution action to drive the robot to perform the corresponding physical operation.

[0011] Furthermore, the preprocessing includes at least one of time synchronization, hand-eye calibration, coordinate transformation, and data format conversion, so that the data from different cameras, the robot body, and the task instructions are under a unified time reference and spatial coordinate system.

[0012] Furthermore, the robot operation decision-making method includes a training phase, a model deployment phase, and an online decision-making phase; The training phase is used to train the adaptation parameters of the action decision model, the action conditional dynamic Gaussian propagation model, and the open vocabulary semantic enhancement module. The action decision model is used to generate a basic action distribution or a set of candidate actions based on the current structured state representation. The action conditional dynamic Gaussian propagation model is used to predict the short-term future dynamic three-dimensional Gaussian state or Gaussian state increment based on the current dynamic three-dimensional Gaussian state representation, candidate actions, and language task conditions. The open vocabulary semantic enhancement module is used to generate target semantic constraints using a set of query phrases constructed based on language task conditions and the current two-dimensional image as input. The three-dimensional semantic index mapping module is used to map the target semantic constraints to three-dimensional semantic index constraints. The action correction and rollback module is used to determine the validity of the three-dimensional semantic index constraints, and to perform action correction when the three-dimensional semantic index constraints are valid, and to roll back to the initial action decision state when they are invalid. The model deployment phase is used to deploy the trained or adapted action decision model, action conditional dynamic Gaussian propagation model, and open vocabulary semantic enhancement module to the robot control end, edge computing end, or server end. Depending on the actual computing power, the action decision model and action conditional dynamic Gaussian propagation model can be deployed on the robot control end or edge computing end, and the open vocabulary semantic enhancement module can be deployed on the edge computing end or server end, and return semantic detection, semantic segmentation, or visual language localization results through a preset interface.

[0013] The online decision-making stage is used to obtain multi-view observations, robot state information and language task conditions at the current moment, and to construct the current dynamic three-dimensional Gaussian state representation. It then calls the deployed action decision model, action condition dynamic Gaussian propagation model and open vocabulary semantic enhancement module to execute steps (1) to (4).

[0014] Furthermore, during the training of the action-conditional dynamic Gaussian propagation model, the next-moment prediction observation is rendered or reconstructed based on the short-term future dynamic three-dimensional Gaussian state, and is supervised according to the image reconstruction error between the next-moment prediction observation and the next-moment actual observation, and / or the state propagation consistency error between the predicted Gaussian state and the actual or estimated Gaussian state.

[0015] Furthermore, the open vocabulary semantic enhancement module needs to undergo domain adaptation training. This domain adaptation training includes performing domain adaptation on the GroundingDINO text encoder in the model of the open vocabulary semantic enhancement module using the LoRA method. The model includes one or a combination of an open vocabulary two-dimensional detection model, an open vocabulary two-dimensional segmentation model, and a visual language localization model.

[0016] Furthermore, the dynamic 3D Gaussian state representation includes multiple explicit 3D Gaussian primitives, each of which includes at least one of the following: center position, scale, rotation, opacity, appearance features, semantic features, or task-related features. The voxel tokenization is obtained by mapping 3D Gaussian primitives to a regular voxel space, aggregating primitive features in the same voxel unit or local voxel block, and forming voxel tokens. The semantically guided compression prioritizes retaining task-related voxel tokens based on language task conditions or semantic category mapping results, and retains pruning indexes for mapping back to the original 3D Gaussian state space.

[0017] Furthermore, a basic action distribution or candidate actions are generated based on the current structured state representation. When short-term future state evaluation is required, a trained action-conditional dynamic Gaussian propagation model is used to predict the short-term future dynamic three-dimensional Gaussian state or Gaussian state increment based on the current dynamic three-dimensional Gaussian state representation, candidate actions, and language task conditions. The basic action distribution or candidate actions are then updated based on at least one of the following: state consistency, task-related region changes, prediction uncertainty, or preset evaluation indicators, to obtain a preliminary action decision state.

[0018] Further, the specific process for determining whether semantic correction is needed is as follows: based on the action uncertainty of the preliminary action decision state, the invocation budget of open vocabulary semantic enhancement, and the cooling condition, it is determined whether semantic correction should be triggered; the action uncertainty is determined based on at least one of the following: the classification interval between the highest probability action and the second highest probability action in the basic action distribution, the entropy of the basic action distribution, the difference in candidate action scores, the variance of the continuous action distribution, and the uncertainty of short-term future state prediction; the invocation budget is used to limit the number of times semantic correction is triggered per unit time, in a single task, or in a single operation sequence; the cooling condition is used to limit the minimum time interval or the minimum number of decision steps between two adjacent semantic corrections.

[0019] Furthermore, when semantic correction is triggered, the open vocabulary semantic enhancement module is invoked to generate target semantic constraints, which are then mapped to three-dimensional semantic index constraints via a three-dimensional semantic index mapping module. The validity of the three-dimensional semantic index constraints is then determined by an action correction and backtracking module. When the three-dimensional semantic index constraints are valid, the basic action distribution or candidate actions are corrected based on these constraints to obtain the final execution action. When semantic correction is not triggered or the three-dimensional semantic index constraints are invalid, the final execution action is obtained based on the preliminary action decision state. Robot control commands are generated based on the final execution action to drive the robot to perform the corresponding physical operations.

[0020] Furthermore, the generation of target semantic constraints based on language task conditions includes: Extract target object words, local component words, attribute words, or spatial modifier words from language task conditions to construct a query phrase set; Using an open vocabulary 2D detection model, an open vocabulary 2D segmentation model, or a visual language localization model, at least one of candidate boxes, candidate categories, detection confidence, and semantic masks is output based on the query phrase set. The candidate semantic results are filtered or sorted based on at least one of the following: semantic mask area, detection confidence, query phrase specificity, and area penalty term, to obtain the target semantic constraints.

[0021] Furthermore, mapping the target semantic constraints to three-dimensional semantic index constraints includes: For pixels in the semantic mask corresponding to the target semantic constraints, back-project them into three-dimensional points in the world coordinate system using the corresponding depth values, camera intrinsic parameters and camera extrinsic parameters. Based on the voxel resolution, the three-dimensional spatial boundary, and the spatial position of the three-dimensional Gaussian unit, the three-dimensional points are discretized into voxel indices, three-dimensional Gaussian unit indices, or three-dimensional state indices. The indexes are merged and deduplicated to obtain three-dimensional semantic index constraints.

[0022] Furthermore, correcting the distribution of basic actions or candidate actions based on the three-dimensional semantic index constraints includes: Based on the three-dimensional semantic index constraints, determine the voxel token, voxel unit, or three-dimensional Gaussian unit corresponding to the target region; improve the candidate action score acting on the target region, reduce the candidate action score acting on non-target regions or confidence semantic regions below the threshold, or correct the basic action distribution or robot end-effector target pose based on the target region; When the three-dimensional semantic index constraint is empty or the corresponding two-dimensional semantic result does not meet the area threshold or confidence threshold, the three-dimensional semantic index constraint is determined to be invalid.

[0023] On the other hand, the present invention also provides a robot operation decision-making system based on three-dimensional Gaussian state and semantic correction, the system comprising: The data acquisition and preprocessing module is used to acquire multi-view observation data, robot state information, and language task conditions, and to perform preprocessing. The dynamic 3D Gaussian state representation module is used to construct a dynamic 3D Gaussian state representation based on multi-view observation data, generate voxel tokens and perform semantically guided compression, and organize the dynamic 3D Gaussian state representation into a structured state representation in combination with language task conditions. The action decision model training module is used to train the action decision model and the action conditional dynamic Gaussian propagation model. An open-vocabulary semantic enhancement adaptation module is used for domain adaptation training of the open-vocabulary semantic enhancement module. The model deployment module is used to deploy trained or adapted models to the robot control end, edge computing end, or server end. The basic action decision module is used to generate basic action distributions or candidate actions based on the current structured state representation input to the action decision model, and to obtain short-term future state prediction results based on the action condition dynamic Gaussian propagation model, and to evaluate the candidate actions. The semantic risk gating module is used to determine whether to call the open vocabulary semantic enhancement module to trigger semantic correction; The three-dimensional semantic index mapping module is used to map the target semantic constraints output by the open vocabulary semantic enhancement module to three-dimensional semantic index constraints; The action correction and rollback module is used to correct actions when the three-dimensional semantic index constraints are valid, or to roll back to the initial action decision state when the three-dimensional semantic index constraints are invalid, so as to obtain the final action to be executed. The robot control interface module is used to convert the final execution action into robot control instructions.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects: First, the present invention uses dynamic three-dimensional Gaussian state representation to organize multi-view observation data, which can express the geometric, appearance and task-related features of target objects and operation scenes with explicit three-dimensional primitives. Compared with methods based solely on two-dimensional images, point clouds or static scene representations, this is beneficial to enhancing the robot's ability to understand occlusion relationships, local structures and short-term state changes.

[0025] Second, this invention organizes dynamic three-dimensional Gaussian states into structured state representations through voxel tokenization and semantically guided compression. This reduces background information and redundant Gaussian meta-inputs that are irrelevant to the current task, lowers the computational burden of the action decision model, and makes the model focus more on the target region pointed to by the language task conditions.

[0026] Third, this invention introduces a dynamic Gaussian propagation model for action conditions, which predicts the short-term future dynamic three-dimensional Gaussian state or state increment before the candidate action is executed. This makes the action selection not only dependent on the current observation, but also able to refer to the short-term state evolution that the action may cause, thereby improving the decision stability in multi-step operations and complex grasping tasks.

[0027] Fourth, this invention constructs a semantic risk gating mechanism through action uncertainty, call budget, and cooling conditions, which triggers the open vocabulary semantic enhancement module only when needed. This can reduce the risk of target confusion in scenarios with multiple similar targets or complex semantic constraints, and avoid online control delays caused by frequent calls to large-scale open vocabulary models.

[0028] Fifth, this invention generates two-dimensional target semantic constraints through an open vocabulary semantic enhancement module, maps them to three-dimensional semantic index constraints through a three-dimensional semantic index mapping module, and corrects candidate actions or action distributions based on the three-dimensional semantic index constraints through an action correction and backtracking module. This enables the open vocabulary detection or segmentation results to be applied to the robot's executable three-dimensional state space, which helps improve the robot's grasping success rate and multi-step operation stability in real-world scenarios. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the core process of the robot operation decision-making method provided in the embodiments of the present invention; Figure 2 This is an overall flowchart of the robot operation decision-making method provided in an embodiment of the present invention; Figure 3 A general framework diagram of the dynamic three-dimensional Gaussian state representation method provided in the embodiments of the present invention; Figure 4 A flowchart illustrating the construction process from three-dimensional Gaussian primitives to voxel tokens, provided for embodiments of the present invention; Figure 5 The overall flowchart of open vocabulary semantic enhancement and gating correction provided for embodiments of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be further described below with reference to embodiments. It should be understood that the following embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, equivalent substitutions or modifications made to sensor types, robot platforms, model structures, threshold settings, deployment locations, or task types without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0032] This embodiment provides a robot operation decision-making method based on three-dimensional Gaussian state and semantic correction. The method is applicable to robotic arms, mobile manipulation robots, humanoid robots, or other robot platforms with end effectors, and can be used for grasping, placing, pushing, pulling, opening and closing drawers, stacking objects, and other multi-step operation tasks. The end effector can be a gripper, a dexterous hand, an adhesive end effector, or other mechanisms capable of performing physical operations.

[0033] The method in this embodiment includes a training phase, a model deployment phase, and an online decision-making phase. The training phase is used to obtain the adaptation parameters of the action decision model, the action conditional dynamic Gaussian propagation model, and the open vocabulary semantic enhancement module; the model deployment phase is used to deploy the trained or adapted model to a computing terminal that can be called online by the robot; the online decision-making phase is used to generate the final execution action based on the current observation, robot state, and language task conditions.

[0034] During the training phase, robot operation task data is first collected. This data includes multi-view observation data, robot state information, and language task conditions. Multi-view observation data can include images, depth maps, camera poses, and timestamps from multiple RGB cameras, RGB-D cameras, depth cameras, head-mounted vision devices, or wrist cameras. Robot state information can include at least one of the following: robot joint angles, joint velocities, end effector pose, gripper opening / closing state, and force or torque information. Language task conditions can be natural language commands, such as "grab the red cube," "open the left drawer," or "put the blue block into the green plate."

[0035] The robot operation task data is preprocessed. Preprocessing includes at least one of time synchronization, hand-eye calibration, coordinate transformation, and data format conversion. Time synchronization aligns observation frames acquired by different cameras with the robot's state information; hand-eye calibration obtains extrinsic parameters between the camera coordinate system and the robot's base coordinate system or world coordinate system; coordinate transformation unifies images, depth data, point clouds, or 3D Gaussian primitives into the same spatial coordinate system; and data format conversion organizes image sequences, robot actions, and language task conditions into data samples suitable for model training.

[0036] A dynamic three-dimensional Gaussian state representation is constructed based on preprocessed multi-view observation data. Let the current time be... A dynamic three-dimensional Gaussian state can be represented as a set of multiple three-dimensional Gaussian elements. ,in, Indicates the number of three-dimensional Gaussian elements. Indicates the current time The Each three-dimensional Gaussian element. It can include at least one of the following: center position, scale, rotation, opacity, color or appearance features, semantic features, and task-related feature embeddings. By using image and depth information from multi-view observation data, the position and appearance attributes of each 3D Gaussian primitive in the scene can be estimated; by observing continuously, the spatial position, visibility, appearance features, or task-related feature embeddings of the 3D Gaussian primitives can be updated, thereby forming a dynamic 3D Gaussian state representation.

[0037] In one optional implementation, the construction of a dynamic 3D Gaussian state representation includes: back-projecting depth pixels based on the intrinsic and extrinsic parameters of the camera acquiring multi-view observation data to obtain 3D points in the world coordinate system; initializing or updating the center position of the 3D Gaussian primitives based on the 3D points; determining the scale and rotation of the 3D Gaussian primitives based on the local point distribution or preset scale parameters; and determining the appearance and semantic features of the 3D Gaussian primitives based on the image color, visual coding features, or semantic coding features of the multi-view observation data. For dynamic objects in continuous operation, the parameters of the 3D Gaussian primitives can be updated based on the observation matching results or state propagation model at adjacent time points.

[0038] To reduce the computational overhead of directly inputting the dynamic 3D Gaussian state representation into the action decision model, this embodiment performs voxel tokenization on the dynamic 3D Gaussian state representation. Specifically, a regular voxel space is established within the robot's workspace, the regular voxel space having a preset spatial boundary and voxel resolution. Each 3D Gaussian element is mapped to a corresponding voxel unit according to its center position, and the features of 3D Gaussian elements in the same voxel unit or a local voxel region containing that voxel unit are aggregated to obtain voxel tokens. Aggregation methods may include average pooling, max pooling, attention-weighted aggregation, distance-based weighted aggregation, or confidence-based weighted aggregation.

[0039] Each voxel token may include at least one of the following: voxel center location, number of 3D Gaussian elements within the voxel, aggregated geometric features, aggregated appearance features, aggregated semantic features, and task-related features. Voxel tokenization transforms a variable number of 3D Gaussian elements into structured state inputs, enabling action decision models to process observation information from different scenarios and with varying numbers of targets in a unified format.

[0040] The voxel tokens are subjected to semantically guided compression. This semantically guided compression calculates the relevance of the voxel tokens to the current task based on task semantic conditions, prioritizing the retention of task-related voxel tokens. The task semantic conditions include at least one of the following: language task conditions, target semantic categories or attribute words parsed from the language task conditions, and semantic category mapping results obtained from open-vocabulary semantic recognition, two-dimensional semantic segmentation, or three-dimensional mapping. The task relevance can be determined based on at least one of the following: the similarity between the semantic features of the voxel tokens and the corresponding text features of the language task conditions; the degree of matching between the region where the voxel tokens are located and the target semantic categories or attribute words; the correspondence between the voxel tokens and the semantic category mapping results; the spatial relationship between the voxel tokens and the current position of the robot's end effector; and the attention weight of the voxel tokens in candidate action prediction.

[0041] In one optional implementation, the language task conditions are text-encoded to obtain language tokens; a similarity score is calculated between each voxel token and the language tokens, and they are sorted according to similarity score, spatial reachability, or target region confidence; voxel tokens with high rankings or those within the target candidate region are retained, while background voxel tokens irrelevant to the current task are deleted or downgraded. Simultaneously, the voxel index, 3D Gaussian meta-index, or pruning index corresponding to the retained voxel tokens are recorded so that the 2D semantic results or action correction results can be mapped back to the original 3D Gaussian state space later.

[0042] The compressed voxel tokens are fused with the language tokens corresponding to the language task conditions to obtain a structured state representation. This structured state representation contains current visual 3D state information and task semantic information corresponding to the language task conditions. The fusion method can be concatenation, cross-attention fusion, multimodal Transformer fusion, gating fusion, or other methods capable of jointly modeling the visual 3D state and language task conditions. This structured state representation serves as input to the action decision model, used to generate a basic action distribution or a candidate action set.

[0043] Training the action decision model. The action decision model can employ a behavior cloning model, a diffusion strategy model, a Transformer strategy model, a classification-regression hybrid model, or other models capable of outputting robot actions based on structured state representations. The actions can include at least one of the following: end effector target position, target posture, gripper opening / closing state, discrete action category, continuous control variable, or action sequence. During training, the action decision model can be optimized using teaching data, simulation data, data collected from real robots, or a mixture of these data, employing supervised learning, imitation learning, offline reinforcement learning, or a combination thereof.

[0044] A dynamic Gaussian propagation model with action conditions is trained. This model receives the current dynamic 3D Gaussian state representation, candidate actions, and language task conditions, and outputs a short-term future dynamic 3D Gaussian state or a Gaussian state increment. The dynamic Gaussian propagation model is a neural network model constructed for predicting changes in 3D Gaussian states. In one optional implementation, the dynamic Gaussian propagation model includes a Gaussian state encoder, an action encoder, a language condition encoder, a feature fusion module, a dynamic propagation module, and a state prediction head. The Gaussian state encoder encodes the current dynamic 3D Gaussian state representation; the action encoder encodes the pose, trajectory, grasping parameters, or action category of candidate actions; the language condition encoder encodes the language task conditions; the feature fusion module fuses Gaussian state features, action features, and language condition features; the dynamic propagation module predicts short-term state changes under the action of candidate actions; and the state prediction head outputs a short-term future dynamic 3D Gaussian state or a Gaussian state increment. The short-term future can correspond to one or more future time steps. The model output may include at least one of the following: changes in the center position of the 3D Gaussian meta-meta-elements, scale changes, rotation changes, opacity changes, appearance feature changes, semantic feature changes, or task-related region changes.

[0045] In one alternative implementation, the action-conditional dynamic Gaussian propagation model uses the current Gaussian state set. Candidate actions and language task conditions Given the input, predict the set of future Gaussian states. or Gaussian state increment Based on the predicted future Gaussian state, the predicted observation for the next time step can be rendered or reconstructed and compared with the actual observation for the next time step to obtain the image reconstruction error; alternatively, the predicted Gaussian state can be compared with the actual or estimated Gaussian state for the next time step to obtain the state propagation consistency error. Based on the image reconstruction error and / or the state propagation consistency error, the action-conditional dynamic Gaussian propagation model is trained.

[0046] Domain-adaptive training is performed on the open-vocabulary semantic enhancement module. This module can include one or a combination of open-vocabulary 2D detection models, open-vocabulary 2D segmentation models, and visual-language localization models. To improve the model's ability to recognize target objects, local parts, color attributes, and spatial modifiers in robot operation scenarios, the LoRA method can be used to domain-adapt the text encoder in the open-vocabulary semantic enhancement module model. For example, training samples can be constructed based on language instructions and target annotations in robot operation tasks, enabling the text encoder to better distinguish operation-related phrases such as "drawer handle," "red square," "green plate," and "left target."

[0047] During the model deployment phase, the trained or adapted action decision model, action-conditional dynamic Gaussian propagation model, and open-vocabulary semantic enhancement module are deployed to the robot control end, edge computing end, or server end. The deployment method can be determined based on real-time requirements and computing resources. For example, the action decision model and action-conditional dynamic Gaussian propagation model are deployed at the robot control end or edge computing end to ensure low-latency action generation; the open-vocabulary semantic enhancement module is deployed at the edge computing end or server end and is only invoked when semantic correction is triggered to reduce online inference overhead.

[0048] At the start of the online decision-making phase, the robot acquires multi-view observation data, robot state information, and language task conditions for the current moment, and constructs a current dynamic 3D Gaussian state representation using the same or corresponding processing methods as in the training phase. Then, the current dynamic 3D Gaussian state representation is voxelized and aggregated into voxel tokens. The voxel tokens are then subjected to semantically guided compression, and the compressed voxel tokens are fused with the language tokens to obtain the current structured state representation.

[0049] The current structured state representation is input into the action decision model to obtain a basic action distribution or a candidate action set. The basic action distribution can be a discrete action category probability distribution, a continuous pose distribution, a gripper opening / closing probability, or an action sequence probability. The candidate action set can include multiple candidate end-effector poses, candidate gripping points, candidate operation trajectories, or candidate action primitives.

[0050] When short-term future state evaluation is required, the current dynamic 3D Gaussian state representation, candidate actions, and language task conditions are input into the action-conditional dynamic Gaussian propagation model to predict the short-term future dynamic 3D Gaussian state or Gaussian state increment corresponding to each candidate action. Based on the prediction results, the candidate actions are evaluated to obtain the future state evaluation result or future state evaluation score. Evaluation indicators may include the consistency between the predicted state and the current task objective, whether the target region changes towards the desired state, whether the target 3D Gaussian primitives are correctly approached or captured, whether non-target regions are disturbed, and whether the uncertainty of the future state prediction is too high. The current task objective can be determined by the language task conditions.

[0051] The action decision model generates a basic action distribution or a set of candidate actions based on the structured state representation and assigns a basic score to each candidate action. The basic score can be at least one of action probability, candidate action confidence, action value estimation, or log probability. Based on the short-term future state evaluation results, the basic scores of each candidate action are updated to obtain updated candidate action scores. For example, candidate actions that can change the target area towards the desired state and have low prediction uncertainty have their scores increased; candidate actions that cause little change in the target area, significant disturbance in non-target areas, or high prediction uncertainty have their scores decreased. Then, the candidate actions are ranked according to the updated candidate action scores, and the candidate action with the highest score, or the top-ranked candidate actions and their scores, are determined as the preliminary action decision state.

[0052] After obtaining the initial action decision state, action uncertainty is calculated. Action uncertainty can be determined based on at least one of the following: the classification margin between the highest and second-highest probability actions in the basic action distribution, the entropy of the basic action distribution, the difference in candidate action scores, the variance of the continuous action distribution, and the short-term future state prediction uncertainty. The smaller the classification margin, the larger the entropy, the smaller the difference in candidate action scores, or the higher the future state prediction uncertainty, the more likely the current action decision is to be ambiguous.

[0053] The decision to trigger semantic correction is based on action uncertainty, the invocation budget for open-vocabulary semantic enhancement, and a cooling-off condition. The invocation budget limits the number of times the open-vocabulary semantic enhancement module is invoked per unit time, per task, or per operation sequence. The cooling-off condition limits the minimum time interval or minimum number of decision steps between two consecutive semantic corrections. Semantic correction is triggered when action uncertainty meets a preset uncertainty condition, the invocation budget does not exceed a preset value, and the cooling-off condition is met; otherwise, semantic correction is not triggered, and the final execution action is generated directly based on the preliminary action decision state.

[0054] When semantic correction is triggered, target object words, local component words, attribute words, or spatial modifier words are extracted from the language task conditions to construct a set of query phrases. For example, for the language task condition "grab the red square on the left", query phrases such as "red square", "red square on the left", and "square" can be constructed; for the language task condition "open the drawer handle", query phrases such as "drawer handle", "handle", and "target drawer" can be constructed.

[0055] The current 2D image and query phrase set are input into the open vocabulary semantic enhancement module to obtain at least one of the following: candidate bounding boxes, candidate categories, detection confidence, and semantic masks. For multiple candidate semantic results, they can be filtered or sorted according to at least one of the following: semantic mask area, detection confidence, query phrase specificity, degree of matching with language task conditions, degree of overlap with robot reachable areas, and area penalty term, to obtain target semantic constraints.

[0056] The 3D semantic index mapping module maps target semantic constraints to 3D semantic index constraints. Specifically, for the semantic mask corresponding to the target semantic constraint, the depth values ​​of the pixels within the mask are obtained; using the intrinsic and extrinsic parameters of the corresponding camera, the pixels within the mask are back-projected into 3D points in the world coordinate system; based on the voxel resolution, 3D spatial boundary, and spatial position of the 3D Gaussian units, the 3D points are discretized into voxel indices, 3D Gaussian unit indices, or 3D state indices; the indices are merged and deduplicated to obtain the 3D semantic index constraints.

[0057] In one optional implementation, for each back-projected 3D point, its corresponding voxel element is first calculated to obtain a voxel index; then, the corresponding voxel token and intra-voxel Gaussian element are found based on the voxel index; alternatively, the 3D Gaussian element whose distance is less than a preset threshold or whose distance is the closest can be selected as the target Gaussian element index based on the distance between the 3D point and the center position of the Gaussian element. If the same target semantic constraint comes from multiple perspectives, the voxel indices or Gaussian element indices obtained from multiple perspectives are fused to improve the stability of the 3D semantic index constraint.

[0058] The action correction and rollback module determines the validity of the 3D semantic index constraints. The 3D semantic index constraints are deemed invalid when: the 3D semantic index constraint is empty; the corresponding 2D semantic result does not meet the area threshold; the detection confidence level is lower than the confidence threshold; the backprojection depth missing ratio is too high; or the mapped 3D region exceeds the robot's workspace or is significantly mismatched with the language task conditions. If the 3D semantic index constraint is invalid, the semantic result is not used for correction, and the module rolls back to the initial action decision state.

[0059] When the 3D semantic index constraints are valid, the action correction and rollback module corrects the distribution of candidate actions or basic actions based on the 3D semantic index constraints. Correction methods include: increasing the score of candidate actions acting on the target region and decreasing the score of candidate actions acting on non-target regions or low-confidence semantic regions; correcting candidate grab points, end-target positions, or end-target poses to within or near the target region; applying semantic mask constraints to the distribution of basic actions to concentrate the distribution of basic actions more in the target region; or deleting candidate actions that clearly act on non-target objects.

[0060] In one optional implementation, if the end target position corresponding to a candidate action falls within the voxel region corresponding to the 3D semantic index constraint, the score of the candidate action is increased; if the end target position corresponding to a candidate action is located in a non-target region, or the distance to the region corresponding to the 3D semantic index constraint is greater than a preset distance threshold, the score of the candidate action is decreased. Finally, the candidate action with the highest score is selected as the final execution action, or the final execution action is obtained by sampling based on the corrected basic action distribution.

[0061] Robot control commands are generated based on the final executed action. These commands may include end-effector pose, gripper opening / closing commands, joint space trajectories, Cartesian space trajectories, motion primitive invocation commands, or low-level control variables. The robot control unit drives the robot to perform corresponding physical operations according to these commands, and continuously collects new multi-view observations and robot state information during execution to facilitate the next online decision-making process.

[0062] This embodiment also provides a robot operation decision-making system based on three-dimensional Gaussian state and semantic correction. The system includes a data acquisition and preprocessing module, a dynamic three-dimensional Gaussian state representation module, an action decision model training module, an open-vocabulary semantic enhancement and adaptation module, a model deployment module, a basic action decision module, a semantic risk gating module, a three-dimensional semantic index mapping module, an action correction and backtracking module, and a robot control interface module.

[0063] The data acquisition and preprocessing module is used to acquire multi-view observation data, robot status information and language task conditions, and to complete time synchronization, hand-eye calibration, coordinate transformation and data format conversion.

[0064] The dynamic 3D Gaussian state representation module is used to construct a dynamic 3D Gaussian state representation based on multi-view observation data, generate voxel tokens, and perform semantically guided compression.

[0065] The action decision model training module is used to train the action decision model and the action conditional dynamic Gaussian propagation model.

[0066] The open vocabulary semantic enhancement adaptation module is used to perform domain adaptation training on the open vocabulary semantic enhancement module. The domain adaptation training includes adapting the GroundingDINO text encoder through the LoRA method.

[0067] The model deployment module is used to deploy the trained or adapted model to the robot control terminal, edge computing terminal, or server terminal.

[0068] The basic action decision module is used to generate a basic action distribution or candidate actions based on the current structured state representation and language task conditions, and update the candidate action scores based on the short-term future state prediction results.

[0069] The semantic risk gating module is used to determine whether to trigger semantic correction based on action uncertainty, call budget, and cooldown conditions.

[0070] The three-dimensional semantic index mapping module is used to map the two-dimensional candidate boxes or semantic masks output by the open vocabulary semantic enhancement module into three-dimensional semantic index constraints.

[0071] The action correction and rollback module is used to correct actions when the three-dimensional semantic index constraints are valid, and to roll back to the initial action decision state when the three-dimensional semantic index constraints are invalid.

[0072] The robot control interface module is used to convert the final execution action into robot control instructions.

[0073] In a specific application example, the robot receives the linguistic task condition "grab the red square on the left". The system constructs a dynamic 3D Gaussian state representation through multi-view observations, maps Gaussian units to voxel tokens, and prioritizes retaining relevant voxel tokens based on linguistic conditions such as "left", "red", and "square". The action decision model generates multiple candidate grasping actions, and the action condition dynamic Gaussian propagation model predicts the short-term future state after each candidate action is executed. When the scores of candidate actions are close or the action distribution entropy is high, the semantic risk gating module triggers the open vocabulary semantic enhancement module to obtain a 2D semantic mask for "red square on the left". The system backprojects this semantic mask and maps it to a 3D semantic index constraint, improving the score of candidate grasping actions acting on the target 3D region and reducing the score of candidate actions acting on other similar squares, finally outputting the corrected grasping action.

[0074] In another specific application example, the robot receives the language task condition "open the drawer handle". The system constructs a dynamic 3D Gaussian state representation including the drawer and handle regions based on multi-view observations, and forms a structured state representation through voxel tokenization. The basic motion decision model generates candidate actions that act on the area in front of the drawer. If the candidate actions are unstable in locating the local component of the "handle", the semantic risk gating module triggers the open vocabulary semantic enhancement module to obtain the 2D semantic constraints of the drawer handle; the 3D semantic index mapping module maps the 2D target semantic constraints to the 3D Gaussian state space. The motion correction and backtracking module corrects the end-effector target pose according to the 3D semantic index constraints corresponding to the handle, so that the robot end effector preferentially acts on the handle region, thereby reducing the risk of performing actions on the drawer panel or non-target locations.

[0075] The model structure, threshold, scoring function, voxel resolution, number of cameras, robot platform, and task type described in the above embodiments are all illustrative. Those skilled in the art can adjust these features according to different robot hardware, sensor configurations, task scenarios, and computing power conditions. As long as they employ a combination of dynamic 3D Gaussian state representation, semantic-guided compression, short-term future state evaluation of action conditions, and open-vocabulary semantic correction, they all fall within the scope of protection of this invention.

Claims

1. A robot operation decision-making method based on three-dimensional Gaussian state and semantic correction, characterized in that, The method includes: (1) Acquire robot operation task data containing multi-view observation data, robot state information and language task conditions and perform preprocessing; (2) A dynamic three-dimensional Gaussian state representation is constructed based on multi-view observation data, and the dynamic three-dimensional Gaussian state representation is organized into a structured state representation by combining language task conditions through voxel tokenization and semantically guided compression; (3) Generate basic action distribution or candidate action set based on structured state representation, predict future dynamic three-dimensional Gaussian state or Gaussian state increment based on current dynamic three-dimensional Gaussian state representation, candidate actions and language task conditions, and evaluate candidate actions based on prediction results; (4) Obtain the preliminary action decision state based on the evaluation results, calculate the action uncertainty, and determine whether semantic correction is required. If so, generate target semantic constraints based on language task conditions and map them to three-dimensional semantic index constraints. When the three-dimensional semantic index constraints are valid, correct the basic action distribution or candidate actions based on the three-dimensional semantic index constraints to obtain the final action to be executed. When semantic correction is not required or the three-dimensional semantic index constraints are invalid, obtain the final action to be executed based on the preliminary action decision state. (5) Generate robot control instructions based on the final execution action to drive the robot to perform the corresponding physical operation.

2. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, The robot operation decision-making method includes a training phase, a model deployment phase, and an online decision-making phase; The training phase is used to train the adaptation parameters of the action decision model, the action conditional dynamic Gaussian propagation model, and the open vocabulary semantic enhancement module. The action decision model is used to generate a basic action distribution or a set of candidate actions based on the current structured state representation. The action conditional dynamic Gaussian propagation model is used to predict the short-term future dynamic three-dimensional Gaussian state or Gaussian state increment based on the current dynamic three-dimensional Gaussian state representation, candidate actions, and language task conditions. The open vocabulary semantic enhancement module is used to generate target semantic constraints using a set of query phrases constructed based on language task conditions and the current two-dimensional image as input. The three-dimensional semantic index mapping module is used to map the target semantic constraints to three-dimensional semantic index constraints. The action correction and rollback module is used to determine the validity of the three-dimensional semantic index constraints, and to perform action correction when the three-dimensional semantic index constraints are valid, and to roll back to the initial action decision state when they are invalid. The model deployment phase is used to deploy the trained or adapted action decision model, action conditional dynamic Gaussian propagation model and open vocabulary semantic enhancement module to the robot control end, edge computing end or server end. The online decision-making stage is used to call the deployed action decision model, action condition dynamic Gaussian propagation model and open vocabulary semantic enhancement module to execute steps (1) to (4).

3. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 2, characterized in that, The action-conditional dynamic Gaussian propagation model is trained based on rendering or reconstructing the next moment's predicted observations in the short-term future dynamic 3D Gaussian state, and is supervised by the image reconstruction error between the next moment's predicted observations and the next moment's actual observations, and / or the state propagation consistency error between the predicted Gaussian state and the actual or estimated Gaussian state.

4. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 2, characterized in that, The open vocabulary semantic enhancement module needs to undergo domain adaptation training. The domain adaptation training includes adapting the text encoder in the model of the open vocabulary semantic enhancement module to the domain using the LoRA method. The model includes one or a combination of an open vocabulary two-dimensional detection model, an open vocabulary two-dimensional segmentation model, and a visual language localization model.

5. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, The dynamic 3D Gaussian state representation includes multiple explicit 3D Gaussian primitives, each of which includes at least one of the following: center position, scale, rotation, opacity, appearance features, semantic features, or task-related features. The voxel tokenization is obtained by mapping 3D Gaussian primitives to a regular voxel space, aggregating primitive features in the same voxel unit or local voxel block, and forming voxel tokens. The semantically guided compression prioritizes retaining task-related voxel tokens based on language task conditions or semantic category mapping results, and retains clipping indexes for mapping back to the original 3D Gaussian state space.

6. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, The specific process for determining whether semantic correction is needed is as follows: based on the action uncertainty of the preliminary action decision state, the invocation budget of open vocabulary semantic enhancement, and the cooling condition, it is determined whether semantic correction should be triggered; the action uncertainty is determined based on at least one of the following: the classification interval between the highest probability action and the second highest probability action in the basic action distribution, the entropy of the basic action distribution, the difference in candidate action scores, the variance of the continuous action distribution, and the uncertainty of short-term future state prediction; the invocation budget is used to limit the number of times semantic correction is triggered per unit time, in a single task, or in a single operation sequence; the cooling condition is used to limit the minimum time interval or the minimum number of decision steps between two adjacent semantic corrections.

7. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, The target semantic constraints generated based on language task conditions include: Extract target object words, local component words, attribute words, or spatial modifier words from language task conditions to construct a query phrase set; Using an open vocabulary 2D detection model, an open vocabulary 2D segmentation model, or a visual language localization model, at least one of candidate boxes, candidate categories, detection confidence, and semantic masks is output based on the query phrase set. The candidate semantic results are filtered or sorted based on at least one of the following: semantic mask area, detection confidence, query phrase specificity, and area penalty term, to obtain the target semantic constraints.

8. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, Mapping target semantic constraints to three-dimensional semantic index constraints includes: For pixels in the semantic mask corresponding to the target semantic constraints, back-project them into three-dimensional points in the world coordinate system using the corresponding depth values, camera intrinsic parameters and camera extrinsic parameters. Based on the voxel resolution, the three-dimensional spatial boundary, and the spatial position of the three-dimensional Gaussian unit, the three-dimensional points are discretized into voxel indices, three-dimensional Gaussian unit indices, or three-dimensional state indices. The indexes are merged and deduplicated to obtain three-dimensional semantic index constraints.

9. The robot operation decision-making method based on three-dimensional Gaussian state and semantic correction according to claim 1, characterized in that, Correcting the basic action distribution or candidate actions based on the three-dimensional semantic index constraints includes: Based on the three-dimensional semantic index constraints, determine the voxel token, voxel unit, or three-dimensional Gaussian unit corresponding to the target region; improve the candidate action score acting on the target region, reduce the candidate action score acting on non-target regions or confidence semantic regions below the threshold, or correct the basic action distribution or robot end-effector target pose based on the target region; When the three-dimensional semantic index constraint is empty or the corresponding two-dimensional semantic result does not meet the area threshold or confidence threshold, the three-dimensional semantic index constraint is determined to be invalid.

10. A robot operation decision-making system based on three-dimensional Gaussian state and semantic correction, implementing the method of any one of claims 1-9, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire multi-view observation data, robot state information, and language task conditions, and to perform preprocessing. The dynamic 3D Gaussian state representation module is used to construct a dynamic 3D Gaussian state representation based on multi-view observation data, generate voxel tokens and perform semantically guided compression, and organize the dynamic 3D Gaussian state representation into a structured state representation in combination with language task conditions. The action decision model training module is used to train the action decision model and the action conditional dynamic Gaussian propagation model. An open-vocabulary semantic enhancement adaptation module is used for domain adaptation training of the open-vocabulary semantic enhancement module. The model deployment module is used to deploy trained or adapted models to the robot control end, edge computing end, or server end. The basic action decision module is used to generate basic action distributions or candidate actions based on the current structured state representation input to the action decision model, and to obtain short-term future state prediction results based on the action condition dynamic Gaussian propagation model, and to evaluate the candidate actions. The semantic risk gating module is used to determine whether to call the open vocabulary semantic enhancement module to trigger semantic correction; The three-dimensional semantic index mapping module is used to map the target semantic constraints output by the open vocabulary semantic enhancement module to three-dimensional semantic index constraints; The action correction and rollback module is used to correct actions when the three-dimensional semantic index constraints are valid, or to roll back to the initial action decision state when the three-dimensional semantic index constraints are invalid, so as to obtain the final action to be executed. The robot control interface module is used to convert the final execution action into robot control instructions.

Images