Body-aware control method

Abstract

The application provides a body intelligence control method, and belongs to the technical field of artificial intelligence. The method comprises the following steps: S10, real-time receiving of external input data; S20, real-time construction and maintenance of a dynamic semantic three-dimensional Gaussian field; S30, analysis of the natural language instruction, extraction of an operation entity, a spatial relationship word and an action word; S40, determination of a target three-dimensional region corresponding to the operation entity in the dynamic semantic three-dimensional Gaussian field; S50, end-to-end action planning in the dynamic semantic three-dimensional Gaussian field, generation of a collision-free action track; and S60, conversion of the collision-free action track into a control instruction sequence, driving of a robot executor to complete the operation corresponding to the action word. The application realizes the whole-process end-to-end integration from language understanding, entity positioning, spatial target calculation to collision-free action planning, so that path planning, collision detection, grasping posture generation and other tasks can be cooperatively completed on the same dynamically updated three-dimensional Gaussian field.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an embodied intelligent control method. Background Technology

[0002] One of the core challenges in the field of embodied intelligence is enabling robots to truly understand the three-dimensional physical world they inhabit and perform complex physical interaction tasks based on human natural language commands. Existing technologies mainly suffer from the following limitations: First, most mainstream vision-language-action (VLA) models (such as RT-2) use two-dimensional image sequences as the basis for perception and decision-making. This 2D vision-based paradigm has inherent defects when dealing with tasks that require precise three-dimensional spatial understanding. For example, for instructions such as "put the cup behind the table", the model has difficulty in accurately inferring the spatial relationship of "behind" that depends on the observation perspective from two-dimensional images, and it also cannot effectively handle the problem of object occlusion, resulting in a high planning failure rate. Secondly, although some studies have attempted to introduce 3D scene representation, traditional methods such as mesh or voxel are difficult to build and update in real time, while Neural Radiation Field (NeRF) can generate high-quality views, but its implicit representation characteristics result in slow rendering speed and make it difficult to use directly for physical interaction and motion planning. Furthermore, most existing work combining language with 3D scenes remains at the level of semantic segmentation or object localization. For example, while advanced works such as LangSplat can embed language features into static 3D Gaussian fields (3DGS) and enable open-word queries for objects such as "red apple", they have not solved how to understand and execute instructions involving complex spatial relationships such as "inside", "above", and "between". Similarly, while works such as SpatialVLM have enhanced the spatial reasoning ability of VLM, their foundation is still a 2D image plus a depth map, rather than a native 3D representation. Finally, in the field of robot manipulation, perception and planning are severely separated. Works such as GaussianGrasper use the precise geometric information of 3DGS to generate grasping postures, but this is limited to single actions. While world models such as GWM use 3DGS to predict the future, they are themselves independent "simulators" and are separated from the decision-making body of VLA. They have failed to form an end-to-end closed-loop system that completes the entire process of "perception-understanding-planning-action" in a unified three-dimensional representation.

[0003] In summary, existing technologies lack a method that enables VLA models to directly perform complex spatial language understanding and end-to-end action planning within a unified, efficient, and explicit representation of the three-dimensional world, resulting in a significant gap between the robot's "cognition" and its "actions." Summary of the Invention

[0004] This invention provides an embodied intelligent control method to solve the problem in the prior art that embodied intelligent robots cannot directly perform complex spatial language understanding and end-to-end motion planning in a unified, efficient, and explicit three-dimensional world representation.

[0005] This invention provides an embodied intelligent control method, comprising: S10. Receive external input data in real time, including: environmental perception data from multi-view cameras, natural language commands, and robot body perception data. S20. Based on the environmental perception data, a dynamic semantic three-dimensional Gaussian field is constructed and maintained in real time. The dynamic semantic three-dimensional Gaussian field is composed of multiple semantic Gaussian elements. Each semantic Gaussian element contains geometric attributes, appearance attributes, and semantic attributes. The semantic attributes include semantic feature vectors and object instance IDs. S30. Parse the natural language instructions and extract the operation entities, spatial relation words, and action words; S40. Based on the spatial relation words obtained from the analysis, determine the target three-dimensional region corresponding to the operation entity in the dynamic semantic three-dimensional Gaussian field, specifically including: Based on the semantic feature vector, the Gaussian cluster corresponding to the operation entity is located in the dynamic semantic three-dimensional Gaussian field by semantic similarity calculation. The spatial relation terms are converted into geometric operation instructions to be executed in a three-dimensional coordinate system; Based on the geometric operation instructions, spatial operations are performed on the located Gaussian clusters in the dynamic semantic three-dimensional Gaussian field to calculate the target three-dimensional region; S50. Based on the target three-dimensional region, the robot body perception data, and the scene geometric information of the dynamic semantic three-dimensional Gaussian field, end-to-end motion planning is performed in the dynamic semantic three-dimensional Gaussian field to generate a collision-free motion trajectory. S60. Convert the collision-free motion trajectory into a sequence of control instructions to drive the robot actuator to complete the operation corresponding to the action word.

[0006] According to an embodied intelligent control method provided by the present invention, the process of generating the semantic feature vector includes: inputting a two-dimensional image from the environmental perception data into a pre-trained visual language model to extract two-dimensional semantic features at the image level; and using a feature distillation network, back-projecting the two-dimensional semantic features using the camera intrinsic and extrinsic parameter matrix and assigning them to the corresponding three-dimensional Gaussian units, so that each semantic Gaussian unit carries an explicit semantic representation aligned with the visual appearance.

[0007] According to the embodied intelligent control method provided by the present invention, the semantic feature vector is a 512-dimensional CLIP embedding vector. The semantic feature vector supports open vocabulary query, that is, for object categories or attribute descriptions not seen during training, semantic matching is achieved by calculating the cosine similarity between the text features and the semantic feature vector.

[0008] According to the embodied intelligent control method provided by the present invention, step S20, namely, the real-time construction and maintenance of a dynamic semantic three-dimensional Gaussian field, includes: Based on the environmental perception data, the semantic Gaussian elements in the dynamic semantic three-dimensional Gaussian field are updated incrementally, including adding, deleting, or adjusting the geometric, appearance, and semantic attributes of the Gaussian elements.

[0009] According to the embodied intelligent control method provided by the present invention, in step S20, the geometric attributes include position coordinates, rotation quaternions, scaling parameters, and opacity; the appearance attributes include RGB color and spherical harmonic coefficients; and the object instance ID is used to distinguish different object instances under the same semantic category.

[0010] According to the embodied intelligent control method provided by the present invention, step S40, which involves converting the spatial relation words into geometric operation instructions executed in a three-dimensional coordinate system, specifically includes: The system queries a predefined mapping table of spatial relation terms to geometric operations, and translates the spatial relation terms into corresponding geometric operations. The geometric operations include at least one of spatial range query, direction sorting, internal region calculation, and relative position offset.

[0011] According to the embodied intelligent control method provided by the present invention, in step S50, the end-to-end motion planning is performed in the dynamic semantic three-dimensional Gaussian field based on the scene geometric information of the target three-dimensional region, the robot body perception data, and the dynamic semantic three-dimensional Gaussian field to generate a collision-free motion trajectory, specifically including: Candidate paths are sampled in the dynamic semantic three-dimensional Gaussian field, and collision detection is performed by intersecting with the Gaussian density field. Optimize the candidate paths that do not collide and select the optimal path; Based on the optimal path and the target 3D region, generate a grasping posture or a placement posture; The optimal path is smoothly interpolated with the grasping or placing posture to generate a complete motion trajectory that is continuous in time and smooth in speed.

[0012] According to the embodied intelligent control method provided by the present invention, during the execution of an action, the method continuously receives updated environmental perception data, updates the dynamic semantic three-dimensional Gaussian field in real time, and dynamically adjusts the collision-free action trajectory to respond to environmental changes.

[0013] According to the embodied intelligent control method provided by the present invention, step S40, which involves calculating the target three-dimensional region in a dynamic semantic three-dimensional Gaussian field based on geometric operation instructions, includes: When multiple spatial relationship constraints exist, the intersection operation of multiple geometric constraints is performed to uniquely determine the target object or target region. The target object includes objects that are occluded from the current viewpoint but physically exist in three-dimensional space.

[0014] According to the embodied intelligent control method provided by the present invention, the natural language instructions include text instructions or text instructions converted from speech; the robot body perception data includes robot joint angles and end effector pose.

[0015] The embodied intelligent control method provided by this invention uses a dynamic semantic 3D Gaussian field as a unified world representation, which can explicitly express the dense geometry and multimodal semantics of 3D physical space, thus providing a high-precision data foundation for cross-perspective physical interaction planning. Furthermore, this invention directly geometrizes spatial relation words in natural language (such as "behind," "inside," "between," and "top layer") into precise spatial operations that can be executed in the 3D Gaussian field, thereby upgrading the model's capabilities from simple semantic queries to deep spatial relation reasoning. This enables robots to truly understand and execute complex instructions that are difficult to accomplish with existing technologies (such as "put the blue cup hidden behind the box behind the table in the middle of the top layer of the TV cabinet"). More importantly, this invention achieves end-to-end integration of the entire process from language understanding, entity localization, spatial target calculation to collision-free action planning within the aforementioned unified 3D representation, avoiding the separation between perception, cognition, and action modules. This allows tasks such as path planning, collision detection, and grasping posture generation to be completed collaboratively on the same dynamically updated 3D Gaussian field, significantly improving the overall efficiency and decision consistency of the system.

[0016] In summary, this invention enables embodied intelligent robots to respond to environmental changes in real time in dynamic and occluded environments, plan smooth, safe, and collision-free motion trajectories, significantly improve the success rate of complex spatial commands and the robustness of physical interactions, and allow users to interact with robots in a more natural and richer spatial language. Thus, it demonstrates significant practical value and broad application prospects in scenarios such as industrial operations, service robots, and home assistants. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the embodied intelligent control method provided by the present invention; Figure 2 This is a schematic diagram of the overall architecture of the embodied intelligent control system provided by the present invention; Figure 3 This is a schematic diagram of the data structure of the dynamic semantic three-dimensional Gaussian field provided by the present invention; Figure 4 This is a schematic diagram of the spatial language geometric anchoring process provided by the present invention; Figure 5 This is an end-to-end action planning flowchart based on a dynamic semantic three-dimensional Gaussian field provided by the present invention; Figure 6 This is a schematic diagram of a task execution scenario under a complex spatial instruction provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following is combined with Figures 1-6 This invention describes an embodied intelligent control method and system, which is applied in the field of embodied artificial intelligence and robotics. Specifically, it relates to a decision control method and system that deeply integrates three-dimensional scene representation, natural language processing, and robot motion planning.

[0021] On one hand, the present invention provides an embodied intelligent control method, comprising: S10. Receive external input data in real time. The external input data includes: environmental perception data from multi-view cameras (RGB-D), natural language commands, and robot body perception data. The environmental perception data includes color image data (RGB) and depth image data. Color image data is used to identify visual features such as color, texture, and edges of objects in the environment, supporting semantic understanding and object recognition. Depth image data is used to provide distance information from each pixel to the camera, which is used to recover the three-dimensional geometry of the scene and supports spatial positioning, collision detection, and three-dimensional reconstruction. Natural language instructions include text instructions or text instructions converted from speech; further, natural language instructions are instructions containing complex spatial relationships, which include multiple spatial constraints and occlusion relationships. Robot body perception data includes robot joint angles and end effector pose.

[0022] S20. Based on environmental perception data, a dynamic semantic three-dimensional Gaussian field is constructed and maintained in real time. The dynamic semantic three-dimensional Gaussian field consists of multiple displayed semantic Gaussian elements. Each semantic Gaussian element contains geometric attributes, appearance attributes, and semantic attributes. The semantic attributes include semantic feature vectors and object instance IDs. The geometric attributes include position coordinates, rotation quaternions, scaling parameters, and opacity; the appearance attributes include RGB color and spherical harmonic coefficients; and the object instance ID is used to distinguish different object instances within the same semantic category. Among them, the semantic feature vector is obtained by distilling the output features of the pre-trained visual language model, and is used to support semantic query and entity localization under open vocabulary conditions; Furthermore, the semantic feature vector generation process includes: inputting the two-dimensional image from the environmental perception data into the pre-trained visual language model to extract the two-dimensional semantic features at the image level; and using a feature distillation network, back-projecting the two-dimensional semantic features using the camera intrinsic and extrinsic parameter matrices and assigning them to the corresponding three-dimensional Gaussian units, so that each semantic Gaussian unit carries an explicit semantic representation aligned with the visual appearance.

[0023] More specifically, the semantic feature vector is a 512-dimensional CLIP embedding vector. The semantic feature vector supports open vocabulary queries, that is, for object categories or attribute descriptions not seen during training, semantic matching is achieved by calculating the cosine similarity between the text features and the semantic feature vector.

[0024] To enable open-vocabulary query capabilities for 3D Gaussian fields, this invention designs a feature distillation network. This network utilizes the intrinsic parameter matrix of a multi-view camera. and extrinsic parameter matrix Pixel coordinates on a two-dimensional image Projected as a line-of-sight ray in three-dimensional space. For any semantic Gaussian unit in space... Its central position is Calculate the projected pixel position of the center point in the current camera coordinate system. : ; like If the pixel falls within the image feature map of the current viewpoint, then extract the two-dimensional feature vector of the pre-trained visual language model (such as 512-dimensional CLIP) corresponding to that pixel location. The Gaussian element is mapped and assigned to it through a lightweight distillation network consisting of a fully connected layer and a residual structure.

[0025] To ensure the generalizability of the distilled 3D features, the feature distillation network is optimized during training by minimizing the difference between the rendered 2D semantic map and the real feature map extracted by the pre-trained model. Its core loss function is... Represented as: ;

[0026] in The set of pixels from the current viewpoint. This indicates rendering to planar pixels via Gaussian rasterization. semantic features at the location For pre-trained models at pixels The true embedding vector at that location, To balance the scalar weighting coefficients.

[0027] Furthermore, to adapt to dynamic production environments, the same Gaussian element may be repeatedly overwritten by multiple frames of observation when the robot moves or the object is displaced. Therefore, the feature distillation network of this invention introduces an incremental update strategy based on Kalman filtering under multi-frame observations: for the first... t Newly observed two-dimensional semantic features The semantic feature vector corresponding to the three-dimensional Gaussian unit. Update using the following formula: ; in, ; in, For dynamic fusion weights, based on historical uncertainty and observation noise from the current perspective Jointly decided. Observational noise. It is proportional to the object distance (depth value) of the camera and the angle between the Gaussian element and the camera's line of sight. This mechanism can effectively suppress boundary noise caused by single-frame 2D image features.

[0028] S30. Parse natural language instructions and extract operation entities, spatial relation words, and action words; The operational entities include, but are not limited to: the object to be operated on (such as "cup," "blue cup," etc., which are target objects that need to be grasped, moved, or manipulated) and reference objects or containers (such as "table," "box," "TV cabinet," etc., used to locate spatial relationships (such as "placed behind the table," "taken out of the drawer"). The operational entity can be a single object (such as "apple") or a composite description (such as "the blue cup behind the table that is blocked by the box" containing multiple entities such as cup, table, and box). These entities are all queried and located in a dynamic semantic three-dimensional Gaussian field using semantic feature vectors. Among them, spatial relation words include at least one of the following: behind, in front, above, below, inside, outside, left, right, between, top layer, bottom layer, middle, obscured; Action words refer to verbs that are extracted from natural language instructions and represent the specific physical operations or behaviors that the robot should perform; action words include, but are not limited to: grasp, place, move, pick up, put down, push, pull, open, close, etc.

[0029] S40. Based on the spatial relation words obtained from the analysis, determine the target three-dimensional region corresponding to the operational entity in the dynamic semantic three-dimensional Gaussian field, specifically including: Based on semantic feature vectors, Gaussian clusters corresponding to operational entities are located in a dynamic semantic 3D Gaussian field by calculating semantic similarity. Convert spatial relation terms into geometric operation instructions that can be executed in a three-dimensional coordinate system; Based on geometric operation instructions, spatial operations are performed on the located Gaussian clusters in a dynamic semantic 3D Gaussian field to calculate the target 3D region; S50. Based on the scene geometry information of the target 3D region, robot body perception data and dynamic semantic 3D Gaussian field, end-to-end motion planning is performed in the dynamic semantic 3D Gaussian field to generate collision-free motion trajectories. S60. Convert the collision-free motion trajectory into a sequence of control instructions to drive the robot actuator to complete the operation corresponding to the action word.

[0030] In this embodiment, a dynamic semantic 3D Gaussian field is constructed and maintained in real time, including: incrementally updating the semantic Gaussian elements in the dynamic semantic 3D Gaussian field based on environmental perception data, including adding, deleting, or adjusting the geometric, appearance, and semantic attributes of the Gaussian elements. That is, when the environment changes, the corresponding semantic Gaussian elements are added, deleted, or updated, so that the dynamic semantic 3D Gaussian field updates in real time with changes in the environment.

[0031] In step S40, spatial relation terms are converted into geometric operation instructions to be executed in a three-dimensional coordinate system. Specifically, this includes: querying a predefined mapping table from spatial relation terms to geometric operations, translating the spatial relation terms into corresponding geometric operations, and the geometric operations including at least one of spatial range query, direction sorting, internal region calculation, and relative position offset.

[0032] The mapping relationship between predefined spatial relation terms and geometric operation instructions is as follows: Following this: Using the observer and the target object as references, perform a spatial search in the region far from the target object; Front: Using the observer and the target object as references, perform a spatial search in the near-end region of the target object; Above: Calculate the spatial region above the target object; Below: Using the target object as a reference, calculate the spatial region below it; Inside: Identify the Gaussian clusters of the target container and calculate its internal three-dimensional spatial region; Outside: Calculate the annular space region that is adjacent to the target object and excludes the inside, based on the target object. Left side: Calculate the spatial region along the left side of the target object (e.g., the negative X-axis direction of the world coordinate system); Right side: Calculate the spatial region along the right side of the target object (e.g., the positive X-axis direction of the world coordinate system); Between: Using two target objects as a reference, calculate the connecting region near the midpoint between them; Top layer: Sort the Gaussian subclusters of the specified set of objects vertically by Y coordinate and select the spatial region corresponding to the highest layer; Bottom layer: Sort the Gaussian subclusters of the specified set of objects vertically by Y coordinate and select the spatial region corresponding to the bottom layer; Center: Using the target object as a reference, calculate the central region of its horizontal projection (XZ plane), or the center position between the two boundaries; Occlusion: Based on the observer and the target object, detect whether there are opaque Gaussian elements on the line of sight in a three-dimensional Gaussian field. If they exist, it is determined to be an occluded state, and a path to bypass the occluder needs to be planned.

[0033] In step S40, calculating the target 3D region in the dynamic semantic 3D Gaussian field based on geometric operation instructions includes: when there are multiple spatial relationship constraints, performing the intersection operation of multiple geometric constraints to uniquely determine the target object or target region. The target object includes objects that are occluded from the current viewpoint but physically exist in 3D space.

[0034] In step S50, based on the scene geometry information of the target 3D region, robot body perception data, and dynamic semantic 3D Gaussian field, end-to-end motion planning is performed in the dynamic semantic 3D Gaussian field to generate a collision-free motion trajectory, specifically including: Candidate paths are sampled in a dynamic semantic 3D Gaussian field, and collision detection is performed by intersecting with the Gaussian density field. Optimize the candidate paths that do not collide and select the optimal path; resample the paths that collide. Based on the optimal path and the target 3D region, a grasping posture or a placement posture is generated; specifically, a stable grasping posture is generated based on the geometric surface normal information of the Gaussian cluster of the target object, and a placement posture is generated in the target region. The optimal path is smoothly interpolated with the grab or place posture to generate a complete motion trajectory that is continuous in time and smooth in speed.

[0035] Among them, the end-to-end motion planning also utilizes the geometric information of occluded objects in the dynamic semantic 3D Gaussian field to plan a collision-free path to bypass the occluders.

[0036] During the execution of the action, it continuously receives updated environmental perception data, updates the dynamic semantic 3D Gaussian field in real time, and dynamically adjusts the collision-free action trajectory to respond to environmental changes.

[0037] Figure 5 The flowchart of the end-to-end motion planning based on a three-dimensional Gaussian field provided by the present invention shows the detailed steps of how to perform path sampling, collision detection, path optimization, and combined with grasping posture generation after obtaining the target three-dimensional region in the three-dimensional Gaussian field, and finally output a smooth, collision-free complete motion trajectory.

[0038] In step S60, the operational relationships corresponding to the action words are as follows: Grab: Pick up a target object from the current location or environment; Placement: Places the target object into the specified target area; Move: Transfer an object from one location to another; Pick up: same as grab; Put down: same as place; Push: To apply a pushing force to an object to change its position; Pull: Applying a pulling force to an object to change its position; Open: to change a container (such as a drawer or door) from a closed state to an open state; Close: Changes a container from an open state to a closed state.

[0039] The embodied intelligent control system provided by the present invention is described below. The embodied intelligent control system described below can be referred to in correspondence with the embodied intelligent control method described above.

[0040] On the other hand, the present invention also provides an embodied intelligent control system, comprising: The external input module is used to receive external input data in real time. The external input data includes: environmental perception data from multi-view cameras, natural language commands, and robot body perception data. The 3D Gaussian field construction and maintenance module is used to construct and maintain a dynamic semantic 3D Gaussian field in real time based on environmental perception data. The dynamic semantic 3D Gaussian field is composed of multiple semantic Gaussian elements. Each semantic Gaussian element contains geometric attributes, appearance attributes, and semantic attributes. The semantic attributes include semantic feature vectors and object instance IDs. The spatial language parsing module is used to parse natural language commands and extract operation entities, spatial relation words, and action words; The spatial relationship geometric anchoring module is used to determine the target 3D region corresponding to the operation entity in a dynamic semantic 3D Gaussian field based on the parsed spatial relationship words. Specifically, it includes: locating the Gaussian cluster corresponding to the operation entity in the dynamic semantic 3D Gaussian field by calculating semantic similarity based on semantic feature vectors; converting the spatial relationship words into geometric operation instructions to be executed in a 3D coordinate system; and performing spatial operations on the located Gaussian cluster in the dynamic semantic 3D Gaussian field based on the geometric operation instructions to calculate the target 3D region. The 3D motion planning module is used to perform end-to-end motion planning in the dynamic semantic 3D Gaussian field based on the scene geometry information of the target 3D region, robot body perception data and dynamic semantic 3D Gaussian field, and generate collision-free motion trajectories. The execution control module is used to convert collision-free motion trajectories into a sequence of control commands, driving the robot actuators to complete the operations corresponding to the action words.

[0041] The core VLA system of this invention consists of the three-dimensional Gaussian field construction and maintenance module, the spatial language parsing module, the spatial relationship geometric anchoring module, and the three-dimensional motion planning module.

[0042] The process of generating semantic feature vectors includes: inputting two-dimensional images from environmental perception data into a pre-trained visual language model to extract image-level two-dimensional semantic features; using a feature distillation network, back-projecting the two-dimensional semantic features using the camera intrinsic and extrinsic parameter matrices and assigning them to the corresponding three-dimensional Gaussian units, so that each semantic Gaussian unit carries an explicit semantic representation aligned with the visual appearance.

[0043] The semantic feature vector is a 512-dimensional CLIP embedding vector. The semantic feature vector supports open vocabulary queries, that is, for object categories or attribute descriptions not seen during training, semantic matching is achieved by calculating the cosine similarity between the text features and the semantic feature vector.

[0044] The 3D Gaussian field construction and maintenance module is specifically used to: incrementally update the semantic Gaussian elements in the dynamic semantic 3D Gaussian field based on environmental perception data, including adding, deleting, or adjusting the geometric, appearance, and semantic attributes of Gaussian elements.

[0045] Geometric attributes include position coordinates, rotation quaternions, scaling parameters, and opacity; appearance attributes include RGB color and spherical harmonic coefficients; object instance IDs are used to distinguish different object instances within the same semantic category.

[0046] The spatial relationship geometric anchoring module specifically includes: The mapping table storage unit is used to store a mapping table from predefined spatial relation terms to geometric operations; The translation unit is used to query the mapping table and translate spatial relation terms into corresponding geometric operations. The geometric operations include at least one of spatial range query, direction sorting, internal region calculation, and relative position offset.

[0047] The 3D motion planning module includes: The path sampling and collision detection unit is used to sample candidate paths in a dynamic semantic 3D Gaussian field and perform collision detection by intersecting with the Gaussian density field. The path optimization unit is used to optimize the candidate paths that do not collide and select the optimal path. The pose generation unit is used to generate a grasping pose or a placement pose based on the optimal path and the target 3D region. The trajectory generation unit is used to smoothly interpolate the optimal path with the grasping or placement posture to generate a complete motion trajectory that is continuous in time and smooth in speed.

[0048] The system also includes: The dynamic adjustment module is used to continuously receive updated environmental perception data during the execution of actions, update the dynamic semantic 3D Gaussian field in real time, and dynamically adjust the collision-free action trajectory to respond to environmental changes.

[0049] The spatial relationship geometric anchoring module is further used to: when there are multiple spatial relationship constraints, perform the intersection operation of multiple geometric constraints to uniquely determine the target object or target region. The target object includes objects that are occluded from the current viewpoint but physically exist in three-dimensional space.

[0050] Natural language instructions include text instructions or text instructions converted from speech; robot body perception data includes robot joint angles and end effector poses.

[0051] The principle of the embodied intelligent control system of the present invention is the same as that of the embodied intelligent control method described above, and will not be repeated here.

[0052] In this embodiment, with Figure 6 Taking the complex instruction "Place the blue cup that is hidden behind the box behind the table into the middle position on the top shelf of the TV cabinet" as an example, the specific process of the embodied intelligent control method of this invention is as follows: I. The core VLA system receives and parses commands. Please refer to [link / reference]. Figure 4 Natural language commands enter the spatial language parsing module through the external input module. The spatial language parsing module performs the language parsing steps, breaking down the command into the operation entity (cup, table, box, TV cabinet), spatial relationship (behind, blocked, top layer, middle), and action (placement).

[0053] II. Performing geometric anchoring of spatial relationships. This is the core step of the invention and is completed in the spatial relationship geometric anchoring module. Specifically, it includes: Entity localization. The system utilizes semantic feature vectors to query within a dynamic semantic 3D Gaussian field, finding the Gaussian cluster that best matches the semantics of "cup," "table," "box," and "TV cabinet," such as... Figure 6 The blue cup, table, box, and TV cabinet are shown in the picture.

[0054] Spatial relationships are converted into geometric operation instructions. The system queries a built-in mapping table of spatial relationship terms to geometric operations, translating text terms such as "behind," "blocked," "top layer," and "middle" into a series of geometric operation instructions that can be executed in a three-dimensional coordinate system.

[0055] Target 3D region calculation. The system performs translated geometric operations: First, by intersecting the geometric constraints of "behind the table" and "hidden by the box," the blue cup, though physically present but invisible from the current viewpoint, is uniquely located in 3D space. Then, by performing geometric operations on the "top layer" and "middle layer" of the Gaussian cluster of the "TV cabinet," the final placement target area is calculated, such as... Figure 6 The target area is shown in the image.

[0056] III. End-to-end motion planning in a 3D Gaussian field. Please refer to [link / reference]. Figure 5 After receiving the target object and target area, the 3D motion planning module performs the following steps: Path sampling and collision detection. First, candidate paths are sampled in a 3D Gaussian field, and then collision detection is performed by intersecting the Gaussian density field.

[0057] Path optimization. For paths that may collide, resampling is performed; for paths that do not collide, optimization is performed to select the optimal path. This step ensures that the planned path can intelligently avoid collisions such as... Figure 6 Obstacles such as "boxes" in the middle.

[0058] Grasping / Placement Pose Generation. By analyzing the geometric surface normal information of the Gaussian cluster of the target object (blue cup), one or more stable grasping poses are generated; similarly, placement poses are generated in the target area.

[0059] Trajectory generation. The optimized path points and grab / place postures are smoothly interpolated to generate a complete motion trajectory that is continuous in time and smooth in velocity.

[0060] IV. Execution. Finally, the trajectory is converted into a sequence of control commands, driving the robot actuators to complete the full task of grasping, transferring, and placing, thus successfully fulfilling the complex instructions issued by the user.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An embodied intelligent control method, characterized in that, include: S10. Receive external input data in real time, including: environmental perception data from multi-view cameras, natural language commands, and robot body perception data. S20. Based on the environmental perception data, a dynamic semantic three-dimensional Gaussian field is constructed and maintained in real time. The dynamic semantic three-dimensional Gaussian field is composed of multiple semantic Gaussian elements. Each semantic Gaussian element contains geometric attributes, appearance attributes, and semantic attributes. The semantic attributes include semantic feature vectors and object instance IDs. S30. Parse the natural language instructions and extract the operation entities, spatial relation words, and action words; S40. Based on the spatial relation words obtained from the analysis, determine the target three-dimensional region corresponding to the operation entity in the dynamic semantic three-dimensional Gaussian field, specifically including: Based on the semantic feature vector, the Gaussian cluster corresponding to the operation entity is located in the dynamic semantic three-dimensional Gaussian field by semantic similarity calculation. The spatial relation terms are converted into geometric operation instructions to be executed in a three-dimensional coordinate system; Based on the geometric operation instructions, spatial operations are performed on the located Gaussian clusters in the dynamic semantic three-dimensional Gaussian field to calculate the target three-dimensional region; S50. Based on the target three-dimensional region, the robot body perception data, and the scene geometric information of the dynamic semantic three-dimensional Gaussian field, end-to-end motion planning is performed in the dynamic semantic three-dimensional Gaussian field to generate a collision-free motion trajectory. S60. Convert the collision-free motion trajectory into a sequence of control instructions to drive the robot actuator to complete the operation corresponding to the action word.

2. The embodied intelligent control method according to claim 1, characterized in that, The process of generating the semantic feature vector includes: inputting the two-dimensional image from the environmental perception data into a pre-trained visual language model to extract the two-dimensional semantic features at the image level; and using a feature distillation network, back-projecting the two-dimensional semantic features using the camera intrinsic and extrinsic parameter matrix and assigning them to the corresponding three-dimensional Gaussian units, so that each semantic Gaussian unit carries an explicit semantic representation aligned with the visual appearance.

3. The embodied intelligent control method according to claim 2, characterized in that, The feature distillation network introduces an incremental update strategy based on Kalman filtering under multi-frame observations: for the first... t Newly observed two-dimensional semantic features The semantic feature vector corresponding to the three-dimensional Gaussian unit. Update using the following formula: ; in, ; in, For dynamic fusion weights, For historical uncertainty, This represents the observation noise from the current perspective.

4. The embodied intelligent control method according to claim 2, characterized in that, The feature distillation network is optimized during training by minimizing the difference between the rendered 2D semantic map and the real feature map extracted by the pre-trained model. Its core loss function is expressed as: ; in, The set of pixels from the current viewpoint. This indicates rendering to planar pixels via forward Gaussian rasterization. semantic features at the location For pre-trained models at pixels The true embedding vector at that location, To balance the scalar weighting coefficients.

5. The embodied intelligent control method according to claim 1, characterized in that, In step S20, the geometric attributes include position coordinates, rotation quaternions, scaling parameters, and opacity; the appearance attributes include RGB color and spherical harmonic coefficients; and the object instance ID is used to distinguish different object instances under the same semantic category.

6. The embodied intelligent control method according to claim 1, characterized in that, In step S40, converting the spatial relation terms into geometric operation instructions executed in a three-dimensional coordinate system specifically includes: The system queries a predefined mapping table of spatial relation terms to geometric operations, and translates the spatial relation terms into corresponding geometric operations. The geometric operations include at least one of spatial range query, direction sorting, internal region calculation, and relative position offset.

7. The embodied intelligent control method according to claim 1, characterized in that, In step S50, the step of performing end-to-end motion planning in the dynamic semantic three-dimensional Gaussian field based on the scene geometry information of the target three-dimensional region, the robot's body perception data, and the dynamic semantic three-dimensional Gaussian field to generate a collision-free motion trajectory specifically includes: Candidate paths are sampled in the dynamic semantic three-dimensional Gaussian field, and collision detection is performed by intersecting with the Gaussian density field. Optimize the candidate paths that do not collide and select the optimal path; Based on the optimal path and the target 3D region, generate a grasping posture or a placement posture; The optimal path is smoothly interpolated with the grasping or placing posture to generate a complete motion trajectory that is continuous in time and smooth in speed.

8. The embodied intelligent control method according to claim 1, characterized in that, During the execution of the action, the system continuously receives updated environmental perception data, updates the dynamic semantic three-dimensional Gaussian field in real time, and dynamically adjusts the collision-free action trajectory to respond to environmental changes.

9. The embodied intelligent control method according to claim 1, characterized in that, In step S40, calculating the target three-dimensional region in a dynamic semantic three-dimensional Gaussian field based on geometric operation instructions includes: When multiple spatial relationship constraints exist, the intersection operation of multiple geometric constraints is performed to uniquely determine the target object or target region. The target object includes objects that are occluded from the current viewpoint but physically exist in three-dimensional space.

10. The embodied intelligent control method according to claim 1, characterized in that, The natural language instructions include text instructions or text instructions converted from speech; the robot body perception data includes robot joint angles and end effector poses.

Images