Robot fine operation method and system based on visual language model and semantic key point representation

Abstract

The invention relates to the field of robot technology and artificial intelligence, and discloses a robot fine operation method and system based on a visual language model and semantic key point characterization, a closed-loop multi-body Agent system is constructed, GPT-4o and Ground DINO are combined to accurately understand and disambiguate a natural language instruction of a user, an innovative bridging layer is introduced, and the robot fine operation method and system based on visual language model and semantic key point characterization are obtained. Abstract decision information is converted into bottom execution parameters represented by three-dimensional semantic key points, the problem of decision transfer distortion is solved, meanwhile, a KMP algorithm is improved, and the stiffness and contact force of trajectory tracking are dynamically optimized by introducing a learning-based time-varying hybrid impedance / force control strategy, so that the accuracy of trajectory tracking is improved. According to the method, the generalization success rate of fine operation tasks of the robot and the stability of physical interaction are remarkably improved, seamless connection from high-level semantic understanding to low-level action generation is achieved, and an efficient and reliable technical scheme is provided for autonomous operation of the robot in a complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot technology and artificial intelligence, in particular to a robot fine operation method based on visual language model and semantic key point representation. BACKGROUND

[0002] Currently, the mainstream robot control architecture usually adopts a hierarchical mode, that is, the upper layer is responsible for task planning and decision-making by a large model, and the lower layer is responsible for generating specific execution trajectories by motion control algorithm. However, this hierarchical architecture has an inherent technical defect: decision-making information transmission distortion. The upper layer model usually generates abstract, text form decision instructions (for example, "pick up that red cup"), while the lower layer motion planning algorithm needs accurate, quantitative parameters (for example, target pose, grasp point coordinates, trajectory key points, etc.). In the conversion process from abstract instructions to specific parameters, key geometric and semantic information is easily lost, resulting in insufficient accuracy of robot action, especially in scenes requiring fine operation (such as threading, plugging, assembly, etc.), the task success rate is low.

[0003] In addition, existing trajectory generation algorithms, such as Dynamic Motion Primitive (DMP) or Kernelized Motion Primitive (KMP), although they can learn and reproduce demonstration trajectories, but their generalization ability is limited when facing new targets or environmental changes. Especially for the start and end stages of the operation, these stages often contain key details for task success (such as alignment, slight adjustment of insertion), traditional algorithms have difficulty in fully retaining these detail features, resulting in the generated trajectories failing to meet the requirements of fine operation.

[0004] Therefore, how to effectively bridge the gap between high-level semantic decision-making of the upper layer and accurate motion control of the lower layer, realize lossless transmission of decision-making information, and improve the fine operation ability of robots in complex tasks, is a technical problem to be solved in the current robot technology field. SUMMARY

[0005] In view of the technical problems of decision-making information transmission distortion in the hierarchical control architecture of the robot and insufficient generalization ability for fine operation tasks in the prior art, the present application provides a robot fine operation method based on visual language model and semantic key point representation, and the technical solution is as follows:

[0006] Step 1, use GPT-4o to receive user input multi-modal instructions, call GPT-4o visual language model for direct semantic understanding; query candidate targets in the visual scene in combination with GroundingDINO, output detection results with confidence; construct a point cloud scene graph of hypergraph structure, and jointly infer it with the logical form output by the language model to obtain the final decision instruction;

[0007] Step 2, introduce the Bridge-Layer, extract and represent the final decision instruction, the extraction process uses a Transformer-based mapping network f_extract, which takes the embedding vectors of the target point cloud and the text instruction as input, and outputs a set of three-dimensional semantic key points;

[0008] Step 3, the Bridge-Layer references the geometric attention network g_predict to predict the target pose of the robot end effector, which receives the set of three-dimensional semantic key points and the set of target position key points and calculates all point pair relationships between the two sets of points, regresses a rotation matrix and a translation vector, and constitutes the target pose;

[0009] Step 4, pass the target pose to the kernel motion primitive algorithm to generate the expected motion trajectory;

[0010] Step 5, the robot performs operations according to the expected motion trajectory, continuously monitors external contact force and execution deviation through sensors and visual servo systems, and triggers the re-planning mechanism to return to step 1 when the external contact force exceeds the preset threshold or the execution deviation is too large.

[0011] Further, in the joint inference of step 1, the final decision instruction is obtained by maximizing the posterior probability, which is expanded according to Bayes' theorem as:

[0012] P(A,O,T|I,G)∝P(I|A,O,T,G)*P(A,O,T|G)

[0013] Where A is the operation, O is the target object, T is the target state, I is the text input, and G is the point cloud scene graph. The likelihood term P(I|A,O,T,G) evaluates the probability of observing the input instruction given the decision hypothesis, which is obtained by weighting the visual detection confidence and the language model score of the instruction. The prior term P(A,O,T|G) evaluates the physical reasonableness of the decision based on the point cloud scene graph G.

[0014] Further, after the joint inference in step 1, if there are multiple candidate targets, the current scene is an ambiguous scene, and an additional human-machine multi-round dialogue is performed for the ambiguous scene, calling a GPT-4o visual language model and sharing historical dialogue information to obtain retrieval information of the candidate targets, and through visual reasoning and inquiry of missing information inspired by humans, a unique target object is selected from multiple candidate targets, realizing task decision-making in ambiguous scenes.

[0015] Furthermore, in step 2, when the mapping network f_extract receives the target point cloud and the embedding vector of the text instruction, it uses a cross-attention mechanism to fuse multimodal information and generate a task-related scene representation modulated by the text instruction.

[0016] Furthermore, in step 2, the mapping network f_extract is trained using the loss function L_keypoint, which is a weighted sum of chamfer distance and bulldozer distance:

[0017] L_keypoint=λ_c*L_chamfer(P_obj,P_gt)+λ_e*L_emd(P_obj,P_gt)

[0018] Where P_gt is the set of real key points annotated manually, P_obj is the set of 3D semantic key points, λ_c and λ_e are weight coefficients, L_chamfer is the chamfer distance, and L_emd is the bulldozer distance.

[0019] Furthermore, in step 3, the geometric attention network g_predict is trained using a loss function L, the formula for which is as follows:

[0020] L=L_trans+γ*L_rot=||p_final-p_gt||^2+γ*(1-<q_final,q_gt> ^2)

[0021] Where L_trans represents the loss of the translation component, L_rot represents the loss of the rotation component, γ is the parameter for adjusting the weights, p_final is the translation vector of the target pose, q_final is the quaternion representing the rotation matrix of the target pose, and p_gt and q_gt are the actual translation vector and quaternion, respectively.

[0022] Furthermore, in step 4, the nucleated motion primitive algorithm simultaneously outputs the desired, time-varying impedance parameter profile and the desired contact force along a specific task direction. The robot executes the desired motion trajectory using a time-varying hybrid impedance / force control law, and its dynamic equation is expressed as:

[0023]

[0024] Where x is the target pose. For speed, Let Λ(x) be the acceleration, and Λ(x) be the inertia matrix in the operating space. Let g(x) be the Coriolis force and centripetal force matrix in the operating space, g(x) be the gravity vector in the operating space, F_ctrl be the control force, which is dynamically adjusted by the impedance parameter profile and the desired contact force, J(q)^(-T) be the transpose of the inverse of the Jacobian matrix from the joint space to the operating space, and F_ext be the actual external contact force.

[0025] Furthermore, in step 5, the process is continuously monitored through sensors and a visual servo system. An extended Kalman filter is used to fuse joint encoder readings, inertial measurement unit data, and visual markers to estimate the robot's end-effector Cartesian state with high accuracy. The state transition equation and observation equation of the extended Kalman filter are nonlinear functions as shown below:

[0026] x_k=f(x_{k-1},u_{k-1})+w_{k-1}

[0027] z_k=h(x_k)+v_k

[0028] Where k is the current time, x_k is the state vector at time k, x_{k-1} is the state vector at the previous time, u_{k-1} is the control input vector from time k-1 to time k, f(·) is the state transition function, w_{k-1} is the process noise; z_k is the observation vector at time k, h(·) is the observation function, and v_k is the observation noise.

[0029] Furthermore, in step 5, the monitoring process sets a dynamic safety threshold and defines a comprehensive anomaly score, Score_anomaly(t):

[0030] Score_anomaly(t)

[0031] =w_f||(IS(t))(F_ext-F_d(t))||^2+w_e||S(t)(x_d(t)-x(t))||^2

[0032] Where t is time, w_f and w_e are weight coefficients, I is the identity matrix, S(t) is the selection matrix, F_ext is the actual external contact force, F_d(t) is the desired force / torque vector, x_d(t) is the desired pose, and x(t) is the actual pose. When the integral of the comprehensive anomaly score Score_anomaly(t) exceeds the threshold within the time window Δt, it is determined to be an anomaly. At this time, step 1 is returned with the anomaly information, and the anomaly information is used as a new constraint to update the point cloud scene map.

[0033] This method also provides a robot fine-grained operating system based on visual language models and semantic key point representations, constructing a closed-loop multi-avatar agent system. When executed, the agent system implements any of the aforementioned robot fine-grained operation methods based on visual language models and semantic key point representations, including the following modules:

[0034] Multimodal decision module: Receives user multimodal commands, performs semantic understanding and queries candidate targets, constructs and infers scenario graphs, and parses the final decision command;

[0035] Bridge layer module: Extracts and represents the final decision instruction as a set of three-dimensional semantic key points, and predicts the termination parameters;

[0036] Trajectory generation module: Generates the desired motion trajectory using an improved kernelized motion primitive algorithm and enhances the local features of the start and end points;

[0037] Execution and Feedback Module: The robot performs operations and continuously monitors the external environment. When an anomaly occurs, it returns to the multimodal decision-making module, which provides feedback on the anomaly and replans the operation.

[0038] Beneficial effects:

[0039] By employing an innovative Bridge-Layer and 3D semantic keypoint representation, high-level abstract decisions are effectively transformed into precise and rich geometric and semantic information, overcoming the information distortion problem in traditional methods and achieving a seamless transition from "thinking" to "action." An improved kernelized motion primitive algorithm combined with a time-varying hybrid impedance / force control law enables the robot to adaptively adjust its stiffness and contact force when performing delicate tasks such as piercing, insertion, and alignment. This ensures high precision in critical stages while achieving smooth and safe interaction, significantly improving task success rate and robustness. Joint reasoning between scene graphs and multi-turn dialogues allows for a deeper understanding of complex scenes and ambiguous instructions, proactively eliminating ambiguity and greatly enhancing the system's intelligence level. Constructing an internal and external dual closed loop encompassing upper-level high-level semantic replanning and lower-level physical feedback enables the system to handle unexpected situations during execution and ensure the reliable completion of long-term complex tasks. Attached Figure Description

[0040] Figure 1 A detailed flowchart of a robot fine manipulation method based on visual language models and semantic key point representations;

[0041] Figure 2 This is a diagram of the overall architecture of a robot's sophisticated operating system based on visual language models and semantic key point representations. Detailed Implementation

[0042] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0043] like Figure 1 As shown, the robot fine manipulation method based on visual language model and semantic key point representation of the present invention includes the following steps:

[0044] 1. Scene graph construction and reasoning.

[0045] After receiving the instruction using GPT4-o, the system first uses GroundingDINO to detect objects in the scene, outputting a detection result B{(b_i,c_i,l_i)} with confidence, including bounding boxes, point clouds, and labels. The point cloud c_i and label l_i of each object are then used as nodes v_i in the point cloud scene graph G. The hyperedge e_j of the point cloud scene graph G is constructed by analyzing the spatial relationships (e.g., distance, orientation, containment) and semantic relationships (e.g., "on top of", "inside") between nodes. The point cloud scene graph G is then jointly reasoned with the logical form output by the language model. The final decision instruction {A,O,T} is obtained by maximizing the posterior probability argmaxP(A,O,T|I,G). This process combines the prior knowledge of the large language model with the physical constraints provided by the scene graph. This posterior probability can be expanded according to Bayes' theorem as follows:

[0046] P(A,O,T|I,G)∝P(I|A,O,T,G)*P(A,O,T|G)

[0047] The likelihood term P(I|A,O,T,G) evaluates the probability of observing the input instruction given the decision assumption, and is obtained by weighting the visual detection confidence score and the language model's score for the instruction. The prior term P(A,O,T|G) evaluates the physical rationality of the decision based on the point cloud scene graph G. For example, the prior probability of a "place" action is higher when the target object T has a stable plane.

[0048] For a single robot operation, if multiple candidate targets exist after joint reasoning, the current task scenario is ambiguous. To address this, additional multi-turn human-robot dialogue is conducted. The GPT4-o visual language model is invoked, and historical dialogue information is shared to obtain retrieval information for the aforementioned candidate targets. Through visual reasoning and querying for missing information heuristics from humans, a unique target object is selected from the multiple candidate targets, thus achieving task decision-making in ambiguous scenarios.

[0049] 2. Semantic key point extraction.

[0050] A Transformer-based encoder-decoder architecture is used as f_extract. The encoder receives point cloud c_i and CLIP embedding vectors of text instructions, fusing multimodal information through a cross-attention mechanism. The decoder takes a set of learnable query vectors as input and regresses the coordinates of n 3D keypoints. The loss function L_keypoint used to train this network employs a weighted sum of Chamfer Distance and Earth Mover's Distance to ensure the accuracy and reasonable distribution of the predicted point set.

[0051] L_keypoint=λ_c*L_chamfer(P_obj,P_gt)+λ_e*L_emd(P_obj,P_gt)

[0052] Where P_gt is the set of real key points annotated manually, P_obj is the set of 3D semantic key points, λ_c and λ_e are weight coefficients, L_chamfer is the chamfer distance, and L_emd is the bulldozer distance.

[0053] 3. Termination of parameter prediction.

[0054] The `g_predict` network is used, whose core is a geometric attention layer. This layer computes all point pair relationships between two sets of points and assigns an attention weight to each relationship, thus focusing on the keypoint pairs most important for predicting the target pose. The `g_predict` network receives the objects of operation—a set of 3D semantic keypoints `P_obj` and a set of target position keypoints `P_target`—and outputs a translation vector `p_final` for the target pose and a quaternion `q_final` (representing a rotation matrix). The loss function L for training this network is:

[0055] L=L_trans+γ*L_rot=||p_final-p_gt||^2+γ*(1-<q_final,q_gt> ^2)

[0056] Where L_trans represents the loss of the translation part, L_rot represents the loss of the rotation part, γ is the parameter for adjusting the weights, and p_gt and q_gt are the actual translation vector and quaternion, respectively.

[0057] 4. Refined trajectory generation and control.

[0058] Trajectories are generated using the Kernelized Motion Primitives (KMP) algorithm. The KMP base model extracts the desired baseline trajectory from multiple sets of human-taught trajectories {Ξ_m}_(m=1)^M using Gaussian Mixture Regression (GMR). The core of GMR is to calculate the state given the current time t. The conditional expectation. For each Gaussian component k, its weight h_k(t) is calculated as follows:

[0059] h_k(t)=(π_kN(t|μ_k^t,∑_k^t)) / (∑_jπ_jN(t|μ_j^t,∑_j^t))

[0060] Where t is the sample, k is the k-th Gaussian distribution component, π_k is the mixing coefficient, representing the prior weight of the k-th Gaussian component, N(t|μ_k^t,∑_k^t) is the probability density function of the k-th Gaussian component, μ_k^t is the mean of the k-th Gaussian component, ∑_k^t is the covariance matrix of the k-th Gaussian component, j is the summation index, traversing all Gaussian components, ∑_jπ_jN(t|μ_j^t,∑_j^t) represents the sum of the Gaussian components:

[0061] The expected trajectory is then the weighted sum of the expectations of each component:

[0062] x_d(t)=∑_kh_k(t)[μ_k^x+∑_k^{xt}(∑_k^{tt})^{-1}(t-μ_k^t)]

[0063] Where μ_k^x is the mean of the k-th component in the output space x, ∑_k^{xt}(∑_k^{tt})^{-1}(t-μ_k^t) is the linear correction term, ∑_k^{xt} is the cross-covariance matrix between x and t, (∑_k^{tt})^{-1} is the inverse of the input space covariance matrix, and t-μ_k^t is the difference between the input value t and the input mean of the k-th component.

[0064] The core improvement of this invention lies in the fact that the robot not only tracks positional trajectories but also tracks force trajectories when interaction with the environment is required, by designing the following time-varying hybrid impedance / force control law:

[0065]

[0066] Where x is the target pose. Let Λ(x) be the velocity, and Λ(x) be the inertia matrix of the operating space. Let g(x) be the Coriolis force and centripetal force matrix in the operating space, g(x) be the gravity vector in the operating space, and F_ctrl be the control force, dynamically adjusted by the impedance parameter profile and the desired contact force. The desired acceleration a_d is composed of the following formula:

[0067]

[0068] in, For acceleration, the impedance control part a_imp is:

[0069]

[0070] The force control part a_force is:

[0071] a_force=K_f(IS)∫(F_d(t)-F_ext)dt

[0072] D_d(t) is the damping matrix, K_d(t) is the stiffness matrix, and S is the diagonal selection matrix. Let x_d(t) be the desired velocity and pose. x(t) represents the actual velocity and pose, K_f is the force control integral gain matrix, I is the identity matrix, F_d(t) is the desired external contact force, and F_ext is the actual external contact force.

[0073] S = diag(s_i), s_i ∈ {0, 1}. When s_i equals 1, the i-th degree of freedom is subjected to impedance control; when s_i equals 0, integral force control is performed, so that the robot can automatically switch to the most suitable control strategy at different stages of the task (such as free space movement, contact, insertion).

[0074] 5. Condition monitoring and replanning.

[0075] An extended Kalman filter (EKF) is used to fuse joint encoder readings, IMU data, and visual fiducials to estimate the robot's end-effector Cartesian state with high accuracy. The state transition equations and observation equations of the EKF are nonlinear functions as shown below:

[0076] x_k=f(x_{k-1},u_{k-1})+w_{k-1}

[0077] z_k=h(x_k)+v_k

[0078] Where k is the current time, x_k is the state vector at time k, x_{k-1} is the state vector at the previous time, u_{k-1} is the control input vector from time k-1 to time k, f(·) is the state transition function, w_{k-1} is the process noise, z_k is the observation vector at time k, h(·) is the observation function, and v_k is the observation noise. Accurate tracking of the robot's state is achieved through iterative updates after linearization.

[0079] Set a dynamic safety threshold and define the comprehensive anomaly score Score_anomaly(t):

[0080] Score_anomaly(t)

[0081] =w_f||(IS(t))(F_ext-F_d(t))||^2+w_e||S(t)(x_d(t)-x(t))||^2

[0082] Where t is time, w_f and w_e are weighting coefficients, I is the identity matrix, S(t) is the selection matrix, F_ext is the actual external contact force, F_d(t) is the desired external contact force, x_d(t) is the desired pose, and x(t) is the actual pose. This score only considers the error under the corresponding control mode. When the integral of this score within the time window Δt exceeds the threshold Θ, it is judged as an anomaly. At this time, the scene graph construction and inference are returned with anomaly information (such as "encountering resistance exceeding F_d along the Z-axis near the target position"), and this information is used as a new constraint to update the point cloud scene graph G and perform replanning. This dual closed loop ensures strong adaptability to dynamic and uncertain environments.

[0083] The present invention relates to a sophisticated robot operating system based on visual language models and semantic key point representations. This system is a closed-loop multi-embodied agent system, and its overall framework is as follows: Figure 2 As shown, it consists of a multimodal decision module 100, a bridging layer module 200, a trajectory generation module 300, and an execution and feedback module 400.

[0084] Multimodal Decision Module 100: Starting from receiving multimodal instructions, the Langchain agent coordinates the work of this module, using GroundingDINO and GPT-4o to understand user intent, introducing ambiguity judgment and clearance, and ensuring that the robot and user's understanding of instructions are completely aligned through human-machine multi-turn dialogue and point cloud SceneGraph, reducing errors from the source.

[0085] Bridge Layer Module 200: The Bridge Layer translates high-level, abstract instructions into 3D semantic keypoint information that the lower layer can understand, and then predicts the termination parameters of the lower-level trajectory.

[0086] Trajectory generation module 300: Through an improved kernelized motion primitive algorithm, the trajectory is parameterized using a set of three-dimensional semantic key points, and local feature enhancement is performed on the start and end points, thereby generating a more accurate and environmentally adaptable refined operation trajectory.

[0087] Execution and Feedback Module 400: The robot executes the generated refined operation trajectory and continuously judges whether the task is completed. If it is not completed or an abnormality occurs, it performs closed-loop reproduction through sensor feedback, feeds the information back to the multimodal decision module 100, and restarts the entire system until the task is successful.

[0088] Through the above implementation methods, the present invention constructs a complete link from high-level semantic understanding to low-level fine control, effectively solving the shortcomings of the prior art and providing strong technical support for the application of robots in complex real-world environments.

[0089] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine manipulation of robots based on visual language models and semantic key point representations, characterized in that... Includes the following steps: Step 1: Use GPT-4o to receive multimodal commands input by the user, call the GPT-4o visual language model to directly perform semantic understanding; combine GroundingDINO to query candidate targets in the visual scene and output detection results with confidence; construct a point cloud scene graph with a hypergraph structure, and perform joint reasoning with the logical form output by the language model to obtain the final decision command; Step 2: Introduce a Bridge-Layer to extract and represent the final decision instruction. This extraction process uses a Transformer-based mapping network f_extract, which takes the target point cloud and the embedding vector of the text instruction as input and a three-dimensional semantic key point set as output. Step 3: The Bridge-Layer uses the geometric attention network g_predict to predict the target pose of the robot's end effector. The geometric attention network g_predict receives the set of 3D semantic key points and the set of target position key points and calculates all point pairs between the two sets of point points, regresses the rotation matrix and translation vector, and constitutes the target pose. Step 4: Pass the target pose to the kernelized motion primitive algorithm to generate the desired motion trajectory; Step 5: The robot performs operations according to the desired motion trajectory. It continuously monitors the external contact force and execution deviation through sensors and a visual servo system. When the external contact force exceeds a preset threshold or the execution deviation continues to be too large, a replanning mechanism is triggered to return to step 1.

2. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 1, during joint reasoning, the final decision instruction is obtained by maximizing the posterior probability, which, according to Bayes' theorem, is as follows: P(A,O,T|I,G)∝P(I|A,O,T,G)*P(A,O,T|G) Where A is the operation, O is the target object, T is the target state, I is the text input, and G is the point cloud scene graph. The likelihood term P(I|A,O,T,G) evaluates the probability of observing the input instruction under a given decision hypothesis, and is obtained by weighting the visual detection confidence and the language model's score of the instruction. The prior term P(A,O,T|G) evaluates the physical rationality of the decision based on the point cloud scene graph G.

3. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: After joint reasoning in step 1, if there are multiple candidate targets, the current scenario is an ambiguous scenario. For this ambiguous scenario, additional human-computer multi-turn dialogue is performed, the GPT-4o visual language model is invoked, and historical dialogue information is shared to obtain the retrieval information of the candidate targets. Through visual reasoning and querying for missing information heuristically from humans, a unique target object is selected from multiple candidate targets to achieve task decision-making in ambiguous scenarios.

4. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 2, when the mapping network f_extract receives the target point cloud and the embedding vector of the text instruction, it uses a cross-attention mechanism to fuse multimodal information and generate a task-related scene representation modulated by the text instruction.

5. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 2, the mapping network f_extract is trained using the loss function L_keypoint, which is a weighted sum of chamfer distance and bulldozer distance. L_keypoint=λ_c*L_chamfer(P_obj,P_gt)+λ_e*L_emd(P_obj,P_gt) Where P_gt is the set of real key points annotated manually, P_obj is the set of 3D semantic key points, λ_c and λ_e are weight coefficients, L_chamfer is the chamfer distance, and L_emd is the bulldozer distance.

6. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 3, the geometric attention network g_predict is trained using a loss function L, the formula for which is as follows: L=L_trans+γ*L_rot=||p_final-p_gt||^2+γ*(1-<q_final,q_gt> ^2) Where L_trans represents the loss of the translation component, L_rot represents the loss of the rotation component, γ is the parameter for adjusting the weights, p_final is the translation vector of the target pose, q_final is the quaternion representing the rotation matrix of the target pose, and p_gt and q_gt are the actual translation vector and quaternion, respectively.

7. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 4, the nucleated motion primitive algorithm simultaneously outputs the desired, time-varying impedance parameter profile and the desired contact force along a specific task direction. The robot executes the desired motion trajectory using a time-varying hybrid impedance / force control law, and its dynamic equation is expressed as: Where x is the target pose. For speed, Let Λ(x) be the acceleration, and Λ(x) be the inertia matrix in the operating space. Let g(x) be the Coriolis force and centripetal force matrix in the operating space, g(x) be the gravity vector in the operating space, F_ctrl be the control force, which is dynamically adjusted by the impedance parameter profile and the desired contact force, J(q)^(-T) be the transpose of the inverse of the Jacobian matrix from the joint space to the operating space, and F_ext be the actual external contact force.

8. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 5, the process is continuously monitored using sensors and a visual servo system. An extended Kalman filter is used to fuse joint encoder readings, inertial measurement unit data, and visual markers to estimate the robot's end effector Cartesian state with high accuracy. The state transition equation and observation equation of the extended Kalman filter are nonlinear functions as shown below: x_k=f(x_{k-1},u_{k-1})+w_{k-1} z_k=h(x_k)+v_k Where k is the current time, x_k is the state vector at time k, x_{k-1} is the state vector at the previous time, u_{k-1} is the control input vector from time k-1 to time k, f(·) is the state transition function, w_{k-1} is the process noise; z_k is the observation vector at time k, h(·) is the observation function, and v_k is the observation noise.

9. The robot fine manipulation method based on visual language model and semantic key point representation as described in claim 1, characterized in that: In step 5, the monitoring process sets a dynamic safety threshold and defines a comprehensive anomaly score, Score_anomaly(t): Score_anomaly(t)=w_f||(IS(t))(F_ext-F_d(t))||^2+w_e||S(t)(x_d(t)-x(t))||^2 Where t is time, w_f and w_e are weight coefficients, I is the identity matrix, S(t) is the selection matrix, F_ext is the actual external contact force, F_d(t) is the desired force / torque vector, x_d(t) is the desired pose, and x(t) is the actual pose. When the integral of the comprehensive anomaly score Score_anomaly(t) exceeds the threshold within the time window Δt, it is determined to be an anomaly. At this time, step 1 is returned with the anomaly information, and the anomaly information is used as a new constraint to update the point cloud scene map.

10. A sophisticated operating system for robots based on visual language models and semantic key point representations, characterized in that: A closed-loop multi-embodied agent system is constructed, which, when executed, implements the robot fine manipulation method based on visual language models and semantic key point representations as described in any one of claims 1 to 9, comprising the following modules: Multimodal decision module (100): Receives user multimodal instructions, performs semantic understanding and queries candidate targets, constructs and infers scenario graphs, and parses final decision instructions; Bridge layer module (200): Extracts and represents the final decision instruction as a set of three-dimensional semantic key points, and predicts the termination parameters; Trajectory generation module (300): Generates the desired motion trajectory using an improved kernelized motion primitive algorithm and enhances the local features of the start and end points; Execution and Feedback Module (400): The robot performs operations and continuously monitors the external environment. When an anomaly occurs, it returns to the multimodal decision module (100), provides feedback on the anomaly information, and replans.

Images