Method and system for realizing intelligent control of body based on VLM + Action fusion architecture

Abstract

The invention discloses a method and a system for realizing intelligent control on the basis of a VLM (Visual Language Model) + Actionality fusion architecture. The method aims at solving the problems that an existing end-to-end vision-language-action (VLA) model is scarce in data, high in hardware coupling degree, poor in controllability, weak in compatibility and the like. The core thought is to decouple high-level semantic understanding and bottom-level action execution, finish scene understanding and task planning through a VLM module, map high-level intentions into standardized API calling through an intention analysis and action calling module, execute specific actions through an Action module library packaging a traditional algorithm, and dispatch and monitor through an action execution engine. A large amount of end-to-end training data is not needed, the research and development cost is reduced, the system modularization, expandability and hardware universality are improved, the high delay problem is avoided, meanwhile, the controllability and safety of the control process are guaranteed, and the method is suitable for various intelligent scenes with bodies such as industrial manufacturing, medical assistance and family service.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to VLM (Vision-Language Model), Embodied AI, and robot motion control technology, and particularly to a method and system for embodied intelligent control based on a VLM+Action (visual language model + action) fusion architecture, which is suitable for realizing fusion control from visual understanding to non-grasping object manipulation. Background Technology

[0002] Currently, the cutting edge in the field of embodied intelligence is the development of end-to-end VLA (Vision-Language Model). These VLA models aim to construct a single, massive neural network that directly maps raw visual input (such as camera footage) and user natural language commands to a continuous sequence of underlying motion control (e.g., torque or speed commands for each joint of a robotic arm).

[0003] However, this purely end-to-end VLA paradigm faces many insurmountable bottlenecks in current engineering practices and commercialization:

[0004] 1. Data scarcity and high training costs: Training an effective end-to-end VLA model requires massive amounts of "visual-language-action" triplet data with precise action annotations. In reality, obtaining such high-quality annotated data is extremely difficult and costly. Publicly available open-source datasets have very limited sample sizes, far from sufficient to support the model's robust generalization ability, resulting in high R&D barriers and long development cycles.

[0005] 2. Strong coupling with specific hardware: The action sequences output by the end-to-end model are highly bound to the physical parameters (size, degrees of freedom, dynamic characteristics) of specific hardware devices (such as specific brands and models of robotic arms, cameras, and mobile chassis). Even minor changes to the hardware require extensive retraining or costly fine-tuning of the entire model, lacking versatility and portability.

[0006] 3. The "black box" problem and lack of controllability and security: The end-to-end model decision-making process is opaque, difficult to explain and debug. When the model produces unexpected or dangerous actions, it is difficult for external parties to effectively intervene and correct them. This uncontrollability is a fatal flaw in scenarios with strict requirements for safety and reliability, such as industrial manufacturing, medical assistance, and home services.

[0007] 4. Poor compatibility with mature algorithms: The robotics field has accumulated a wealth of mature, efficient, and proven traditional control algorithms over decades, such as visual servoing, motion planning (RRT, A*), and SLAM (Simultaneous Localization and Mapping). End-to-end models attempt to "replace" all these classic algorithms with a single neural network, which is not only difficult but also wastes existing technological achievements and makes it hard to effectively integrate these robust algorithm modules.

[0008] Therefore, constructing an embodied intelligent control architecture that can leverage the powerful understanding and reasoning capabilities of large language models, avoid the inherent defects of the end-to-end model, and is more efficient, modular, controllable, and easy to engineer has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] This invention aims to address the following core problems of existing end-to-end VLA models: strong dependence on large-scale, high-quality "vision-language-action" training data, resulting in high R&D costs and long development cycles; strong coupling between the model and specific hardware devices, leading to poor generalization ability and weak portability; lack of modular design in the system, making it difficult to integrate and reuse mature traditional control algorithms; and opaque and uncontrollable control process, making it difficult to guarantee the safety and reliability of operation.

[0010] To address the aforementioned issues, this invention proposes an innovative VLM+Action fusion architecture. The core idea is to decouple "high-level semantic understanding" from "low-level action execution," achieving fully controllable intelligent control with functionality equivalent to or even surpassing end-to-end VLA using a non-end-to-end technology stack. The specific technical solution is as follows:

[0011] The first objective of this invention is to provide a method for implementing embodied intelligent control based on a VLM+Action fusion architecture, comprising the following steps:

[0012] S1, High-level semantic understanding: Employs a pre-trained VLM (Visual Language Model) module to receive external visual information and user natural language instructions, and outputs a high-level description of action intent or decomposed task steps, wherein the description of action intent or task steps is natural language text or semi-structured text.

[0013] S2, Intent Resolution and Action Invocation: The Action-Call module receives and parses the output of the VLM module, mapping it to one or more standardized predefined Action API calls.

[0014] S3, Standardized motion skill execution: The motion execution engine receives the API call instruction, schedules hardware resources, and calls the corresponding Action module in the Action module library that is pre-encapsulated with underlying hardware operations and traditional robot algorithms;

[0015] S4, Action Execution Feedback: After the called Action module completes the task, it returns the execution status and result to the upper layer.

[0016] In one embodiment, the visual information includes real-time camera images and screenshots; the traditional robot algorithm includes one or more of the following: visual servoing algorithm, motion planning algorithm, force control algorithm, and SLAM (Simultaneous Localization and Mapping) algorithm.

[0017] In one embodiment, the Action-Call module implements intent parsing and API mapping through preset rules, keyword matching, or a lightweight intent recognition model.

[0018] In one embodiment, each Action module in the Action module library has a unified and well-defined API interface, forming an extensible system skill library. When adding new functions, only the new Action module needs to be developed and its API registered, and the upper-layer VLM module and core architecture do not need to be modified.

[0019] In one embodiment, the action execution engine is responsible for managing the lifecycle of each Action module, executing API call instructions in sequence, and monitoring the execution status.

[0020] The second objective of this invention is to provide a system for embodied intelligent control based on a VLM+Action fusion architecture, characterized by comprising:

[0021] VLM module: Used to receive external visual information and user natural language commands, perform scene understanding and task planning, and output high-level action intent descriptions or decomposed task steps;

[0022] 1) Action-Call module: Used to parse the output of the VLM module and map it to standardized predefined Action API calls;

[0023] 2) Action module library: Contains multiple independent modules pre-packaged with low-level hardware operations and traditional robot algorithms.

[0024] Action modules, each with a unified API interface;

[0025] 3) Action Execution Engine: Used to receive API call instructions, schedule hardware resources, call the corresponding Action module, manage the lifecycle of the Action module, and monitor the execution status;

[0026] 4) Hardware layer: This includes hardware devices such as robotic arms, cameras, and mobile chassis that perform actions, and receive control commands from the Action module to execute corresponding actions.

[0027] In one embodiment, the VLM module employs a large-scale multimodal model, either open-source or closed-source, including the Doubao multimodal model and the CogVLM model.

[0028] In one embodiment, the Action module in the Action module library includes a visual positioning module, a push posture alignment module, a visual servo push module, and a robotic arm reset module. The API of the visual positioning module is defined as find_object(object_name:str)->Coordinates, which is used to obtain the three-dimensional coordinates of the object in the robot coordinate system. The API of the push posture alignment module is defined as align_for_push(target_coordinates:Coordinates,push_direction_vector:Vector)->bool, which is used to move the robotic arm end effector to a suitable position and posture behind the target object. The API of the visual servo push module is defined as visual_servo_push(target_coordinates:Coordinates,end_coordinates:Coordinates,force_limit:float)->bool, which is used to push the object according to a predetermined trajectory and control the pushing force. The API of the robotic arm reset module is defined as retract_arm(to_home_position:bool), which is used to control the robotic arm to return to a preset position.

[0029] In one embodiment, when the hardware layer replaces the hardware device, only the underlying Action module needs to be rewritten or adapted, while the upper-layer VLM module and core business logic remain unchanged.

[0030] In one embodiment, the system further includes a human-computer interaction module for receiving user input commands and providing feedback on task execution results to the user, wherein the feedback methods include voice feedback and text feedback.

[0031] Compared with existing technologies, the VLM+Action fusion architecture of this invention brings revolutionary advantages:

[0032] 1. Achieve training-free end-to-end intelligent control: By cleverly utilizing the powerful and general understanding capabilities of the mature VLM, actions are executed through deterministic API calls, completely avoiding end-to-end model training for "vision to action". This fundamentally solves the problem of data scarcity, significantly reduces R&D costs, and shortens the R&D cycle from months or even years to weeks.

[0033] 2. Superior Modularity and Scalability: Complex servo algorithms are encapsulated into standardized Action modules, enabling "plug-and-play" skill expansion. When adding new system functions (such as adding the "wiping the table" skill), only the new Action module needs to be developed and its API registered. The upper-layer VLM model and core architecture do not require any changes, significantly improving the system's modularity and scalability.

[0034] 3. Solving the high latency problem: Although it is not an end-to-end architecture, its working mode is fundamentally different from traditional multi-turn query end-to-end systems. VLM is only responsible for high-level, one-time task planning, while the specific service processes that require real-time feedback and high-frequency control are completely encapsulated locally and executed in a closed loop within the efficient Action module. This avoids the high latency problem caused by repeatedly querying large cloud models during action execution, ensuring the smoothness and real-time performance of actions.

[0035] 4. High hardware versatility and portability: VLM is completely decoupled from specific hardware implementations. When changing hardware devices (such as changing from brand A robotic arm to brand B), only the underlying Action module (i.e. the specific implementation of the API) needs to be rewritten or adapted. The upper-layer VLM and core business logic do not need to be changed, enabling the system to adapt to different hardware platforms quickly and at low cost.

[0036] 5. Extremely high controllability, reliability, and debuggability: Each Action module is a functionally defined, fully tested "white-box" unit with predictable and reliable behavior. When system problems occur, the Action module that failed to execute can be easily located for targeted debugging. This determinism provides a solid guarantee for system security and reliability. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the framework of a method and system for implementing embodied intelligent control based on a VLM+Action fusion architecture in one embodiment of the present invention;

[0039] Figure label:

[0040] 110. VLM (Visual Language Model) module; 120. Action-Call (Intent Resolution and Action Invocation) module; 130. Action library; 140. Action execution engine; 131. Visual positioning module; 132. Push alignment module; 133. Servo push module; 134. Robotic arm reset module. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly set on the other component; when a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to the other component.

[0043] Furthermore, in the description of this application, "multiple" or "several" means two or more, unless otherwise explicitly specified.

[0044] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which this application can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size should still fall within the scope of the technical content disclosed in this application, provided that they do not affect the effects and purposes that this application can produce.

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The core idea of ​​this invention is to construct a decoupled, modular embodied intelligent control system. The following description uses a robotic arm lamp robot (hereinafter referred to as "lamp robot") performing the task of "Please push the water cup on the table onto the coaster on the right" as an example.

[0046] Example 1

[0047] 1. Detailed System Architecture Explanation

[0048] System architecture of the present invention (see appendix) Figure 1 It mainly includes the following core modules, each of which is labeled in the attached diagram:

[0049] 1.1 VLM (Visual Language Model) Module (Figure 110)

[0050] Role: The system's "cognitive brain".

[0051] Composition: Select mature open-source or closed-source large-scale multimodal models in the industry, such as Doubao Multimodal, CogVLM, etc.

[0052] Functions: The system receives real-time images of the desktop captured by the robot's camera (external visual information), as well as natural language commands from the user ("Please push the water glass on the table to the coaster on the right"). It understands the objects in the scene (water glass, coaster) and the user's intent, performs scene understanding and task planning, and outputs a high-level task plan (a description of the action intent or decomposed task steps). In practical applications, external visual information may include real-time camera images, screenshots, etc.

[0053] 1.2 Action-Call (Intent Resolution and Action Invocation) Module (120)

[0054] Role: The "nerve center" or "translator" connecting the "brain" and "limbs".

[0055] Function: Receives high-level task planning (such as a text describing steps) from the VLM module (110), parses and translates it into a series of precise call instructions to the underlying standardized action API (i.e., mapped to standardized predefined Action API calls). The Action-Call module can achieve intent parsing and API mapping through preset rules, keyword matching, or a lightweight intent recognition model.

[0056] 1.3 Action Module Library (130)

[0057] Role: The "skill toolbox" of the lamp robot.

[0058] Composition: This invention comprises multiple independent Action modules pre-encapsulated with underlying hardware operations and traditional robot algorithms. Each Action module has a unified API interface, and each module represents a specific skill. Providing services through a unified and well-defined API interface is one of the key innovations of this invention: the API interface constitutes an extensible system "skill library." When adding new functions, only the development of a new Action module and registration of its API are required; the upper-layer VLM module and core architecture remain unchanged, achieving complex algorithm encapsulation and plug-and-play functionality. The aforementioned traditional robot algorithms include one or more of the following: visual servoing algorithms, motion planning algorithms, force control algorithms, and Simultaneous Localization and Mapping (SLAM) algorithms.

[0059] Example of the Action module for this case:

[0060] ·131. Visual positioning module (find_object):

[0061] ·API definition: find_object(object_name:str)->Coordinates

[0062] • Function: Receives an object name (such as "water cup" or "cup coaster"), locates the object in the camera's field of view using a built-in object detection algorithm, and returns its 3D coordinates in the robot coordinate system.

[0063] 132. Push Attitude Alignment Module (align_for_push):

[0064] ·API definition: align_for_push(target_coordinates:Coordinates,push_direction_vector:Vector)->bool

[0065] • Function: Receives the coordinates of the target object and the desired pushing direction vector. This module encapsulates inverse kinematics and path planning algorithms, responsible for moving the end effector of the robotic arm (e.g., the lamp head of a desk lamp or a specific contact point on the arm) to a suitable starting position and posture behind the target object, ready to begin pushing.

[0066] ·133. Visual Servo Push Module (visual_servo_push):

[0067] ·API definition: visual_servo_push(target_coordinates:Coordinates,end_coordinates:Coordinates,force_limit:float)->bool

[0068] • Function: Receives the current coordinates of the target object, the coordinates of the endpoint of the push, and a force limit threshold. This module is the core motion module, internally encapsulating complex visual servoing and force control algorithms. It will:

[0069] a. Use a camera to track the position of the pushed object in real time.

[0070] b. Based on the deviation between the object's current position and the target path, adjust the pushing speed and direction of the robotic arm in real time to ensure that the object moves along the predetermined trajectory.

[0071] c. Use a force sensor at the end to monitor the pushing force and ensure that the force does not exceed the force_limit to prevent the cup from being knocked over or damaged.

[0072] ·134. Robotic Arm Reset Module (retract_arm):

[0073] API definition: retract_arm(to_home_position:bool)

[0074] • Function: Controls the robotic arm to safely retract to a preset initial or safe position.

[0075] 1.4 Motion Execution Engine (140)

[0076] Roles: The "scheduler" and "supervisor" of the mission.

[0077] Function: The action execution engine is responsible for managing the lifecycle of each Action module, receiving and executing API call instructions from the Action-Call module (120) in sequence, and monitoring the execution status.

[0078] 2. Workflow Details: Pushing the Water Cup Task

[0079] The following is a detailed breakdown of the complete process of the desk lamp robot performing the task "Please push the water glass on the table onto the coaster on the right":

[0080] 2.1 Step 1 (Perception and Planning)

[0081] The lamp robot's camera captures images of the desktop (external visual information), which, along with the user's voice or text commands (user's natural language commands), are sent to the VLM module (110).

[0082] 2.2 Step 2 (High-level plan output)

[0083] After analysis by the VLM module (110), the task objective is understood, and the decomposed high-level plan (action intent description or decomposed task steps, which can be natural language text or semi-structured text) is output. The plan can be in the following JSON format:

[0084]

[0085]

[0086] 2.3 Step 3 (Intent Resolution and API Mapping)

[0087] The Action-Call module (120) receives the above JSON plan, parses it line by line, and translates it into one or more standardized, precisely executable, predefined Action API call sequences:

[0088] i.cup_coords=find_object(object_name="water_cup")

[0089] ii.coaster_coords=find_object(object_name="coaster")

[0090] iii.push_vector=calculate_direction(from=cup_coords,to=coaster_coords)

[0091] iv.align_success=align_for_push(target_coordinates=cup_coords, push_direction_vector=push_vector)

[0092] v.push_success = visual_servo_push(target_coordinates = cup_coords, end_coordinates = coaster_coords, force_limit = 5.0) (assuming the force limit is 5 Newtons)

[0093] vi.retract_arm(to_home_position=True)

[0094] The Action module library mentioned above includes a visual positioning module, a push posture alignment module, a visual servo push module, and a robotic arm reset module. The API definition of the visual positioning module is find_object(object_name:str)->Coordinates, which is used to obtain the three-dimensional coordinates of the object in the robot coordinate system. The API definition of the push posture alignment module is align_for_push(target_coordinates:Coordinates,push_direction_vector:Vector)->bool, which is used to move the robotic arm end effector to a suitable position and posture behind the target object. The API definition of the visual servo push module is visual_servo_push(target_coordinates:Coordinates,end_coordinates:Coordinates,force_limit:float)->bool, which is used to push the object according to a predetermined trajectory and control the pushing force. The API definition of the robotic arm reset module is retract_arm(to_home_position:bool), which is used to control the robotic arm to return to a preset position.

[0095] 2.4 Step 4 (API Call and Execution):

[0096] The action execution engine (140) begins to call the APIs in the Action library (130) in sequence:

[0097] i. Call find_object twice: the visual positioning module (131) finds the cup and coaster respectively and returns their coordinates.

[0098] ii. Calculate the push direction: The Action-Call module (120) or an auxiliary module calculates the push direction vector push_vector based on the coordinates of the start and end points.

[0099] iii. Call align_for_push: Start the alignment module (132) to move the robotic arm behind the water cup in preparation for pushing.

[0100] iv. Calling visual_servo_push: This is the core execution step. The visual servo push module (133) is activated. The robotic arm of the lamp robot begins to smoothly push the water cup, the camera continuously tracks the position of the water cup, and the servo algorithm continuously corrects the trajectory of the arm based on visual feedback to ensure that the water cup is accurately pushed to the center of the coaster.

[0101] v. Calling retract_arm: The robotic arm reset module (134) is started, and the robotic arm is retracted to its initial position.

[0102] 2.5 Step 5 (Task Completed):

[0103] All API calls were executed successfully. The action execution engine (140) reports task completion to the upper layer. The lamp robot may respond via voice (implemented by another Action module): "Okay, I've pushed the cup onto the coaster."

[0104] This example demonstrates that the architecture of this invention decomposes a complex physical manipulation task into "high-level planning of the VLM" and "a series of localized, deterministic calls to Action modules that encapsulate complex algorithms." This approach leverages the intelligence of the large model while ensuring high efficiency, reliability, and controllability in the execution process. Preferably, when hardware devices are replaced at the hardware layer, only the underlying Action modules need to be rewritten or adapted, while the upper-layer VLM modules and core business logic remain unchanged. The system may also include a human-computer interaction module for receiving user input commands and providing feedback on task execution results, including voice feedback, text feedback, etc.

[0105] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for realizing embodied intelligent control based on a VLM+Action fusion architecture, characterized in that, Comprising the following steps: S1, high-level semantic understanding: using a pre-trained VLM module, receiving external visual information and user natural language instructions, outputting high-level action intent description or decomposed task steps, the action intent description or task steps being natural language text or semi-structured text; S2, intent analysis and action calling: through the Action-Call module, receiving and analyzing the content output by the VLM module, mapping it to one or more standardized pre-defined Action API calls; S3, standardized action skill execution: the action execution engine receives the Action API call instructions, schedules hardware resources, and calls the corresponding Action module in the Action module library pre-encapsulated with underlying hardware operations and traditional robot algorithms; S4, action execution feedback: after the called Action module completes the task, it returns the execution status and results to the upper layer.

2. The method of claim 1, wherein, The visual information includes real-time camera images and screenshots; the traditional robot algorithms include one or more of visual servoing algorithms, motion planning algorithms, force control algorithms, and SLAM algorithms.

3. The method of claim 1, wherein, The Action-Call module realizes intent analysis and API mapping through preset rules, keyword matching, or lightweight intent recognition models.

4. The method of claim 1, wherein, Each Action module in the Action module library has a unified API interface, forming an extensible system skill library.

5. The method of claim 1, wherein, The action execution engine is responsible for managing the life cycle of each Action module, executing API call instructions in order, and monitoring execution status.

6. A system for realizing embodied intelligent control based on VLM+Action fusion architecture, characterized in that, Comprising: VLM module: for receiving external visual information and user natural language instructions, scene understanding and task planning, Output high-level action intent description or decomposed task steps; Action-Call module: for analyzing the content output by the VLM module, mapping it to a standardized pre-defined Action API call; Action module library: contains multiple independent Action modules pre-encapsulated with underlying hardware operations and traditional robot algorithms, each Action module has a unified API interface; Action execution engine: for receiving API call instructions, scheduling hardware resources, calling corresponding Action modules, managing Action module life cycle and monitoring execution status; Hardware layer: including mechanical arm, camera, mobile chassis and other hardware devices for executing actions, receiving control instructions from Action modules to execute corresponding actions.

7. The system of claim 6, wherein, The VLM module uses open-source or closed-source large multi-modal models, including the beanbag multi-modal model and the CogVLM model.

8. The system of claim 6, wherein, The Action modules in the Action module library include a visual positioning module, a push posture alignment module, a visual servo push module, and a robot arm reset module, the API definition of the visual positioning module is find_object(object_name:str)->Coordinates, which is used to obtain the three-dimensional coordinates of an object in the robot coordinate system; the API definition of the push posture alignment module is align_for_push(target_coordinates:Coordinates,push_direction_vector:Vector)->bool, which is used to move the robot arm end effector to a suitable position and posture behind the target object; the API definition of the visual servo push module is visual_servo_push(target_coordinates:Coordinates,end_coordinates:Coordinates,force_limit:float)->bool, which is used to push an object along a predetermined trajectory and control the pushing force; the API definition of the robot arm reset module is retract_arm(to_home_position:bool), which is used to control the robot arm to return to a preset position. When the hardware layer is replaced with a hardware device, only the underlying Action module needs to be rewritten or adapted, and the upper VLM module and the core business logic remain unchanged.

9. The system of claim 6, wherein, The human-computer interaction module is also included, which is used to receive user instruction input and feed back task execution results to the user, and the feedback methods include voice feedback and text feedback.

10. The system of claim 6, wherein, ​

Images