A robot control method, system, electronic device and readable storage medium

Abstract

This application discloses a robot control method and system. The method includes: extracting visual features from the current environmental state information based on a pre-trained visual language model, and converting the natural language task description into a semantic feature representation; mapping the fused features obtained by fusing visual and semantic features to the action space through an adapter unit to obtain action control commands; controlling the robot to execute the actions corresponding to the action control commands and obtaining execution feedback information; calculating a reward value based on the feedback information, and updating the action generation strategy based on the reward value through reinforcement learning, optimizing the parameters of the adapter unit during the update process while keeping the main parameters of the visual language model unchanged; repeating the process until the action generation strategy meets the preset convergence conditions. This application can effectively improve the action generation accuracy and generalization ability of robots in complex tasks.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to a robot control method, system, resistive device, and computer-readable storage medium. Background Technology

[0002] With the deep integration of artificial intelligence and robotics, enabling robots to understand natural language instructions and autonomously perform tasks in complex environments has become an important research direction in the field of embodied intelligence. Currently, robot control technology is evolving from traditional methods based on precise modeling and trajectory planning to data-driven intelligent decision-making and control methods, and end-to-end control frameworks integrating visual perception and language understanding are gradually becoming a research hotspot.

[0003] In existing technologies, a mainstream approach is imitation learning based on behavioral cloning. This method trains a neural network model by collecting demonstration data of human actions to achieve a direct mapping from perceptual input to action output. Such methods perform well in structured tasks, but their performance is highly dependent on the coverage and quality of the demonstration data. When faced with unseen scenarios or task changes, the model's generalization ability is limited, and this method lacks a mechanism for policy correction using environmental feedback.

[0004] Another approach attempts to incorporate large-scale pre-trained models, particularly visual language models, to enhance the robot's semantic understanding of complex instructions and environments. By encoding images and text into a unified semantic space, the model can achieve cross-modal alignment and support semantic-based task reasoning. However, the output of such models is typically a semantic representation, which is difficult to directly translate into the continuous and precise control instructions required by the robot's execution layer. Furthermore, the large number of model parameters results in high computational costs when tuning for specific tasks, limiting deployment efficiency.

[0005] In addition, some studies have explored autonomous optimization methods based on reinforcement learning. This method involves the robot repeatedly interacting with its environment and gradually updating its control strategy based on pre-set reward signals. While this method theoretically possesses self-optimization capabilities, its sample efficiency is low under high-dimensional perceptual inputs and complex task settings, requiring a large amount of interactive trial-and-error data. Furthermore, it is prone to policy oscillations or performance degradation during training, and is highly dependent on the quality of the initial strategy, thus limiting its application in real-world robotic systems.

[0006] In summary, existing robot control technologies have their own limitations in terms of the connection between semantic understanding and action execution, the self-optimization capability of strategies, and the efficiency of models in actual deployment, which restrict the overall performance of robots in complex tasks. Summary of the Invention

[0007] To address the aforementioned problems, the purpose of this application is to provide a robot control method and system. This solution can effectively improve the robot's motion generation accuracy and generalization ability in complex tasks without increasing excessive computing resource consumption, and has good industrial application prospects and promotion value.

[0008] To achieve the above objectives: In a first aspect, embodiments of this application provide a robot control method, the control method comprising: Obtain current environmental status information and natural language task description; Based on a pre-trained visual language model, visual features are extracted from the acquired current environment state information, and the received natural language task description is converted into a semantic feature representation. The fused features, obtained by fusing the visual features and the semantic features, are mapped to the action space through an adapter unit to obtain action control instructions; wherein, the adapter unit is a lightweight network structure inserted into the visual language model, serving as a carrier of the action generation strategy; Control the robot to execute the actions corresponding to the action control commands, and obtain execution feedback information; The reward value is calculated based on the feedback information, and the action generation strategy is updated by reinforcement learning based on the reward value with the goal of maximizing the cumulative reward. During the update process, the parameters of the adapter unit are optimized while keeping the main parameters of the visual language model unchanged. Repeat the steps from acquiring environmental state information to updating the action generation strategy until the action generation strategy meets the preset convergence condition.

[0009] Optionally, the adapter unit includes at least one parameter-efficient fine-tuning structure selected from the following: a low-rank adapter structure, a prefix adjustment structure, or a serial adapter structure.

[0010] Optionally, the visual language model includes a visual encoding unit and a text encoding unit; The visual features are extracted from the environmental state information by the visual encoding unit, and the environmental state information includes at least one of RGB image information and depth information. The semantic feature representation is obtained by encoding the natural language task description through the text encoding unit.

[0011] Optionally, the step of updating the action generation strategy by reinforcement learning based on the reward value with the goal of maximizing the cumulative reward is performed using a proximal policy optimization algorithm and / or a policy gradient algorithm.

[0012] Optionally, the reward value is calculated based on the task execution result, and the calculation items of the reward value include at least one of the following: success reward item, distance constraint item, and action smoothness constraint item.

[0013] Optionally, when outputting motion control commands, the range of the motion control commands is also limited to constrain the range of motion of the robot during execution.

[0014] Secondly, embodiments of this application provide a control system for a robot, the system comprising: The sensing module is used to acquire current environmental status information; The instruction processing module is used to receive a natural language task description and convert the natural language task description into a semantic feature representation; The visual language model processing module includes a pre-trained visual language model and an adapter unit inserted therein. The pre-trained visual language model is used to extract visual features from the environmental state information and align the visual features with the language features to a unified semantic space for fusion. The adapter unit, as a carrier of the action generation strategy, is used to map the fused visual features and semantic features to the action space and output action control commands. The control execution module is used to control the robot to perform corresponding actions according to the action control instructions, and to obtain feedback information during the execution process; The reinforcement learning optimization module is used to calculate the reward value based on the feedback information, and with the goal of maximizing the cumulative reward, drive the action generation strategy to perform reinforcement learning updates based on the reward value. During the update process, the parameters of the adapter unit are optimized while keeping the main parameters of the visual language model unchanged.

[0015] Optionally, the control execution module includes: A motion planning unit is used to generate an execution trajectory based on the motion control command; The actuator control unit is used to control the robot actuator to complete the corresponding operation; The feedback unit is used to output feedback on the robot's execution status.

[0016] Optionally, the reinforcement learning optimization module includes: A reward calculation unit is used to calculate a reward value based on the feedback information. The calculation items of the reward value include at least one of a success reward item, a distance constraint item, and a motion smoothness constraint item. A strategy evaluation unit is used to evaluate the current action generation strategy based on the reward value; The parameter update unit is used to update the parameters of the action generation strategy based on the strategy evaluation results, with the goal of maximizing the cumulative reward, and optimize the parameters of the adapter unit during the update process.

[0017] Optionally, the control system is deployed in a host computer computing unit; The host computer computing unit is connected to the robot controller and sensor module through a communication interface; The sensor module includes an RGB camera and / or a depth camera, used to collect the environmental state information; The robot controller is used to receive and parse the motion control commands output by the control system in order to drive the robot actuator to complete the corresponding operation.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing a computer program, wherein when the processor runs the computer program, it implements the steps of the robot control method described above.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the robot control method described above.

[0020] The present invention, through the above technical solution, has at least the following beneficial effects: The technical solution provided in this application constructs a robot control system based on a pre-trained visual language model and adapter units, and introduces a closed-loop reinforcement learning mechanism for policy optimization. This solution can effectively improve the robot's motion generation accuracy and generalization ability in complex tasks without increasing excessive computational resource consumption, and has good prospects for industrial application and promotion value. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0022] Figure 1 This is a flowchart illustrating the robot control method provided in an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the control system of the robot provided in an embodiment of this application.

[0024] Figure 3 This is a hardware system structure diagram of the robot provided in an embodiment of this application.

[0025] The objectives, functional features, and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0028] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, may be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., as used in this application, may be interpreted as inclusive, or mean any one or any combination thereof. For example, "including at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0029] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0030] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0031] It should be noted that step designations such as S1 are used in this paper for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial restriction on the order.

[0032] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0033] Example 1: refer to Figure 1 As shown, the robot control method provided in this application includes: A method for controlling a robot, characterized in that the control method includes: S1. Obtain current environmental status information and natural language task description.

[0034] In the embodiments of this application, the robot acquires current environmental state information through its perception module (such as an RGB camera, depth camera, LiDAR, etc.). This environmental state information includes, but is not limited to, RGB image information and depth information. The perception module may include an RGB image acquisition unit and a depth information acquisition unit, used to acquire RGB color images of the work scene and three-dimensional spatial distance data, respectively. Simultaneously, it receives natural language task descriptions input by the user, such as "Place the red cup in the upper right corner of the table" or "Pick up the small cube in front of you and put it in the box on the left."

[0035] It should be understood that the environmental state information in this embodiment is not limited to static images, but may also include time-series video streams or multimodal sensor fusion data to provide dynamic environmental perception.

[0036] S2. Based on a pre-trained visual language model, visual features are extracted from the acquired current environment state information, and the received natural language task description is converted into a semantic feature representation.

[0037] In the embodiments of this application, the environmental state information and natural language task description obtained in step S1 are input into a pre-trained visual language model for processing.

[0038] Specifically, the visual language model includes a visual encoding unit and a text encoding unit. The visual encoding unit extracts features from the input environmental state information (such as an RGB image) to obtain visual features. In embodiments of this application, the visual encoding unit can be implemented using a convolutional neural network structure or a Transformer structure based on an attention mechanism. The text encoding unit encodes the natural language task description, converting it into a semantic feature representation. In this embodiment, the visual language model is pre-trained on large-scale data, enabling it to align visual and linguistic information into a unified semantic space.

[0039] In a limited embodiment, the main parameters of the visual language model are frozen during training and inference to preserve its general visual-language alignment capabilities acquired through pre-training on massive amounts of data and prevent catastrophic forgetting.

[0040] S3. The fused feature obtained by fusing the visual feature and the semantic feature is mapped to the action space through the adapter unit to obtain the action control command.

[0041] In the embodiments of this application, the visual features and semantic features obtained in step S2 are fused to obtain fused features. This fusion process can be implemented by a multimodal fusion unit, for example, by using a cross-modal attention mechanism to interactively fuse visual features and semantic features.

[0042] Furthermore, the fused features are input to the adapter unit, which maps the fused features to the action space and outputs action control commands.

[0043] It is important to note that this adapter unit is a lightweight network structure inserted into the visual language model (e.g., located between layers of a Transformer, or inserted as a prefix), and serves as the physical carrier of the action generation strategy. It is responsible for mapping high-dimensional fused features to the robot's motion space and outputting specific motion control commands (such as the displacement velocity of the robotic arm's end effector, the gripper's opening and closing angle, etc.). In a preferred embodiment, the adapter unit can be implemented using a parameter-efficient fine-tuning structure, including but not limited to low-rank adaptation structures (such as LoRA), prefix adjustment structures, and serial adapter structures. Since the number of parameters in the adapter unit is much smaller than that in the main visual language model, subsequent strategy optimization only needs to operate on this small portion of parameters.

[0044] In this embodiment, the motion control commands may include at least one of the following: robotic arm end-effector pose control commands, joint angle control commands, and gripper opening / closing degree control commands. The motion space may be a continuous motion space and / or a discrete motion space.

[0045] In a preferred embodiment, the motion control commands output by the adapter unit also need to undergo range limiting processing, that is, the range of motion changes in each degree of freedom direction is limited to constrain the motion amplitude during robot execution and prevent dangerous or abnormal motion situations.

[0046] S4. Control the robot to execute the action corresponding to the action control command and obtain execution feedback information.

[0047] In the embodiments of this application, the robot is controlled to perform corresponding actions according to the motion control instructions generated in step S3. Specifically, firstly, an execution trajectory is generated according to the motion control instructions, and then the robot's actuators (such as robotic arms and end effectors) are controlled to complete the corresponding operation actions, such as grasping, transporting, or assembling.

[0048] During the robot's actions, sensor modules installed on the robot's end monitor environmental changes and the robot's state in real time, acquiring feedback information during the execution process. This feedback information includes, but is not limited to, task execution result information (such as whether the target object was successfully grasped), state change information (such as the change in the position of the robotic arm's end effector), and sensor feedback information (such as readings from force sensors and tactile sensors).

[0049] S5. Calculate the reward value based on the feedback information.

[0050] In the embodiments of this application, after obtaining the feedback information of the robot in the process of executing motion control commands, the reward value is further calculated using a preset reward function. The reward value calculation items include success reward items (whether the task is completed), distance constraint items (such as the distance between the end effector and the target object), and motion smoothness constraint items (penalty for severe jitter).

[0051] Specifically, the success reward term is used to characterize whether the task is successfully completed. For example, a positive reward is given when the robot successfully places the target object in the designated position. The distance constraint term is used to measure the change in distance between the robot's end effector and the target position. A positive reward is given when the distance decreases. The motion smoothness constraint term is used to constrain the smoothness of the robot's motion. A negative penalty is given for excessive acceleration or sudden stop and start behavior.

[0052] S6. The action generation strategy is updated by reinforcement learning based on the reward value.

[0053] In the embodiments of this application, after calculating the reward value through feedback information, a policy network and a value network are further constructed, and an advantage function (such as a generalized advantage estimation function) is calculated based on the reward value. The policy gradient is calculated using the Proximal Policy Optimization (PPO) algorithm or the Policy Gradient algorithm, and the parameters of the policy network are updated according to the policy gradient. In this embodiment, during the parameter update process, only the parameters of the adapter unit are subjected to gradient backpropagation and optimization, keeping the main parameters of the visual language model unchanged. Preferably, the Proximal Policy Optimization (PPO) algorithm or the Policy Gradient algorithm is used for training. This mechanism ensures that the rich prior knowledge already learned by the visual language model is fully preserved, preventing catastrophic forgetting due to the exploration process of reinforcement learning. Simultaneously, since only the adapter unit with a very small number of parameters needs to be updated, the computational overhead is significantly reduced, making online learning on real robot systems possible. It should be understood that by fine-tuning only the adapter, the system can allow the robot to learn fine operational skills for specific tasks through trial and error without compromising the general capabilities of the visual language model.

[0054] S7. Repeat the steps from obtaining environmental state information to updating the action generation strategy until the action generation strategy meets the preset convergence condition.

[0055] In the embodiments of this application, steps S1 to S6 are repeated to form a closed-loop control process of "perception → understanding → mapping → execution → feedback → evaluation → update".

[0056] In each iteration, the robot executes actions according to the current strategy, obtains environmental feedback, and continuously optimizes the parameters of the adapter unit based on the feedback. After multiple iterations, training stops when the action generation strategy meets the preset convergence condition.

[0057] In this embodiment, the preset convergence condition may include at least one of the following: the average reward value changes less than a preset threshold over multiple consecutive training rounds; the task success rate continuously reaches a preset target value (e.g., the success rate reaches more than 95% in 50 consecutive evaluations); the update magnitude of the strategy parameters is continuously lower than a preset lower limit; or the preset maximum number of training rounds is reached.

[0058] In a preferred embodiment, before executing the aforementioned reinforcement learning loop, the action generation strategy (i.e., the parameters of the adapter unit) can be pre-trained using collected robot operator expert demonstration data to obtain an initial strategy with basic behavior cloning capabilities. Then, this initial strategy is used as the starting point for reinforcement learning, proceeding to the aforementioned loop for further optimization. This two-stage training method of "supervised learning pre-training + reinforcement learning optimization" enables the strategy to have a good initialization foundation, significantly accelerating the convergence speed of reinforcement learning.

[0059] In the embodiments of this application, the adapter unit is a key structure for realizing "semantic-action mapping" and also the physical carrier of the action generation strategy. This adapter unit is inserted into the pre-trained visual language model and serves as an additional lightweight network module outside the main structure of the visual language model. Its parameters specifically refer to all learnable connection weights and bias terms contained within this lightweight network module. Depending on the actual application scenario and computing resources, the adapter unit can be implemented in, but is not limited to, the following structural forms.

[0060] In a preferred embodiment, the adapter unit employs a bottleneck structure. This structure consists of a dimensionality reduction layer, a nonlinear activation layer, and a dimensionality increase layer connected in series. Dimensionality reduction layer: contains the first weight matrix ∈ This is used to map high-dimensional fusion features (dimension D, e.g., D=4096) into a low-dimensional latent space (dimension d, e.g., d=64, satisfying d...). D). The dimensionality reduction layer may also contain corresponding bias terms. .

[0061] Nonlinear activation layer: This layer acts on the output of the dimensionality reduction layer to introduce nonlinear transformation capabilities. The activation function can be ReLU (Rectified Linear Unit) or GELU (Gaussian Error Linear Unit), etc. This layer typically does not contain trainable parameters, or the parameters remain fixed during training.

[0062] Upgraded layer: Contains the second weight matrix ∈ This layer is used to map features from the low-dimensional latent space back to the original high-dimensional space, enabling residual connections with the output of the next layer in the visual language model. The upscaling layer may also contain corresponding bias terms. .

[0063] In this structure, the total number of parameters in the adapter unit is approximately D×d+d×D. Since the bottleneck dimension d is much smaller than the feature dimension D, this portion of the parameters accounts for a very small percentage of the total number of parameters in the entire model. For example, when D=4096 and d=64, the adapter parameters account for approximately one-thousandth of the total number of parameters.

[0064] In another implementation, the adapter unit employs a low-rank adaptation structure. This structure achieves efficient parameter fine-tuning by introducing a low-rank decomposition matrix into the pre-trained weight matrix. Specifically, for a certain layer in the visual language model, the pre-trained weight matrix W∈ The low-rank adaptor structure is introduced in its bypass: The lower projection matrix A∈ This compresses the input features from dimension d to a low-rank dimension r; Up projection matrix B∈ This maps the low-rank features back to the original output dimension k.

[0065] Here, the rank *r* is a hyperparameter much smaller than *d* and *k*, typically taking values ​​of 4, 8, or 16. During training, the original weight matrix *W* remains frozen, with only matrices *A* and *B* being trainable parameters. During inference, the output of the low-rank adaptation structure is added to the output of the original layer via residuals.

[0066] In another implementation, the adapter unit employs a prefix adjustment structure. The parameters in this structure are represented as a set of learnable prefix embedding vectors, i.e., consecutive virtual token embeddings. These prefix embedding vectors are concatenated at the beginning of the feature sequence of the input visual language model, guiding the model's attention mechanism to favor semantic mappings related to action generation. The number and dimension of the prefix embedding vectors are preset hyperparameters, all of which are trainable parameters.

[0067] In this embodiment, regardless of the specific structure used internally by the adapter unit, its output is also connected to a motion space projection layer. This projection layer is used to ultimately map the feature vectors output by the adapter unit to the robot's target motion space, outputting specific motion control commands. The parameters of the motion space projection layer are the weight matrix and bias terms of the fully connected layer, which are also part of the adapter unit parameters and participate in subsequent reinforcement learning updates.

[0068] In the embodiments of this application, during the reinforcement learning phase of the adapter unit, the parameters of the adapter unit have clearly different update states from the main parameters of the visual language model.

[0069] In this embodiment, in each iteration of reinforcement learning, the environmental feedback information obtained after the robot performs an action is used to calculate the reward value. With the goal of maximizing the cumulative reward, the gradient of the policy loss relative to the adapter unit parameters is calculated using a policy gradient algorithm or a proximal policy optimization algorithm. During backpropagation, the gradient signal is only transmitted to the aforementioned learnable parameters within the adapter unit, and these parameters are numerically updated by an optimizer (such as the Adam optimizer).

[0070] This process means that the robot gradually "learns" better operating strategies through interaction with the environment. Essentially, the parameters of the adapter unit are constantly adjusted, thereby changing the mapping function from fused semantic features to action control commands.

[0071] Unlike the parameters of the adapter unit, the main parameters of the pre-trained visual language model remain frozen throughout the reinforcement learning training process. These parameters include: All parameters in the visual encoder, including convolutional kernel weights or self-attention weights of Transformer layers, weights of multilayer perceptrons, etc. All parameters in the text encoder, including embedding layer weights, self-attention weights, multilayer perceptron weights, etc. All original weights in the cross-modal alignment and fusion layer.

[0072] These core parameters do not participate in gradient calculations and do not receive parameter updates from the optimizer. This mechanism ensures that the rich semantic understanding capabilities learned by the visual language model during the pre-training phase are fully preserved, preventing catastrophic forgetting during the reinforcement learning exploration process. Simultaneously, because only a very small number of adapter units need to be updated, the computational cost of each iteration is significantly reduced, making online learning possible on real robotic systems.

[0073] It should be noted that the above description of the specific structure of the adapter unit is merely illustrative and not intended to limit the invention. In specific implementations, those skilled in the art can select different parameter fine-tuning structures to implement the adapter unit according to actual needs. As long as it is inserted into the pre-trained visual language model as an independent trainable parameter module, and only the parameters of this module are optimized during reinforcement learning updates while keeping the main parameters of the visual language model unchanged, it falls within the protection scope of this invention.

[0074] Example 2: refer to Figure 2 As shown, the robot control system provided in this application embodiment includes: a perception module 100, an instruction processing module 200, an action generation module 300 (including a visual language model processing module), a control execution module 400, and a reinforcement learning optimization module 500. The modules are connected through a communication interface to form a closed-loop control system.

[0075] The perception module 100 is used to acquire current environmental status information, including an RGB image acquisition unit 101 and a depth information acquisition unit 102. In a preferred embodiment, the RGB image acquisition unit can be a color sensor of an industrial camera or a depth camera, used to acquire color images of the work scene. The depth information acquisition unit can be a depth camera, a structured light sensor, or a LiDAR, used to acquire three-dimensional spatial distance data of objects in the scene. The perception module outputs the acquired information to subsequent modules for processing.

[0076] The instruction processing module 200 receives a natural language task description input by the user and converts the natural language task description into a semantic feature representation. This instruction processing module 200 includes a text encoding unit 201, which is constructed based on a text encoder in a pre-trained visual language model. This text encoding unit is capable of encoding natural language instructions into high-dimensional semantic vectors for use by the action generation module.

[0077] The visual language model processing model is the core component of the action generation module 300, comprising a pre-trained visual language model and an adapter unit 303 inserted therein. In this embodiment, the pre-trained visual language model includes a visual encoding unit 301, used to extract visual features from the environmental state information acquired by the perception module 100 and align them with the semantic features output by the instruction processing module 200 to a unified semantic space for fusion. The multimodal fusion unit 302 is used to fuse visual and semantic features, for example, by using a cross-modal attention mechanism to achieve deep interaction between visual and linguistic information.

[0078] In this embodiment, the adapter unit 303, serving as the carrier of the action generation strategy, is a lightweight network structure inserted into the visual language model. The adapter unit 303 receives the fused features output by the multimodal fusion unit 302, maps them to the action space, and outputs action control commands. The action output unit 304 converts the mapping result into a control command format executable by the robot.

[0079] In one implementation, the adapter unit 303 may employ a low-rank adaptation structure (LoRA), which achieves efficient parameter fine-tuning by adding a low-rank decomposition matrix next to the weight matrix of the pre-trained model. In other implementations, a prefix adjustment structure or a serial adapter structure may also be used. Regardless of the specific structure used, the adapter unit 303 has independent parameters that are isolated from the parameters of the main visual language model.

[0080] The control execution module 400 is used to drive the robot to perform corresponding actions according to the action control commands output by the action generation module, and to acquire feedback information during the execution process. The control execution module 400 includes: a motion planning unit 401, used to generate the motion trajectory of the robot's actuators according to the action control commands, including path planning and velocity planning; an actuator control unit 402, used to convert the motion planning results into joint control signals or end effector control signals for the robot, controlling the robot's robotic arm and end effector to complete operations such as grasping, handling, and assembly; and a feedback unit 403, used to collect feedback information such as changes in the environmental state and task completion status after the robot performs actions and output it to the reinforcement learning optimization module 500.

[0081] The reinforcement learning optimization module 500 is used to iteratively optimize the action generation strategy based on execution feedback. This module includes: a reward calculation unit 501, used to calculate a reward value based on feedback information, wherein the reward value calculation includes at least one of a success reward term, a distance constraint term, and an action smoothness constraint term; a policy evaluation unit 502, used to evaluate the performance of the current action generation strategy based on the reward value, for example, by calculating an advantage function to measure the performance of the current strategy relative to the average; and a parameter update unit 503, used to update the parameters of the action generation strategy based on the policy evaluation results, with the goal of maximizing the cumulative reward.

[0082] It is particularly important to emphasize that the parameter update unit 503 only optimizes the parameters of the adapter unit 303 during the update process, keeping the main parameters of the visual language model unchanged.

[0083] Example 3: refer to Figure 3 As shown, in one embodiment, the control system is deployed in a host computer computing unit. The host computer computing unit is connected to the robot controller and sensor module via a communication interface.

[0084] The host computer computing unit includes a model inference module and a policy optimization module. The model inference module loads the visual language model and the adapter unit, and generates action control commands based on the input environmental state information and task instructions. The policy optimization module updates the parameters of the adapter unit through reinforcement learning based on execution feedback.

[0085] The sensor module includes an RGB camera and / or a depth camera, used to collect status information of the working environment. The robot controller receives motion control commands output from the host computer computing unit, parses and processes them, and then drives the robot actuators (such as robotic arms and end effectors) to complete the corresponding operations.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0087] In summary, by adopting the technical solution provided by this invention, the following technical effects are achieved: Efficiently utilize prior knowledge: By freezing the backbone parameters of the pre-trained visual language model, the general visual-language alignment capability learned from massive amounts of data can be directly utilized, avoiding the huge cost of training a perception model from scratch, and preventing overfitting due to insufficient data.

[0088] Efficient parameter fine-tuning: The introduction of a lightweight vision-language-action adapter allows for adaptation to specific control tasks with only a small number of parameters trained, significantly reducing memory usage and computational overhead, making it possible to deploy large-scale model control strategies on resource-constrained devices.

[0089] Strong closed-loop optimization and generalization capabilities: By introducing a reinforcement learning mechanism, the robot uses reward signals from environmental feedback to drive adapter parameter updates, enabling the robot's strategy to transcend the limitations of expert demonstration data and learn better or even entirely new behavioral strategies through trial and error, significantly improving the success rate and robustness of tasks in complex and dynamic environments.

[0090] Bridging the semantic-action gap: The adapter structure, acting as a "bridge" between the semantic space and the action space, effectively solves the technical bottleneck that existing visual language models cannot directly output precise and continuous control commands, and realizes end-to-end closed-loop control from "understanding intent" to "precise execution".

[0091] This application provides an electronic device, characterized in that it includes: a processor and a memory storing a computer program, wherein when the processor runs the computer program, it implements the steps of the robot control method described above.

[0092] This application also provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, implements the steps of the robot control method described above.

[0093] The embodiments of the electronic devices and storage media provided in this application may include all the technical features of any of the above-described processing method embodiments. The extended and explanatory content of the specification is basically the same as that of the embodiments of the above methods, and will not be repeated here.

[0094] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0095] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0096] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0097] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0098] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0099] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0102] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, storage disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0103] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for controlling a robot, characterized in that, The control method includes: Obtain current environmental status information and natural language task description; Based on a pre-trained visual language model, visual features are extracted from the acquired current environment state information, and the received natural language task description is converted into a semantic feature representation. The fused features, obtained by fusing the visual features and the semantic features, are mapped to the action space through an adapter unit to obtain action control instructions; wherein, the adapter unit is a lightweight network structure inserted into the visual language model, serving as a carrier of the action generation strategy; Control the robot to execute the actions corresponding to the action control commands, and obtain execution feedback information; The reward value is calculated based on the feedback information, and the action generation strategy is updated by reinforcement learning based on the reward value with the goal of maximizing the cumulative reward. During the update process, the parameters of the adapter unit are optimized while keeping the main parameters of the visual language model unchanged. Repeat the steps from acquiring environmental state information to updating the action generation strategy until the action generation strategy meets the preset convergence condition.

2. The control method according to claim 1, characterized in that, The adapter unit includes at least one parameter-efficient fine-tuning structure selected from the following: a low-rank adapter structure, a prefix adjustment structure, or a serial adapter structure.

3. The control method according to claim 1, characterized in that, The visual language model includes visual encoding units and text encoding units; The visual features are extracted from the environmental state information by the visual encoding unit, and the environmental state information includes at least one of RGB image information and depth information. The semantic feature representation is obtained by encoding the natural language task description through the text encoding unit.

4. The control method according to claim 1, characterized in that, The goal is to maximize the cumulative reward. Based on the reward value, the action generation strategy is updated through reinforcement learning, which is performed using a proximal policy optimization algorithm and / or a policy gradient algorithm.

5. The control method according to claim 1, characterized in that, The reward value is calculated based on the task execution result, and the calculation items of the reward value include at least one of the following: success reward item, distance constraint item, and action smoothness constraint item.

6. The control method according to claim 1, characterized in that, When outputting motion control commands, the range of the motion control commands is also limited to constrain the range of motion of the robot during execution.

7. A control system for a robot, characterized in that, The system includes: The sensing module is used to acquire current environmental status information; The instruction processing module is used to receive a natural language task description and convert the natural language task description into a semantic feature representation; The visual language model processing module includes a pre-trained visual language model and an adapter unit inserted therein. The pre-trained visual language model is used to extract visual features from the environmental state information and align the visual features with the language features to a unified semantic space for fusion. The adapter unit, as a carrier of the action generation strategy, is used to map the fused visual features and semantic features to the action space and output action control commands. The control execution module is used to control the robot to perform corresponding actions according to the action control instructions, and to obtain feedback information during the execution process; The reinforcement learning optimization module is used to calculate the reward value based on the feedback information, and with the goal of maximizing the cumulative reward, drive the action generation strategy to perform reinforcement learning updates based on the reward value. During the update process, the parameters of the adapter unit are optimized while keeping the main parameters of the visual language model unchanged.

8. The system according to claim 7, characterized in that, The control execution module includes: A motion planning unit is used to generate an execution trajectory based on the motion control command; The actuator control unit is used to control the robot actuator to complete the corresponding operation; The feedback unit is used to output feedback on the robot's execution status.

9. An electronic device, characterized in that, include: A processor and a memory storing a computer program, wherein, when the processor executes the computer program, the steps of the robot control method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the robot control method according to any one of claims 1 to 6.

Images