A mechanical arm energy-saving control method and system based on a large language model and distributed reinforcement learning

Abstract

This invention belongs to the field of intelligent control and robot energy-saving optimization technology. It discloses an energy-saving control method and system for robotic arms based on a large language model and distributed reinforcement learning. The system includes semantic policy generation, adaptive double-buffer sampling, energy-sensitive policy optimization, vision-language joint representation, and an octopus-like distributed execution unit. An initial policy is generated by fusing natural language instructions and multi-source perceptual data. The policy is optimized by combining dynamic weighted sampling and a hybrid loss function with energy consumption penalties. A distributed architecture with local autonomy and global collaboration is employed to achieve high energy efficiency, strong fault tolerance, and cross-task generalization capabilities.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and robot energy-saving optimization technology, specifically a method and system for energy-saving control of robotic arms based on large language models and distributed reinforcement learning. Background Technology

[0002] Robotic arms can be used to perform operations such as grasping, carrying, and placing objects. In the fintech scenario, for example, robotic arms can help banks count and sort banknotes or coins, thereby improving work efficiency and reducing human error. In the healthcare scenario, for example, robotic arms can be used to grasp and operate surgical instruments to assist in medical surgeries. In addition, robotic arms can also be used to sort, verify, and distribute medicines, thereby reducing the manpower required for manual medicine delivery and improving the efficiency and accuracy of medicine distribution.

[0003] In related technologies, the manipulation of robotic arms mainly relies on a task planning model for task planning and a strategy control model for control execution. These two models are trained independently, which not only consumes significant time and computational resources, but also, because the collaboration between the two models is not naturally close, information transmission often fails during the transition from task planning to actual control, leading to inaccurate robotic arm movements. Therefore, improving the accuracy of robotic arm manipulation has become an urgent technical problem to be solved.

[0004] Furthermore, current technologies are mostly trained and designed for specific scenarios and items. When the environment changes slightly, such as the angle of the item's placement, the lighting conditions, or when faced with items of new shapes or materials, the robotic arm often struggles to grasp accurately. The model struggles to learn general grasping patterns and strategies, and cannot effectively transfer experience learned in one scenario to other scenarios. For example, in a logistics warehouse, if a previously trained model can only grasp regularly shaped packaging boxes, it will be unable to complete the grasping task when encountering irregularly shaped packages.

[0005] Furthermore, in current embodied intelligence technologies, the vision module is responsible for recognizing objects, and the language module is responsible for understanding task descriptions. However, there is a lack of deep integration between these two modules and the final motion execution module. Information acquired by the vision and language modules suffers significant loss when transmitted to the motion module, making it difficult for the robotic arm to accurately generate and execute actions based on visual and linguistic information. For example, when receiving the instruction to "grab the blue cylindrical part," the vision module may correctly identify the blue cylinder, but due to coordination issues, the robotic arm cannot grasp the part with the appropriate posture and force when the instruction is transmitted to the motion execution module. Summary of the Invention

[0006] To address the problems existing in the prior art, the main objective of this invention is to propose an energy-saving control method and system for robotic arms based on large language models and distributed reinforcement learning. This invention can solve the problems of high energy consumption, low sample efficiency, weak fault tolerance, and insufficient cross-task generalization ability of existing robotic arms in complex dynamic environments.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: An energy-saving control method for robotic arms based on large language models and distributed reinforcement learning includes the following process: Acquire multi-source sensor data and natural language commands, wherein the multi-source sensor data includes RGB-D images, joint torque feedback and tactile sensing signals; The natural language instructions are parsed, and an initial action strategy sequence is generated; The initial action policy sequence and the collected state-action-reward-next state quadruples are weighted and sampled to obtain a mixed sample. The policy network parameters are updated using the mixed samples and the total loss function to obtain the policy parameters after policy update. The RGB-D image is encoded to extract image features; the natural language instructions are encoded to extract instruction embeddings; the image features and instruction embeddings are fused to generate a joint state representation. Based on the joint state representation, the energy consumption value of the current path is predicted to obtain the predicted energy consumption value of the current path; the predicted energy consumption value is compared with a preset threshold, and if the predicted energy consumption value exceeds the preset threshold, the action strategy is regenerated. The robotic arm execution unit executes the updated strategy parameters and the regenerated motion strategy, and controls the robotic arm based on the joint torque feedback and tactile sensing signals to achieve energy-saving control of the robotic arm.

[0008] Preferably, based on the dynamic sampling weight function Weighted sampling is performed to obtain a pooled sample; where, As the initial weights, The attenuation coefficient is... t This represents the current training iteration step. T This represents the total number of training iterations. This is the dynamic sampling weight decay coefficient. Rate the difficulty of the task. This represents the energy consumption coefficient for the current action. This is the preset attenuation coefficient.

[0009] Preferably, the total loss function is

[0010] in, To enhance the learning loss, the PPO algorithm is used to calculate the policy gradient; The intrinsic motivation loss is constructed based on the prediction error of state access frequency; This is a distributed consistency loss used to constrain the consistency of the actions output by each local Actor network in the global coordinate system. The trajectory imitation loss is used to measure the Euclidean distance between the current policy trajectory and the trajectory generated by LLM. This is a rule-constrained loss, used to penalize actions that violate kinematic boundaries or collision constraints. As an energy consumption penalty item, , To achieve the theoretically optimal energy consumption, the dynamic model of the robotic arm, constructed based on the Lagrange equation, is determined according to the joint trajectory. It is calculated from its first and second derivative integrals; The actual energy consumption is obtained by real-time sampling and integration of current and voltage sensors on the joint actuator.

[0011] Preferably, the RGB-D image is encoded to extract image features; the natural language instruction is encoded to extract instruction embeddings; and the image features and instruction embeddings are fused through a cross-attention mechanism to generate a joint state representation, specifically including the following process: The RGB-D image is preprocessed, including depth map normalization, color space conversion to HSV domain, and background removal, followed by encoding and extraction of image features. Natural language instructions are tagged with part-of-speech tags and subjected to dependency parsing to extract action verbs, target objects, and spatial relation triples. The triples are mapped to semantic slot vectors to obtain instruction embeddings. The instruction embeddings are then aligned and fused with the image features through a multi-head cross-attention mechanism to generate joint state representations.

[0012] This invention also provides an energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning, used to implement the energy-saving control method for a robotic arm based on a large language model and distributed reinforcement learning as described above, comprising: Natural language instruction input module: used to acquire natural language instructions; Multi-source sensing data acquisition module: used to acquire multi-source sensing data, including RGB-D images, joint torque feedback and tactile sensing signals; Semantic policy generation unit: used to parse the natural language instructions and generate an initial action policy sequence; Adaptive double-buffer sampling unit: used to perform weighted sampling on the initial action policy sequence and the collected state-action-reward-next state quadruple to obtain mixed samples; Energy-sensitive policy optimization unit: used to update policy network parameters using the mixed samples and total loss function, obtain policy parameters with updated policy, and synchronize the updated policy parameters to the octopus-like distributed execution unit; Visual-Language Joint Representation Unit: Used to encode RGB-D images and extract image features; encode natural language instructions and extract instruction embeddings; fuse image features and instruction embeddings to generate joint state representations; Gradient boosting tree model: used to predict the energy consumption value of the current path based on the joint state representation, and obtain the predicted energy consumption value of the current path; compare the predicted energy consumption value with a preset threshold, and if the predicted energy consumption value exceeds the preset threshold, trigger the semantic policy generation unit to regenerate the action policy sequence. Octopus-like distributed execution unit: used to execute the updated strategy parameters and regenerated motion strategy through the robotic arm execution unit, and control the robotic arm according to the joint torque feedback and tactile sensing signals to achieve energy-saving control of the robotic arm.

[0013] Preferably, the natural language command input module receives high-level task commands input by the user in natural language form through a standard API interface; the multi-source sensing data acquisition module connects to an RGB-D camera, a joint torque sensor, and a tactile array sensor to acquire multi-source sensing data respectively, and packages the multi-source sensing data in a timestamp-aligned manner and transmits it to the vision-language joint representation unit and the octopus-like distributed execution unit; the multi-source sensing data includes RGB-D images of the environment, real-time torque feedback of each joint, and tactile pressure distribution signals of the end effector contact surface.

[0014] Preferably, it also includes an LLM demonstration buffer and an RL interaction buffer. The semantic policy generation unit is connected to the natural language instruction input module. The semantic policy generation unit integrates a fine-tuned large language model. During the training phase, the large language model is supervised and fine-tuned using a paired dataset containing industrial operation instructions and corresponding action sequences. Upon receiving a natural language instruction, the semantic policy generation unit calls the large language model to parse the action verbs, target objects, and spatial relationships in the instruction, and generates an initial action policy sequence. The initial action policy sequence consists of a series of joint angle target values ​​or Cartesian space path points. The generated initial action policy sequence is written into the LLM demonstration buffer, and simultaneously associated with the corresponding natural language instruction and the current environment state snapshot to form a structured demonstration sample. Both the LLM demonstration buffer and the RL interaction buffer are circular buffer structures, deployed in different address segments of system memory. The LLM demonstration buffer stores high-confidence policy samples generated by the semantic policy generation unit, and the RL interaction buffer is filled with a state-action-reward-next state quadruple returned by the octopus-like distributed execution unit during real interaction.

[0015] Preferably, the adaptive double-buffered sampling unit establishes data reading channels with both the LLM demonstration buffer and the RL interaction buffer; during each policy update iteration, the adaptive double-buffered sampling unit adjusts its sampling weights according to the dynamic sampling weight function. Calculate the sampling ratio of LLM samples at the current time t, where, For the preset attenuation coefficient, Total training steps The result is calculated by the task planning module based on the geometric complexity of the target object, the operating distance, and the density of obstacles. The energy-sensitive strategy optimization unit provides real-time feedback; the adaptive double-buffer sampling unit samples from the LLM demonstration buffer according to the weight function. Samples are taken from the RL interaction buffer. The samples are merged to form a mixed sample, and the mixed sample is input into the energy-sensitive strategy optimization unit.

[0016] Preferably, the input end of the visual-language joint representation unit is connected to the multi-source perception data acquisition module and the natural language command input module, respectively. The visual-language joint representation unit includes a CLIP visual encoder and an LLM semantic encoder, a semantic parsing layer, a multi-head cross-attention fusion module, and an output layer. First, the RGB-D image is processed by depth map normalization, mapping the depth values ​​to the [0,1] interval. Then, the color space is converted to the HSV domain, and a background segmentation algorithm is used to remove static background areas, retaining only the foreground operation area. Then, the processed image is input into the CLIP visual encoder to extract multi-dimensional image feature vectors. The natural language command is then fed into the LLM semantic encoder. A dependency parsing layer is added to the general LLM, outputting action verb-target object-spatial relation triples. These triples are mapped to semantic slot vectors through an embedding layer and then input together with the multi-dimensional image feature vectors into a multi-head cross-attention fusion module. The multi-head cross-attention fusion module performs cross-modal alignment through multiple attention heads and finally outputs a multi-dimensional joint state representation. The joint state representation is simultaneously fed into an energy-sensitive policy optimization unit and an independently deployed gradient boosting tree model. The gradient boosting tree model is trained based on historical energy consumption data and is used to predict the energy consumption value of the current policy path. If the predicted value exceeds a preset threshold, the semantic policy generation unit is triggered to re-parse the instructions and generate a corrected policy.

[0017] Preferably, the octopus-like distributed execution unit adopts a local Actor-global Critic architecture. The hardware deployment of the octopus-like distributed execution unit is as follows: each joint of the robotic arm embeds a local Actor network, labeled as joint 1 local Actor network, joint 2 local Actor network, up to joint n local Actor networks, which are all deployed on the embedded processors of the joint controllers of the robotic arm; the global Critic network is deployed in the central processing unit; each local Actor network is connected to the central processing unit to form a star topology; the inputs of the local Actor network include the angle output by the encoder of this joint, the angular velocity output by the gyroscope, and the torque sensor... The output torque value and the pressure distribution map output by the tactile sensor are used. The output of the local Actor network is the joint target position increment and impedance control parameters, including damping coefficient and stiffness coefficient. The global Critic network receives state observations of all joints and joint state representations, including angle, angular velocity, and torque. The global Critic network outputs a state value estimate. During the training phase, the local Actor network uses an asynchronous dominant actor-critic algorithm to calculate the policy gradient based on the local reward signal and the benchmark value provided by the global Critic network and updates it asynchronously. The global Critic network uses a temporal difference method to fit the value function using multi-step rewards.

[0018] The present invention has the following beneficial effects: This invention relates to an energy-saving control method and system for robotic arms based on large language models and distributed reinforcement learning. By parsing natural language commands, it directly generates initial action policy sequences, significantly reducing the policy search space of reinforcement learning, improving policy convergence speed, and lowering the computational resource consumption required for training. The method utilizes initial policy sequences and states... action award Next-state quadruple weighted sampling yields hybrid samples, which can balance demonstration experience and real interaction data, improving sample utilization efficiency and policy learning stability. Policy network parameters are updated based on hybrid samples and the total loss function. The joint state representation obtained by fusing image features and instruction embeddings predicts path energy consumption, and the policy is regenerated when thresholds are exceeded. This allows for energy control from the planning stage, effectively reducing the robot arm's operating power consumption. The unified joint state representation formed by fusing image features and instruction embeddings eliminates the loss of visual and linguistic information during transmission, enabling the robot arm to more accurately understand task intent and environmental states, improving action execution accuracy. Simultaneously, RGB data is fused... Closed-loop control is achieved through 3D image processing, joint torque feedback, and tactile sensing signals. This enables the robot to adapt to changes in the environment, such as object shape, placement angle, and lighting, enhancing its adaptability and generalization in complex scenarios and ensuring stable and reliable control. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture of the energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning, provided in an embodiment of the present invention.

[0020] Figure 2 This is a flowchart illustrating the semantic policy generation and double-buffer sampling mechanism in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the visual-language joint representation unit in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the loss function composition of the energy-sensitive strategy optimization unit in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the "local Actor-global Critic" architecture of the octopus-like distributed execution unit in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the execution flow of the single-joint fault compensation mechanism in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described clearly and in detail below with reference to specific embodiments and the accompanying drawings. Those skilled in the art will be able to implement the present invention based on these descriptions. Furthermore, the embodiments of the present invention described below are generally only some, not all, of the embodiments of the present invention. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0026] This invention provides an energy-saving control method and system for robotic arms based on large language models and distributed reinforcement learning. The specific embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the overall architecture of the system includes a natural language command input module, a multi-source sensor data acquisition module, a semantic policy generation unit, an LLM demonstration buffer, an RL interaction buffer, an adaptive double-buffer sampling unit, an energy-sensitive policy optimization unit, a vision-language joint representation unit, an octopus-inspired distributed execution unit, and a robotic arm. The natural language command input module receives high-level task commands input by the user in natural language form through a standard API interface, such as "grab the red cylinder from position A and place it at position B." The multi-source sensor data acquisition module connects to an RGB-D camera, joint torque sensors, and a tactile array sensor to acquire RGB-D images of the environment, real-time torque feedback from each joint, and tactile pressure distribution signals from the end effector contact surface. This multi-source sensor data is then packaged and transmitted to the vision-language joint representation unit and the octopus-inspired distributed execution unit in a timestamp-aligned manner.

[0028] The semantic policy generation unit is connected to the natural language instruction input module and integrates a fine-tuned Large Language Model (LLM). During the training phase, this LLM is supervised and fine-tuned using a paired dataset containing industrial operation instructions and corresponding action sequences. Upon receiving a natural language instruction, the semantic policy generation unit invokes the LLM to parse the action verbs, target objects, and spatial relationships within the instruction, generating an initial action policy sequence. This initial action policy sequence consists of a series of joint angle target values ​​or Cartesian space path points. The generated initial action policy sequence is then written to the LLM demonstration buffer, simultaneously associated with the corresponding natural language instruction and a snapshot of the current environment state, forming a structured demonstration sample. Both the LLM demonstration buffer and the RL interaction buffer are circular buffer structures, deployed in different address segments of system memory. The LLM demonstration buffer stores high-confidence policy samples generated by the semantic policy generation unit, while the RL interaction buffer is filled with state-action-reward-next-state quadruples returned by the octopus-like distributed execution unit during real-world interactions.

[0029] like Figure 2 As shown, the adaptive double-buffered sampling unit establishes data read channels with both the LLM demonstration buffer and the RL interaction buffer. During each policy update iteration, the adaptive double-buffered sampling unit adjusts its sampling weights according to the dynamic sampling weight function. Calculate the sampling ratio of LLM samples at the current time t, where, For the preset attenuation coefficient, Total training steps The result is calculated by the task planning module based on the geometric complexity of the target object, the operating distance, and the density of obstacles. This is then fed back in real time by the energy-sensitive strategy optimization unit. Specifically, , =0.3, then the dynamic sampling weight function Each time the policy is updated, the adaptive double-buffered sampling unit samples from the LLM demo buffer according to this weight. Samples are taken from the RL interaction buffer. 1 sample, of which To determine the batch size, merge samples to form a mixed sample, and input the mixed sample into the energy-sensitive policy optimization unit.

[0030] After receiving mixed training samples, the energy-sensitive policy optimization unit uses, for example... Figure 4 The total loss function shown is: Update the policy network parameters. This includes the reinforcement learning loss. The policy gradient is calculated using the PPO algorithm; intrinsic motivation loss is also considered. Construction of prediction error based on state access frequency; distributed consistency loss Constraining the consistency of the actions output by each local Actor network in the global coordinate system; trajectory imitation loss. Measure the Euclidean distance between the current policy trajectory and the LLM-generated trajectory; rule-constrained loss. Penalize actions that violate kinematic boundaries or collision constraints; energy consumption penalty term. In the middle, the theoretical energy consumption integral value The robotic arm dynamics model, built based on the Lagrange equations, is based on the joint trajectories. It is calculated from its first and second derivative integrals. The parameters are obtained by real-time sampling and integration of current and voltage sensors on the joint driver. The energy-sensitive strategy optimization unit synchronizes the updated strategy parameters to the octopus-like distributed execution unit via a high-speed bus.

[0031] When the energy-sensitive strategy optimization unit detects task failure or action energy consumption exceeding the threshold, it generates a structured feedback report. The structured feedback report includes a failure type identifier, the current energy consumption value, and the action deviation vector. The structured feedback report is used as context input to the large language model, which outputs a corrected strategy code fragment. The strategy code fragment includes parameter constraints on high-energy-consuming actions, including the maximum acceleration limit and the minimum smoothing time constant.

[0032] The structure of the visual-language joint representation unit is as follows: Figure 3 As shown, the input terminals of the visual-language joint representation unit are connected to the multi-source sensing data acquisition module and the natural language command input module, respectively. It can receive the RGB-D image output from the multi-source sensing data acquisition module and the natural language command output from the natural language command input module. In the input layer of the visual-language joint representation unit, the RGB-D image first undergoes depth map normalization processing, mapping the depth values ​​to the [0,1] interval. Then, the color space is converted to the HSV domain, and a background segmentation algorithm is used to remove static background areas, retaining only the foreground operation area. The processed image is input to the CLIP visual encoder to extract a 768-dimensional image feature vector. Natural language commands are fed into an LLM semantic encoder, which adds a dependency parsing layer (i.e., a semantic analysis layer) to the general LLM. The dependency parsing layer performs semantic parsing on the result encoded by the LLM semantic encoder and outputs a triplet of action verb-target object-spatial relation, such as (grab, red cylinder, above). The triplet is mapped to a semantic slot vector (i.e., command embedding) by the embedding layer. The command embedding and the CLIP image feature vector (i.e., the 768-dimensional image feature vector mentioned above) are input into a multi-head cross-attention fusion module. Cross-modal alignment is performed through 8 attention heads, and the final output is a joint state representation with a dimension of 768. This joint state indicates The data is simultaneously fed into an energy-sensitive policy optimization unit and an independently deployed gradient boosting tree model. The latter (i.e., the gradient boosting tree model) is trained based on historical energy consumption data and is used to predict the energy consumption value of the current policy path. The predicted energy consumption value is compared with a preset threshold. If the predicted value exceeds the preset threshold, the semantic policy generation unit is triggered to re-parse the instructions and generate a corrected policy.

[0033] The octopus-like distributed execution unit adopts, for example Figure 5The "Local Actor-Global Critic" architecture shown is illustrated with the following hardware deployment of the octopus-like distributed execution unit: Each robotic arm joint embeds a local Actor network, labeled as Joint 1 Local Actor Network, Joint 2 Local Actor Network, and so on, up to Joint n Local Actor Network. These local Actor networks are deployed on the embedded processors of the joint controllers of each robotic arm. The global Critic network is deployed in the central processing unit (CPU). Each local Actor network is connected to the CPU via a CAN bus communication link, forming a star topology. The inputs to the local Actor networks include the angle output from the joint encoder, the angular velocity output from the gyroscope, the torque value output from the torque sensor, and the pressure distribution map output from the tactile sensor. The output of the local Actor network is the increment of the joint target position. With impedance control parameters (damping coefficient) and (Stiffness coefficient). The global Critic network receives state observations (including angles, angular velocities, and moments) and joint state representations from all joints. Global Critic Network Output State Value Estimation During the training phase, the local Actor network employs the Asynchronous Advantage Actor-Critic (A3C) algorithm, which calculates the policy gradient and updates it asynchronously based on the local reward signal and the benchmark value provided by the global Critic; the global Critic network, on the other hand, uses the TD(λ) temporal difference method to fit the value function using multi-step rewards.

[0034] In terms of fault tolerance, such as Figure 6 As shown, the system periodically performs single-joint fault simulations during the training phase: a joint is randomly selected (e.g., joint 2), and its sensor inputs are masked via software instructions. At this point, the local Actor network of that joint only receives zero-value inputs. Upon detecting this anomaly, the fault detection module generates a fault detection signal and broadcasts it to the local Actor networks of the remaining joints. The remaining joints (e.g., the local Actor networks of joint 1 and joint n) adjust the force distribution ratio and attitude parameters according to the current task objective and remaining degrees of freedom through local strategies. For example, they increase the torque output of adjacent joints to compensate for missing degrees of freedom, and adjust the end effector trajectory to maintain task feasibility. When the system determines that the end effector has stabilized near the target pose, it marks the attitude compensation as complete and continues to execute subsequent actions until the task ends. The data collected during this process is written to the RL interaction buffer to enhance the robustness of the strategy.

[0035] In actual operation, the user issues the command "move the blue cube from the conveyor belt to the right tray" via the natural language command input module. The multi-source sensing data acquisition module simultaneously acquires RGB-D images of the scene, torques of each joint, and end-effector tactile signals. The vision-language joint representation unit parses the command and fuses visual features to generate a joint state representation. The semantic policy generation unit calls the LLM to generate the initial grab-move-place sequence and stores it in the LLM demonstration buffer. The adaptive double-buffer sampling unit assigns higher LLM weights based on the current training stage (early stage) and samples more demonstration samples. The energy-sensitive policy optimization unit updates the policy by combining mixed samples and predicts energy consumption through a gradient boosting tree model. If the predicted energy consumption exceeds the limit, an LLM reconstruction instruction is triggered. The final policy is executed by an octopus-like distributed execution unit, with each joint's local Actor network cooperating to complete the action. If a joint fails during execution, the other joints automatically compensate to ensure the task is completed without interruption. Throughout the process, all modules achieve low-latency data interaction through shared memory and the CAN bus, ensuring the real-time performance and energy efficiency of the control loop.

[0036] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below with a specific application scenario.

[0037] In the actual operation scenario of a logistics sorting center, the robotic arm needs to grab randomly appearing irregularly shaped packages on the conveyor belt and sort and place them into designated areas. The user issues the command "move the green irregular soft package from the conveyor belt to the yellow bin on the left" through the natural language command input module. This command is transmitted to the system via a standard API interface. The multi-source sensing data acquisition module simultaneously activates the RGB-D camera, torque sensors of each joint, and end-effector tactile array sensor to acquire RGB-D images containing depth information, real-time torque feedback of each joint, and contact surface pressure distribution signals. After being aligned with a unified timestamp, these are transmitted to the vision-language joint representation unit and the octopus-like distributed execution unit, respectively.

[0038] After receiving the RGB-D image and natural language instructions, the visual-language joint representation unit first normalizes the depth map, linearly mapping the original depth values ​​to the [0,1] interval. Then, it converts the RGB channels to the HSV color space to enhance color robustness and uses a Gaussian mixture model-based background segmentation algorithm to remove static backgrounds, retaining only the foreground operation area on the conveyor belt. The processed image is input into the CLIP visual encoder to extract a 768-dimensional image feature vector. Simultaneously, the natural language instructions are fed into the LLM semantic encoder, which embeds a dependency parsing layer on top of a general large language model to parse action verb-target object-spatial relation triples, such as (move, green irregular soft package, yellow bin on the left). These triples are mapped to fixed-dimensional semantic slot vectors through a learnable embedding layer and then input together with the CLIP image features into a multi-head cross-attention fusion module. Eight parallel attention heads calculate cross-modal association weights, ultimately outputting a joint state representation. It simultaneously encodes the spatial location, geometric outline, material properties, and task semantics of the visual target in the language instructions.

[0039] The semantic policy generation unit calls the internally integrated fine-tuning large language model, based on joint state representation. Based on the original instruction context, an initial action policy sequence is generated. This sequence contains complete Cartesian space pathpoints from approach and grasp to move and release, along with the corresponding gripper opening and closing timing. This sequence is then structured and encapsulated into a (instruction, environment snapshot, action sequence) triple and written to the LLM demo buffer. The adaptive double-buffered sampling unit, within the policy update cycle, adjusts the sampling based on a dynamic sampling weight function. Calculate the current percentage of LLM samples, including the task difficulty score. It is derived from a combination of the target object's surface curvature, centroid offset, and the number of surrounding obstacles. Feedback is provided by the energy-sensitive strategy optimization unit; since this is the initial stage of the mission and the target is a soft, irregularly shaped object. The higher the value, the higher the weight of the LLM demonstration sample, thus sampling more high-confidence samples from the LLM demonstration buffer and mixing them with a small number of real interaction samples in the RL interaction buffer to form a training set.

[0040] After receiving mixed training samples, the energy-sensitive policy optimization unit constructs the total loss function. End-to-end optimization is performed, including trajectory imitation loss. The Euclidean distance between the joint trajectory output by the current policy and the trajectory generated by LLM is constrained to not exceed a threshold to ensure semantic consistency; an energy consumption penalty term is also included. Theoretical optimal energy consumption Based on the dynamic model of the Lagrange equations, according to the joint angle angular velocity and angular acceleration Integral calculation, actual energy consumption The parameters are obtained in real time by integrating the voltage and current sensors of each joint actuator. When the deviation between the two is too large, the loss function applies gradient backpropagation pressure, forcing the policy network to adjust the motion smoothness and acceleration curve. The optimized policy parameters are synchronized to the octopus-like distributed execution unit via a high-speed bus.

[0041] In the octopus-like distributed execution unit, the local actor networks of each joint (i.e., the local actor network of joint 1, the local actor network of joint 2, and the local actor network of joint 3) are deployed in the embedded processor of the corresponding joint controller. They receive in real time the angle output by the encoder of the joint, the angular velocity output by the gyroscope, the torque value output by the torque sensor, and the pressure distribution map output by the haptic array as inputs, and output the incremental target position of the joint. With impedance control parameters , The global Critic network is deployed at the central processing unit and aggregates all joint states and joint states via CAN bus communication link 38. Output state value estimation Used to guide local strategy updates. When performing a grasping action, if the tactile signal at the end of the device changes abruptly due to slippage of the package surface, the local Actor network can instantly fine-tune the gripping force and posture based on local force feedback, without waiting for central decision-making, achieving millisecond-level response.

[0042] If a joint sensor suddenly fails during the grasping process, the fault detection module immediately generates a fault detection signal and broadcasts it to the local Actor networks of the remaining joints. The local Actor networks of joint 1 and joint 3 then activate the compensation mechanism, re-planning the force distribution ratio according to the remaining degrees of freedom. For example, they increase the output torque of adjacent joints to compensate for the missing pitch degree of freedom, and solve for a new feasible end effector trajectory through inverse kinematics, so that the end effector can still stably reach the target pose. After completing the attitude compensation, the placement action continues. The state-action-reward sequence recorded in this fault compensation process is written into the RL interaction buffer for subsequent policy robustness enhancement training.

[0043] Throughout the task process, the gradient boosting tree model continuously receives joint state representations. As input, the energy consumption of the current strategy path is predicted. If the predicted value exceeds a preset threshold, a reconstruction trigger signal is sent to the semantic strategy generation unit, which re-parses the instruction and generates a low-energy alternative strategy, such as using a smoother approach trajectory or reducing unnecessary joint swaying. All modules achieve microsecond-level data synchronization via shared memory and the CAN bus, ensuring that the closed-loop latency from instruction understanding to action execution is less than 50ms, meeting industrial real-time requirements.

[0044] As can be seen, the energy-saving control method and system for robotic arms based on large language models and distributed reinforcement learning proposed in this invention transforms natural language instructions into executable initial policies through a semantic policy generation unit, reducing the exploration space of reinforcement learning; it introduces high-confidence LLM demonstration data in the early stage of training through an adaptive double-buffer sampling mechanism, gradually transitioning to real interaction data in the later stage, and dynamically adjusting the sampling ratio based on task difficulty and action energy consumption; it explicitly introduces an energy consumption penalty term through an energy-sensitive hybrid loss function, guiding the policy to converge to a Pareto front with high success rate and low energy consumption; it achieves cross-modal alignment through a vision-language joint representation unit, improving the generalization ability to new tasks and objects; and it endows each joint with local autonomy through an octopus-like distributed execution architecture, maintaining task continuity through force redistribution and posture reconstruction even when a single joint fails, avoiding additional energy consumption caused by rework. The synergistic effect of the above technical means enables the robotic arm to have high energy efficiency, strong robustness, and good generalization ability in complex industrial scenarios.

[0045] All contents not described in detail in the specification are existing technologies known to those skilled in the art, and the model parameters of each electrical appliance are not specifically limited; conventional equipment can be used. Electrical control components not mentioned in this technical solution are not shown in the figures because they are existing technologies, and will not be described here.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for energy-saving control of a robot arm based on a large language model and distributed reinforcement learning, characterized in that, The process includes the following: Acquire multi-source sensor data and natural language commands, wherein the multi-source sensor data includes RGB-D images, joint torque feedback and tactile sensing signals; The natural language instructions are parsed, and an initial action strategy sequence is generated; The initial action policy sequence and the collected state-action-reward-next state quadruples are weighted and sampled to obtain a mixed sample. The policy network parameters are updated using the mixed samples and the total loss function to obtain the policy parameters after policy update. Encode the RGB-D image and extract its features; Natural language instructions are encoded and instruction embeddings are extracted; image features are fused with instruction embeddings to generate joint state representations; Based on the joint state representation, the energy consumption value of the current path is predicted to obtain the predicted energy consumption value of the current path; the predicted energy consumption value is compared with a preset threshold, and if the predicted energy consumption value exceeds the preset threshold, the action strategy is regenerated. The robotic arm execution unit executes the updated strategy parameters and the regenerated motion strategy, and controls the robotic arm based on the joint torque feedback and tactile sensing signals to achieve energy-saving control of the robotic arm.

2. The energy-saving control method for a robotic arm based on a large language model and distributed reinforcement learning according to claim 1, characterized in that, Based on the dynamic sampling weight function Weighted sampling is performed to obtain a pooled sample; where, As the initial weights, The attenuation coefficient is... t This represents the current training iteration step. T This represents the total number of training iterations. This is the dynamic sampling weight decay coefficient. Rate the difficulty of the task. This represents the energy consumption coefficient for the current action. This is the preset attenuation coefficient.

3. The energy-saving control method for a robotic arm based on a large language model and distributed reinforcement learning according to claim 1, characterized in that, The total loss function is in, To enhance the learning loss, the PPO algorithm is used to calculate the policy gradient; The intrinsic motivation loss is constructed based on the prediction error of state access frequency; This is a distributed consistency loss used to constrain the consistency of the actions output by each local Actor network in the global coordinate system. The trajectory imitation loss is used to measure the Euclidean distance between the current policy trajectory and the trajectory generated by LLM. This is a rule-constrained loss, used to penalize actions that violate kinematic boundaries or collision constraints. As an energy consumption penalty item, , To achieve the theoretically optimal energy consumption, the dynamic model of the robotic arm, constructed based on the Lagrange equation, is determined according to the joint trajectory. It is calculated from its first and second derivative integrals; The actual energy consumption is obtained by real-time sampling and integration of current and voltage sensors on the joint actuator.

4. The energy-saving control method for a robotic arm based on a large language model and distributed reinforcement learning according to claim 1, characterized in that, Encode the RGB-D image and extract its features; The natural language instructions are encoded, and instruction embeddings are extracted. Image features and instruction embeddings are then fused using a cross-attention mechanism to generate a joint state representation. The specific process includes the following steps: The RGB-D image is preprocessed, including depth map normalization, color space conversion to HSV domain, and background removal, followed by encoding and extraction of image features. Natural language instructions are tagged with part-of-speech tags and subjected to dependency parsing to extract action verbs, target objects, and spatial relation triples. The triples are mapped to semantic slot vectors to obtain instruction embeddings. The instruction embeddings are then aligned and fused with the image features through a multi-head cross-attention mechanism to generate joint state representations.

5. An energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning, characterized in that, The method for energy-saving control of a robotic arm based on a large language model and distributed reinforcement learning as described in any one of claims 1-4 includes: Natural language instruction input module: used to acquire natural language instructions; Multi-source sensing data acquisition module: used to acquire multi-source sensing data, including RGB-D images, joint torque feedback and tactile sensing signals; Semantic policy generation unit: used to parse the natural language instructions and generate an initial action policy sequence; Adaptive double-buffer sampling unit: used to perform weighted sampling on the initial action policy sequence and the collected state-action-reward-next state quadruple to obtain mixed samples; Energy-sensitive policy optimization unit: used to update policy network parameters using the mixed samples and total loss function, obtain policy parameters with updated policy, and synchronize the updated policy parameters to the octopus-like distributed execution unit; Visual-Language Joint Representation Unit: Used to encode RGB-D images and extract image features; encode natural language instructions and extract instruction embeddings; fuse image features and instruction embeddings to generate joint state representations; Gradient boosting tree model: used to predict the energy consumption value of the current path based on the joint state representation, and obtain the predicted energy consumption value of the current path; compare the predicted energy consumption value with a preset threshold, and if the predicted energy consumption value exceeds the preset threshold, trigger the semantic policy generation unit to regenerate the action policy sequence. Octopus-like distributed execution unit: used to execute the updated strategy parameters and regenerated motion strategy through the robotic arm execution unit, and control the robotic arm according to the joint torque feedback and tactile sensing signals to achieve energy-saving control of the robotic arm.

6. The energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning according to claim 5, characterized in that, The natural language command input module receives high-level task commands input by the user in natural language form through a standard API interface; the multi-source sensing data acquisition module connects to an RGB-D camera, a joint torque sensor, and a tactile array sensor to acquire multi-source sensing data, and packages the multi-source sensing data in a timestamp-aligned manner and transmits it to the vision-language joint representation unit and the octopus-like distributed execution unit; the multi-source sensing data includes RGB-D images of the environment, real-time torque feedback of each joint, and tactile pressure distribution signals of the end effector contact surface.

7. The energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning as described in claim 5, characterized in that, It also includes an LLM demonstration buffer and an RL interaction buffer. The semantic policy generation unit is connected to the natural language instruction input module. The semantic policy generation unit integrates a fine-tuned large language model. During the training phase, the large language model is supervised and fine-tuned using a paired dataset containing industrial operation instructions and corresponding action sequences. Upon receiving a natural language instruction, the semantic policy generation unit calls the large language model to parse the action verbs, target objects, and spatial relationships in the instruction, and generates an initial action policy sequence. The initial action policy sequence consists of a series of joint angle target values ​​or Cartesian space path points. The generated initial action policy sequence is written into the LLM demonstration buffer, and simultaneously associated with the corresponding natural language instruction and the current environment state snapshot to form a structured demonstration sample. Both the LLM demonstration buffer and the RL interaction buffer are circular buffer structures, deployed in different address segments of system memory. The LLM demonstration buffer stores high-confidence policy samples generated by the semantic policy generation unit, and the RL interaction buffer is filled with the state-action-reward-next state quadruple returned by the octopus-like distributed execution unit during real interaction.

8. The energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning according to claim 7, characterized in that, The adaptive double-buffered sampling unit establishes data read channels with both the LLM demonstration buffer and the RL interaction buffer; during each policy update iteration, the adaptive double-buffered sampling unit adjusts its sampling weights according to the dynamic sampling weight function. Calculate the sampling ratio of LLM samples at the current time t, where, For the preset attenuation coefficient, Total training steps The result is calculated by the task planning module based on the geometric complexity of the target object, the operating distance, and the density of obstacles. The energy-sensitive strategy optimization unit provides real-time feedback; the adaptive double-buffer sampling unit samples from the LLM demonstration buffer according to the weight function. Samples are taken from the RL interaction buffer. The samples are merged to form a mixed sample, and the mixed sample is input into the energy-sensitive strategy optimization unit.

9. The energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning according to claim 5, characterized in that, The input terminals of the visual-language joint representation unit are connected to the multi-source perception data acquisition module and the natural language command input module, respectively. The visual-language joint representation unit includes a CLIP visual encoder, an LLM semantic encoder, a semantic parsing layer, a multi-head cross-attention fusion module, and an output layer. First, the RGB-D image is processed by depth map normalization, mapping the depth values ​​to the [0,1] interval. Then, the color space is converted to the HSV domain, and a background segmentation algorithm is used to remove static background areas, retaining only the foreground operation area. Then, the processed image is input into the CLIP visual encoder to extract multi-dimensional image feature vectors. Natural language commands are sent to the LLM semantic encoder, which... A dependency parsing layer is added to the LLM framework to output action verb-target object-spatial relation triples. These triples are then mapped to semantic slot vectors via an embedding layer and input together with the multidimensional image feature vectors into a multi-head cross-attention fusion module. The multi-head cross-attention fusion module performs cross-modal alignment through multiple attention heads, ultimately outputting a multi-dimensional joint state representation. This joint state representation is simultaneously fed into an energy-sensitive policy optimization unit and an independently deployed gradient boosting tree model. The gradient boosting tree model is trained based on historical energy consumption data and is used to predict the energy consumption value of the current policy path. If the predicted value exceeds a preset threshold, the semantic policy generation unit is triggered to re-parse the instructions and generate a corrected policy.

10. The energy-saving control system for a robotic arm based on a large language model and distributed reinforcement learning according to claim 5, characterized in that, The octopus-like distributed execution unit adopts a local Actor-global Critic architecture. The hardware deployment of the octopus-like distributed execution unit is as follows: each joint of the robotic arm embeds a local Actor network, which is labeled as the local Actor network of joint 1, local Actor network of joint 2, up to local Actor network of joint n. These networks are deployed on the embedded processors of the controllers of each joint of the robotic arm; the global Critic network is deployed in the central processing unit. Each local Actor network is connected to the central processing unit, forming a star topology. The inputs to each local Actor network include the angle output from the joint encoder, the angular velocity output from the gyroscope, the torque value output from the torque sensor, and the pressure distribution map output from the tactile sensor. The outputs of each local Actor network are the joint target position increment and impedance control parameters, including damping coefficients and stiffness coefficients. The global Critic network receives state observations from all joints and a joint state representation, including angles, angular velocities, and torques. The global Critic network outputs a state value estimate. During the training phase, the local Actor networks employ an asynchronous dominant actor-critic algorithm, calculating and asynchronously updating the policy gradient based on the local reward signal and the benchmark value provided by the global Critic network. The global Critic network uses a temporal difference method, fitting the value function using multi-step rewards.

Images