A multi-fingered gripper robot control method, system and electronic device

Abstract

The application discloses a multi-fingered gripper robot control method, system and electronic equipment, and belongs to the technical field of robot control. A camera and a force sensor are synchronously triggered to sample and obtain sampling data of different modes, the sampling data of different modes is fused by filtering, and optimal state estimation after filtering is output; based on the optimal state estimation after filtering, position error, velocity error and force error are calculated, and an initial joint torque command is generated through linear gain; by using the optimal state estimation, a control instruction after semantic injection adjustment is obtained; based on the control instruction after semantic injection adjustment, an optimal control sequence is solved in a quadratic optimization problem with safety constraints to obtain a final execution instruction; the final execution instruction is issued to a driver for execution, and an execution result is collected, and the execution result is fed back to a multi-modal perception and state estimation module to realize closed-loop iteration. The application ensures high precision and high safety of the multi-fingered gripper robot in a complex environment.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, and specifically relates to a control method, system and electronic device for a multi-fingered gripper robot. Background Technology

[0002] In recent years, multi-fingered gripper robots have been widely used in precision grasping and manipulation tasks, but existing control schemes have the following shortcomings in terms of multi-source information fusion and safety assurance:

[0003] Visual servo control methods rely solely on camera-measured end-effector pose for closed-loop adjustment, ignoring force disturbances during object contact, which can easily lead to loss of grip or collision. Pure force control methods rely on tactile sensors to adjust the gripping force, making it difficult to achieve precise spatial alignment of the target object and sensitive to positional deviations. In separate control architectures, visual and force feedback are isolated and do not compensate for each other, making it difficult for the two types of information to work together. Once one sensor fails, control performance drops significantly. Traditional PID control methods have limited robustness to model uncertainties and perceived noise, and cannot quickly adapt after visual occlusion or force sensor drift. Existing research only uses large-scale language models for high-level task planning, lacking a mechanism to inject natural language inference results into low-level control laws in real time. Safety mechanisms are lacking: they rely heavily on task interruption or emergency stops to deal with collision and overload risks, lacking a closed-loop strategy that integrates forward prediction and safety constraints.

[0004] To address the aforementioned issues, this application proposes a control method, system, and electronic device for a multi-fingered gripper robot. Summary of the Invention

[0005] To address the shortcomings of the prior art, this application provides a multi-fingered gripper robot control method, system, and electronic device. This method can solve the technical problems in the prior art, such as only using large-scale language models for high-level task planning, lacking a mechanism to inject natural language reasoning results into low-level control laws in real time, and lacking security mechanisms.

[0006] The technical effect to be achieved in this application is accomplished through the following solution:

[0007] Firstly, this application provides a control method for a multi-fingered gripper robot, comprising:

[0008] Step 1: The multimodal perception and state estimation module synchronously triggers the camera and force sensor to sample, obtains sampling data of different modes, and uses filtering and extended Kalman filtering to fuse the sampling data of different modes and output the filtered optimal state estimate.

[0009] Step 2: The initial vision and force fusion control law calculation module calculates the position error, velocity error and force error based on the filtered optimal state estimate, and generates the initial joint torque command through linear gain;

[0010] Step 3: The LLM-assisted real-time parameter adjustment module uses optimal state estimation to obtain the control commands after semantic adjustment.

[0011] Step 4: The model prediction and safety constraint optimization module solves for the optimal control sequence in a quadratic optimization problem with safety constraints based on the injected semantically adjusted control instructions, and obtains the final execution instructions;

[0012] Step 5: The driver execution and closed-loop iteration module sends the final execution instruction to the driver for execution and collects the execution results. The execution results are then fed back to the multimodal perception and state estimation module in Step 1 to achieve closed-loop iteration.

[0013] In some embodiments, the sampling data includes: the end position x measured by the camera. t and contact force f t The system utilizes filtering to fuse sampled data from different modes, outputting the filtered optimal state estimate, including:

[0014] Obtain the end position after filtering Wherein, LPF(·; ω c ) represents a first-order low-pass filter function, ω c Indicates the cutoff frequency;

[0015] Obtain the corrected contact force Among them, f bias This indicates static drift of the finger force sensor;

[0016] Constructing the multimodal state vector s t , in, Indicates the final velocity after filtering;

[0017] Determining the optimal state estimate after filtering using the extended Kalman filter algorithm function. Where A represents the system matrix, C represents the observation matrix, Q represents the process noise covariance, R represents the observation noise covariance, EKF(·) represents the extended Kalman filter algorithm function, and u t-1 This represents the optimal state estimate after filtering, based on the control command from the previous cycle. The component representation is as follows:

[0018] in, This indicates the estimated end position. This indicates the terminal velocity estimate. This indicates the contact force estimate.

[0019] In some embodiments, the position error e is calculated using the following formula. x，t Speed ​​error Sum of force error e f，t :

[0020]

[0021] Where, x d，t Indicates the desired position. f represents the desired velocity. d，t It indicates the desired contact force.

[0022] In some embodiments, the initial joint torque command τ is generated using the following formula via linear gain. 0，t :

[0023]

[0024] Among them, K p K d K f These represent the gain matrices for position, velocity, and contact force, respectively.

[0025] In some embodiments, the LLM-assisted real-time parameter adjustment module utilizes optimal state estimation to obtain control commands after semantic adjustment, including:

[0026] Constructing LLM input vectors Where [·;·] represents vector concatenation, and task represents a natural language task description;

[0027] Calculate the increment Δ c , Where LLM represents a large language model, ΔK p ΔK d ΔK f These represent fine-tuning of the gain matrix for position, velocity, and contact force, respectively, Δx d , Δf d This indicates the desired position fine-tuning and desired contact force fine-tuning;

[0028] Determine the control command τ after semantic adjustment. 1,t , Where, x′ d，t and f′ d，t This represents the desired position after LLM correction and the desired contact force after calibration; This represents the adjusted gain matrix, specifically:

[0029]

[0030] x′ d，t =x d，t +Δx d ,

[0031] f′ d，t =f d，t +Δf d .

[0032] In some embodiments, the model prediction and safety constraint optimization module solves for the optimal control sequence in a quadratic optimization problem with safety constraints based on the injected semantically adjusted control instructions to obtain the final execution instructions, including:

[0033] Predicted state at time t+1 based on time t information Where f(·) represents the dynamic mapping, w t Indicates process noise;

[0034] Security constraint definition C(x) t f t )for:

[0035] C(x t f t )=max{0,||x t -x obs ||-d min , ||f t ||-f max}, where x obs Indicates the location of the obstacle, d min f max Let C represent the distance threshold and the contact force threshold, respectively, and ||·|| represent the norm of the vector. If C>0, it indicates that the vector is out of bounds.

[0036] In some embodiments, the model prediction and safety constraint optimization module, based on the injected semantically adjusted control instructions, solves for the optimal control sequence in a quadratic optimization problem with safety constraints to obtain the final execution instructions, and further includes:

[0037] Solve the following constrained quadratic programming problem to obtain the optimal control sequence from time t to t+H-1:

[0038]

[0039] Where H represents the prediction window length, and Q and R represent the state error weighting matrix and the control input weighting matrix, respectively. This represents the optimizer's optimal joint torque command at time t+k within the prediction window. t+kThis represents the control input at time t+k. It means that in all feasible {τ} t , τ t+1 , τ t+2 ,...,τ t+H-1 Choose the sequence that minimizes the objective function from the given set. This represents the expected future state vector, with components being the expected three-dimensional position, expected three-dimensional velocity, and expected three-dimensional contact force.

[0040] The final execution instruction is

[0041] In some embodiments, implementing closed-loop iteration includes: stopping the closed loop when the completion conditions defined in the natural language task are met; wherein the completion conditions include: reaching a predetermined position or reaching a preset number of iterations.

[0042] Secondly, this application provides a multi-fingered gripper robot control system, including: a multimodal perception and state estimation module, an initial vision and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a drive execution and closed-loop iteration module, wherein the system is used to execute any of the methods described above.

[0043] Thirdly, this application provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the aforementioned methods.

[0044] The multi-finger gripper robot control method, system, and electronic equipment provided in this application simultaneously consider end-effector position error, velocity error, and contact force error within the same control loop, achieving complementary correction of multi-source information. Natural language commands and multimodal states are input into a large-scale language model, generating dynamic adjustment suggestions for control gain and expected value online, realizing real-time parameter adjustment and robot control. Based on the model predictive control framework, multimodal prediction results and safety hard constraints are incorporated into the optimization problem, correcting potential risk commands in advance and improving the accuracy of robot control. Attached Figure Description

[0045] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1This is a flowchart of a multi-fingered gripper robot control method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the structure of a multi-fingered gripper robot control system according to one embodiment of this application;

[0048] Figure 3 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in one or more embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0051] The various non-limiting embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0052] First, refer to Figure 1 The control method for the multi-fingered gripper robot of this application is described in detail below:

[0053] Step 1 (S1): The multimodal perception and state estimation module synchronously triggers the camera and force sensor to sample, obtains sampling data of different modes, uses filtering to fuse the sampling data of different modes, and outputs the filtered optimal state estimate; for example, the central coordinator sends a synchronous trigger signal.

[0054] Step 2 (S2): The initial vision and force fusion control law calculation module calculates the position error, velocity error and force error based on the filtered optimal state estimate, and generates the initial joint torque command through linear gain;

[0055] Step 3 (S3): The LLM-assisted real-time parameter adjustment module uses optimal state estimation to obtain the control command after semantic adjustment.

[0056] Step 4 (S4): The model prediction and safety constraint optimization module solves for the optimal control sequence in a quadratic optimization problem with safety constraints based on the injected semantically adjusted control instructions, and obtains the final execution instructions;

[0057] Step 5 (S5): The drive execution and closed-loop iteration module sends the final execution instruction to the driver for execution and collects the execution results. The execution results are then fed back to the multimodal perception and state estimation module in Step 1 to achieve closed-loop iteration.

[0058] The above method considers end-effector position error, velocity error, and contact force error simultaneously within the same control loop, achieving complementary correction of multi-source information. It inputs natural language commands and multimodal states into a large-scale language model, generating dynamic adjustment suggestions for control gain and expected value online, thus realizing real-time parameter adjustment and robot control. Based on the model predictive control framework, it incorporates multimodal prediction results and safety hard constraints into the optimization problem, correcting potential risk commands in advance and improving the accuracy of robot control.

[0059] In some embodiments, the sampling data includes: the end position x measured by the camera. t and contact force f t By filtering and fusing sampled data from different modes, the filtered optimal state estimate is output, providing accurate input for subsequent control. Specifically, this includes:

[0060] Obtain the end position after filtering Wherein, LPF(·; ω c ) represents a first-order low-pass filter function, ω c This indicates the cutoff frequency, which is obtained through experimental calibration. For example, it can be 5Hz. This is just an example, and other cutoff frequency values ​​known to those skilled in the art can also be applied here without limitation.

[0061] Obtain the corrected contact force Among them, f bias This indicates the static drift of the finger force sensor, which can be obtained through a static zero-force test.

[0062] Constructing the multimodal state vector s t , in, The final velocity after filtering is calculated using finite difference.

[0063] Determining the optimal state estimate after filtering using the extended Kalman filter algorithm function. Where A represents the system matrix, C represents the observation matrix, Q represents the process noise covariance, R represents the observation noise covariance (A, C, Q, and R are all obtained from offline system identification and sensor calibration), EKF(·) represents the extended Kalman filter algorithm function, u t-1 This represents the optimal state estimate after filtering, based on the control command from the previous cycle. The component representation is as follows:

[0064] in, This indicates the estimated end position. This indicates the terminal velocity estimate. This indicates the contact force estimate.

[0065] For example, the output here This will be used as input for steps 2 and 3 to ensure that control and parameter adjustments are based on the optimal state.

[0066] In some embodiments, the position error e is calculated using the following formula. x，t Speed ​​error Sum of force error e f，t :

[0067]

[0068] Where, x d，t Indicates the desired position. f represents the desired velocity. d，t It indicates the desired contact force.

[0069] These correspond to the end position estimate, end velocity estimate, and contact force estimate output by EKF in step 1, respectively, and are used to calculate the difference with the expected value.

[0070] x d,t , f d，t These represent the desired end position, desired velocity, and desired contact force, respectively.

[0071] In some embodiments, the initial joint torque command τ is generated using the following formula via linear gain. 0，t :

[0072]

[0073] Among them, Kp K d K f These represent the gain matrices for position, velocity, and contact force, respectively, which can be obtained through offline gain tuning or online adaptive algorithms.

[0074] τ 0，t The initial control instructions, which are not injected with semantics or optimized, will be adjusted by LLM in step 3 and optimized by MPC in step 4.

[0075] For example, the desired position x d，t Expected speed Contact force f d，t These are generated by the task planning module. During task initialization or switching, this module calculates the expected state sequence for the entire process using cubic spline interpolation or optimal control algorithms based on the natural language task description and environmental geometric information, and then distributes it to the central coordinator.

[0076] In some embodiments, the LLM-assisted real-time parameter adjustment module utilizes optimal state estimation to obtain control commands after semantic adjustment, including:

[0077] Constructing LLM input vectors Where [·;·] represents vector concatenation, and task represents a natural language task description;

[0078] Calculate the increment Δ c , Where LLM represents a large language model, ΔK p ΔK d ΔK f These represent fine-tuning of the gain matrix for position, velocity, and contact force, respectively, Δx d , Δf d This indicates the desired position fine-tuning and desired contact force fine-tuning;

[0079] Determine the control command τ after semantic adjustment. 1,t , Where, x′ d，t and f′ d，t This represents the desired position after LLM correction and the desired contact force after calibration; This represents the adjusted gain matrix, specifically:

[0080]

[0081] x′ d，t =x d，t +Δx d ,

[0082] f′ d，t =fd，t +Δf d .

[0083] For example, τ 1，t Step 4 will be used as input for MPC optimization to balance tracking accuracy, command smoothness, and safety constraints.

[0084] In some embodiments, the model prediction and safety constraint optimization module solves for the optimal control sequence in a quadratic optimization problem with safety constraints based on the injected semantically adjusted control instructions to obtain the final execution instructions, including:

[0085] Predicted state at time t+1 based on time t information Where f(·) represents the dynamic mapping, derived from system identification, w t This represents process noise, noise The covariance W was determined by offline statistical analysis.

[0086] Security constraint definition C(x) t f t )for:

[0087] C(x t f t )=max{0,||x t -x obs ||-d min , ||f t ||-f max}, where x obs Indicates the location of the obstacle, d min f max Let C represent the distance threshold and the contact force threshold, respectively. Let ||·|| represent the norm of the vector, i.e., the magnitude of the vector. If C>0, it indicates that the vector is out of bounds.

[0088] For example, v represents the end velocity, compared with the end velocity estimate in the state vector components. Or predict speed Similarly, they are all parameters related to speed.

[0089] For example, the optimal sequence in the MPC optimization problem is defined as follows: First, by solving the following constrained quadratic programming problem, the optimal control sequence from time t to t+H-1 is obtained.

[0090] In some embodiments, the model prediction and safety constraint optimization module, based on the injected semantically adjusted control instructions, solves for the optimal control sequence in a quadratic optimization problem with safety constraints to obtain the final execution instructions, and further includes:

[0091] Solve the following constrained quadratic programming problem to obtain the optimal control sequence from time t to t+H-1:

[0092]

[0093] Where H represents the prediction window length, and Q and R represent the state error weighting matrix and the control input weighting matrix, respectively. This represents the optimizer's optimal joint torque command at time t+k within the prediction window. t+k This represents the control input at time t+k. It means that in all feasible {τ} t , τ t+1 , τ t+2 ,...,τ t+H-1 Choose the sequence that minimizes the objective function from the given set. This represents the expected future state vector, with components being the expected three-dimensional position, expected three-dimensional velocity, and expected three-dimensional contact force.

[0094] Let represent the weighted quadratic norm of matrix M, which measures the squared length of vector y under the metric defined by matrix M.

[0095] The final execution instruction is The final execution instruction represents the optimal joint torque actually executed in this control cycle.

[0096] The drive execution and closed-loop iteration module sends the final execution instruction to the driver for execution and collects the execution results. The execution results are then fed back to the multimodal perception and state estimation module in step 1 to achieve closed-loop iteration, including:

[0097] Command issued: The central coordinator will The data is transmitted to each joint driver via a high-speed bus.

[0098] Motion feedback sampling is performed, and afterward, the camera and force sensor resample to obtain x. t+1 , f t+1 Wait for the execution result, where x t+1 This indicates the position of the end point at time t+1. f represents the terminal velocity at time t+1. t+1 This represents the contact force at time t+1, and the execution result is returned to step 1 to achieve closed-loop iteration.

[0099] In some embodiments, implementing closed-loop iteration includes: stopping the closed loop when the completion conditions defined in the natural language task are met; wherein the completion conditions include: reaching a predetermined position or reaching a preset number of iterations.

[0100] This application realizes multi-source information fusion control of vision, force, language, prediction and safety, ensuring high precision and high safety of multi-fingered gripper robots in complex environments.

[0101] This application provides a multi-fingered gripper robot control system (such as...). Figure 2 As shown, the system includes: a multimodal perception and state estimation module, an initial vision and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a drive execution and closed-loop iteration module. The system is used to execute any of the methods described above.

[0102] The modules mentioned above exchange information in real time via a high-speed data bus.

[0103] For example, the function of the multimodal perception and state estimation module is to simultaneously acquire data from the camera and force sensor, fuse the data using a filtering algorithm (such as extended Kalman filter), and output the optimal state estimate.

[0104] For example, the function of the initial vision and force fusion control law calculation module (Primary Controller) is to calculate the position error, velocity error and force error based on the estimated state and the desired trajectory, and generate the initial joint torque command by weighting it with a linear gain matrix.

[0105] For example, the function of the LLM-assisted real-time parameter tuning module (LLM Tuner) is to construct the multimodal state vector and the natural language task description into an LLM input, obtain the gain and expected value adjustment increment, and update the control parameters.

[0106] For example, the Model Prediction and Safety Constraint Optimizer (MPC Optimizer) module functions to predict future multi-step states based on the discrete dynamics model and the corrected commands, and solve the optimal control sequence with safety hard constraints online.

[0107] For example, the function of the drive execution and closed-loop iteration module (Actuation & Feedback) is to issue the final control command to the driver, execute the motion, and feed back the execution result to the sensing module through the sensor to enter the next control cycle.

[0108] The above system is managed by a central coordinator, which is responsible for the startup sequence and data synchronization of each module, ensuring that the end-to-end latency is within the millisecond range. At each control time t, the system executes the five steps of the multi-finger gripper robot control method in sequence.

[0109] The above system can achieve all the technical effects of multi-finger gripper robot control methods, which will not be elaborated here.

[0110] It should be noted that the methods of one or more embodiments of this application can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this application, and the multiple devices will interact with each other to complete the method described.

[0111] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Based on the same inventive concept, and corresponding to any of the above-described embodiments, this application also discloses an electronic device;

[0113] Specifically, Figure 3 This diagram illustrates the hardware structure of an electronic device for a multi-fingered gripper robot control method provided in this embodiment. The device may include a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, memory 420, input / output interface 430, and communication interface 440 are interconnected internally via the bus 450.

[0114] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0115] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0116] Input / output interface 430 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0117] The communication interface 440 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable, etc.) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth, etc.).

[0118] Bus 450 includes a pathway for transmitting information between various components of the device (e.g., processor 410, memory 420, input / output interface 430, and communication interface 440).

[0119] It should be noted that although the above-described device only shows the processor 410, memory 420, input / output interface 430, communication interface 440, and bus 450, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0120] The electronic devices described above are used to implement the corresponding multi-finger gripper robot control methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0121] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, one or more embodiments of this application also provide a computer-readable storage medium storing computer instructions for causing the computer to execute the multi-finger gripper robot control method as described in any of the above embodiments.

[0122] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0123] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the multi-finger gripper robot control method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0125] Additionally, to simplify the description and discussion, and to avoid obscuring one or more embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring one or more embodiments of this application, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which one or more embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) are set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that one or more embodiments of this application may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0126] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0127] One or more embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of this application.

Claims

1. A multi-fingered gripper robot control method, characterized by, The method comprises the following steps: Step 1: A multi-modal perception and state estimation module synchronously triggers a camera and a force sensor to sample, obtains sampling data of different modalities, fuses the sampling data of different modalities by filtering, and outputs filtered optimal state estimation; Step 2: An initial visual and force fusion control law calculation module calculates position error, velocity error and force error based on the filtered optimal state estimation, and generates an initial joint torque command through a linear gain; Step 3: An LLM-assisted real-time parameter adjustment module obtains a control instruction adjusted by injecting semantics by using the optimal state estimation; Step 4: A model prediction and safety constraint optimization module solves an optimal control sequence in a quadratic optimization problem with safety constraints based on the control instruction adjusted by injecting semantics, and obtains a final execution instruction; Step 5: A driving execution and closed-loop iteration module feeds the final execution instruction to a driver for execution, collects execution results, and feeds the execution results back to the multi-modal perception and state estimation module in step 1 to realize closed-loop iteration. where the camera measures the end position and the contact force ; fusing the sampled data of different modalities with filtering to output filtered optimal state estimation, comprising: Obtaining filtered end position , wherein denotes a first order low pass filter function, denotes a cut-off frequency; Obtaining corrected contact force , wherein represents the finger force sensor static drift; Constructing a multi-modal state vector , wherein, denotes the filtered end velocity; determining a filtered optimal state estimate using an extended Kalman filter algorithm function , wherein, represents a system matrix, represents an observation matrix, represents a process noise covariance, represents an observation noise covariance, represents an extended Kalman filter algorithm function, represents a previous cycle control command, a filtered optimal state estimate is represented in component form as: , represents an end position estimate, represents an end velocity estimate, represents a contact force estimate; The position error is calculated using the following equation , the velocity error , and the force error : ， ， ， wherein, represents a desired position, represents a desired velocity, desired contact force; Generating initial joint torque commands by linear gains : ， wherein respectively represent gain matrix; The LLM-assisted real-time parameter adjustment module obtains a control instruction adjusted by injecting semantics by using the optimal state estimation, and comprises the following steps: Constructing an LLM input vector , wherein denotes vector concatenation, denotes a natural language task description; Computing the increment , where The gain matrix is fine-tuned for position, velocity and contact force, denotes the desired position fine-tuning and the desired contact force fine-tuning; Determine the injection semantics adjusted control instruction wherein, represents the LLM corrected desired position and calibrated desired contact force; , represents the adjusted gain matrix, in particular: ， ， ， ， 。 2. The multi-fingered hand robot control method of claim 1, wherein, The model prediction and safety constraint optimization module solves an optimal control sequence in a quadratic optimization problem with safety constraints based on the control instruction adjusted by injecting semantics, and obtains a final execution instruction, and comprises the following steps: State at time t+1 predicted based on time t information , where Kinetic mapping, denotes process noise; Security constraint definition is: wherein, represents an obstacle position, respectively represent a distance threshold and a contact force threshold, represents a violation, if represents a violation.

3. The multi-fingered hand robot control method of claim 2, wherein, The model prediction and safety constraint optimization module solves an optimal control sequence in a quadratic optimization problem with safety constraints based on the control instruction adjusted by injecting semantics, and obtains a final execution instruction, and further comprises the following steps: Solve the following constrained quadratic program to obtain the optimal control sequence to at time ， wherein, denotes the prediction window length denote the state error weighting matrix and the control input weighting matrix, respectively, Within the prediction window, the optimizer solves for the optimal joint torque command, at time t + k, denotes the control input at time t + k, denotes the set of sequences that minimizes the objective function among all feasible sequences, denotes the desired future state vector, with components being the desired three-dimensional position, the desired three-dimensional velocity, and the desired three-dimensional contact force, respectively; The final execution instruction is .

4. The multi-fingered hand robot control method of claim 3, wherein, The closed-loop iteration comprises: when a completion condition defined in a natural language task is met, the system stops the closed loop; wherein the completion condition comprises: reaching a predetermined position or reaching a preset number of iterations.

5. A multi-fingered gripper robot control system, characterized by, The system comprises a multi-modal perception and state estimation module, an initial visual and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a driving execution and closed-loop iteration module, and is used to execute the method according to any one of claims 1 to 4. A memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 4.

6. An electronic device, comprising: ​

Images