Multi-finger gripper robot control method and system and electronic equipment

Abstract

The invention discloses a multi-finger gripper robot control method and system and electronic equipment, and belongs to the technical field of robot control. Synchronously triggering the camera and the force sensor to sample to obtain sampling data of different modes, fusing the sampling data of different modes by utilizing filtering, and outputting optimal state estimation after filtering; calculating a position error, a speed error and a force error based on the filtered optimal state estimation, and generating an initial joint torque command through linear gain; the optimal state estimation is utilized to obtain a control instruction after semantic adjustment is injected; solving an optimal control sequence in a quadratic optimization problem with security constraints on the basis of the control instruction after semantic adjustment is injected, and obtaining a final execution instruction; and issuing the final execution instruction to a driver for execution, collecting an execution result, and feeding back the execution result to the multi-mode perception and state estimation module to realize closed-loop iteration. According to the method, high precision and high safety of the multi-finger gripper robot in a complex environment are ensured.

Description

Technical Field

[0001] The present application belongs to the field of robot control technology, and in particular relates to a multi-finger gripper robot control method, system and electronic equipment. Background Art

[0002] In recent years, multi-finger gripper robots have been widely used in precise grasping and manipulation tasks. However, existing control schemes have the following deficiencies in multi-source information fusion and safety assurance:

[0003] The visual servo control method relies solely on the end-position measured by the camera for closed-loop adjustment, ignoring the force disturbance during object contact, which can easily lead to loss of grasp or collision; the pure force control method relies on tactile sensors to adjust the grasping force, making it difficult to achieve precise spatial alignment of the target object and sensitive to position deviation; in the separate control architecture method, visual and force feedback are separated and do not compensate each other, making it difficult for the two types of information to work together. Once one sensor fails, the control performance is greatly reduced; the traditional PID control method has limited robustness to model uncertainty and perceptual noise, and cannot quickly adapt after visual occlusion or force sensor drift; existing research only uses large-scale language models for high-level task planning, and lacks a mechanism to inject natural language reasoning results into low-level control laws in real time; safety mechanisms are lacking: they often rely on task interruption or emergency stop to deal with collision and overload risks, and lack a closed-loop strategy that integrates forward-looking prediction and safety constraints.

[0004] In response to the above problems, the present application proposes a multi-finger gripper robot control method, system and electronic equipment. Summary of the Invention

[0005] In order to address the deficiencies of the prior art, the present application provides a multi-finger gripper robot control method, system and electronic device, which can solve technical problems in the prior art such as only using large-scale language models for high-level task planning, lacking a mechanism to inject natural language reasoning results into low-level control laws in real time, and lacking a safety mechanism.

[0006] The technical effects to be achieved by this application are achieved through the following solutions:

[0007] In a first aspect, the present application provides a multi-finger gripper robot control method, comprising:

[0008] Step 1: The multimodal perception and state estimation module synchronously triggers the camera and force sensor to sample, obtains sampled data of different modalities, fuses the sampled data of different modalities using filtering and extended Kalman filtering, and outputs the filtered optimal state estimate;

[0009] Step 2: The initial vision and force fusion control law calculation module calculates the position error, velocity error, and force error based on the filtered optimal state estimate, and generates the initial joint torque command through linear gain;

[0010] Step 3: The LLM-assisted real-time parameter adjustment module uses the optimal state estimation to obtain the control instructions after the injection semantic adjustment;

[0011] Step 4: The model prediction and safety constraint optimization module solves the optimal control sequence in the quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment, and obtains the final execution instructions;

[0012] Step 5: The drive execution and closed-loop iteration module sends the final execution instruction to the driver for execution, collects the execution results, and feeds the execution results back to the multimodal perception and state estimation module in step 1 to achieve closed-loop iteration.

[0013] In some embodiments, the sampled data includes: the end position x measured by the camera t and contact force f t ; Use filtering to fuse the sampled data of different modes and output the filtered optimal state estimate, including:

[0014] Get the end position after filtering Among them, LPF(·;ω c ) represents the first-order low-pass filter function, ω c represents the cutoff frequency;

[0015] Get the corrected contact force Among them, f bias Indicates the static drift of the finger force sensor;

[0016] Construct a multimodal state vector s t , in, Indicates the terminal velocity after filtering;

[0017] Use the extended Kalman filter algorithm function to determine the optimal state estimate after filtering Where A represents the system matrix, C represents the observation matrix, Q represents the process noise covariance, R represents the observation noise covariance, EKF(·) represents the extended Kalman filter algorithm function, and u t-1 Represents the control command of the previous cycle, the optimal state estimate after filtering The component representation of is:

[0018] in, represents the end position estimate, represents the terminal velocity estimate, Represents the contact force estimate.

[0019] In some embodiments, the position error e is calculated using the following formula: x，t , speed error and force error e f，t :

[0020]

[0021] Among them, x d，t Indicates the desired position, represents the expected speed, f d，t represents the expected contact force.

[0022] In some embodiments, the initial joint torque command τ is generated by linear gain using the following formula: 0，t :

[0023]

[0024] Among them, K p , K d , K f denote the gain matrices for position, velocity, and contact force, respectively.

[0025] In some embodiments, the LLM-assisted real-time parameter adjustment module uses optimal state estimation to obtain control instructions after injection semantic adjustment, including:

[0026] Construct LLM input vector Where [·;·] represents vector concatenation, and task represents the natural language task description;

[0027] Calculate the increment Δ c , Where LLM represents the large language model, ΔK p , ΔK d , ΔK f Denote the gain matrix fine-tuning for position, velocity, and contact force, Δx d , Δf d represents the desired position fine-tuning and the desired contact force fine-tuning;

[0028] Determine the control instruction τ after injection semantic adjustment 1,t , Where x′ d，t and f′ d，t represents the expected position after LLM correction and the expected contact force after calibration; Represents the adjusted gain matrix, specifically:

[0029]

[0030] x′ d，t =x d，t +Δx d ,

[0031] f′ d，t =f d，t +Δf d .

[0032] In some embodiments, the model prediction and safety constraint optimization module solves the optimal control sequence in a quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, including:

[0033] The state at time t+1 predicted based on the information at time t where f(·) represents the dynamical mapping, w t represents process noise;

[0034] Security constraint definition C(x t , f t )for:

[0035] C(x t , f t )=max{0,||x t -x obs ||-d min ,||f t ||-f max}, where x obs Indicates the obstacle position, d min , f max They represent the distance threshold and contact force threshold respectively, ||·|| represents the norm of the vector, and if C>0, it means out of bounds.

[0036] In some embodiments, the model prediction and safety constraint optimization module solves the optimal control sequence in a quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, and further includes:

[0037] Solve the following constrained quadratic programming to obtain the optimal control sequence from time t to t+H-1:

[0038]

[0039] Among them, H represents the prediction window length, Q and R represent the state error weighting matrix and the control input weighting matrix respectively. Indicates the optimal joint torque command of the optimizer at time t+k within the prediction window, τ t+krepresents the control input at time t+k, Indicates that in all feasible {τ t , τ t+1 , τ t+2 ,...,τ t+H-1} select the set of sequences that minimize the objective function, represents the expected future state vector, whose components are the three-dimensional expected position, three-dimensional expected velocity and three-dimensional expected contact force;

[0040] The final execution instruction is

[0041] In some embodiments, implementing closed-loop iteration includes: when the completion conditions defined in the natural language task are met, the system stops the closed loop; wherein the completion conditions include: reaching a predetermined position or reaching a preset number of iterations.

[0042] In the second aspect, the present application provides a multi-finger gripper robot control system, including: a multimodal perception and state estimation module, an initial vision and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a drive execution and closed-loop iteration module. The system is used to execute any of the methods described above.

[0043] In a third aspect, the present application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the aforementioned methods when executing the computer program.

[0044] Through the multi-finger gripper robot control method, system and electronic equipment provided by this application, the end position error, velocity error and contact force error are simultaneously considered in the same control loop, complementary correction of multi-source information is achieved, natural language instructions and multimodal states are input into a large-scale language model, and dynamic adjustment suggestions for control gains and expected values ​​are generated online, realizing real-time parameter adjustment and robot control. Based on the model predictive control framework, the multimodal prediction results and safety hard constraints are jointly incorporated into the optimization problem, and potential risk commands are corrected in advance, thereby improving the accuracy of robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1This is a flow chart of a multi-finger gripper robot control method according to an embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of the structure of a multi-finger gripper robot control system in one embodiment of the present application;

[0048] Figure 3 This is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0051] Various non-limiting embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0052] First, refer to Figure 1 , the multi-finger gripper robot control method of this application is described in detail:

[0053] Step 1 (S1): The multimodal perception and state estimation module synchronously triggers the camera and force sensor to sample, obtains sampling data of different modalities, fuses the sampling data of different modalities using filtering, and outputs the filtered optimal state estimation; illustratively, a synchronous trigger signal is issued by the central coordinator.

[0054] Step 2 (S2): The initial vision and force fusion control law calculation module calculates the position error, velocity error, and force error based on the filtered optimal state estimate, and generates the initial joint torque command through linear gain;

[0055] Step 3 (S3): The LLM-assisted real-time parameter adjustment module uses the optimal state estimation to obtain the control instructions after the injection semantic adjustment;

[0056] Step 4 (S4): The model prediction and safety constraint optimization module solves the optimal control sequence in the quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions;

[0057] Step 5 (S5): The drive execution and closed-loop iteration module sends the final execution instruction to the driver for execution, collects the execution results, and feeds the execution results back to the multimodal perception and state estimation module in step 1 to achieve closed-loop iteration.

[0058] The above method simultaneously considers the end position error, velocity error and contact force error in the same control loop, realizes the complementary correction of multi-source information, inputs natural language instructions and multimodal states into the large-scale language model, generates dynamic adjustment suggestions for control gains and expected values ​​online, realizes real-time parameter adjustment and robot control, and based on the model predictive control framework, incorporates multimodal prediction results and safety hard constraints into the optimization problem, corrects potential risk commands in advance, and improves the accuracy of robot control.

[0059] In some embodiments, the sampled data includes: the end position x measured by the camera t and contact force f t ; Use filtering to fuse the sampled data of different modes and output the filtered optimal state estimate to provide accurate input for subsequent control, including:

[0060] Get the end position after filtering Among them, LPF(·;ω c ) represents the first-order low-pass filter function, ω c represents the cutoff frequency, which is obtained by experimental calibration, and can be, for example, 5 Hz. This is for illustrative purposes only, and other cutoff frequency values ​​known to those skilled in the art can also be applied here, without limitation.

[0061] Get the corrected contact force Among them, f bias It indicates the static drift of the force sensor, which can be obtained through static zero-force test;

[0062] Construct a multimodal state vector s t , in, represents the terminal velocity after filtering, calculated by finite difference;

[0063] Use the extended Kalman filter algorithm function to determine the optimal state estimate after filtering Where A represents the system matrix, C represents the observation matrix, Q represents the process noise covariance, R represents the observation noise covariance (A, C, Q, R are all obtained by offline system identification and sensor calibration), EKF(·) represents the extended Kalman filter algorithm function, u t-1 Represents the control command of the previous cycle, the optimal state estimate after filtering The component representation of is:

[0064] in, represents the end position estimate, represents the terminal velocity estimate, Represents the contact force estimate.

[0065] For example, the output here is It will serve as the input to steps 2 and 3 to ensure that the control and parameter adjustments are based on the optimal state.

[0066] In some embodiments, the position error e is calculated using the following formula: x，t , speed error and force error e f，t :

[0067]

[0068] Among them, x d，t Indicates the desired position, represents the expected speed, f d，t represents the expected contact force.

[0069] They correspond to the end position estimate, end velocity estimate, and contact force estimate output by the EKF in step 1, and are used to calculate the difference with the expected value.

[0070] x d,t , f d，t represent the desired end position, desired velocity, and desired contact force, respectively.

[0071] In some embodiments, the initial joint torque command τ is generated by linear gain using the following formula: 0，t :

[0072]

[0073] Among them, Kp , K d , K f They represent the gain matrices for position, velocity, and contact force, respectively, and can be obtained by offline gain tuning or online adaptive algorithm.

[0074] τ 0，t The preliminary control instructions without semantic injection or optimization will be adjusted by LLM in step 3 and optimized by MPC in step 4.

[0075] For example, the desired position x d，t , expected speed and the expected contact force f d，t They are generated by the task planning module. When the task is initialized or switched, the module calculates the expected state sequence of the entire process based on the natural language task description and environmental geometry information through cubic spline interpolation or optimal control algorithm, and then sends it to the central coordinator.

[0076] In some embodiments, the LLM-assisted real-time parameter adjustment module uses optimal state estimation to obtain control instructions after injection semantic adjustment, including:

[0077] Construct LLM input vector Where [·;·] represents vector concatenation, and task represents the natural language task description;

[0078] Calculate the increment Δ c , Where LLM represents the large language model, ΔK p , ΔK d , ΔK f Denote the gain matrix fine-tuning for position, velocity, and contact force, Δx d , Δf d represents the desired position fine-tuning and the desired contact force fine-tuning;

[0079] Determine the control instruction τ after injection semantic adjustment 1,t , Where x′ d，t and f′ d，t represents the expected position after LLM correction and the expected contact force after calibration; Represents the adjusted gain matrix, specifically:

[0080]

[0081] x′ d，t =x d，t +Δx d ,

[0082] f′ d，t =fd，t +Δf d .

[0083] For example, τ 1，t The input to step 4 is used as the input for MPC optimization to balance tracking accuracy, command smoothness, and safety constraints.

[0084] In some embodiments, the model prediction and safety constraint optimization module solves the optimal control sequence in a quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, including:

[0085] The state at time t+1 predicted based on the information at time t where f(·) represents the dynamical mapping, which comes from system identification, and w t represents process noise, noise The covariance W was determined by offline statistical analysis.

[0086] Security constraint definition C(x t , f t )for:

[0087] C(x t , f t )=max{0,||x t -x obs ||-d min ,||f t ||-f max}, where x obs Indicates the obstacle position, d min , f max They represent the distance threshold and contact force threshold respectively, ||·|| represents the norm of the vector, that is, the modulus of the calculated vector. If C>0, it means out of bounds.

[0088] For example, v represents the terminal velocity, which is the terminal velocity estimate in the state vector component Or predicted speed Similarly, both are speed-related parameters.

[0089] Exemplarily, the optimal sequence definition in the MPC optimization problem is: first, the optimal control sequence from time t to t+H-1 is obtained by solving the following constrained quadratic programming.

[0090] In some embodiments, the model prediction and safety constraint optimization module solves the optimal control sequence in a quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, and further includes:

[0091] Solve the following constrained quadratic programming to obtain the optimal control sequence from time t to t+H-1:

[0092]

[0093] Among them, H represents the prediction window length, Q and R represent the state error weighting matrix and the control input weighting matrix respectively. Indicates the optimal joint torque command of the optimizer at time t+k within the prediction window, τ t+k represents the control input at time t+k, Indicates that in all feasible {τ t , τ t+1 , τ t+2 ,...,τ t+H-1} select the set of sequences that minimize the objective function, represents the expected future state vector, whose components are the three-dimensional expected position, three-dimensional expected velocity and three-dimensional expected contact force;

[0094] represents the weighted quadratic norm of the matrix M, which measures the squared length of the vector y under the metric defined by the matrix M4.

[0095] The final execution instruction is The final execution instruction represents the optimal joint torque actually executed in this control cycle.

[0096] The driver execution and closed-loop iteration module sends the final execution instruction to the driver for execution, collects the execution results, and feeds the execution results back to the multimodal perception and state estimation module in step 1 to achieve closed-loop iteration, including:

[0097] Command issued: The central coordinator will Sent to each joint driver via a high-speed bus.

[0098] Motion feedback sampling, after execution, the camera and force sensor resample to obtain x t+1 、 f t+1 Wait for the execution result, where x t+1 represents the end position at time t+1, represents the terminal velocity at time t+1, f t+1 Represents the contact force at time t+1, and returns the execution result to step 1 to achieve closed-loop iteration.

[0099] In some embodiments, implementing closed-loop iteration includes: when the completion conditions defined in the natural language task are met, the system stops the closed loop; wherein the completion conditions include: reaching a predetermined position or reaching a preset number of iterations.

[0100] This application realizes multi-source information fusion control of vision-force-language-prediction-safety, ensuring the high precision and high safety of the multi-finger gripper robot in complex environments.

[0101] This application provides a multi-finger gripper robot control system (such as Figure 2 As shown), it includes: a multimodal perception and state estimation module, an initial vision and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a drive execution and closed-loop iteration module. The system is used to execute any of the methods described above.

[0102] The above modules exchange information in real time via a high-speed data bus.

[0103] For example, the function of the multimodal perception and state estimation module (Perception & Estimation) is to synchronously collect data from the camera and force sensor, fuse them using a filtering algorithm (such as extended Kalman filter), and output the optimal state estimate.

[0104] For example, the function of the initial vision and force fusion control law calculation module (Primary Controller) is to calculate the position error, velocity error and force error based on the estimated state and the desired trajectory, and generate the initial joint torque command through linear gain matrix weighting.

[0105] Exemplarily, the functions of the LLM-assisted real-time parameter adjustment module (LLM Tuner) are: constructing the multimodal state vector and the natural language task description into LLM input, obtaining the gain and expected value adjustment increments, and updating the control parameters.

[0106] For example, the function of the model prediction and safety constraint optimization module (MPC Optimizer) is to predict the future multi-step state based on the discrete dynamics model and the corrected command, and solve the optimal control sequence with safety hard constraints online.

[0107] For example, the functions of the drive execution and closed-loop iteration module (Actuation & Feedback) are: sending the final control command to the driver, executing the movement, and feeding back the execution result to the perception module through the sensor to enter the next control cycle.

[0108] The system's central coordinator is responsible for synchronizing the startup sequence and data of each module, ensuring end-to-end latency within milliseconds. At each control time t, the system sequentially executes the five steps of the multi-finger gripper robot control method.

[0109] The above system can achieve all the technical effects of the multi-finger gripper robot control method, which will not be repeated here.

[0110] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0111] It should be noted that the above description is of specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Based on the same inventive concept, corresponding to any of the above embodiments and methods, the present application also discloses an electronic device;

[0113] Specifically, Figure 3 The hardware structure diagram of an electronic device for a multi-finger gripper robot control method provided in this embodiment is shown. The device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other within the device via the bus 450.

[0114] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0115] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 420 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present application are implemented through software or firmware, the relevant program codes are stored in the memory 420 and are called and executed by the processor 410.

[0116] The input / output interface 430 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0117] The communication interface 440 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (e.g., USB, network cable, etc.) or a wireless method (e.g., mobile network, Wi-Fi, Bluetooth, etc.).

[0118] The bus 450 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 ).

[0119] It should be noted that although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440, and the bus 450, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in the figure.

[0120] The electronic device of the above embodiment is used to implement the corresponding multi-finger gripper robot control method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0121] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-finger gripper robot control method described in any of the above embodiments.

[0122] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0123] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multi-finger gripper robot control method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0125] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the following fact, that is, the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0126] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0127] The one or more embodiments of the present application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the one or more embodiments of the present application shall be included within the scope of protection of the present application.

Claims

1. A multi-finger gripper robot control method, characterized in that: include: Step 1: The multimodal perception and state estimation module synchronously triggers the camera and force sensor to sample, obtains sampled data of different modalities, fuses the sampled data of different modalities using filtering, and outputs the filtered optimal state estimate; Step 2: The initial vision and force fusion control law calculation module calculates the position error, velocity error, and force error based on the filtered optimal state estimate, and generates the initial joint torque command through linear gain; Step 3: The LLM-assisted real-time parameter adjustment module uses the optimal state estimation to obtain the control instructions after the injection semantic adjustment; Step 4: The model prediction and safety constraint optimization module solves the optimal control sequence in the quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment, and obtains the final execution instructions; Step 5: The drive execution and closed-loop iteration module sends the final execution instruction to the driver for execution, collects the execution results, and feeds the execution results back to the multimodal perception and state estimation module in step 1 to achieve closed-loop iteration.

2. The multi-finger gripper robot control method according to claim 1, characterized in that: The sampling data includes: the end position x measured by the camera t and contact force f t ; Use filtering to fuse the sampled data of different modes and output the filtered optimal state estimate, including: Get the end position after filtering Among them, LPF(·;ω c ) represents the first-order low-pass filter function, ω c represents the cutoff frequency; Get the corrected contact force Among them, f bias Indicates the static drift of the finger force sensor; Construct a multimodal state vector s t , in, Indicates the terminal velocity after filtering; Use the extended Kalman filter algorithm function to determine the optimal state estimate after filtering Where A represents the system matrix, C represents the observation matrix, Q represents the process noise covariance, R represents the observation noise covariance, EKF(·) represents the extended Kalman filter algorithm function, u t-1 Represents the control command of the previous cycle, the optimal state estimate after filtering The component representation of is: in, represents the end position estimate, represents the terminal velocity estimate, Represents the contact force estimate.

3. The multi-finger gripper robot control method according to claim 2, characterized in that: The position error e is calculated using the following formula: x，t , speed error and force error e f，t : Among them, x d，t Indicates the desired position, represents the expected speed, f d，t represents the expected contact force.

4. The multi-finger gripper robot control method according to claim 3, characterized in that: The initial joint torque command τ is generated by linear gain using the following formula: 0，t : Among them, K p , K d , K f denote the gain matrices for position, velocity, and contact force, respectively.

5. The multi-finger gripper robot control method according to claim 4, characterized in that: The LLM-assisted real-time parameter adjustment module uses optimal state estimation to obtain control instructions after injection semantic adjustment, including: Construct LLM input vector Where [·;·] represents vector concatenation, and task represents the natural language task description; Calculate the increment Δ c , Where LLM represents the large language model, ΔK p , ΔK d , ΔK f Denote the gain matrix fine-tuning for position, velocity, and contact force, Δx d , Δf d represents the desired position fine-tuning and the desired contact force fine-tuning; Determine the control instruction τ after injection semantic adjustment 1,t , Where x′ d，t and f′ d，t represents the expected position after LLM correction and the expected contact force after calibration; Represents the adjusted gain matrix, specifically: x′ d，t =x d，t +Δx d , f′ d，t =f d，t +Δf d 。 6. The multi-finger gripper robot control method according to claim 5, characterized in that: The model prediction and safety constraint optimization module solves the optimal control sequence in the quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, including: The state at time t+1 predicted based on the information at time t where f(·) represents the kinetic map, w t represents process noise; Security constraint definition C(x t , f t )for: C(x t , f t )=max{0,||x t -x obs ||-d min ,||f t ||-f max }, where x obs Indicates the obstacle position, d min , f max They represent the distance threshold and contact force threshold respectively, ||·|| represents the norm of the vector, and if C>0, it means out of bounds.

7. The multi-finger gripper robot control method according to claim 6, characterized in that: The model prediction and safety constraint optimization module solves the optimal control sequence in the quadratic optimization problem with safety constraints based on the control instructions after injection semantic adjustment to obtain the final execution instructions, and also includes: Solve the following constrained quadratic programming to obtain the optimal control sequence from time t to t+H-1: Among them, H represents the prediction window length, Q and R represent the state error weighting matrix and the control input weighting matrix respectively. Indicates the optimal joint torque command of the optimizer at time t+k within the prediction window, τ t+k represents the control input at time t+k, Indicates that in all feasible {τ t , τ t+1 , τ t+2 ,...,τ t+H-1 } select the set of sequences that minimize the objective function, represents the expected future state vector, whose components are the three-dimensional expected position, three-dimensional expected velocity and three-dimensional expected contact force; The final execution instruction is 8. The multi-finger gripper robot control method according to claim 7, characterized in that: Implementing closed-loop iteration includes: when the completion conditions defined in the natural language task are met, the system stops the closed loop; wherein the completion conditions include: reaching a predetermined position or reaching a preset number of iterations.

9. A multi-finger gripper robot control system, characterized in that: include: A multimodal perception and state estimation module, an initial vision and force fusion control law calculation module, an LLM-assisted real-time parameter adjustment module, a model prediction and safety constraint optimization module, and a drive execution and closed-loop iteration module. The system is used to execute the method described in any one of claims 1 to 8.

10. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images