Mobile manipulator car-arm coordinated visual predictive control method based on multi-objective optimization

Abstract

The application relates to the field of robot motion control and discloses a mobile manipulator vehicle-arm coordination visual predictive control method based on multi-target optimization, which comprises the following steps: taking motion impact as a control input, establishing a linearized state space motion model of a mobile manipulator system predictive control; designing a mobile manipulator vehicle-arm coordination motion constraint model based on operability optimization, improving the speed operability of the mobile manipulator, and making the mobile manipulator far away from a singular point position; and designing a mobile manipulator visual multi-target predictive control algorithm based on operability adaptive weighted optimization, tracking visual features and optimizing vehicle-arm coordination, motion distribution accuracy and smoothness; while ensuring accurate tracking of visual features, the application autonomously optimizes vehicle-arm coordination, motion distribution accuracy and motion smoothness, and finally effectively realizes accurate tracking of visual features of the mobile manipulator.

Description

Technical Field

[0001] This invention relates to the field of robot motion control, and specifically to a visual predictive control method for mobile robotic arm coordination based on multi-objective optimization. Background Technology

[0002] With the increasing demands for autonomy and flexibility in fields such as intelligent manufacturing and special operations, mobile robotic arms that integrate environmental perception, mobile operation, and dexterous manipulation capabilities have become an important development direction for robotics technology. Mobile robotic arms consist of a mobile platform and a multi-degree-of-freedom robotic arm. They overcome the limitations of fixed-base robotic arms in terms of working space and can achieve target recognition, state estimation, and closed-loop tracking based on visual sensing, making them suitable for complex tasks such as assembly, grinding, wiping, and inspection. However, there are redundancies in degrees of freedom, differences in motion constraints, and nonlinear coupling between the mobile platform and the robotic arm. Visual information is also susceptible to noise, time delay, and model uncertainty. In unstructured contact operations, changes in target pose, changes in contact state, and external disturbances further reduce the system's trajectory tracking accuracy and interaction stability. Therefore, research on a unified control method that balances visual feature tracking, coordinated arm movement, motion smoothness, and end-effector force control has significant engineering and application value.

[0003] Currently, visual control of mobile robotic arms mainly employs methods such as image-based Jacobian matrix-based visual servoing, task priority or null-space projection, pseudo-inverse velocity allocation, and model predictive control. Traditional visual servoing methods typically prioritize visual error convergence and allocate the motion of the mobile platform and robotic arm through fixed weights or preset priorities. This approach struggles to adjust the robot-arm coordination in real time based on system configuration, joint motion redundancy, and task phase, easily leading to problems such as ineffective platform motion, near-singular robotic arm configurations, abrupt joint velocity changes, and unsmooth end-effector trajectories. While some multi-objective optimization methods can simultaneously consider tracking error, energy consumption, and obstacle avoidance, the weights of each optimization objective often rely on empirical tuning, lacking an adaptive adjustment mechanism that reflects the system's instantaneous motion capability. Existing operability optimization methods also often use single scalar geometric indices, failing to accurately describe the end-effector's velocity generation capability along different task directions, thus limiting the accuracy of robot-arm motion allocation. Therefore, developing an advanced control method that can adaptively optimize robot-arm motion allocation based on system motion capability, balance visual tracking accuracy and motion stability, and achieve precise and stable tracking of visual features is crucial for improving the autonomous operation capability of mobile robotic arms in complex environments. Summary of the Invention

[0004] To address the technical problems of existing technologies, such as the inability to simultaneously achieve visual feature tracking, coordinated arm movement, motion stability, and end-effector force control, as well as the difficulty in real-time and adaptive adjustment of the arm-vehicle coordination relationship, this invention provides a mobile robotic arm vehicle-arm coordination visual predictive control method based on multi-objective optimization. The technical solution is as follows:

[0005] A mobile robotic arm-vehicle coordination vision predictive control method based on multi-objective optimization includes the following steps:

[0006] Step 1: Using motion impact as the control input, establish a linearized state-space motion model for predictive control of the mobile robotic arm system;

[0007] Step 2: Design a mobile robotic arm-vehicle coordinated motion constraint model based on operability optimization to improve the speed operability of the mobile robotic arm and keep it away from singularity positions;

[0008] Step 3: Design a vision-based multi-object predictive control algorithm for a mobile robotic arm based on operability adaptive weighted optimization, track visual features and optimize arm coordination, motion distribution accuracy and stability.

[0009] Furthermore, in step 1, the state variables of the mobile robotic arm system are defined as follows:

[0010] ,

[0011] in, and Represents generalized joint space velocity and acceleration; impact of motion. Defined as the control input, the linearized state-space motion model of the predictive control of the mobile robotic arm system is as follows:

[0012] ,

[0013] in, and This represents the time-invariant state transition matrix and input matrix.

[0014] Furthermore, the state transition matrix and input matrix for:

[0015] ,

[0016] ,

[0017] in, Represents the identity matrix.

[0018] Further, step 2 is based on the image Jacobian matrix. The translation velocity Jacobian matrix of the first three rows Define the translational velocity operability ellipsoid index for:

[0019] ,

[0020] in, Determine the determinant of the matrix; calculate the operability ellipsoid index of the translational velocity. First-order time differential The formula is:

[0021] ,

[0022] in, It is the Jacobian of the translational speed of the mobile robotic arm; optimize the state variables. make maximize.

[0023] Furthermore, the operability of the translation speed is Jacobian. for:

[0024] ,

[0025] in, A vector operator that transforms a matrix into column vectors; ; Hessian matrix representing a mobile robotic arm The translation component.

[0026] Furthermore, step 3 involves designing a hybrid control strategy for the position and velocity of visual features in the image space, as shown in the formula:

[0027] ,

[0028] in, The nominal expected speed representing visual characteristics; The expected trajectory represents visual features; It is a positive definite diagonal matrix of gain coefficients; Represents visual feature tracking error. The actual location representing visual features; in the future time span Internal mobile robotic arm system state trajectory and control input To optimize the variables, a visual adaptive predictive controller is designed, with the following formula:

[0029] ,

[0030] in, The cost function for multi-objective optimization. and Represents system state variables The lower and upper limits; and Represents system control input The lower and upper limits.

[0031] Furthermore, the cost function for:

[0032] ,

[0033] in, The velocity tracking error represents the visual feature, while the actual velocity of the visual feature is... ; The error optimization weight matrix represents the positive definite diagonal; and The state variable optimization weight matrix and the input variable optimization weight matrix represent the positive definite diagonal; For the adaptive weight coefficient matrix of the state quadratic optimization term of the mobile robotic arm system; For the adaptive weighting coefficients of the coordination optimization term of the mobile robotic arm; The linear coefficient vector represents the optimization of the arm coordination operability.

[0034] Furthermore, the linear term coefficient vector for:

[0035] ;

[0036] The adaptive weighting coefficient for:

[0037] ,

[0038] in, It is the rate of change of the adaptive weight coefficients; the adaptive weight coefficient matrix for:

[0039] ,

[0040] in, For matrix block elements, Assign factors to the arm movement.

[0041] Furthermore, the arm motion allocation factor for:

[0042] ,

[0043] in, It is the allocation threshold value that only requires the movement of the manipulator; The formula for evaluating the speed of arm joint movement is:

[0044] ,

[0045] in, The speed of movement of the operating arm joint; and This represents the lower and upper limits of the speed of the manipulator joints.

[0046] Furthermore, in step 3, each step predicts the continuous time range of the control. Based on a fixed time step Converting to discrete time points, the formula is:

[0047] ,

[0048] The continuous-time optimization problem is transformed into a discrete-time nonlinear programming problem, and solved using the open-source framework OCS2, which employs a sequential quadratic programming algorithm, with one iteration within each control cycle. The sequential quadratic programming algorithm utilizes the cost function in each iteration. The second-order approximation and the first-order approximation of the constraints are used to construct the corresponding quadratic programming subproblems. The Jacobian of the first-order approximation constraints is calculated using the automatic differentiator CppAD, and the Hessian matrix of the second-order approximation cost is calculated using the Gauss-Newton approximation.

[0049] Beneficial effects

[0050] The mobile robotic arm vehicle-arm coordination visual prediction control method of the present invention adopts a mobile robotic arm visual adaptive prediction control algorithm based on speed operability adaptive weighted optimization. While ensuring accurate tracking of visual features, it autonomously optimizes vehicle-arm coordination, motion allocation accuracy and motion stability, and ultimately effectively achieves accurate tracking of the visual features of the mobile robotic arm. Attached Figure Description

[0051] Figure 1 The flowchart shows a vision-based predictive control method for mobile robotic arm vehicle-arm coordination based on multi-objective optimization.

[0052] Figure 2a This is a schematic diagram of the expected trajectory of the mobile robotic arm at the end of the world coordinate system.

[0053] Figure 2b A schematic diagram of the expected trajectory of a mobile robotic arm based on visual features in image space;

[0054] Figure 3 A schematic diagram illustrating the visual feature tracking error of a mobile robotic arm;

[0055] Figure 4a A schematic diagram illustrating the motion speed of the mobile platform during the visual feature tracking process of a mobile robotic arm.

[0056] Figure 4b A schematic diagram illustrating the movement speed of the manipulator arm during visual feature tracking by a mobile robotic arm.

[0057] Figure 5 This is a schematic diagram showing the optimization results of the vehicle-arm coordination operability during the visual feature tracking process of a mobile robotic arm. Detailed Implementation

[0058] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0059] like Figure 1 As shown, the specific steps of the mobile robotic arm vehicle-arm coordination visual prediction control method based on multi-objective optimization of the present invention are as follows:

[0060] Step 1: Using motion impact as the control input, establish a linearized state-space motion model for predictive control of the mobile robotic arm system to ensure the smoothness of the mobile robotic arm's motion.

[0061] To achieve adaptive predictive control for visual feature tracking, the state variables of the mobile robotic arm system are defined as follows: ,in and Represents the generalized joint space velocity and acceleration, while also including motion impact. Defined as the control input, a linearized state-space motion model for predictive control of the mobile robotic arm system is constructed, as shown in the following formula (1):

[0062] ,

[0063] In formula (1), and Representing the time-invariant state transition matrix and input matrix respectively, they are defined as follows:

[0064] ,

[0065] ,

[0066] in, The unit matrix represents the joint motion. Using joint motion impact as a control input can effectively ensure the smoothness of the mobile robotic arm's movement. In practice, the joint velocity value is obtained through two time integrations and then issued for execution.

[0067] Step 2: Design a mobile robotic arm-vehicle coordinated motion constraint model based on operability optimization to improve the speed operability of the mobile robotic arm and keep it away from singularity positions.

[0068] In order to make the mobile robotic arm move away from the singularity during the movement, that is, to make its end effector generate the same task space movement speed with a smaller joint space movement speed, the speed operability ellipsoid index is defined as shown in the following formula (2):

[0069] ,

[0070] In formula (2), Represents the determinant value of the matrix. It is the image Jacobian matrix. (Add...) This value can improve the robot's speed maneuverability and move it away from motion singularities. However, in the autonomous operation of a mobile robotic arm in an unstructured environment, the translational speed performance of the end effector in the task space is far more important than its rotational speed performance. Therefore, only the translational speed Jacobian matrix of the first three rows of the image Jacobian matrix is ​​selected. The translational velocity operability ellipsoid index is defined as shown in the following formula (3):

[0071] ,

[0072] To improve speed operability and avoid singularities during visual feature tracking of a mobile robotic arm, the translational speed operability ellipsoid index is determined. The first-order time derivative is shown in the following formula (4):

[0073] ,

[0074] In formula (4), It is the Jacobian of the translational speed operability of the mobile robotic arm, that is, the mapping relationship between the rate of change of translational speed operability and the joint movement speed, which can be derived as shown in the following formula (5):

[0075] ,

[0076] In formula (5), Vector operators Used to convert a matrix into a column vector; Hessian matrix representing a mobile robotic arm The translation component is shown in the following formula (6):

[0077] ,

[0078] Optimize joint space state variables Make Maximizing this will improve the speed and maneuverability of the mobile robotic arm, allowing it to move away from singularity points.

[0079] Step 3: Design a vision-based multi-target predictive control algorithm for a mobile robotic arm based on operability adaptive weighted optimization, accurately track visual features and optimize arm coordination, motion distribution accuracy and stability.

[0080] To prevent infinite position drift during velocity tracking, a hybrid control strategy for visual feature position and velocity in image space is first designed, as shown in the following formula (7):

[0081] ,

[0082] In formula (7), The nominal expected speed, representing visual characteristics It is a positive definite diagonal matrix of gain coefficients. Represents visual feature tracking error. The actual location representing the visual feature is calculated by the camera. The expected trajectory represents visual features.

[0083] In the future time span Internal mobile robotic arm system state trajectory and control input To optimize the variables, a visual adaptive predictive controller is designed as shown in the following formula (8):

[0084] ,

[0085] In formula (8), and Representing system state variables respectively The lower and upper limits, and They represent the system control inputs respectively. The lower and upper limits.

[0086] Design a cost function for multi-objective optimization. As shown in the following formula (9):

[0087] ,

[0088] In formula (9), The velocity tracking error represents the visual feature, whose actual velocity is mapped from the joint motion of the moving robotic arm. Calculations show that The error optimization weight matrix represents the positive definite diagonal. In practice, increasing this weight coefficient can improve the tracking accuracy of visual features. and These represent the state variable optimization weight matrix and the input variable optimization weight matrix, respectively, representing positive definite diagonal states. The relevant quadratic terms minimize joint impact to improve the smoothness of the mobile robotic arm's motion. The linear coefficient vector representing the optimization of the arm coordination operability can be derived based on formulas (4) and (5), as shown in formula (10) below:

[0089] ,

[0090] To avoid affecting the tracking performance of visual features during the optimization of the maneuverability of the mobile arm coordination, the adaptive weight coefficients of the coordination optimization term of the mobile maneuver are designed as shown in the following formula (11):

[0091] ,

[0092] In formula (11), It is the rate of change of the weighting coefficient, that is, the larger its value is... The descent is steeper near the origin. When the visual feature tracking error is small... It can continuously optimize the operability of the boom coordination; however, when the visual feature tracking error is large... It will then automatically switch to a mode that prioritizes performance improvement.

[0093] Due to significant differences in motion characteristics between the mobile platform and the manipulator, the mobile platform suffers from poor motion accuracy due to factors such as uneven ground disturbances and wheel slippage. Furthermore, its high mass inertia results in a slower dynamic response speed compared to the manipulator. Therefore, during contact operations, the manipulator can bear more of the motion. Since the mobile manipulator is driven by a joint velocity loop during visual feature tracking control, the speed of the manipulator joints... Exceeding its permitted range When this occurs, it indicates that the movement of the manipulator alone is insufficient to meet the motion requirements of the mobile robotic arm's end effector task. Therefore, more motion needs to be allocated to the mobile platform; conversely, the manipulator will bear the main task motion. The lower limit of the joint velocity... and upper limit The motion allocation factor is determined by a combination of the robot's own motion capabilities and task requirements. Based on this, the motion allocation factor of the robot arm is designed as shown in the following formula (12):

[0094] ,

[0095] Among them, the evaluation index of the movement speed of the manipulator joint As shown in the following formula (13):

[0096] ,

[0097] In formulas (12) and (13), It is the allocation threshold value that only requires the movement of the manipulator.

[0098] Further design of the adaptive weight coefficient matrix for the state quadratic optimization term of the mobile robotic arm system. As shown in the following formula (14):

[0099] ,

[0100] In formula (14), the matrix block elements As shown in the following formula (15):

[0101] ,

[0102] When the movement speed of the manipulator joint approaches or exceeds the allowable upper or lower limits, In this case, the joint position, velocity, acceleration, and other motion states of the manipulator are minimized, while the motion state of the moving platform is unrestricted. When the motion velocity of the manipulator joint is in the intermediate range, the end effector motion of the moving robotic arm is achieved jointly by the manipulator and the moving platform. If the manipulator needs to undertake more end effector motion in small-scale or precision operations, the displacement can be appropriately increased. The set value.

[0103] In summary, by solving the adaptive predictive control optimization problem defined by formulas (7) to (15) in each control cycle, the optimal control input for the mobile robotic arm can be obtained, enabling accurate tracking of visual features, while autonomously optimizing the coordination of the mobile robotic arm, the smoothness of motion, and the accuracy of motion distribution.

[0104] To solve the above nonlinear optimization problem, each step involves the continuous time range of the predictive control. Based on a fixed time step Converted to discrete time points:

[0105] ,

[0106] The continuous-time optimization problem is then transformed into a discrete-time nonlinear programming problem, and solved using the open-source framework OCS2, a sequential quadratic programming algorithm. Real-time performance is ensured by performing one iteration within each control cycle. The sequential quadratic programming algorithm utilizes the aforementioned cost function in each iteration. The second-order approximation and the first-order approximation of the constraints are used to construct the corresponding quadratic programming subproblems. The Jacobian of the first-order approximation constraints is calculated using the automatic differentiator CppAD, while the Hessian matrix of the second-order approximation cost is calculated using the Gauss-Newton approximation.

[0107] Example

[0108] The flowchart of the mobile robotic arm-vehicle coordination visual predictive control method based on multi-objective optimization of the present invention is as follows: Figure 1 As shown, the specific implementation object is a mobile robotic arm consisting of a Mecanum wheeled mobile platform and a six-degree-of-freedom robotic arm. The focal length of the vision system is defined as 800mm, and the center coordinates of the image plane are... By combining pixels with the end effector of the mobile robotic arm and the camera pose, the visual feature trajectory corresponding to the visual feature point can be calculated, and finally online feedback of the visual trajectory can be achieved.

[0109] In the implementation of this invention, the mobile robotic arm-vehicle coordinated visual prediction control law is shown in formulas (7) to (15), where the discrete time step is set to... The prediction step size is set to The motion trajectory of the mobile robotic arm's end effector is defined as follows:

[0110] ,

[0111] That is, the working plane is located at At the location, the end effector of the mobile robotic arm is , Both axes move sinusoidally upwards; simultaneously , Centered on, with Four visual feature points are defined on the working plane for each side length. Based on this, the expected trajectory of the visual features is obtained as follows: Figure 2a , Figure 2b As shown.

[0112] The above describes the implementation effect of the mobile robotic arm's coordinated visual predictive control: For example... Figure 3 As shown, the designed visual adaptive predictive controller can reduce the visual error to within ±0.05 pixels, achieving high-precision tracking of visual features; Figure 4a , Figure 4b As shown, the joint space velocity input to the vision-adaptive predictive control exhibits smooth and continuous characteristics, ensuring the stability of the mobile robotic arm's movement; the actual operability value of the mobile robotic arm is as follows: Figure 5 As shown, the operability of the coordinated movement of the arm can be improved simultaneously during the tracking of visual feature trajectories. The above results verify the effectiveness of the multi-objective optimized visual adaptive predictive control algorithm of this invention.

[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile robotic arm-vehicle coordination visual predictive control method based on multi-objective optimization, characterized in that, Includes the following steps: Step 1: Using motion impact as the control input, establish a linearized state-space motion model for predictive control of the mobile robotic arm system; Step 2: Design a mobile robotic arm-vehicle coordinated motion constraint model based on operability optimization to improve the speed operability of the mobile robotic arm and keep it away from singularity positions; Step 3: Design a vision-based multi-object predictive control algorithm for a mobile robotic arm based on operability adaptive weighted optimization, track visual features and optimize arm coordination, motion distribution accuracy and stability.

2. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 1, characterized in that: Step 1 defines the state variables of the mobile robotic arm system as follows: ， in, and Represents generalized joint space velocity and acceleration; impact of motion. Defined as the control input, the linearized state-space motion model of the predictive control of the mobile robotic arm system is as follows: ， in, and This represents the time-invariant state transition matrix and input matrix.

3. The mobile robotic arm vehicle-arm coordination visual prediction control method based on multi-objective optimization as described in claim 2, characterized in that: The state transition matrix and input matrix for: ， ， in, Represents the identity matrix.

4. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 2, characterized in that: Step 2 is based on the image Jacobian matrix. The translation velocity Jacobian matrix of the first three rows Define the translational velocity operability ellipsoid index for: ， in, Determine the determinant of the matrix; calculate the operability ellipsoid index of the translational velocity. First-order time differential The formula is: ， in, It is the Jacobian of the translational speed of the mobile robotic arm; optimize the state variables. make maximize.

5. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 4, characterized in that: The translation speed operability Jacobi for: ， in, A vector operator that transforms a matrix into column vectors; ; Hessian matrix representing a mobile robotic arm The translation component.

6. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 4, characterized in that: Step 3 involves designing a hybrid control strategy for the location and velocity of visual features in the image space, using the following formula: ， in, The nominal expected speed representing visual characteristics; The expected trajectory represents visual features; It is a positive definite diagonal matrix of gain coefficients; Represents visual feature tracking error. The actual location representing visual features; in the future time span Internal mobile robotic arm system state trajectory and control input To optimize the variables, a visual adaptive predictive controller is designed, with the following formula: ， in, The cost function for multi-objective optimization. and Represents system state variables The lower and upper limits; and Represents system control input The lower and upper limits.

7. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 6, characterized in that: The cost function for: ， in, Speed ​​tracking error, representing visual characteristics ; The error optimization weight matrix represents the positive definite diagonal; and The state variable optimization weight matrix and the input variable optimization weight matrix represent the positive definite diagonal; For the adaptive weight coefficient matrix of the state quadratic optimization term of the mobile robotic arm system; For the adaptive weighting coefficients of the coordination optimization term of the mobile robotic arm; The linear coefficient vector represents the optimization of the arm coordination operability.

8. The mobile robotic arm-vehicle coordination visual predictive control method based on multi-objective optimization as described in claim 7, characterized in that: The linear term coefficient vector for: ； The adaptive weighting coefficient for: ， in, It is the rate of change of the adaptive weight coefficients; the adaptive weight coefficient matrix for: ， in, For matrix block elements, Assign factors to the arm movement.

9. The mobile robotic arm-vehicle coordination visual predictive control method based on multi-objective optimization as described in claim 8, characterized in that: The arm motion distribution factor for: ， in, It is the allocation threshold value that only requires the movement of the manipulator; The formula for evaluating the speed of arm joint movement is: ， in, The movement speed of the operating arm joint; and This represents the lower and upper limits of the speed of the manipulator joints.

10. The mobile robotic arm-vehicle coordination visual prediction control method based on multi-objective optimization as described in claim 9, characterized in that: In step 3, each step will predict the continuous time range of the control. Based on a fixed time step Converting to discrete time points, the formula is: ， The continuous-time optimization problem is transformed into a discrete-time nonlinear programming problem, and the open-source framework OCS2, which is a sequential quadratic programming algorithm, is used to solve it, and one iteration is used in each control cycle. The sequential quadratic programming algorithm utilizes the cost function in each iteration. The second-order approximation and the first-order approximation of the constraints are used to construct the corresponding quadratic programming subproblems. The Jacobian of the first-order approximation constraints is calculated using the automatic differentiator CppAD, and the Hessian matrix of the second-order approximation cost is calculated using the Gauss-Newton approximation.

Images