Quadruped robot autonomous decision-making control system based on multi-mode perception

Abstract

The invention discloses a quadruped robot autonomous decision control system based on multi-modal perception, and belongs to the technical field of robot motion, and the system specifically comprises the steps: collecting visual, force and sound information through multiple sensors of a camera, a force sensor and a microphone array, and carrying out the preprocessing; according to the processed information, a diagonal gait, leg track and motion path planning scheme is generated by adopting a motion planning strategy; generating a specific control instruction to drive the robot to move, and monitoring and adjusting in real time; after the task is completed, the motion performance of the robot is evaluated, and a motion planning and control method is optimized according to an evaluation result; the autonomous motion ability and adaptability of the quadruped robot in a complex environment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot motion, in particular to an autonomous decision-making control system of a quadruped robot based on multi-modal perception. Background Art

[0002] In the research of quadruped robots, autonomous decision-making and control methods are a core issue, which involves multiple aspects such as the robot's perception, planning, decision-making and execution in complex environments. In order to achieve autonomous decision-making and control, quadruped robots need to be equipped with various sensors, such as depth cameras, lidars and inertial measurement units, for environmental perception and navigation. At the same time, advanced computing boards and algorithms are also needed to process these sensor data and make corresponding decisions and plans.

[0003] The autonomous decision-making control methods for quadruped robots include rule-based control methods, model-based control methods, learning-based control methods and hybrid control methods. The rule-based control method judges the robot's behavior through preset rules and logic, but has poor flexibility; the model-based control method establishes the robot's dynamic model and kinematic model, and uses mathematical models to predict and control the robot's behavior, but due to the complexity of the actual robot system, the established model often has errors, resulting in unsatisfactory control effects; the learning-based control method uses machine learning or reinforcement learning and other technologies to allow the robot to continuously learn and optimize its own behavior in the process of interacting with the environment, but it requires a large amount of training data and time to optimize the model, and the performance of the model depends to a large extent on the quality and quantity of the training data; although the hybrid control method combines the advantages of multiple control methods, it needs to process multiple control strategies at the same time, which increases the complexity of the system and the difficulty of implementation.

[0004] For example, a Chinese patent application with authorization announcement number CN113671976B discloses a motion positioning control method for a three-legged supported pipeline robot, including: system startup; obtaining posture information; establishing a static posture model; obtaining distance measurement information; switching straight / bent pipe control mode; motor operation; current PID module operation; segmented control mode; system stop; the invention uses information feedback from an external perception module, a positioning module, and a motion module, and forms the robot's posture information after processing, and the controller decides to adopt the optimal control strategy, so that the pipeline robot has a certain degree of autonomous recognition capability, reduces the burden and risk of misoperation of operators, and adjusts the robot's posture at any time according to the motion path, so that it can autonomously bend in the pipeline and operate in straight pipes, curved pipes and obstacles.

[0005] For example, the patent application with publication number CN118625847A discloses a lightweight quadruped robot motion method based on a cross-modal perception attention mechanism, including: obtaining proprioceptive perception data and visual perception data of the quadruped robot; performing multimodal data fusion based on the proprioceptive perception data and visual perception data to obtain the fused multimodal features; obtaining a trained cross-modal perception attention controller and a low-level reinforcement learning motion controller; and controlling the motion of the quadruped robot through the trained cross-modal perception attention controller and the low-level reinforcement learning motion controller. This technical solution adds a cross-modal attention control module to the autonomous obstacle avoidance decision-making task of the quadruped robot, thereby realizing the lightweight quadruped robot obstacle avoidance motion with dynamic and contextual perception on the robot, and the robot can autonomously choose to switch information between different modes according to task requirements.

[0006] The above existing technologies all have the following problems: lack of optimization strategies and real-time performance; lack of sufficient experimental verification and performance evaluation; more suitable for specific tasks or scenarios, and limited scope of application. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes an autonomous decision-making and control system for a quadruped robot based on multimodal perception, which collects visual, force and acoustic information through multiple sensors including cameras, force sensors and microphone arrays, and performs preprocessing; based on the processed information, a motion planning strategy is adopted to generate diagonal gait, leg trajectory and motion path planning schemes; specific control instructions are generated to drive the robot movement, and the robot movement is monitored and adjusted in real time; after completing the task, the robot movement performance is evaluated, and the motion planning and control methods are optimized according to the evaluation results; the present invention improves the autonomous movement ability and adaptability of the quadruped robot in complex environments.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The autonomous decision-making control system of a quadruped robot based on multimodal perception includes: a perception module, a decision-making planning module, an execution module, and a feedback monitoring module;

[0010] The perception module is used to collect the robot's internal state information and external environment information through multiple sensors, and integrate them to form multimodal perception information;

[0011] The decision-making planning module includes a planning strategy unit, wherein the planning strategy unit is configured with a gait generation strategy and a motion planning strategy; the gait generation strategy is used to determine the time period, the support phase and the swing phase of each step, and to determine each joint angle using inverse kinematics; the motion planning strategy is used to perform motion planning and decision-making according to the gait generation strategy;

[0012] The execution module includes an instruction generation unit and an execution unit; the instruction generation unit is configured with a PID control algorithm, and the PID control algorithm is used to generate control instructions for controlling different joints and drivers of the quadruped robot according to the motion planning scheme;

[0013] The feedback monitoring module includes a monitoring unit and an optimization strategy generation unit. The monitoring unit is configured with a three-dimensional obstacle avoidance strategy, which is used to monitor the robot's posture, speed, acceleration, ground contact force and changes in the surrounding environment in real time, and avoid obstacles based on the monitoring results; the optimization strategy generation unit is configured with a spider bee optimization strategy, which is used to optimize the evaluation results.

[0014] Specifically, the specific steps of the gait generation strategy include:

[0015] A1: Obtain multimodal perception information generated by the perception module and set gait parameters according to task requirements, wherein the gait parameters include step length, step speed, and diagonal gait type;

[0016] A2: According to the set gait parameters and the robot's walking speed requirements, the time of the stance phase and the swing phase is calculated and allocated by the table lookup method, and the time of the stance phase and the swing phase are added to obtain the gait cycle; at the same time, according to the diagonal gait type, the switching rules between the stance phase and the swing phase of different legs are determined;

[0017] A3: Set the movement speed or acceleration state threshold h1, and calculate the switching timing between the stance phase and the swing phase of different legs based on the gait cycle and switching rules and the movement rules of different legs;

[0018] If v sp >h1, or v sw

[0019] If v sp ≤h1, or v sw ≥h1, the switch from the support phase to the swing phase is triggered, where v sp is the velocity or acceleration of the leg at the end of the stance phase, v sw represents the velocity or acceleration of the leg at the beginning of the swing phase;

[0020] A4: Based on the preset target position and orientation of the robot end effector, use the Newton-Raphson algorithm to calculate the angles θ of different joints i , based on the obtained joint angles and the switching timing obtained in A3, calculate the speed u of different joints i =θ i ′, and integrate the calculated different joint angles, speeds and switching timings into gait information, where θ​i represents the angle of the i-th joint, and i∈[1,n], u i represents the velocity of the i-th joint, θ i ′ represents θ i The first-order derivative of with respect to time, n represents the number of joints;

[0021] A5: Perform stability analysis on the generated gait information, and adjust and optimize gait parameters and joint control based on the analysis results.

[0022] Specifically, the steps for calculating the angles of different joints in A4 include:

[0023] A4.1: Collect the robot's mechanical structure parameters and set the target position and direction of the robot's end effector;

[0024] A4.2: Take the target position and orientation as input, run the Newton-Raphson algorithm, and output different joint angles that meet the target position and orientation requirements;

[0025] A4.3: Check whether the calculated joint angles meet the physical limitations of the robot;

[0026] If not satisfied, readjust the target position and direction;

[0027] If satisfied, the calculated joint angle is output.

[0028] Specifically, the specific steps of A4.2 include:

[0029] A4.21: Receive the target position and direction of the robot end effector and convert the target position and direction into vector format to obtain the target position vector x vector and the target direction vector θ vector ;

[0030] A4.22: Read the current joint angle from the sensor based on Calculate the transformation matrix M of this joint relative to the previous joint through translation transformation bian , the total transformation matrix M from the base to the end effector is obtained by accumulation total =M bian,1 ×…×M bian,n , and extract the actual position v of the end effector from the total transformation matrix ad and direction θ ad , where M bian,n Represents the transformation matrix of the nth joint.

[0031] Specifically, the specific steps of A4.2 also include:

[0032] A4.23: Use sensors to obtain the actual position of the current joint angle and direction Calculate the error vector between the target position and orientation and the actual position and orientation using the formula:

[0033]

[0034] Among them, e vector represents the error vector between the target position and direction and the actual position and direction, w1 represents the position weight, w2 represents the direction weight, and |·| represents the absolute value;

[0035] A4.24: Use the Jacobian matrix to solve the first derivative of the error vector with respect to the current joint angle And according to the first-order derivative and the error vector, calculate the increment of the joint angle in, The Jacobian matrix represents the relationship between the velocity of the robot end effector and the joint velocity, and λ represents the learning rate;

[0036] A4.25: Increase Add to current joint angle Iterate and update until the preset number of iterations is reached, and then output the final joint angle.

[0037] Specifically, the monitoring unit adopts a three-dimensional obstacle avoidance strategy, and the specific steps of the three-dimensional obstacle avoidance strategy include:

[0038] B1: Use binocular cameras to capture the 3D information of the surrounding environment and the distance of obstacles in real time, and perform preprocessing to obtain environmental perception image information;

[0039] B2: Perform stereo matching on the environment perception images taken by the left and right cameras of the binocular camera, calculate the three-dimensional coordinates and depth information of the obstacle by calculating the disparity map and combining the internal and external parameters of the camera;

[0040] B3: Use machine learning convolutional neural network algorithm to identify and classify the three-dimensional information of obstacles calculated in B2;

[0041] B4: Based on the identified and classified obstacle information, as well as the robot’s current position and posture, a path planning algorithm is used to plan multiple feasible obstacle avoidance paths.

[0042] Specifically, the specific steps of B2 include:

[0043] B2.1: Obtaining pre-processed environment perception image information, where the environment perception image information is taken by both left and right cameras of a binocular camera;

[0044] B2.2: Based on the environment perception image, for each pixel in the left image, find its corresponding point in the right image, and calculate the matching cost of each pixel in the left and right images. Store the matching costs of all pixels in a three-dimensional matrix to obtain the disparity three-dimensional matrix M DSI , the formula is:

[0045] Q(q1,q2,q3)=|g L (q1,q2)-g R (q1-q3,q2)|;

[0046] Where q1 represents the width of the environment perception image, q2 represents the height of the environment perception image, q3 represents the disparity search range, Q(q1,q2,q3) represents the matching cost when the disparity is q3 at the pixel position (q1,q2) in the environment perception image, and g L (q1, q2) represents the grayscale value of the left image of the binocular camera at the pixel position (q1, q2) in the environmental perception image, g R (q1-q3,q2) represents the grayscale value of the right image of the binocular camera at the pixel position (q1-q3,q2) in the environmental perception image;

[0047] B2.3: Aggregate the matching costs to obtain a new disparity 3D matrix The cost value of each pixel reflects its correlation with the candidate pixels.

[0048] Specifically, the specific steps of B2 also include:

[0049] B2.4: Create a disparity map of the same size as the left image to store the optimal disparity value for each pixel;

[0050] B2.5: For the new disparity 3D matrix For each pixel in , traverse all possible disparity values ​​of each pixel, and find the cost value of the pixel under each disparity value, and select the disparity value with the smallest cost value as the optimal disparity of the pixel. The formula is:

[0051]

[0052] Among them, q 3,k represents the optimal disparity value of pixel k, Q(k,q3) represents the cost value of pixel k when the disparity is q3, and argmin(·) represents the optimal disparity value of pixel k in the domain The value of the independent variable corresponding to the minimum value of inner Q(k,q3);

[0053] B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map, and obtain a disparity map containing the optimal disparity value of each pixel;

[0054] B2.7: Calculate the depth value corresponding to each pixel based on the disparity map, the focal length of the camera, and the distance between the two eyes The depth value is mapped to the three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle, where d guang Represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q4 represents the parallax between the two cameras.

[0055] The optimization strategy generation unit adopts the spider bee optimization strategy, and the specific steps of the spider bee optimization strategy include:

[0056] D1: Set the search space and randomly distribute a certain number of search agents in the search space, where each search agent represents a potential solution;

[0057] D2: Each search agent moves in the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value;

[0058] D3: If the search agent finds a potential high-quality solution, it will follow the solution. If the quality of the solution decreases or encounters an obstacle, the search agent will choose to escape, that is, move away from the current solution and try a new search direction;

[0059] D4: Once a satisfactory solution is found, the search agent builds a nest around it, i.e., performs local optimization;

[0060] D5: Combine the solutions of different search agents through crossover operation to generate new offspring, and repeat iteratively until the preset maximum number of iterations is reached.

[0061] The switching rule in A2 includes that when the left front leg and the right hind leg are supporting, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg are supporting on the ground, the left front leg and the right hind leg start to swing.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. The present invention proposes a quadruped robot autonomous decision-making control system based on multimodal perception, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.

[0064] 2. The present invention proposes an autonomous decision-making and control system for a quadruped robot based on multimodal perception, which forms multimodal perception information through multi-sensor fusion, and realizes comprehensive monitoring of the robot's internal state and external environment; the gait generation strategy and motion planning strategy in the planning strategy unit can accurately determine the robot's gait and motion path, ensuring that the robot walks stably and effectively avoids obstacles. This comprehensive perception and planning capability improves the robot's adaptability and walking efficiency in complex environments.

[0065] 3. The present invention proposes an autonomous decision-making control system for a quadruped robot based on multimodal perception, which generates precise control instructions through a PID control algorithm to drive the various joints and drivers of the quadruped robot to work together; at the same time, the three-dimensional obstacle avoidance strategy and optimization strategy generation unit in the feedback monitoring module can monitor the robot's motion state in real time and optimize the evaluation results to ensure that the robot maintains optimal performance during walking; this closed-loop control mechanism not only improves the robot's motion accuracy and stability, but also provides a guarantee for its continuous and stable operation in a complex and changeable environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a diagram of the architecture of the autonomous decision-making control system of a quadruped robot based on multimodal perception of the present invention;

[0067] Figure 2 This is a flowchart of the overall implementation of the autonomous decision-making control system of a quadruped robot based on multimodal perception of the present invention;

[0068] Figure 3 This is a flow chart for implementing the gait generation strategy of the autonomous decision-making control system of a quadruped robot based on multimodal perception of the present invention;

[0069] Figure 4 The present invention is a flowchart for implementing the three-dimensional obstacle avoidance strategy of the autonomous decision-making control system of the quadruped robot based on multimodal perception. DETAILED DESCRIPTION

[0070] Example 1

[0071] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: a quadruped robot autonomous decision-making control system based on multimodal perception, comprising the following steps:

[0072] Perception module, decision-making and planning module, execution module, feedback monitoring module;

[0073] The perception module is used to collect the robot's internal state information and external environment information through multiple sensors, and integrate them into multi-modal perception information to provide data support for the decision-making and planning module;

[0074] The decision-making and planning module is used to formulate motion plans and decisions based on multimodal perception information and generate motion planning solutions;

[0075] An execution module is used to generate specific control instructions according to the motion planning scheme and send the control instructions to the actuator of the quadruped robot;

[0076] The feedback monitoring module is used to monitor the operating status of the entire system in real time, and feed back the monitoring results to the decision-making planning module, and evaluate the robot's motion performance after the robot completes the target task.

[0077] The perception module includes: an internal sensor unit, an external sensor unit, and a preprocessing unit;

[0078] The internal sensor unit includes an inertial measurement unit and a joint motor encoder; the inertial measurement unit is used to measure the robot's angular velocity, acceleration, posture angle and other information; the joint motor encoder is used to measure the rotation angle and speed of the joint;

[0079] The external sensor unit includes a laser radar, a camera, and a foot-end pressure sensor. The laser radar is used to accurately measure the distance between the robot and surrounding objects; the camera is used to capture environmental images and achieve visual perception; and the foot-end pressure sensor is used to monitor the contact between the robot's legs and the ground.

[0080] The preprocessing unit is used to perform operations such as data cleaning, format unification, and noise filtering to ensure the accuracy and reliability of perceived information.

[0081] The decision-making and planning module includes: planning strategy unit, program formulation unit, and evaluation unit;

[0082] A planning strategy unit, which is used to perform motion planning and decision-making based on the preprocessed multimodal perception information using gait generation strategy and motion planning strategy, such as diagonal gait selection, leg trajectory planning, and motion path determination;

[0083] Among them, the gait generation strategy is a key sub-strategy in the planning strategy unit. Its main task is to determine the time period, support phase and swing phase of each step according to the movement requirements of the quadruped robot and the current environmental information; the support phase refers to which feet of the robot are in contact with the ground during walking; and the swing phase refers to which feet are off the ground; in addition, the gait generation strategy also needs to use inverse kinematics to determine the angle of each joint to ensure that the robot can walk according to the predetermined gait.

[0084] The motion planning strategy is based on the results of the gait generation strategy, and further carries out detailed motion planning and decision-making. It needs to consider the robot's motion path, speed, acceleration and other parameters, as well as possible obstacles and terrain changes. The goal of the motion planning strategy is to generate a safe and efficient motion planning plan so that the robot can successfully complete its target task.

[0085] Furthermore, the specific steps of the motion planning strategy include:

[0086] (1) In a known environment map, a path planning algorithm is used to plan an optimal path from a starting point to an end point. In the present invention, the path planning algorithm adopts a Dijkstra algorithm. The Dijkstra algorithm is a prior art in the art and is not an inventive solution of the present application, and is not described in detail herein.

[0087] (2) Optimize motion parameters such as step length, step frequency, and joint angles based on the quadruped robot’s dynamic model and kinematic constraints, as well as terrain features in the environment map;

[0088] (3) According to the gait generation strategy, combined with the optimal path and motion parameters, the specific execution plan for each step is decided, and the execution plan is converted into control instructions and sent to the actuator of the quadruped robot;

[0089] 1) Select a gait strategy from a preset gait generation strategy library based on current environmental conditions and task requirements;

[0090] 2) Integrate the optimal path generated by the path planning algorithm with the motion parameters obtained by the motion parameter optimization algorithm;

[0091] 3) Generate a specific execution plan for each step based on the gait strategy and motion parameters. The execution plan should include detailed information such as the starting position, end position, walking path, and joint angle changes of each step;

[0092] 4) Convert the execution plan into control instructions that can be understood by the robot actuator, where the control instructions usually include parameters such as joint angle, motor speed, torque, etc. These parameters will directly control the robot's walking.

[0093] A plan making unit, used for generating a motion planning plan according to the motion planning and decision results;

[0094] The evaluation unit is used to consider the stability and energy consumption factors of the robot during the implementation of the motion planning scheme to ensure the feasibility and efficiency of the motion planning scheme.

[0095] The execution module includes: an instruction generation unit and an execution unit;

[0096] An instruction generation unit, used to generate control instructions for controlling different joints and drivers of the quadruped robot through a PID control algorithm according to the motion planning scheme, wherein the PID control algorithm is a prior art content in the art and is not an inventive solution of the present application, and will not be described in detail here;

[0097] Actuator units, such as leg actuators, are used to execute control instructions and drive the quadruped robot to move.

[0098] The feedback monitoring module includes: monitoring unit, performance evaluation unit, optimization strategy generation unit, feedback guidance unit;

[0099] The monitoring unit is used to monitor the robot's posture, speed, acceleration, ground contact force and other motion states and changes in the surrounding environment in real time, and provide feedback to the quadruped robot's autonomous decision-making control system based on multimodal perception for adjustment and optimization;

[0100] A performance evaluation unit, used to quantitatively evaluate the robot's motion performance and generate evaluation results;

[0101] An optimization strategy generation unit, used to generate improvement measures and optimization strategies based on the evaluation results;

[0102] The feedback guidance unit is used to use the evaluation results and improvement measures as feedback to guide the new multimodal perception information acquisition, preprocessing, motion planning and decision-making process.

[0103] In summary, in the present invention, the overall implementation process includes:

[0104] S1: Collect multimodal perception information through multiple sensors and perform preprocessing. The multimodal perception information includes visual information, force information and acoustic information. The visual information includes terrain features and obstacle locations. The force information includes softness and hardness, friction, and acoustic information refers to capturing surrounding sounds using a microphone array. The multiple sensors include cameras, force sensors, and microphone arrays. The preprocessing includes data cleaning, format unification, and noise filtering to obtain accurate and reliable environmental perception information.

[0105] S2: Based on the processed multimodal perception information, use the motion planning strategy to perform motion planning and decision-making, and generate a motion planning scheme, including selecting diagonal gaits, planning leg trajectories, and determining motion paths. At the same time, the robot's stability and energy consumption factors are considered;

[0106] S3: Generate specific control instructions based on the motion planning scheme, which will be used to control different joints and actuators of the quadruped robot to achieve the desired motion;

[0107] S4: Send control commands to the actuators of the quadruped robot, such as the leg drives, and monitor the robot's motion state and changes in the surrounding environment in real time, including posture, speed, acceleration, and ground contact force. At the same time, make necessary adjustments and optimizations to the control commands based on the feedback information monitored in real time;

[0108] S5: After the robot completes the target task, the robot's motion performance is evaluated, including the robot's stability, motion efficiency, energy consumption, etc., and based on the evaluation results, the motion planning strategy and control instruction generation method are improved and optimized, and improvement measures are generated;

[0109] S6: Use the evaluation results and improvement measures as feedback to guide new multimodal perception information acquisition, preprocessing, and motion planning and decision-making processes.

[0110] Example 2

[0111] See also Figure 3 , the specific steps of the gait generation strategy in this embodiment include:

[0112] A1: Obtain multimodal perception information generated by the perception module and set gait parameters according to task requirements, wherein the gait parameters include step length, step speed, and diagonal gait type;

[0113] Among them, the diagonal gait types include: a diagonal gait in which the left front leg and the right hind leg are supported at the same time, and the right front leg and the left hind leg swing at the same time; a diagonal gait in which the right front leg and the left hind leg are supported at the same time, and the left front leg and the right hind leg swing at the same time.

[0114] A2: According to the set gait parameters and the robot's walking speed requirements, the time of the stance phase and the swing phase is calculated and allocated by the table lookup method, and the time of the stance phase and the swing phase are added to obtain the gait cycle. At the same time, according to the diagonal gait type, the switching rules between the stance phase and the swing phase of different legs are determined;

[0115] Among them, the table lookup method is based on previous experimental or simulation data; the gait cycle refers to the process from the heel touching the ground on one side to the heel touching the ground again on that side, which includes two stages: the support phase and the swing phase. In one gait cycle, the robot will experience a transition from the support phase to the swing phase, and then from the swing phase back to the support phase.

[0116] A3: Set the movement speed or acceleration state threshold h1, and calculate the switching timing between the stance phase and the swing phase of different legs based on the gait cycle and switching rules and the movement rules of different legs;

[0117] If v sp >h1, or v sw ​

[0118] If v sp ≤h1, or v sw ≥h1, the switch from the support phase to the swing phase is triggered, where v sp is the velocity or acceleration of the leg at the end of the stance phase, v sw represents the velocity or acceleration of the leg at the beginning of the swing phase;

[0119] A4: Based on the preset target position and orientation of the robot end effector, use the Newton-Raphson algorithm to calculate the angles θ of different joints i , based on the obtained joint angles and the switching timing obtained in A3, calculate the speed u of different joints i =θ i ′, and integrate the calculated different joint angles, speeds and switching timings into gait information, where θ i represents the angle of the i-th joint, and i∈[1,n], u i represents the velocity of the i-th joint, θ i ′ represents θ i The first-order derivative of with respect to time, n represents the number of joints;

[0120] A5: Perform stability analysis on the generated gait information, and adjust and optimize gait parameters and joint control based on the analysis results.

[0121] Furthermore, the specific steps of A5 include:

[0122] (1) Gait stability assessment:

[0123] 1) Use gait analysis instruments, such as three-dimensional gait analyzers and pressure gait analyzers, to collect gait data, including gait cycle, stride length, stride speed, stride frequency, and joint angles;

[0124] 2) Use the scale evaluation method to perform stability analysis on the collected gait data, including the consistency of the gait cycle, the uniformity of the stride, and the stability of the gait speed. The scale evaluation method is a prior art in this field and is not an inventive solution of the present application, and will not be described in detail here;

[0125] 3) assess gait continuity, which is the ability to smoothly transition between phases of walking;

[0126] 4) Based on the evaluation results, determine whether there are any problems with gait stability, such as prolonged or shortened gait cycle, reduced stride length, or reduced gait speed;

[0127] (2) Gait parameters and joint control adjustment and optimization:

[0128] 1) Identify key gait parameters that affect gait stability, such as step length, step width, step speed, and joint angles;

[0129] 2) Develop targeted adjustment strategies based on the evaluation results, such as increasing step length, adjusting step speed, and optimizing joint angles;

[0130] 3) Using biomechanical principles to optimize joint control strategies, such as improving joint coordination;

[0131] 4) Implement adjustment strategies, collect feedback data, evaluate adjustment effects, and further adjust and optimize gait parameters and joint control strategies based on feedback data.

[0132] The steps for calculating the angles of different joints in A4 include:

[0133] A4.1: Collect the robot's mechanical structure parameters from the robot's instruction manual and set the target position and direction of the robot's end effector. The mechanical structure parameters include the link length, joint type, and joint range. The target position is usually a coordinate point in three-dimensional space, and the target direction is usually a three-dimensional vector or rotation matrix.

[0134] A4.2: Take the target position and orientation as input, run the Newton-Raphson algorithm, and output different joint angles that meet the target position and orientation requirements;

[0135] A4.3: Check whether the calculated joint angles meet the physical limitations of the robot;

[0136] If not satisfied, readjust the target position and direction;

[0137] If satisfied, the calculated joint angle is output.

[0138] The specific steps of A4.2 include:

[0139] A4.21: Receive the target position and direction of the robot end effector and convert the target position and direction into vector format to obtain the target position vector x vector and the target direction vector θ vector ;

[0140] The specific process of converting the target position and orientation into vector format includes:

[0141] (1) Convert the target position into vector format:

[0142] 1) Determine the target location: The target location is a point in three-dimensional space, represented by its x, y, and z coordinates in a Cartesian coordinate system;

[0143] 2) Construct the position vector:

[0144] Construct a three-dimensional vector using the x, y, and z coordinates of the target location as the components of the vector.

[0145] For example, if the target position is a point P(x, y, z), the position vector can be expressed as

[0146] (2) Convert the target direction into vector format:

[0147] 1) Determine the target direction: The target direction is a unit vector pointing in a specific direction;

[0148] 2) If the target direction is already a unit vector, use it directly; if the target direction is a non-unit vector, it needs to be normalized to obtain a unit vector.

[0149] For example, if the direction vector is Its length is Then the unit direction vector is

[0150] A4.22: Read the current joint angle from the sensor based on Calculate the transformation matrix M of this joint relative to the previous joint through translation transformation bian , the total transformation matrix M from the base to the end effector is obtained by accumulation total =M bian,1 ×…×M bian,n , and extract the actual position v of the end effector from the total transformation matrix ad and direction θ ad , where M bian,n Represents the transformation matrix of the nth joint;

[0151] Furthermore, the implementation steps of the transformation matrix include:

[0152] (1) Determine the joint type: First determine whether the joint is a rotational joint or a translational joint;

[0153] (2) Select transformation matrix: Select the corresponding transformation matrix from the above formula according to the joint type and current angle or displacement;

[0154] (3) Apply transformation matrix: Apply the selected transformation matrix to the coordinate system of the previous joint to obtain the coordinate system of the current joint.

[0155] A4.23: Use sensors to obtain the actual position of the current joint angle and direction Calculate the error vector between the target position and orientation and the actual position and orientation using the formula:

[0156]

[0157] Among them, e vector represents the error vector between the target position and direction and the actual position and direction, w1 represents the position weight, w2 represents the direction weight, and |·| represents the absolute value;

[0158] Furthermore, the specific steps of A4.23 include:

[0159] (1) Set the target position and direction that the robot end effector is expected to reach;

[0160] (2) Using sensors or kinematic models, the actual position and orientation of the robot end effector is calculated based on the current joint angles;

[0161] (3) Calculate the vector difference between the target position and the actual position, i.e., the position error vector, which is usually a three-dimensional vector representing the deviation in the x, y, and z coordinates;

[0162] (4) Calculate the error between the target direction and the actual direction;

[0163] (5) The position error and direction error are combined into a unified error vector through a weighted algorithm.

[0164] A4.24: Use the Jacobian matrix to solve the first-order derivative of the error vector with respect to the current joint angle And according to the first-order derivative and the error vector, calculate the increment of the joint angle in, The Jacobian matrix represents the relationship between the velocity of the robot end effector and the joint velocity, and λ represents the learning rate;

[0165] A4.25: Increase Add to current joint angle Iterate and update until the preset number of iterations is reached, and then output the final joint angle.

[0166] Example 3

[0167] See also Figure 4 In this embodiment, the monitoring unit adopts a three-dimensional obstacle avoidance strategy. The specific steps of the three-dimensional obstacle avoidance strategy include:

[0168] B1: Use binocular cameras to capture the 3D information of the surrounding environment and the distance of obstacles in real time, and perform preprocessing to obtain environmental perception image information;

[0169] B2: Perform stereo matching on the environment perception images taken by the left and right cameras of the binocular camera, calculate the three-dimensional coordinates and depth information of the obstacle by calculating the disparity map and combining the internal and external parameters of the camera;

[0170] B3: using a convolutional neural network algorithm to identify and classify the three-dimensional information of obstacles calculated in B2, wherein the convolutional neural network algorithm is a prior art in the art and is not an inventive solution of the present application, and is not described in detail here;

[0171] B4: Based on the identified and classified obstacle information, as well as the current position and posture of the robot, a path planning algorithm is used to plan multiple feasible obstacle avoidance paths. The path planning algorithm is a prior art in this field and is not an inventive solution of the present application, so it will not be elaborated here.

[0172] The specific steps of B2 include:

[0173] B2.1: Obtaining pre-processed environment perception image information, where the environment perception image information is taken by both left and right cameras of a binocular camera;

[0174] B2.2: Based on the environment perception image, for each pixel in the left image, find its corresponding point in the right image, and calculate the matching cost of each pixel in the left and right images. Store the matching costs of all pixels in a three-dimensional matrix to obtain the disparity three-dimensional matrix M DSI , the formula is:

[0175] Q(q1,q2,q3)=|g L (q1,q2)-g R (q1-q3,q2)|;

[0176] Where q1 represents the width of the environment perception image, q2 represents the height of the environment perception image, q3 represents the disparity search range, Q(q1,q2,q3) represents the matching cost when the disparity is q3 at the pixel position (q1,q2) in the environment perception image, and g L (q1, q2) represents the grayscale value of the left image of the binocular camera at the pixel position (q1, q2) in the environmental perception image, g R (q1-q3,q2) represents the grayscale value of the right image of the binocular camera at the pixel position (q1-q3,q2) in the environmental perception image;

[0177] B2.3: Aggregate the matching costs to obtain a new disparity 3D matrix The cost value of each pixel reflects its correlation with the candidate pixels;

[0178] Furthermore, the specific steps of aggregating the matching costs include:

[0179] (1) Obtain matching cost and set aggregation rules;

[0180] It should be noted that cost aggregation requires establishing connections between adjacent pixels, usually based on certain criteria, such as adjacent pixels should have continuous disparity values. These criteria are used to optimize the cost matrix so that the new cost value of each pixel at a certain disparity can be recalculated based on the cost values ​​of its adjacent pixels at the same disparity value or nearby disparity values.

[0181] (2) Use the semi-global matching algorithm to aggregate the matching costs of all paths around the pixel in one dimension under all disparities. Specifically, for each pixel k, along the path U around it, such as horizontal, vertical, diagonal, etc., calculate the path cost Q U (k, q3), wherein the semi-global matching algorithm is a prior art in the art and is not an inventive solution of the present application, and is not described in detail here;

[0182] (3) Add up all path costs to obtain the matching cost value of the pixel after aggregation, and update the disparity three-dimensional matrix.

[0183] B2.4: Create a disparity map of the same size as the left image to store the optimal disparity value for each pixel;

[0184] B2.5: For the new disparity 3D matrix For each pixel in , traverse all possible disparity values ​​of each pixel, and find the cost value of the pixel under each disparity value, and select the disparity value with the smallest cost value as the optimal disparity of the pixel. The formula is:

[0185]

[0186] Among them, q 3,k represents the optimal disparity value of pixel k, Q(k,q3) represents the cost value of pixel k when the disparity is q3, and argmin(·) represents the optimal disparity value of pixel k in the domain The value of the independent variable corresponding to the minimum value of inner Q(k,q3);

[0187] B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map, and obtain a disparity map containing the optimal disparity value of each pixel;

[0188] B2.7: Calculate the depth value corresponding to each pixel based on the disparity map, the focal length of the camera, and the distance between the two eyes The depth value is mapped to the three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle, where d guang Represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q4 represents the parallax between the two cameras.

[0189] It should be noted that, in practical applications, since the disparity value is discrete, the optimal disparity can be found by traversing all possible disparity values.

[0190] The optimization strategy generation unit adopts a spider bee optimization strategy. The specific steps of the spider bee optimization strategy include:

[0191] D1: Set the search space and randomly distribute a certain number of search agents in the search space, where each search agent represents a potential solution;

[0192] D2: Each search agent moves in the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value, wherein the gradient descent and fitness value calculation formulas are prior art contents in this field, and are not the inventive solutions of this application, and are not described in detail here;

[0193] D3: If the search agent finds a potential high-quality solution, it will follow the solution. If the quality of the solution decreases or encounters an obstacle, the search agent will choose to escape, that is, move away from the current solution and try a new search direction;

[0194] D4: Once a satisfactory solution is found, the search agent builds a nest around it, i.e., performs local optimization;

[0195] D5: Combine the solutions of different search agents through crossover operation to generate new offspring, and repeat iteratively until the preset maximum number of iterations is reached.

[0196] The switching rules in A2 include that when the left front leg and the right hind leg are supporting, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg land on the ground for support, the left front leg and the right hind leg start to swing.

[0197] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

Claims

1. A quadruped robot autonomous decision-making control system based on multimodal perception, characterized in that: include: Perception module, decision-making and planning module, execution module, feedback monitoring module; The perception module is used to collect the robot's internal state information and external environment information through multiple sensors, and integrate them to form multimodal perception information; The decision-making planning module includes a planning strategy unit, in which a gait generation strategy and a motion planning strategy are configured; The gait generation strategy is used to determine the time period, support phase and swing phase of each step, and to determine each joint angle using inverse kinematics; the motion planning strategy is used to perform motion planning and decision-making according to the gait generation strategy; The execution module includes an instruction generation unit and an execution unit; the instruction generation unit is configured with a PID control algorithm, and the PID control algorithm is used to generate control instructions for controlling different joints and drivers of the quadruped robot according to the motion planning scheme; The feedback monitoring module includes a monitoring unit and an optimization strategy generation unit. The monitoring unit is configured with a three-dimensional obstacle avoidance strategy, which is used to monitor the robot's posture, speed, acceleration, ground contact force and changes in the surrounding environment in real time, and avoid obstacles based on the monitoring results; the optimization strategy generation unit is configured with a spider bee optimization strategy, which is used to optimize the evaluation results.

2. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 1, characterized in that: The specific steps of the gait generation strategy include: A1: Obtain multimodal perception information generated by the perception module and set gait parameters according to task requirements, wherein the gait parameters include step length, step speed, and diagonal gait type; A2: According to the set gait parameters and the robot's walking speed requirements, the time of the stance phase and the swing phase is calculated and allocated by the table lookup method, and the time of the stance phase and the swing phase are added to obtain the gait cycle; at the same time, according to the diagonal gait type, the switching rules between the stance phase and the swing phase of different legs are determined; A3: Set the movement speed or acceleration state threshold h1, and calculate the switching timing between the stance phase and the swing phase of different legs based on the gait cycle and switching rules and the movement rules of different legs; If v sp >h1, or v sw <h1, then keep moving; If v sp ≤h1, or v sw ≥h1, the switch from the support phase to the swing phase is triggered, where v sp is the velocity or acceleration of the leg at the end of the stance phase, v sw represents the velocity or acceleration of the leg at the beginning of the swing phase; A4: Based on the preset target position and orientation of the robot end effector, use the Newton-Raphson algorithm to calculate the angles θ of different joints i , based on the obtained joint angles and the switching timing obtained in A3, calculate the speed u of different joints i =θ i ′, and integrate the calculated different joint angles, speeds and switching timings into gait information, where θ i represents the angle of the i-th joint, and i∈[1,n], u i represents the velocity of the i-th joint, θ i ′ represents θ i The first-order derivative of with respect to time, n represents the number of joints; A5: Perform stability analysis on the generated gait information, and adjust and optimize gait parameters and joint control based on the analysis results.

3. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 2, characterized in that: The steps for calculating the angles of different joints in A4 include: A4.1: Collect the robot's mechanical structure parameters and set the target position and direction of the robot's end effector; A4.2: Take the target position and orientation as input, run the Newton-Raphson algorithm, and output different joint angles that meet the target position and orientation requirements; A4.3: Check whether the calculated joint angles meet the physical limitations of the robot; If not satisfied, readjust the target position and direction; If satisfied, the calculated joint angle is output.

4. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 3, characterized in that: The specific steps of A4.2 include: A4.21: Receive the target position and direction of the robot end effector and convert the target position and direction into vector format to obtain the target position vector x vector and the target direction vector θ vector ; A4.22: Read the current joint angle from the sensor based on Calculate the transformation matrix M of this joint relative to the previous joint through translation transformation bian , the total transformation matrix M from the base to the end effector is obtained by accumulation total =M bian,1 ×…×M bian,n , and extract the actual position v of the end effector from the total transformation matrix ad and direction θ ad , where M bian,n Represents the transformation matrix of the nth joint.

5. The autonomous decision-making control system of a quadruped robot based on multimodal perception as claimed in claim 4, characterized in that: The specific steps of A4.2 also include: A4.23: Use sensors to obtain the actual position of the current joint angle and direction Calculate the error vector between the target position and orientation and the actual position and orientation using the formula: Among them, e vector represents the error vector between the target position and direction and the actual position and direction, w1 represents the position weight, w2 represents the direction weight, and |·| represents the absolute value; A4.24: Use the Jacobian matrix to solve the first derivative of the error vector with respect to the current joint angle And according to the first-order derivative and the error vector, calculate the increment of the joint angle in, The Jacobian matrix represents the relationship between the velocity of the robot end effector and the joint velocity, and λ represents the learning rate; A4.25: Increase Add to current joint angle Iterate and update until the preset number of iterations is reached, and then output the final joint angle.

6. The autonomous decision-making control system of a quadruped robot based on multimodal perception as claimed in claim 5, characterized in that: The monitoring unit adopts a three-dimensional obstacle avoidance strategy, and the specific steps of the three-dimensional obstacle avoidance strategy include: B1: Use binocular cameras to capture the 3D information of the surrounding environment and the distance of obstacles in real time, and perform preprocessing to obtain environmental perception image information; B2: Perform stereo matching on the environment perception images taken by the left and right cameras of the binocular camera, calculate the three-dimensional coordinates and depth information of the obstacle by calculating the disparity map and combining the internal and external parameters of the camera; B3: Use machine learning convolutional neural network algorithm to identify and classify the three-dimensional information of obstacles calculated in B2; B4: Based on the identified and classified obstacle information, as well as the robot’s current position and posture, a path planning algorithm is used to plan multiple feasible obstacle avoidance paths.

7. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 6, characterized in that: The specific steps of B2 include: B2.1: Obtaining pre-processed environment perception image information, where the environment perception image information is taken by both left and right cameras of a binocular camera; B2.2: Based on the environment perception image, for each pixel in the left image, find its corresponding point in the right image, and calculate the matching cost of each pixel in the left and right images. Store the matching costs of all pixels in a three-dimensional matrix to obtain the disparity three-dimensional matrix M DSI , the formula is: Q(q1,q2,q3)=|g L (q1,q2)-g R (q1-q3,q2)|; Where q1 represents the width of the environment perception image, q2 represents the height of the environment perception image, q3 represents the disparity search range, Q(q1,q2,q3) represents the matching cost when the disparity is q3 at the pixel position (q1,q2) in the environment perception image, and g L (q1, q2) represents the grayscale value of the left image of the binocular camera at the pixel position (q1, q2) in the environmental perception image, g R (q1-q3,q2) represents the grayscale value of the right image of the binocular camera at the pixel position (q1-q3,q2) in the environmental perception image; B2.3: Aggregate the matching costs to obtain a new disparity 3D matrix The cost value of each pixel reflects its correlation with the candidate pixels.

8. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 7, characterized in that: The specific steps of B2 also include: B2.4: Create a disparity map of the same size as the left image to store the optimal disparity value for each pixel; B2.5: For the new disparity 3D matrix For each pixel in , traverse all possible disparity values ​​of each pixel, and find the cost value of the pixel under each disparity value, and select the disparity value with the smallest cost value as the optimal disparity of the pixel. The formula is: Among them, q 3,k represents the optimal disparity value of pixel k, Q(k,q3) represents the cost value of pixel k when the disparity is q3, and argmin(·) represents the optimal disparity value of pixel k in the domain The value of the independent variable corresponding to the minimum value of inner Q(k,q3); B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map, and obtain a disparity map containing the optimal disparity value of each pixel; B2.7: Calculate the depth value corresponding to each pixel based on the disparity map, the focal length of the camera, and the distance between the two eyes The depth value is mapped to the three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle, where d guang Represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q4 represents the parallax between the two cameras.

9. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 8, characterized in that: The optimization strategy generation unit adopts the spider bee optimization strategy, and the specific steps of the spider bee optimization strategy include: D1: Set the search space and randomly distribute a certain number of search agents in the search space, where each search agent represents a potential solution; D2: Each search agent moves in the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value; D3: If the search agent finds a potential high-quality solution, it will follow the solution. If the quality of the solution decreases or encounters an obstacle, the search agent will choose to escape, that is, move away from the current solution and try a new search direction; D4: Once a satisfactory solution is found, the search agent builds a nest around it, i.e., performs local optimization; D5: Combine the solutions of different search agents through crossover operation to generate new offspring, and repeat iteratively until the preset maximum number of iterations is reached.

10. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 9, characterized in that: The switching rule in A2 includes that when the left front leg and the right hind leg are supporting, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg are supporting on the ground, the left front leg and the right hind leg start to swing.

Images