Autonomous Decision-making Control System for Quadruped Robot Based on Multimodal Sensing

Abstract

The present invention discloses an autonomous decision-making control system for a quadruped robot based on multi-modal perception, belonging to the field of robot motion technology. Specifically, it includes: collecting visual, tactile, and auditory information through multi-sensors such as cameras, force sensors, and microphone arrays, and performing preprocessing; according to the processed information, adopting a motion planning strategy to generate a diagonal gait, leg trajectories, and a motion path planning scheme; generating specific control instructions to drive the robot to move and monitoring and adjusting in real time; after completing the task, evaluating the motion performance of the robot, and optimizing the motion planning and control methods according to the evaluation results; the present invention improves the autonomous motion ability and adaptability of the quadruped robot in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot motion, specifically an autonomous decision-making control system for a quadruped robot based on multi-modal perception. Background Technique

[0002] In the research of quadruped robots, the autonomous decision-making control method is a core issue, which involves multiple aspects such as perception, planning, decision-making, and execution of the robot in a complex environment. To achieve autonomous decision-making control, quadruped robots need to be equipped with various sensors, such as depth cameras, lidar, and inertial measurement units, for environmental perception and navigation. At the same time, advanced computing boards and algorithms are also required to process the sensor data and make corresponding decisions and plans.

[0003] The autonomous decision-making control methods for quadruped robots include rule-based control methods, model-based control methods, learning-based control methods, and hybrid control methods. The rule-based control method judges the behavior of the robot through preset rules and logics, but has poor flexibility; the model-based control method establishes the dynamic model and kinematic model of the robot and uses the mathematical model to predict and control the behavior of the robot. However, due to the complexity of the actual robot system, the established model often has errors, resulting in unsatisfactory control effects; the learning-based control method enables the robot to continuously learn and optimize its behavior during the interaction with the environment through technologies such as machine learning or reinforcement learning. However, a large amount of training data and time are required to optimize the model. Moreover, the performance of the model depends to a large extent on the quality and quantity of the training data; although the hybrid control method combines the advantages of multiple control methods, it needs to process multiple control strategies simultaneously, increasing the complexity and implementation difficulty of the system.

[0004] For example, the Chinese patent application with the authorization announcement number CN113671976B discloses a motion positioning control method for a three-legged support type pipeline robot, including: system startup; obtaining attitude information; establishing a static attitude model; obtaining ranging information; switching the straight / bent pipe control mode; motor operation; current PID module operation; segmented control mode; system stop; through the information feedback of the external perception module, positioning module, and motion module, the attitude information of the robot is formed after processing, and the controller decides to adopt the optimal control strategy, enabling the pipeline robot to have a certain autonomous recognition ability, reducing the burden and risk of misoperation of the operator, and adjusting the robot attitude at any time according to the motion path, so that it can autonomously turn in the pipeline and operate in straight pipes, bent pipes, and obstacles.

[0005] The patent application with the publication number CN118625847A discloses a lightweight quadruped robot motion method based on a cross-modal perception attention mechanism, including: obtaining the body perception data and visual perception data of the quadruped robot; performing multi-modal data fusion based on the body perception data and visual perception data to obtain the fused multi-modal features; obtaining a trained cross-modal perception attention controller and a low-level reinforcement learning motion controller; and controlling the motion of the quadruped robot through the trained cross-modal perception attention controller and the low-level reinforcement learning motion controller. This technical solution adds a cross-modal attention control module to the autonomous obstacle avoidance decision-making task of the quadruped robot, thereby realizing lightweight quadruped robot obstacle avoidance motion with dynamic and context awareness on the robot, and the robot can autonomously select information switching between different modalities according to the task requirements.

[0006] The above existing technologies all have the following problems: lack of optimization strategies and real-time performance; lack of sufficient experimental verification and performance evaluation; more suitable for specific tasks or scenarios, and there are limitations in the application scope. Summary of the Invention

[0007] In view of the deficiencies of the prior art, the present invention proposes an autonomous decision-making control system for a quadruped robot based on multi-modal perception. Visual, force, and auditory information are collected through multi-sensors such as cameras, force sensors, and microphone arrays and preprocessed; according to the processed information, a motion planning strategy is used to generate a diagonal gait, leg trajectories, and a motion path planning scheme; specific control instructions are generated to drive the robot to move and are monitored and adjusted in real time; after the task is completed, the motion performance of the robot is evaluated, and the motion planning and control methods are optimized according to the evaluation results; the present invention improves the autonomous motion ability and adaptability of the quadruped robot in complex environments.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] An autonomous decision-making control system for a quadruped robot based on multi-modal perception, including: a perception module, a decision-making and planning module, an execution module, and a feedback monitoring module;

[0010] The perception module is used to collect the internal state information and external environment information of the robot through multi-sensors and integrate them into multi-modal perception information;

[0011] The decision-making and planning module includes a planning strategy unit, and a gait generation strategy and a motion planning strategy are configured in the planning strategy unit; the gait generation strategy is used to determine the time period, support phase, and swing phase of each step, and use inverse kinematics to determine the angle of each joint; the motion planning strategy is used to perform motion planning and decision-making according to the gait generation strategy;

[0012] The execution module includes an instruction generation unit and an execution unit; a PID control algorithm is configured in the instruction generation unit, and the PID control algorithm is used to generate control instructions for controlling different joints and drivers of the quadruped robot according to the motion planning scheme.

[0013] The feedback monitoring module includes a monitoring unit and an optimization strategy generation unit. A three-dimensional obstacle avoidance strategy is configured in the monitoring unit, and the three-dimensional obstacle avoidance strategy is used to monitor the changes in the robot's posture, speed, acceleration, ground contact force, and surrounding environment in real time and perform obstacle avoidance according to the monitoring results. A spider wasp optimization strategy is configured in the optimization strategy generation unit, and the spider wasp optimization strategy is used to optimize the evaluation results.

[0014] Specifically, the specific steps of the gait generation strategy include:

[0015] A1: Obtain the multi-modal perception information generated by the perception module and set the gait parameters according to the task requirements. The gait parameters include step length, step speed, and diagonal gait type.

[0016] A2: Calculate and allocate the time of the support phase and the swing phase by means of a look-up table according to the set gait parameters and the robot's walking speed requirement, and add the time of the support phase and the swing phase to obtain the gait cycle. At the same time, according to the diagonal gait type, determine the switching rules between different legs during the support phase and the swing phase.

[0017] A3: Set the motion speed or acceleration state threshold h 1 , and based on the gait cycle and the switching rules, combined with the motion laws of different legs, calculate the switching timing between the support phase and the swing phase of different legs.

[0018] If v sp > h 1 , or v sw < h 1 , then maintain the motion state.

[0019] If v sp ≤ h 1 , or v sw ≥ h 1 , then trigger the switch from the support phase to the swing phase, where v sp represents the speed or acceleration of the leg at the end of the support phase, and v sw represents the speed or acceleration of the leg at the start of the swing phase.

[0020] A4: According to the preset target position and direction of the robot's end effector, use the Newton-Raphson algorithm to calculate the angles θ i of different joints, and based on the obtained joint angles, combined with the switching timing obtained in A3, calculate the speeds u i= θ i ′, and integrate the calculated different joint angles, speeds, and switching times into gait information, where θ i represents the angle of the i-th joint, and i ∈ [1, n], u i represents the speed of the i-th joint, θ i ′ represents the first derivative of θ i with respect to time, and n represents the number of joints;

[0021] A5: Conduct a stability analysis on the generated gait information, and adjust and optimize the gait parameters and joint control according to the analysis results.

[0022] Specifically, the calculation steps of the angles of different joints in A4 include:

[0023] A4.1: Collect the mechanical structure parameters of the robot, and set the target position and direction of the robot's end effector;

[0024] A4.2: Take the target position and direction as inputs, run the Newton-Raphson algorithm, and output different joint angles that meet the requirements of the target position and direction;

[0025] A4.3: Check whether the calculated joint angles meet the physical limitations of the robot;

[0026] If not, readjust the target position and direction;

[0027] If so, output the calculated joint angles.

[0028] Specifically, the specific steps of A4.2 include:

[0029] A4.21: Receive the target position and direction of the robot's end effector, and convert the target position and direction into vector format to obtain the target position vector x vector and the target direction vector θ vector ;

[0030] A4.22: Read the current joint angles from the sensor Based on Calculate the transformation matrix M bian of this joint relative to the previous joint through translational transformation, and obtain the total transformation matrix M total from the base to the end effector by accumulation = M bian,1 ×…×M bian,n , and extract the actual position v ad and direction θ ad of the end effector from the total transformation matrix, where M bian,n represents the transformation matrix of the n-th joint.

[0031] Specifically, the specific steps of A4.2 further include:

[0032] A4.23: Use a sensor to obtain the actual position and direction at the current joint angle, and calculate the error vector between the target position and direction and the actual position and direction. The formula is:

[0033]

[0034] where e vector represents the error vector between the target position and direction and the actual position and direction, w 1 represents the position weight, w 2 represents the direction weight, and |·| represents taking the absolute value;

[0035] A4.24: Use the Jacobian matrix to solve the first-order derivative of the error vector with respect to the current joint angle and calculate the increment of the joint angle based on the first-order derivative and the error vector where represents the Jacobian matrix that represents the relationship between the speed of the robot's end effector and the joint speed, and λ represents the learning rate;

[0036] A4.25: Add the increment to the current joint angle and perform iterative update until the preset number of iterations is reached, and output the final joint angle.

[0037] Specifically, the monitoring unit adopts a three-dimensional obstacle avoidance strategy. The specific steps of the three-dimensional obstacle avoidance strategy include:

[0038] B1: Use a binocular camera to capture the three-dimensional information and obstacle distance of the surrounding environment in real time, and perform preprocessing to obtain environmental perception image information;

[0039] B2: Perform stereo matching on the environmental perception images captured by the left and right cameras of the binocular camera, calculate the disparity map, and combine the internal and external parameters of the camera to calculate the three-dimensional coordinates and depth information of the obstacle;

[0040] B3: Use the machine learning convolutional neural network algorithm to identify and classify the three-dimensional information of the obstacle calculated in B2;

[0041] B4: According to the identified and classified obstacle information, as well as the current position and posture of the robot, use the path planning algorithm to plan multiple feasible obstacle avoidance paths.

[0042] Specifically, the specific steps of B2 include:

[0043] B2.1: Obtain the pre - processed environmental perception image information, where the environmental perception image information is captured jointly by the left and right cameras of the binocular camera;

[0044] B2.2: Based on the environmental perception image, for each pixel in the left image, find its corresponding point in the right image, calculate the matching cost of each pixel in the left and right images, store the matching costs of all pixels in a three - dimensional matrix, and obtain the disparity three - dimensional matrix M DSI , the formula is:

[0045] Q(q 1 ,q 2 ,q 3 ) = |g L (q 1 ,q 2 ) - g R (q 1 - q 3 ,q 2 )|;

[0046] Among them, q 1 represents the width of the environmental perception image, q 2 represents the height of the environmental perception image, q 3 represents the disparity search range, Q(q 1 ,q 2 ,q 3 ) represents the matching cost at the pixel position (q 1 ,q 2 ) in the environmental perception image when the disparity is q 3 , g L (q 1 ,q 2 ) represents the gray value of the left image of the binocular camera at the pixel position (q 1 ,q 2 ), g R (q 1 - q 3 ,q 2 ) represents the gray value of the right image of the binocular camera at the pixel position (q 1 - q 3 ,q 2 );

[0047] B2.3: Aggregate the matching costs to obtain a new disparity three - dimensional matrix where the cost value of each pixel reflects its correlation with the candidate pixels.

[0048] Specifically, the specific steps of B2 further include:

[0049] B2.4: Create a disparity map with the same size as the left image to store the optimal disparity value for each pixel;

[0050] B2.5: For each pixel in the new disparity three-dimensional matrix , traverse all possible disparity values for each pixel, and at each disparity value, find the cost value of the pixel, and select the disparity value with the minimum cost value as the optimal disparity of the pixel. The formula is:

[0051]

[0052] where q 3,k represents the optimal disparity value of pixel k, and Q(k, q 3 ) represents the cost value of pixel k at a disparity of q 3 . argmin(·) represents the independent variable value corresponding to the minimum value of Q(k, q ) within the domain; 3

[0053] B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map to obtain a disparity map containing the optimal disparity value of each pixel;

[0054] B2.7: Calculate the depth value corresponding to each pixel according to the disparity map, the focal length of the camera, and the distance between the two eyes, and map the depth value into three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle. Among them, d guang represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q 4 represents the disparity between the two cameras.

[0055] The optimization strategy generation unit adopts the spider wasp optimization strategy. The specific steps of the spider wasp optimization strategy include:

[0056] D1: Set the search space, and randomly distribute a certain number of search agents within the search space. Among them, each search agent represents a potential solution;

[0057] D2: Each search agent moves within the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value;

[0058] D3: If a search agent discovers a potential high-quality solution, the search agent will follow the solution. If the quality of the solution deteriorates or it encounters an obstacle, the search agent will choose to flee, that is, move away from the current solution and try a new search direction;

[0059] D4: Once a satisfactory solution is found, the search agent will build a nest around it, that is, perform local optimization;

[0060] D5: Combine the solutions of different search agents through crossover operation to generate new offspring, and iterate repeatedly until the preset maximum number of iterations is reached.

[0061] The switching rules in the above-mentioned A2 include that when the left front leg and the right hind leg are supporting, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg land to support, the left front leg and the right hind leg start to swing.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0063] 1. The present invention proposes an autonomous decision-making control system for a quadruped robot based on multi-modal perception, and has optimized improvements in the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production and working costs.

[0064] 2. The present invention proposes an autonomous decision-making control system for a quadruped robot based on multi-modal perception. Through multi-sensor fusion, multi-modal perception information is formed, realizing comprehensive monitoring of the internal state and external environment of the robot; the gait generation strategy and motion planning strategy in the planning strategy unit can accurately determine the gait and motion path of the robot, ensuring that the robot walks stably and effectively avoids obstacles. This comprehensive perception and planning ability improves the adaptability and walking efficiency of the robot in complex environments.

[0065] 3. The present invention proposes an autonomous decision-making control system for a quadruped robot based on multi-modal perception. Precise control instructions are generated through the PID control algorithm to drive the various joints and actuators of the quadruped robot to work together; at the same time, the stereo obstacle avoidance strategy and the optimization strategy generation unit in the feedback monitoring module can monitor the motion state of the robot in real time and optimize the evaluation results, ensuring that the robot maintains the best performance during walking; this closed-loop control mechanism not only improves the motion accuracy and stability of the robot, but also provides guarantee for its continuous and stable operation in complex and changeable environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is the architecture diagram of the autonomous decision-making control system for a quadruped robot based on multi-modal perception of the present invention;

[0067] Figure 2 It is the overall implementation flowchart of the autonomous decision-making control system for a quadruped robot based on multi-modal perception of the present invention;

[0068] Figure 3 It is the implementation flowchart of the gait generation strategy of the autonomous decision-making control system for a quadruped robot based on multi-modal perception of the present invention;

[0069] Figure 4This is the flowchart for implementing the three-dimensional obstacle avoidance strategy of the autonomous decision-making control system for a quadruped robot based on multi-modal perception in the present invention. Specific implementation manners

[0070] Example 1

[0071] Please refer to Figure 1 and Figure 2 An example provided by the present invention: An autonomous decision-making control system for a quadruped robot based on multi-modal perception, including the following steps:

[0072] A perception module, a decision-making and planning module, an execution module, and a feedback and monitoring module;

[0073] The perception module is used to collect the internal state information and external environment information of the robot through multiple sensors, and integrate them to form multi-modal perception information, providing data support for the decision-making and planning module;

[0074] The decision-making and planning module is used to formulate a motion plan and decision according to the multi-modal perception information, and generate a motion planning scheme;

[0075] The execution module is used to generate specific control instructions according to the motion planning scheme, and send the control instructions to the execution mechanism of the quadruped robot;

[0076] The feedback and monitoring module is used to monitor the running state of the entire system in real time, and feedback the monitoring results to the decision-making and planning module, and evaluate the motion performance of the robot after the robot completes the target task.

[0077] The perception module includes: an internal sensor unit, an external sensor unit, and a preprocessing unit;

[0078] The internal sensor unit includes an inertial measurement unit and a joint motor encoder; the inertial measurement unit is used to measure information such as the angular velocity, acceleration, and attitude angle of the robot; the joint motor encoder is used to measure the rotation angle and speed of the joint;

[0079] The external sensor unit includes a lidar, a camera, and a foot-end pressure sensor. The lidar is used to accurately measure the distance between the robot and surrounding objects; the camera is used to capture environmental images to achieve visual perception; the foot-end pressure sensor is used to monitor the contact situation between the robot's legs and the ground;

[0080] The preprocessing unit is used to perform operations such as data cleaning, format unification, and noise filtering to ensure the accuracy and reliability of the perception information.

[0081] The decision-making and planning module includes: a planning strategy unit, a scheme formulation unit, and an evaluation unit;

[0082] A planning strategy unit is used to perform motion planning and decision-making based on the preprocessed multi-modal perception information, using a gait generation strategy and a motion planning strategy, such as diagonal gait selection, leg trajectory planning, and motion path determination.

[0083] Among them, the gait generation strategy is a key sub-strategy in the planning strategy unit. Its main task is to determine the time period, stance phase, and swing phase of each step according to the motion requirements of the quadruped robot and the current environmental information. The stance phase refers to which feet of the robot are in contact with the ground during walking, while the swing phase refers to which feet are off the ground. In addition, the gait generation strategy also needs to use inverse kinematics to determine the angles of each joint to ensure that the robot can walk according to the predetermined gait.

[0084] The motion planning strategy is based on the results of the gait generation strategy and further performs detailed motion planning and decision-making. It needs to consider parameters such as the motion path, speed, and acceleration of the robot, as well as possible obstacles and terrain changes. The goal of the motion planning strategy is to generate a safe and efficient motion planning scheme so that the robot can successfully complete the target task.

[0085] Furthermore, the specific steps of the motion planning strategy include:

[0086] (1) In the known environmental map, use a path planning algorithm to plan the optimal path from the starting point to the ending point. In the present invention, the path planning algorithm adopts the Dijkstra algorithm. The Dijkstra algorithm is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here.

[0087] (2) According to the dynamic model and kinematic constraints of the quadruped robot, as well as the terrain features in the environmental map, optimize the motion parameters, such as step length, step frequency, and joint angles.

[0088] (3) According to the gait generation strategy, combine the optimal path and motion parameters to decide the specific execution plan for each step, and convert the execution plan into control instructions and send them to the execution mechanism of the quadruped robot.

[0089] 1) Select a gait strategy from the preset gait generation strategy library according to the current environmental conditions and task requirements.

[0090] 2) Integrate the optimal path generated by the path planning algorithm with the motion parameters obtained by the motion parameter optimization algorithm.

[0091] 3) Generate the specific execution plan for each step according to the gait strategy and motion parameters. Among them, the execution plan should include detailed information such as the starting position, ending position, walking path, and joint angle change of each step.

[0092] 4) Convert the execution plan into control instructions that can be understood by the robot's actuators. Among them, the control instructions usually include parameters such as joint angles, motor speeds, and torques, and these parameters will directly control the robot's movement.

[0093] The scheme formulation unit is used to generate a motion planning scheme according to the motion planning and decision-making results;

[0094] The evaluation unit is used to consider the stability and energy consumption factors of the robot during the implementation of the motion planning scheme to ensure the feasibility and efficiency of the motion planning scheme.

[0095] The execution module includes: an instruction generation unit and an execution unit;

[0096] The instruction generation unit is used to generate control instructions for different joints and actuators of the quadruped robot through the PID control algorithm according to the motion planning scheme. Among them, the PID control algorithm is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here;

[0097] The execution unit, such as a leg actuator, is used to execute the control instructions and drive the quadruped robot to move.

[0098] The feedback monitoring module includes: a monitoring unit, a performance evaluation unit, an optimization strategy generation unit, and a feedback guidance unit;

[0099] The monitoring unit is used to continuously monitor the motion states such as the robot's posture, speed, acceleration, and ground contact force and the changes in the surrounding environment, and feedback them to the autonomous decision-making control system of the quadruped robot based on multi-modal perception for adjustment and optimization;

[0100] The performance evaluation unit is used to quantitatively evaluate the motion performance of the robot and generate an evaluation result;

[0101] The optimization strategy generation unit is used to generate improvement measures and optimization strategies according to the evaluation result;

[0102] The feedback guidance unit is used to use the evaluation result and improvement measures as feedback to guide the acquisition, preprocessing, and motion planning and decision-making processes of new multi-modal perception information.

[0103] In summary, in the present invention, the overall implementation process includes:

[0104] S1: Collect multi-modal perception information through multi-sensors and perform preprocessing. The multi-modal perception information includes visual information, force perception information, and auditory perception information. The visual information includes terrain features and obstacle positions. The force perception information includes softness / hardness and friction. The auditory perception information refers to capturing surrounding sounds using a microphone array. The multi-sensors include cameras, force sensors, and microphone arrays. The preprocessing includes data cleaning, format unification, and noise filtering to obtain accurate and reliable environmental perception information;

[0105] S2: According to the processed multi-modal perception information, use motion planning strategies to perform motion planning and decision-making, generate a motion planning scheme, including selecting a diagonal gait, planning leg trajectories, and determining a motion path. At the same time, consider the stability and energy consumption factors of the robot;

[0106] S3: Generate specific control instructions according to the motion planning scheme. These instructions will be used to control different joints and actuators of the quadruped robot to achieve the desired motion;

[0107] S4: Send the control instructions to the actuators of the quadruped robot, such as leg actuators, and real-time monitor the motion state of the robot and changes in the surrounding environment, including posture, speed, acceleration, and ground contact force. At the same time, make necessary adjustments and optimizations to the control instructions according to the real-time monitored feedback information;

[0108] S5: When the robot completes the target task, evaluate the motion performance of the robot, including evaluating aspects such as the stability, motion efficiency, and energy consumption of the robot. And according to the evaluation results, improve and optimize the motion planning strategy and the control instruction generation method to generate improvement measures;

[0109] S6: Use the evaluation results and improvement measures as feedback to guide the new multi-modal perception information acquisition, preprocessing, and motion planning and decision-making processes.

[0110] Embodiment 2

[0111] Please refer to Figure 3 , the specific steps of the gait generation strategy in this embodiment include:

[0112] A1: Obtain the multi-modal perception information generated by the perception module and set gait parameters according to the task requirements. The gait parameters include step length, step speed, and diagonal gait type;

[0113] Among them, the diagonal gait types include: a diagonal gait where the left front leg and the right hind leg support simultaneously, and the right front leg and the left hind leg swing simultaneously; a diagonal gait where the right front leg and the left hind leg support simultaneously, and the left front leg and the right hind leg swing simultaneously.

[0114] A2: According to the set gait parameters and the walking speed requirements of the robot, calculate and allocate the time of the stance phase and the swing phase through the look-up table method, and add the time of the stance phase and the swing phase to obtain the gait cycle. At the same time, according to the diagonal gait type, determine the switching rules between different legs during the stance phase and the swing phase;

[0115] Among them, the look-up table method is based on previous experimental or simulation data; the gait cycle refers to the process from the heel strike of one side to the heel strike of the same side again, including two stages: the stance phase and the swing phase. During one gait cycle, the robot will experience the transition from the stance phase to the swing phase and then back from the swing phase to the stance phase.

[0116] A3: Set the motion speed or acceleration state threshold h 1 , and based on the gait cycle and the switching rules, combined with the motion laws of different legs, calculate the switching timing between different legs during the stance phase and the swing phase;

[0117] If v sp > h 1 , or v sw < h 1 , then maintain the motion state;

[0118] If v sp ≤ h 1 , or v sw ≥ h 1 , then trigger the switch from the stance phase to the swing phase, where v sp represents the speed or acceleration of the leg at the end of the stance phase, and v sw represents the speed or acceleration of the leg at the start of the swing phase;

[0119] A4: According to the preset target position and direction of the robot end effector, use the Newton-Raphson algorithm to calculate the angles θ i of different joints. Based on the obtained joint angles, combined with the switching timing obtained in A3, calculate the speeds u i = θ i ′, and integrate the calculated different joint angles, speeds, and switching timing into gait information, where θ i represents the angle of the i-th joint, and i ∈ [1, n], u i represents the speed of the i-th joint, and θ i ′ represents the first derivative of θ i with respect to time, and n represents the number of joints;

[0120] A5: Conduct a stability analysis on the generated gait information, and adjust and optimize the gait parameters and joint control according to the analysis results.

[0121] Furthermore, the specific steps of A5 include:

[0122] (1) Gait stability assessment:

[0123] 1) Use gait analysis instruments, such as three-dimensional gait analyzers and pressure gait analyzers, to collect gait data, including gait cycle, step length, walking speed, step frequency, and joint angles;

[0124] 2) Use the scale assessment method to perform stability analysis on the collected gait data, including the consistency of the gait cycle, the uniformity of the step length, and the stability of the walking speed. Among them, the scale assessment method is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here;

[0125] 3) Evaluate gait coherence, that is, the ability to smoothly transition between different stages during walking;

[0126] 4) According to the evaluation results, judge whether there are problems with gait stability, such as an extended or shortened gait cycle, a reduced step length, or a decreased walking speed;

[0127] (2) Optimization of gait parameters and joint control:

[0128] 1) Identify key gait parameters that affect gait stability, such as step length, step width, walking speed, and joint angles;

[0129] 2) According to the evaluation results, formulate targeted adjustment strategies, such as increasing the step length, adjusting the walking speed, and optimizing the joint angles;

[0130] 3) Use biomechanical principles to optimize joint control strategies, such as improving joint coordination;

[0131] 4) Implement the adjustment strategies and collect feedback data to evaluate the adjustment effect. According to the feedback data, further adjust and optimize the gait parameters and joint control strategies.

[0132] The calculation steps for the angles of different joints in A4 include:

[0133] A4.1: Through the robot user manual, collect the mechanical structure parameters of the robot and set the target position and direction of the robot end effector. Among them, the mechanical structure parameters include link lengths, joint types, and joint ranges; the target position is usually a coordinate point in three-dimensional space, and the target direction is usually a three-dimensional vector or rotation matrix;

[0134] A4.2: Use the target position and direction as inputs, run the Newton-Raphson algorithm, and output the different joint angles that meet the requirements of the target position and direction;

[0135] A4.3: Check whether the calculated joint angles meet the physical limitations of the robot;

[0136] If not satisfied, readjust the target position and orientation;

[0137] If satisfied, output the calculated joint angles.

[0138] The specific steps of A4.2 include:

[0139] A4.21: Receive the target position and orientation of the robot end effector, and convert the target position and orientation into vector format to obtain the target position vector x vector and the target direction vector θ vector ;

[0140] The specific process of converting the target position and orientation into vector format includes:

[0141] (1) Conversion of the target position into vector format:

[0142] 1) Determine the target position: The target position is a point in three-dimensional space, represented by its x, y, and z coordinates in the Cartesian coordinate system;

[0143] 2) Construct the position vector:

[0144] Use the x, y, and z coordinates of the target position as the components of the vector to construct a three-dimensional vector.

[0145] Exemplarily, if the target position is the point P(x, y, z), the position vector can be represented as

[0146] (2) Conversion of the target direction into vector format:

[0147] 1) Determine the target direction: The target direction is a unit vector pointing in a specific direction;

[0148] 2) If the target direction is already a unit vector, directly use this vector; if the target direction is a non-unit vector, it needs to be normalized to obtain a unit vector.

[0149] Exemplarily, if the direction vector is whose length is then the unit direction vector is

[0150] A4.22: Read the current joint angles from the sensor Based on Calculate the transformation matrix M of this joint relative to the previous joint through translational transformation bian , and obtain the total transformation matrix M from the base to the end effector through accumulation total = M bian,1 ×…×M bian,nand extract the actual position v of the end effector from the total transformation matrix ad and the direction θ ad where M bian,n represents the transformation matrix of the nth joint;

[0151] Furthermore, the implementation steps of the transformation matrix include:

[0152] (1) Determine the joint type: First, determine whether the joint is a rotational joint or a translational joint;

[0153] (2) Select the transformation matrix: Select the corresponding transformation matrix from the above formula according to the joint type and the current angle or displacement;

[0154] (3) Apply the transformation matrix: Apply the selected transformation matrix to the coordinate system of the previous joint to obtain the coordinate system of the current joint.

[0155] A4.23: Use sensors to obtain the actual position and direction at the current joint angle and calculate the error vector between the target position and direction and the actual position and direction. The formula is:

[0156]

[0157] where e vector represents the error vector between the target position and direction and the actual position and direction, w 1 represents the position weight, w 2 represents the direction weight, and |·| represents taking the absolute value;

[0158] Furthermore, the specific steps of A4.23 include:

[0159] (1) Set the target position and target direction that the end effector of the robot is expected to reach;

[0160] (2) Use sensors or kinematic models to calculate the actual position and direction of the end effector of the robot according to the current joint angles;

[0161] (3) Calculate the vector difference between the target position and the actual position, that is, the position error vector, which is usually a three-dimensional vector representing the deviations in the x, y, and z coordinates;

[0162] (4) Calculate the error between the target direction and the actual direction;

[0163] (5) Combine the position error and the direction error into a unified error vector through a weighted algorithm.

[0164] A4.24: Use the Jacobian matrix to solve the first-order derivative of the error vector with respect to the current joint angles And calculate the increment of the joint angle based on the first derivative and the error vector Wherein The Jacobian matrix representing the relationship between the velocity of the robot end effector and the joint velocity, and λ represents the learning rate;

[0165] A4.25: Add the increment To the current joint angle For iterative update until the preset number of iterations is reached, and output the final joint angle.

[0166] Embodiment 3

[0167] Please refer to Figure 4 , In this embodiment, the monitoring unit adopts a three-dimensional obstacle avoidance strategy, and the specific steps of the three-dimensional obstacle avoidance strategy include:

[0168] B1: Real-time capture the three-dimensional information and obstacle distance of the surrounding environment through a binocular camera, and perform preprocessing to obtain environmental perception image information;

[0169] B2: Perform stereo matching on the environmental perception images captured by the left and right cameras of the binocular camera, calculate the disparity map, and combine the internal and external parameters of the camera to calculate the three-dimensional coordinates and depth information of the obstacle;

[0170] B3: Use the convolutional neural network algorithm to identify and classify the three-dimensional information of the obstacle calculated in B2. Among them, the convolutional neural network algorithm is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here;

[0171] B4: According to the identified and classified obstacle information, as well as the current position and attitude of the robot, use the path planning algorithm to plan multiple feasible obstacle avoidance paths. Among them, the path planning algorithm is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here.

[0172] The specific steps of B2 include:

[0173] B2.1: Obtain the preprocessed environmental perception image information, and the environmental perception image information is jointly captured by the left and right cameras of the binocular camera;

[0174] B2.2: Based on the environmental perception image, for each pixel in the left image, find its corresponding point in the right image, calculate the matching cost of each pixel in the left and right images, store the matching costs of all pixels in a three-dimensional matrix, and obtain the disparity three-dimensional matrix M DSI , The formula is:

[0175] Q(q 1 ,q 2 ,q3 ) = |g L (q 1 , q 2 ) - g R (q 1 - q 3 , q 2 )|;

[0176] Among them, q 1 represents the width of the environmental perception image, q 2 represents the height of the environmental perception image, q 3 represents the disparity search range, Q(q 1 , q 2 , q 3 ) represents that in the environmental perception image, at the pixel position (q 1 , q 2 ), when the disparity is q 3 , the matching cost, g L (q 1 , q 2 ) represents that in the environmental perception image, the grayscale value of the left image of the binocular camera at the pixel position (q 1 , q 2 ), g R (q 1 - q 3 , q 2 ) represents that in the environmental perception image, the grayscale value of the right image of the binocular camera at the pixel position (q 1 - q 3 , q 2 );

[0177] B2.3: Aggregate the matching cost to obtain a new disparity three-dimensional matrix where the cost value of each pixel reflects its correlation with the candidate pixels;

[0178] Furthermore, the specific steps for aggregating the matching cost include:

[0179] (1) Obtain the matching cost and set the aggregation rule;

[0180] It should be noted that cost aggregation needs to establish the connection between adjacent pixels. Usually, based on certain criteria, such as adjacent pixels should have continuous disparity values, these criteria are used to optimize the cost matrix so that the new cost value of each pixel at a certain disparity can be recalculated according to the cost values of its adjacent pixels at the same disparity or nearby disparities.

[0181] (2) Use the semi-global matching algorithm to perform one-dimensional aggregation of the matching costs at all disparities of the pixels along all paths around the pixels. Specifically, for each pixel k, calculate the path cost Q U (k, q 3 ) along its surrounding paths U, such as horizontal, vertical, diagonal, etc. The semi-global matching algorithm is the prior art content in the art and is not the creative solution of this application, so it will not be elaborated here;

[0182] (3) Add up all the path costs to obtain the aggregated matching cost value of the pixel and update the disparity three-dimensional matrix.

[0183] B2.4: Create a disparity map with the same size as the left image to store the optimal disparity value of each pixel;

[0184] B2.5: For each pixel in the new disparity three-dimensional matrix , traverse all possible disparity values of each pixel, and at each disparity value, find the cost value of the pixel, and select the disparity value with the minimum cost value as the optimal disparity of the pixel. The formula is:

[0185]

[0186] where q 3,k represents the optimal disparity value of pixel k, Q(k, q 3 ) represents the cost value of pixel k at a disparity of q 3 , and argmin(·) represents the independent variable value corresponding to the minimum value of Q(k, q ) within the domain; 3 ;

[0187] B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map to obtain a disparity map containing the optimal disparity value of each pixel;

[0188] B2.7: Calculate the depth value corresponding to each pixel according to the disparity map, the focal length of the camera, and the distance between the two eyes, and map the depth value into three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle. Among them, d guang represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q 4 represents the disparity between the two cameras.

[0189] It should be noted that in practical applications, since the disparity values are discrete, the optimal disparity can be found by traversing all possible disparity values.

[0190] The optimization strategy generation unit adopts a spider wasp optimization strategy. The specific steps of the spider wasp optimization strategy include:

[0191] D1: Set the search space, and randomly distribute a certain number of search agents within the search space. Among them, each search agent represents a potential solution.

[0192] D2: Each search agent moves within the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value. Among them, the calculation formulas of gradient descent and fitness value are the prior art content in this field and are not the creative solutions of this application, so they will not be elaborated here.

[0193] D3: When a search agent discovers a potential high-quality solution, the search agent will follow this solution. If the quality of the solution deteriorates or an obstacle is encountered, the search agent will choose to escape, that is, move away from the current solution and try a new search direction.

[0194] D4: Once a satisfactory solution is found, the search agent will build a nest around it, that is, perform local optimization.

[0195] D5: Combine the solutions of different search agents through crossover operations to generate new offspring, and iterate repeatedly until the preset maximum number of iterations is reached.

[0196] The switching rule in A2 includes that when the left front leg and the right hind leg support, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg land and support, the left front leg and the right hind leg start to swing.

[0197] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make changes, modifications, substitutions, and variations to the above embodiments without departing from the purpose and scope protected by the present invention. All of these fall within the protection scope of the present invention.

Claims

1. A quadruped robot autonomous decision-making control system based on multimodal perception, characterized in that: include: Perception module, decision-making and planning module, execution module, feedback monitoring module; The perception module is used to collect the robot's internal state information and external environment information through multiple sensors, and integrate them to form multimodal perception information; The decision-making planning module includes a planning strategy unit, in which a gait generation strategy and a motion planning strategy are configured; The gait generation strategy is used to determine the time period, support phase and swing phase of each step, and to determine each joint angle using inverse kinematics; the motion planning strategy is used to perform motion planning and decision-making according to the gait generation strategy; The execution module includes an instruction generation unit and an execution unit; the instruction generation unit is configured with a PID control algorithm, and the PID control algorithm is used to generate control instructions for controlling different joints and drivers of the quadruped robot according to the motion planning scheme; The feedback monitoring module includes a monitoring unit and an optimization strategy generation unit. The monitoring unit is configured with a three-dimensional obstacle avoidance strategy, which is used to monitor the robot's posture, speed, acceleration, ground contact force and changes in the surrounding environment in real time, and avoid obstacles according to the monitoring results; the optimization strategy generation unit is configured with a spider bee optimization strategy, which is used to optimize the evaluation results; The specific steps of the gait generation strategy include: A1: Obtain multimodal perception information generated by the perception module and set gait parameters according to task requirements, wherein the gait parameters include step length, step speed, and diagonal gait type; A2: According to the set gait parameters and the robot's walking speed requirements, the time of the stance phase and the swing phase is calculated and allocated by the table lookup method, and the time of the stance phase and the swing phase are added to obtain the gait cycle; at the same time, according to the diagonal gait type, the switching rules between the stance phase and the swing phase of different legs are determined; A3: Set the movement speed or acceleration state threshold h1, and calculate the switching timing between the stance phase and the swing phase of different legs based on the gait cycle and switching rules and the movement rules of different legs; If v sp >h1, or v sw <h1, then keep moving; If v sp ≤h1, or v sw ≥h1, the switch from the support phase to the swing phase is triggered, where v sp is the velocity or acceleration of the leg at the end of the stance phase, v sw represents the velocity or acceleration of the leg at the beginning of the swing phase; A4: Based on the preset target position and orientation of the robot end effector, use the Newton-Raphson algorithm to calculate the angles θ of different joints i , based on the obtained joint angles and the switching timing obtained in A3, calculate the speed u of different joints i =θ i ′, and integrate the calculated different joint angles, speeds and switching timings into gait information, where θ i represents the angle of the i-th joint, and i∈[1,n], u i represents the velocity of the i-th joint, θ i ′ represents θ i The first-order derivative of with respect to time, n represents the number of joints; A5: Perform stability analysis on the generated gait information, and adjust and optimize gait parameters and joint control based on the analysis results; 2. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 1, characterized in that: The steps for calculating the angles of different joints in A4 include: A4.1: Collect the robot's mechanical structure parameters and set the target position and direction of the robot's end effector; A4.2: Take the target position and orientation as input, run the Newton-Raphson algorithm, and output different joint angles that meet the target position and orientation requirements; A4.3: Check whether the calculated joint angles meet the physical limitations of the robot; If not satisfied, readjust the target position and direction; If satisfied, the calculated joint angle is output.

3. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 2, characterized in that: The specific steps of A4.2 include: A4.21: Receive the target position and direction of the robot end effector and convert the target position and direction into vector format to obtain the target position vector x vector and the target direction vector θ vector ; A4.22: Read the current joint angle from the sensor based on Calculate the transformation matrix M of this joint relative to the previous joint through translation transformation bian , the total transformation matrix M from the base to the end effector is obtained by accumulation total =M bian,1 ×…×M bian,n , and extract the actual position v of the end effector from the total transformation matrix ad and direction θ ad , where M bian,n Represents the transformation matrix of the nth joint.

4. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 3, characterized in that: The specific steps of A4.2 also include: A4.23: Use sensors to obtain the actual position of the current joint angle and direction Calculate the error vector between the target position and orientation and the actual position and orientation using the formula: Among them, e vector represents the error vector between the target position and direction and the actual position and direction, w1 represents the position weight, w2 represents the direction weight, and |·| represents the absolute value; A4.24: Use the Jacobian matrix to solve the first-order derivative of the error vector with respect to the current joint angle And according to the first-order derivative and the error vector, calculate the increment of the joint angle in, The Jacobian matrix represents the relationship between the velocity of the robot end effector and the joint velocity, and λ represents the learning rate; A4.25: Increase Add to current joint angle Iterate and update until the preset number of iterations is reached, and then output the final joint angle.

5. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 4, characterized in that: The monitoring unit adopts a three-dimensional obstacle avoidance strategy, and the specific steps of the three-dimensional obstacle avoidance strategy include: B1: Use binocular cameras to capture the 3D information of the surrounding environment and the distance of obstacles in real time, and perform preprocessing to obtain environmental perception image information; B2: Perform stereo matching on the environment perception images taken by the left and right cameras of the binocular camera, calculate the three-dimensional coordinates and depth information of the obstacle by calculating the disparity map and combining the internal and external parameters of the camera; B3: Use the machine learning convolutional neural network algorithm to identify and classify the three-dimensional information of obstacles calculated in step B2; B4: Based on the identified and classified obstacle information, as well as the robot’s current position and posture, a path planning algorithm is used to plan multiple feasible obstacle avoidance paths.

6. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 5, characterized in that: The specific steps of B2 include: B2.1: Obtaining pre-processed environment perception image information, where the environment perception image information is taken by both left and right cameras of a binocular camera; B2.2: Based on the environment perception image, for each pixel in the left image, find its corresponding point in the right image, and calculate the matching cost of each pixel in the left and right images. Store the matching costs of all pixels in a three-dimensional matrix to obtain the disparity three-dimensional matrix M DSI , the formula is: Q(q1,q2,q3)=|g L (q1,q2)-g R (q1-q3,q2)|; Where q1 represents the width of the environment perception image, q2 represents the height of the environment perception image, q3 represents the disparity search range, Q(q1,q2,q3) represents the matching cost when the disparity is q3 at the pixel position (q1,q2) in the environment perception image, and g L (q1, q2) represents the grayscale value of the left image of the binocular camera at the pixel position (q1, q2) in the environmental perception image, g R (q1-q3,q2) represents the grayscale value of the right image of the binocular camera at the pixel position (q1-q3,q2) in the environmental perception image; B2.3: Aggregate the matching costs to obtain a new disparity 3D matrix The cost value of each pixel reflects its correlation with the candidate pixels.

7. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 6, characterized in that: The specific steps of B2 also include: B2.4: Create a disparity map of the same size as the left image to store the optimal disparity value for each pixel; B2.5: For the new disparity 3D matrix For each pixel in , traverse all possible disparity values ​​of each pixel, and find the cost value of the pixel under each disparity value, and select the disparity value with the smallest cost value as the optimal disparity of the pixel. The formula is: Among them, q 3,k represents the optimal disparity value of pixel k, Q(k,q3) represents the cost value of pixel k when the disparity is q3, and argmin(·) represents the optimal disparity value of pixel k in the domain The value of the independent variable corresponding to the minimum value of inner Q(k,q3); B2.6: Assign the optimal disparity value of each pixel to the corresponding position in the disparity map, and obtain a disparity map containing the optimal disparity value of each pixel; B2.7: Calculate the depth value corresponding to each pixel based on the disparity map, the focal length of the camera, and the distance between the two eyes The depth value is mapped to the three-dimensional space to obtain the three-dimensional coordinates and depth information of the obstacle, where d guang Represents the distance between the optical centers of the two cameras of the binocular camera, f gc represents the focal length from the optical center of the camera to the imaging plane, and q4 represents the parallax between the two cameras.

8. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 7, characterized in that: The optimization strategy generation unit adopts the spider bee optimization strategy, and the specific steps of the spider bee optimization strategy include: D1: Set the search space and randomly distribute a certain number of search agents in the search space, where each search agent represents a potential solution; D2: Each search agent moves in the search space according to gradient descent to find the optimal solution, and evaluates the solution at the current position of each search agent to calculate its fitness value; D3: If the search agent finds a potential high-quality solution, it will follow the solution. If the quality of the solution decreases or encounters an obstacle, the search agent will choose to escape, that is, move away from the current solution and try a new search direction; D4: Once a satisfactory solution is found, the search agent builds a nest around it, i.e., performs local optimization; D5: Combine the solutions of different search agents through crossover operation to generate new offspring, and repeat iteratively until the preset maximum number of iterations is reached.

9. The quadruped robot autonomous decision-making control system based on multimodal perception as claimed in claim 8, characterized in that: The switching rule in A2 includes that when the left front leg and the right hind leg are supporting, the right front leg and the left hind leg start to swing; when the right front leg and the left hind leg are supporting on the ground, the left front leg and the right hind leg start to swing.

Images