Foot type robot terrain perception modeling method fusing vision and touch

Abstract

The invention relates to a foot type robot terrain perception modeling method fusing vision and touch, and belongs to the technical field of walking control of foot type cross-country robots. According to the method, comprehensive perception of a geometric structure, mechanical properties and material characteristics of a terrain is realized by integrating various sensor data, and a local terrain model is established on the basis, so that passable characteristics of an unknown road surface in front are predicted, and finally autonomous selection of a safe area in a complex environment is helped to be realized by the foot-type robot.

Description

Technical Field

[0001] The present invention belongs to the technical field of walking control of legged off-road robots, and particularly relates to a terrain perception modeling method for legged robots that integrates vision and touch. Background Technique

[0002] With the development of robot technology, legged off-road robots have been widely used in fields such as wilderness search and rescue and geological exploration due to their excellent terrain adaptability. However, when facing complex and changeable natural environments, how to accurately grasp the current and future possible terrain conditions has become a key factor in determining whether the robot can successfully complete tasks. Currently, most robots mainly rely on single vision or touch sensors for terrain perception. This method is difficult to comprehensively and accurately reflect the actual terrain characteristics. Especially when facing soft, loose, or severely occluded ground, it is prone to misjudgment and even out-of-control accidents.

[0003] Traditional vision modeling or touch modeling only includes the geometric elements of the terrain (a method and system for measuring terrain parameters of legged robots, CN201710391624.4). Currently, there is no modeling method that includes the physical properties of the ground (such as bearing capacity, slip rate, etc.). Therefore, it is difficult to accurately predict the unknown road conditions ahead based on known terrain features, which limits the autonomous movement ability of the robot in unknown environments. Summary of the Invention

[0004] (I) Technical Problems to be Solved

[0005] The technical problem to be solved by the present invention is to design a terrain perception modeling method for legged robots to predict the passable features of the unknown road surface ahead, so as to help legged robots autonomously select safe areas in complex environments.

[0006] (II) Technical Solutions

[0007] To solve the above technical problems, the present invention provides a terrain perception modeling method for legged robots that integrates vision and touch, including the following steps:

[0008] Step 1, Visual Terrain Perception: Real-time capture the terrain color RGB and the geometric information of each plot that makes up the terrain through the optical sensor installed on the robot, rasterize the perception range, and assign visual semantic information of the plot to each grid; a terrain is composed of one or more plots, and the plot is a two-dimensional plane;

[0009] Assign the visual semantic information of the plot as (x i , y j , R ij , G ij , B ij ), where xi , y j represent the horizontal and vertical coordinates of the plot, R ij , G ij , B ij represent the RGB information of the terrain color. The subscripts i and j are used to distinguish the information on two dimensions of the two-dimensional plane of the plot. One plot corresponds to one foothold;

[0010] Step 2. Tactile terrain perception: Based on the inertial measurement unit (IMU) installed on the robot, solve the body pose information. Through the pressure sensor installed at the end of the robot's foot, record the pressure change process generated during the contact between the end of the robot's foot and the ground, and express it as the tactile semantic information of the plot, specifically including:

[0011] (1) Calculate the position of the foot end in combination with the body pose: Obtain the absolute position of the body centroid from the IMU of the body centroid. According to the body size data, calculate the relative position between the foot end and the centroid to obtain the absolute position of the foot end;

[0012] (2) Record the pressure change process generated during the contact between the end of the robot's foot and the ground: The start of the contact process is from when the foot end starts to be stressed, and the end of the contact process is when the absolute position of the foot end remains unchanged. The change amount in the vertical direction of the absolute position of the foot end during this process is the subsidence amount c of the foot end ij , and the change amount in the forward direction of the absolute position of the foot end is the slip amount s ij , and the horizontal component force during slip is the maximum friction force f ij ; among them, the start and end moments of the contact process are determined according to the output data of the pressure sensor;

[0013] Assign the tactile semantic information of the plot as (x i , y j , c ij , s ij , f ij );

[0014] Step 3. Construction of mapping relationship: First, construct a system of equations for visual-tactile mapping relationship based on the following formula:

[0015]

[0016] Among them, is the coefficient matrix to be solved;

[0017] Secondly, establish a grid of the terrain within a certain range based on the visual semantic information of the plot obtained by using the method in Step 1. Then, use the method in Step 2 to collect the tactile semantic information of the plot at the foothold once every step, and continuously obtain three footholds (n0, n 0+1 , n 0+2After obtaining the tactile semantic information of (), three sets of visual-tactile mapping relationship equations are established, and the unique coefficient matrix is solved.

[0018] Step 4: Terrain mechanics condition prediction: First, according to the solved unique coefficient matrix and the equations, the ground tactile information parameters (c ij , G ij , B ij ) of the next foothold are calculated from the terrain color RGB information (R ij , s ij , f ij ) obtained from the visual terrain perception in Step 1 as predicted values.

[0019] Then iterative adjustment is carried out: After the robot walks to the plot corresponding to the predicted value, an actual value (c' ij , s' ij , f' ij ) is obtained based on the method in Step 2. If the difference between the predicted value and the actual value is greater than the preset threshold, the average of the predicted value and the actual value is used as the data n 0+3 of the fourth point, the first valid point n0 is removed, and three sets of equations are re-solved to obtain a new unique solution. This process is repeated until the difference between the predicted value and the actual value is less than or equal to the preset threshold, and the iteration ends. Finally, the adjusted visual-tactile mapping relationship equation of the coefficient matrix is used to predict the passable characteristics of the unknown road surface ahead through this visual-tactile mapping relationship equation set to obtain the ground tactile information parameters, so as to realize the autonomous selection of the safe area by the robot.

[0020] The present invention also provides a system for implementing the above method.

[0021] The present invention also provides a legged robot autonomous walking control method implemented by using the above method.

[0022] The present invention also provides a legged robot autonomous walking control system implemented by using the above method.

[0023] (III) Beneficial effects

[0024] The present invention aims to provide a method for terrain perception modeling of a legged robot that integrates vision and touch. By integrating various sensor data, this method enables comprehensive perception of the terrain's geometric structure, mechanical properties, and material characteristics. Based on this, a local terrain model is established, and then the passable characteristics (ground touch information parameters) of the unknown road surface ahead are predicted. Ultimately, it helps the legged robot autonomously select safe areas in complex environments. By comprehensively using various sensor data, this method achieves accurate all-round perception and prediction of the terrain, providing reliable guarantee for the autonomous movement of the legged robot in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] To make the objectives, contents, and advantages of the present invention clearer, the following further describes the specific embodiments of the present invention in detail with reference to the drawings and embodiments.

[0027] Refer to Figure 1 , a method for terrain perception modeling that integrates vision and touch provided by the present invention includes four steps: visual terrain perception, tactile terrain perception, mapping relationship construction, and prediction adjustment, which are specifically described as follows:

[0028] Step 1: Visual terrain perception: The optical sensor system installed on the robot is used to capture the RGB color of the terrain and the geometric information of each plot that makes up the terrain in real time, and the perception range is rasterized. Visual semantic information of the plot is assigned to each grid; a terrain consists of one or more plots, and a plot is a two-dimensional plane.

[0029] The visual semantic information of the plot is assigned as (x i , y j , h ij , R ij , G ij , B ij ), where x i , y j represent the horizontal and vertical coordinates of the plot (foot landing point), h ij represents the elevation of the plot, R ij , G ij , B ij represent the RGB color information of the terrain. Subscripts i and j are used to distinguish the information on the two dimensions of the two-dimensional plane of the plot. x i represents the distance of the i-th plot on the horizontal coordinate, y j represents the distance of the j-th plot on the vertical coordinate. One plot corresponds to one foot landing point.

[0030] Step 2. Tactile terrain perception: Calculate the body pose (position and attitude) of the robot through the sensor IMU (Inertial Measurement Unit) installed on the robot, and record the pressure change process generated during the contact between the robot's foot end (foot) and the ground through the pressure sensor installed at the foot end of the robot, and express it as the tactile semantic information of the plot;

[0031] (1) Calculate the foot end position in combination with the body attitude: Obtain the absolute position of the body center of mass according to the IMU of the body center of mass, and calculate the relative position between the foot end and the center of mass based on the body size data (including the installation position of the joint motor and the length of the leg (linkage)) to obtain the absolute position of the foot end;

[0032] (2) Record the pressure change process generated during the contact between the robot's foot end and the ground: The start of the contact process is from when the foot end is stressed, and the end of the contact process is when the absolute position of the foot end remains unchanged. The change amount of the absolute position of the foot end in the vertical direction during this process is the foot end settlement amount c ij , and the change amount of the absolute position of the foot end in the forward direction is the slip amount s ij , and the horizontal component force when slipping occurs is the maximum friction force f ij , and the start and end times of the contact process are determined according to the output data of the pressure sensor;

[0033] Assign the tactile semantic information of the plot as (x i , y j , c ij , s ij , f ij ), where the height difference is the foot end settlement amount c ij , the distance change amount in the forward direction is the slip amount s ij , and the horizontal component force when slipping occurs is the maximum friction force f ij .

[0034] Step 3. Construction of mapping relationship: First, based on the body space geometric relationship and the terrain information grid established based on vision, during the robot's walking, for each landing point (x i , y j ), the (R ij , G ij , B ij ) obtained by visual information sampling and the (c ij , s ij , f ij ) obtained by tactile information sampling are used to construct the visual-tactile mapping relationship equation based on the following formula:

[0035]

[0036] Among them, is the coefficient matrix to be solved;

[0037] Secondly, based on the plot visual semantic information obtained by using the method in Step 1, establish a grid of plot visual semantic information for a certain large-scale terrain. Then, using the method in Step 2, collect the plot tactile semantic information of the landing point once in each step, and continuously obtain the tactile semantic information of three landing points (n0, n 0+1 , n 0+2 ). After that, establish three groups of visual-tactile mapping relationship equations and solve for the unique coefficient matrix The three landing points that can solve for the unique coefficient matrix are valid landing points;

[0038] If the landing point where the tactile semantic information is collected has no corresponding visual semantic information or cannot solve for the unique coefficient matrix, the robot continues to walk until a landing point with corresponding visual semantic information is collected;

[0039] If the three groups of visual-tactile mapping relationship equations corresponding to the landing point (n0, n 0+1 , n 0+2 ) have no unique solution, that is, there are infinitely many solutions (the equations are repeated) or no solution (the equations are contradictory), then delete the equations of the third landing point n 0+2 , and use the next landing point n 0+3 until the unique coefficient matrix is solved.

[0040] Step 4: Terrain mechanics condition prediction: First, according to the solved unique coefficient matrix and the equations, calculate the ground tactile information parameters (c ij , s ij , f ij ) of the next landing point from the terrain color RGB information (R ij , G ij , B ij ) obtained by the visual terrain perception (visual information sampling) in Step 1 as the predicted values. The predicted ground tactile information parameters (tactile attributes) are used for pre-motion feedforward control.

[0041] Then perform iterative adjustment: When the robot walks onto the plot corresponding to the predicted value, based on the method in Step 2, collect an actual value (c' ij , s' ij , f' ij ) through the sensor. If the difference between the predicted value and the actual value is greater than the preset threshold, then take the average of the predicted value and the actual value as the data of the fourth point n 0+3, remove the first valid point n0, and re-solve the three sets of equations to obtain a new unique solution (coefficient matrix). Repeat this process until the difference between the predicted value and the actual value is less than or equal to the preset threshold, at which point the iteration ends. Finally, obtain the adjusted visual-tactile mapping relationship equation of the coefficient matrix for predicting the passable features (ground tactile information parameters) of the unknown road surface ahead through this visual-tactile mapping relationship equation, so as to realize the autonomous selection of safe areas by the robot in complex environments and provide guarantee for the autonomous movement of the legged robot in complex environments. The present invention effectively increases the prediction accuracy by iteratively eliminating errors.

[0042] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the technical principle of the present invention, several improvements and deformations can still be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A terrain perception modeling method for a legged robot that integrates vision and touch, characterized in that Including the following steps: Step 1, visual terrain perception: The RGB color of the terrain and the geometric information of each plot that makes up the terrain are captured in real time through the optical sensors installed on the robot, and the perception range is rasterized. Visual semantic information of the plot is assigned to each grid. A terrain consists of one or more plots, and the plot is a two-dimensional plane. Assign the visual semantic information of the plot as (x i , y i , R ij , G ij , B ij ), where x i , y j represent the horizontal and vertical coordinates of the plot, and R ij , G ij , B ij represent the RGB information of the terrain color. The subscripts i and j are used to distinguish the information on the two dimensions of the two-dimensional plane of the plot, and one plot corresponds to one foothold; Step 2, tactile terrain perception: Based on the inertial measurement unit (IMU) installed on the robot, the body pose information is solved. Through the pressure sensors installed at the feet of the robot, the pressure change process generated during the contact between the feet of the robot and the ground is recorded and expressed as the tactile semantic information of the plot, specifically including: (1) Calculate the foot position in combination with the body pose: The absolute position of the body centroid is obtained from the IMU of the body centroid. According to the body size data, the relative position between the foot and the centroid is calculated to obtain the absolute position of the foot. (2) Record the pressure change process generated during the contact between the robot's foot end and the ground: The start of the contact process is when the foot end starts to be stressed, and the end of the contact process is when the absolute position of the foot end remains unchanged. During this process, the change amount of the absolute position of the foot end in the vertical direction is the foot end subsidence amount c ij , and the change amount of the absolute position of the foot end in the forward direction is the slip amount s ij , and the horizontal component force during slip is the maximum frictional force f ij ; among them, the start and end times of the contact process are determined according to the output data of the pressure sensor; Assign the tactile semantic information of the plot as (x i , y j , c ij , s ij , f ij ); Step 3, construction of the mapping relationship: First, a system of equations for the visual-tactile mapping relationship is constructed based on the following formula: Among them, is the coefficient matrix to be solved; Secondly, a grid of the terrain within a certain range is established based on the visual semantic information of the plot obtained by using the method in Step 1. Then, by using the method in Step 2, the tactile semantic information of the plot at the landing point is collected once in each step. After continuously obtaining the tactile semantic information of three landing points (n0, n 0+1 , n 0+2 ), three groups of visual-tactile mapping relationship equations are established, and the unique coefficient matrix is solved Step 4: Terrain mechanics condition prediction: First, based on the uniquely solved coefficient matrix and the equations, the ground tactile information parameters (c ij , G ij , B ij ) of the next foothold are calculated from the terrain color RGB information (R ij , s ij , f ij ) obtained from the visual terrain perception in Step 1 as the predicted values; Then perform iterative adjustment: After the robot reaches the plot corresponding to the predicted value, an actual value (c' ij , s' ij , f' ij ) is obtained based on the method in Step 2. If the difference between the predicted value and the actual value is greater than the preset threshold, the average of the predicted value and the actual value is calculated as the data n of the fourth point 0+3 , the first valid point n0 is removed, and the three groups of equations are re-solved to obtain a new unique solution. Repeat this process until the difference between the predicted value and the actual value is less than or equal to the preset threshold to end the iteration. Finally, the adjusted visual-tactile mapping relationship equation of the coefficient matrix is obtained, which is used to predict the passable feature of the unknown road surface ahead through the visual-tactile mapping relationship equations to obtain the ground tactile information parameters, so as to realize the autonomous selection of the safe area by the robot.

2. The method according to claim 1, characterized in that The body size data includes the installation positions of the joint motors of the robot and the lengths of the legs.

3. The method according to claim 1, characterized in that, Step 1 and Step 2: According to the geometric relationship of the fuselage space and the terrain information grid established based on vision, during the walking process of the robot, visual information sampling is performed on each landing point (x i , y j ) to obtain (R ij , G ij , B ij ), and tactile information sampling is performed to obtain (c ij , s ij , f ij ).

4. The method according to claim 1, wherein In Step 3, if there is no corresponding visual semantic information for the footfall point where the tactile semantic information is collected, the robot continues to walk until a footfall point with corresponding visual semantic information is collected. If the three sets of visual-tactile mapping relationship equations corresponding to the foot landing points (n0, n 0+1 , n 0+2 ) do not have a unique solution, that is, there are infinitely many solutions or no solution, then delete the third foot landing point n 0+2 corresponding visual-tactile mapping relationship equations, and use the next foot landing point n 0+3 corresponding visual-tactile mapping relationship equations until a unique coefficient matrix is solved.

5. The method according to claim 1, characterized in that, Three footfall points that can solve for a unique coefficient matrix are valid footfall points.

6. The method according to claim 1, wherein In Step 4, the predicted ground tactile information parameters are used for pre-motion feedforward control.

7. The method according to claim 1, characterized in that The visual semantic information of the plot also includes the elevation information of the plot.

8. A system for implementing the method according to any one of claims 1 to 7.

9. A method for autonomous walking control of a legged robot implemented by using the method according to any one of claims 1 to 7.

10. A system for autonomous walking control of a legged robot implemented by using the method according to any one of claims 1 to 7.

Images