Terrain recognition system

Abstract

The invention discloses a terrain recognition system, and relates to the technical field of robot environment perception, and the system comprises an external sensibility sensor module, an internal sensibility sensor module and an additional module. The external sensitivity sensor module at least comprises two different types of sensors such as an RGB camera and a radar, and the internal sensitivity module at least comprises two different types of sensors such as an IMU and a force sensor. The additional module comprises a data transmission unit, a data storage unit, a data processing unit and a terrain recognition unit. According to the terrain recognition system, the time sequence condition of internal and external sensibility data is fully considered, firstly, the external sensibility data is used for pre-judging the terrain which is about to pass through, and when the terrain is pre-judged, the terrain is recognized again in combination with the internal and external sensibility data. The advantage complementation of multi-sensor data is effectively realized, and meanwhile, the support can be provided for the follow-up control strategy or gait adjustment of the quadruped robot.

Description

Technical Field

[0001] The present invention relates to the field of robot environment perception technology, and in particular to a terrain recognition system suitable for use by a quadruped robot and combined with time-series internal and external perception data fusion. Background Art

[0002] Unlike the structured environment indoors, the driving environment of quadruped robots in outdoor scenes is more complex and prone to "non-geometric damage", such as sinking, slipping and other dangerous situations when crossing mudflats, sand and snow with poor ground bearing capacity. Therefore, the robot must have accurate terrain perception and recognition capabilities to make appropriate motion control and gait planning based on this.

[0003] At present, there are two main methods for terrain recognition: interoceptive and exteroceptive methods. Among them, exteroceptive methods (for example, visual terrain classification) are usually sensitive to changes in lighting and can be easily deceived by terrain cover (for example, fallen leaves). The main sensors are cameras, light detection and ranging (LiDAR), and laser detection and ranging (LADAR). Interoceptive methods, including acoustics, touch, and vibration, mainly use the physical reactions produced by the interaction between terrain and motion mechanisms. The biggest problem is that the method of obtaining physical properties through contact between the robot and the terrain lags behind the task requirements. The main sensors are inertial sensors, microphones, force / torque sensors, tactile sensors, ground penetrating radar (GPR), etc.

[0004] The current research focuses on terrain recognition based on pure exteroceptive methods or pure interoceptive methods, which cannot overcome the inherent defects of the two types of perception methods. The exteroceptive method is still easily disturbed by light and occlusion. The time for the interoceptive method to identify the terrain type lags significantly behind the demand. For example, after a robot falls into a swamp and then recognizes that the current terrain is a swamp, it cannot avoid risks.

[0005] In addition, some studies have combined internal and external perception methods, such as "A terrain recognition method for outdoor mobile robots based on visual and tactile fusion (application number: 201911088893.9)", "A terrain classification method based on audio-visual information fusion (application number: 202210204930.3)", "A terrain recognition system based on vehicle vision and intelligent tire tactile perception (application number: 202410647203.3)", etc., without considering the temporal relationship between internal and external perception data, and inputting the internal and external perception data at the same time into the classification algorithm for terrain recognition, which does not conform to the actual situation of real perception. Summary of the invention

[0006] In view of the above problems, the present invention provides a terrain recognition system for overcoming the above problems or at least partially solving the above problems. The system can be mounted on a quadruped robot and perform pre-recognition of external perception and re-recognition of internal and external perception fusion based on time sequence. The problem that traditional terrain recognition does not consider the data time sequence when combining internal and external perception methods is solved.

[0007] The present invention provides the following scheme:

[0008] A terrain recognition system includes an exteroceptive sensor module, an interoceptive sensor module and an additional module.

[0009] The external perception module includes at least two different types of sensors, such as an RGB camera and a radar. The RGB camera collects image information of the ground in front, and the laser radar collects radar point cloud data of the ground in front.

[0010] The internal perception module includes at least two different types of sensors, such as an IMU and a force sensor; the IMU inertial measurement unit collects vibration information of the current ground, and the force sensor collects pressure information of the current ground;

[0011] The additional module includes a data transmission unit, a data storage unit, a data processing unit and a terrain recognition unit. The interaction between the units is as follows:

[0012] The data obtained by the exteroceptive sensor and the interoceptive sensor are transmitted to the data processing unit through the data transmission unit.

[0013] In the data processing unit, data preprocessing and feature value extraction are completed.

[0014] The extracted data features are then transferred to the data storage unit.

[0015] The terrain recognition unit will obtain relevant features of the exteroceptive data and the interoceptive data from the data storage unit in a time sequence, and simultaneously run the exteroceptive terrain recognition algorithm and the internal and external perception fusion terrain recognition algorithm to achieve pre-recognition and re-recognition of the terrain.

[0016] That is, at the current moment, the ground in front of the robot will be pre-identified based on the exteroceptive data received at the current moment, and at the same time, the ground where the robot is currently located will be re-identified based on the fusion of the exteroceptive data received at the previous moment and the internal sensory data received at the current moment.

[0017] The present invention aims to propose a system framework for the fusion of internal and external perception combined with time series. In the specific implementation process, other types of sensors can be used, other types of features can be extracted from sensor data, and the terrain classification model can also be replaced by other machine learning models.

[0018] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0019] The embodiment of the present application provides a terrain recognition system, which uses exteroceptive data to predict the terrain to be passed, and can adjust the control strategy in advance, ensuring the forward-looking nature of the adjustment and reducing potential risks. When passing through the predicted terrain, combined with the internal perception method, the terrain can be recognized again, and possible errors in the prediction can be corrected to adjust the control strategy in real time, further reducing the risk of passing through unknown terrain. Combining the method of terrain recognition with internal and external perception fusion can achieve more accurate terrain recognition.

[0020] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of a terrain recognition process at time k of a terrain recognition system provided by an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of the pre-identification and re-identification process of ground b provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0025] See also Figure 2 , is a terrain recognition system provided by an embodiment of the present invention, such as Figure 2 As shown, the system may include:

[0026] An exteroceptive sensor module, an interoceptive sensor module and an additional module mounted on a quadruped robot; the exteroceptive sensor module includes at least two different types of exteroceptive data acquisition sensors, and the interoceptive sensor module includes at least two different types of interoceptive data acquisition sensors;

[0027] The additional modules are used to perform the following operations:

[0028] Acquire multi-source heterogeneous exteroceptive data of a target ground area collected by the exteroceptive sensor module at a previous moment; the target ground area is a ground area passed by the quadruped robot at a current moment;

[0029] Extracting features from the multi-source heterogeneous exteroceptive data to obtain multi-source heterogeneous exteroceptive high-dimensional features;

[0030] Using a multi-layer perceptron in combination with the multi-source heterogeneous exteroceptive high-dimensional features to pre-identify the terrain of the target ground area;

[0031] Acquire multi-source heterogeneous interoceptive data of the target ground area currently collected by the interoceptive sensor module;

[0032] Performing feature extraction on the multi-source heterogeneous interoceptive data to obtain multi-source heterogeneous interoceptive features;

[0033] The terrain of the target ground area is re-identified by using a support vector machine in combination with the multi-source heterogeneous exteroceptive high-dimensional features and the multi-source heterogeneous interoceptive features.

[0034] The present invention proposes a terrain recognition system that combines time-series internal and external perception fusion for quadruped robots to identify unknown terrain. That is, before the quadruped robot enters the unknown terrain, it predicts the terrain to be crossed through external perception, adjusts the robot's own state in advance, enters the predicted terrain, and then uses internal perception to further judge the current terrain to correct possible errors in the previous prediction. The perception method provided by the system is consistent with the human perception of unknown terrain, and at the same time fully considers the complementary characteristics of different sensors, which helps to estimate the ground type more accurately and efficiently.

[0035] In order to improve the robustness and accuracy of terrain recognition, the exteroceptive sensor module and the interoceptive sensor module provided in the embodiments of the present invention respectively use a variety of sensors, and provide different feature extraction methods for data collected by different sensors.

[0036] In a specific implementation, the embodiment of the present invention may provide that the exteroceptive sensor module includes at least an RGB camera and a laser radar, and the multi-source heterogeneous exteroceptive data includes at least RGB image information and radar point cloud data.

[0037] Further, extracting features from the multi-source heterogeneous exteroceptive data to obtain multi-source heterogeneous exteroceptive high-dimensional features includes:

[0038] Extract the RGB image information using ResNet to obtain high-dimensional features of the image;

[0039] Optical histogram features are created using the radar point cloud data, and high-dimensional features of the radar are obtained by extracting features from the optical histogram features using the multi-layer perceptron, so as to obtain the multi-source heterogeneous external perception high-dimensional features after combining the high-dimensional features of the image with the high-dimensional features of the radar.

[0040] Furthermore, the step of creating an optical histogram feature using the radar point cloud data includes:

[0041] Filtering out lidar points below an intensity threshold in the radar point cloud data, and projecting and clipping the remaining points onto corresponding images;

[0042] By calculating the numerical distribution of the three channels of ambient light, reflectivity and intensity, average frequency histograms of the three channels of different ground types are drawn to create the optical histogram feature.

[0043] The interoceptive sensor module includes an IMU inertial measurement unit and a force sensor, and the multi-source heterogeneous interoceptive data includes vibration information and pressure information of the ground. Vibration information refers to the linear acceleration perpendicular to the ground measured by the IMU. What is directly measured is time domain information, which can be directly analyzed by power spectrum after conversion to frequency domain. Pressure information refers to the signal collected by the force sensor, which is directly measured in time domain information, and time domain feature extraction can be performed directly.

[0044] The extracting features of the multi-source heterogeneous interoceptive data to obtain multi-source heterogeneous interoceptive features comprises:

[0045] Performing power spectrum analysis and extraction on the vibration information to obtain the minimum translation acceleration, the maximum translation acceleration, the average translation acceleration, the translation acceleration variance, the root mean square of the translation acceleration, and the translation acceleration norm;

[0046] The pressure information is subjected to time domain feature extraction to obtain sample expectation, sample standard deviation, sample coefficient of variation and peak value.

[0047] Furthermore, the minimum translational acceleration is extracted by the following formula:

[0048]

[0049] The maximum translational acceleration is extracted by the following formula:

[0050]

[0051] The average translational acceleration is extracted by the following formula:

[0052]

[0053] The translational acceleration variance is extracted by the following formula:

[0054]

[0055] The root mean square of the translational acceleration is extracted by the following formula:

[0056]

[0057] The translational acceleration norm is extracted by the following formula:

[0058]

[0059] Where: The frequency domain data representing the vibration information, N represents the number of data.

[0060] The sample expects:

[0061]

[0062] Sample standard deviation:

[0063]

[0064] Sample coefficient of variation:

[0065]

[0066] crest:

[0067] b 4 =max(x(t))

[0068] Where: x(t) represents the data collected by the force sensor.

[0069] The additional module provided by the embodiment of the present invention may include a data transmission unit, a data storage unit, a data processing unit and a terrain recognition unit.

[0070] The terrain recognition system provided by the present invention is introduced in detail below.

[0071] The present invention mainly provides a basis for the stable walking of the quadruped robot by accurately identifying the terrain. Taking into account the time sequence of the internal and external perception data, the present invention first uses the external perception data to predict the terrain to be passed, and then when passing through the predicted terrain, the terrain is recognized again in combination with the internal and external perception method, which effectively realizes the complementary advantages of multi-sensor data and can provide support for the subsequent adjustment of the control strategy or gait.

[0072] The system provided by the embodiment of the present invention adopts a method combining internal and external perception for the stable walking of a quadruped robot. It can not only predict the terrain type to adjust the gait or other control strategies in advance, but also further judge the predicted terrain based on proprioception, effectively reducing the loss caused by prediction errors.

[0073] For complex unstructured terrain in the wild, robots are often unable to rely on one sensor to accurately complete the task of terrain recognition. The present invention uses more than one sensor in both the external perception part and the internal perception part, realizing the complementary advantages of multiple sensors, which can improve the robustness and accuracy of terrain recognition.

[0074] A terrain recognition method combining internal and external perception fusion with time series is proposed. Taking into account the actual situation of perception control, it not only avoids blindly entering unknown terrain and relying on the robot's adaptive adjustment, but also avoids relying solely on visual prediction of the terrain. After entering the terrain, the robot no longer adjusts based on the current perception situation. This makes the robot's control strategy or gait adjustment more flexible.

[0075] The present invention provides a terrain recognition system combining time-series internal and external perception fusion, comprising a sensor and an additional module.

[0076] The sensor includes an external perception sensor module and an internal perception sensor module. The external perception sensor module uses an RGB camera and a laser radar, and the internal perception sensor module uses an IMU inertial measurement unit and a force sensor.

[0077] The additional module includes a data transmission unit, a data storage unit, a data processing unit and a terrain recognition unit, and the units interact as follows:

[0078] The RGB camera collects image information of the ground ahead, the LiDAR collects radar point cloud data of the ground ahead, and the IMU inertial measurement unit and force sensor collect the robot's response to the current ground. The image information and radar point cloud data containing the features of the ground ahead, as well as the vibration and pressure information of the robot on the current ground, are transmitted to the data processing unit through the data transmission unit.

[0079] In the data processing unit, data preprocessing and feature value extraction are completed.

[0080] (1) Radar data preprocessing and feature value extraction.

[0081] Lidar points below an intensity threshold in the LiDAR data are first filtered out, and the remaining points are then projected and cropped onto the corresponding image (matching is done based on the closest timestamp).

[0082] Since the return wave of the laser radar carries material information when it is projected onto the surface, the present invention selects three available channels (ambient light, reflectivity and intensity) to distinguish different terrain types. The optical histogram feature is created by calculating the numerical distribution of the three channels and plotting the average frequency histogram of the three channels for different ground types.

[0083] (2) IMU acceleration data preprocessing and feature extraction

[0084] The IMU inertial measurement unit is used to measure the linear acceleration (z-axis) perpendicular to the ground plane, and high-frequency sampling and filtering are performed on it to reduce noise. Then the original time domain data is converted into the frequency domain, and power spectrum analysis is performed to extract the following six features (minimum translation acceleration, maximum translation acceleration, average translation acceleration, translation acceleration variance, root mean square of translation acceleration, and translation acceleration norm).

[0085] The minimum translational acceleration is extracted by the following formula:

[0086]

[0087] The maximum translational acceleration is extracted by the following formula:

[0088]

[0089] The average translational acceleration is extracted by the following formula:

[0090]

[0091] The translational acceleration variance is extracted by the following formula:

[0092]

[0093] The root mean square of the translational acceleration is extracted by the following formula:

[0094]

[0095] The translational acceleration norm is extracted by the following formula:

[0096]

[0097] Where: The frequency domain data representing the vibration information, N represents the number of data.

[0098] (3) Pressure sensing signal preprocessing and feature extraction

[0099] The signal collected by the pressure sensor is sampled at high frequency and filtered for noise reduction, and the corresponding time domain features are extracted. The following four features (sample expectation, sample standard deviation, sample coefficient of variation, and peak) are extracted.

[0100] The sample expects:

[0101]

[0102] Sample standard deviation:

[0103]

[0104] Sample coefficient of variation:

[0105]

[0106] crest:

[0107] b 4 =max(x(t))

[0108] Where: x(t) represents the data collected by the force sensor.

[0109] The extracted data features are then transferred to the data storage unit.

[0110] The terrain recognition unit obtains relevant features of the external perception data and the internal perception data from the data storage unit in time sequence, and simultaneously runs the external perception terrain recognition algorithm and the internal and external perception fusion terrain recognition algorithm to realize the pre-recognition and re-recognition of the terrain.

[0111] Specific circumstances such as Figure 1 As shown, at time k (current time), the ground c will be pre-identified based on the exteroceptive data received at time k, and the ground b will be re-identified based on the fusion of the exteroceptive data received at time ki (previous time) and the interoceptive data received at time k.

[0112] For one of the ground pre-recognition and re-recognition Figure 2 As shown, Figure 2 Take the recognition of ground b as an example. At time ki, the RGB image and the corresponding radar histogram features are obtained from the data storage unit. ResNet is used to extract the high-dimensional features of the RGB image. The radar histogram features are further extracted using a multi-layer perceptron (MLP). After that, the high-dimensional features of the image and radar are combined together to realize the pre-recognition of ground b using a multi-layer perceptron (MLP).

[0113] At time k, the 6-dimensional features extracted from the IMU and the 4-dimensional features extracted from the force signal are obtained from the data storage unit and combined with the high-dimensional features of the image and radar obtained at time ki to re-identify the ground b using the support vector machine (SVM).

[0114] Figure 2It is just a process of testing the data. The training process needs to follow the above process of collecting sensor data and extracting features. Figure 2 The mentioned ResNet, MLP, SVM and other models are trained, and the trained models can be used for testing.

[0115] The present invention aims to propose a system framework for the fusion of internal and external perception combined with time series. In the specific implementation process, other types of sensors can be used, and other types of features can be extracted from sensor data. Classification models such as MLP and SVM can also be replaced by other machine learning models.

[0116] It can be seen that the system provided by the present invention first uses exteroceptive data to predict the terrain to be passed, which can realize the early adjustment of the control strategy, ensure the forward-looking nature of the adjustment, and reduce potential risks. When passing through the predicted terrain, combined with the internal perception method, the terrain can be identified again, and the possible errors in the prediction can be corrected to adjust the control strategy in real time, further reducing the risk of passing through unknown terrain. Combining the method of terrain recognition with internal and external perception fusion can achieve more accurate terrain recognition.

[0117] In summary, the terrain recognition system provided by the present invention ensures a more stable walking of the quadruped robot by accurately identifying the terrain. Whether it is external perception or internal perception, multi-sensor fusion perception is used to achieve complementary advantages and improve the robustness and accuracy of terrain recognition. For the stable walking of the quadruped robot, a combination of internal and external perception is adopted, and a terrain recognition method that combines internal and external perception can also be realized in combination with time sequence.

[0118] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0120] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A terrain recognition system, characterized in that: It includes an exteroceptive sensor module, an interoceptive sensor module and an additional module mounted on a quadruped robot; the exteroceptive sensor module includes at least two different types of exteroceptive data acquisition sensors, and the interoceptive sensor module includes at least two different types of interoceptive data acquisition sensors; The additional modules are used to perform the following operations: Acquire multi-source heterogeneous exteroceptive data of a target ground area collected by the exteroceptive sensor module at a previous moment; the target ground area is a ground area passed by the quadruped robot at a current moment; Extracting features from the multi-source heterogeneous exteroceptive data to obtain multi-source heterogeneous exteroceptive high-dimensional features; Using a multi-layer perceptron in combination with the multi-source heterogeneous exteroceptive high-dimensional features to pre-identify the terrain of the target ground area; Acquire multi-source heterogeneous interoceptive data of the target ground area currently collected by the interoceptive sensor module; Performing feature extraction on the multi-source heterogeneous interoceptive data to obtain multi-source heterogeneous interoceptive features; The terrain of the target ground area is re-identified by using a support vector machine in combination with the multi-source heterogeneous exteroceptive high-dimensional features and the multi-source heterogeneous interoceptive features.

2. The terrain recognition system according to claim 1, characterized in that: The exteroceptive sensor module includes at least an RGB camera and a laser radar, and the multi-source heterogeneous exteroceptive data includes at least RGB image information and radar point cloud data.

3. The terrain recognition system according to claim 2, characterized in that: Extracting features from the multi-source heterogeneous exteroceptive data to obtain multi-source heterogeneous exteroceptive high-dimensional features includes: Extract the RGB image information using ResNet to obtain high-dimensional features of the image; Optical histogram features are created using the radar point cloud data, and high-dimensional features of the radar are obtained by extracting features from the optical histogram features using the multi-layer perceptron, so as to obtain the multi-source heterogeneous external perception high-dimensional features after combining the high-dimensional features of the image with the high-dimensional features of the radar.

4. The terrain recognition system according to claim 3, characterized in that: The step of creating and obtaining an optical histogram feature using the radar point cloud data comprises: Filtering out lidar points below an intensity threshold in the radar point cloud data, and projecting and clipping the remaining points onto corresponding images; By calculating the numerical distribution of the three channels of ambient light, reflectivity and intensity, average frequency histograms of the three channels of different ground types are drawn to create the optical histogram feature.

5. The terrain recognition system according to claim 1, characterized in that: The interoceptive sensor module includes an IMU inertial measurement unit and a force sensor, and the multi-source heterogeneous interoceptive data includes vibration information and pressure information of the ground.

6. The terrain recognition system according to claim 5, characterized in that: Extracting features from the multi-source heterogeneous interoceptive data to obtain multi-source heterogeneous interoceptive features includes: Performing power spectrum analysis and extraction on the vibration information to obtain the minimum translation acceleration, the maximum translation acceleration, the average translation acceleration, the translation acceleration variance, the root mean square of the translation acceleration, and the translation acceleration norm; The pressure information is subjected to time domain feature extraction to obtain sample expectation, sample standard deviation, sample coefficient of variation and peak value.

7. The terrain recognition system according to claim 6, characterized in that: The minimum translational acceleration is extracted by the following formula: The maximum translational acceleration is extracted by the following formula: The average translational acceleration is extracted by the following formula: The translational acceleration variance is extracted by the following formula: The root mean square of the translational acceleration is extracted by the following formula: The translational acceleration norm is extracted by: Where: The frequency domain data representing the vibration information, N represents the number of data.

8. The terrain recognition system according to claim 6, characterized in that: The sample expects: Sample standard deviation: Sample coefficient of variation: crest: b4=max(x(t)) Where: x(t) represents the data collected by the force sensor.

9. The terrain recognition system according to claim 1, characterized in that: The additional module includes a data transmission unit, a data storage unit, a data processing unit and a terrain recognition unit.

Images