SVM-based difficult sample mining and CNN-based weighted training method for terrain classification

Abstract

This invention proposes a terrain classification method based on SVM hard sample mining and CNN weighted training, belonging to the field of robot environmental perception technology. The method includes the following steps: collecting multimodal monitoring data of a robot moving on various terrains; preprocessing the multimodal monitoring data to generate multimodal feature vectors at different times; and forming a training sample set from these multimodal feature vectors. This invention uses a support vector machine to perform prior difficulty assessment on the training samples, quantifying the distance from the sample to the classification decision boundary as a difficulty index, and generating dynamic training weights accordingly. Then, during the training of the convolutional neural network, these weights are used as weighting factors to construct a loss function, allowing the model optimization process to focus on learning the features of samples with higher classification difficulty, thereby effectively improving the system's accuracy in overall terrain classification and the model's robustness in real-world complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception technology, and in particular to a terrain classification method based on SVM hard sample mining and CNN weighted training. Background Technology

[0002] With the increasing demand for autonomous operation of intelligent mobile platforms such as wheeled and quadruped robots in complex environments, accurate and robust terrain perception capabilities have become a key prerequisite for achieving stable movement, adaptive gait planning, and safe navigation. Currently, mainstream technologies rely on fusing multi-source sensor information from the robot itself for terrain recognition. This involves collecting multimodal time-series data generated by the robot's movement on different terrains, extracting features, and using classification algorithms to determine the current terrain category, providing crucial environmental perception input for higher-level decision-making.

[0003] Existing terrain classification methods can be mainly divided into two categories: methods based on traditional machine learning and methods based on deep learning. Methods based on traditional machine learning, such as Support Vector Machines (SVMs), require manual feature design and extraction, and the construction of a classifier within these fixed feature spaces. SVMs are widely used due to their good generalization ability in small sample sizes and high-dimensional scenarios. However, their classification performance heavily depends on the quality of feature representation. When the distributions of different terrain samples in the feature space overlap or are in a transitional state, even with optimized model parameters, some misclassified samples will still exist. Simply adjusting the hyperparameters of the SVM itself cannot fundamentally solve this problem. Deep learning-based methods, especially Convolutional Neural Networks (CNNs), can automatically learn hierarchical feature representations from raw data, avoiding tedious manual feature engineering. However, in actual training, standard training strategies are often dominated by a large number of easily classifiable samples, leading to insufficient learning of the few but crucial difficult-to-classify samples near the decision boundary. This limits the overall classification performance of the model, especially further improvement in generalization ability. Summary of the Invention

[0004] In view of this, this invention proposes a terrain classification method based on SVM hard sample mining and CNN weighted training. By using a support vector machine to perform prior difficulty assessment on the training samples, the distance from the sample to the classification decision boundary is quantified as a difficulty index, and dynamic training weights are generated accordingly. Then, during the training process of the convolutional neural network, these weights are used as weighting factors to construct a loss function, allowing the model optimization process to focus on learning the features of samples with higher classification difficulty. This combines the advantages of traditional machine learning models in defining the discrimination boundary with the powerful feature learning capabilities of deep learning models. Without changing the original data distribution or relying on information from the testing phase, this method systematically guides the model to strengthen its feature recognition ability for boundary samples and difficult-to-classify terrain, thereby effectively improving the overall accuracy of terrain classification and the robustness of the model in real-world complex scenarios.

[0005] The technical solution of this invention is implemented as follows: In a first aspect, the present invention provides a terrain classification method based on SVM hard sample mining and CNN weighted training, comprising the following steps: Multimodal monitoring data of the robot moving on various terrains is collected, the multimodal monitoring data is preprocessed to generate multimodal feature vectors at different times, and a training sample set is composed of multiple multimodal feature vectors at different times. A support vector machine terrain classification model is constructed. The support vector machine terrain classification model is trained using a training sample set. The distance from each multimodal feature vector in the training sample set to the classification decision is calculated using the trained support vector machine terrain classification model to obtain a quantitative index of classification difficulty. Based on the classification difficulty quantification index of each multimodal feature vector, the training weight corresponding to each multimodal feature vector is dynamically calculated, and the training weight is negatively correlated with the classification difficulty quantification index value. A convolutional neural network classification model is constructed. The training sample set is used as input, and the training weights are used as weighting factors. A weighted loss function is constructed to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify. The multimodal monitoring data collected in real time by the robot is input into the trained convolutional neural network classification model to obtain the terrain classification results.

[0006] In some embodiments, the multimodal monitoring data collected by the robot during its movement across various terrains is preprocessed to generate multimodal feature vectors at different times. A training sample set is then composed of these multimodal feature vectors, specifically including: Control the robot to move on various terrains, and simultaneously collect motion data output by the inertial measurement unit and joint angle data output by the joint encoder; The synchronously acquired motion data and joint angle data are time-stamp aligned and synchronized to obtain synchronized sensor timing data; The synchronized sensor time-series data are subjected to feature extraction to generate inertial measurement unit feature vectors and joint encoder feature vectors. The inertial measurement unit feature vector and the joint encoder feature vector are standardized and then fused to generate a multimodal feature vector corresponding to each sampling time. The multimodal feature vectors from multiple sampling times are collected to form a training sample set.

[0007] In some embodiments, the construction of the support vector machine terrain classification model involves training the support vector machine terrain classification model using a training sample set, and then using the trained support vector machine terrain classification model to calculate the distance from each multimodal feature vector in the training sample set to the classification decision, thereby obtaining a quantitative index of classification difficulty. Specifically, this includes: The support vector machine terrain classification model is trained using the training sample set and its corresponding terrain category labels to construct the classification decision hyperplane; Using the decision function in the trained support vector machine terrain classification model, calculate the output value of the decision function for each multimodal feature vector in the training sample set; Based on the decision function output value of each obtained multimodal feature vector, calculate the geometric distance from the corresponding multimodal feature vector to the classification decision hyperplane; The absolute value of the geometric distance is defined as a quantitative index of the classification difficulty of the corresponding multimodal feature vector. The smaller the absolute value of the geometric distance, the higher the classification difficulty of the corresponding multimodal feature vector.

[0008] In some embodiments, the construction of the convolutional neural network classification model, using a training sample set as input and training weights as weighting factors, and constructing a weighted loss function to train the convolutional neural network classification model, specifically includes: Construct a convolutional neural network classification model, wherein the input layer of the convolutional neural network classification model is configured to receive multimodal feature vectors; Construct a weighted loss function, which is obtained by combining the standard cross-entropy loss function with the training weights; The convolutional neural network classification model is trained using the training sample set and weighted loss function. During training, the contribution of each multimodal feature vector to the total loss of the convolutional neural network classification model is dynamically adjusted according to the training weights. The multimodal feature vector with larger training weights has a greater impact on the parameter update of the convolutional neural network classification model during backpropagation.

[0009] In some embodiments, inputting the multimodal monitoring data collected in real time by the robot into a trained convolutional neural network classification model to obtain terrain classification results specifically includes: The raw data stream from the multimodal sensors acquired in real time by the robot is preprocessed to generate the real-time multimodal feature vector at the current moment. The generated real-time multimodal feature vectors are input into the already trained convolutional neural network classification model; The convolutional neural network classification model performs forward propagation calculation on the input real-time multimodal feature vector, and obtains the probability distribution vector corresponding to each terrain category at the output of the fully connected classification layer; The terrain category with the highest probability value in the probability distribution vector is selected as the final real-time terrain classification result output.

[0010] In some embodiments, the weighted loss function Represented as:

[0011] Weight Represented as:

[0012] in, It is a standard loss function. It is the first The true label of each sample It is the model's prediction result. It is the first The weights of each sample, It is the total number of training samples. It is the first The distance from each sample to the classification decision boundary. It is a small constant that prevents the denominator from being zero.

[0013] In a further embodiment, the multimodal sensor further includes a microphone. The process of collecting multimodal monitoring data of the robot moving across various terrains, and preprocessing the multimodal monitoring data to generate multimodal feature vectors at different times, further includes: The sound signals generated by the robot as it moves across different terrains are collected using a microphone. Mel frequency cepstral coefficients are extracted from the sound signal to obtain the acoustic feature vector; The acoustic feature vector is fused with the feature vectors of the inertial measurement unit and the joint encoder to generate a multimodal feature vector.

[0014] Secondly, the present invention provides a terrain classification system based on SVM hard sample mining and CNN weighted training, for implementing the aforementioned terrain classification method based on SVM hard sample mining and CNN weighted training, comprising: The data acquisition and preprocessing module is used to acquire multimodal monitoring data of the robot when it moves on various terrains, preprocess the multimodal monitoring data to generate multimodal feature vectors at different times, and form a training sample set by the multimodal feature vectors at multiple different times. The sample difficulty assessment module is used to construct a support vector machine terrain classification model. The support vector machine terrain classification model is trained using the training sample set, and the distance from each multimodal feature vector in the training sample set to the classification decision is calculated using the trained support vector machine terrain classification model to obtain a quantitative index of classification difficulty. The training weight generation module is used to dynamically calculate the training weight corresponding to each multimodal feature vector based on the classification difficulty quantification index, wherein the training weight is negatively correlated with the classification difficulty quantification index value. The neural network training module is used to construct a convolutional neural network classification model. It uses the training sample set as input and the training weights as weighting factors to construct a weighted loss function to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify. The real-time classification module is used to input the multimodal monitoring data collected by the robot in real time into the trained convolutional neural network classification model to obtain the terrain classification result.

[0015] Thirdly, the present invention provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the above-described method.

[0016] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0017] The terrain classification method based on SVM hard sample mining and CNN weighted training of the present invention has the following advantages over the prior art: 1. By using support vector machines to perform prior difficulty assessment of training samples, the distance from the sample to the classification decision boundary is quantified as a difficulty index, and dynamic training weights are generated accordingly. Then, during the training process of the convolutional neural network, the loss function is constructed using these weights as weighting factors, so that the model optimization process focuses on learning the features of samples with higher classification difficulty. This combines the advantages of traditional machine learning models in characterizing the discrimination boundary with the powerful feature learning capabilities of deep learning models. Without changing the original data distribution and without relying on information from the testing phase, the model is systematically guided to strengthen its feature recognition capabilities for boundary samples and difficult-to-classify terrain, thereby effectively improving the accuracy of the system in overall terrain classification and the robustness of the model in real complex scenarios. 2. By synchronously acquiring, feature-level fusion, and standardizing the inertial measurement unit, joint encoder, and audio signals, a multimodal feature vector with complementary information was constructed. This processing method fully utilizes the unique advantages of different sensing modes, namely, the IMU reflects the overall motion state, the encoder reflects the details of joint motion, and the audio captures the terrain contact features, while eliminating dimensional differences. This provides high-quality and highly consistent input for subsequent models, fundamentally enhancing the system's ability to represent and distinguish complex terrain. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the terrain classification method based on SVM hard sample mining and CNN weighted training of the present invention. Figure 2 This is a schematic diagram of the convolutional neural network classification model of the terrain classification method based on SVM hard sample mining and CNN weighted training according to the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] like Figure 1 As shown, the terrain classification method based on SVM hard sample mining and CNN weighted training of the present invention can be divided into five steps in this embodiment, namely steps S100-S500.

[0022] Step S100: Collect multimodal monitoring data of the robot moving on various terrains, preprocess the multimodal monitoring data to generate multimodal feature vectors at different times, and form a training sample set from the multimodal feature vectors at multiple different times.

[0023] This step is used for data acquisition and the construction of the training sample set, while preparing high-quality basic data for subsequent model training. Specifically, the robot is controlled to perform standard movements on various terrains to be identified, such as grass, sand, and stone paths, while simultaneously collecting multimodal monitoring data. Key preprocessing is performed on the raw data stream, such as aligning and synchronizing multi-source data based on a unified timestamp. Then, discriminative features are extracted from each sensor data, such as motion statistics from IMU data and angle and velocity information from joint encoder data. The extracted features are standardized to eliminate the bias caused by differences in the units and numerical ranges of different sensor data. Finally, different modal features at each synchronization moment are obtained. The processed modal features at each synchronization moment are spliced ​​and fused to form a multimodal feature vector. The vectors from multiple consecutive moments constitute the training sample set for model training.

[0024] Step S200: Construct a support vector machine terrain classification model. Train the support vector machine terrain classification model using a training sample set, and use the trained support vector machine terrain classification model to calculate the distance from each multimodal feature vector in the training sample set to the classification decision, thereby obtaining a quantitative index of classification difficulty.

[0025] After obtaining the training sample set, a Support Vector Machine (SVM) terrain classification model is trained using the training sample set constructed in step S100 and its corresponding terrain labels. The essence of training the SVM terrain classification model is to find an optimal classification decision hyperplane in the feature space to maximize the margin between samples of different classes. After training, it is used as a difficulty estimator to calculate the distance from each multimodal feature vector in the training sample set to the classification decision hyperplane. This distance is defined as a quantitative indicator of the classification difficulty of the sample; that is, the closer the distance, the closer the sample is to the classification boundary, the lower the confidence of the SVM model in classifying it, and the higher the classification difficulty; conversely, the farther the distance, the lower the classification difficulty.

[0026] Step S300: Based on the classification difficulty quantification index of each multimodal feature vector, dynamically calculate the training weight corresponding to each multimodal feature vector, wherein the training weight is negatively correlated with the classification difficulty quantification index value.

[0027] This step transforms the abstract difficulty index obtained in the previous step into specific weights that can guide the training of the deep network. Based on the classification difficulty quantification index of each sample calculated earlier, a pre-defined weight calculation function dynamically calculates the training weight corresponding to that sample. The design of this function ensures that the training weight is negatively correlated with the classification difficulty quantification index value; that is, samples with smaller difficulty index values ​​are assigned larger training weights.

[0028] Step S400: Construct a convolutional neural network classification model. Use the training sample set as input and the training weights as weighting factors to construct a weighted loss function to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify.

[0029] This step constructs a convolutional neural network (CNN) classification model. A one-dimensional CNN can be chosen to adapt to temporal feature vectors. Instead of using a standard uniform loss function, a weighted loss function is constructed when training this CNN model. Specifically, the previously generated training weights are used as weighting factors and combined with the standard loss function. In each iteration of model optimization, the weighted loss function adjusts its contribution to the overall loss differently based on the weight of each sample. This amplifies the loss of difficult samples with high weights, thus having a greater impact on model parameter updates during backpropagation, while the impact of easy samples with low weights is relatively reduced. This mechanism forces the CNN model to pay more attention to and strive to learn the features of samples with higher classification difficulty during parameter optimization, thereby effectively improving the model's ability to identify terrain with blurred boundaries.

[0030] Step S500: Input the multimodal monitoring data collected by the robot in real time into the trained convolutional neural network classification model to obtain the terrain classification result.

[0031] Finally, actual data can be collected. During the robot's real-time operation, the multimodal monitoring data collected is preprocessed using the same procedure as in step S100 to generate a real-time multimodal feature vector for the current moment. This vector is then input into the convolutional neural network classification model trained in step S400, which possesses strong hard sample learning capabilities. The model calculates through forward propagation and outputs a distribution representing the probability of belonging to each terrain category. Finally, the category with the highest probability is output as the terrain classification result, thus completing the real-time and accurate terrain recognition task.

[0032] In some embodiments, step S100 can be further divided into multiple sub-steps, namely steps S101-S105, to achieve accurate data synchronization and the construction of high-quality feature vectors.

[0033] Step S101: Control the robot to move on various terrains, and simultaneously collect motion data output by the inertial measurement unit and joint angle data output by the joint encoder.

[0034] First, the robot is controlled to perform standardized walking gait on various preset terrains, such as concrete, grass, and gravel roads. At the same time, the data streams of its onboard inertial measurement unit (IMU) and joint encoders are triggered and recorded. The IMU publishes six-dimensional data, including three-axis acceleration and three-axis angular velocity, at a fixed frequency, such as 50Hz. The joint encoders simultaneously publish the angular position information of each joint. For a typical 12-joint quadruped robot, 12-dimensional data is generated. During the acquisition, a precise system timestamp is recorded for each data point to lay the foundation for subsequent synchronization.

[0035] Step S102: Perform timestamp alignment and synchronization processing on the synchronously acquired motion data and joint angle data to obtain synchronized sensor timing data.

[0036] Because inherent sensor frequencies or transmission delays may cause data to be incompletely aligned in time, this step performs timestamp alignment and synchronization processing on the raw data acquired by S101. Specifically, using a unified reference clock, such as the ROS system time, the IMU data stream and the joint encoder data stream are interpolated or matched using their corresponding timestamps to ensure that each processing point in time has a strictly corresponding set of IMU data packets and joint encoder data packets, forming synchronized sensor timing data. If the sampling frequencies are inconsistent, the data can be resampled to the same frequency, such as 50Hz, using methods such as linear interpolation.

[0037] Step S103: Extract features from the synchronized sensor time-series data to generate feature vectors for the inertial measurement unit and joint encoder.

[0038] Then, targeted feature extraction is performed on the previously obtained synchronous timing data. For IMU data, statistical features can be extracted from each aligned data window, such as the mean, variance, standard deviation, maximum, minimum values, and spectral energy of acceleration and angular velocity for each axis, ultimately forming an inertial measurement unit feature vector, which can be an 18-dimensional vector. For joint encoder data, statistical features of the angle sequence of each joint can be extracted, such as the mean, range, mean and variance of motion velocity, or the instantaneous joint angle values ​​at the alignment moment can be directly used to form a joint encoder feature vector, which can be a 12-dimensional or higher-dimensional vector based on statistics. This step transforms the original timing signal into a more discriminative static feature representation.

[0039] Step S104: Standardize the feature vectors of the inertial measurement unit and the joint encoder respectively, and then fuse them to generate a multimodal feature vector corresponding to each sampling time.

[0040] Because IMU features and joint encoder features differ significantly in numerical scale and distribution range (e.g., acceleration units m / s² vs. angle units rad), direct fusion can lead to unstable model training. Therefore, this step standardizes the inertial measurement unit feature vector and joint encoder feature vector generated by S103 separately. Specifically, Z-score standardization can be used, which calculates the mean and standard deviation of each feature dimension over the entire training set, and then transforms the feature values. After processing, each feature dimension follows a standard normal distribution with a mean of 0 and a standard deviation of 1, eliminating the influence of dimensions and enabling the subsequent model to learn information from different sensors equally.

[0041] Then, the standardized IMU feature vector and the joint encoder feature vector are concatenated and fused. For example, if the IMU feature vector is 18-dimensional and the joint encoder feature vector is 12-dimensional, they are directly concatenated to form a 30-dimensional multimodal feature vector. This vector comprehensively expresses the robot's overall motion state and leg joint configuration state at a specific moment, with complementary information, providing rich joint feature representations for terrain classification. Such a vector is generated for each aligned sampling moment.

[0042] Step S105: Collect multimodal feature vectors from multiple sampling times to form a training sample set.

[0043] Finally, all multimodal feature vectors generated by the robot across all terrains and multiple motion cycles at different sampling times are collected, and each vector is labeled with the corresponding real terrain category label at the time of collection, such as "grassland" or "sand". The set of these labeled feature vectors constitutes the training sample set for subsequent model training and evaluation. During construction, it can be randomly divided into training and test sets in a certain ratio to ensure the fairness of model evaluation; the ratio of training to test sets can be set to 7:3.

[0044] In some embodiments, in order to further enhance the characterization capability and classification robustness of terrain features, step S100 includes the acquisition and fusion processing of sound signals in addition to the acquisition of inertial measurement unit and joint encoder data, which can be implemented through sub-steps S107-S109.

[0045] Step S107: Collect sound signals generated by the robot as it moves on different terrains using a microphone.

[0046] The robot uses a microphone integrated into the robot body or connected externally via a USB expansion port to synchronously collect sound signals generated by the interaction between the robot's feet or body and the ground when the robot moves on different terrains. The sound signals are recorded in the form of raw pulse code modulation stream. To coordinate with data from other sensors, an independent ROS audio acquisition node is created. This node captures audio data from the microphone hardware interface, adds a precise system timestamp to it, and publishes it to the ROS topic to achieve a synchronous acquisition framework with the IMU and joint encoder data streams in the time dimension.

[0047] Step S108: Extract the Mel frequency cepstral coefficient features from the sound signal to obtain the acoustic feature vector.

[0048] This step is used to extract Mel-frequency cepstral coefficient features from the raw sound signal acquired in step S107. First, the audio stream is pre-emphasized and windowed in frames. For example, a Hamming window with a width of 20 milliseconds is used, sliding at 10-millisecond intervals. A fast Fourier transform is performed on each frame of the signal to obtain the spectrum. Then, a set of Mel-scale triangular filter banks is passed through, and the logarithmic energy of each filter output is calculated. Finally, a discrete cosine transform is performed on the filtered logarithmic energy sequence, retaining the first 13 coefficients, which yields a 13-dimensional Mel-frequency cepstral coefficient feature vector. This vector can effectively characterize the short-time power spectral envelope characteristics of terrain contact sound.

[0049] Step S109: Fuse the acoustic feature vector with the inertial measurement unit feature vector and the joint encoder feature vector to generate a multimodal feature vector.

[0050] To ensure that the multi-source information is aligned in time and frequency, the acoustic feature sequence extracted in step S108 needs to be processed. For example, if the sampling frequency of the IMU and joint encoder data is 50Hz, and the MFCC feature sequence has an equivalent frequency of 100Hz due to a window shift of 10ms, then the continuous MFCC feature vectors need to be downsampled to reduce their frequency to 50Hz. Then, the standardized inertial measurement unit feature vector, joint encoder feature vector, and downsampled and aligned acoustic feature vector are concatenated and fused. Finally, a multimodal feature vector containing motion, attitude, and acoustic information, for example, 31 dimensions, is generated, specifically 6-dimensional IMU features + 12-dimensional joint encoder features + 13-dimensional MFCC features. This vector provides a more comprehensive and complementary terrain perception input for subsequent models.

[0051] In one specific embodiment, each fused multimodal feature vector contains 31 dimensions, represented as follows:

[0052] in, The fused multimodal feature vector The feature vector of the IMU, The feature vector of the joint encoder, This represents the feature vector of the sound signal. Essentially, each data point is a 1×31 vector. By having the robot move across various terrains to collect data, n×31 training samples can be obtained.

[0053] In some embodiments, step S200 can be further divided into multiple sub-steps, namely steps S201-S204, to achieve sample difficulty quantification based on support vector machine geometric margin.

[0054] Step S201: Use the training sample set and its corresponding terrain category labels to train the support vector machine terrain classification model to construct the classification decision hyperplane.

[0055] The training process aims to find an optimal classification decision hyperplane, mathematically expressed as w*x + b = 0, where w is the normal vector of the hyperplane and b is the bias term. The training objective of SVM is to maximize the geometric margin between the support vectors of different classes and this hyperplane in the feature space. For non-linearly separable terrain data, a kernel function can be used to map the original features to a high-dimensional space, in which a linear hyperplane is constructed. After model training is complete, the optimal classification decision hyperplane is determined, serving as a benchmark for subsequent sample difficulty evaluation.

[0056] Step S202: Using the decision function in the trained support vector machine terrain classification model, calculate the output value of the decision function for each multimodal feature vector in the training sample set.

[0057] Using its decision function f(x) = w*φ(x) + b, where φ(x) is the feature vector after possible kernel mapping, calculate each multimodal feature vector x in the training sample set. i The output value of the decision function f(x) i The output value is a continuous scalar. For a binary classification problem, f(x) i The sign of the symbol () indicates the class predicted by the model, and its absolute value represents the class of the sample point x. i The distance from the classification decision hyperplane is the distance; the larger the absolute value, the farther the distance and the higher the classification confidence.

[0058] Step S203: Calculate the geometric distance from the corresponding multimodal feature vector to the classification decision hyperplane based on the decision function output value of each obtained multimodal feature vector.

[0059] Based on each multimodal feature vector x obtained through calculation i The output value of the decision function f(x) i ), calculate the geometric distance d from the sample to the classification decision hyperplane w*x+b=0. i The formula for calculating geometric distance is d. i =|f(x i )| / ||w||, where ||w|| is the norm of the hyperplane normal vector, and the distance d i It is a non-negative scalar value representing the sample point x. i The absolute offset perpendicular to the hyperplane in the feature space is a precise geometric measure of the sample's spatial location. For multi-class SVM, it is the distance from the sample to its corresponding class decision hyperplane.

[0060] Step S204: Define the absolute value of the geometric distance as a quantitative index of the classification difficulty of the corresponding multimodal feature vector. The smaller the absolute value of the geometric distance, the higher the classification difficulty of the corresponding multimodal feature vector.

[0061] Finally, the calculated geometric distance d i The absolute value of x is formally defined as the absolute value of the sample x. i The classification difficulty quantification index, its physical meaning lies in the geometric distance d i The smaller the distance, the closer the multimodal feature vector is to the classification decision hyperplane in the feature space, placing it in the fuzzy boundary region of classification. This results in a lower certainty of the support vector machine model in classifying it, thus increasing the classification difficulty. Conversely, a larger distance indicates that the sample is further from the boundary, leading to a clearer classification and lower difficulty. This step ensures that all training samples obtain comparable difficulty scores, providing a data foundation for generating differentiated training weights.

[0062] In some embodiments, step S300 converts the classification difficulty quantification index into specific training weights that can directly guide the optimization process of the deep neural network. The core of this step is to assign appropriate weight values ​​to each sample based on its computational difficulty, thereby providing differentiated guidance for the model's learning focus in subsequent training.

[0063] Specifically, the classification difficulty quantification index d for each multimodal feature vector defined in this step is... i This refers to the geometric distance from the sample to the classification decision hyperplane. The training weight ω corresponding to the sample is dynamically calculated using a pre-defined weight calculation function. i The design principle of this function is that the values ​​of the training weights ωi should be proportional to the input classification difficulty quantification index d. i The values ​​of ω and d are negatively correlated; samples with smaller di values ​​will be assigned a larger training weight ω. i Conversely, samples with larger di values ​​are assigned smaller weights.

[0064] In an optional embodiment, weights It can be represented as:

[0065] in, It is the first The weights of each sample, It is the first The distance from each sample to the classification decision boundary. It is a small constant that prevents the denominator from being zero.

[0066] The classification difficulty quantification index d is calculated using this function. i With training weights ω i A strict reciprocal relationship exists between them. For example, a d i Very small hard samples have very small denominators, which will produce a very large ω. i ; and a d i A large number of easy samples will result in a large denominator, leading to a very small ω. i Once all samples have been calculated, a set of training weights matching the difficulty of the samples is obtained. These weights, as key auxiliary information, will be input into step S400 to construct a weighted loss function, thereby achieving precise intervention in the training process of the convolutional neural network model and forcing the model to allocate optimization resources to difficult-to-classify samples.

[0067] In some embodiments, step S400 can be further divided into multiple sub-steps, namely steps S401-S403, which utilize the training weights generated in the preceding steps to construct a loss function with a clear optimization orientation, thereby performing targeted training on the convolutional neural network classification model.

[0068] Step S401: Construct a convolutional neural network classification model, wherein the input layer of the convolutional neural network classification model is configured to receive multimodal feature vectors.

[0069] First, a convolutional neural network (CNN) classification model is constructed. Its network structure is specifically designed to process the multimodal feature vectors generated in step S105. Since the input feature vectors are one-dimensional arrays describing multimodal sensing information at a given moment, a one-dimensional CNN architecture is preferred. The model's input layer is configured to receive multimodal feature vectors of a fixed dimension, such as the aforementioned 30-dimensional vector. The network can contain multiple stacked one-dimensional convolutional layers, pooling layers, and fully connected layers. The one-dimensional convolutional layers, by sliding along the sequence dimension of the feature vectors, can effectively extract local correlations and patterns between features from different sensors. The pooling layers are used to reduce dimensionality and enhance the translation invariance of the features. Finally, the fully connected layers are responsible for mapping the learned high-level features to specific terrain category probability distributions.

[0070] Specifically, such as Figure 2 As shown, the constructed convolutional neural network classification model adopts a specific one-dimensional convolutional neural network architecture, whose network structure is connected sequentially from top to bottom. The specific configuration and data flow are as follows: Input layer: The network input is configured to receive a 31-dimensional multimodal feature vector. This vector is composed of 6-dimensional features from the inertial measurement unit, 12-dimensional features from the joint encoder, and 13-dimensional MFCC features from the sound signal, thus fully representing the multi-source sensing information at a given moment.

[0071] The first one-dimensional convolutional layer follows the input layer. This layer has 64 convolutional kernels of size 3, which extract local feature combinations and primary patterns by performing sliding convolution operations on the input one-dimensional feature sequence. Each convolutional kernel is followed by a non-linear activation function to introduce the non-linear expressive power of the model.

[0072] Max pooling layer: Connected after the first convolutional layer. This layer is configured with a pooling window of size 2. Its function is to downsample the feature map output by the convolutional layer. By taking the maximum value within the window, the pooling layer can reduce the spatial size of the data and the amount of subsequent computation while retaining the most salient features, and give the model a certain degree of translation invariance.

[0073] The second one-dimensional convolutional layer follows the max-pooling layer. This layer is configured with 128 convolutional kernels of size 3, used to further extract and combine features at a more abstract level. The increased number of convolutional kernels aims to learn more complex feature representations.

[0074] Flattening layer: Connected after the second convolutional layer. This layer converts the two-dimensional feature map output after multiple convolutions and pooling into a one-dimensional feature vector, which is then input into the subsequent fully connected layers for processing.

[0075] The first fully connected layer follows the flattened layer. This layer contains 64 neurons and is responsible for performing nonlinear transformations and integration on the flattened high-level features, learning the global correlations between features.

[0076] The second fully connected layer follows the first fully connected layer. It also contains 64 neurons and further refines and maps the features, preparing for the final classification.

[0077] Output layer: Also known as the classification layer, it is connected after the last fully connected layer. This layer can choose a Softmax classifier with the number of neurons equal to the total number of terrain categories to be classified. It maps the final feature representation to the probability distribution of each terrain category, and the category with the highest probability value is the model's classification result.

[0078] Step S402: Construct a weighted loss function, which is obtained by combining the standard cross-entropy loss function with the training weights.

[0079] In this step, a weighted loss function is constructed to differentiate between different samples during model training. This function is obtained by combining the standard cross-entropy loss function with the training weights calculated in step S300.

[0080] In an optional embodiment, the weighted loss function Represented as:

[0081] in, This represents the total value of the constructed weighted loss function. It is a standard loss function. It is the first The true label of each sample It is the model's prediction result. It is the first The weights of each sample, This is the total number of training samples.

[0082] With this construction method, the standard cross-entropy loss for each sample All of them are scaled by their corresponding training weights ωi, thus directly encoding the prior classification difficulty information of the samples into the optimization objective.

[0083] Step S403: Train the convolutional neural network classification model using the training sample set and weighted loss function. During the training process, the contribution of each multimodal feature vector to the total loss of the convolutional neural network classification model is dynamically adjusted according to the training weights. The multimodal feature vector with larger training weights has a greater impact on the parameter update of the convolutional neural network classification model during backpropagation.

[0084] The constructed convolutional neural network classification model is trained using the built training sample set and the constructed weighted loss function. In each forward propagation of the training process, the model calculates the predicted value and its standard loss for each sample in the batch, and then calculates the loss based on the training weight ω for each sample. i A weighted sum is performed to obtain the total loss for this batch. In the subsequent backpropagation and parameter update phases, the gradient calculation of the total loss with respect to the model parameters naturally includes the weight factor ω. i The effect is due to the training weights ω. i Larger samples have a higher proportion of their corresponding loss term in the total loss, thus exerting a greater pull on the overall parameter update direction from their gradient. Conversely, samples with smaller weights have a weaker influence. This mechanism dynamically adjusts the contribution of each multimodal feature vector to the model's total loss, forcing the model to prioritize reducing the prediction error of high-weight samples when optimizing parameters. This achieves the fundamental goal of focusing the model on learning features from samples with higher classification difficulty. With each training iteration, the model's ability to represent features from difficult samples is specifically enhanced, ultimately improving overall classification accuracy and generalization robustness.

[0085] In some embodiments, step S500 can be further divided into multiple sub-steps, namely steps S501-S504, to fully realize the end-to-end inference process from real-time data acquisition to the final classification result output.

[0086] Step S501: Preprocess the raw data stream from the multimodal sensors acquired by the robot in real time to generate the real-time multimodal feature vector at the current moment.

[0087] For the raw multimodal sensor data streams continuously acquired during the robot's real-time operation, perform the same preprocessing operations as in the model training phase of step S100. Specifically, this includes: aligning and synchronizing the real-time IMU data stream and the joint encoder data stream with timestamps; extracting inertial measurement unit features and joint encoder features from the synchronized real-time data using the same algorithm and parameter settings as in the training phase; standardizing the extracted real-time features using the mean and standard deviation parameters of each feature dimension calculated and saved in the training phase; and finally concatenating and fusing the standardized real-time features of each modality to generate a real-time multimodal feature vector that is completely consistent with the training sample format and represents the current environment and state information.

[0088] Step S502: Input the generated real-time multimodal feature vector into the already trained convolutional neural network classification model.

[0089] Real-time multimodal feature vectors are used as input data and fed into the trained convolutional neural network classification model. Upon deployment, the trained model parameters are already stored in memory or an embedded computing unit, ready for inference.

[0090] Step S503: The convolutional neural network classification model performs forward propagation calculation on the input real-time multimodal feature vector, and obtains the probability distribution vector corresponding to each terrain category at the output of the fully connected classification layer.

[0091] After receiving the real-time multimodal feature vector as input, the convolutional neural network classification model automatically performs a complete forward propagation computation. The data flows sequentially through all layers of the network, undergoing a series of linear and nonlinear transformations. Finally, at the output layer of the network, a probability distribution vector is generated. The dimension of this vector is equal to the total number of terrain categories, and each element in the vector represents the predicted probability that the current input data belongs to the corresponding terrain category. The sum of the probabilities of all elements is 1.

[0092] Step S504: Select the terrain category with the highest probability value in the step probability distribution vector as the final real-time terrain classification result output.

[0093] The probability distribution vector obtained in step S503 is analyzed, and the terrain category corresponding to the element with the highest probability value is selected. This category is determined as the final real-time terrain classification result for the robot's current location. For example, if the probability vector is [0.02, 0.85, 0.13], corresponding to "concrete", "grass", and "sand" respectively, the output classification result is "grass". This result can be published to the robot's motion planning and control system in real time, serving as a key environmental perception basis for gait adjustment, speed adaptation, or path replanning, thereby completing the closed loop from perception to decision-making.

[0094] The terrain classification system based on SVM hard sample mining and CNN weighted training of the present invention is used to implement the above-mentioned terrain classification method based on SVM hard sample mining and CNN weighted training. It includes a data acquisition and preprocessing module, a sample difficulty assessment module, a training weight generation module, a neural network training module, and a real-time classification module.

[0095] The data acquisition and preprocessing module is used to collect multimodal monitoring data of the robot when it moves on various terrains, preprocess the multimodal monitoring data to generate multimodal feature vectors at different times, and form a training sample set by the multimodal feature vectors at multiple different times. The sample difficulty assessment module is used to construct a support vector machine terrain classification model. The support vector machine terrain classification model is trained using the training sample set, and the distance from each multimodal feature vector in the training sample set to the classification decision is calculated using the trained support vector machine terrain classification model to obtain a quantitative index of classification difficulty. The training weight generation module is used to dynamically calculate the training weight corresponding to each multimodal feature vector based on the classification difficulty quantification index, wherein the training weight is negatively correlated with the classification difficulty quantification index value. The neural network training module is used to construct a convolutional neural network classification model. It uses the training sample set as input and the training weights as weighting factors to construct a weighted loss function to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify. The real-time classification module is used to input the multimodal monitoring data collected by the robot in real time into the trained convolutional neural network classification model to obtain the terrain classification result.

[0096] The electronic device of the present invention includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the above-described method.

[0097] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0098] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A terrain classification method based on SVM hard sample mining and CNN weighted training, characterized in that, Includes the following steps: Multimodal monitoring data of the robot moving on various terrains is collected, the multimodal monitoring data is preprocessed to generate multimodal feature vectors at different times, and a training sample set is composed of multiple multimodal feature vectors at different times. A support vector machine terrain classification model is constructed. The support vector machine terrain classification model is trained using a training sample set. The distance from each multimodal feature vector in the training sample set to the classification decision is calculated using the trained support vector machine terrain classification model to obtain a quantitative index of classification difficulty. Based on the classification difficulty quantification index of each multimodal feature vector, the training weight corresponding to each multimodal feature vector is dynamically calculated, and the training weight is negatively correlated with the classification difficulty quantification index value. A convolutional neural network classification model is constructed. The training sample set is used as input, and the training weights are used as weighting factors. A weighted loss function is constructed to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify. The multimodal monitoring data collected in real time by the robot is input into the trained convolutional neural network classification model to obtain the terrain classification results.

2. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 1, characterized in that, The data collection robot moves across various terrains, generating multimodal monitoring data. This data is preprocessed to generate multimodal feature vectors at different times. A training sample set is then composed of these multimodal feature vectors, specifically including: Control the robot to move on various terrains, and simultaneously collect motion data output by the inertial measurement unit and joint angle data output by the joint encoder; The synchronously acquired motion data and joint angle data are time-stamp aligned and synchronized to obtain synchronized sensor timing data; The synchronized sensor time-series data are subjected to feature extraction to generate inertial measurement unit feature vectors and joint encoder feature vectors. The inertial measurement unit feature vector and the joint encoder feature vector are standardized and then fused to generate a multimodal feature vector corresponding to each sampling time. The multimodal feature vectors from multiple sampling times are collected to form a training sample set.

3. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 1, characterized in that, The construction of the support vector machine terrain classification model involves training the model using a training sample set, and then using the trained model to calculate the distance from each multimodal feature vector in the training sample set to the classification decision, thereby obtaining a quantitative index of classification difficulty. Specifically, this includes: The support vector machine terrain classification model is trained using the training sample set and its corresponding terrain category labels to construct the classification decision hyperplane; Using the decision function in the trained support vector machine terrain classification model, calculate the output value of the decision function for each multimodal feature vector in the training sample set; Based on the decision function output value of each obtained multimodal feature vector, calculate the geometric distance from the corresponding multimodal feature vector to the classification decision hyperplane; The absolute value of the geometric distance is defined as a quantitative index of the classification difficulty of the corresponding multimodal feature vector. The smaller the absolute value of the geometric distance, the higher the classification difficulty of the corresponding multimodal feature vector.

4. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 1, characterized in that, The construction of the convolutional neural network classification model, using the training sample set as input and the training weights as weighting factors, involves constructing a weighted loss function to train the convolutional neural network classification model, specifically including: Construct a convolutional neural network classification model, wherein the input layer of the convolutional neural network classification model is configured to receive multimodal feature vectors; Construct a weighted loss function, which is obtained by combining the standard cross-entropy loss function with the training weights; The convolutional neural network classification model is trained using the training sample set and weighted loss function. During training, the contribution of each multimodal feature vector to the total loss of the convolutional neural network classification model is dynamically adjusted according to the training weights. The multimodal feature vector with larger training weights has a greater impact on the parameter update of the convolutional neural network classification model during backpropagation.

5. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 1, characterized in that, The process of inputting real-time multimodal monitoring data collected by the robot into a trained convolutional neural network classification model to obtain terrain classification results specifically includes: The raw data stream from the multimodal sensors acquired in real time by the robot is preprocessed to generate the real-time multimodal feature vector at the current moment. The generated real-time multimodal feature vectors are input into the already trained convolutional neural network classification model; The convolutional neural network classification model performs forward propagation calculation on the input real-time multimodal feature vector, and obtains the probability distribution vector corresponding to each terrain category at the output of the fully connected classification layer; The terrain category with the highest probability value in the probability distribution vector is selected as the final real-time terrain classification result output.

6. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 1, characterized in that, The weighted loss function Represented as: Weight Represented as: in, It is a standard loss function. It is the first The true label of each sample It is the model's prediction result. It is the first The weights of each sample, It is the total number of training samples. It is the first The distance from each sample to the classification decision boundary. It is a small constant that prevents the denominator from being zero.

7. The terrain classification method based on SVM hard sample mining and CNN weighted training as described in claim 2, characterized in that, The multimodal sensor also includes a microphone. The process of collecting multimodal monitoring data of the robot moving across various terrains, preprocessing the multimodal monitoring data, and generating multimodal feature vectors at different times further includes: The sound signals generated by the robot as it moves across different terrains are collected using a microphone. Mel frequency cepstral coefficients are extracted from the sound signal to obtain the acoustic feature vector; The acoustic feature vector is fused with the feature vectors of the inertial measurement unit and the joint encoder to generate a multimodal feature vector.

8. A terrain classification system based on SVM hard sample mining and CNN weighted training, characterized in that, To implement the terrain classification method based on SVM hard sample mining and CNN weighted training as described in any one of claims 1-7, comprising: The data acquisition and preprocessing module is used to acquire multimodal monitoring data of the robot when it moves on various terrains, preprocess the multimodal monitoring data to generate multimodal feature vectors at different times, and form a training sample set by the multimodal feature vectors at multiple different times. The sample difficulty assessment module is used to construct a support vector machine terrain classification model. The support vector machine terrain classification model is trained using the training sample set, and the distance from each multimodal feature vector in the training sample set to the classification decision is calculated using the trained support vector machine terrain classification model to obtain a quantitative index of classification difficulty. The training weight generation module is used to dynamically calculate the training weight corresponding to each multimodal feature vector based on the classification difficulty quantification index, wherein the training weight is negatively correlated with the classification difficulty quantification index value. The neural network training module is used to construct a convolutional neural network classification model. It uses the training sample set as input and the training weights as weighting factors to construct a weighted loss function to train the convolutional neural network classification model. The weighted loss function is used to adjust the loss contribution of different multimodal feature vectors according to the training weights during the model optimization process, so that the convolutional neural network classification model focuses on learning sample features that are more difficult to classify. The real-time classification module is used to input the multimodal monitoring data collected by the robot in real time into the trained convolutional neural network classification model to obtain the terrain classification result.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-7.

Images