A Terrain Classification Method Based on Audiovisual Information Fusion
Through the combination of audio-visual information fusion and multi-classifier, visual and sound features are extracted and integrated, and the problems of low accuracy and high cost of terrain classification in the prior art are solved, and efficient and accurate in unstructured environments in the wild Terrain classification enhances the vehicle's terrain perception ability.
Patent Information
- Application Number
- CN202210204930.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-02
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-03-02
AI Technical Summary
The existing terrain classification methods have low accuracy in unstructured environments in the wild, especially based on visual methods, high cost based on tactile methods, low accuracy in classification of single sound signals, and cannot effectively solve the terrain perception of off-road vehicles. need.
Using the terrain classification method based on audio-visual information fusion, the terrain image and sound sensor are obtained through vehicle-mounted cameras to collect sound signals, extract multiple features of images and sound, perform feature fusion and dimensionality reduction, and combine fully connected neural networks and residual networks for classification. Training to realize real-time online terrain classification.
It realizes efficient and accurate terrain classification in complex outdoor environments, with low cost and strong robustness, enhances the terrain perception ability of the vehicle, and is suitable for environments such as poor lighting, with high accuracy.
Smart Images
Figure CN114792375B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a terrain classification method based on audio-visual information fusion, which aims to obtain efficient and accurate terrain classification effects with cheap audio-visual sensors to provide services for the environmental perception capability of vehicles, and belongs to the technical field of unstructured terrain classification. Background Art
[0002] With the rapid development of intelligent driving technology, more and more people are devoted to the research of vehicle terrain perception ability. The classification of terrain by off-road vehicles in complex outdoor environments is a basic issue of vehicle terrain perception ability.
[0003] For the terrain classification problem in the wild unstructured environment, common methods include vision-based methods and touch-based methods. For example, the patent "A terrain classification method for mobile robots based on vision" (publication number: CN108960294B) proposes a terrain classification method based on visual image features. However, the terrain classification method based on vision is greatly affected by lighting conditions, and the classification effect is often not ideal when the light is not good. For example, the patent "An outdoor terrain classification method for wheeled mobile robots" (publication number: CN111598142 A) proposes a method for terrain classification using wheel force signals. Although the terrain classification method based on touch can make up for the problem of changing lighting conditions, the force sensor or vibration sensor adopted is generally expensive. In addition, the terrain classification method based on sound signals has gradually attracted people's attention due to its low cost. However, the accuracy of terrain classification using only sound signals is often not high. Therefore, the present invention proposes a terrain classification method based on audio-visual information fusion, which has low cost, high efficiency, and good robustness. It can perform terrain classification in wild unstructured environments and provide services for vehicle terrain perception. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to overcome the shortcomings of existing terrain classification methods and to propose a method for terrain classification based on audio-visual information, which has the characteristics of effectiveness, low cost and good robustness.
[0005] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is:
[0006] A terrain classification method based on audio-visual information. The invention aims at the terrain classification ability of off-road vehicles in field environments. First, the vehicle is controlled to travel on different terrains, and a forward-facing camera mounted on the bottom of the vehicle is used to obtain terrain images when the vehicle is traveling. At the same time, a sound sensor is used to collect sound signals of the interaction between the vehicle and the ground during the driving process; the visual image and the sound sensor are preprocessed; then, the features of the visual image and the sound signal are extracted respectively. The extracted image features include the color histogram features and local binary features of the image, and the extracted sound signal features include the mean, maximum value, minimum value, and zero crossing times. Eight time domain features, including standard deviation, total variation, number of sign changes and autocorrelation function, six frequency domain features, including average power frequency, median frequency, power sum, power spectrum density, fast Fourier transform coefficient, discrete cosine transform coefficient, and three time-frequency domain features, including peak value, mean value and mean square error value of each layer decomposition coefficient after wavelet decomposition; the visual features are fused with the sound features and feature dimension reduction is performed; then the reduced feature vector and the original image are respectively input into the designed fully connected neural network classifier and residual network classifier for classification training; the final classification decision result is obtained by weighting the output of the two classifiers. After the model is trained, the visual and sound information can be obtained online in real time, and the model can be used for online terrain classification.
[0007] The specific implementation steps include:
[0008] 1) Image and sound information collection.
[0009] The vehicle is controlled to travel on various terrains, and the visual image information of the ground is collected through the vehicle-mounted camera, and the sound signal of the vehicle is collected through the vehicle-mounted sound sensor. The sound signal collection frequency is 100Hz, and the collected data is divided into several sample segments at 1 second intervals. The camera collects image information once per second during the vehicle's driving process.
[0010] 2) Preprocessing of image and sound information.
[0011] For image data, we first downsampled the image to get a 224*224 image to avoid overly complex calculations, and then used the median filter operator to smooth and denoise the image. For sound signals, we normalized each sound vector of the sample segment to a mean of 0 and a standard deviation of 1. The above preprocessed image and sound data were matched one by one to form a sample set.
[0012] 3) Image and sound feature extraction.
[0013] The image after preprocessing is subjected to feature extraction. The image features extracted by this method include color histogram and local binary pattern (LBP) features. Since the HSV color space focuses more on the color, saturation and brightness of the image than the RGB color space, the original image is first converted to the HSV color space, and then the color histogram of the image is extracted as the color feature of the image. The LBP algorithm is used to extract the texture features of the image. The specific calculation method of the LBP value is as follows: First, the original color image is converted into a grayscale image and an array is created to store the LBP value of the image. Then a 3*3 window is created according to the formula Calculate the LBP value, where x c ,y c is the pixel position of the center point, b c Indicates the brightness value of the center point, b i Represents the brightness value of adjacent points, and p=8 is the number of adjacent pixels around the center point. The extracted sound signal features are divided into time domain features, frequency domain features, and time-frequency domain features. The time domain features include mean, maximum, minimum, zero crossing times, standard deviation, total variation, sign change times, and autocorrelation function. The specific standard deviation calculation formula is: Among them, σ v is the standard deviation of the sound signal, N is the number of sampling points, V i represents the sound signal of the i-th sampling point, V mean is the average value of the sound signal, and the total variation is calculated as: The extracted frequency domain features include average power frequency, median frequency, power sum, power spectrum density, fast Fourier transform (FFT) coefficients, and discrete cosine transform (DCT) coefficients. The specific FFT coefficients are obtained by the formula The calculation results show that, where k = 0, 1, 2, ..., N-1, x(i) is the original sound signal, N is the number of sampling points, and X(k) is the coefficient after FFT transformation. The DCT coefficient calculation method is: in, N is the number of sampling points. The extracted time-frequency domain features include the peak value, mean value and mean square error value of each layer of decomposition coefficients after wavelet decomposition. This method uses the db4 wavelet basis to perform wavelet decomposition on the sound signal to obtain three layers of decomposition coefficients, and then obtains the peak value, mean value and mean square error of each layer of decomposition coefficients as time-frequency domain features.
[0014] 4) Feature fusion and dimensionality reduction.
[0015] The features extracted in step 3) are connected in series to form an n-dimensional feature vector X = (x1, x2, x3, ..., x n). After normalizing the matrix composed of the eigenvectors of the sample segments, the matrix DATA is obtained, and the eigencovariance matrix C of DATA is calculated. Then the eigenvalues and eigenvectors of C are calculated and the largest first k vectors are selected to form the eigenvector matrix, where k is the number of eigenvectors selected. Then all sample points are projected onto the selected eigenvectors, the result after dimensionality reduction is obtained and the result after dimensionality reduction is put into the new eigenvector set.
[0016] 5) The new feature vector and the original image after dimension reduction in step 4) are respectively input into the designed fully connected neural network classifier and residual network classifier for training to obtain a classification model, where the output of the classification model uses one-hot vectors [1,0,0,0,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1] to represent the five types of terrain: cement road, silt, sand, gravel and grass. The specific terrain classification model is as follows: Figure 2 As shown in , the input layer size of the designed fully connected network classifier is equal to the input feature vector size, and the output layer size is equal to the number of terrain categories. Figure 3 As shown in the figure, the designed residual network classifier is based on resnet50. Since multi-channel features are generated after the image is input into the convolutional neural network, but the features in each channel and the importance of each channel are different, the network design adds channel attention mechanism and spatial attention mechanism after the first 7*7 convolution of resnet50 and before the final full connection, and adds residual connection to the attention model to better extract image semantic features. The designed spatial attention mechanism and channel attention mechanism are shown in Figure 4 shown.
[0017] 6) Obtain visual and sound information online in real time, execute steps 1) to 4), and use the terrain classification model trained in step 5) to perform online classification and recognition to obtain the terrain category. The final terrain classification result is obtained by weighting the fully connected neural network classifier and the residual network classifier. The decision weight of each classifier is generated by the classification training accuracy, which can be obtained and normalized by each classifier. The specific formula for each classifier weight is: w r represents the weight of the rth classifier, A r Represents the accuracy of the rth classifier during training.
[0018] Beneficial effects:
[0019] The present invention proposes a method for terrain classification based on audio-visual signals, which is suitable for classification and recognition of complex unstructured terrain in the wild.
[0020] The method provided by the present invention can not only effectively perform terrain classification prediction, but also the operation mode of audio-visual information fusion greatly enhances the vehicle's terrain perception ability, making terrain classification robust, avoiding the deficiency of low accuracy of using single information features, and is also suitable for environments such as poor lighting.
[0021] The present invention only uses a camera and a microphone to collect audio-visual information, which is low in cost. In addition, the terrain classification method introduces an attention mechanism and combines multiple classifiers, which has a high accuracy rate and is easy to promote on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 is a flow chart of the present invention;
[0023] Figure 2 is the terrain classification model of the present invention;
[0024] Figure 3 It is the Resnet+attention classifier framework of the present invention;
[0025] Figure 4 It is the attention module. DETAILED DESCRIPTION
[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0027] With the rapid development of intelligent driving technology, more and more people are devoted to the research of vehicle terrain perception ability. The classification of terrain by off-road vehicles in complex outdoor environments is a basic issue of vehicle terrain perception ability.
[0028] The present invention proposes a terrain classification method based on audio-visual information fusion. First, a vehicle-mounted camera and an audio sensor are used to collect visual and audio information when the vehicle is driving; the visual image and the audio sensor are preprocessed; then the features of the visual image and the audio signal are extracted respectively; the visual features and the audio features are fused and feature dimension reduction is performed; then the reduced feature vector and the original image are respectively input into the designed fully connected neural network classifier and the residual network classifier for classification training; the final classification decision result is obtained by weighting the outputs of the two classifiers. After the model is trained, the visual and audio information can be obtained online in real time, and the model can be used for online terrain classification. The present invention proposes a method for terrain classification based on audio-visual signals, which is suitable for classification prediction of complex unstructured terrain in the wild. The method provided by the present invention can not only effectively perform terrain classification prediction, but also the operation mode of audio-visual information fusion greatly enhances the terrain perception ability of the vehicle, makes the terrain classification robust, avoids the deficiency of low accuracy of using a single information feature, and is also suitable for environments such as poor lighting. The present invention only uses a camera and a microphone to collect audio-visual information, which is low in cost. In addition, the terrain classification method introduces an attention mechanism and combines multiple classifiers, which has a high accuracy rate and is easy to promote on a large scale.
[0029] In the implementation method, a camera is used for visual image acquisition. When acquiring image information in each type of terrain environment, the influence of different lighting conditions and weather on subsequent feature extraction must also be considered. Therefore, images should be acquired under different lighting and weather conditions to generate a training set. The sound sensor uses a high-precision (maximum error is less than or equal to 0.5db) and high-frequency (output frequency is greater than or equal to 20Hz) sound sensor. The designed residual network classifier adds an attention mechanism. The input layer size of the fully connected network classifier is equal to the input feature vector size, and the output layer size is equal to the number of terrain categories.
[0030] The specific implementation steps include:
[0031] 1) Image and sound information collection.
[0032] The vehicle is controlled to travel on various terrains, and the visual image information of the ground is collected through the vehicle-mounted camera, and the sound signal of the vehicle is collected through the vehicle-mounted sound sensor. The sound signal collection frequency is 100Hz, and the collected data is divided into several sample segments at 1 second intervals. The camera collects image information once per second during the vehicle's driving process.
[0033] 2) Preprocessing of image and sound information.
[0034] For image data, we first downsampled the image to get a 224*224 image to avoid overly complex calculations, and then used the median filter operator to smooth and denoise the image. For sound signals, we normalized each sound vector of the sample segment to a mean of 0 and a standard deviation of 1. The above preprocessed image and sound data were matched one by one to form a sample set.
[0035] 3) Image and sound feature extraction.
[0036] The image after preprocessing is subjected to feature extraction. The image features extracted by this method include color histogram and local binary pattern (LBP) features. Since the HSV color space focuses more on the color, saturation and brightness of the image than the RGB color space, the original image is first converted to the HSV color space, and then the color histogram of the image is extracted as the color feature of the image. The LBP algorithm is used to extract the texture features of the image. The specific calculation method of the LBP value is as follows: First, the original color image is converted into a grayscale image and an array is created to store the LBP value of the image. Then a 3*3 window is created according to the formula Calculate the LBP value, where x c ,y c is the pixel position of the center point, b c Indicates the brightness value of the center point, b i Represents the brightness value of adjacent points, and p=8 is the number of adjacent pixels around the center point. The extracted sound signal features are divided into time domain features, frequency domain features, and time-frequency domain features. The time domain features include mean, maximum, minimum, zero crossing times, standard deviation, total variation, sign change times, and autocorrelation function. The specific standard deviation calculation formula is: Among them, σ v is the standard deviation of the sound signal, N is the number of sampling points, V i represents the sound signal of the i-th sampling point, V mean is the average value of the sound signal, and the total variation is calculated as: The extracted frequency domain features include average power frequency, median frequency, power sum, power spectrum density, fast Fourier transform (FFT) coefficients, and discrete cosine transform (DCT) coefficients. The specific FFT coefficients are obtained by the formula The calculation results show that, where k = 0, 1, 2, ..., N-1, x(i) is the original sound signal, N is the number of sampling points, and X(k) is the coefficient after FFT transformation. The DCT coefficient calculation method is: in, N is the number of sampling points. The extracted time-frequency domain features include the peak value, mean value and mean square error value of each layer of decomposition coefficients after wavelet decomposition. This method uses the db4 wavelet basis to perform wavelet decomposition on the sound signal to obtain three layers of decomposition coefficients, and then obtains the peak value, mean value and mean square error of each layer of decomposition coefficients as time-frequency domain features.
[0037] 4) Feature fusion and dimensionality reduction.
[0038] The features extracted in step 3) are connected in series to form an n-dimensional feature vector X = (x1, x2, x3, ..., x n ). After normalizing the matrix composed of the eigenvectors of the sample segments, the matrix DATA is obtained, and the eigencovariance matrix C of DATA is calculated. Then the eigenvalues and eigenvectors of C are calculated and the largest first k vectors are selected to form the eigenvector matrix, where k is the number of eigenvectors selected. Then all sample points are projected onto the selected eigenvectors, the result after dimensionality reduction is obtained and the result after dimensionality reduction is put into the new eigenvector set.
[0039] 5) The new feature vector and the original image after dimension reduction in step 4) are respectively input into the designed fully connected neural network classifier and residual network classifier for training to obtain a classification model, where the output of the classification model uses one-hot vectors [1,0,0,0,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1] to represent the five types of terrain: cement road, silt, sand, gravel and grass. The specific terrain classification model is as follows: Figure 2 As shown in , the input layer size of the designed fully connected network classifier is equal to the input feature vector size, and the output layer size is equal to the number of terrain categories. Figure 3 As shown in the figure, the designed residual network classifier is based on resnet50. Since multi-channel features are generated after the image is input into the convolutional neural network, but the features in each channel and the importance of each channel are different, the network design adds channel attention mechanism and spatial attention mechanism after the first 7*7 convolution of resnet50 and before the final full connection, and adds residual connection to the attention model to better extract image semantic features. The designed spatial attention mechanism and channel attention mechanism are shown in Figure 4 shown.
[0040] 6) Obtain visual and sound information online in real time, execute steps 1) to 4), and use the terrain classification model trained in step 5) to perform online classification and recognition to obtain the terrain category. The final terrain classification result is obtained by weighting the fully connected neural network classifier and the residual network classifier. The decision weight of each classifier is generated by the classification training accuracy, which can be obtained and normalized by each classifier. The specific formula for each classifier weight is: w r represents the weight of the rth classifier, A r Represents the accuracy of the rth classifier during training.
Claims
1. A terrain classification method based on audio-visual information fusion, characterized in that: The terrain includes cement road, silt, sand, gravel and grass; firstly, the vehicle is controlled to travel on different terrains, and an on-board camera is used to obtain an image of the terrain when the vehicle is traveling, the camera is mounted on the bottom of the vehicle and points to the ground near the wheels, and at the same time, an acoustic sensor is used to collect the sound signal of the interaction between the vehicle and the ground during driving; the visual image and the sound signal are preprocessed; then, the features of the visual image and the sound signal are extracted respectively, the extracted image features include the color histogram features and the local binary features of the image, and the extracted sound signal features include the mean, maximum value, minimum value, number of zero crossings, standard deviation, total variation, sign variation The eight time-domain features of the number of conversions and autocorrelation function, the six frequency-domain features of the average power frequency, median frequency, power sum, power spectral density, fast Fourier transform coefficient, discrete cosine transform coefficient, and the three time-frequency domain features of the peak value, mean value, and mean square error value of the decomposition coefficients of each layer after wavelet decomposition; the image features are fused with the sound features and feature dimension reduction is performed; then the feature vector after dimension reduction and the original image are respectively input into the designed fully connected neural network classifier and residual network classifier for classification training; the final classification decision result is obtained by weighting the outputs of the two classifiers, the fully connected neural network classifier and the residual network classifier; After the model is trained, it is used to obtain visual and sound information online in real time, and the model is used for online terrain classification; The specific implementation steps include: 1) Image and sound signal acquisition Control the vehicle to drive on different terrains, and use the on-board camera to obtain images of the vehicle while driving. The camera is mounted on the bottom of the vehicle and points to the ground near the wheels. At the same time, the sound sensor is used to collect the sound signal of the interaction between the vehicle and the ground during driving. The sampling frequency of the sound signal is 100Hz, and the camera collects image information of the terrain once per second during the vehicle driving; 2) Image and sound information preprocessing For the collected image data, first downsample to obtain a 224*224 image to avoid overly complex calculations. Since the images collected by the camera inevitably contain noise and redundant information, the downsampled images are smoothed and denoised using a median filter operator. For sound signals, each sound vector of the sample segment is normalized to a mean of 0 and a standard deviation of 1, and the above preprocessed image and sound data are matched one by one to form a sample set. 3) Image and sound feature extraction The image features extracted by this method include color histogram and local binary pattern features. Since the HSV color space focuses more on the color, saturation and brightness of the image than the RGB color space, the original image is first converted to the HSV color space, and then the color histogram of the image is extracted as the color feature of the image. The LBP algorithm is used to extract the texture feature of the image. The specific calculation method of the LBP value is as follows: First, the original color image is converted to a grayscale image and an array is created to store the LBP value of the image. Then a 3*3 window is created according to the formula Calculate the LBP value, where x c ,y c is the pixel position of the center point, b c Indicates the brightness value of the center point, b i represents the brightness value of adjacent points, and p=8 is the number of adjacent pixels around the center point; the extracted sound signal features are divided into time domain features, frequency domain features and time-frequency domain features, among which the time domain features include mean, maximum, minimum, number of zero crossings, standard deviation, total variation, number of sign changes and autocorrelation function; The specific formula for calculating the standard deviation is: Among them, σ v is the standard deviation of the sound signal, N is the number of sampling points, V i represents the sound signal of the i-th sampling point, V mean is the average value of the sound signal; The calculation method of total variation δ is: V i-1 Represents the sound signal of the i-1th sampling point; The extracted frequency domain features include average power frequency, median frequency, power sum, power spectral density, fast Fourier transform coefficients, and discrete cosine transform coefficients; The specific calculation formula of FFT coefficients is: Where k = 0, 1, 2, ..., N-1, x(i) is the original sound signal, and X(k) is the coefficient after FFT transformation; The DCT coefficient calculation method is: in, The extracted time-frequency domain features include the peak value, mean value and mean square error value of each layer of decomposition coefficients after wavelet decomposition. This method uses the db4 wavelet basis to perform wavelet decomposition on the sound signal to obtain three layers of decomposition coefficients, and then obtains the peak value, mean value and mean square error of each layer of decomposition coefficients as time-frequency domain features; 4) Feature fusion and dimensionality reduction The features extracted in step 3) are connected in series to form an n-dimensional feature vector X = (x1, x2, x3, ..., x n ), standardize the matrix composed of the eigenvectors of the sample segments to obtain the matrix DATA, calculate the characteristic covariance matrix C of DATA, then calculate the eigenvalues and eigenvectors of C and select the largest first k vectors to form the eigenvector matrix, k is the number of eigenvectors selected, then project all sample points onto the selected eigenvectors, obtain the result after dimensionality reduction and put the result after dimensionality reduction into the new eigenvector set; 5) The new feature vector after dimensionality reduction in step 4) and the original image are respectively input into the designed fully connected neural network classifier and residual network classifier for training to obtain a classification model, wherein the output of the classification model uses one-hot vectors [1,0,0,0,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1] to represent the five types of terrain: cement road, silt, sand, gravel and grass; the input layer size of the designed fully connected network classifier is equal to the input feature vector size, and the output layer size is equal to the number of terrain categories; the designed residual network classifier is based on resnet50; since multi-channel features will be generated after the image is input into the convolutional neural network, but the features in each channel and the importance of each channel are different, the network design adds channel attention mechanism and spatial attention mechanism after the first 7*7 convolution of resnet50 and before the last full connection, and adds residual connection to the attention model to better extract image semantic features; 6) Obtain visual and sound information online in real time, execute steps 1) to 4), and use the terrain classification model trained in step 5) to perform online classification and recognition to obtain the terrain category, where the final terrain classification result is obtained by weighting the fully connected neural network classifier and the residual network classifier. The decision weight of each classifier is generated by the classification training accuracy, which can be obtained and normalized by each classifier. The specific formula for the weight of each classifier is: w r represents the weight of the rth classifier, A r Represents the accuracy of the rth classifier during training.