Pavement type identification method and equipment based on image and tire noise fusion

Abstract

The invention discloses a road surface type identification method and equipment based on image and tire noise fusion, belongs to the technical field of road surface identification, and aims to solve the problems that the existing road surface type identification technology is not perfect enough, the color, texture, resolution and other features of a road surface are difficult to accurately calculate and identify, and the identification of the road surface type has large defects. And generalization identification of the pavement type is not facilitated. The method comprises the following steps: acquiring a pavement type image data set and an acoustic data set; performing cascade fusion processing on the spectrogram and the road surface type map to obtain a fusion data set; constructing a multi-level neural network model, and training the neural network model to obtain an audio-visual fusion model; performing model training under related vector dimensions on the road surface type recognition model to determine the road surface type recognition model; and carrying out online identification processing on the current fusion data through the pavement type identification model to obtain current pavement type data.

Description

Technical Field

[0001] The present application relates to the technical field of road surface recognition, and in particular to a method and device for road surface type recognition based on the fusion of image and tire noise. Background Art

[0002] Different road types have different effects on intelligent vehicles, because parameters such as adhesion coefficient, sideslip angle and damping coefficient will change according to the road type, and the changes in these parameters can directly affect the adjustment of the tracking control system, suspension control system and anti-lock braking system. Therefore, accurate road type identification is of great significance for vehicle active control and trajectory planning. Existing road type identification methods are generally based on the following five sensors: lidar, millimeter wave radar, ultrasonic radar, visual sensor, and acoustic sensor.

[0003] The road type recognition method using radar (lidar, millimeter wave radar, ultrasonic radar) is an active recognition method. It is to identify the road type by emitting electromagnetic waves or mechanical waves to the road surface according to the different echo intensity of the road surface. Although lidar and millimeter wave radar have high resolution and good recognition effect, they are expensive and have no obvious cost advantage. Ultrasonic radar has good recognition effect on short-distance road surface, but when the road surface is far away, the radar reflection signal is weak and the recognition effect will be significantly reduced. The road type recognition method using visual sensors can collect high-resolution road surface images containing characteristic information such as color and texture, but it is easily affected by changes in lighting conditions, and when the texture features of the collected road surface images are very close, this method is difficult to distinguish the road type, such as gravel road surface and damaged asphalt road surface, asphalt road surface with shallow water accumulation and dry asphalt road surface. The road type identification method using acoustic sensors can collect tire noise through acoustic sensors, and then better identify the road material and determine whether there is water film on the road. Acoustic sensors are cheap and easy to install, but they cannot collect road color and texture information. Moreover, the vehicle perception system can only identify the road type corresponding to the current vehicle position through acoustic sensors, and the identification range is limited.

[0004] In order to further enhance the robustness and accuracy of intelligent vehicles in identifying different road types (asphalt road, snow road, cement road, gravel road, flooded road, etc.), a road type recognition method based on the fusion of image and tire noise is urgently needed to realize the complementary advantages of visual sensors and acoustic sensors in road type recognition, and it can have a lower cost. Summary of the invention

[0005] The embodiments of the present application provide a method and device for identifying road type based on the fusion of image and tire noise, which is used to solve the following technical problems: the existing road type recognition technology is not perfect, and it is difficult to accurately calculate and identify the color, texture, resolution and other characteristics of the road surface. The recognition of road type has major defects and is not conducive to the generalized recognition of road type.

[0006] The present application embodiment adopts the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a road type recognition method based on the fusion of image and tire noise, including: based on a preset camera and an acoustic sensor, respectively performing image acquisition and vehicle tire noise data acquisition for different road types to obtain a road type image data set and an acoustic data set; performing data conversion on the acoustic data set related to a tire noise waveform diagram to obtain a spectrogram data set; and performing cascade fusion processing on the spectrogram in the spectrogram data set and the road type map in the road type image data set to obtain a fused data set; constructing a multi-level neural network model through a preset visual and auditory fusion network, and training the configured neural network model on the fused data set to obtain an audio-visual fusion model; performing model training on a road type recognition model under the relevant vector dimension according to the fused image feature vector data set extracted and reduced by the audio-visual fusion model, and determining a road type recognition model with the highest accuracy; collecting current fused data under the same spatiotemporal dimension, and performing online recognition processing on the current fused data through the road type recognition model to obtain current road type data of the current vehicle driving.

[0008] The embodiment of the present application uses the advantages of visual sensors and acoustic sensors in road type identification, and realizes the complementary information of each sensor through multi-sensor fusion. Through the fusion of visual sensors and acoustic sensors, various characteristic information such as the color, texture and material of the road surface can be accurately obtained, which makes up for the defects when using visual sensors or acoustic sensors alone to identify road surface types. In addition, the present invention converts acoustic data into information-rich two-dimensional images through graphical modeling, extracts image feature vectors based on a targeted convolutional neural network, realizes the automation of feature extraction, and makes full use of the advantage of the ViT model in accurately capturing time relationship features, effectively improving the generalization ability of the road surface type recognition model.

[0009] In a feasible implementation, based on a preset camera and an acoustic sensor, image acquisition and vehicle tire noise data acquisition are performed for different road types to obtain a road type image dataset and an acoustic dataset, specifically including: acquiring an original road information image during vehicle driving through the camera installed at a preset angle; cropping the bottom edge pixel area in the original road information image to obtain a road information image; and normalizing the road information image to obtain a labeled original road type image; wherein the original road type image includes: an asphalt road image, a snow road image, a cement road image, a gravel road image, and a waterlogged road image; performing equalization and enhancement processing on the original road type image through a preset adaptive histogram to obtain a road type image, and generating the road type image dataset.

[0010] In a feasible implementation, the acoustic sensor is pre-installed on the front of the vehicle wheel arch to collect vehicle tire noise data generated when the tire is running on different road types; based on the road type, the vehicle tire noise data is type-labeled to obtain acoustic data, and the acoustic data set is generated.

[0011] In a feasible implementation manner, the acoustic data set is subjected to data conversion related to a tire noise waveform to obtain a spectrogram data set, specifically comprising: performing adaptive filtering processing on the acoustic data set, and performing division processing on the processed acoustic data set under relevant time length and motion sampling to obtain a tire noise waveform; wherein the abscissa in the tire noise waveform is time, and the ordinate is amplitude; based on short-time Fourier transform, converting the tire noise waveform into spectrum data to obtain a spectrogram; wherein the abscissa in the spectrogram is time, and the ordinate is frequency; performing normalization processing related to color images on an image data set corresponding to the spectrogram to obtain the spectrogram data set.

[0012] In a feasible implementation, the spectrogram in the spectrogram dataset and the pavement type map in the pavement type image dataset are cascade-fused to obtain a fused dataset, specifically including: based on the same pavement type scale, extracting the spectrogram in the spectrogram dataset and the pavement type map in the pavement type image dataset that correspond to each other; cascade-fusion processing is performed on the spectrogram and the pavement type map under the same scale characteristics to obtain fused image data; wherein the RGB size of the spectrogram and the pavement type map is 224×224×3, and the RGB size of the fused data image is 224×224×6; based on the fused image data, the spectrogram dataset and the pavement type image dataset are repeatedly fused with each other to generate the fused dataset.

[0013] In a feasible implementation, a multi-level neural network model is constructed through a preset visual and auditory fusion network, specifically including: through the visual and auditory fusion network, the neural network model is configured with multiple layers, including: the first layer of the neural network model is configured as a convolution layer, the first layer parameters are 32 3*3 size convolution kernels, the step size is 1, and the filling method is SAME; and the second layer of the neural network model is configured as a batch normalization layer and the third layer is configured as a nonlinear activation layer, and the second layer and the third layer use ReLU as the activation function; the fourth layer is configured as a random inactivation layer, and the inactivation rate is configured to 0.5; and the fifth layer is configured as an average pooling layer, and the fifth layer parameters are configured as a 5*5 size pooling kernel, and the step size is 2; and the sixth layer is configured as a convolution layer, and the sixth layer parameters are configured as 128 The seventh layer is configured as a batch normalization layer, and the eighth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function; the ninth layer is a random dropout layer, and the dropout rate is configured to 0.5; the tenth layer is configured as a convolution layer, and the tenth layer parameters are configured as 512 3*3 convolution kernels, the step size is 1, and the padding method is SAME; the eleventh layer is configured as a batch normalization layer, and the twelfth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function; the thirteenth layer is configured as a random dropout layer, and the dropout rate is configured to 0.5; the fourteenth layer is configured as a global average pooling layer; and the fifteenth layer is configured as a fully connected layer, and the number of neurons is configured to 5 to correspond to the five road surface types; the sixteenth layer is configured as a Softmax layer.

[0014] In a feasible implementation, after training the configured neural network model on the fusion data set, an audio-visual fusion model is obtained, which specifically includes: dividing the fusion data set into a test data set and a training data set, and performing sample training on the neural network model based on a maximum iteration cycle and a fixed learning rate to obtain the audio-visual fusion model; wherein the maximum iteration cycle is 15, and the fixed learning rate is 10 -3 , the small batch size of samples during training is 32.

[0015] In a feasible implementation, based on the fused image feature vector dataset extracted and reduced in dimension by the audio-visual fusion model, the road type recognition model is trained in the relevant vector dimension to determine the road type recognition model with the highest accuracy, specifically including: extracting all the image feature vectors in the fused dataset through the fourteenth global pooling layer in the audio-visual fusion model; performing dimensionality reduction processing on the image feature vector in terms of cumulative contribution rate to obtain a fused image feature vector, and generating the fused image feature vector dataset; dividing the fused image feature vector dataset into a training dataset and a test dataset; performing dimension correspondence on the fused image feature vector through the temporal position encoding of the vector in the Transformer, and performing model training on the road type recognition model in the relevant vector dimension based on the training dataset and the test dataset to determine the road type recognition model with the highest accuracy.

[0016] In a feasible implementation, current fused data in the same time and space dimensions are collected, and the current fused data are processed online through the road type recognition model to obtain the current road type data of the current vehicle driving, specifically including: aligning the timestamps of the camera and the acoustic sensor pre-installed in the vehicle at the sampling frequency; based on the average driving speed and driving distance of the vehicle, lagging the aligned timestamps in the camera to obtain the lagging timestamps of the camera; collecting the current road type map and the corresponding current spectrogram through the lagging timestamps of the camera and the aligned timestamps of the acoustic sensor; and cascadingly fusing the current road type map and the current spectrogram to obtain the current fused data; inputting the current fused data into the road type recognition model to obtain the current road type data.

[0017] On the other hand, an embodiment of the present application also provides a road type identification device based on the fusion of image and tire noise, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute a road type identification method based on the fusion of image and tire noise described in any of the above embodiments.

[0018] The embodiment of the present application provides a method and device for identifying road type based on the fusion of image and tire noise. Compared with the prior art, the embodiment of the present application has the following beneficial technical effects:

[0019] The embodiment of the present application uses the advantages of visual sensors and acoustic sensors in road type identification, and realizes the complementary information of each sensor through multi-sensor fusion. Through the fusion of visual sensors and acoustic sensors, various characteristic information such as the color, texture and material of the road surface can be accurately obtained, which makes up for the defects when using visual sensors or acoustic sensors alone to identify road surface types. In addition, the present invention converts acoustic data into information-rich two-dimensional images through graphical modeling, extracts image feature vectors based on a targeted convolutional neural network, realizes the automation of feature extraction, and makes full use of the advantage of the ViT model in accurately capturing time relationship features, effectively improving the generalization ability of the road surface type recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A flow chart of a road surface type recognition method based on image and tire noise fusion provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the installation of a camera and a microphone provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of image cropping and normalization provided in an embodiment of the present application;

[0024] Figure 4 A tire noise waveform diagram provided in an embodiment of the present application;

[0025] Figure 5 A tire noise spectrum provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a road surface type recognition model structure provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a road surface type recognition device based on the fusion of image and tire noise provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0029] It should be noted that by analyzing the existing road type recognition methods, the following problems can be found: the road type recognition method using visual sensors can collect high-resolution road images containing characteristic information such as color and texture, but it is easily affected by changes in lighting conditions, and when the texture features of the collected road images are very close, this method is difficult to distinguish the road type, such as gravel road and damaged asphalt road. The road type recognition method using acoustic sensors can collect tire noise through acoustic sensors, and then better identify the road material and determine whether there is water film on the road. Acoustic sensors are cheap and easy to install, but they cannot collect road color and texture information, and the vehicle perception system can only identify the road type corresponding to the current vehicle position through acoustic sensors.

[0030] In order to make up for the shortcomings of these two methods, the present application discloses a road surface type recognition method based on the fusion of image and tire noise. It can give full play to the complementary advantages of visual sensors and acoustic sensors in road surface type recognition, and has an absolute cost advantage compared to the road surface type recognition method of lidar or millimeter wave radar. This method converts the perceptual information extracted from the road surface in the acoustic sensor into a spectrogram, and then fuses the spectrogram with the road surface image collected by the camera to obtain a fused data set. In order to better extract the image feature vector, a model called VAAF-Net (Visual and acoustic fusion network) is constructed at the same time, that is, an audio-visual fusion model for extracting image feature vectors. Compared with other deep learning models, the image feature extraction network can extract road surface features in a targeted manner, and has the advantages of lightweight and high efficiency. Based on VAAF-Net, the fused data set is used for training and extracting multimodal feature information to generate road surface type recognition features fused with multimodal information. The ViT (Vision Transformer) model, i.e., the pavement type recognition model, is also applied to the field of pavement type recognition. A pavement type recognition model is established based on ViT, and the pavement type recognition problem is transformed into a time series prediction problem. Finally, the pavement type recognition is completed through supervised learning.

[0031] Therefore, this application proposes a road type recognition method based on the fusion of image and tire noise, which is mainly used to enhance the robustness and accuracy of intelligent vehicles in identifying different road types (asphalt road, snow road, cement road, gravel road, and flooded road).

[0032] The present application embodiment provides a road surface type recognition method based on the fusion of image and tire noise, such as Figure 1 As shown, the road type recognition method based on the fusion of image and tire noise specifically includes steps S101-S105:

[0033] S101. Based on a preset camera and acoustic sensor, collect images and vehicle tire noise data for different road types to obtain road type image data sets and acoustic data sets.

[0034] Specifically, it is necessary to first collect original road information images during the vehicle's driving process through a camera installed at a preset angle.

[0035] Furthermore, the bottom pixel area in the original road surface information image is cropped to obtain a road surface information image. The road surface information image is normalized to obtain a labeled original road surface type image. The original road surface type image includes: asphalt road surface image, snow road surface image, cement road surface image, gravel road surface image and waterlogged road surface image.

[0036] Furthermore, the original road surface type image is subjected to equalization enhancement processing through a preset adaptive histogram to obtain a road surface type image, and a road surface type image data set is generated.

[0037] In one embodiment, Figure 2 A schematic diagram of the installation of a camera and a microphone provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the vehicle camera needs to be installed. First, the camera is installed above the front windshield of the vehicle. To ensure that the camera captures road information as much as possible, the main optical axis of the camera is tilted to the ground at a fixed angle β. Then, to ensure that the image information obtained is as much as possible from the collected road surface, Figure 3 A schematic diagram of image cropping and normalization provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, 50% of the pixel area is cropped along the bottom edge of the captured image, and the cropped image is normalized into a standard image of 224×224, and each frame of the image is labeled (asphalt road, snow road, cement road, gravel road, flooded road). It is also necessary to enhance the original road type collected: In order to enhance the generalization performance of the model during training and prevent overfitting, adaptive histogram equalization is used to enhance the original data set containing 30,000 images, thereby generating a road type image data set.

[0038] Furthermore, the vehicle tire noise data generated when the tire runs on different road types is collected by pre-installing an acoustic sensor on the front of the vehicle wheel arch.

[0039] Furthermore, based on the road surface type, the vehicle tire noise data is type-labeled to obtain acoustic data and generate an acoustic data set.

[0040] In one embodiment, an acoustic sensor installed on a vehicle is used to collect tire noise generated when the vehicle is driving on different types of roads, and the acoustic data is annotated. That is, after the acoustic sensor is installed, a microphone is installed on the front of the vehicle wheel housing, such as Figure 2 As shown. The acoustic data obtained on different road surfaces are labeled (asphalt road surface, snow road surface, cement road surface, gravel road surface, and flooded road surface), the corresponding labeling processing of vehicle tire noise data and road surface type is completed, the acoustic data is obtained, and an acoustic data set is generated.

[0041] S102, converting the acoustic data set into data related to the tire noise waveform to obtain a spectrogram data set, and cascadingly fusing the spectrogram in the spectrogram data set with the road surface type image in the road surface type image data set to obtain a fused data set.

[0042] Specifically, the acoustic data set is first subjected to adaptive filtering, and the processed acoustic data set is divided according to the time length and the moving sampling to obtain a tire noise waveform diagram, wherein the horizontal axis of the tire noise waveform diagram is time and the vertical axis is amplitude.

[0043] Furthermore, based on the short-time Fourier transform, the tire noise waveform is converted into spectrum data to obtain a spectrogram, wherein the abscissa in the spectrogram is time and the ordinate is frequency.

[0044] Furthermore, the spectrogram corresponding image dataset is normalized with respect to the color image to obtain the spectrogram dataset.

[0045] In one embodiment, Figure 4 A tire noise waveform diagram provided in an embodiment of the present application, such as Figure 4 As shown in the figure, after loading the acoustic data set, in order to eliminate the influence of wind noise and engine noise, an adaptive filtering algorithm is used to denoise the acoustic data set to improve signal quality. Then the data set is divided according to the time length of 1 second, and the step length of the moving sampling is 0.25 seconds. After division, the tire noise waveform is obtained, where the horizontal axis represents time and the vertical axis represents amplitude. Then based on the principle of short-time Fourier transform (STFT), Figure 5 A tire noise spectrum provided in an embodiment of the present application, such as Figure 5The gravel road tire noise spectrogram shown in the figure converts the tire noise waveform into a spectrogram, where the horizontal axis represents time and the vertical axis represents frequency. Finally, the images in the spectrogram dataset are normalized into 224×224×3 color images, totaling 30,000 images.

[0046] Furthermore, based on the same road surface type scale, the spectrograms in the spectrogram dataset and the road surface type images in the road surface type image dataset that correspond to each other are extracted.

[0047] Furthermore, it is necessary to perform cascade fusion processing on the spectrogram and the road type map under the same scale characteristics to obtain fused image data. Among them, the RGB of the spectrogram and the road type map are both 224×224×3, and the RGB of the fused data image is 224×224×6.

[0048] Furthermore, based on the fused image data, the spectrogram dataset and the road surface type image dataset are repeatedly fused with each other to generate a fused dataset.

[0049] As a feasible implementation method, the spectrogram (224×224×3) and the road type map (224×224×3) are fused in a cascade manner to ultimately obtain a fused image of 224×224×6. The operation is repeated to obtain a fused data set. For example, the number of fused images in the fused data set may be 30,000.

[0050] S103, constructing a multi-level neural network model through a preset visual and auditory fusion network, and training the configured neural network model with relevant fusion data sets to obtain an audio-visual fusion model.

[0051] Specifically, Figure 6 A schematic diagram of a road surface type recognition model structure provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the neural network model is configured with multiple layers using the visual and auditory fusion network, including:

[0052] Preferably, the first layer of the neural network model is configured as a convolution layer, the first layer parameters are 32 3*3 convolution kernels, the step size is 1, and the padding method is SAME. The second layer of the neural network model is configured as a batch normalization layer and the third layer is configured as a nonlinear activation layer, and the second and third layers use ReLU as the activation function.

[0053] Preferably, the fourth layer is configured as a random dropout layer, and the dropout rate is configured to 0.5. The fifth layer is configured as an average pooling layer, and the fifth layer parameters are configured as a pooling kernel of 5*5 size, and the step size is 2. The sixth layer is configured as a convolution layer, and the sixth layer parameters are configured as 128 3*3 size convolution kernels, the step size is 1, and the filling method is SAME.

[0054] Preferably, the seventh layer is configured as a batch normalization layer, and the eighth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function. The ninth layer is a random dropout layer, and the dropout rate is configured to 0.5. The tenth layer is configured as a convolution layer, and the parameters of the tenth layer are configured as 512 3*3 size convolution kernels, the step size is 1, and the filling method is SAME.

[0055] Preferably, the eleventh layer is configured as a batch normalization layer, and the twelfth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function. The thirteenth layer is configured as a random inactivation layer, and the inactivation rate is configured as 0.5. The fourteenth layer is configured as a global average pooling layer. The fifteenth layer is configured as a fully connected layer, and the number of neurons is configured as 5 to achieve mutual correspondence with the five road surface types. The sixteenth layer is configured as a Softmax layer.

[0056] Furthermore, the fusion data set is divided into a test data set and a training data set, and the neural network model is trained based on the maximum iteration cycle and a fixed learning rate to obtain an audio-visual fusion model. The maximum iteration cycle is 15 and the fixed learning rate is 10. -3 , the mini-batch size of the sample during training is 32. That is, the Mini-Batch Size is set to 32 and the fixed learning rate is 110 -3 ,The maximum iteration cycle is set to 15. When training the VAAF-Net model (audio-visual fusion model), 20% of the fusion data set is extracted as the training data set, and the remaining 80% is the test data set. After the training, the model is tested using the test data set to obtain the VAAF-Net model.

[0057] S104. Based on the fusion image feature vector data set extracted and reduced in dimension by the audio-visual fusion model, the road surface type recognition model is trained in the relevant vector dimension to determine the road surface type recognition model with the highest accuracy.

[0058] Specifically, the fourteenth global pooling layer in the audio-visual fusion model is used to extract all image feature vectors in the fusion dataset.

[0059] Furthermore, the image feature vector is subjected to dimension reduction processing related to the cumulative contribution rate to obtain a fused image feature vector, and a fused image feature vector data set is generated.

[0060] In one embodiment, principal component analysis (PCA) is used to reduce the dimension of the image feature vector. The PCA algorithm is used to reduce the dimension of the image feature vector, and only the image feature vector with a cumulative contribution rate exceeding 95% is retained. The image feature vector after dimension reduction has a total of 52 dimensions. The image feature vector after dimension reduction is recorded as a fused feature vector. The fused feature vector data set includes: image label, dimension of the fused feature vector, and total number of samples.

[0061] In one embodiment, according to Then we can get the fused feature vector dataset Dataset, where y is the image label, n is the dimension of the fused feature vector, and m is the total number of samples, n=52, m=30000.

[0062] Furthermore, the fused image feature vector data set is divided into a training data set and a test data set.

[0063] Furthermore, the fusion image feature vector is dimensionally mapped through the temporal position encoding of the vector in the Transformer, and the road type recognition model is trained under the relevant vector dimensions based on the training data set and the test data set to determine the road type recognition model with the highest accuracy.

[0064] In one embodiment, the Train-Batch Size is set to 32 and the fixed learning rate is 10. -3 , the maximum iteration cycle is set to 18. Then, the first 80% of the fused image feature vector data set is extracted as the training set, and the remaining 20% ​​is used as the test set. The ViT model is trained using the training set, and the test set is used for verification. The specific steps include:

[0065] (1) Standard Transformer module, the input is a token (vector), each token has a position code representing its temporal order, and the input token dimension is modified to 52 to correspond to the dimension of the fused image feature vector.

[0066] (2) The ViT model is trained using the training set, and the model accuracy is verified using the test set. The training is stopped after the maximum number of rounds. The model with the highest accuracy is retained and used as the ViT road type recognition model.

[0067] S105, collecting current fused data in the same space-time dimension, and performing online recognition processing on the current fused data through a road surface type recognition model to obtain current road surface type data of the current vehicle driving.

[0068] Specifically, firstly, the camera and acoustic sensor pre-installed in the vehicle are controlled to align the timestamps at the sampling frequency.

[0069] Furthermore, based on the average driving speed and driving distance of the vehicle, the aligned timestamps in the camera are delayed to obtain the delayed timestamps of the camera.

[0070] Furthermore, the current road type map and the corresponding current sound spectrogram are collected through the delayed timestamp of the camera and the timestamp after the acoustic sensor is aligned. The current road type map and the current sound spectrogram are cascaded and fused to obtain the current fused data.

[0071] In one embodiment, Figure 2 As shown in the figure, the sampling frequency of the camera is set to 30HZ, the sampling frequency of the acoustic sensor is set to 30kHZ, and the time alignment of the perception data is achieved by using the timestamp alignment method. Since the perception information collected by the camera is ahead of the perception information collected by the microphone, there is a time difference t between the two. 0 , so the image data needs to be delayed by t 0 time to achieve spatial alignment of camera image data and acoustic data, t 0 =s / v, v is t 0 The average speed of the vehicle at time t, s is 0 The vehicle’s driving distance within a certain period of time is calculated by using the camera’s delayed timestamp to collect the current sound spectrogram to achieve the correspondence with the current road type map.

[0072] Furthermore, the current fusion data is input into a road surface type recognition model to obtain current road surface type data, including: asphalt road surface, snow road surface, cement road surface, gravel road surface, flooded road surface, etc.

[0073] In one embodiment, the trained ViT road type recognition model is optimized with an optimizer (TensorRT) to generate an inference engine suitable for running on edge computing devices, and then the inference engine is deployed on NVIDIA Jetson AGX Xavier (high-performance edge computing). It is also necessary to define the communication interface between the camera (camera) and the microphone and the on-board computer. Taking the current fusion data as input, the ViT road type recognition model performs online recognition of the road type based on the perception information, and finally obtains the current road type data.

[0074] In addition, the present application also provides a road type recognition device based on the fusion of image and tire noise, such as Figure 7 As shown, the road type recognition device 700 based on the fusion of image and tire noise specifically includes:

[0075] At least one processor 701. And a memory 702 in communication with the at least one processor 701. The memory 702 stores instructions that can be executed by the at least one processor 701, so that the at least one processor 701 can execute:

[0076] Based on the preset cameras and acoustic sensors, images and vehicle tire noise data are collected for different road types to obtain road type image datasets and acoustic datasets;

[0077] The acoustic data set is converted into data related to the tire noise waveform to obtain a spectrogram data set; and the spectrogram in the spectrogram data set is cascade-fused with the road type image in the road type image data set to obtain a fused data set;

[0078] Through the preset visual and auditory fusion network, a multi-level neural network model is constructed, and the configured neural network model is trained with the relevant fusion data set to obtain an audio-visual fusion model;

[0079] According to the fusion image feature vector data set extracted and reduced by the audio-visual fusion model, the road type recognition model is trained under the relevant vector dimension to determine the road type recognition model with the highest accuracy;

[0080] The current fused data under the same time and space dimensions are collected, and the current fused data is processed online through the road type recognition model to obtain the current road type data of the current vehicle driving.

[0081] The embodiment of the present application uses the advantages of visual sensors and acoustic sensors in road type identification, and realizes the complementary information of each sensor through multi-sensor fusion. Through the fusion of visual sensors and acoustic sensors, various characteristic information such as the color, texture and material of the road surface can be accurately obtained, which makes up for the defects when using visual sensors or acoustic sensors alone to identify road surface types. In addition, the present invention converts acoustic data into information-rich two-dimensional images through graphical modeling, extracts image feature vectors based on a targeted convolutional neural network, realizes the automation of feature extraction, and makes full use of the advantage of the ViT model in accurately capturing time relationship features, effectively improving the generalization ability of the road surface type recognition model.

[0082] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0083] The above describes specific embodiments of the present application. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the scope of the specification of the present application.

Claims

1. A road type recognition method based on the fusion of image and tire noise, It is characterized in that The method comprises: Based on the preset cameras and acoustic sensors, images and vehicle tire noise data are collected for different road types to obtain road type image datasets and acoustic datasets; The acoustic data set is subjected to data conversion related to the tire noise waveform diagram to obtain a spectrogram data set; and the spectrogram in the spectrogram data set is subjected to cascade fusion processing with the road surface type image in the road surface type image data set to obtain a fused data set; A multi-layer neural network model is constructed through a preset visual and auditory fusion network, and the configured neural network model is trained on the fusion data set to obtain an audio-visual fusion model; According to the fusion image feature vector data set extracted and reduced in dimension by the audio-visual fusion model, the road surface type recognition model is trained in the relevant vector dimension to determine the road surface type recognition model with the highest accuracy; The current fused data under the same space-time dimension is collected, and the current fused data is processed online through the road type recognition model to obtain the current road type data of the current vehicle driving.

2. A road type recognition method based on image and tire noise fusion according to claim 1, It is characterized in that Based on the preset cameras and acoustic sensors, we collect images and vehicle tire noise data for different road types, and obtain road type image datasets and acoustic datasets, including: The camera installed at a preset angle is used to collect original road information images during the vehicle's driving process; The bottom edge pixel area in the original road surface information image is cropped to obtain a road surface information image; and the road surface information image is normalized to obtain a labeled original road surface type image; wherein the original road surface type image includes: asphalt road surface image, snow road surface image, cement road surface image, gravel road surface image and waterlogged road surface image; The original road surface type image is subjected to equalization enhancement processing by using a preset adaptive histogram to obtain a road surface type image, and the road surface type image data set is generated.

3. The method for identifying road types based on the fusion of image and tire noise according to claim 2, It is characterized in that The acoustic sensor pre-installed on the front of the vehicle wheel cover is used to collect vehicle tire noise data generated by tire operation on different road types; Based on the road surface type, the vehicle tire noise data is subjected to type labeling processing to obtain acoustic data, and the acoustic data set is generated.

4. The method for identifying road types based on the fusion of image and tire noise according to claim 1, It is characterized in that The acoustic data set is subjected to data conversion related to the tire noise waveform diagram to obtain a spectrogram data set, which specifically includes: Adaptively filter the acoustic data set, and divide the processed acoustic data set into time length and motion sampling to obtain a tire noise waveform; wherein the abscissa of the tire noise waveform is time, and the ordinate is amplitude; Based on short-time Fourier transform, the tire noise waveform is converted into spectrum data to obtain a spectrogram; wherein the abscissa in the spectrogram is time and the ordinate is frequency; The spectrogram corresponding image data set is subjected to normalization processing related to the color image to obtain the spectrogram data set.

5. The method for identifying road types based on the fusion of image and tire noise according to claim 1, It is characterized in that The spectrogram in the spectrogram dataset and the road type image in the road type image dataset are cascaded and fused to obtain a fused dataset, specifically comprising: Based on the same road surface type scale, extracting the sound spectrogram in the sound spectrogram dataset and the road surface type image in the road surface type image dataset that correspond to each other; The spectrogram and the road type map are subjected to cascade fusion processing under the same scale feature to obtain fused image data; wherein the RGB size of the spectrogram and the road type map is 224×224×3, and the RGB size of the fused data image is 224×224×6; Based on the fused image data, the spectrogram dataset and the road type image dataset are repeatedly fused with each other to generate the fused dataset.

6. The method for road type recognition based on image and tire noise fusion according to claim 1, It is characterized in that Through the preset visual and auditory fusion network, a multi-level neural network model is constructed, including: Through the visual and auditory fusion network, the neural network model is configured with multiple layers, including: The first layer of the neural network model is configured as a convolution layer, the first layer parameters are 32 3*3 convolution kernels, the step size is 1, and the padding mode is SAME; and the second layer of the neural network model is configured as a batch normalization layer and the third layer is configured as a nonlinear activation layer, and the second layer and the third layer use ReLU as the activation function; The fourth layer is configured as a random dropout layer with a dropout rate of 0.5; the fifth layer is configured as an average pooling layer with a 5*5 pooling kernel and a step size of 2; the sixth layer is configured as a convolution layer with 128 3*3 convolution kernels, a step size of 1, and a padding mode of SAME; The seventh layer is configured as a batch normalization layer, and the eighth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function; the ninth layer is configured as a random dropout layer, and the dropout rate is configured to 0.5; the tenth layer is configured as a convolution layer, and the parameters of the tenth layer are configured as 512 3*3 convolution kernels, the step size is 1, and the padding method is SAME; The eleventh layer is configured as a batch normalization layer, and the twelfth layer is configured as a nonlinear activation layer, and ReLU is used as the activation function; the thirteenth layer is configured as a random dropout layer, and the dropout rate is configured to 0.5; the fourteenth layer is configured as a global average pooling layer; and the fifteenth layer is configured as a fully connected layer, and the number of neurons is configured to 5 to correspond to the five road types; the sixteenth layer is configured as a Softmax layer.

7. The method for identifying road type based on fusion of image and tire noise according to claim 6, It is characterized in that After training the configured neural network model on the fusion data set, an audio-visual fusion model is obtained, which specifically includes: The fusion data set is divided into a test data set and a training data set, and based on a maximum iteration cycle and a fixed learning rate, the neural network model is trained with samples to obtain the audio-visual fusion model; The maximum iteration period is 15, and the fixed learning rate is 10. -3 , the small batch size of samples during training is 32.

8. The method for road type recognition based on image and tire noise fusion according to claim 1, It is characterized in that According to the fusion image feature vector data set extracted and reduced by the audio-visual fusion model, the road type recognition model is trained under the relevant vector dimension to determine the road type recognition model with the highest accuracy, specifically including: Extracting all image feature vectors in the fusion data set through the fourteenth global pooling layer in the audio-visual fusion model; Performing dimension reduction processing on the image feature vector in terms of cumulative contribution rate to obtain a fused image feature vector, and generating a fused image feature vector data set; The fused image feature vector data set is divided into a training data set and a test data set; The fusion image feature vector is dimensionally matched by encoding the temporal position of the vector in the Transformer, and the road type recognition model is trained under the relevant vector dimension based on the training data set and the test data set to determine the road type recognition model with the highest accuracy.

9. The method for road type recognition based on image and tire noise fusion according to claim 1, It is characterized in that The current fused data under the same time and space dimension is collected, and the current fused data is processed online through the road type recognition model to obtain the current road type data of the current vehicle driving, specifically including: Performing timestamp alignment control on the camera and the acoustic sensor pre-installed in the vehicle at a sampling frequency; Based on the average driving speed and driving distance of the vehicle, the aligned timestamps in the camera are delayed to obtain the delayed timestamps of the camera; The current road surface type map and the corresponding current sound spectrogram are collected through the lagging timestamp of the camera and the timestamp after the acoustic sensor is aligned; and the current road surface type map and the current sound spectrogram are cascade-fused to obtain the current fused data; The current fusion data is input into the road surface type recognition model to obtain the current road surface type data.

10. A road type recognition device based on the fusion of image and tire noise, It is characterized in that The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the road type recognition method based on the fusion of image and tire noise according to any one of claims 1-9.

Images