Road surface type recognition method and device, vehicle, and cloud server

Abstract

A road surface type recognition method, comprising: acquiring first detection data, the first detection data comprising a tire vibration acceleration signal; preprocessing the first detection data to acquire a time-frequency transformation diagram corresponding to the first detection data; and on the basis of the time-frequency transformation diagram corresponding to the first detection data, acquiring a road surface type of a road surface to be recognized. The method can implement accurate and rapid recognition of the road surface condition. Further provided are a road surface type recognition device, a vehicle provided with the road surface type recognition device, and a cloud server implementing the road surface type recognition method.

Description

Road surface type identification methods, devices, vehicles, and cloud servers

[0001] This application claims priority to Chinese patent application No. 202410931030.8, filed on July 11, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure belongs to the field of intelligent vehicle technology, and particularly relates to a road surface recognition method, device, vehicle, and cloud server. Background Technology

[0003] The continuous evolution of intelligent driving technology has made accurate identification of road conditions crucial, which not only affects driving safety but also passenger comfort. Summary of the Invention

[0004] This disclosure provides a road surface type identification method, device, vehicle, and cloud server, which can solve the problem of low accuracy in road surface type identification.

[0005] In a first aspect, a road surface type identification method is provided, comprising: acquiring first detection data, the first detection data including: tire vibration acceleration signal; preprocessing the first detection data to obtain a time-frequency transformation diagram corresponding to the first detection data; and obtaining the road surface type of the road surface to be identified based on the time-frequency transformation diagram corresponding to the first detection data.

[0006] In some embodiments, the first detection data further includes: a road surface image; obtaining the road surface type of the road surface to be identified based on the time-frequency transformation map corresponding to the first detection data includes: performing a first fusion process on the road surface image and the time-frequency transformation map to obtain a fused image; and obtaining the road surface type of the road surface to be identified based on the fused image.

[0007] In some embodiments, the first detection data further includes: tire noise signal;

[0008] The step of preprocessing the first detection data to obtain the time-frequency transformation map corresponding to the first detection data includes: preprocessing the tire vibration acceleration signal to obtain a first sub-time-frequency transformation map corresponding to the tire vibration acceleration signal; preprocessing the tire noise signal to obtain a second sub-time-frequency transformation map corresponding to the tire noise signal; and performing a second fusion processing on the first sub-time-frequency transformation map and the second sub-time-frequency transformation map to obtain the time-frequency transformation map corresponding to the first detection data.

[0009] In some embodiments, obtaining the road surface type of the road surface to be identified based on the time-frequency transformation diagram of the first detection data includes: obtaining the road surface type of the road surface to be identified based on the speed information corresponding to the first detection data and the time-frequency transformation diagram.

[0010] In some embodiments, obtaining the road surface type of the road surface to be identified based on the speed information corresponding to the first detection data and the time-frequency transformation diagram includes: determining the target recognition model corresponding to the speed interval where the speed information is located based on the speed information corresponding to the first detection data; inputting the time-frequency transformation diagram into the target recognition model to obtain the road surface type of the road surface to be identified.

[0011] In some embodiments, the tire vibration acceleration signal includes: the tire's vertical vibration acceleration signal.

[0012] In some embodiments, preprocessing the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data includes:

[0013] The first detection data is processed by any of the following methods to obtain the time-frequency transformation diagram corresponding to the first detection data: wavelet transform processing, short-time Fourier transform processing, Cohen class processing, and Wigner-Ville distribution processing.

[0014] In some embodiments, performing a first fusion process on the road surface image and the time-frequency transformation map to obtain a fused image includes: vertically stitching the road surface image and the time-frequency transformation map to obtain the fused image.

[0015] In some embodiments, the second fusion process of the first sub-time-frequency transformation map and the second sub-time-frequency transformation map to obtain the time-frequency transformation map corresponding to the first detection data includes: horizontally stitching the first sub-time-frequency transformation map and the second sub-time-frequency transformation map to obtain the time-frequency transformation map corresponding to the first detection data.

[0016] In some embodiments, the target recognition model is obtained by training a candidate recognition model based on sample data, wherein the sample data includes: detection data samples, speed information, and corresponding road surface type label information.

[0017] In some embodiments, the method further includes: determining detection data samples within the same speed range based on the speed information of the detection data samples; preprocessing the detection data samples to obtain image feature samples corresponding to the detection data samples; and training the candidate recognition model based on the image feature samples to obtain a target recognition model corresponding to the speed range.

[0018] In some embodiments, the detection data samples include: at least two types of detection data samples; the preprocessing of the detection data samples to obtain image feature samples corresponding to the detection data samples includes: preprocessing the at least two types of detection data samples within the same speed range respectively to obtain image feature samples corresponding to the at least two types of detection data respectively; fusing the image feature samples corresponding to the at least two types of detection data respectively to obtain fused image feature samples corresponding to the detection data; the training of the candidate recognition model based on the image feature samples to obtain a target recognition model corresponding to the speed range includes: training the candidate recognition model based on the fused image feature samples to obtain a target recognition model corresponding to the speed range.

[0019] In some embodiments, training the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range includes: performing masking processing on the fused image feature samples to obtain masked fused image feature samples; and training the candidate recognition model based on the masked fused image feature samples to obtain the target recognition model corresponding to the speed range.

[0020] In some embodiments, before training the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range, the method further includes:

[0021] Multiple fused image feature samples are obtained by processing the fused image feature samples in at least one of the following ways:

[0022] Adjust image brightness; adjust image contrast; flip; crop; misaligned stitching or masking.

[0023] In some embodiments, before training the candidate recognition model based on the sample data to obtain the target recognition model, the method further includes: pre-training the initial recognition model based on preset sample data to obtain the candidate recognition model.

[0024] Secondly, a road surface type identification device is provided, comprising: an acquisition module, a processing module, and an identification module.

[0025] The acquisition module is used to acquire first detection data, which includes tire vibration acceleration signals. The processing module is used to preprocess the first detection data to acquire a time-frequency transformation diagram corresponding to the first detection data. The identification module is used to acquire the road surface type of the road surface to be identified based on the time-frequency transformation diagram corresponding to the first detection data.

[0026] In some embodiments, the first detection data further includes: a road surface image; the recognition module is used to perform a first fusion process on the road surface image and the time-frequency transformation map to obtain a fused image; and based on the fused image, obtain the road surface type of the road surface to be identified.

[0027] In some embodiments, the first detection data further includes: a tire noise signal; the processing module is configured to preprocess the tire vibration acceleration signal to obtain a first sub-time-frequency transformation diagram corresponding to the tire vibration acceleration signal; preprocess the tire noise signal to obtain a second sub-time-frequency transformation diagram corresponding to the tire noise signal; and perform a second fusion processing on the first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram to obtain a time-frequency transformation diagram corresponding to the first detection data.

[0028] In some embodiments, the identification module is used to obtain the road surface type of the road surface to be identified based on the speed information corresponding to the first detection data and the time-frequency transformation diagram.

[0029] In some embodiments, the identification module is used to determine the target identification model corresponding to the speed interval where the speed information is located based on the speed information corresponding to the first detection data; and input the time-frequency transformation diagram into the target identification model to obtain the road surface type of the road surface to be identified.

[0030] In some embodiments, the tire vibration acceleration signal includes: the tire's vertical vibration acceleration signal.

[0031] In some embodiments, the processing module is configured to perform any of the following processing on the first detection data to obtain a time-frequency transformation diagram corresponding to the first detection data:

[0032] Wavelet transform processing, short-time Fourier transform processing, Cohen class processing, and Wigner-Ville distribution processing.

[0033] In some embodiments, the processing module is used to vertically stitch the road surface image and the time-frequency transformation image to obtain the fused image.

[0034] In some embodiments, the processing module is used to horizontally stitch the first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram to obtain the time-frequency transformation diagram corresponding to the first detection data.

[0035] In some embodiments, the target recognition model is obtained by training the candidate recognition model based on sample data, wherein the sample data includes: detection data samples, speed information, and corresponding road surface type label information.

[0036] In some embodiments, the road surface type recognition device further includes: a model training module, which is used to determine the detection data samples within the same speed range based on the speed information of the detection data samples; preprocess the detection data samples to obtain image feature samples corresponding to the detection data samples; and train the candidate recognition model based on the image feature samples to obtain the target recognition model corresponding to the speed range.

[0037] In some embodiments, the detection data samples include: at least two types of detection data samples; the model training module is used to preprocess the at least two types of detection data samples within the same speed range to obtain image feature samples corresponding to the at least two types of detection data respectively; to perform fusion processing on the image feature samples corresponding to the at least two types of detection data respectively to obtain fused image feature samples corresponding to the detection data; and to train the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range.

[0038] In some embodiments, the model training module is used to perform masking processing on the fused image feature samples to obtain masked fused image feature samples; and to train the candidate recognition model based on the masked fused image feature samples to obtain the target recognition model corresponding to the speed range.

[0039] In some embodiments, the model training module is further configured to process the fused image feature samples in at least one of the following ways to obtain multiple fused image feature samples:

[0040] Adjust image brightness; adjust image contrast; flip; crop; misaligned stitching or masking.

[0041] In some embodiments, the model training module is further configured to pre-train the initial recognition model based on preset sample data to obtain the candidate recognition model.

[0042] Thirdly, a vehicle is provided, comprising: a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the road surface type identification method according to the first aspect.

[0043] Fourthly, a cloud server is provided, comprising: a processor and a memory, wherein the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the road surface type identification method according to the first aspect.

[0044] Fifthly, some embodiments of the present invention provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the road surface type identification method according to the first aspect.

[0045] In a sixth aspect, some embodiments of the present invention provide a computer program product that, when executed by a processor of a vehicle or a cloud server, implements the road surface type recognition method according to the first aspect.

[0046] Some embodiments of this disclosure acquire first detection data, which includes tire vibration acceleration signals. These tire vibration acceleration signals are unaffected by factors such as light and weather. The tire vibration acceleration signals reflect the road surface condition. The first detection data is preprocessed to obtain a time-frequency transformation diagram corresponding to the first detection data. Since the time-frequency transformation diagram is a two-dimensional feature, it contains richer information reflecting the road surface condition. Therefore, based on the time-frequency transformation diagram corresponding to the first detection data, the road surface type of the road to be identified can be obtained, improving the accuracy of the obtained road surface type. Attached Figure Description

[0047] Figure 1 is a flowchart of a road surface type identification method according to some embodiments;

[0048] Figure 2 is a flowchart of another road surface type identification method according to some embodiments;

[0049] Figure 3 is a schematic diagram of a fused image according to some embodiments;

[0050] Figure 4 is a flowchart of another road surface type identification method according to some embodiments;

[0051] Figure 5 is a schematic diagram of time-frequency transformation according to some embodiments;

[0052] Figure 6 is a schematic diagram of another fused image according to some embodiments;

[0053] Figure 7 is a flowchart of another road surface type identification method according to some embodiments;

[0054] Figure 8 is a flowchart of a model training method for a target recognition model according to some embodiments;

[0055] Figure 9A is a schematic diagram of masking a road surface image according to some embodiments;

[0056] Figure 9B is a schematic diagram of masking the time-frequency transformation diagram corresponding to the tire noise signal according to some embodiments;

[0057] Figure 9C is a schematic diagram of masking the time-frequency transformation diagram corresponding to the tire vibration acceleration signal according to some embodiments;

[0058] Figure 10A is a schematic diagram of a normal fused image according to some embodiments;

[0059] Figure 10B is a schematic diagram of a fused image with reduced brightness and contrast according to some embodiments;

[0060] Figure 10C is a schematic diagram of a fused image with enhanced brightness and contrast according to some embodiments;

[0061] Figure 11 is a block diagram of a road surface type identification device according to some embodiments;

[0062] Figure 12 is a block diagram of a vehicle according to some embodiments;

[0063] Figure 13 is a block diagram of a cloud server according to some embodiments. Detailed Implementation

[0064] The technical solutions of some embodiments of this disclosure will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure are within the scope of protection of this disclosure.

[0065] In related technologies, vehicles are typically equipped with a visual perception system to capture and record visual information about the road surface during driving. By analyzing the captured road surface images, the visual perception system can identify the road surface type and select the appropriate driving mode accordingly.

[0066] However, the visual perception system using this technology is susceptible to environmental factors, such as changes in light intensity and weather conditions, which may affect the clarity of the image and thus reduce the accuracy of road type recognition.

[0067] In order to improve the accuracy of road surface type identification, some embodiments of this disclosure utilize the characteristic that tire vibration acceleration signals are not affected by factors such as light and weather. The road surface condition is reflected through the tire vibration acceleration signals, thereby improving the accuracy of road surface type identification. Here, road surface types include, but are not limited to: dry asphalt, wet asphalt, dry grass, wet grass, dry mud, wet mud, dry sand, wet sand, hard snow, soft snow, etc.

[0068] In some embodiments of this disclosure, to improve the accuracy of road surface type identification, the tire vibration acceleration signal is preprocessed, and road surface type identification is performed based on the time-frequency transformation diagram corresponding to the tire vibration acceleration signal. Since the time-frequency transformation diagram is a two-dimensional feature, it has richer information reflecting the road surface condition, thus improving the accuracy of road surface type identification.

[0069] In some embodiments of this disclosure, to improve the accuracy of road surface type identification, a first fusion process is performed between the time-frequency transformation map and the road surface image to obtain a fused image, and the road surface type of the road surface to be identified is obtained based on the fused image. Since this further reflects the road surface condition information more comprehensively and richly from different perspectives, the accuracy of road surface type identification is further improved.

[0070] In some embodiments of this disclosure, the time-frequency transformation diagrams of some embodiments can be combined with the time-frequency transformation diagrams of tire vibration acceleration signals and tire noise signals, thus enriching the information reflecting road conditions and further improving the accuracy of road type identification.

[0071] Some embodiments of this disclosure combine road surface features collected in different ways through image fusion, avoiding the problem of single feature recognition failure and improving the reliability and accuracy of road surface type recognition. In scenarios where road surface type recognition is performed using a recognition model, the image fusion method is simple, and the use of shallow feature fusion makes the recognition model more generalizable without increasing the complexity of the recognition model.

[0072] The following describes some embodiments of this disclosure.

[0073] Figure 1 is a flowchart of an embodiment of a road surface type identification method according to some embodiments. The method of this embodiment can be executed by a cloud server that communicates with the vehicle, or by the vehicle, or by the cloud server and the vehicle working together. The executing entity of the method of this embodiment can be determined according to the actual application scenario. This disclosure does not limit it. As shown in Figure 1, the method of this embodiment includes S11 to S13.

[0074] S11: Obtain the first detection data.

[0075] Here, the first detection data includes: tire vibration acceleration signal.

[0076] Here, the tire vibration acceleration signal can be understood as the tactile signal of the vehicle to the road surface, which can be acquired by a tactile sensor. The tactile sensor can be, for example, an acceleration sensor (e.g., a six-axis acceleration sensor). In some embodiments, the acceleration sensor can be mounted on the suspension linkage between the vehicle's shock absorbers and the vehicle's wheels to prevent the acquired tire vibration acceleration signal from being attenuated by the shock absorbers.

[0077] In some embodiments, the tire vibration acceleration signal may include vibration acceleration signals in three directions: lateral (X) vibration acceleration signal, longitudinal (Y) vibration acceleration signal, and vertical (Z) vibration acceleration signal. Here, X, Y, and Z are three directions in a three-phase coordinate system.

[0078] In some embodiments, the vertical vibration acceleration signal of the tire can reflect the tire's movement in the vertical direction, and the tire's movement in the vertical direction can reflect the smoothness of the road surface. Therefore, the tire vibration acceleration signal includes: the vertical vibration acceleration signal of the tire.

[0079] S12: Preprocess the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data.

[0080] In some embodiments, the preprocessing of the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data includes, but is not limited to, the following implementation methods:

[0081] Wavelet transform processing;

[0082] Short-time Fourier transform processing;

[0083] Cohen class processing;

[0084] Wigner-Ville Distribution (WVD) processing.

[0085] Through the above preprocessing, the time-frequency transformation diagram corresponding to the first detection data can be obtained.

[0086] S13: Based on the time-frequency transformation diagram corresponding to the first detection data, obtain the road surface type of the road surface to be identified.

[0087] Since the time-frequency transformation map is a two-dimensional feature, it contains richer information reflecting the road surface condition. Therefore, it is more accurate to obtain the road surface type of the road surface to be identified based on the time-frequency transformation map.

[0088] In this embodiment, by acquiring first detection data, which includes tire vibration acceleration signals, the tire vibration acceleration signals are not affected by factors such as light and weather. The tire vibration acceleration signals reflect the road surface condition. The first detection data is preprocessed to obtain a time-frequency transformation map corresponding to the first detection data. Since the time-frequency transformation map is a two-dimensional feature, it has richer information reflecting the road surface condition. Therefore, based on the time-frequency transformation map corresponding to the first detection data, the road surface type of the road surface to be identified can be obtained, which can improve the accuracy of the obtained road surface type.

[0089] Figure 2 is a flowchart of another embodiment of the road surface type identification method according to some embodiments. In some embodiments, based on the embodiment shown in Figure 1, the first detection data further includes: a road surface image, performing a first fusion process on the time-frequency transformation map and the road surface image to obtain a fused image, and obtaining the road surface type of the road surface to be identified based on the fused image. As shown in Figure 2, S13 further includes S131 to S132.

[0090] S131: Perform a first fusion process on the road surface image and the time-frequency transformation image to obtain a fused image.

[0091] Here, the road surface image can be obtained based on an in-vehicle camera. In some embodiments, in order to eliminate the interference of irrelevant information in the road surface image, the road surface image obtained by the in-vehicle camera can be processed. For example, the road surface image obtained by the front-view camera can only be cropped to the lower half of the road surface image, because the lower half of the road surface image mainly contains road surface information.

[0092] In some embodiments, one possible implementation of the first fusion process between the road surface image and the time-frequency transformation image is as follows:

[0093] The road surface image and the time-frequency transformation image are vertically stitched together to obtain the fused image.

[0094] To stitch together a road surface image and a time-frequency transformation image, one method is to perform vertical stitching. Vertical stitching requires maintaining the same width information for both the road surface image and the time-frequency transformation image at the same resolution. For example, if both the road surface image and the time-frequency transformation image have a resolution of 512×256, the resulting fused image after vertical stitching will have a resolution of 512×512. As shown in Figure 3, which is a schematic diagram of a fused image according to some embodiments, the upper half of Figure 3 shows the road surface image, and the lower half shows the time-frequency transformation image.

[0095] S132: Based on the fused image, obtain the road surface type of the road surface to be identified.

[0096] Because the fused image combines the road surface image and the time-frequency transformation map, it provides richer information reflecting the road surface. Therefore, the road surface type obtained based on the fused image is more accurate.

[0097] In this embodiment, by acquiring first detection data, which includes tire vibration acceleration signals and road surface images, and performing a first fusion process on the road surface images and the time-frequency transformation diagrams corresponding to the tire vibration acceleration signals, a fused image is obtained, which enriches the information reflecting the road surface. Therefore, based on the fused image, the road surface type of the road surface to be identified is more accurate.

[0098] Figure 4 is a flowchart of another embodiment of a road surface type identification method according to some embodiments. Based on the embodiment shown in Figure 1 or Figure 2, the first detection data further includes: tire noise signal. As shown in Figure 4, S12 also includes S121 to S123.

[0099] S121: Preprocess the tire vibration acceleration signal to obtain the first sub-time-frequency transformation diagram corresponding to the tire vibration acceleration signal.

[0100] The method for preprocessing the tire vibration acceleration signal can be found in the description of S12 in Figure 1, and will not be repeated here.

[0101] S122: Preprocess the tire noise signal to obtain the second sub-time-frequency transformation diagram corresponding to the tire noise signal.

[0102] Here, tire noise signals can be acquired through acoustic sensors, which can be installed on the suspension near the point where the tire contacts the ground. Since the sounds produced by tires rubbing against different types of ground are different, road surface types can be classified based on the differences in frequency and amplitude of the tire noise signals.

[0103] Preprocessing the tire noise signal to obtain the second sub-time-frequency transform diagram corresponding to the tire noise signal includes, but is not limited to, the following possible implementation methods:

[0104] Wavelet transform processing;

[0105] Short-time Fourier transform processing;

[0106] Cohen class processing;

[0107] Wigner-Ville distribution processing.

[0108] Through the above preprocessing, the second sub-time-frequency transformation diagram corresponding to the tire noise signal can be obtained.

[0109] In some embodiments, before preprocessing the tire noise signal, the control method further includes: removing outliers and filtering, and then segmenting the tire noise signal according to a preset time length, for example, into segments of tire noise signal every 5 seconds, i.e., the interval between two adjacent segments of tire noise signal is 5 seconds. The preprocessing described above is performed on the tire noise signal segments of 5 seconds each.

[0110] S123: Perform a second fusion process on the first sub-time-frequency transformation map and the second sub-time-frequency transformation map to obtain the time-frequency transformation map corresponding to the first detection data.

[0111] In some embodiments, the first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram can be stitched together to obtain the time-frequency transformation diagram corresponding to the first detection data.

[0112] In some embodiments, the first and second sub-time-frequency transform maps can be horizontally stitched together. Horizontal stitching requires that the length information of the resolution of the first and second sub-time-frequency transform maps be consistent. For example, if the resolution of the first sub-time-frequency transform map is 256×256, and the resolution of the second sub-time-frequency transform map is also 256×256, then the resolution of the time-frequency transform map formed after horizontal stitching of the first and second sub-time-frequency transform maps is 512×256. As shown in Figure 5, which is a schematic diagram of a time-frequency transform map according to some embodiments, the right half of Figure 5 represents the first sub-time-frequency transform map, and the left half represents the second sub-time-frequency feature.

[0113] Figure 6 is a schematic diagram of another fused image according to some embodiments. Accordingly, in S131, the road surface image and the time-frequency transformation diagram are subjected to a first fusion process to obtain a fused image. As shown in Figure 6, the fused image is obtained by fusing the first sub-time-frequency transformation diagram corresponding to the tire vibration acceleration signal, the second sub-time-frequency transformation diagram corresponding to the tire noise signal, and the road surface image. The resolution of the fused image is 512×512.

[0114] In this embodiment, by fusing the first sub-time-frequency transformation map corresponding to the tire vibration acceleration signal, the second sub-time-frequency transformation map corresponding to the tire noise signal, and the road surface image, road surface type identification is performed based on the fused image. This enriches the information reflecting the road surface from different perspectives. Therefore, the road surface type obtained based on the fused image is more accurate.

[0115] This disclosure, through multi-feature image fusion, can distinguish more detailed road surface type features, thereby enabling the identification of more detailed road surface types, such as asphalt roads, cement roads, grass roads, sand roads, snow roads, ice roads, gravel roads, dirt roads, mud roads, and wading roads.

[0116] The response characteristics of a vehicle on different road surface types are affected by vehicle speed. For example, the response characteristics of a vehicle traveling at high speed on a relatively smooth road surface are similar to those of a vehicle traveling at low speed on an uneven road surface. Because the vehicle response characteristics of different road surface types are similar at different speeds, in order to further improve the accuracy of road surface type identification, some embodiments of this disclosure also consider speed information when performing road surface type identification. The road surface type of the road surface to be identified is obtained based on the speed information corresponding to the first detection data and the time-frequency transformation diagram, or the speed information and the fused image.

[0117] In some embodiments, road surface type identification can be based on a target recognition model, where the target recognition model corresponds to a speed range. For example, the speed range can be set to (0 km / h, 120 km / h). In some embodiments, the interval between adjacent speed ranges is 10 km / h. Taking the speed range as a unit, 12 target recognition models can be set for each speed range, namely: the target recognition model corresponding to the speed range (0 km / h, 10 km / h), the target recognition model corresponding to the speed range (10 km / h, 20 km / h), the target recognition model corresponding to the speed range (20 km / h, 30 km / h), the target recognition model corresponding to the speed range (30 km / h, 40 km / h), and so on. Target recognition models for speed ranges: (40km / h, 50km / h), (50km / h, 60km / h), (60km / h, 70km / h), (70km / h, 80km / h), (80km / h, 90km / h), (90km / h, 100km / h), (100km / h, 110km / h), and (110km / h, 120km / h).

[0118] In some embodiments, a possible implementation of road surface type identification based on a target recognition model is shown in Figure 7. Figure 7 is a flowchart of another embodiment of a road surface type identification method according to some embodiments. Figure 7 describes a possible implementation of S13 based on the above embodiments. As shown in Figure 7, S13 may also include S131' to S132'.

[0119] S131': Based on the speed information corresponding to the first detection data, determine the target recognition model corresponding to the speed interval where the speed information is located.

[0120] For example, if the speed information corresponding to the first detection data is 66 km / h, and the speed range where the speed information is located is determined to be (60 km / h, 70 km / h], then the target recognition model corresponding to the speed range of (60 km / h, 70 km / h) will be used for road surface type recognition.

[0121] S132': Input the time-frequency transformation diagram into the target recognition model to obtain the road surface type of the road surface to be identified.

[0122] By combining the aforementioned embodiments, inputting the time-frequency transformation diagram into the target recognition model corresponding to the speed range of (60km / h, 70km / h) can yield the road surface type of the road surface to be identified.

[0123] The training weights of the target recognition model are different for different speed ranges. In application, the input of the prediction network can include image input and vehicle speed. The image input can be a time-frequency transformation map or a fused image. The speed range is determined based on the vehicle speed. The training weights of the target recognition model corresponding to the speed range are loaded, and the road surface type recognition result and confidence score are output.

[0124] In this embodiment, by combining speed information and using the target recognition model corresponding to the speed information to identify the road surface type, the accuracy of road surface type identification is further improved.

[0125] The above target recognition model is obtained by training the candidate recognition model based on sample data. The training process of the model will be described below.

[0126] Figure 8 is a flowchart of a model training method for a target recognition model according to some embodiments. As shown in Figure 8, the method of this embodiment includes S81 to S83.

[0127] S81: Based on the speed information of the detection data samples, determine the detection data samples within the same speed range.

[0128] After obtaining the detection data samples, the samples are classified based on their speed information. Detection data samples within the same speed range are grouped into a training sample set to train the candidate recognition model corresponding to that speed range. For example, there are 12 speed ranges, each corresponding to a training sample set.

[0129] In some embodiments, the detection data sample includes at least two types of detection data samples.

[0130] At least two types of detection data samples refer to at least two of the following: detection data samples of tire vibration acceleration signals, detection data samples of tire noise signals, or detection data samples of road surface images.

[0131] S82: Preprocess the detection data samples to obtain the image feature samples corresponding to the detection data samples.

[0132] One possible implementation is as follows:

[0133] The detection data samples are preprocessed to obtain the image feature samples corresponding to the detection data samples. This can be done by referring to the description in the previous embodiments. For example, preprocessing can be performed by wavelet transform, short-time Fourier transform, Cohen class processing, or Wigner-Ville distribution processing to obtain the corresponding image feature samples. The label of the image feature samples can be "speed range and road surface type", for example, "(10km / h, 20km / h] asphalt road".

[0134] In some embodiments, if the detection data samples include at least two types of detection data samples, then the at least two types of detection data samples within the same speed range are preprocessed to obtain image feature samples corresponding to the at least two types of detection data respectively; the image feature samples corresponding to the at least two types of detection data are fused to obtain fused image feature samples corresponding to the detection data.

[0135] Image feature samples corresponding to at least two types of detection data within the same speed range and for the same road surface type are fused to obtain fused image feature samples corresponding to the detection data. The label of the fused image feature samples is the corresponding road surface type. The fusion processing method can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0136] S83: Train the candidate recognition model based on image feature samples to obtain the target recognition model corresponding to the speed range.

[0137] In some embodiments, the candidate recognition model can be built and trained based on a Mobile Vision Transformer (MobileViT) network for mobile devices.

[0138] One possible implementation is to train the corresponding candidate recognition model based on image feature samples within the same speed range until the model converges, or until a preset condition is met, to obtain the target recognition model corresponding to that speed range.

[0139] In some embodiments, if the detection data samples include at least two types of detection data samples, the candidate recognition model is trained based on the fused image feature samples until the model converges, or until a preset condition is met, to obtain the target recognition model corresponding to the speed range. Thus, target recognition models corresponding to different speed ranges can be obtained.

[0140] In this embodiment, by determining the detection data samples within the same speed range based on the speed information of the detection data samples, the detection data samples are preprocessed to obtain the image feature samples corresponding to the detection data samples. The candidate recognition model is then trained based on the image feature samples to obtain the target recognition model corresponding to the speed range. Since the training samples used by this target recognition model are detection data samples within the same speed range, the accuracy of the target recognition model in identifying road surface types is improved.

[0141] In some scenarios, certain information acquisition devices of a vehicle may malfunction. For example, one or two of the acoustic sensors, accelerometers, or cameras may fail, resulting in incomplete information and the fused image lacking information from one or two sensors. To address this situation and improve the reliability and robustness of the model, some embodiments of this disclosure further perform masking processing on the fused image feature samples during model training to obtain masked fused image feature samples. Based on the masked fused image feature samples, the candidate recognition model is trained to obtain the target recognition model corresponding to the speed range.

[0142] Here, the masking process involves setting the pixel values ​​of one or two feature images in the fused image to zero, thereby simulating the fused image obtained when at least one of the multiple information acquisition devices malfunctions. The masked images are shown in Figures 9A, 9B, and 9C. Figures 9A to 9C are schematic diagrams of image masking processing according to some embodiments. Here, Figure 9A is a schematic diagram of masking a road surface image according to some embodiments, Figure 9B is a schematic diagram of masking a time-frequency transformation diagram corresponding to a tire noise signal according to some embodiments, and Figure 9C is a schematic diagram of masking a time-frequency transformation diagram corresponding to a tire vibration acceleration signal according to some embodiments.

[0143] In this embodiment, by masking the fused image feature samples, masked fused image feature samples are obtained. Based on the masked fused image feature samples, the candidate recognition model is trained to obtain the target recognition model corresponding to the speed range, thereby improving the reliability and robustness of the target recognition model.

[0144] During model training, in order to expand the sample capacity, some embodiments of this disclosure also use image enhancement techniques to increase the number of samples. In some embodiments, the fused image feature samples are processed in at least one of the following possible implementations to obtain multiple fused image feature samples:

[0145] Adjust image brightness; adjust image contrast; flip; crop; misaligned stitching or masking.

[0146] For example, a road surface image with a resolution of 512×256 is cropped into two road surface images of 256×256 resolution. Then, the two road surface images and two images processed by wavelet transform are randomly stitched together. The two images processed by wavelet transform can be the time-frequency transformation diagram corresponding to the tire vibration acceleration signal and the time-frequency transformation diagram corresponding to the tire noise signal. The fused image is shown in Figures 10A to 10C. Figures 10A to 10C are schematic diagrams of image enhancement processing of the fused image according to some embodiments. Here, Figure 10A is a schematic diagram of a normal fused image according to some embodiments, Figure 10B is a schematic diagram of a fused image with reduced brightness and contrast according to some embodiments, and Figure 10C is a schematic diagram of a fused image with enhanced brightness and contrast according to some embodiments.

[0147] In this embodiment, by enhancing the image to expand the sample capacity, the number of training samples is increased, thereby improving the accuracy of the target recognition model in identifying road surface types.

[0148] Building upon the above embodiments, to further accelerate the convergence speed of the model, the initial recognition model can be pre-trained using preset samples to obtain candidate recognition models. For example, the ImageNet dataset can be used for pre-training to obtain pre-training weights, which will then be used to obtain the initial training weights for the candidate recognition models. Then, the candidate recognition models for each speed range can be trained based on the sample data corresponding to each speed range to obtain the target recognition model, thereby accelerating the convergence speed of the model.

[0149] Figure 11 is a block diagram of a road surface type identification device according to some embodiments. As shown in Figure 11, the road surface type identification device 1100 of this embodiment includes: an acquisition module 1101, a processing module 1102 and an identification module 1103. Here, the acquisition module 1101 is used to acquire first detection data, which includes: tire vibration acceleration signal.

[0150] The processing module 1102 is used to preprocess the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data.

[0151] The identification module 1103 is used to obtain the road surface type of the road surface to be identified based on the time-frequency transformation diagram corresponding to the first detection data.

[0152] In the above embodiments, the first detection data further includes: road surface image.

[0153] The recognition module 1103 can also be used to perform a first fusion process on the road surface image and the time-frequency transformation image to obtain a fused image; and based on the fused image, obtain the road surface type of the road surface to be recognized.

[0154] In the above embodiments, the first detection data further includes: tire noise signal.

[0155] The processing module 1102 can also be used to preprocess the tire vibration acceleration signal to obtain a first sub-time-frequency transformation diagram corresponding to the tire vibration acceleration signal; preprocess the tire noise signal to obtain a second sub-time-frequency transformation diagram corresponding to the tire noise signal; and perform a second fusion processing on the first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram to obtain a time-frequency transformation diagram corresponding to the first detection data.

[0156] In the above embodiments, the identification module 1103 is used to obtain the road surface type of the road surface to be identified based on the speed information corresponding to the first detection data and the time-frequency transformation diagram.

[0157] In the above embodiment, the identification module 1103 is used to determine the target identification model corresponding to the speed interval where the speed information is located based on the speed information corresponding to the first detection data; and input the time-frequency transformation diagram into the target identification model to obtain the road surface type of the road surface to be identified.

[0158] In the above embodiments, the tire vibration acceleration signal includes: the vertical vibration acceleration signal of the tire.

[0159] In the above embodiments, the processing module 1102 is used to perform any of the following processing on the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data: wavelet transform processing, short-time Fourier transform processing, Cohen class processing, and Wigner-Ville distribution processing.

[0160] In the above embodiment, the processing module 1102 is used to vertically stitch the road surface image and the time-frequency transformation image to obtain the fused image.

[0161] In the above embodiment, the processing module 1102 is used to horizontally stitch the first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram to obtain the time-frequency transformation diagram corresponding to the first detection data.

[0162] In the above embodiments, the target recognition model is obtained by training the candidate recognition model based on sample data, and the sample data includes: detection data samples, speed information and corresponding road surface type label information.

[0163] In the above embodiments, the road surface type recognition device further includes: a model training module, which is used to determine the detection data samples within the same speed range based on the speed information of the detection data samples; preprocess the detection data samples to obtain image feature samples corresponding to the detection data samples; and train the candidate recognition model based on the image feature samples to obtain the target recognition model corresponding to the speed range.

[0164] In the above embodiments, the detection data samples include at least two types of detection data samples.

[0165] The model training module can also be used to preprocess at least two types of detection data samples within the same speed range to obtain image feature samples corresponding to the at least two types of detection data respectively; perform fusion processing on the image feature samples corresponding to the at least two types of detection data to obtain fused image feature samples corresponding to the detection data; and train the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range.

[0166] In the above embodiments, the model training module can also be used to perform masking processing on the fused image feature samples to obtain masked fused image feature samples; and to train the candidate recognition model based on the masked fused image feature samples to obtain the target recognition model corresponding to the speed range.

[0167] In the above embodiments, the model training module is further configured to process the fused image feature samples in at least one of the following ways to obtain multiple fused image feature samples: adjusting image brightness; adjusting image contrast; flipping; cropping; misaligned stitching or masking.

[0168] In the above embodiments, the model training module is further used to pre-train the initial recognition model based on preset sample data to obtain the candidate recognition model.

[0169] The road surface type identification device in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0170] Some embodiments of this disclosure also provide a vehicle 1000, as shown in FIG12. The vehicle 1000 includes a processor 1001 and a memory 1002, the processor 1001 and the memory 1002 being coupled together. The memory 1002 stores programs or instructions that can run on the processor 1001. When the programs or instructions are executed by the processor 1001, they implement the above-described road surface type recognition method.

[0171] Some embodiments of this disclosure also provide a cloud server 2000, as shown in FIG13. The cloud server 2000 includes a processor 2001 and a memory 2002. The memory 2002 stores programs or instructions that can run on the processor 2001. When the programs or instructions are executed by the processor 2001, they implement the above-described road surface type identification method.

[0172] Some embodiments of this disclosure also provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the road surface type identification method as described above.

[0173] Some embodiments of this disclosure also provide a computer program product that, when executed by a processor of a vehicle or a cloud server, implements the steps of the road surface type recognition method embodiments described above.

[0174] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this disclosure is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain embodiments may be combined in other embodiments.

[0175] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this disclosure.

[0176] The embodiments of this disclosure have been described above with reference to the accompanying drawings. However, this disclosure is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this disclosure without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this disclosure.

Claims

1. A road surface type identification method, comprising: Acquire first detection data, which includes: tire vibration acceleration signal; The first detection data is preprocessed to obtain the time-frequency transformation diagram corresponding to the first detection data; Based on the time-frequency transformation diagram corresponding to the first detection data, the road surface type of the road surface to be identified is obtained.

2. The method according to claim 1, wherein, The first detection data also includes: road surface images; The step of obtaining the road surface type of the road surface to be identified based on the time-frequency transformation map corresponding to the first detection data includes: The road surface image and the time-frequency transformation image are subjected to a first fusion process to obtain a fused image; Based on the fused image, the road surface type of the road surface to be identified is obtained.

3. The method according to claim 2, wherein, The first fusion process of the road surface image and the time-frequency transformation image to obtain a fused image includes: The road surface image and the time-frequency transformation image are vertically stitched together to obtain the fused image.

4. The method according to claim 2, wherein, The first detection data also includes: tire noise signal; The step of preprocessing the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data includes: The tire vibration acceleration signal is preprocessed to obtain the first sub-time-frequency transformation diagram corresponding to the tire vibration acceleration signal; The tire noise signal is preprocessed to obtain the second sub-time-frequency transform diagram corresponding to the tire noise signal; A second fusion process is performed on the first sub-time-frequency transformation map and the second sub-time-frequency transformation map to obtain the time-frequency transformation map corresponding to the first detection data.

5. The method according to claim 4, wherein, The second fusion process of the first sub-time-frequency transform map and the second sub-time-frequency transform map to obtain the time-frequency transform map corresponding to the first detection data includes: The first sub-time-frequency transformation diagram and the second sub-time-frequency transformation diagram are horizontally stitched together to obtain the time-frequency transformation diagram corresponding to the first detection data.

6. The method according to claim 1, wherein, The process of obtaining the road surface type of the road surface to be identified based on the time-frequency transformation diagram of the first detection data includes: Based on the speed information corresponding to the first detection data and the time-frequency transformation diagram, the road surface type of the road surface to be identified is obtained.

7. The method according to claim 6, wherein, The step of obtaining the road surface type of the road surface to be identified based on the speed information corresponding to the first detection data and the time-frequency transformation diagram includes: Based on the speed information corresponding to the first detection data, determine the target recognition model corresponding to the speed interval where the speed information is located; The time-frequency transformation diagram is input into the target recognition model to obtain the road surface type of the road surface to be identified.

8. The method according to any one of claims 1 to 7, wherein, The tire vibration acceleration signal includes: the vertical vibration acceleration signal of the tire.

9. The method according to any one of claims 1 to 7, wherein, The step of preprocessing the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data includes: Perform any of the following processing steps on the first detection data to obtain the time-frequency transformation diagram corresponding to the first detection data: Wavelet transform processing, short-time Fourier transform processing, Cohen class processing, and Wigner-Ville distribution processing.

10. The method according to claim 7, wherein, The target recognition model is obtained by training the candidate recognition model based on sample data, which includes: detection data samples, speed information, and corresponding road surface type label information.

11. The method of claim 10, further comprising: Based on the speed information of the detection data samples, the detection data samples within the same speed range are determined; The detection data samples are preprocessed to obtain the image feature samples corresponding to the detection data samples; The candidate recognition model is trained based on the image feature samples to obtain the target recognition model corresponding to the speed range.

12. The method according to claim 11, wherein, The detection data samples include at least two types of detection data samples; The preprocessing of the detection data samples to obtain the image feature samples corresponding to the detection data samples includes: Preprocessing is performed on at least two types of detection data samples within the same speed range to obtain image feature samples corresponding to the at least two types of detection data respectively; The image feature samples corresponding to the at least two types of detection data are fused to obtain the fused image feature samples corresponding to the detection data. The step of training the candidate recognition model based on the image feature samples to obtain the target recognition model corresponding to the speed range includes: The candidate recognition model is trained based on the fused image feature samples to obtain the target recognition model corresponding to the speed range.

13. The method according to claim 12, wherein, The step of training the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range includes: The fused image feature samples are masked to obtain masked fused image feature samples; The candidate recognition model is trained based on the fused image feature samples after the masking process to obtain the target recognition model corresponding to the speed range.

14. The method according to claim 12, wherein, Before training the candidate recognition model based on the fused image feature samples to obtain the target recognition model corresponding to the speed range, the method further includes: Multiple fused image feature samples are obtained by processing the fused image feature samples in at least one of the following ways: Adjust image brightness; adjust image contrast; flip; crop; misaligned stitching or masking.

15. The method according to any one of claims 10 to 14, wherein, Before training the candidate recognition model based on the sample data to obtain the target recognition model, the method further includes: The initial recognition model is pre-trained based on preset sample data to obtain the candidate recognition model.

16. A vehicle comprising: A processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the road surface type identification method according to any one of claims 1 to 15.

17. A cloud server, comprising: A processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the road surface type identification method according to any one of claims 1 to 15.

18. A computer program product, wherein, When the computer program product is executed by the processor of a vehicle or a cloud server, it implements the road surface type identification method according to any one of claims 1 to 15.

Images