Road condition detection model training, road condition detection method and device

Abstract

The present disclosure provides a road condition detection model training method, a road condition detection method and device, and relates to the technical field of artificial intelligence. The road condition detection model training method comprises: constructing a first training sample data set; using a first road condition detection model to label the road condition category of the first training sample data set; comprehensively sampling the labeled first training sample data set to obtain a second training sample data set; and training a second road condition detection model according to the second training sample data set. Through the above road condition detection model training method, the performance of the trained road condition detection model can be improved, and the accuracy of the road condition detection result can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a road condition detection model training method and device and a road condition detection method and device. BACKGROUND

[0002] The working environment of an open-pit mine is harsh, and it is of great significance to promote the unmanned operation of the mine truck in the open-pit mine.

[0003] In the unmanned transportation scenario of the open-pit mine, due to the complex mine area road environment and large road surface undulation, the vehicle transportation efficiency is reduced, the tire wears out quickly, and the fuel consumption is high. The mine area road condition detection based on big data analysis is of great significance to solve the above problems. SUMMARY

[0004] The present disclosure provides a road condition detection model training method and device and a road condition detection method and device.

[0005] According to a first aspect of the present disclosure, a road condition detection model training method is provided, comprising: constructing a first training sample data set, the first training sample data set comprising: vehicle driving feature sample data and road environment feature sample data on a road; using a first road condition detection model to label the road condition categories of the first training sample data set; comprehensively sampling the labeled first training sample data set to obtain a second training sample data set; and training a second road condition detection model according to the second training sample data set.

[0006] In some embodiments, the first road condition detection model is an unsupervised model.

[0007] In some embodiments, the comprehensive sampling of the labeled first training sample data set comprises: comprehensively sampling the samples of a specified road condition category in the labeled first training sample data set, wherein the specified road condition category is N road condition categories with the smallest data volume proportion in the labeled first training sample data set, or N road condition categories with a data volume less than a preset threshold, and N is an integer greater than or equal to 1.

[0008] In some embodiments, the comprehensive sampling of the samples of the specified road condition category in the labeled first training sample data set comprises: oversampling the samples of the specified road condition category; and undersampling the oversampled samples.

[0009] In some embodiments, the vehicle driving feature data comprises at least one of vehicle posture, speed, steering angle, angular velocity, and acceleration; and / or the road environment feature data comprises at least one of freezing degree and skidding degree of the road.

[0010] In some embodiments, the constructing the first training sample dataset comprises: collecting multi-sensor data of the vehicle during driving on the road; performing feature data screening and / or feature data construction based on the multi-sensor data to obtain vehicle driving feature sample data and road environment feature sample data; and constructing the first training sample dataset according to the vehicle driving feature sample data and the road environment feature sample data.

[0011] In some embodiments, the multi-sensor data comprises at least one of vehicle pose, speed, steering angle, angular velocity, and acceleration, vehicle load, and at least one of temperature, snowfall, and rainfall.

[0012] In some embodiments, the second road condition detection model comprises a first road condition detection module and a second road condition detection module; the training the second road condition detection model according to the second training sample dataset comprises: inputting the vehicle driving feature sample data in the second training sample dataset into the first road condition detection module to obtain a first road condition category detection result; determining fusion feature data according to the vehicle driving feature sample data and the road environment feature sample data in the second training sample dataset; inputting the fusion feature data into the second road condition detection module to obtain a second road condition category detection result; determining a loss function value according to the first road condition category detection result and the second road condition category detection result; and updating the second road condition detection model according to the loss function value.

[0013] In some embodiments, the first road condition detection module comprises a feature extraction sub-module; and the determining the fusion feature data according to the vehicle driving feature sample data and the road environment feature sample data in the second training sample dataset comprises: processing the vehicle driving feature sample data in the second training sample dataset based on the feature extraction sub-module to obtain intermediate feature data; and splicing the intermediate feature data with the road environment feature sample data to obtain the fusion feature data.

[0014] In some embodiments, the feature extraction sub-module comprises a residual network structure and an attention network structure; and the processing the vehicle driving feature sample data in the second training sample dataset based on the feature extraction sub-module to obtain intermediate feature data comprises: processing the vehicle driving feature sample data based on the residual network structure to obtain processed feature data; processing the vehicle driving feature sample data based on the attention network structure to determine attention weights; and determining the intermediate feature data according to the vehicle driving feature sample data, the processed feature data output by the residual network structure, and the attention weights.

[0015] In some embodiments, the second road condition detection module comprises a feature extraction submodule and a road condition category detection submodule; the inputting of the fusion feature data into the second road condition detection module to obtain a second road condition category detection result comprises: processing the fusion feature data based on the feature extraction submodule to output processed feature data; and classifying the processed feature data based on the road condition category detection submodule to obtain the second road condition category detection result.

[0016] In some embodiments, further comprising: during the use of the second road condition detection model, obtaining corrected road condition category prediction data; and retraining the second road condition detection model based on the corrected road condition category prediction data.

[0017] According to a second aspect of the present disclosure, a road condition detection method is provided, comprising: obtaining feature data of a to-be-detected road, wherein the feature data of the to-be-detected road comprises vehicle driving feature data on the to-be-detected road and environment feature data of the to-be-detected road; and inputting the feature data of the to-be-detected road into a road condition detection model to obtain a road condition category detection result of the to-be-detected road.

[0018] In some embodiments, the road condition detection model is a second road condition detection model trained according to the road condition detection model training method of any of the embodiments of the present disclosure.

[0019] In some embodiments, further comprising: outputting position point information corresponding to an abnormal road condition category in the to-be-detected road; and / or obtaining corrected road condition category prediction data, and retraining the road condition detection model based on the corrected road condition category prediction data.

[0020] According to a third aspect of the present disclosure, a road condition detection model training apparatus is provided, comprising: a construction module configured to construct a first training sample data set, the first training sample data set comprising: vehicle driving feature sample data on a road and road environment feature sample data; a labeling module configured to label road condition categories of the first training sample data set by using a first road condition detection model; a data enhancement module configured to comprehensively sample the labeled first training sample data set to obtain a second training sample data set; and a training module configured to train a second road condition detection model based on the second training sample data set.

[0021] According to a fourth aspect of the present disclosure, a road condition detection device is provided, comprising: an acquisition module configured to acquire feature data of a to-be-detected road, wherein the feature data of the to-be-detected road comprises vehicle driving feature data on the to-be-detected road and environment feature data of the to-be-detected road; and a detection module configured to input the feature data of the to-be-detected road into a road condition detection model to obtain a road condition category detection result of the to-be-detected road.

[0022] According to a fifth aspect of the present disclosure, a road condition detection model training device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a road condition detection model training method as described above based on instructions stored in the memory.

[0023] According to a sixth aspect of the present disclosure, a road condition detection device is also provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a road condition detection method as described above based on instructions stored in the memory.

[0024] According to a seventh aspect of the present disclosure, a computer readable storage medium having computer program instructions stored thereon is also provided, the instructions being executed by a processor to implement a road condition detection model training method or a road condition detection method as described above.

[0025] Other features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] The present disclosure can be more clearly understood with reference to the following detailed description in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A flowchart of a road condition detection model training method according to some embodiments of the present disclosure.

[0029] Figure 2 A flowchart of constructing a first training sample data set according to some embodiments of the present disclosure.

[0030] Figure 3 A flowchart of a comprehensive sampling step according to some embodiments of the present disclosure.

[0031] Figure 4 A flowchart of a model training step according to some embodiments of the present disclosure.

[0032] Figure 5 A structure diagram of a second road condition detection model according to some embodiments of the present disclosure.

[0033] Figure 6 Flowchart of a road condition detection method according to some embodiments of the present disclosure.

[0034] Figure 7 Structural diagram of a road condition detection model training device according to some embodiments of the present disclosure.

[0035] Figure 8 Structural diagram of a road condition detection device according to some embodiments of the present disclosure.

[0036] Figure 9 Structural diagram of a road condition detection model training device or a road condition detection device according to some other embodiments of the present disclosure.

[0037] Figure 10 Structural diagram of a computer system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Various exemplary embodiments of the present disclosure will now be described in detail by referring to the drawings. It should be noted that the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless specifically stated otherwise.

[0039] It should be understood that the sizes of the various portions shown in the drawings are not necessarily drawn to scale for ease of description.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the disclosure and its applications or uses.

[0041] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the described technology.

[0042] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0043] It should be noted that like numbers and letters refer to like elements throughout the several views of the drawings and that the exemplified embodiments can not reflect the patentable scope or claim of the present disclosure.

[0044] For the purpose of making the objects, technical solutions, and advantages of the present disclosure more clear, the present disclosure is further described in detail below by combining with specific embodiments and referring to the drawings.

[0045] In the related art 1, road data is collected by a vehicle sensor, and a deep learning model is used to classify the collected road data to obtain the type of abnormal road surface. This method needs to collect a large amount of abnormal road surface data, and needs manual annotation of the abnormal road surface. In the open-pit mine scene, it is difficult to obtain abnormal road surface data, and manual annotation has high cost and is highly subjective. Therefore, this method is not suitable for the open-pit mine scene.

[0046] In the related art 2, a road condition detection model is trained based on road image data, and subsequent road condition detection is performed. In the open-pit mine scene, there are problems such as a lot of dust. Only relying on road image data to train a road condition detection model and subsequent road condition detection has problems such as poor model training effect and unreliable detection result.

[0047] Moreover, the mine environments in different regions are very different, and it is difficult to directly reuse the model trained based on the data of the conventional road scene to evaluate the road conditions. This further exacerbates the problems such as small scale of abnormal data in the mine, high cost and high subjectivity of manual annotation.

[0048] To solve the above problems, the present disclosure provides a road condition detection model training method, a road condition detection method and a device. By means of automatic data annotation and abnormal data enhancement, the advantages of unsupervised learning and supervised learning are combined to improve the performance of the trained road condition detection model, which is especially suitable for the open-pit mine scene. The road condition detection model trained by the road condition detection model training method based on the embodiments of the present disclosure can evaluate the overall road conditions of the vehicle, find out the poor road sections, assist safety personnel in road condition investigation and vehicle inspection, and assist developers in program testing.

[0049] Figure 1 A flowchart of the road condition detection model training method according to some embodiments of the present disclosure is shown. As shown in FIG. 1, the road condition detection model training method according to some embodiments of the present disclosure includes the following steps. Figure 1

[0050] Step S110: Construct a first training sample data set.

[0051] The first training sample data set includes vehicle driving feature data on the road and road environment feature sample data.

[0052] In some embodiments, the vehicle driving feature data on the road includes at least one of vehicle attitude, speed, steering angle, angular velocity, and acceleration.

[0053] In some embodiments, the road environment feature sample data includes at least one of the freezing degree and the skidding degree of the road.

[0054] ​In some embodiments, the road environment feature sample data is determined according to a vehicle load, a weather condition (such as temperature, rainfall, snowfall) during vehicle driving, and the like.

[0055] In the embodiments of the present disclosure, by training the second road condition detection model based on the vehicle driving feature data on the road and the road environment feature sample data, compared with the related art of only based on road image data or only based on vehicle driving data for road condition detection, the performance of the trained road condition detection model can be improved, and thus the accuracy of road condition detection is improved, which is especially suitable for complex and changeable road environments such as open-pit mines.

[0056] Step S120: performing road condition category labeling on the first training sample data set by using the first road condition detection model.

[0057] In some embodiments, the first road condition detection model is an unsupervised model. In these embodiments, unsupervised anomaly detection is performed on the first training sample data set by using the unsupervised model, and the road condition categories are divided according to the degree of outlying. For example, the road condition categories can be divided into three categories: normal, warning, and abnormal, to complete the labeling of the road condition categories.

[0058] For example, the first road condition detection model is an isolation forest model. The isolation forest model is an anomaly detection algorithm that isolates each anomaly sample by randomly extracting features and constructing multiple binary search trees. Due to the outlying nature of most samples, anomaly values are closer to the root node of the isolated tree. The closer to the root node, the higher the degree of abnormality, and based on the above principle, the road condition categories (such as abnormal, warning, and normal) can be divided.

[0059] In the embodiments of the present disclosure, by step S120, automatic labeling of road condition categories can be achieved, which not only alleviates the problems of high cost and high subjectivity of manual road condition category labeling, but also improves the data labeling efficiency, and thus helps to improve the training efficiency of the road condition detection model.

[0060] Step S130: performing comprehensive sampling on the labeled first training sample data set to obtain a second training sample data set.

[0061] In some embodiments, the samples of the specified road condition categories in the labeled first training sample data set are comprehensively sampled.

[0062] The specified road condition categories are N road condition categories with the smallest data amount proportion in the labeled first training sample data set, or N road condition categories with a data amount less than a preset threshold, where N is an integer greater than or equal to 1.

[0063] For example, the road condition categories in the first training sample data set include three categories of normal, warning and abnormal, and comprehensive sampling is performed on the abnormal category and the warning category in the first training sample data set.

[0064] For example, the road condition categories in the first training sample data set include three categories of normal, warning and abnormal, and comprehensive sampling is performed on the abnormal category and the warning category in the first training sample data set.

[0065] In the embodiments of the present disclosure, data augmentation is realized through step S130, which can expand the data amount of abnormal samples and alleviate the imbalance problem of abnormal samples. Furthermore, the problem that it is difficult to obtain abnormal road condition sample data in an open-pit mine, the amount of abnormal road condition sample data is small, and the model is difficult to train is solved.

[0066] Step S140: training the second road condition detection model according to the second training sample data set.

[0067] In some embodiments, the second road condition detection model is a supervised neural network model. By labeling sample data using an unsupervised neural network model and training the supervised neural network model based on the labeled samples, the advantages of unsupervised learning and supervised learning can be fully combined in the road condition detection of the open-pit mine, thereby helping to improve the accuracy and reliability of the road condition detection result.

[0068] In some embodiments, it further includes: in the process of using the second road condition detection model, obtaining corrected road condition category prediction data; and retraining the second road condition detection model according to the corrected road condition category prediction data.

[0069] For example, after the second road condition detection model performs road condition detection on the to-be-predicted road, the position point information corresponding to the abnormal category in the road condition category detection result is output to a user terminal (such as a user terminal of a road patrol personnel). The patrol personnel investigates according to the abnormal data position point information, corrects the road condition category prediction data according to the investigation result, and uploads the corrected road condition category prediction data. After receiving the corrected road condition category prediction data, the online learning module re-trains the second road condition detection model according to the corrected road condition category prediction data. Through the above processing, the performance of the second road condition detection model can be further optimized, and the accuracy and reliability of the road condition category detection result can be improved.

[0070] In the embodiments of the present disclosure, on one hand, by training the second road condition detection model based on the vehicle driving feature data and the road environment feature sample data, compared with the related art which only uses road image data or only uses vehicle driving data for road condition detection, the performance of the trained road condition detection model can be improved, and thus the accuracy and reliability of the road condition detection can be improved; on the other hand, by automatically labeling the first training sample data set and data enhancement, the data amount of the training sample set, especially the sample data amount of the abnormal road condition category, can be expanded, and thus the performance of the trained road condition detection model can be improved, and thus the accuracy of the road condition detection based on the model can be improved, which is especially suitable for complex road scenes such as open-pit mines.

[0071] Figure 2 A flowchart for constructing the first training sample data set according to some embodiments of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the flow of constructing the first training sample data set according to some embodiments of the present disclosure includes the following steps. Figure 2

[0072] Step S111: Collecting multi-sensor data of a vehicle driving on a road.

[0073] In some embodiments, the multi-sensor data includes real-time time series data and cross-scale time series data. The real-time time series data includes at least one of the vehicle pose, the speed of the vehicle, the steering angle of the vehicle, the angular velocity of the vehicle, and the acceleration of the vehicle. The cross-scale time series data includes at least one of the temperature, the rainfall, the snowfall, and the current vehicle load.

[0074] In some embodiments, the multi-sensor data of the vehicle driving on the road is collected by manually driving the vehicle.

[0075] Step S112: Based on the multi-sensor data, feature data screening and / or feature data construction are performed to obtain vehicle driving feature sample data and road environment feature sample data.

[0076] In some embodiments, for the real-time time series data, feature extraction and selection are performed to filter out key features with high difference. For example, the following feature extraction method can be adopted: a time window is constructed to obtain sequence data with a fixed length, and statistical features, entropy features, segmented features, and event features of each sequence data are counted. After the feature extraction is completed, the following feature selection method can be adopted: features with a variance less than a preset threshold are removed.

[0077] ​In some embodiments, for cross-scale time series data, the freezing degree, the skidding degree and the like of the road surface are constructed according to the temperature, the rainfall, the snowfall and the like. For example, the temperature and the like data of each day are counted, new index data are constructed according to the temperature and the like data, the new index data are normalized, and then are subjected to exponential smoothing processing to obtain the final road environment feature sample data.

[0078] Step S113: constructing a first training sample data set according to the vehicle driving feature sample data and the road environment feature sample data.

[0079] In some embodiments, the vehicle driving feature sample data and the road environment feature sample data are taken as a whole as the first training sample data set

[0080] In the embodiments of the present disclosure, the training sample data set is constructed through the above steps. By including both the vehicle driving feature data on the road and the road environment feature sample data in the training sample data set, the performance of the road condition detection model trained can be improved, and thus the accuracy and reliability of the road condition detection can be improved.

[0081] Figure 3 A flowchart of the comprehensive sampling step according to some embodiments of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the comprehensive sampling step according to some embodiments of the present disclosure includes: Figure 3

[0082] Step S131: oversampling the samples of the specified road condition category.

[0083] In some embodiments, the samples of the specified road condition category are oversampled according to the following formula:

[0084] x new = x + random(0, 1) · |x - x near |

[0085] wherein: x new represents the newly added sample; x represents the sample of the specified road condition category; x near represents the near neighbor sample of the same category as x; and random(0, 1) represents a random function with a value of 0 or 1.

[0086] Step S132: undersampling the oversampled samples.

[0087] In some embodiments, the oversampled samples are undersampled according to the following formula:

[0088]

[0089] wherein: x i represents the sample of the specified category in the first training sample data set; and y​j x represents i The nearest neighbor samples; d(x) i y j ) represents sample x i With sample y j The distance between them; d limit Represents the distance threshold; class(x) i )≠class(y j ) represents sample x i With sample y j For different road condition categories; when sample x i With sample y j When the distance between them is less than the distance threshold, and they belong to different road condition categories, sample x is... i With sample y j Delete; otherwise, remove sample x. i With sample y j reserve.

[0090] In some embodiments, the Euclidean distance formula is used to calculate sample x. i With sample y j The distance between them.

[0091] In this embodiment of the disclosure, by performing oversampling followed by undersampling, sample data that are too similar between different categories can be removed, thereby improving the discriminative power between samples. Comprehensive sampling ensures sample balance, which in turn facilitates the subsequent training process of the second road condition detection model.

[0092] Figure 4 This is a flowchart illustrating the model training steps according to some embodiments of the present disclosure. Figure 4 As shown, the model training steps in some embodiments of this disclosure include:

[0093] Step S141: Input the vehicle driving feature sample data from the second training sample dataset into the first road condition detection module to obtain the first road condition category detection result.

[0094] In this embodiment of the disclosure, the second road condition detection model includes two road condition detection modules, specifically a first road condition detection module and a second road condition detection module.

[0095] In some embodiments, the first road condition detection module includes a first feature extraction submodule and a first road condition category detection submodule. In these embodiments, the first feature extraction submodule extracts features from vehicle driving feature sample data in the second training sample dataset to obtain intermediate feature data; the first road condition category detection submodule processes the intermediate feature data to obtain a first road condition category detection result.

[0096] Step S142: determining the fusion feature data according to the vehicle driving feature sample data in the second training sample data set and the road environment feature sample data.

[0097] In some embodiments, the first road condition detection module includes a first feature extraction sub-module and a first road condition category detection sub-module. In these embodiments, step S142 includes: performing feature extraction on the vehicle driving feature sample data in the second training sample data set based on the first feature extraction sub-module to obtain intermediate feature data; and splicing the intermediate feature data with the road environment feature sample data to obtain the fusion feature data.

[0098] In some embodiments, the first feature extraction sub-module includes a residual network structure and an attention network structure. In these embodiments, step S142 includes: processing the vehicle driving feature sample data based on the residual network structure to obtain processed feature data; processing the vehicle driving feature sample data based on the attention network structure to determine an attention weight; determining the intermediate feature data according to the vehicle driving feature sample data, the processed feature data output by the residual network structure, and the attention weight; and splicing the intermediate feature data with the road environment feature sample data to obtain the fusion feature data.

[0099] In some embodiments, the residual network structure includes a plurality of short-hop residual units. The residual network structure processes the vehicle driving feature sample data according to the following formula:

[0100] x l = relu(BN((W l ·x l-1 +b l )+x l-1 ))

[0101] wherein x l represents the feature data output by the lth short-hop residual unit in the residual network structure; x l-1 represents the feature data output by the (l-1)th short-hop residual unit in the residual network structure; BN() represents batch normalization processing; W l represents a weight variable that is trained and automatically updated during training; b l is a bias coefficient; and relu represents a linear rectifier function, which is a kind of neural network activation function.

[0102] By setting the residual network structure, the model representation ability can be improved, and model degradation can be avoided.

[0103] In some embodiments, the attention network structure compresses and restores the input vehicle driving feature sample data, and finally obtains the attention weight value. The attention weight value is obtained by nonlinear mapping of the initial feature data, and the nonlinear mapping relationship can be obtained by training.

[0104] In some embodiments, the vehicle driving feature sample data is processed according to the following formula:

[0105]

[0106]

[0107] wherein x0[C, 1] represents the input vehicle driving feature sample; relu represents a linear rectification function; W0 represents a weight variable to be trained, which is randomly initialized first and then updated by back propagation during training; C represents the feature dimension of the sample x0; r represents a hyperparameter; represents the compressed feature; a1[C, 1] represents the attention weight; b0 represents a bias coefficient during training; W1 represents a weight variable to be trained; b1 represents a bias coefficient during training; and σ represents a sigmod function, which is a neural network activation function.

[0108] In the embodiments of the present disclosure, by setting the attention network structure, different attention weights can amplify or reduce different features, useful features are amplified, and useless features are reduced, thereby realizing the correction of data.

[0109] In some embodiments, after obtaining the processed feature data output by the residual network structure and the attention weight, the intermediate feature data is determined according to the following formula:

[0110] x out = relu(BN[a1·x li +x0])

[0111] wherein x out represents the intermediate feature data; x li represents the processed feature data output by the residual network structure; a1 represents the attention weight; x0 represents the vehicle driving feature sample data; BN represents batch normalization processing; and relu represents a linear rectification function.

[0112] Step S143: input the fusion feature data into the second road condition detection module to obtain a second road condition category detection result.

[0113] In some embodiments, the second road condition detection module includes a second feature extraction submodule and a second road condition category detection submodule. In these embodiments, the fused feature data is processed based on the second feature extraction submodule to output processed feature data; the processed feature data is then classified based on the second road condition category detection submodule to obtain a second road condition category detection result.

[0114] In some embodiments, the second feature extraction submodule includes a residual network structure.

[0115] Step S144: Determine the loss function value based on the detection results of the first road condition category and the second road condition category.

[0116] The first road condition category detection result includes the predicted probability of the road condition category for multiple sample points, and the second road condition category detection result includes the predicted probability of the road condition category for multiple sample points.

[0117] In some embodiments, step S144 includes: determining a first loss function value based on the first road condition category detection result and the category label carried by the sample data; determining a second loss function value based on the second road condition category detection result and the category label carried by the sample data; and determining a total loss function value based on the first loss function value and the second loss function value.

[0118] Step S145: Update the second road condition detection model based on the loss function value.

[0119] In this embodiment of the disclosure, the above steps enable the training of a second road condition detection model based on real-time time-series data (such as vehicle driving feature sample data) and cross-scale time-series data (such as road environment feature sample data), thereby optimizing the network parameters of the second road condition detection model and improving the performance of the trained second road condition detection model.

[0120] Figure 5 This is a schematic diagram of the structure of a second road condition detection model according to some embodiments of this disclosure. Figure 5 As shown, the second road condition detection model includes: a first road condition detection module 510 and a second road condition detection module 520.

[0121] The first road condition detection module 510 includes: a first feature extraction submodule 511 and a first road condition category detection submodule 512.

[0122] The first feature extraction submodule 511 includes: a residual network structure and an attention network structure.

[0123] In some embodiments, the residual network structure includes two short-jump residual units.

[0124] In some embodiments, each short-hop residual unit processes the input vehicle driving feature sample data according to the following formula:

[0125] x l = relu(BN((W l ·x l-1 +b l )+x l-1 ))

[0126] wherein x l represents the feature data output by the lth short-hop residual unit in the residual network structure; x l-1 represents the feature data output by the (l-1)th short-hop residual unit in the residual network structure; BN represents batch normalization processing; W l represents a weight variable to be trained, which is automatically updated during training; b l is a bias coefficient; and relu represents a linear rectifier function, which is a kind of neural network activation function.

[0127] In some embodiments, the attention network structure processes the input vehicle driving feature sample data according to the following formula:

[0128]

[0129]

[0130] wherein x0[C, 1] represents the input vehicle driving feature sample; relu represents a linear rectifier function; W0 represents a weight variable to be trained, which is randomly initialized at first and then updated through back propagation during training; C represents the feature dimension of the sample x0; r represents a hyperparameter; represents the compressed feature; a1[C, 1] represents the attention weight; b0 represents a bias coefficient during training; W1 represents a weight variable to be trained; b1 represents a bias coefficient during training; and σ represents a sigmod function, which is a kind of neural network activation function.

[0131] In some embodiments, after obtaining the processed feature data output by the residual network structure and the attention weight, the intermediate feature data output by the first feature extraction sub-module 511 is determined according to the following formula:

[0132] x out = relu(BN[a1·x li +x0])

[0133] wherein x out represents the intermediate feature data; and x liThe processed feature data representing the output of the residual network structure; a1 represents the attention weight; x0 represents the vehicle driving feature sample data; BN represents the batch normalization processing; and relu represents the linear rectification function.

[0134] The first road condition category detection submodule 512 is configured to process the input intermediate feature data to obtain a first road condition category detection result.

[0135] The second road condition detection module 520 includes a second feature extraction submodule 521 and a second road condition category detection submodule 522.

[0136] In some embodiments, the intermediate feature data output by the first feature extraction submodule is spliced with the road environment feature sample data, and the spliced feature data is input into the second feature extraction submodule 521 as the fusion feature data.

[0137] The second feature extraction submodule 521 includes a residual network structure.

[0138] In some embodiments, the residual network structure of the second feature extraction submodule 521 includes two short-hop residual units. After the fusion feature data is processed based on the two short-hop residual units, the processed feature data of the second feature extraction submodule 521 is input into the second road condition category detection submodule 522 after a feature connection processing is performed through a long-hop residual.

[0139] The second road condition category detection submodule 522 is configured to process the input feature data to obtain a second road condition category detection result.

[0140] In the embodiments of the present disclosure, by constructing a model including the first road condition detection module and the second road condition detection module, and performing model training and subsequent road condition prediction based on the real-time time sequence feature data and the cross-scale time sequence feature data, the road condition category can be comprehensively evaluated based on multiple data and multiple detection modules, and the accuracy and reliability of the road condition detection result can be improved.

[0141] Figure 6 FIG. 1 is a flowchart of a road condition detection method according to some embodiments of the present disclosure. As shown in FIG. 1, the road condition detection method according to some embodiments of the present disclosure includes the following steps. Figure 6

[0142] Step S610: Obtain feature data of a road to be detected.

[0143] The feature data of the road to be detected includes vehicle driving feature data of the road to be detected and environment feature data of the road to be detected.

[0144] ​In some embodiments, the vehicle driving feature data to be detected for the road entropy comprises at least one of vehicle posture, speed, steering angle, angular velocity, acceleration.

[0145] In some embodiments, the environmental feature data to be detected for the road comprises at least one of freezing degree, skidding degree of the road.

[0146] Step S620: inputting the feature data of the road to be detected into the road condition detection model to obtain a road condition category detection result of the road to be detected.

[0147] In some embodiments, the road condition detection model is a second road condition detection model trained according to the road condition detection model training method of any of the embodiments of the present disclosure.

[0148] In some embodiments, the method further comprises: outputting position point information corresponding to the abnormal road condition category in the road to be detected; and / or, obtaining corrected road condition category prediction data, and re-optimizing the second road condition detection model according to the corrected road condition category prediction data.

[0149] In the embodiments of the present disclosure, on the one hand, by detecting the road condition based on the vehicle driving feature data on the road and the road environmental feature sample data, compared with the related art which only detects the road condition based on road image data or only based on vehicle driving data, the accuracy and reliability of the road condition detection can be improved; on the other hand, by training the road condition detection model by using the above method, the performance of the trained road condition detection model can be improved, and thus the accuracy of the road condition detection based on the model can be improved. The road condition detection method of the embodiments of the present disclosure is especially suitable for complex and changeable road scenes such as open-pit mines.

[0150] Figure 7 FIG. 7 is a structural schematic diagram of a road condition detection model training device according to some embodiments of the present disclosure. As shown in FIG. 7, the road condition detection model training device of the embodiments of the present disclosure comprises a construction module 710, a labeling module 720, a data enhancement module 730, and a training module 740. Figure 7

[0151] The construction module 710 is configured to construct a first training sample data set.

[0152] The first training sample data set comprises vehicle driving feature data on the road and road environmental feature sample data.

[0153] In some embodiments, the vehicle driving feature data on the road comprises at least one of vehicle posture, speed, steering angle, angular velocity, and acceleration.

[0154] In some embodiments, the road environmental feature sample data comprises at least one of freezing degree and skidding degree of the road. ​

[0155] In some embodiments, the road environment feature sample data is determined according to vehicle load, weather conditions (such as temperature, rainfall, snowfall) during vehicle driving, and the like.

[0156] In the embodiments of the present disclosure, by training the second road condition detection model based on the vehicle driving feature data on the road and the road environment feature sample data, compared with the related art of only based on road image data or only based on vehicle driving data for road condition detection, the performance of the road condition detection model trained can be improved, and thus the accuracy of road condition detection is improved, which is especially suitable for complex and changeable road environments such as open-pit mines.

[0157] The labeling module 720 is configured to label the first training sample data set with the first road condition detection model.

[0158] In some embodiments, the first road condition detection model is an unsupervised model. In these embodiments, the first training sample data set is subjected to unsupervised anomaly detection by using the unsupervised model, and the road condition categories are divided according to the degree of outlying. For example, the road condition categories can be divided into three categories: normal, warning, and abnormal, to complete the labeling of the road condition categories.

[0159] For example, the first road condition detection model is an isolation forest model. The isolation forest model is an anomaly detection algorithm that isolates each anomaly sample by randomly extracting features and constructing multiple binary search trees. Due to the outlying nature of most samples, anomaly values are closer to the root node of the isolated tree. The closer to the root node, the higher the degree of abnormality, and based on the above principle, the road condition categories (such as abnormal, warning, and normal) can be divided.

[0160] In the embodiments of the present disclosure, by setting the labeling module, automatic labeling of road condition categories can be achieved, which not only alleviates the problems of high cost and high subjectivity of manual road condition category labeling, but also improves the data labeling efficiency, and thus helps to improve the training efficiency of the road condition detection model.

[0161] The data enhancement module 730 is configured to comprehensively sample the labeled first training sample data set to obtain a second training sample data set.

[0162] In some embodiments, the data enhancement module 730 comprehensively samples the samples of the specified road condition categories in the labeled first training sample data set.

[0163] The specified road condition categories are N road condition categories with the smallest data amount proportion in the labeled first training sample data set, or N road condition categories with a data amount less than a preset threshold, where N is an integer greater than or equal to 1.

[0164] For example, the road condition categories in the first training sample data set include three categories of normal, warning and abnormal, and comprehensive sampling is performed on the abnormal category and the warning category in the first training sample data set.

[0165] For example, the road condition categories in the first training sample data set include three categories of normal, warning and abnormal, and comprehensive sampling is performed on the abnormal category and the warning category in the first training sample data set.

[0166] In the embodiments of the present disclosure, by setting the data enhancement module 730, the data amount of the abnormal sample can be expanded, and the imbalance problem of the abnormal sample can be alleviated. Furthermore, the problem that the abnormal road condition sample data is difficult to obtain, the amount of abnormal road condition sample data is small, and the model is difficult to train is solved.

[0167] The training module 740 is configured to train the second road condition detection model according to the second training sample data set.

[0168] In some embodiments, the second road condition detection model is a supervised neural network model. By labeling the sample data using an unsupervised neural network model and training the supervised neural network model based on the labeled samples, the advantages of unsupervised learning and supervised learning can be fully combined in the road condition detection of the open-pit mine, thereby helping to improve the accuracy and reliability of the road condition detection result.

[0169] In the embodiments of the present disclosure, on the one hand, by training the second road condition detection model based on the vehicle driving feature data and the road environment feature sample data on the road, compared with the related art which only performs road condition detection based on road image data or only based on vehicle driving data, the performance of the trained road condition detection model can be improved, thereby improving the accuracy and reliability of the road condition detection; on the other hand, by automatically labeling the road condition categories of the first training sample data set and data enhancement, the data amount of the training sample set can be expanded, especially the sample data amount of the abnormal road condition category, thereby helping to improve the performance of the trained road condition detection model, thereby improving the accuracy of the road condition detection based on the model, and especially suitable for complex and changeable road scenes such as open-pit mines.

[0170] Figure 8 A structural schematic diagram of a road condition detection device according to some embodiments of the present disclosure is shown. As shown in Figure 8 The road condition detection device of the embodiments of the present disclosure includes an acquisition module 810 and a detection module 820.

[0171] The acquisition module 810 is configured to acquire feature data of a to-be-detected road.

[0172] The feature data of the to-be-detected road includes vehicle driving feature data of the to-be-detected road entropy and environment feature data of the to-be-detected road.

[0173] In some embodiments, the vehicle driving characteristic data of the road entropy to be detected includes at least one of vehicle attitude, speed, steering angle, angular velocity, and acceleration.

[0174] In some embodiments, the environmental characteristic data of the road to be detected includes at least one of the road's degree of freezing and its degree of slippage.

[0175] The detection module 820 is configured to input the feature data of the road to be detected into the road condition detection model in order to obtain the road condition category detection result of the road to be detected.

[0176] In some embodiments, the road condition detection model is a second road condition detection model trained according to the road condition detection model training method of any embodiment of the present disclosure.

[0177] In some embodiments, the method further includes: outputting location point information corresponding to abnormal road condition categories in the road to be detected; and / or, obtaining corrected road condition category prediction data, and further optimizing the second road condition detection model based on the corrected road condition category prediction data.

[0178] In this embodiment, on the one hand, by performing road condition detection based on vehicle driving characteristic data and road environment characteristic sample data, the accuracy and reliability of road condition detection can be improved compared to related technologies that rely solely on road image data or vehicle driving data. On the other hand, by training the road condition detection model using the above method, the performance of the trained road condition detection model can be improved, thereby increasing the accuracy of road condition detection based on the model. The road condition detection method of this embodiment is particularly suitable for scenarios with complex and variable roads, such as open-pit mines.

[0179] Figure 9 This is a block diagram illustrating a road condition detection model training apparatus or a road condition detection apparatus according to other embodiments of the present disclosure.

[0180] like Figure 9 As shown, the road condition detection model training apparatus or road condition detection apparatus 900 includes a memory 910 and a processor 920 coupled to the memory 910. The memory 910 is used to store instructions for executing the road condition detection model training method or road condition detection method corresponding to an embodiment. The processor 920 is configured to execute the road condition detection model training method or road condition detection method in any of the embodiments of this disclosure based on the instructions stored in the memory 910.

[0181] Figure 10 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0182] like Figure 10As shown, computer system 1000 can be in the form of a general- purpose computing device. Computer system 1000 includes memory 1010, processor 1020, and bus 1030 that couples various system components including the memory 1010 to the processor 1020.

[0183] The memory 1010 can include, for example, system memory, non-volatile storage media, and / or the like. The system memory can store, among other things, an operating system, application programs, a Boot Loader, and other programs. The system memory can include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media can store, among other things, instructions to implement at least one of the road condition detection model training method or the road condition detection method.

[0184] The processor 1020 can be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field

[0185] The bus 1030 can use any of a variety of bus structures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus.

[0186] The interfaces 1040, 1050, 1060, the memory 1010, and the processor 1020 of the computer system 1000 can be connected by the bus 1030. The input / output interface 1040 can provide a connection interface for display, mouse, keyboard, and other input / output devices. The network interface 1050 provides a connection interface for various networking devices. The storage interface 1060 provides a connection interface for external storage devices such as floppy disks, U disks, SD cards, and the like.

[0187] Here, various aspects of the disclosure are described with reference to flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks, can be implemented by computer readable program instructions.

[0188] These computer readable program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable device to produce a machine, so that the instructions executed by the processor produce the apparatus that implements the functions specified in one or more blocks of the flowcharts and / or block diagrams.

[0189] These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions

[0190] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both software and hardware aspects.

[0191] Through the road condition detection model training, the road condition detection method and the device in the above embodiment, the performance of the path detection model trained can be improved, and then the accuracy of the road condition detection result is improved.

[0192] Thus far, the road condition detection model training, the road condition detection method and the device according to the present disclosure have been described in detail. In order to avoid obscuring the concepts of the present disclosure, some details well-known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

Claims

1. A method for training a road condition detection model, comprising: Construct a first training sample dataset, which includes: vehicle driving feature sample data and road environment feature sample data on the road; The first road condition detection model is used to label the first training sample dataset with road condition categories. The first training sample dataset after annotation is comprehensively sampled to obtain the second training sample dataset; A second road condition detection model is trained based on a second training sample dataset. The second road condition detection model includes a first road condition detection module and a second road condition detection module. Training the second road condition detection model based on the second training sample dataset includes: inputting vehicle driving feature sample data from the second training sample dataset into the first road condition detection module to obtain a first road condition category detection result; determining fused feature data based on the vehicle driving feature sample data and road environment feature sample data from the second training sample dataset; inputting the fused feature data into the second road condition detection module to obtain a second road condition category detection result; determining a loss function value based on the first and second road condition category detection results; and updating the second road condition detection model based on the loss function value.

2. The road condition detection model training method according to claim 1, wherein, The first road condition detection model is an unsupervised model.

3. The road condition detection model training method according to claim 1, wherein, The comprehensive sampling of the first labeled training sample dataset includes: For samples of a specified road condition category in the first labeled training sample dataset, a comprehensive sampling is performed. The specified road condition category is either the N road condition categories with the smallest data volume in the first labeled training sample dataset, or the N road condition categories with a data volume less than a preset threshold, where N is an integer greater than or equal to 1.

4. The road condition detection model training method according to claim 3, wherein, For the samples of the specified road condition category in the first labeled training sample dataset, comprehensive sampling is performed, including: Oversampling is performed on samples of the specified road condition category; Undersample the oversampled samples.

5. The road condition detection model training method according to claim 1, wherein: The vehicle driving characteristic sample data includes at least one of the following: vehicle attitude, speed, steering angle, angular velocity, and acceleration; and / or The road environment characteristic sample data includes at least one of the following: the degree of road freezing and the degree of slippage.

6. The road condition detection model training method according to claim 1, wherein, The construction of the first training sample dataset includes: Collect multi-sensor data during vehicle operation on the road; Based on the multi-sensor data, feature data filtering and / or feature data construction are performed to obtain vehicle driving feature sample data and road environment feature sample data. A first training sample dataset is constructed based on the vehicle driving feature sample data and the road environment feature sample data.

7. The road condition detection model training method according to claim 6, wherein, The multi-sensor data includes: Vehicle attitude, speed, steering angle, angular velocity, and acceleration, vehicle load, and temperature, snowfall, and rainfall.

8. The road condition detection model training method according to claim 1, wherein, The first road condition detection module includes a feature extraction submodule; The step of determining the fused feature data based on vehicle driving feature sample data and road environment feature sample data in the second training sample dataset includes: Based on the feature extraction submodule, the vehicle driving feature sample data in the second training sample dataset is processed to obtain intermediate feature data; The intermediate feature data is concatenated with the road environment feature sample data to obtain the fused feature data.

9. The road condition detection model training method according to claim 8, wherein, The feature extraction submodule includes a residual network structure and an attention network structure; The step of processing the vehicle driving feature sample data in the second training sample dataset based on the feature extraction submodule to obtain intermediate feature data includes: The vehicle driving feature sample data is processed based on the residual network structure to obtain the processed feature data. The vehicle driving feature sample data is processed based on an attention network structure to determine attention weights; Intermediate feature data are determined based on the vehicle driving feature sample data, the processed feature data output by the residual network structure, and the attention weights.

10. The road condition detection model training method according to claim 1, wherein, The second road condition detection module includes a feature extraction submodule and a road condition category detection submodule; The step of inputting the fused feature data into the second road condition detection module to obtain the second road condition category detection result includes: The fused feature data is processed based on the feature extraction submodule to output the processed feature data; The processed feature data is classified based on the road condition category detection submodule to obtain the second road condition category detection result.

11. The road condition detection model training method according to any one of claims 1 to 6, further comprising: During the use of the second road condition detection model, the corrected road condition category prediction data is obtained; The second road condition detection model is retrained based on the corrected road condition category prediction data.

12. A road condition detection method, comprising: Acquire feature data of the road to be detected, wherein the feature data of the road to be detected includes vehicle driving feature data and environmental feature data of the road to be detected; The feature data of the road to be detected is input into the road condition detection model to obtain the road condition category detection result of the road to be detected, wherein the road condition detection model is a second road condition detection model trained by the road condition detection model training method according to any one of claims 1-11.

13. The road condition detection method according to claim 12 further includes: Output the location information of the abnormal road condition categories in the road to be detected; And / or, Obtain the corrected road condition category prediction data, and retrain the road condition detection model based on the corrected road condition category prediction data.

14. A road condition detection model training device, comprising: The construction module is configured to construct a first training sample dataset, which includes: vehicle driving feature sample data and road environment feature sample data on the road; The annotation module is configured to use the first road condition detection model to annotate the first training sample dataset with road condition categories; The data augmentation module is configured to perform comprehensive sampling on the labeled first training sample dataset to obtain the second training sample dataset; The training module is configured to train a second road condition detection model based on a second training sample dataset. The second road condition detection model includes a first road condition detection module and a second road condition detection module. The training module is configured to: input vehicle driving feature sample data from the second training sample dataset into the first road condition detection module to obtain a first road condition category detection result; determine fused feature data based on the vehicle driving feature sample data and road environment feature sample data from the second training sample dataset; input the fused feature data into the second road condition detection module to obtain a second road condition category detection result; determine a loss function value based on the first and second road condition category detection results; and update the second road condition detection model based on the loss function value.

15. A road condition detection device, comprising: The acquisition module is configured to acquire feature data of the road to be detected, wherein the feature data of the road to be detected includes vehicle driving feature data and environmental feature data of the road to be detected. The detection module is configured to input the feature data of the road to be detected into the road condition detection model to obtain the road condition category detection result of the road to be detected, wherein the road condition detection model is a second road condition detection model trained according to any one of the road condition detection model training methods according to claims 1-11.

16. A road condition detection model training device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the road condition detection model training method as described in any one of claims 1 to 11 based on instructions stored in the memory.

17. A road condition detection device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the road condition detection method as described in any one of claims 12 to 13 based on instructions stored in the memory.

18. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the road condition detection model training method according to any one of claims 1 to 11, or the road condition detection method according to any one of claims 12 to 13.

Images