Decision-level fusion identification method for road surface damage based on machine learning confusion matrix

Abstract

The present invention discloses a decision-level fusion identification method for road surface damage based on machine learning confusion matrix, which relates to the field of road detection technology, including designing a set of road surface vibration data and image data acquisition devices, and proposing a data matching method; constructing a road surface vibration data set and an image data set, respectively training a vibration identification model and an image identification model for road surface damage; based on the confusion matrix of each road surface damage type obtained by the model, calculating the probability distribution model of the actual road section identification result; according to the given fusion identification formula, weighted fusion is performed on the vibration identification result and the image identification result to obtain a fusion identification result that is closer to the actual situation. The method of the present invention provides a specific data matching method to ensure the temporal and spatial consistency of vibration data and image data, and adopts a decision-level fusion method to more accurately count the damage of the target road section, which is applicable to a variety of road conditions and has good practicality and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of road detection, and in particular to a road surface damage decision-level fusion identification method based on a machine learning confusion matrix. Background Art

[0002] With the rapid development of my country's road infrastructure, more and more road sections have entered the maintenance era. Road condition assessment is the core link of road maintenance, which is of great significance for ensuring road safety, improving traffic efficiency and extending the service life of roads. In order to ensure that the assessment results truly reflect the actual situation of the road surface, an accurate and reliable road surface damage identification method is needed. At present, there are three main methods for road surface damage identification: image recognition method, vehicle-mounted vibration method and multi-sensor fusion method.

[0003] The image recognition method uses a camera to collect road surface images, and then uses image processing and analysis and other related technologies to intuitively identify cracks, potholes, patches and other road surface damage. It has been widely used in the field of road surface detection. The vehicle-mounted vibration method uses a vibration sensor to monitor the vibration of the vehicle during driving. When the detection vehicle passes through different types of road surface damage, different vibration characteristics will be generated to identify the road surface damage. The multi-sensor fusion method uses two or more sensors (such as cameras, vibration sensors, laser scanners, etc.) to collect road surface damage information, comprehensively utilizing the advantages of different types of sensors to improve the accuracy and reliability of road surface damage identification; for example, the fusion of image and vibration information can realize detection under different vehicle speeds and different lighting conditions; the fusion of two-dimensional images and three-dimensional images can avoid the impact of uneven lighting.

[0004] However, although the multi-sensor fusion method can improve the accuracy of identification and expand its scope of application, it still faces great challenges in practical application. The main difficulty lies in how to effectively fuse and analyze data from different types of sensors. The data collected by different sensors have great differences in spatiotemporal scale, data format, information redundancy, etc., which requires the design of a reasonable data fusion mechanism to extract complementary information between different sensor data and achieve more accurate identification of road damage. Summary of the invention

[0005] Aiming at the problem of multi-sensor fusion in road surface damage detection, the present invention proposes a road surface damage decision-level fusion identification method based on machine learning confusion matrix. The method analyzes how to realize the fusion of different sensor data, obtains the identification result closer to the actual situation based on the identification results of different methods, and improves the accuracy and reliability of road surface damage identification.

[0006] Technical solution: A decision-level fusion identification method for road surface damage based on machine learning confusion matrix, including the following steps:

[0007] Step 1: Use the inspection vehicle to collect road surface data and obtain vibration data and image data;

[0008] Step 2: determining a matching method for the vibration data and the image data, so as to obtain matched vibration data and image data;

[0009] Step 3: Process the vibration data and image data collected in step 1 to construct a vibration data set and an image data set; after preprocessing the vibration data set and the image data set, divide the vibration data set and the image data set into a training set, a validation set, and a test set respectively;

[0010] Step 4: Based on the machine learning method and the training set and validation set in step 3, a vibration recognition model and an image recognition model for road surface damage recognition are trained;

[0011] Step 5: Determine the fusion recognition formula of the vibration recognition model and the image recognition model of the road surface damage. The fusion recognition formula is:

[0012] E=mE m +nE n

[0013] Among them, E is the fusion recognition result of the two models, that is, the number of a certain type of road surface damage, m is the vibration method weight of a certain type of road surface damage, n is the image method weight of a certain type of road surface damage, and E m Vibration identification expectation of a certain type of road surface damage; E n Expected image recognition of certain types of road surface damage;

[0014] Step six: implement detection on the target road section, obtain matching vibration data and image data according to the method of step two, and input the vibration data and image data into the vibration recognition model and image recognition model for recognition, obtain the vibration recognition results and image recognition results of each type of road surface damage, and then obtain the fusion recognition results through the fusion recognition formula.

[0015] Furthermore, in step 1, a vibration data and image data acquisition device is installed on the inspection vehicle, the acquisition device includes a vibration acceleration sensor, a high-definition motion camera and a positioning device, and the sampling frequencies of the vibration sensor, the motion camera and the positioning device are determined;

[0016] 1) The vibration acceleration sensor is used to collect vibration data of road surface damage. The vibration sensor is arranged on a hard plane on the vehicle body and is sensitive to road conditions;

[0017] 2) The high-definition motion camera is used to collect image data of road surface damage. The high-definition motion camera is aimed at the road surface and can adjust its shooting range to obtain continuous road surface images with a unit length of l0;

[0018] 3) The positioning device is installed at the wheel and is used to match the vibration data of road surface damage with the image data.

[0019] Furthermore, in step 2, the method for matching the vibration data and the image data is: determine the unit length l0 of the road section, determine the time interval experienced by the inspection vehicle through each unit length l0 of the road section through the positioning device, divide the vibration data and the image data according to the time interval of each unit length of the road section, and obtain matched vibration data and image data.

[0020] Furthermore, in step three, the vibration data set and the image data set are preprocessed, and the preprocessing includes:

[0021] 1) In the vibration dataset and image dataset, different types of road surface damage (sinking, transverse cracks, transverse repairs, potholes, etc.) must be covered;

[0022] 2) For vibration data, denoising is performed. First, obvious outliers are removed. Then, the vibration signal is denoised using the wavelet denoising method to extract effective vibration characteristics of road surface damage.

[0023] 3) Make the number of each type of road surface damage in the image data set roughly equal, and at the same time make the number of each type of road surface damage in the vibration data set roughly equal, to ensure that the vibration data set and the image data set are balanced;

[0024] Furthermore, in step three, the training set, validation set, and test set obtained by dividing the vibration data set and the image data set are recorded as vibration data training set, vibration data validation set, vibration data test set, image data training set, image data validation set, and image data test set, respectively.

[0025] Furthermore, in step 4,

[0026] 1) Based on the YOLO target detection algorithm, a road surface damage recognition model based on image method, namely, an image recognition model, is established by relying on the preprocessed image data training set and image data verification set;

[0027] 2) Based on the support vector machine algorithm, a road surface damage identification model using the vibration method, namely the vibration identification model, is established relying on the preprocessed vibration data training set and vibration data verification set.

[0028] Furthermore, in step 5, E m The method to obtain is as follows:

[0029] (1) The vibration data test set is tested using the vibration recognition model to obtain a confusion matrix corresponding to each type of road surface damage, which contains road surface background information and recognition result decomposition information. The formula of the confusion matrix is:

[0030]

[0031] Among them, TP is the true positive class identified by the vibration recognition model, FP is the false positive class identified by the vibration recognition model; TN is the true negative class identified by the vibration recognition model; FN is the false negative class identified by the vibration recognition model;

[0032] Then the probability confusion matrix of this type of road surface damage vibration method is obtained:

[0033]

[0034] in,

[0035] (2) According to the vibration method probability confusion matrix of each type of road surface damage, the vibration recognition probability distribution model is obtained, and then the vibration recognition expectation E is calculated. m , the calculation formula is as follows:

[0036]

[0037] Among them, a is the number of road surface damages that are considered to be of a certain type by the vibration recognition results; b is the number of road surface damages that are not considered to be of a certain type by the vibration recognition results; x is the independent variable, which is the possible actual number of road surface damages of a certain type, and its value range is [0, a+b]; Pb(x) is the probability that the actual number of road surface damages of a certain type is x.

[0038] Furthermore, in step 5, E n The method to obtain is as follows:

[0039] (1) Use the image recognition model to detect the image data test set and obtain the confusion matrix corresponding to each road surface damage type, which contains the road surface background information and the recognition result decomposition information. The formula of the confusion matrix is:

[0040]

[0041] Among them, TP' is the true positive class identified by the image recognition model; FP' is the false positive class identified by the image recognition model; TN' is the true negative class identified by the image recognition model; FN' is the false negative class identified by the image recognition model;

[0042] Then the probability confusion matrix of each road surface damage type image method is obtained:

[0043]

[0044] in,

[0045] (2) According to the image method probability confusion matrix of each type of road surface damage, the image recognition probability distribution model is obtained, and then the image recognition expectation E is calculated. n , the calculation formula is as follows:

[0046]

[0047] Where a' is the number of road surface damages that are considered to be of a certain type by the image recognition result; b' is the number of road surface damages that are not considered to be of a certain type by the image recognition result; x is the independent variable, which is the possible actual number of road surface damages of a certain type, and its value range is [0, a'+b']; Pb'(x) is the probability that the actual number of road surface damages of a certain type is x.

[0048] Furthermore, in step 5, the method for determining m and n is as follows:

[0049] (1) Compare the recognition results of the vibration data test set based on the vibration recognition model with the actual results of the vibration data test set;

[0050] (2) Compare the recognition results of the image data test set by the image recognition model with the actual results of the image data test set;

[0051] (3) Based on the comparison results, the accuracy of the vibration recognition method and the image recognition method is determined, and then the weights are allocated: the method with a higher accuracy rate obtains a higher weight, and the method with a lower accuracy rate obtains a lower weight, and the sum of the weights is 1; finally, the weight m of the vibration method for each type of road surface damage and the weight n of the image method for each type of road surface damage are obtained, and m+n=1.

[0052] Beneficial effects: (1) The present invention adopts a decision-level fusion method to comprehensively utilize the recognition results of the vehicle-mounted vibration method and the image method, avoiding the problem of incompatibility of heterogeneous data sources faced by sensor-level and feature-level data fusion, and has higher inclusiveness and scalability. In view of the respective defects of the vehicle-mounted vibration method and the image method (such as the image method is easily affected by vehicle speed and light, and the vibration method has a limited detection range, etc.), a fusion recognition method is adopted to take advantage of each other's strengths and overcome the limitations of a single method, and can be used in more scenarios. (2) The present invention constructs respective probability distribution models through the confusion matrix of the recognition results of the vibration method and the image method, which can quantify the recognition accuracy of different types of road surface damage under the two methods, and then obtain the expected value of the comprehensive recognition result, which has higher accuracy and reliability. (3) The present invention proposes an effective method for matching vibration data and image data. By reasonably setting the unit length of the road section and using a positioning device to record the time interval of the inspection vehicle passing through each unit length section, the vibration data and image data can be accurately corresponded in time and space. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the method of the present invention;

[0054] Figure 2 It is a schematic diagram of the installation position of the vibration data acquisition device in the present invention;

[0055] Figure 3 It is a schematic diagram of the installation positions of the image data acquisition device and the positioning device in the present invention;

[0056] Figure 4 Schematic diagram of the probability distribution of the vibration recognition results and image recognition results of the sinking in implementation. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0058] The present invention discloses a road surface damage decision-level fusion identification method based on machine learning confusion matrix, which is characterized by comprising the following steps:

[0059] Step 1: Use the inspection vehicle to collect road surface data and obtain vibration data and image data;

[0060] Vibration data and image data acquisition equipment is installed on the inspection vehicle. The acquisition equipment includes a vibration acceleration sensor, a high-definition motion camera, a positioning device, etc., and the sampling frequencies of the vibration sensor, the motion camera, and the positioning device are determined.

[0061] 1) The vibration acceleration sensor is used to collect vibration data of road surface damage; the vibration sensor is arranged on a hard plane on the vehicle body and is sensitive to road conditions.

[0062] like Figure 2 As shown, in this embodiment, the vibration acceleration sensor is arranged on the upper swing arm 3 of the right front wheel steering knuckle 2, where there is no vibration reduction system and is relatively sensitive to road conditions.

[0063] 2) The high-definition motion camera is used to collect image data of road surface damage; the high-definition motion camera is aimed at the road surface and can adjust its shooting range to obtain continuous road surface images with a unit length of l0.

[0064] like Figure 3 As shown, in this embodiment, the high-definition motion camera is installed at the rear of the vehicle using a magnetic device 4 and an adjustable bracket 5, and is aimed at the road surface for shooting, and its shooting range is adjusted by the adjustable bracket 5.

[0065] 3) The positioning device is installed at the wheel to accurately match the vibration data of road surface damage with the image data. Figure 3 As shown, in this embodiment, the positioning device is installed at the wheel, and the main components are a single-point laser ranging radar 7 and a reflective positioning plate 8, which monitors the number of wheel rotations in real time. The reflective positioning plate 8 is installed on the spoke of the detection vehicle and rotates with the wheel; the single-point laser ranging radar 7 is installed on the vehicle body above the spoke, and the laser is vertically pointed to the ground, and when the wheel rotates, the reflective positioning plate passes through the laser aiming line.

[0066] Properly increasing the sampling frequency can obtain more information, which is beneficial to the identification of road conditions; but at the same time, increasing the sampling frequency means an increase in data storage and a decrease in data analysis efficiency, so a better sampling frequency needs to be determined.

[0067] In view of the fact that the highest resonance frequency of a vehicle is generally 80-100Hz, the sampling frequency of the vibration sensor is set to 1000Hz in this embodiment to capture the vibration characteristics of the vehicle under different working conditions. For high-definition motion cameras, in order to ensure that the complete road section can be captured in the end and to avoid excessive data volume, the shooting frame rate is set to 60 frames per second. For the positioning device, in order to ensure that the wheel rotation can be monitored in any vehicle speed range (0-120km / h), the sampling frequency of the wheel is set to 300Hz.

[0068] During the collection, the vibration acceleration sensor, high-definition motion camera and positioning device are turned on to synchronously collect vibration data and image data; the present invention uses the vehicle-mounted vibration method and image method to synchronously collect data and obtain training data. When the detection vehicle is driving, it stays in the same lane to achieve lane-level data collection.

[0069] Step 2: determining a matching method for the vibration data and the image data, so as to obtain a matched vibration data set and image data set;

[0070] Matching method: determine the unit length l0 of the road section, determine the time interval that the detection vehicle experiences when passing through each unit length l0 of the road section through the positioning device, divide the vibration data and image data according to the time interval of each unit length of the road section, and obtain the matched vibration data set and image data set.

[0071] The unit length l0 is preferably 4 to 6 meters. If the unit length is short, such as 1 meter, when the vehicle is traveling at high speed, there are few sampling points within 1 meter, which affects the recognition of road surface damage; if the unit length is long, the difference in damage characteristics of different road sections will be greatly reduced, which is not conducive to recognition. In this embodiment, the length l0 is the circumference of two wheels 2C (the wheel rotates two circles, and the shooting range just moves forward one unit length); the road surface width photographed by the high-definition camera is generally 3.75 meters (the same as the width of one lane of the road section to be tested).

[0072] Combined with the positioning device, the time interval experienced by the inspection vehicle passing through each unit length section is determined. According to the time interval of each unit length section, the vibration data and image data are divided to achieve accurate matching of the vibration data and image data.

[0073] Step 3: Use the matching method in step 2 to process the vibration data and image data collected in step 2 to obtain matched vibration data sets and image data sets. After preprocessing the vibration data sets and image data sets, divide the vibration data sets and image data sets into a training set, a validation set, and a test set.

[0074] In this embodiment, a vibration data set and an image data set are obtained by collecting road surface data over a large area. Before the data set is used to construct a road surface damage recognition model, it is necessary to perform preprocessing. The steps for preprocessing the data set are as follows:

[0075] 1) In the vibration data set and image data set, different types of road surface damage (including subsidence, transverse cracks, transverse repairs, potholes, etc.) must be covered;

[0076] 2) The collected vibration data needs to be denoised; first, the obvious outliers are removed, and then the vibration signal is denoised using the wavelet denoising method to extract effective vibration characteristics of road surface damage;

[0077] 3) The number of each type of road surface damage in the image data set is made roughly equal, and the number of each type of road surface damage in the vibration data set is made roughly equal, so as to obtain a balanced image data set and vibration data set.

[0078] The vibration dataset and image dataset are divided into training set, verification set and test set respectively, and recorded as vibration data training set, vibration data verification set, vibration data test set, image data training set, image data verification set and image data test set respectively.

[0079] Step 4: Based on the machine learning method and the training set and validation set in step 3, a vibration recognition model and an image recognition model for road surface damage identification are trained.

[0080] Select a suitable machine learning method, use the vibration data training set and the vibration data verification set to train to obtain a road surface damage recognition model based on vibration data (vibration recognition model), and use the image data training set and the image data verification set to train to obtain a road surface damage recognition model based on image data (image recognition model).

[0081] The specific steps of step 4 are as follows:

[0082] 1) Based on the YOLO target detection algorithm, a road surface damage recognition model (image recognition model) based on image methods is established relying on the preprocessed image data training set and image data verification set.

[0083] 2) Based on the support vector machine (SVM) algorithm, a road surface damage identification model (vibration identification model) based on the vibration method is established relying on the preprocessed vibration data training set and vibration data verification set.

[0084] Step 5: Determine the fusion recognition formula of the vibration recognition model and the image recognition model of the road surface damage. The fusion recognition formula is:

[0085] E=mE m +nE n

[0086] Among them, E is the fusion recognition result of the two models, that is, the number of a certain type of road surface damage, m is the vibration method weight of a certain type of road surface damage, n is the image method weight of a certain type of road surface damage, and E m Vibration identification expectation of a certain type of road surface damage; E n It is expected to identify images of certain types of road surface damage.

[0087] Step 5.1, E m The method to obtain is as follows:

[0088] (1) The vibration recognition model is used to detect the test set of vibration data to obtain the confusion matrix corresponding to each type of road surface damage (such as subsidence) with road surface background information and recognition result decomposition information.

[0089] The formula for the confusion matrix is:

[0090]

[0091] Among them, TP is the true positive identified by the vibration recognition model (True positive, the actual number of positive examples correctly predicted by the model as positive examples); FP is the false positive identified by the vibration recognition model (False positive, the actual number of negative examples incorrectly predicted by the model as positive examples); TN is the true negative identified by the vibration recognition model (True negative, the actual number of negative examples correctly predicted by the model as negative examples); FN is the false negative identified by the vibration recognition model (False negative, the actual number of positive examples incorrectly predicted by the model as negative examples).

[0092] Then the probability confusion matrix formula of this type of road surface damage vibration method is obtained:

[0093]

[0094] in,

[0095] (2) According to the vibration method probability confusion matrix of each type of road surface damage, the vibration recognition probability distribution model is obtained, and then the vibration recognition expectation E is calculated. m , the calculation formula is as follows:

[0096]

[0097] Among them, a is the number of road surface damages that are considered to be of a certain type by the vibration recognition results; b is the number of road surface damages that are not considered to be of a certain type by the vibration recognition results; x is the independent variable, which is the possible actual number of road surface damages of a certain type, and its value range is [0, a+b]; Pb(x) is the probability that the actual number of road surface damages of a certain type is x.

[0098] ①When 0≤x≤a,

[0099] When the recognition result is considered to be the vibration data of the specified road surface damage, and the actual number of the specified road surface damage increases from 0 to x, the recognition result is considered not to be the data of the specified road surface damage, but the actual number of the specified road surface damage decreases from x to 0 at the same time (keeping the actual total amount of the specified road surface damage as x):

[0100]

[0101] Among them, in the first term of Pb(x), It indicates the probability that there are actually 0 road surface damages among the a road surface damages identified. It represents the probability that among the b cases identified as not being this road surface damage, actually x cases are this road surface damage. Also, since each identification is an independent and repeated event, multiplying the two represents the probability that actually x cases are this road surface damage. This is the first case;

[0102] In the second term of Pb(x), It represents the probability that among the a cases identified as being this road surface damage, actually 1 case is this road surface damage, It represents the probability that among the b cases identified as not being this road surface damage, actually (x - 1) cases are this road surface damage. Multiplying the two also represents the probability that actually x cases are this road surface damage. Similarly, each term represents a situation that makes the probability that actually x cases are this road surface damage. Under the condition of 0 ≤ x ≤ a, there are a total of (x + 1) such situations. Therefore, Pb(x) is the sum of the probabilities of (x + 1) situations.

[0103] ② When a < x ≤ b,

[0104]

[0105] In this case, the construction idea of Pb(x) is exactly the same as that in ①, but due to the different value range of x, the number of situations is different. In this case, the maximum number of cases where the identification is actually this road surface damage can only be a.

[0106] ③ When b < x ≤ a + b,

[0107]

[0108] The construction ideas of the probability functions in these three cases are the same. It's just that due to the change in the value range of x, the upper and lower limits of the number of cases where the identification is actually this road surface damage and the number of cases where the identification is not this road surface damage but actually is this road surface damage are different, and the two fluctuate synchronously, but their sum is always x.

[0109] Step 5.2. The method for obtaining E n is as follows:

[0110] (1) Similarly, use the image recognition model to detect the test set of image data, and obtain the confusion matrix corresponding to each type of road surface damage (such as subsidence) with road surface background information and recognition result decomposition information.

[0111] The formula for the confusion matrix is:

[0112]

[0113] Among them, TP' is the true positive class identified by the image recognition model (Truepositive, the actual number of positive examples correctly predicted by the model as positive examples); FP' is the false positive class identified by the image recognition model (Falsepositive, the actual number of negative examples incorrectly predicted by the model as positive examples); TN' is the true negative class identified by the image recognition model (True negative, the actual number of negative examples correctly predicted by the model as negative examples); FN' is the false negative class identified by the image recognition model (False negative, the actual number of positive examples incorrectly predicted by the model as negative examples).

[0114] Then the probability confusion matrix of each type of road surface damage image method is obtained:

[0115]

[0116] in,

[0117] (2) According to the image method probability confusion matrix of each type of road surface damage, the image recognition probability distribution model is obtained, and then the image recognition expectation E is calculated. n , the calculation formula is as follows:

[0118]

[0119] Among them, a' is the number of road surface damages that the image recognition results consider to be a certain type; b' is the number of road surface damages that the image recognition results consider not to be a certain type; x is an independent variable, which is the possible true number of road surface damages of a certain type, and its value range is [0, a'+b']; Pb'(x) is the probability that the true number of road surface damages of a certain type is x. The calculation formula of Pb'(x) is also the same as the formula of Pb(x) of the vibration method, except that the probabilities P'1, P'2, P'3, and P'4 in the confusion matrix of the image recognition model should be used here.

[0120] For each group of vibration-image data in the vibration data test set and the image data test set, the vibration recognition model and the image recognition model are used to identify the data, and the recognition results are counted to obtain the confusion matrix of the fusion recognition results of each type of road surface damage (sinking, transverse cracks, transverse repairs, potholes, etc.); taking sinking as an example, there are 200 groups in the test set, and the recognition results of the vibration recognition model on the test set are counted to obtain the confusion matrix of the vibration recognition model for sinking:

[0121]

[0122] Its probability form is:

[0123]

[0124] The recognition results of the image recognition model on the image data test set are statistically analyzed, and the confusion matrix of the image recognition model for sinkholes is obtained as follows:

[0125]

[0126] Its probability form is:

[0127]

[0128] The above confusion matrix reflects the performance of the recognition model.

[0129] Compared with the recognition results of a single model, E is the fusion recognition result that is closest to the actual situation. By analyzing the confusion matrix of the recognition results of the vibration method and the image method, a probability distribution model is constructed, and finally the fusion recognition results of the two models are obtained.

[0130] Step 5.3: m and n are determined as follows:

[0131] (1) Compare the recognition results of the vibration data test set based on the vibration recognition model with the actual results of the vibration data test set;

[0132] (2) Compare the recognition results of the image data test set by the image recognition model with the actual results of the image data test set;

[0133] (3) According to the comparison results, the accuracy of the vibration recognition method and the image recognition method is determined, and then the weights are assigned: the method with higher accuracy gets a higher weight, the method with lower accuracy gets a lower weight, and the sum of the weights is 1. Finally, the weight m of the vibration method for each type of road surface damage and the weight n of the image method for each type of road surface damage are obtained, and m+n=1.

[0134] In this embodiment, based on the comparison between the vibration and image recognition results and the true results, it is determined that for sinking, the weight m of the vibration method is 0.81, and the weight n of the image method is 0.19.

[0135] Step 6: Detect the target road section, obtain the matched vibration data set and image data set, and input the vibration data set and image data set into the vibration recognition model and image recognition model for recognition, obtain the vibration recognition expectation and image recognition expectation of each type of road surface damage, and obtain the fusion recognition result through the fusion recognition formula;

[0136] 1) The detection is carried out on a certain section of road (with a total length of 2 km), and the detection data is divided according to 4.17 m (the circumference of two wheels 2C), resulting in a total of 480 groups of detection data. In order to obtain the number of subsidence in a certain target section of road (with a length of 300 m), the detection data of this section (a total of 24 groups) is identified by vibration method and image method. The identification result of the vibration method is 5, and the identification result of the image method is 2. Assume that the true number of subsidence in the target section is x.

[0137] 2) For the vibration identification result, the number of identified subsidence is 5, and the number of non - subsidence is 19.

[0138] When 0 ≤ x ≤ 5,

[0139]

[0140] When 5 < x ≤ 19,

[0141]

[0142] When 19 < x ≤ 24,

[0143]

[0144] Among them, a = 5, b = 19, P1 = 0.85, P2 = 0.15, P3 = 0.86, P4 = 0.14.

[0145] According to the above probability distribution model, the vibration recognition expectation of the subsidence in the target section is obtained. Due to the complexity of the formula, it will be very time - consuming to calculate each Pb(x) manually, and it is completed by programming in practical applications.

[0146]

[0147] 3) For the image recognition result, the number of identified subsidence is 2, and the number of non - subsidence is 22.

[0148] When 0 ≤ x ≤ 2, calculate Pb'(x);

[0149] When 2 < x ≤ 22, calculate Pb'(x);

[0150] When 22 < x ≤ 24, calculate Pb'(x);

[0151] Among them, a' = 2, b' = 22, P1' = 0.77, P2' = 0.23, P3' = 0.73, P4' = 0.27.

[0152] According to the above probability distribution model, the image recognition expectation of the subsidence in the target section is obtained:

[0153]

[0154] Figure 4 Schematic diagram of the probability distribution of the vibration recognition results and image recognition results of the sinking in implementation.

[0155] 4) Identify the expected E based on the vibration of the sinking m 6.91, image recognition expectation E n is 7.48, and the vibration method weight m of the sinking is 0.81, and the image method weight n is 0.19, and the fusion recognition result of the sinking is obtained:

[0156] E=0.81×6.91+0.19×7.48=7.02

[0157] The present invention adopts image method and vibration method to perform fusion identification of road surface damage, but unlike the prior art, the present invention focuses on the decision-level fusion level, that is, fusion is performed on the basis of the independent decision results of image recognition and vibration recognition. By analyzing the recognition results of the two methods and their corresponding confusion matrices, a fusion recognition technology with better reliability is formed. Compared with sensor-level data fusion and feature-level data fusion, decision-level fusion has higher tolerance for heterogeneous data sources. In addition, under the premise that the error of the results of a single method is controllable, decision-level fusion can better process the comprehensive results, thereby improving the accuracy and reliability of road surface damage identification.

[0158] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A decision-level fusion identification method for road surface damage based on machine learning confusion matrix, characterized in that: The steps include: Step 1: Use the inspection vehicle to collect road surface data and obtain vibration data and image data; Step 2: determining a matching method for vibration data and image data to obtain a matched vibration data set and image data set; Step 3: Process the vibration data and image data collected in step 1 to construct a vibration data set and an image data set; after preprocessing the vibration data set and the image data set, divide the vibration data set and the image data set into a training set, a validation set, and a test set respectively; Step 4: Based on the machine learning method and the training set and validation set in step 3, a vibration recognition model and an image recognition model for road surface damage recognition are trained; Step 5: Determine the fusion recognition formula of the vibration recognition model and the image recognition model of the road surface damage. The fusion recognition formula is: E=mE m +nE n Among them, E is the fusion recognition result of the two models, that is, the number of a certain type of road surface damage, m is the vibration method weight of a certain type of road surface damage, n is the image method weight of a certain type of road surface damage, and E m Vibration identification expectation for a certain type of road surface damage; E n Expected image recognition of certain types of road surface damage; Step 6: Detect the target road section, obtain matching vibration data and image data according to the method in step 2, and input the vibration data and image data into the vibration recognition model and image recognition model for recognition, obtain the vibration recognition result and image recognition result of each type of road surface damage, and then obtain the fusion recognition result through the fusion recognition formula; In step 5, E m The method to obtain is as follows: (1) The vibration data test set is tested using the vibration recognition model to obtain a confusion matrix corresponding to each type of road surface damage, which contains road surface background information and recognition result decomposition information. The formula of the confusion matrix is: Among them, TP is the true positive class identified by the vibration recognition model, FP is the false positive class identified by the vibration recognition model; TN is the true negative class identified by the vibration recognition model; FN is the false negative class identified by the vibration recognition model; Then the probability confusion matrix of this type of road surface damage vibration method is obtained: in, (2) According to the vibration method probability confusion matrix of each type of road surface damage, the vibration recognition probability distribution model is obtained, and then the vibration recognition expectation E is calculated. m , the calculation formula is as follows: Where a is the number of road surface damages that are considered to be of a certain type by the vibration recognition results; b is the number of road surface damages that are considered not to be of a certain type by the vibration recognition results; x is an independent variable, which is the possible true number of road surface damages of a certain type, and its value range is [0, a+b]; Pb(x) is the probability that the true number of road surface damages of a certain type is x; In step 5, E n The method to obtain is as follows: (1) Use the image recognition model to detect the image data test set and obtain the confusion matrix corresponding to each road surface damage type, which contains the road surface background information and the recognition result decomposition information. The formula of the confusion matrix is: Among them, TP ′ The true class identified by the image recognition model; FP ′ TN is the false positive class identified by the image recognition model; ′ True negative class identified by the image recognition model; FN ′ False negatives identified by image recognition models; Then the probability confusion matrix of each road surface damage type image method is obtained: in, (2) According to the image method probability confusion matrix of each type of road surface damage, the image recognition probability distribution model is obtained, and then the image recognition expectation E is calculated. n , the calculation formula is as follows: Where a′ is the number of road surface damages that are considered to be of a certain type by the image recognition result; b′ is the number of road surface damages that are considered not to be of a certain type by the image recognition result; x is an independent variable, which is the possible true number of road surface damages of a certain type, and its value range is [0, a′+b′]; Pb′(x) is the probability that the true number of road surface damages of a certain type is x; In step 5, m and n are determined as follows: (1) Compare the recognition results of the vibration data test set based on the vibration recognition model with the actual results of the vibration data test set; (2) Compare the recognition results of the image data test set by the image recognition model with the actual results of the image data test set; (3) Based on the comparison results, the accuracy of the vibration recognition method and the image recognition method is determined, and then the weights are allocated: the method with high accuracy is given a high weight, and the method with low accuracy is given a low weight, and the sum of the weights is 1; finally, the weight m of the vibration method for each type of road surface damage and the weight n of the image method for each type of road surface damage are obtained, and m+n=1.

2. The method for road surface damage decision-level fusion identification based on machine learning confusion matrix according to claim 1 is characterized in that: In step 1, a vibration data and image data acquisition device is installed on the inspection vehicle, wherein the acquisition device includes a vibration acceleration sensor, a high-definition motion camera and a positioning device, and the sampling frequencies of the vibration sensor, the motion camera and the positioning device are determined; 1) The vibration acceleration sensor is used to collect vibration data of road surface damage. The vibration sensor is arranged on a hard plane on the vehicle body and is sensitive to road conditions; 2) The high-definition motion camera is used to collect image data of road surface damage. The high-definition motion camera is aimed at the road surface and can adjust its shooting range to obtain continuous road surface images with a unit length of l0; 3) The positioning device is installed at the wheel and is used to match the vibration data of road surface damage with the image data.

3. The method for road surface damage decision-level fusion identification based on machine learning confusion matrix according to claim 1 is characterized in that: In step 2, the matching method of the vibration data and the image data is as follows: determine the unit length l0 of the road section, determine the time interval experienced by the inspection vehicle through each unit length l0 of the road section through the positioning device, divide the vibration data and the image data according to the time interval of each unit length of the road section, and obtain the matched vibration data set and image data set.

4. The method for road surface damage decision-level fusion identification based on machine learning confusion matrix according to claim 1 is characterized in that: In step three, the vibration data training set and the image data set are preprocessed, and the preprocessing includes: 1) Make the vibration dataset and image dataset include data of different types of road surface damage; 2) De-noising the vibration data. First, remove the obvious outliers. Then use the wavelet denoising method to reduce the noise of the vibration signal and extract the effective vibration characteristics of the road surface damage. 3) The number of each type of road surface damage in the image data set is made equal, and the number of each type of road surface damage in the vibration data set is made equal accordingly, so as to obtain a balanced image data set and vibration data set.

5. The method for road surface damage decision-level fusion identification based on machine learning confusion matrix according to claim 1 is characterized in that: In step three, the training set, validation set, and test set after the vibration data set and the image data set are divided are recorded as vibration data training set, vibration data validation set, vibration data test set, image data training set, image data validation set, and image data test set, respectively.

6. The method for road surface damage decision-level fusion identification based on machine learning confusion matrix according to claim 5 is characterized in that: The specific steps of step 4 are as follows: 1) Based on the YOLO target detection algorithm, a road surface damage recognition model based on image method, namely, an image recognition model, is established by relying on the preprocessed image data training set and image data verification set; 2) Based on the support vector machine algorithm, a road surface damage identification model using the vibration method, namely the vibration identification model, is established relying on the preprocessed vibration data training set and vibration data verification set.

Images