An intelligent detection method for sidewalk flatness based on multi-source data fusion analysis

Abstract

The present invention discloses an intelligent detection method for sidewalk flatness based on multi-source data fusion analysis, including data collection, data analysis, image analysis, using GIS to classify and layer abnormal points of 9 common uneven types of sidewalks on a map, and overall flatness evaluation. A laser sensor is used to collect the height data of each detection point of a sidewalk section, and the standard deviation σ of the vertical displacement values ​​of all detection points of the sidewalk section is calculated, and the IRI index is further solved according to the relationship. Finally, the FQI index of the sidewalk section is calculated using the relationship, and the flatness of the sidewalk section is evaluated in an overall graded manner.

Description

Technical Field

[0001] The present invention relates to the field of data fusion analysis and machine learning technology, and in particular to an intelligent detection method for sidewalk flatness based on multi-source data fusion analysis. Background Art

[0002] In recent years, as the government advocates healthy walking and cycling activities, citizens' attention to healthy and low-carbon travel has continued to increase, and people's demand for slow traffic has increased significantly. Cycling has become an important way for citizens to relax and exercise. During the process of walking and cycling, citizens can strengthen their bodies, get close to nature, and appreciate the charm of the city. This puts higher requirements on the quality of sidewalks, and the flatness of sidewalks is one of the important indicators of sidewalk quality. However, at present, the sidewalk flatness detection method is still insufficient: lack of unified standards and specifications, lack of efficient detection technology and equipment, lack of systematic detection data and analysis, so it is not possible to carry out efficient, large-scale, and intelligent evaluation of sidewalk flatness. With the rise of data fusion analysis and machine learning technology, the present invention combines traditional multi-source sensor technology with emerging machine learning and data analysis technology to form a complete and intelligent sidewalk flatness intelligent detection system with multi-source data image fixed-point acquisition, data fusion analysis, driving machine learning algorithm pattern recognition, flatness pattern recognition result feedback and GPS dynamic display points, which can realize dynamic, efficient, fixed-point and intelligent monitoring and diagnosis of the flatness of urban pedestrian road network to ensure the travel safety of pedestrians, especially vulnerable groups such as the elderly, children, and the disabled, improve the travel efficiency of citizens, improve the comfort of walking and cycling, promote the enthusiasm of urban residents for healthy cycling, and drive the development of green travel in cities.

[0003] Related Introduction

[0004] 1. DQN algorithm (DeepQNetwork): Reinforcement learning is usually based on the Markov decision process. The agent observes the state of the environment. t , and then do a corresponding action (Action) a t , the state of the environment changes after executing this action, and a reward (Reward) r is obtained t As feedback, and get the next state s t+1 This interactive process can obtain a four-tuple (s t ,a t ,r t ,s t+1), representing the current state, action, reward, and next state respectively, which is the source of the training set. Each recursion repeats a series of steps such as "obtaining the environment state, executing the action and obtaining the reward, obtaining the new state, recording the sample, and updating the gradient" to train the neural network. After a series of sample training, the feature vector is finally formed. The execution principle of each sample is as follows:

[0005] First, set the previous data in the input data set to state s t , the action taken is a t , set to read the next data and subtract it from the previous data to get the reward value r t That is, the difference, then get the next data and set it to state s t+1 , this interaction results in a four-tuple (s t ,a t ,r t ,s t+1 )(t represents the number of the state, t+1 represents the number of the next state). Then, all the data in a category of collected data fragments are identified in sequence (where a data fragment refers to a data group composed of several data that reflects the characteristics of the category under study) to form a sample. Finally, all the data fragments contained in a category are identified, and a large number of sample gradients trained in each category are stored in the DQN neural network. The sample library in the neural network is updated with real-time gradients, thereby realizing the training of the DQN algorithm and finally generating the feature vectors corresponding to each category.

[0006] 2. Decision Tree Algorithm: Decision Tree Algorithm is a typical classification method. Classification (Categorization or Classification) is to label the research object according to a certain standard, and then distinguish and classify it according to the artificial label. Classification requires manual definition of categories in advance, and the classifier needs to be trained with manually labeled classification training corpus, which belongs to the category of supervised learning. The decision tree algorithm consists of three steps: feature selection, decision tree generation and decision tree pruning.

[0007] First of all, the role of feature selection is to screen out features that are highly correlated with the classification results, that is, features with strong classification capabilities for the studied categories. The various feature vectors generated by DQN can be input into the decision tree classification algorithm to achieve feature selection. Then the decision tree is generated. After selecting the features, starting from the root node, the information gain of all features of the node is calculated, and the feature with the largest information gain is selected as the node feature. Leaf nodes are established according to the different characteristics of the studied category, that is, the different characteristics of the reward value in DQN. Finally, the decision tree is pruned and corrected. The main purpose of pruning is to combat "overfitting". The evaluation effect of the decision tree model is obtained based on the test data, and some branches are actively removed to reduce the risk of overfitting.

[0008] 3. IRI: International Roughness Index is the cumulative vertical displacement value of a quarter of a car at a speed of 80km / h, and the unit is m / km. IRI is actually a dimensionless index because it comes from the simulated statistical value of a quarter of a car, but it is usually expressed in m / km.

[0009] 4.FQI: refers to the sidewalk condition index, which represents the quality indicator of the sidewalk flatness.

[0010] 5. Laser sensor data analysis principle: refer to the attached Figure 5 Schematic diagram of the acquisition equipment. It can be seen that the laser sensor collects the height H of each point from the ground K Therefore, when the laser sensor is fixed on the inspection vehicle, if the road surface flatness at the vehicle wheels meets the standard, the height of the laser sensor basically does not change, so it can measure smaller pits or smaller bumps (diameter <45cm) on the road surface between the wheels of the inspection vehicle.

[0011] From the above figure, we can see that the height difference from the ground is ΔH = H K+1 -H K , (K represents the number of the detection point in the height from the ground data). When the height from the rear detection point to the ground is less than the height from the front detection point to the ground (such as detection points 1 and 2), that is, ΔH < 0, it means that the actual road surface is on an upward trend. On the contrary, when the height from the rear detection point to the ground is greater than the height from the front detection point to the ground (such as detection points 2 and 3), that is, ΔH > 0, it means that the actual road surface is on a downward trend. It is used in road surface state pattern recognition.

[0012] 6. Gyroscope sensor data analysis principle:

[0013]

[0014]

[0015]

[0016] When stationary, the gyroscope measures the value of the stationary acceleration in the Z direction. From the above equations 1.1 and 1.2, we can know that the difference in the acceleration in the Z direction Δa is Z =(a Z+1 -a 静 )-(a Z -a 静 )=(a Z+1 -a Z), (where Z represents the number of the current detection point in the vertical acceleration data, and Z+1 represents the number of the next detection point in the vertical acceleration data) is the change trend of the acceleration of the detection vehicle at adjacent detection points, and the acceleration value can reflect the displacement value. Combined with formula 1.3, the detection vehicle is subjected to changes in the vertical force due to the changes in the convexity and concavity of the ground, which in turn causes the acceleration to change. If Δa Z >0 means a Z+1 >a Z Because the ground is uphill between two adjacent detection points, the resistance and impact force increase, resulting in an increase in the resultant force in the Z+ (vertical upward) direction, a Z+1 becomes larger; if Δa Z <0 means a Z+1 <a Z Because the ground is downhill between two adjacent detection points, the resistance and impact force are reduced, resulting in a decrease in the resultant force in the Z+ (vertical upward) direction. Z+1 Therefore, according to the above characteristics, the vertical acceleration a detected by the gyroscope Z+1 The change of the value can be used to judge the changing trend of the road surface undulation. Summary of the invention

[0017] The technical solution adopted by the present invention is an intelligent detection method for sidewalk flatness based on multi-source data fusion analysis, which includes the following steps, wherein steps 3-10 are the construction and training of the DQN and decision tree fusion algorithm model, which can be completed for the first time, and steps 1-2 and steps 11-19 are necessary steps for each subsequent on-site sidewalk flatness detection.

[0018] Step 1, collect data:

[0019] Determine the sidewalk section to be tested that needs to be evaluated for flatness, use a collection device equipped with a laser sensor TOF, a gyroscope sensor IMU, a GPS sensor GNSS, and a camera, collect data from the same detection point every 25 cm along the sidewalk section to be tested that needs to be evaluated for flatness, and record the entire field video in real time. The data collected at the same detection point includes: collection time t, height above the ground H K , vertical acceleration a Z , multi-source data information of the point latitude and longitude, and on-site pictures of the detection point. The schematic diagram of the collection equipment is attached. Figure 5 As shown;

[0020] Step 2: Clean the data:

[0021] The height data H collected by the laser sensor K Referred to as laser data, K represents the number of the detection point in the laser data, and H in the laser data K∈[10,+∞) is abnormal data. The vertical acceleration data a collected by the gyroscope sensor Z Gyroscope data, where Z represents the number of the detection point in the gyroscope data.

[0022] a Z ∈(-∞,0]∪[30000,+∞) is abnormal data. Delete all the collected height data H K And all vertical acceleration data a Z Abnormal data in

[0023] Step 3, prepare data segments and classification datasets:

[0024] Step 3.1, prepare data segments: All cleaned height data H K The total data set formed is set as A H , A H The H in the data set indicates that all data in the data set are height data from the ground. All the cleaned vertical acceleration data a Z The total data set formed is set as A a , A a a in the data set indicates that all data in the data set are vertical acceleration data. From the two data sets, find the flatness anomalies on the sidewalk to be tested based on the full field video. Flatness anomalies refer to points among all detection points that have six types of unevenness problems, namely, road bumps, road convexities, road concave points, road height differences, large road depressions, and large road convexities, and record the collection time t corresponding to the flatness anomaly points respectively. According to the collection time t corresponding to the flatness anomaly point, retrieve a total of 3 data collected by the laser sensor and the gyroscope sensor at the flatness anomaly point and one detection point before and after the flatness anomaly point. According to the category to which the flatness anomaly point belongs, these 3 detection point data are regarded as an anomaly fragment of one of the 6 categories. According to the above method, find all the flatness anomaly points in turn and obtain their corresponding anomaly fragments;

[0025] Step 3.2, prepare the classification data set: classify all the abnormal point fragments contained in each category according to the above 6 categories to construct the data set C i (i=1,2,3,4,5,6), the number of outlier fragments contained in each data set is set to

[0026] N i (i=1,2,3,4,5,6), i represents the category number of the unevenness problem;

[0027] Step 4: Divide the test set and training set:

[0028] Step 3 . The classification data sets C obtained in 2 i(i=1,2,3,4,5,6) for division, let 3N i / 4 (i=1,2,3,4,5,6) outlier segments are used as training set and N i / 4 (i=1,2,3,4,5,6) pieces of abnormal point segments are the test set;

[0029] Step 5: Train the laser data analysis DQN algorithm model:

[0030] Step 5.1, input laser sensor training data: the first four datasets C corresponding to the four categories of road bumps, road convex points, road concave points, and road height difference in step 4 i (i=1,2,3,4) corresponds to the divided training set 3N i / 4 (i=1,2,3,4) copies of data are input into the laser data analysis DQN algorithm model;

[0031] Step 5.2, train the laser data analysis DQN algorithm model:

[0032] Step 5.2.1, set the initial four-tuple elements: According to the principle of DQN algorithm, initialize the state s t The state vector is set to the first distance height H in the first outlier segment of a certain category K (K=1) value as the reference, in state s t Next, perform action a t (t=1), action a t The uniform setting is to obtain the next ground height H in the same abnormal point segment K+1 , K+1 represents the number of the next detection point, and the height data H of the next detection point from the ground K+1 The height data H from the current detection point K Make the difference and get ΔH as the current reward value r t , then get the next state s t+1 The next height from the ground is H K+1 Set to s t+1 . At this time, we get a four-tuple (s t ,a t ,r t ,s t+1 ), which represent the current state, action, reward, and next state respectively, and are stored in the neural network memory pool;

[0033] Step 5.2.2, determine the trend of actual road surface changes: according to the method of setting the initial four-tuple, traverse the other data in the same abnormal point segment of a certain category in turn. t When the traversal is completed in a certain abnormal point segment of the same category, a sample of this category is formed based on the two output quadruples.t The positive and negative characteristics of the value and the laser sensor data analysis principle can analyze the change trend of the actual road surface: if r t <0, the actual road surface has an upward trend between the two detection points. t > 0, the actual road surface has a downward trend between the two detection points. Machine learning captures r t The mutation point of the positive and negative value is determined as an abnormal point. t , determine its positive or negative:

[0034] If two r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. The former is negative and the latter suddenly becomes positive. A smaller convex feature vector D1 is generated and stored in the neural network. If r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. The former is positive and the latter suddenly becomes negative. A smaller concave feature vector D2 is generated and stored in the neural network. t If one has an absolute value less than 0.5 cm and the other is negative or positive and has an absolute value greater than 0.5 cm, a road elevation difference feature vector D3 is generated and stored in the neural network;

[0035] Step 6, training the laser data analysis decision tree algorithm model;

[0036] Step 6.1, feature selection: The laser data analysis DQN algorithm model in step 5 is referred to as "DQN laser sensor neural network" to generate three types of feature vectors: small convexity D1, small concave D2, and road height difference D3. The number of feature vectors of each type is set to M j (j=1, 2, 3), where j represents the sequence number of the feature vector, is input into the constructed decision tree model as feature selection;

[0037] Step 6.2, training the laser data analysis decision tree algorithm model:

[0038] Step 6.2.1, obtain the two quadruplets that form the above three types of feature vectors from the DQN laser neural network, and take the reward r in the two quadruplets t The absolute value with the largest value is |r t | max ,According to the actual measurement situation, 0.5cm, 2cm and 5cm are set as leaf node feature values ​​and input into the training decision tree model;

[0039] Step 6.2.1.1, call each smaller convex feature vector D1 generated by the DQN neural network in turn, if |r t | max ∈(0.5cm~2.0cm), generate the manual label E1 of the road bump category; if |rt | max ∈(2.0cm~5.0cm), generate the manual labels E2 of the road surface convex points;

[0040] Step 6.2.1.2, call each smaller concave feature vector D2 generated by the DQN neural network in turn. If |r t | max ∈(0.5cm~2.0cm), generate the manual label E1 of the road bump category; if |r t | max ∈(2.0cm~5.0cm), generate the manual labels of road surface concave points E3;

[0041] Step 6.2.1.3, call each road elevation difference feature vector D3 generated by the DQN neural network in turn. If |r t |x ax ∈(5.0cm~1000cm), generate the large category of manual labels E4 for road elevation difference;

[0042] Step 7: Train the gyroscope data analysis DQN algorithm model:

[0043] Step 7.1: Input the gyroscope sensor training data set: the dataset C corresponding to the two categories of large surface convexity and large road depression in step 4 i The training set 3N divided in (i=5,6) i / 4 (i=5,6) parts of data are input into the gyroscope data analysis DQN algorithm model;

[0044] Step 7.2, train the gyroscope data analysis DQN algorithm model:

[0045] Step 7.2.1, set the initial four-tuple elements: According to the principle of DQN algorithm, initialize the state s′ t Set to C 5 and C 6 The first vertical acceleration value a in the first point segment included Z (Z=1). In state s′ t Next, set the execution action a′ t To obtain C 5 and C 6 The next vertical acceleration value a in the same abnormal point segment Z+1 , and make a difference Z+1 -a Z , we get Δa Z As the reward value r′ t . At this time, a state s′ is obtained t+1 Set to the next vertical acceleration value a Z+1 . Now we get a 4-tuple (s' t,a' t ,r' t ,s' t+1 ), which represent the current state, action, reward, and next state respectively, and are stored in the neural network memory pool;

[0046] Step 7.2.2, determine the trend of actual road surface changes: according to the method of initially setting the four-tuple, traverse the other data in the same abnormal point segment of a certain category in turn. t When the traversal is completed in a certain abnormal point segment of the same category, a sample of this category is formed based on the two output quadruple groups. t The positive and negative characteristics of the value and the gyroscope sensor data analysis principle can be used to analyze the change trend of the actual road surface: t <0, then the actual road surface has a downward trend between the two detection points, r′ t > 0, it means that there is an upward trend between the two detection points on the actual road surface. Machine learning captures r′ t The mutation point of the positive and negative value is judged as an abnormal point. Retrieve the two corresponding r's in the two quadruples contained in the same sample t , determine its positive or negative:

[0047] If two r′ t One positive and one negative, and the former is positive and the latter suddenly becomes negative, a larger concave feature vector D4 is generated and stored in the neural network; if r′ t One is positive and the other is negative, and the former is negative, and the latter suddenly becomes positive, generating a larger convex feature vector D5, which is stored in the neural network;

[0048] Step 8: Train the gyroscope data analysis decision tree algorithm model:

[0049] Step 8.1, feature selection: The gyroscope data analysis DQN algorithm model is referred to as DQN gyroscope sensor neural network. Set all feature vectors of the two categories of larger concave D4 and larger convex D5 generated in the DQN gyroscope sensor neural network in step 7, and set the number of feature vectors of each category to M j (j=5, 6) parts, input the constructed decision tree algorithm model as feature selection;

[0050] Step 8.2, train the gyroscope data analysis decision tree algorithm:

[0051] Step 8.2.1, obtain the two quadruplets that form the above two types of feature vectors from the DQN gyroscope network, and take the reward value r′ in the two quadruplets t The absolute value with the largest value is |r t |′ max ,According to the previous measured situation, 1000 is set as the leaf node feature value and input into the training decision tree model;

[0052] Step 8.2.1.1, call the feature vector D4 of each larger depression in the DQN gyroscope neural network in turn, if |r t |′ max >1000, generate the road subsidence category manual label E5. If the data is not within this range, it is not considered as an abnormal point and outputs "no abnormality";

[0053] Step 8.2.1.2, call up the feature vector D5 of each larger bump in the DQN gyroscope neural network in turn, if |r t |′ max >1000, generate the road bulge category artificial label E6. Data outside this range is not considered as an abnormal point, and the output is "no abnormality";

[0054] Step 9, test the effect of DQN and decision tree fusion model:

[0055] C i (i=1,2,3,4) corresponds to the divided abnormal point segment N i / 4 (i = 1, 2, 3, 4) is the laser data analysis algorithm model of DQN and decision tree fusion generated by the test set input; the two types of data sets C i (i=5,6) The corresponding abnormal point segment N i / 4(i=5,6) is the DQN and decision tree fusion gyroscope data analysis algorithm model generated by the test set input. Test the detection results of each category, verify the model, and evaluate the model accuracy. If the accuracy meets the actual requirements, save the model; if the accuracy does not meet the requirements, adjust the decision tree leaf node values ​​in steps 6.2 and 8.2 until the accuracy of the two models meets the actual requirements. Among them, the basic structure of the laser data analysis algorithm model fused with DQN and decision tree is shown in the attached figure. Figure 3 The basic structure of the gyroscope data analysis algorithm model fused with DQN and decision tree is shown in the attached figure. Figure 4 As shown;

[0056] Step 10: Train the image recognition neural network:

[0057] Obtain relevant pictures of six categories, including blind paths, broken bricks, abnormal cracks on manhole covers, common obstacles, missing manhole covers, and normal manhole covers, mark the pictures, and input the marked pictures into the picture set to train the YOLO image recognition neural network, evaluate the recognition effect, and generate the final weight file. Based on the two DQN and decision tree fusion algorithm models that can identify six types of unevenness problems, image recognition can refine the pattern recognition to identify the expected road bumps, obstacles, broken bricks, abnormal manhole covers, road bumps or road depressions, missing manhole covers, road height differences, road subsidence, and road arches, a total of 9 common sidewalk flatness problems;

[0058] Step 11, generate the laser sensor DQN feature vector:

[0059] Step 11.1, input laser sensor data: start the laser data analysis algorithm model fused with DQN and decision tree trained in step 9, and input the total data set A of the height from the ground collected by the laser sensor H ;

[0060] In DQN model training, the reward value under a certain state can be output in real time according to the f(S,a) function. The f(S,a) idea is the difference-by-difference idea, and the difference ΔH is also set as the reward value output;

[0061] Step 11.2, set the initial quaternion: According to the working principle of DQN, the first ground height H collected by the laser sensor is K+1 Set to the initial state s t , action a t Set to get the next height H from the ground K+1 (K=1), and make the difference ΔH=H K+1 -H K . Set the reward value r t =ΔH, the reward value corresponding to the system output is r t , and proceed to the next state s t+1 Get, output a four-tuple (s t ,a t ,r t ,s t+1 );

[0062] Step 11.3, generate the feature vector corresponding to the abnormal point: According to the principle of laser sensor data analysis, it can be known that the result corresponding to the actual road surface undulation state can be output according to the positive and negative value of the reward value. If r t >0, is a positive number, the distance from the latter point to the ground is larger, the output is "down", on the contrary, if r t<0, which is a negative number. The distance from the latter point to the ground is small, and the output is "rising". At the mutation point of the positive and negative reward value, it indicates that there is an abnormal flatness point. The r of the two quadruples before and after the mutation point is retrieved from the DQN neural network according to the sampling time. t , and determine the positive and negative:

[0063] If two r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm and the former is positive, while the latter suddenly becomes negative. A smaller convex feature vector D1 is generated and stored in the neural network. If r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. If the former is negative and the latter suddenly becomes positive, a smaller concave feature vector D2 is generated and stored in the neural network. If r t If one absolute value is less than 0.5 cm, and the other absolute value of the reward value is less than 0.5 cm and is negative or positive, then a road height difference feature vector D3 is generated and stored in the neural network;

[0064] Step 12, laser sensor decision tree classification pattern recognition:

[0065] Step 12.1, classification pattern recognition: When the system no longer outputs the new road state "up" or "down", it means that the decision tree is used to generate artificial labels and classification pattern recognition is started:

[0066] Step 12.1.1: If a certain abnormal point has generated a small convex feature vector D1 in the DQN neural network, the |r of the two quadruplets generated before and after the abnormal point is further retrieved. t | max The value of is compared with the set leaf node. t | max ∈(0.5cm~2.0cm), the final pattern recognition result will be output as "road bumps" and the acquisition time of the point, which will be automatically stored in the "road bumps abnormal point" text document; if it meets the requirements of |r t | max ∈(2.0cm~5.0cm), the final pattern recognition result will be output as "Road surface convex point" and the acquisition time of the point, which will be automatically stored in the "Road surface convex point abnormal point" text document; if both do not meet the range, "No abnormality" will be output;

[0067] Step 12.1.2: If a certain abnormal point has generated a smaller concave feature vector D2 in the DQN neural network, the |r of the two quadruplets generated before and after the abnormal point is further retrieved. t | max The value of is compared with the set leaf node. t |max ∈(0.5cm~2.0cm), the final pattern recognition result will be output as "road bumps" and the acquisition time of the point, which will be automatically stored in the "road bumps abnormal point" text document; if it meets the requirements of |r t | max ∈(2.0cm~5.0cm), the final pattern recognition result will be output as "road concave point" and the acquisition time of the point, which will be automatically stored in the "road concave point abnormal point" text document; if both do not meet the range, "no abnormality" will be output;

[0068] Step 12.1.3: If a road elevation difference feature vector D3 is generated in the DQN neural network for a certain abnormal point, the |r t | max The value of the leaf node is compared with the set leaf node. t | max ∈(5.0cm~1000cm), the final pattern recognition result is output as "road height difference" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road height difference abnormal point" text document; if both do not meet the range, "no abnormality" will be output;

[0069] Step 13, generate the gyroscope sensor DQN feature vector:

[0070] Step 13.1, input gyroscope sensor data: start the gyroscope data analysis algorithm model fused with DQN and decision tree trained in step 9, and input the vertical acceleration a of all monitoring points collected by the gyroscope sensor Z Total Dataset A a ;

[0071] In DQN model training, the reward value under a certain state can be output in real time according to the f(S,a) function, where the f(S,a) idea is the difference idea. Z Set as reward value output;

[0072] Step 13.2, set the initial quaternion: set the first vertical acceleration a collected by the gyroscope sensor Z Set to the initial state s′ t Action a′ t Set to get the next vertical acceleration a Z+1 , and make a difference Δa Z =a Z+1 -a Z . Set the reward value r′ t =Δa Z , the system outputs the corresponding reward value r′ t , and proceed to the next state s′t+1 Get, output a four-tuple (s' t ,a' t ,r' t ,s' t+1 );

[0073] Step 13.3, generate the feature vector corresponding to the outlier point: output the corresponding result according to its positive or negative value. If r′ t >0, is a positive number, the vibration acceleration generated by the latter point is small, and the output is "rising", otherwise if r' t <0, which is a negative number. The vibration acceleration generated by the latter point is larger, and the output "drops". The point where the reward value is positive or negative is the abnormal point. Retrieve r' from the two quadruples before and after the mutation point from the neural network t If r′ t One positive and one negative, and the former is negative, and the latter suddenly becomes positive, generating both positive and negative characteristics, and the positive and negative characteristics are less than zero in the early stage and suddenly become greater than zero, generating a larger concave feature vector D4, which is stored in the neural network; if r′ t One is positive and the other is negative, and the former is positive and the latter suddenly becomes negative, which generates a larger convex feature vector D5 and stores it in the neural network;

[0074] Step 14, gyroscope sensor decision tree classification pattern recognition:

[0075] Step 14.1, classification pattern recognition: When the system no longer outputs the new road state "upward" or "downward", it means that the decision tree is used to generate artificial labels and the classification pattern recognition is started;

[0076] Step 14.1.1: If a certain abnormal point has generated a relatively large concave feature vector D4 in the DQN neural network, further retrieve the |r′ in the two quadruple generated before and after the abnormal point. t | max The value of is compared with the set leaf node. t |′ max >1000, the final pattern recognition result is output as "road subsidence" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road subsidence abnormal point" text document; if both do not meet the range, "no abnormality" will be output;

[0077] Step 14.1.2: If an abnormal point is obtained and generates a large convex feature vector D5 in the DQN neural network, if it meets the condition |r′ t | max>1000, the final pattern recognition result is output as "road bulge" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road bulge abnormal point" text document; if both do not meet the range, "no abnormality" will be output;

[0078] Step 15, image classification and recognition:

[0079] Start the trained image YOLO model, and filter out all the pictures corresponding to the abnormal points according to the abnormal point moments in the 6 "xxx abnormal point" files generated in steps 12 and 14, and delete the normal point pictures. Retrieve the pictures corresponding to the 6 major categories of abnormal points for YOLO image recognition, which is used to further filter and refine the recognition result categories and improve the accuracy of pattern recognition. The picture pattern recognition corresponding to the 6 major categories of abnormal points obtained in the previous step is as follows:

[0080] Step 15.1, for the field images of abnormal points of road bumps: distinguish road bumps from normal blind roads through image recognition, delete the detection point images whose image recognition results are normal blind roads, retain all bumpy point images caused by road damage, and store the collection time data of abnormal road bumps and pattern recognition results in the abnormal road bumps result data set C′ 1 middle;

[0081] Step 15.2, for the two categories of real-world images of road convex points and road concave points: image recognition is required to distinguish between brick damage, abnormal cracks on manhole covers, common road obstacles and road convex points or concave points, a total of 4 categories. The images taken at the first three abnormal points are saved, and the time information and pattern recognition results of the corresponding abnormal points are classified and stored in the brick damage C′ 2 、Abnormal manhole cover C′ 3 , obstacle C′ 4 The remaining abnormal points that do not contain the above three categories are uniformly retained in the original image and the abnormal point collection time and pattern recognition results are stored in the abnormal point result data set C′ of the road surface convex or concave points. 5 middle;

[0082] Step 15.3, for field images with large road elevation differences: image recognition is required to distinguish between missing manhole covers, normal holes in manhole covers, and road elevation differences, a total of 3 categories. The point collection time information and pattern recognition results of the first type of missing manhole covers are saved in the manhole cover missing abnormal point result dataset C′ 6 The real-life pictures of abnormal points are retained; the second type of points identified as abnormal due to the voids in normal manhole covers are excluded and the corresponding real-life pictures are deleted; the remaining abnormal points that do not contain the above two types are uniformly retained in the original pictures and the corresponding point collection time information and pattern recognition results are saved to the abnormal point result data set C′ stored in the road height difference 7 middle;

[0083] Step 15.4: For the two types of field pictures of road subsidence and road bulge, image recognition is not required. The corresponding point time information and pattern recognition results are directly saved to the classification and saved to the corresponding abnormal point result data set C′. 8 and C′ 9 Just in the middle;

[0084] Step 16, obtain the pattern recognition result:

[0085] The final output of common sidewalk unevenness types includes road bumps, obstacles, brick damage, abnormal manhole covers, road convex or concave points, missing manhole covers, road height difference, road subsidence, and road bulges, a total of 9 categories. If new categories are discovered later, they can be supplemented according to the same operation steps as the present invention. Among them, the 9 common types of sidewalk unevenness and their classification relationships are shown in the attached figure. Figure 2 As shown;

[0086] Step 17, GIS batch deployment:

[0087] For 9 different categories C'={C' 1 ,C' 2 ,C' 3 ,C' 4 ,C' 5 ,C' 6 ,C' 7 ,C' 8 ,C' 9}The pattern recognition result dataset C' corresponding to each category i (i=1,2,3,4,5,6,7,8,9). Match the time and longitude information corresponding to all points contained in the 9 abnormal point category result sets, and classify and layer the points on the GIS map according to the longitude and latitude corresponding to the points and the identification results, so as to obtain the purpose of fine-grained fixed-point identification and early warning of the flatness of the pedestrian road network;

[0088] Step 18: Get the latitude and longitude of the starting and ending points of the sidewalk:

[0089] Find the latitude and longitude values ​​of the starting and ending points of each sidewalk to be tested from Baidu Maps. If the sidewalk to be tested is north-south, obtain the latitude of the starting and ending points; if the sidewalk to be tested is east-west, obtain the longitude of the starting and ending points;

[0090] Step 19, conduct overall smoothness assessment on the observed pedestrian road section by section:

[0091] Step 19.1, extracting the longitude and latitude of all test points and abnormal points and dividing the road section area according to the longitude and latitude of the starting point and the end point of a road section;

[0092] Step 19.2, calculate the IRI value of each road section. Calculate the height H of the laser above the ground at each detection point on each road section. K The mean value ε of each detection point is calculated, and the height H of each detection point from the ground is calculated. K The absolute value of the difference between the mean ε and the vertical displacement value of each point is obtained. Finally, the standard deviation of the vertical displacement values ​​of all detection points is calculated, which is the flatness standard deviation σ, and then the IRI is calculated:

[0093] Step 19.3, further calculate the sidewalk quality index FQI. Given the IRI value, the FQI can be calculated according to the following relationship:

[0094] FQI=4.98-0.34*IRI

[0095] The value of FQI is 0-5;

[0096] Judgment index: Grade A is excellent FQI = 5, Grade B is good FQI = 4, Grade C is average FQI = 3,

[0097] Grade D is poor FQI = 2, Grade E is extremely poor FQI = 0

[0098] Step 19.4, obtaining an overall evaluation of the smoothness of each section of the sidewalk to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a flow chart of the steps of the present invention.

[0100] Figure 2 This is a diagram of the common uneven types of sidewalk pavement (9 types) and their classification relationships.

[0101] Figure 3 This is a schematic diagram of the laser data analysis algorithm model that integrates DQN and decision tree.

[0102] Figure 4 This is a schematic diagram of the gyroscope data analysis algorithm model that integrates DQN and decision tree.

[0103] Figure 5 It is a schematic diagram of the acquisition device of the present invention.

[0104] In the figure: 1-collector display screen; 2-gyro sensor; 3-GPS sensor; 4-camera; 5-laser emission source;

[0105] 6-laser beam; 7-detection point; 8-laser sensor; 9-GPS antenna; 10-height from the ground. DETAILED DESCRIPTION

[0106] The following will provide a clear and complete description of the implementation cases of the present invention in conjunction with the drawings in the implementation of the present invention. The implementation cases described are only a part of the present invention and do not represent all implementation scenarios of the present invention. Based on the embodiments of the present invention, if new types of uneven sidewalks are discovered later, additional supplements are added according to the operating steps of the present invention, and other embodiments obtained by those skilled in the art without making other creative ideas and efforts are all within the scope of protection of the present invention.

[0107] As attached Figure 1 As shown, the present invention includes five steps. The first step is data collection. The data collected at the same detection point using laser, gyroscope, GPS multi-sensor and camera include: collection time t, height H above the ground K , vertical acceleration a Z and longitude and latitude, and take real-life images; subsequently, the abnormal data is cleaned and data segments and data sets are prepared as the data and image basis for the subsequent road surface roughness analysis. The second is data analysis. Construct and train the laser data analysis algorithm model that integrates DQN and decision tree and the gyroscope data analysis algorithm model that integrates DQN and decision tree; classify and construct data sets based on 6 major categories of uneven sidewalks, including road bumps, road convex points, road concave points, road height differences, large road convexities, and large road concavenesses, and divide them into training sets and test sets for each category; use the training sets divided into each category to train the DQN data analysis model; obtain the feature vectors generated by the DQN network for the 6 major categories of uneven sidewalks as decision tree feature selection; combine the set leaf node differentiation value, set artificial labels and use the feature vectors generated by the DQN network to train two decision tree data analysis models respectively; finally, use the test set to evaluate the pattern recognition effect of the two DQN and decision tree fusion algorithm models. The third is image analysis. The trained YOLO neural network is used to further perform image recognition on the pictures collected on the spot. On the basis of the data pattern recognition of 6 major types of unevenness, the pattern recognition content of each major category is further refined and distinguished. Finally, the data analysis and image recognition are integrated to accurately recognize the abnormal points, which can identify road bumps, obstacles, broken bricks, abnormal manhole covers, road convex or concave points, missing manhole covers, road height differences, road subsidence, and road arches, a total of 9 types of common uneven sidewalks. Fourth, GIS is used to classify and layer the abnormal points of the 9 common uneven sidewalks on the map. Fifth, the overall evaluation of flatness. The height data of each detection point of a certain sidewalk section is collected using a laser sensor, and the standard deviation σ of the vertical displacement values ​​of all detection points of the sidewalk section is calculated. The IRI index is further solved according to the relationship, and finally the FQI index of the sidewalk section is calculated using the relationship, and the flatness of the sidewalk section is evaluated as a whole.

[0108] The implementation steps are further explained below with reference to specific data examples.

[0109] Step 1: Field data collection:

[0110] Before starting the test, the collector with laser, gyroscope, GPS multi-sensor and camera needs to be installed and fixed on the inspection vehicle. To ensure the accuracy of the data, the camera shooting direction and the laser beam direction need to be perpendicular to the ground. Then turn on the collector and calibrate the time online, run the data collection related programs, and turn on the switches of each sensor. Push the inspection vehicle at a constant speed along the sidewalk whose flatness needs to be tested. The system will record the entire process and collect the time, height from the ground, vertical acceleration, GPS longitude and latitude valid data of the same sampling point at an interval of 25cm in real time, and take a photo of the actual sidewalk surface. The data and pictures are automatically stored in the collector. If there are no abnormalities in the program throughout the process, no operation is required. Finally, the sampling is completed and each sensor is turned off, and the data, road surface pictures and the entire road surface condition video are obtained. The collected data are shown in Table 1, Table 2, and Table 3:

[0111] Table 1: Data snippet collected by laser sensor

[0112]

[0113] Table 2: Gyroscope sensor data collection snippet

[0114]

[0115] Table 3: GPS sensor data collection snippet

[0116]

[0117] Step 2: Clean the data:

[0118] The height data H collected by the laser sensor K Referred to as laser data, K represents the number of the detection point in the laser data, and H in the laser data K ∈[10,+∞) is abnormal data. The vertical acceleration data a collected by the gyroscope sensor Z Referred to as gyroscope data, where Z represents the number of the detection point in the gyroscope data, and a in the gyroscope data Z ∈(-∞,0]∪[30000,+∞) is abnormal data. Delete all the collected height data H K And all vertical acceleration data a Z Abnormal data in

[0119] Step 3, laser sensor DQN feature vector generation:

[0120] 3.1, start the laser data analysis algorithm model that integrates the previously trained DQN and decision tree, and input the list data set consisting of the columns of the height from the ground collected by the laser sensor, that is, the sixth column of data in "Result_TOF_Laser_20240xxx.txt" in the field collection data;

[0121] 3.2, set the initial quaternion: the first height H from the ground collected by the laser sensor 1 =0.839, set as initial state s t , action a t Set to get the next height H from the ground 2 =0.846, and make the difference ΔH=H 2 -H 1 ; In the reward value r t Set to obtain the height difference between the next point and the ground in each state nextstate(H 2 ) and the current point currentstate(H 1 ) is the difference between two variables, that is, r t =ΔH, the reward value corresponding to the system output is r t , and proceed to the next state s t+1 Get, output a four-tuple (s t ,a t ,r t ,s t+1 );

[0122] 3.3, Generate the corresponding feature vector of the outlier point: Output the corresponding result according to its positive or negative value. If r t >0, is a positive number, the distance from the latter point to the ground is larger, the system output "decreases", on the contrary, if r t <0, is a negative number, the distance from the latter point to the ground is small, the output is "rising", and the point where the positive and negative mutation of the reward value occurs is an abnormal point. The specific method for generating a smaller concave feature vector identified by laser data at each abnormal point is as follows:

[0123] Retrieve the r of the two quadruplets before and after the mutation point from the neural network t If two r t One positive and one negative, the absolute values ​​of the two reward values ​​are both greater than 0.5cm=0.005m and the positive and negative values ​​are the same as the previous r t Greater than zero and there is a subsequent r t If it suddenly becomes less than zero, a smaller concave feature vector D2 is generated, capturing the reward value r before and after the fourth point tThey are +0.021m and -0.022m respectively, and the absolute values ​​of the two reward values ​​are both greater than 0.005. Therefore, it is determined that the feature vector of the smaller concave abnormal point should be marked at the fourth detection point and stored in the DQN laser neural network. The following results are obtained:

[0124]

[0125] Step 4, laser sensor decision tree classification pattern recognition:

[0126] When the system no longer outputs new road conditions "up" or "down", it means that the decision tree is used to generate artificial labels. Use the decision tree for classification pattern recognition. The acquired abnormal point generates a small concave feature vector D2 in the DQN neural network. The specific pattern recognition method using the trained decision tree algorithm is as follows:

[0127] In the above laser data segment|r t | max =0.022m=2.2cm∈(2.0cm~5.0cm), so the road surface concave point category artificial label E3 is generated. The final pattern recognition result is output as "road surface concave point" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road surface concave point abnormal point" text document;

[0128] Step 5, gyroscope sensor DQN feature vector generation:

[0129] 5.1, start the gyroscope data analysis algorithm model that integrates the trained DQN and decision tree, and read the vertical acceleration a collected by the gyroscope sensor in the input data list Z List of columns in data set C 6 , that is, the 9th column of data in "Result_IMU_20240xxx" in the field data;

[0130] 5.2, set the initial quaternion: the first vertical acceleration a collected by the gyroscope sensor Z =21939, set as initial state s′ t Action a′ t Set to get the next vertical acceleration a Z+1 =15902, and make a difference Δa Z =a Z+1 -a Z =-6037, set reward value r' t =Δa Z =-6037, the system outputs the corresponding reward value r' t , and proceed to the next state s' t+1 Get, output a four-tuple (s't ,a' t ,r' t ,s' t+1 );

[0131] 5.3, Generate the corresponding feature vector of the outlier point: Output the corresponding result according to its positive or negative value. If r t >0, is a positive number, the vibration acceleration generated by the latter point is small, and the output is "rising", otherwise if r' t <0, which is a negative number. The vibration acceleration generated at the latter point is larger, and the output is "dropped". The abnormal point is at the point where the positive and negative mutation of the reward value occurs. The following steps are used to output a larger concave feature vector for the gyroscope test data:

[0132] Retrieve r′ from the two quadruples before and after the mutation point from the neural network t If two r′ t One is positive and the other is negative, and the positive and negative values ​​show that the previous reward value is 9009-15902=-6893, which is less than zero (negative), and the next reward value suddenly becomes 14605-9009=5596, which is greater than zero (positive). Then a larger concave feature vector D4 is generated and stored in the DQN gyroscope neural network, and the following results are obtained:

[0133]

[0134] Step 6, gyroscope sensor decision tree classification pattern recognition:

[0135] When the system no longer outputs the new road status "up" or "down", it means that the decision tree is used to generate artificial labels. Use decision trees for classification pattern recognition. The acquired abnormal point generates a large concave feature vector D4 in the DQN neural network. The trained decision tree algorithm is used for specific pattern recognition. In the above gyroscope data segment, |r t | max =6893>1000, so the final pattern recognition result is output as "road subsidence" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road subsidence abnormal point" text document;

[0136] Step 7, image classification pattern recognition:

[0137] Start the trained YOLO model, and filter out the pictures corresponding to all the abnormal points (named by timestamp) according to the abnormal point times in the two files of "road bump abnormal points" and "road settlement abnormal points". Retrieve the pictures corresponding to all abnormal points of the two abnormal types for YOLO image recognition. The picture pattern recognition corresponding to the abnormal points of the road concave points and road settlement obtained in the previous step is as follows:

[0138] Concave spots on the road surface: image recognition is required to distinguish broken bricks, abnormal cracks on manhole covers, common road obstacles, and convex or concave spots on the road surface. There are four categories in total. The images of the first three abnormal points are saved, and the time information and pattern recognition results of the corresponding abnormal points are classified and stored in the brick damage C′ 2 、Abnormal manhole cover C′ 3 , obstacle C′ 4 The remaining abnormal points that do not contain the above three categories are uniformly retained in the original image and the abnormal point collection time and pattern recognition results are stored in the abnormal point result data set C′ of the road surface convex or concave points. 5 middle;

[0139] There is no need to perform image recognition on road subsidence. The corresponding point time information and pattern recognition results are directly saved to the classification and saved to the corresponding abnormal point result data set C′ 8 Just in the middle;

[0140] Step 8: GIS batch deployment:

[0141] Match the time and longitude information of each abnormal point in the two categories of local road concave points and road subsidence, and display the points in different levels on the GIS map according to the longitude and latitude corresponding to the points and the recognition results;

[0142] Step 9, obtain the latitude and longitude of the starting and ending points of the sidewalk:

[0143] Find the latitude and longitude values ​​of the starting point and the ending point of the sidewalk where the sample data segment to be measured is located from Baidu Maps. If the sidewalk to be measured is north-south, obtain the latitude of the starting and ending points; if the sidewalk to be measured is east-west, obtain the longitude of the starting and ending points;

[0144] Step 10: Evaluate the overall smoothness of the observed pedestrian road section by section:

[0145] 10.1, according to the latitude and longitude of the starting point and the end point of a road section, the longitude and latitude of all test points and abnormal points are extracted and the road section area is divided;

[0146] 10.2, calculate the IRI value of each road section. Calculate the laser height difference H of each road section respectively K The mean value ε of each detection point is calculated, and the height H of each detection point from the ground is calculated. K The absolute value of the difference between the mean ε and the vertical displacement value of each point is obtained. Finally, the standard deviation of the vertical displacement values ​​of all detection points is calculated, which is the flatness standard deviation σ, and then the IRI is calculated:

[0147] 10.3, further calculate the sidewalk quality index FQI. Given the IRI value, the FQI can be calculated according to the following relationship:

[0148] FQI=4.98-0.34*IRI (1.5)

[0149] The value of FQI is 0-5, and the result is rounded off;

[0150] Judgment index: Grade A is excellent FQI = 5, Grade B is good FQI = 4, Grade C is average FQI = 3,

[0151] Grade D is poor FQI = 2, Grade E is extremely poor FQI = 0

[0152] 10.4, and obtain the overall evaluation of the smoothness of the pedestrian road section to be tested.

Claims

1. An intelligent detection method for sidewalk flatness based on multi-source data fusion analysis, characterized in that: The following steps are involved: Step 1, collect data: Determine the sidewalk section to be tested that needs to be evaluated for smoothness, use data collection equipment equipped with laser sensor TOF, gyroscope sensor IMU, GPS sensor GNSS and camera, collect data from the same detection point every 25 cm along the sidewalk section to be tested that needs to be evaluated for smoothness, and record the entire field video in real time; The data collected at the same detection point include: collection time t, height from the ground H K , vertical acceleration a Z , multi-source data information of the point's latitude and longitude, and on-site pictures of the detection point; Step 2: Clean the data: The height data H collected by the laser sensor K Referred to as laser data, K represents the number of the detection point in the laser data, and H in the laser data K ∈[10,+∞) is abnormal data; the vertical acceleration data a collected by the gyroscope sensor Z Referred to as gyroscope data, where Z represents the number of the detection point in the gyroscope data, and a in the gyroscope data Z ∈(-∞,0]∪[30000,+∞) is abnormal data; delete all the collected height data H K And all vertical acceleration data a Z Abnormal data in Step 3, prepare data segments and classification datasets: Step 4: Divide the test set and training set: For each classification data set C obtained in step 3 i Divide, i = 1, 2, 3, 4, 5, 6, let 3N i / 4 outlier segments are used as training set and N i / 4 abnormal point segments are test sets; Step 5: Train the laser data analysis DQN algorithm model: Step 6, training the laser data analysis decision tree algorithm model; Step 7: Train the gyroscope data analysis DQN algorithm model: Step 8, training the gyroscope data analysis decision tree algorithm model; Step 9, test the effect of DQN and decision tree fusion model: C i The corresponding abnormal point segment N i / 4 is the laser data analysis algorithm model of DQN and decision tree fusion generated by the test set input, i = 1, 2, 3, 4; the two types of data sets C i The corresponding abnormal point segment N i / 4 is the gyroscope data analysis algorithm model of DQN and decision tree fusion generated by the test set input, i=5,6; each category is tested for detection results, the model is verified, and the model accuracy is evaluated; if the accuracy meets the actual requirements, the model is saved; if the accuracy does not meet the requirements, the decision tree leaf node value is adjusted until the accuracy of the two models meets the actual requirements; Step 10: Train the image recognition neural network: Obtain relevant pictures of six categories, including blind paths, broken bricks, abnormal cracks on manhole covers, common obstacles, missing manhole covers, and normal manhole covers, mark the pictures, and input the marked pictures into the picture set to train the YOLO image recognition neural network, evaluate the recognition effect, and generate the final weight file; based on the two DQN and decision tree fusion algorithm models that can identify six types of unevenness problems, image recognition can refine the pattern recognition to identify the expected road bumps, obstacles, broken bricks, abnormal manhole covers, road bumps or road depressions, missing manhole covers, road height differences, road subsidence, and road arches, a total of 9 types of common sidewalk flatness problems; Step 11, generate the laser sensor DQN feature vector: Step 12, laser sensor decision tree classification pattern recognition: Step 13, generate the gyroscope sensor DQN feature vector: Step 13.1, input gyroscope sensor data: start the gyroscope data analysis algorithm model fused with DQN and decision tree trained in step 9, and input the vertical acceleration a of all monitoring points collected by the gyroscope sensor Z Total Dataset A a : DQN model training is based on the f(S,a) function, which can output the reward value in a certain state in real time. The f(S,a) idea is the difference-by-difference idea, and the difference is also set as the reward value output; Step 13.2, set the initial quaternion: set the first vertical acceleration a collected by the gyroscope sensor Z Set to the initial state s' t Action a' t Set to get the next vertical acceleration a Z+1 , and make a difference Δa Z =a Z+1 -a Z ; Set reward value r' t =Δa Z , the system outputs the corresponding reward value r' t , and proceed to the next state s' t+1 Get, output a four-tuple (s' t ,a' t ,r' t ,s' t+1 ); Step 13.3, generate the feature vector corresponding to the outlier point: output the corresponding result according to its positive or negative value. If r' t >0, is a positive number, the vibration acceleration generated by the latter point is small, and the output is "rising", on the contrary, if r' t <0, which is a negative number. The vibration acceleration generated by the latter point is larger, and the output "drops". The point where the positive and negative mutation of the reward value occurs is the abnormal point. The r' in the two quadruples before and after the mutation point is retrieved from the neural network. t ; if r' t One positive and one negative, and the former is negative, and the latter suddenly becomes positive, generating both positive and negative characteristics, and the positive and negative characteristics are less than zero in the early stage and suddenly become greater than zero, generating a larger concave feature vector D4, which is stored in the neural network; if r' t One is positive and the other is negative, and the former is positive and the latter suddenly becomes negative, which generates a larger convex feature vector D5 and stores it in the neural network; Step 14, gyroscope sensor decision tree classification pattern recognition: Step 15, image classification and recognition: Start the trained image YOLO model, filter out the images corresponding to all abnormal points according to the abnormal point moments in the 6 "xxx abnormal point" files generated in steps 12 and 14, and delete the normal point images; Step 16, obtain the pattern recognition result: The final output of common sidewalk unevenness types includes bumpy road surface, obstacles, broken bricks, abnormal manhole covers, convex or concave road surfaces, missing manhole covers, road height differences, road subsidence, and road bulges, a total of 9 categories; Step 17, GIS batch deployment: For 9 different categories C'={C'1,C'2,C'3,C'4,C'5,C'6,C'7,C'8,C'9}, we get the pattern recognition result dataset C' corresponding to each category. i (i=1,2,3,4,5,6,7,8,9); Match the time and longitude information corresponding to all points contained in the 9 abnormal point category result sets, and classify and layer the points on the GIS map according to the longitude and latitude corresponding to the points and the identification results, so as to obtain the purpose of fine-grained fixed-point identification and early warning of the flatness of the pedestrian road network; Step 18: Get the latitude and longitude of the starting and ending points of the sidewalk: Find the latitude and longitude values ​​of the starting and ending points of each sidewalk to be tested from Baidu Maps. If the sidewalk to be tested is north-south, obtain the latitude of the starting and ending points; if the sidewalk to be tested is east-west, obtain the longitude of the starting and ending points; Step 19, performing an overall smoothness assessment on the observed pedestrian road section.

2. According to claim 1, a method for intelligent detection of sidewalk flatness based on multi-source data fusion analysis is characterized in that: Step 3 includes, Step 3.1, prepare data segments: All cleaned height data H K The total data set formed is set as A H , A H The H in the data set indicates that all data in the data set are height data from the ground. All the cleaned vertical acceleration data a Z The total data set formed is set as A a , A a a in the data set indicates that all data in the data set are vertical acceleration data; find the flatness anomaly points on the sidewalk to be tested from the two data sets based on the full field video; the flatness anomaly points refer to points with six types of unevenness problems among all the detection points, namely, road bumps, road convex points, road concave points, road height differences, large road depressions and large road convexities, and record the collection time t corresponding to the flatness anomaly points respectively; retrieve a total of three data collected by the laser sensor and the gyroscope sensor at the flatness anomaly point and one detection point before and after the flatness anomaly point through the collection time t corresponding to the flatness anomaly point; according to the category to which the flatness anomaly point belongs, take these three detection point data as an anomaly segment of one of the six categories; find all the flatness anomaly points in turn according to the above method and obtain their corresponding anomaly segment; Step 3.2, prepare the classification data set: classify all the abnormal point fragments contained in each category according to the above 6 categories to construct the data set C i , the number of outlier fragments contained in each data set is set to N i , i represents the category number of the unevenness problem.

3. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 5 includes, Step 5.1, input laser sensor training data: the first four datasets C corresponding to the four categories of road bumps, road convex points, road concave points, and road height difference in step 4 i Corresponding to the divided training set 3N i / 4 copies of data are input into the laser data analysis DQN algorithm model, i=1,2,3,4; Step 5.2, train the laser data analysis DQN algorithm model: Step 5.2.1, set the initial four-tuple elements: According to the principle of DQN algorithm, initialize the state s t The state vector is set to the first distance height H in the first outlier segment of a certain category K value as the benchmark, K = 1, in state s t Next, perform action a t , t = 1, action a t The uniform setting is to obtain the next ground height H in the same abnormal point segment K+1 , K+1 represents the number of the next detection point, and the height data H of the next detection point from the ground K+1 The height data H of the current detection point from the ground K Make the difference and get ΔH as the current reward value r t , then get the next state s t+1 The next height above the ground is H K+1 Set to s t+1 , t represents the number of the state, t+1 represents the number of the next state; at this time, a four-tuple (s t , a t ,r t ,s t+1 ), which represent the current state, action, reward, and next state respectively, and are stored in the neural network memory pool; Step 5.2.2, determine the trend of actual road surface changes: according to the method of setting the initial four-tuple, traverse the other data in the same abnormal point segment of a certain category in turn; when the state s t When the traversal is completed in a certain abnormal point segment of the same category, a sample of the category is formed according to the two output quadruple groups; according to r t The positive and negative characteristics of the value and the laser sensor data analysis principle can analyze the change trend of the actual road surface: if r t <0, the actual road surface has an upward trend between the two detection points. t >0, the actual road surface has a downward trend between the two detection points; Machine learning captures r t The mutation point of the positive and negative value is determined as an abnormal point; the two corresponding r t , determine its positive or negative: If two r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. The former is negative and the latter suddenly becomes positive. A smaller convex feature vector D1 is generated and stored in the neural network. If r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. The former is positive and the latter suddenly becomes negative. A smaller concave feature vector D2 is generated and stored in the neural network. t If one has an absolute value less than 0.5 cm and the other is negative or positive and has an absolute value greater than 0.5 cm, a road elevation difference feature vector D3 is generated and stored in the neural network.

4. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 6 includes, Step 6.1, feature selection: The laser data analysis DQN algorithm model in step 5 is referred to as "DQN laser sensor neural network" to generate three types of feature vectors: small convexity D1, small concave D2, and road height difference D3. The number of feature vectors of each type is set to M j Part, j represents the sequence number of the feature vector, j = 1, 2, 3 input into the constructed decision tree model as feature selection; Step 6.2, training the laser data analysis decision tree algorithm model: Step 6.2.1, obtain the two quadruplets that form the above three types of feature vectors from the DQN laser neural network, and take the reward r in the two quadruplets t The absolute value with the largest value is |r t | max ,According to the actual measurement situation, 0.5cm, 2cm and 5cm are set as leaf node feature values ​​and input into the training decision tree model; Step 6.2.1.1, call each smaller convex feature vector D1 generated by the DQN neural network in turn, if |r t | max ∈(0.5cm~2.0cm), generate the manual label E1 of the road bump category; if |r t | max ∈(2.0cm~5.0cm), generate the manual labels E2 of the road surface convex points; Step 6.2.1.2, call each smaller concave feature vector D2 generated by the DQN neural network in turn. If |r t | max ∈(0.5cm~2.0cm), generate the manual label E1 of the road bump category; if |r t | max ∈(2.0cm~5.0cm), generate the manual labels of road surface concave points E3; Step 6.2.1.3, call each road elevation difference feature vector D3 generated by the DQN neural network in turn. If |r t | max ∈(5.0cm~1000cm), generate the large category manual label E4 of road elevation difference.

5. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 7 includes, Step 7.1, input the gyroscope sensor training set data: the dataset C corresponding to the two categories of large road bumps and large road depressions i The training set 3N is divided into i / 4 copies of data are input into the gyroscope data analysis DQN algorithm model, i=5,6; Step 7.2, train the gyroscope data analysis DQN algorithm model: Step 7.2.1, set the initial four-tuple elements: According to the DQN algorithm principle, initialize the state s' t Set to the first vertical acceleration value a in the first point segment included in C5 and C6 Z , Z=1; in state s' t Next, set the execution action a' t To obtain the next vertical acceleration value a in the same abnormal point segment of C5 and C6 Z+1 , and make a difference Z+1 -a Z , we get Δa Z As the reward value r' t ; At this time, get a state s' t+1 Set to the next vertical acceleration value a Z+1 ; Now we get a 4-tuple (s' t ,a' t ,r' t ,s' t+1 ), which represent the current state, action, reward, and next state respectively, and are stored in the neural network memory pool; Step 7.2.2, determine the trend of actual road surface changes: according to the method of initially setting the four-tuple, traverse the other data in the same abnormal point segment of a certain category in turn; when the state s' t When the traversal is completed in a certain abnormal point segment of the same category, a sample of this category is formed according to the two output quadruple groups; according to r' t The positive and negative characteristics of the value and the gyroscope sensor data analysis principle can be used to analyze the change trend of the actual road surface: if r' t <0, then the actual road surface has a downward trend between the two detection points, r' t > 0, it means that there is an upward trend between the two detection points on the actual road surface. Machine learning captures r' t The mutation point of the positive and negative value is judged as an abnormal point; call the two corresponding r' in the two quadruples contained in the same sample t , determine its positive or negative: If two r' t One positive and one negative, and the former is positive and the latter suddenly becomes negative, a larger concave feature vector D4 is generated and stored in the neural network; if r' t One is positive and the other is negative, and the former is negative, while the latter suddenly becomes positive, thus generating a larger convex feature vector D5, which is stored in the neural network.

6. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 8 includes, Step 8.1, feature selection: the gyroscope data analysis DQN algorithm model is referred to as DQN gyroscope sensor neural network; all feature vectors of the two categories of larger concave D4 and larger convex D5 generated in the DQN gyroscope sensor neural network are set to M for each category of feature vectors j , j = 5, 6, input the constructed decision tree algorithm model as feature selection; Step 8.2, train the gyroscope data analysis decision tree algorithm: Step 8.2.1, obtain the two quadruplets that form the above two types of feature vectors from the DQN gyroscope network, and take the reward value r' in the two quadruplets t The absolute value with the largest value is |r t |' max ,According to the previous measured situation, 1000 is set as the leaf node feature value and input into the training decision tree model; Step 8.2.1.1, call the feature vector D4 of each larger depression in the DQN gyroscope neural network in turn, if |r t |' max >1000, generate the road subsidence category manual label E5; if the data is not within this range, it is not considered as an abnormal point and outputs "no abnormality"; Step 8.2.1.2, call up the feature vector D5 of each larger bump in the DQN gyroscope neural network in turn, if |r t |' max >1000, generate the road arching category artificial label E6; data outside this range are not considered abnormal points and output "no abnormality".

7. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 11 includes, Step 11.1, input laser sensor data: start the laser data analysis algorithm model fused with DQN and decision tree trained in step 9, and input the total data set A of the height from the ground collected by the laser sensor H : In DQN model training, the reward value under a certain state can be output in real time according to the f(S,a) function. The f(S,a) idea is the difference-by-difference idea, and the difference is also set as the reward value output; Step 11.2, set the initial quaternion: According to the working principle of DQN, the first ground height H collected by the laser sensor is K+1 Set to the initial state s t , action a t Set to get the next height H from the ground K+1 , K = 1, and make a difference ΔH = H K+1 -H K ; Set reward value r t =ΔH, the reward value corresponding to the system output is r t , and proceed to the next state s t+1 Get, output a four-tuple (s t ,a t ,r t ,s t+1 ); Step 11.3, generate the feature vector corresponding to the abnormal point: According to the principle of laser sensor data analysis, it can be known that the result corresponding to the actual road surface undulation state can be output according to the positive and negative value of the reward value. If r t >0, is a positive number, the distance from the latter point to the ground is larger, the output is "down", on the contrary, if r t <0, which is a negative number. The distance from the latter point to the ground is small, and the output is "rising". At the mutation point of the positive and negative reward value, it indicates that there is an abnormal flatness point. The r of the two quadruplets before and after the mutation point is retrieved from the DQN neural network according to the sampling time. t , and determine the positive and negative: If two r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm and the former is positive, while the latter suddenly becomes negative. A smaller convex feature vector D1 is generated and stored in the neural network. If r t One is positive and the other is negative. The absolute values ​​of the two reward values ​​are both greater than 0.5 cm. If the former is negative and the latter suddenly becomes positive, a smaller concave feature vector D2 is generated and stored in the neural network. If r t If one absolute value is less than 0.5 cm and the other absolute value of the reward value is less than 0.5 cm and is negative or positive, a road height difference feature vector D3 is generated and stored in the neural network.

8. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 12 includes, Step 12.1, classification pattern recognition: When the system no longer outputs the new road state "up" or "down", it means that the decision tree is used to generate artificial labels and classification pattern recognition is started: Step 12.1.1: If a certain abnormal point obtained generates a small convex feature vector D1 in the laser data analysis DQN algorithm model, further retrieve the |r of the two quadruplets generated before and after the abnormal point t | max The value of is compared with the set leaf node; if it meets the |r t | max ∈(0.5cm~2.0cm) range, the final pattern recognition result will be output as "road bumps" and the acquisition time of the point, which will be automatically stored in the "road bumps abnormal point" text document; if it meets the |r t | max ∈(2.0cm~5.0cm) range, the final pattern recognition result will be output as "Road bump" and the acquisition time of the point, which will be automatically stored in the "Road bump" text document; if both do not meet the range, "No abnormality" will be output; Step 12.1.2: If a certain abnormal point obtained generates a small concave feature vector D2 in the laser data analysis DQN algorithm model, further retrieve the |r of the two quadruples generated before and after the abnormal point t | max The value of is compared with the set leaf node; if it meets the |r t | max ∈(0.5cm~2.0cm) range, the final pattern recognition result will be output as "road bumps" and the acquisition time of the point, which will be automatically stored in the "road bumps abnormal point" text document; if it meets the |r t | max ∈(2.0cm~5.0cm) range, the final pattern recognition result will be output as "Road Concave Point" and the acquisition time of the point, which will be automatically stored in the "Road Concave Point" text document; if both do not meet the range, "No abnormality" will be output; Step 12.1.3: If a road elevation difference feature vector D3 is generated in the laser data analysis DQN algorithm model for a certain abnormal point, the |r t | max The value of is compared with the set leaf node; if it meets the |r t | max ∈(5.0cm~1000cm) range, the final pattern recognition result is output as "road height difference" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road height difference abnormal point" text document; If none of them meet the range, "No exception" will be output.

9. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 14 includes, Step 14.1, classification pattern recognition: When the system no longer outputs the new road state "up" or "down", it means that the decision tree is used to generate artificial labels and classification pattern recognition is started: Step 14.1.1, if a certain abnormal point obtained generates a relatively large concave feature vector D4 in the gyro data analysis DQN algorithm model, further retrieve the |r' in the two quadruples generated before and after the abnormal point t | max The value of is compared with the set leaf node; if it meets the |r' t | max >1000, the final pattern recognition result is output as "road subsidence" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road subsidence abnormal point" text document; if both do not meet the range, "no abnormality" will be output; Step 14.1.2, if the acquired abnormal point generates a large convex feature vector D5 in the gyro data analysis DQN algorithm model, if it meets the |r' t | max >1000, the final pattern recognition result is output as "road bulge" and the acquisition time of the point, and the corresponding time of the abnormal point and the recognition result are automatically stored in the "road bulge abnormal point" text document; If none of them meet the range, "No exception" will be output.

10. The intelligent detection method for sidewalk flatness based on multi-source data fusion analysis according to claim 1 is characterized in that: Step 15 includes, Retrieve the images corresponding to the six categories of abnormal points for YOLO image recognition to further screen and refine the recognition result categories and improve the accuracy of pattern recognition. The image pattern recognition corresponding to the six categories of abnormal points obtained in the previous step is as follows: Step 15.1, for the field images of abnormal points of road bumps: distinguish road bumps from normal blind roads through image recognition, delete the detection point images with image recognition results of normal blind roads, retain all bumpy point images caused by road damage, and store the collection time data of road bump abnormal points and pattern recognition results in the road bump abnormal point result data set C'1; Step 15.2, for the field pictures of the two categories of road surface convex points and road surface concave points: image recognition is required to distinguish broken bricks, abnormal cracks on manhole covers, common road obstacles and road surface convex points or concave points, a total of 4 categories, the pictures taken at the first three abnormal points are saved, and the time information and pattern recognition results of the corresponding abnormal points are classified and stored in the abnormal point result data sets of broken bricks C'2, abnormal manhole covers C'3, and obstacles C'4; for the remaining abnormal points that do not contain the above three categories, the original pictures are uniformly retained and the abnormal point acquisition time and pattern recognition results are stored in the abnormal point result data set of road surface convex points or concave points C'5; Step 15.3, for the real-world images of the road elevation difference category: image recognition is required to distinguish between missing manhole covers, normal holes in manhole covers, and road elevation differences, a total of 3 categories. The point collection time information and pattern recognition results of the first type of missing manhole covers are saved in the missing manhole cover abnormal point result dataset C'6, and the real-world images of the abnormal points are retained; The second type of points identified as abnormal due to the voids in normal manhole covers are excluded and the corresponding real-life pictures are deleted; the remaining abnormal points that do not contain the above two types are uniformly retained in the original pictures and the corresponding point collection time information and pattern recognition results are saved in the abnormal point result data set C'7 stored in the road height difference; Step 15.4: For the two types of field pictures of road subsidence and road bulging, image recognition is not required. The corresponding point time information and pattern recognition results can be directly saved in the corresponding abnormal point result data sets C'8 and C'9; Step 19 includes, Step 19.1, extracting the longitude and latitude of all test points and abnormal points and dividing the road section area according to the longitude and latitude of the starting point and the end point of a road section; Step 19.2, calculate the IRI value of each road section; calculate the laser height H of each detection point on each road section K The mean value ε of each detection point is calculated, and the height H of each detection point from the ground is calculated. K The absolute value of the difference between the mean ε and the vertical displacement value of each point is obtained. Finally, the standard deviation of the vertical displacement values ​​of all detection points is calculated, which is the flatness standard deviation σ, and then the IRI is calculated: Step 19.3, calculate the sidewalk quality index FQI; given the IRI value, calculate the FQI according to the following relationship: FQI=4.98-0.34*IRI The value of FQI is 0-5; Judgment index: Grade A is excellent FQI = 5, Grade B is good FQI = 4, Grade C is average FQI = 3, Grade D means poor FQI = 2, and Grade E means extremely poor FQI = 0; Step 19.4, obtaining an overall evaluation of the smoothness of each section of the sidewalk to be tested.

Images