Traffic safety driving evaluation method based on road surface pit slot influence of digital twinning
Through digital twin technology and deep learning models, the impact of road pits on traffic safety is identified and evaluated, and the problem that the existing technology cannot accurately evaluate the impact of pits on traffic is solved, and the traffic safety driving evaluation and road maintenance optimization is achieved.
Patent Information
- Application Number
- CN202411594935.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2024-12-06
AI Technical Summary
The existing road surface pit trough detection methods cannot accurately identify and evaluate the specific impact of pit troughs on traffic safety, resulting in the inability to carry out road maintenance and repair work in a targeted manner, increasing the risk of road accidents and maintenance costs.
Using a traffic safety driving evaluation method based on digital twins, road surface images are collected through vehicle-mounted three-dimensional structured light cameras, combined with deep learning models and image processing technology, the impact of road surface pits is identified and evaluated, and traffic safety driving evaluation index is provided. .
Accurate assessment of the impact of pavement pits is achieved, helping to optimize road maintenance plans, reduce accident rates, and improve road safety and economic efficiency.
Smart Images

Figure CN119091264A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of traffic safety driving assessment, and specifically relates to a traffic safety driving assessment method based on digital twins for the impact of road potholes. Background Art
[0002] Potholes have a significant impact on road use and traffic. First, potholes can cause uneven driving, affect the vehicle's handling stability, and increase the risk of accidents. Drivers may make sudden lane changes or slow down to avoid potholes, which is particularly dangerous when driving at high speeds and can easily lead to chain reaction accidents. In addition, water accumulation in potholes increases the possibility of sliding, especially in rainy and snowy weather, increasing the risk of the vehicle skidding or losing control. Long-term pothole problems can also accelerate the wear of the vehicle's suspension system and increase vehicle maintenance costs. Therefore, potholes not only directly affect road driving safety, but also indirectly increase the economic burden of traffic operations.
[0003] The traffic safety driving assessment method under the influence of potholes can systematically analyze the specific impact of potholes on traffic flow and vehicle behavior, and identify high-risk areas and situations.
[0004] The invention patent with application number 202211373680.2 and invention name "A method for identifying and evaluating potholes in roads, electronic equipment and storage medium" describes a high-precision method for detecting and evaluating potholes in roads, which involves using a collection vehicle equipped with a global navigation positioning system, an inertial navigation system, an odometer, a camera and an edge computing module to collect pothole data. This method uses the UNet deep learning network structure to realize intelligent recognition and data extraction of pothole images in roads, and distinguishes pothole areas from non-pothole areas through pixel operation processing. It further uses the convolution processing method to extract edge pixel data, and finally calculates evaluation indicators such as the length, area and angle of the pothole, thereby improving the accuracy of detection and the efficiency of information extraction. However, there is no assessment method for the impact of potholes on traffic safety, and it is impossible to accurately identify and evaluate the specific impact of potholes on traffic safety, resulting in the inability to carry out road maintenance and repair work in a targeted manner, increasing the risk of road accidents. The lack of effective assessment methods also means that road managers are unable to optimize the investment of maintenance funds, and resources will be over- or under-allocated, resulting in some road problems that require emergency treatment not being solved in a timely manner, affecting the overall operational efficiency and safety of the road. Summary of the invention
[0005] The problem to be solved by the present invention is to accurately evaluate the traffic safety driving affected by road potholes, and propose a traffic safety driving evaluation method based on digital twins.
[0006] To achieve the above object, the present invention is implemented through the following technical solutions: A traffic safety driving assessment method based on digital twins for the impact of road potholes includes the following steps: S1. Using a vehicle-mounted three-dimensional structured light camera to collect a road image, and then performing image pixel update processing on the collected road image to obtain a processed road image; S2. performing comparative processing on the processed road surface image obtained in step S1, and extracting all road surface images containing road potholes and road surface images of the disease type most similar to the image features of road potholes; S3. Based on all the road images obtained in step S2, a data set for road pothole identification is constructed; S4. Construct a deep learning model for pothole recognition, including a basic model, an auxiliary model, and an enhanced model; S5. constructing a loss function between the output layers of the basic model, the auxiliary model, and the enhanced model, and then constructing an overall loss function between the basic model, the auxiliary model, and the enhanced model, and training the constructed basic model, the auxiliary model, and the enhanced model using the data set for road pothole recognition obtained in step S3, and obtaining the trained basic model as a deep learning model for road pothole recognition; S6. Build a digital twin model based on the deep learning model for pothole recognition; S7. Based on the digital twin model obtained in step S6, a traffic safety driving assessment method based on the impact of road potholes is established.
[0007] Furthermore, the specific implementation method of step S1 includes the following steps: S1. Use a vehicle-mounted 3D structured light camera to collect road images, and control the vehicle speed within 70km / h during the collection process; S1.2. compressing the collected road surface image using Huffman coding to obtain a compressed road surface image, and then performing plane projection to obtain a projected road surface image P; S1.3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , nu is the gray level number, nu =1,2,3,…,16; S1.4. Divide the projected road surface image P obtained in step S1.2 into N equal parts, and record the image matrix after equal division as Pnum, where num is the number of the image matrix after equal division, num=1,2,…,N; S1.5. Count each grayscale interval in the equally divided image The frequency of the corresponding pixel ; S1.6. Calculate each gray level in Pnum The corresponding grayscale frequency cumulative value , the calculation formula is: ; in, no is less than or equal to nu The gray level number, no =0,1,2,…, nu ; S1.7. Construct a pixel value update method, the calculation formula is: ; in, is the minimum non-zero value among all frequency accumulation values, The image after equal division P The total number of pixels corresponding to num, is the maximum value of the gray level, Grayscale The corresponding pixel update value; S1.8 Perform pixel value update processing on all equally divided images to obtain processed road surface images.
[0008] Furthermore, the specific implementation method of step S2 includes the following steps: S2.1. Similar defects of road potholes include longitudinal cracks, transverse cracks, cracking, rutting, bulging, spalling, oiling, subsidence, and undercutting; S2.2. Manually select the processed road surface images obtained in step S1, select all road surface images including potholes and similar diseases, and select the corresponding disease areas, and mark the road surface potholes, longitudinal cracks, transverse cracks, cracks, rutting, bumps, peeling, oil spills, subsidence, and edge gnawing areas in the road surface images in turn. PHO 1. PHO 2. PHO 3. PHO 4. PHO 5. PHO 6. PHO 7. PHO 8. PHO 9. PHO 10; S2.3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , nu =1,2,3,…,16; S2.4. Count the number of pixels corresponding to each gray level in the potholes Then, the number of pixels corresponding to each gray level of similar diseases of potholes such as longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oiling, subsidence, and edge biting is counted. , , , , , , , , ; S2.5. Standardize the number of pixels corresponding to each gray level to obtain the standard number of pixels, where the standard number of pixels corresponding to the potholes is , the calculation formula is: ; The standard number of pixels corresponding to longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oil spilling, subsidence, and edge biting is , , , , , , , , ; S2.6. As a benchmark, the relationship function between road potholes and other diseases is established in turn, and the calculation formula is: ; Among them, ii represents the serial number of other similar diseases, ii=2,3,4,5…10, and we get , , , , , , , , Respectively represent the relationship functions between road potholes and longitudinal cracks, transverse cracks, cracks, rutting, bumps, spalling, oiling, subsidence, and undercutting; S2.7. Comparison , , , , , , , , , and sort them in order from large to small, select the disease types corresponding to the top 3 relationship functions as the disease types most similar to the image features of road potholes, and then select all road surface images containing road potholes and the disease types most similar to the image features of road potholes for use.
[0009] Furthermore, the specific implementation method of step S3 includes the following steps: S3.1. By manually labeling, the pothole area and the other three diseases are selected from all the road images containing potholes and the disease types with the most similar image features to the potholes obtained in step S2, and the potholes are labeled as Pit, and the other three diseases are labeled as other1, other2, and other3 respectively, to obtain a data set for pothole recognition; S3.2. The dataset for pothole identification is divided into a training set and a test set in a ratio of 8:2.
[0010] Furthermore, the specific implementation method of step S4 includes the following steps: S4.1. Constructing a basic model for pothole identification: S4.1.1. The basic model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. S4.1.2. Input the training set obtained in step S3 into the basic model A, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the basic model A, and obtain the trained basic model A; The calculation formula of the loss function of the basic model A is: ; Among them, La is the loss function of the basic model A; Na is the total number of data samples in the basic model A; For the ia The true labels of samples, Based on the prediction of model A ia The probability that a sample is class 1; S4.1.3. The node output values of the four output units corresponding to the basic model A are a 1. a 2. a 3. a 4. Perform initial conversion on the node output values of the four output units to obtain the output values The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as ; S4.1.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the basic model A. After the training is completed, the output values of the four types of diseases corresponding to the basic model A are obtained, which are recorded as , , , ; S4.2. Constructing an auxiliary model for pothole identification: S4.2.1. Auxiliary model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer: S4.2.2. Input the training set obtained in step S3 into the auxiliary model B, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the auxiliary model B, and obtain the trained auxiliary model B; The calculation formula of the loss function of auxiliary model B is: ; Where Lb is the loss function of auxiliary model B; Nb is the total number of data samples in auxiliary model B; For the ib The true labels of samples, For auxiliary model B, predict ib The probability that a sample is class 1; S4.2.3. The node output values of the four output units corresponding to auxiliary model B are b 1. b 2. b 3. b 4. Perform initial conversion on the node output values of the four output units to obtain the output values b i The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value b i The corresponding enhancement probability value The calculation formula is: ; S4.2.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the auxiliary model B. After the training is completed, the disease output value corresponding to the auxiliary model B is obtained as , among the 4 nodes in the output layer, the actual disease corresponding to each node is marked as , i =1,2,3,4; S4.3. Building an enhanced model for pothole identification: S4.3.1. The enhanced model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers, and 1 output layer; S4.3.2. Input the training set obtained in step S3 into the enhanced model C, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the enhanced model C, and obtain the trained enhanced model C; The loss function of the reinforcement model C is calculated as: ; Where Lc is the loss function of the enhanced model C; Nc is the total number of data samples in the enhanced model C; For the ic The true labels of samples, To strengthen the model C to predict the ic The probability that a sample is class 1; S4.3.3. The node output values of the four output units corresponding to the enhanced model C are , the node output values of the four output units are initially converted to obtain The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as ; S4.3.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the enhanced model C. After the training is completed, the output values of the four types of diseases corresponding to the enhanced model C are obtained as follows: .
[0011] Furthermore, the specific implementation method of step S5 includes the following steps: S5.1. Based on and , establish the loss function between the auxiliary model B and the output layer of the basic model A, and the calculation formula is: ; based on and , construct the output value The corresponding probability value The loss function of the base model A , the calculation formula is: ; based on and , establish the relationship loss function between the auxiliary model B and the output layer of the basic model A , the calculation formula is: , in, for The weight coefficient of for The weight coefficient of S5.2. Establish the relationship loss function between the output layer of the reinforcement model C and the auxiliary model B , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function of Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.3. Establishing the relationship loss function between the output layer of the enhanced model C and the base model A , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function is Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.4. Define the overall loss function corresponding to the basic model A, auxiliary model B, and enhanced model C , the calculation formula is: ; S5.5. Using the data set for pothole recognition obtained in step S3, the basic model, auxiliary model and enhanced model are trained by error back propagation based on the chain rule. When the overall loss function obtained in step S4.7 is When is minimum, the corresponding model parameters are optimal, and the trained basic model A is a deep learning model for road pothole recognition.
[0012] Furthermore, the specific implementation method of step S6 includes the following steps: S6.1. Export the deep learning model for pothole recognition obtained in step S5 as a deployment file for independent services, including one of a pt file, a pth file, a json file, and a txt file; S6.2. Construct a road information model; S6.2.1. Collect information on the location, length, width, and layout of roads; S6.2.2. The data obtained in step S6.2.1 is stored in a database, and a basic three-dimensional model of the road is constructed, wherein the basic three-dimensional model of the road includes geometric information of the road; S6.2.3. Based on step S6.2.2, material properties, design specifications, and historical maintenance data are added to the basic three-dimensional model of the road to obtain a road information model; S6.2.4. Export the road information model to a format file supported by the web application, including a json file or a js file; S6.3. In the digital twin platform, import the deployment file of the deep learning model for pothole recognition obtained in step S5.1, and then import the road information model format file obtained in step S5.2; S6.4. Use Python to construct an API interface between image data and the deep learning model for pothole recognition, which is used for intelligent pothole recognition. Use Python to construct an API interface between the deep learning model for pothole recognition and the road information model to complete the construction of the digital twin model.
[0013] Furthermore, the specific implementation method of step S7 includes the following steps: S7.1. Use a 3D structured light camera to collect road surface data and upload it to the digital twin model established in step S6. Then use the deep learning model for pothole recognition deployed in the system to intelligently identify pothole areas in the road image, and select all images containing potholes to obtain pothole data. S7.2. Based on the images containing road potholes obtained in step S6.7, identify the number of road potholes in each image and calculate the total number of potholes NM ; S7.3. Construct a method for randomly selecting the number of pits and obtain the number of randomly selected pits The calculation formula is: ; in, The confidence level is selected z The score is obtained by looking up the z-value table; is an estimate of the proportion of units in a population that have a particular characteristic; is an acceptable error; S7.4. By number of pits Randomly draw pits and ditches, and after the drawing is completed, measure the size information of the pits and ditches on site and calculate the volume of the pits; Take the first pothole on the first road as an example and measure the length of the pothole Then divide the pit into 10 equal parts along its length, and measure the width of the section at each equal section. and depth ,in, Refers to the width of the ijth section of the first pit; Refers to the depth of the ijth section of the first pit, ij=1,2,3,…,9; Then calculate the pit volume V 1. The calculation formula is: ; Calculate all the first road The pit volume of the pit , , …, , then calculate the mean pothole volume of the first road : ; S7.5. Orientation NP roads, collect road images, and calculate all NP The average pothole volume corresponding to the road, , , …, ; Then calculate the average pothole volume corresponding to all roads , the calculation formula is: ; S7.6. Set traffic safety driving evaluation indicators, including the number of accidents, number of injured persons, number of vehicle damages, driving speed reduction value, and number of severe vehicle jolts; Get NP The number of accidents corresponding to the roads is recorded as: ; Get NP The number of injured people corresponding to the roads is recorded as: ; Get NP The number of vehicle damages corresponding to the roads is recorded as: ; Get NP The driving speed reduction values corresponding to the roads are recorded as follows: ; Get NP The number of severe vehicle bumps corresponding to the roads is recorded as: ; Then calculate the average number of severe bumps of the vehicle and record it as ; Then calculate the average value of each traffic safety driving evaluation index ZB: ; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB. That is, the average values of the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively ; S7.7. Based on the traffic safety driving evaluation index of step S7.6, the correlation relationship between the pothole and the traffic safety driving evaluation index is established as follows: Establish the correlation between potholes and traffic safety driving evaluation index ZB, ; in, jj for NP One of the roads, Indicates jj The average volume of potholes corresponding to the roads; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB; that is, the correlation between potholes and the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively , ; S7.8. Set critical values for the degree of correlation between potholes and the number of accidents, the number of injured persons, the number of vehicle damages, the value of reduced driving speed, and the number of severe vehicle jolts, which are respectively: ; S7.9. Compare the correlation between potholes and the number of accidents, number of injured persons, number of vehicle damages, speed reduction, and number of severe vehicle jolts to the critical value. ; make comparisons; when , , , , When any one of the following is true, it means that potholes have an impact on traffic safety.
[0014] Beneficial effects of the present invention: The traffic safety driving assessment method based on digital twins for the impact of potholes on the road surface described in the present invention, potholes on the road surface are usually irregularly shaped damages, and they have different sizes, depths and edge features on the road surface. In combination with the characteristics of potholes, the model structure is designed to include an input layer, a convolution layer, an activation layer, a pooling layer, a splicing layer, a convolution upsampling layer, and an output layer. In the model design structure, the convolution layer is used to extract image features at different levels, the activation layer is used to increase nonlinearity to identify complex pothole patterns, the pooling layer is used to reduce feature dimensions and enhance robustness to position changes, the splicing layer and the convolution upsampling layer are used to combine features of different resolutions, maintain edge details, and finally achieve accurate pixel-level pothole recognition through the output layer. This design allows the model to capture the subtle differences in potholes on the road surface while maintaining high computational efficiency and accuracy, and is suitable for processing complex road image data provided by on-board cameras.
[0015] The traffic safety driving assessment method based on digital twins for the impact of potholes in the road surface, and the method for constructing the output layer loss function of the basic model, auxiliary model and enhanced model described in the present invention can compress the model to the required capacity while ensuring the accuracy of pothole recognition, thereby increasing the operating speed and reducing the consumption of computing resources, while maintaining the recognition performance of the model.
[0016] The digital twin-based road pothole impact assessment method of the present invention combines digital twin technology with a road pothole rapid identification method, which can combine high-precision digital models with real-time data to achieve instant and accurate diagnosis of road conditions. By integrating sensor data, image recognition technology, and machine learning models, it is not only possible to detect and locate road potholes in a timely manner, but also to predict future damage development, thereby optimizing maintenance plans and resource allocation. This helps improve road safety, reduce maintenance costs, and extend road service life, which is of great value in improving public safety and economic efficiency.
[0017] The digital twin-based road pothole impact assessment method of traffic safety driving can provide scientific data support to help road managers identify the most dangerous pothole locations and prioritize repairs, thereby significantly reducing the accident rate. This method can help optimize budget allocation, ensure that maintenance funds are effectively used, and improve the economic efficiency of road maintenance. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a flow chart of the method for evaluating traffic safety driving based on the impact of road potholes based on digital twins described in the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.
[0020] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0021] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 The detailed instructions are as follows:
[0022] Embodiment 1: A traffic safety driving assessment method based on digital twins for the impact of road potholes includes the following steps: S1. Using a vehicle-mounted three-dimensional structured light camera to collect a road image, and then performing image pixel update processing on the collected road image to obtain a processed road image; Furthermore, the specific implementation method of step S1 includes the following steps: S1. Use a vehicle-mounted 3D structured light camera to collect road images, and control the vehicle speed within 70km / h during the collection process; S1.2. compressing the collected road surface image using Huffman coding to obtain a compressed road surface image, and then performing plane projection to obtain a projected road surface image P; S1.3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , nu is the gray level number, nu =1,2,3,…,16; S1.4. Divide the projected road surface image P obtained in step S1.2 into N equal parts, and record the image matrix after equal division as Pnum, where num is the number of the image matrix after equal division, num=1,2,…,N; S1.5. Count each grayscale interval in the equally divided image The frequency of the corresponding pixel ; S1.6. Calculate each gray level in Pnum The corresponding grayscale frequency cumulative value , the calculation formula is: ; in, no is less than or equal to nu The gray level number, no =0,1,2,…, nu ; S1.7. Construct a pixel value update method, the calculation formula is: ; in, is the minimum non-zero value among all frequency accumulation values, The image after equal division P The total number of pixels corresponding to num, is the maximum value of the gray level, Grayscale The corresponding pixel update value; S1.8 Perform pixel value update processing on all equally divided images to obtain processed road surface images.
[0023] S2. performing comparative processing on the processed road surface image obtained in step S1, and extracting all road surface images containing road potholes and road surface images of the disease type most similar to the image features of road potholes; Furthermore, the specific implementation method of step S2 includes the following steps: S2.1. Similar defects of road potholes include longitudinal cracks, transverse cracks, cracking, rutting, bulging, spalling, oiling, subsidence, and undercutting; S2.2. Manually select the processed road surface images obtained in step S1, select all road surface images including potholes and similar diseases, and select the corresponding disease areas, and mark the road surface potholes, longitudinal cracks, transverse cracks, cracks, rutting, bumps, peeling, oil spills, subsidence, and edge gnawing areas in the road surface images in turn. PHO 1. PHO 2. PHO 3. PHO 4. PHO 5. PHO 6. PHO 7. PHO 8. PHO 9. PHO 10; S2.3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , nu =1,2,3,…,16; S2.4. Count the number of pixels corresponding to each gray level in the potholes Then, the number of pixels corresponding to each gray level of similar diseases of potholes such as longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oiling, subsidence, and edge biting is counted. , , , , , , , , ; S2.5. Standardize the number of pixels corresponding to each gray level to obtain the standard number of pixels, where the standard number of pixels corresponding to the potholes is , the calculation formula is: ; The standard number of pixels corresponding to longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oil spilling, subsidence, and edge biting is , , , , , , , , ; S2.6. As a benchmark, the relationship function between road potholes and other diseases is established in turn, and the calculation formula is: ; Among them, ii represents the serial number of other similar diseases, ii=2,3,4,5…10, and we get , , , , , , , , Respectively represent the relationship functions between road potholes and longitudinal cracks, transverse cracks, cracks, rutting, bumps, spalling, oiling, subsidence, and undercutting; S2.7. Comparison , , , , , , , , , and sort them in order from large to small, select the disease types corresponding to the top 3 relationship functions as the disease types most similar to the image features of road potholes, and then select all road surface images containing road potholes and the disease types most similar to the image features of road potholes for use.
[0024] S3. Based on all the road images obtained in step S2, a data set for road pothole identification is constructed; Furthermore, the specific implementation method of step S3 includes the following steps: S3.1. By manually labeling, the pothole area and the other three diseases are selected from all the road images containing potholes and the disease types with the most similar image features to the potholes obtained in step S2, and the potholes are labeled as Pit, and the other three diseases are labeled as other1, other2, and other3 respectively, to obtain a data set for pothole recognition; S3.2. The dataset for pothole identification is divided into a training set and a test set in a ratio of 8:2.
[0025] S4. Construct a deep learning model for pothole recognition, including a basic model, an auxiliary model, and an enhanced model; Furthermore, the specific implementation method of step S4 includes the following steps: S4.1. Constructing a basic model for pothole identification: S4.1.1. The basic model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3; number of convolution kernels 32; stride: 1; Activation layer 1: The activation function is LeakyReLU; Convolutional layer 2: convolution kernel size 3×3; number of convolution kernels 32; Stride: 1; Activation layer 2: The activation function is LeakyReLU; Pooling layer 1: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 3: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 3: The activation function is LeakyReLU; Convolutional layer 4: convolution kernel size 3×3; number of convolution kernels 64; Stride: 1; Activation layer 4: The activation function is LeakyReLU; Pooling layer 2: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 5: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 5: The activation function is LeakyReLU; Convolution layer 6: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 6: The activation function is LeakyReLU; Pooling layer 3: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 7: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 7: The activation function is LeakyReLU; Convolution layer 8: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 8: The activation function is LeakyReLU; Pooling layer 4: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 9: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 9: The activation function is LeakyReLU; Convolution layer 10: kernel size 3×3; number of kernels 512; stride: 1; Activation layer 10: The activation function is LeakyReLU; Pooling layer 5: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 512; Convolution layer 11: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 11: The activation function is LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 256; Convolution layer 12: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 12: The activation function is LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 128; Convolution layer 13: kernel size 3×3; number of kernels 64; stride: 1; Activation layer 13: The activation function is LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32; Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 64; Convolution layer 14: kernel size 3×3; number of kernels 32; stride: 1; Activation layer 14: the activation function is LeakyReLU; Output layer: type is normalized exponential function Softmax; number of neurons: 4; S4.1.2. Input the training set obtained in step S3 into the basic model A, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the basic model A, and obtain the trained basic model A; The calculation formula of the loss function of the basic model A is: ; Among them, La is the loss function of the basic model A; Na is the total number of data samples in the basic model A; For the ia The true labels of samples, Based on the prediction of model A ia The probability that a sample is class 1; S4.1.3. The node output values of the four output units corresponding to the basic model A are a 1. a 2. a 3. a 4. Perform initial conversion on the node output values of the four output units to obtain the output values The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as , specifically, for potholes in the road 1, for other diseases is 0; S4.1.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the basic model A. After the training is completed, the output values of the four types of diseases corresponding to the basic model A are obtained, which are recorded as , , , ; S4.2. Constructing an auxiliary model for pothole identification: S4.2. Constructing an auxiliary model for pothole identification: S4.2.1. Auxiliary model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3; number of convolution kernels 64; Stride: 1; Activation layer 1: The activation function is LeakyReLU; Convolutional layer 2: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 2: The activation function is LeakyReLU; Pooling layer 1: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 3: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 3: The activation function is LeakyReLU; Convolutional layer 4: convolution kernel size 3×3; number of convolution kernels 128; Stride: 1; Activation layer 4: The activation function is LeakyReLU; Pooling layer 2: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 5: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 5: The activation function is LeakyReLU; Convolution layer 6: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 6: The activation function is LeakyReLU; Pooling layer 3: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 7: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 7: The activation function is LeakyReLU; Convolution layer 8: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 8: The activation function is LeakyReLU; Pooling layer 4: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 9: convolution kernel size 3×3; number of convolution kernels 1024; stride: 1; Activation layer 9: The activation function is LeakyReLU; Convolution layer 10: kernel size 3×3; number of kernels 1024; stride: 1; Activation layer 10: The activation function is LeakyReLU; Pooling layer 5: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512; Concatenation layer 1: connects the feature map with the output of activation layer 8, and the number of channels increases to 1024; Convolution layer 11: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 11: The activation function is LeakyReLU; Convolution layer 12: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 12: The activation function is LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 2: connects the feature map with the output of activation layer 6, and the number of channels increases to 512; Convolution layer 13: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 13: The activation function is LeakyReLU; Convolution layer 14: kernel size 3×3; number of kernels 256; stride: 1; Activation layer 14: the activation function is LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 3: connects the feature map with the output of activation layer 4, and the number of channels increases to 256; Convolution layer 15: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 15: The activation function is LeakyReLU; Convolution layer 16: kernel size 3×3; number of kernels 128; stride: 1; Activation layer 16: The activation function is LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 4: connects the feature map with the output of activation layer 2, and the number of channels increases to 128; Convolution layer 17: kernel size 3×3; number of kernels 64; stride: 1; Activation layer 17: The activation function is LeakyReLU; Convolution layer 18: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 18: The activation function is LeakyReLU; Output layer: type is normalized exponential function Softmax; number of neurons: 4; S4.2.2. Input the training set obtained in step S3 into the auxiliary model B, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the auxiliary model B, and obtain the trained auxiliary model B; The calculation formula of the loss function of auxiliary model B is: ; Where Lb is the loss function of auxiliary model B; Nb is the total number of data samples in auxiliary model B; For the ib The true labels of samples, For auxiliary model B, predict ib The probability that a sample is class 1; S4.2.3. The node output values of the four output units corresponding to auxiliary model B are b 1. b 2. b 3. b 4. Perform initial conversion on the node output values of the four output units to obtain the output values b i The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value b i The corresponding enhancement probability value The calculation formula is: ; S4.2.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the auxiliary model B. After the training is completed, the disease output value corresponding to the auxiliary model B is obtained as , among the 4 nodes in the output layer, the actual disease corresponding to each node is marked as , i =1,2,3,4; Specifically, for potholes in the road 1, for other diseases is 0; S4.3. Building an enhanced model for pothole identification: S4.3.1. The enhanced model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: Input layer: image size 227×227×3; Convolution layer 1: convolution kernel size 3×3; number of convolution kernels 32; Stride: 1; Activation layer 1: The activation function is LeakyReLU; Convolutional layer 2: convolution kernel size 3×3; number of convolution kernels 32; Stride: 1; Activation layer 2: The activation function is LeakyReLU; Pooling layer 1: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 3: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 3: The activation function is LeakyReLU; Convolutional layer 4: convolution kernel size 3×3; number of convolution kernels 64; Stride: 1; Activation layer 4: The activation function is LeakyReLU; Pooling layer 2: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 5: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 5: The activation function is LeakyReLU; Convolution layer 6: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 6: The activation function is LeakyReLU; Pooling layer 3: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 7: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 7: The activation function is LeakyReLU; Convolution layer 8: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 8: The activation function is LeakyReLU; Pooling layer 4: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 9: convolution kernel size 3×3; number of convolution kernels 512; stride: 1; Activation layer 9: The activation function is LeakyReLU; Convolution layer 10: kernel size 3×3; number of kernels 512; stride: 1; Activation layer 10: The activation function is LeakyReLU; Pooling layer 5: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution layer 11: convolution kernel size 3×3; number of convolution kernels 1024; stride: 1; Activation layer 11: The activation function is LeakyReLU; Convolution layer 12: kernel size 3×3; number of kernels 1024; stride: 1; Activation layer 12: The activation function is LeakyReLU; Pooling layer 6: Type is maximum pooling; pooling size: 2×2; Stride: 2; Convolution upsampling 1: Use 2×2 transposed convolution upsampling, and the output channels are halved to 512; Concatenation layer 1: connects the feature map with the output of activation layer 10, and the number of channels increases to 1024; Convolution layer 13: kernel size 3×3; number of kernels 512; stride: 1; Activation layer 13: The activation function is LeakyReLU; Convolution layer 14: kernel size 3×3; number of kernels 512; stride: 1; Activation layer 14: the activation function is LeakyReLU; Convolution upsampling 2: Use 2×2 transposed convolution upsampling, and the output channels are halved to 256; Concatenation layer 2: connects the feature map with the output of activation layer 8, and the number of channels increases to 512; Convolution layer 15: kernel size 3×3; number of kernels 256; stride: 1; Activation layer 15: The activation function is LeakyReLU; Convolution layer 16: convolution kernel size 3×3; number of convolution kernels 256; stride: 1; Activation layer 16: The activation function is LeakyReLU; Convolution upsampling 3: Use 2×2 transposed convolution upsampling, and the output channels are halved to 128; Concatenation layer 3: connects the feature map with the output of activation layer 6, and the number of channels increases to 256; Convolution layer 17: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 17: The activation function is LeakyReLU; Convolution layer 18: convolution kernel size 3×3; number of convolution kernels 128; stride: 1; Activation layer 18: The activation function is LeakyReLU; Convolution upsampling 4: Use 2×2 transposed convolution upsampling, and the output channels are halved to 64; Concatenation layer 4: connects the feature map with the output of activation layer 4, and the number of channels increases to 128; Convolution layer 19: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 19: The activation function is LeakyReLU; Convolutional layer 20: convolution kernel size 3×3; number of convolution kernels 64; stride: 1; Activation layer 20: The activation function is LeakyReLU; Convolution upsampling 5: Use 2×2 transposed convolution upsampling, and the output channels are halved to 32; Concatenation layer 5: connects the feature map with the output of activation layer 2, and the number of channels increases to 64; Convolution layer 21: convolution kernel size 3×3; number of convolution kernels 32; stride: 1; Activation layer 21: The activation function is LeakyReLU; Convolutional layer 22: convolution kernel size 3×3; number of convolution kernels 32; stride: 1; Activation layer 22: The activation function is LeakyReLU; Output layer: type is normalized exponential function Softmax; number of neurons: 4; S4.3.2. Input the training set obtained in step S3 into the enhanced model C, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the enhanced model C, and obtain the trained enhanced model C; The loss function of the reinforcement model C is calculated as: ; Where Lc is the loss function of the enhanced model C; Nc is the total number of data samples in the enhanced model C; For the ic The true labels of samples, To strengthen the model C to predict the ic The probability that a sample is class 1; S4.3.3. The node output values of the four output units corresponding to the enhanced model C are , the node output values of the four output units are initially converted to obtain The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as , specifically, for potholes in the road 1, for other diseases is 0; S4.3.4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the enhanced model C. After the training is completed, the output values of the four types of diseases corresponding to the enhanced model C are obtained as follows: .
[0026] S5. constructing a loss function between the output layers of the basic model, the auxiliary model, and the enhanced model, and then constructing an overall loss function between the basic model, the auxiliary model, and the enhanced model, and training the constructed basic model, the auxiliary model, and the enhanced model using the data set for road pothole recognition obtained in step S3, and obtaining the trained basic model as a deep learning model for road pothole recognition; Furthermore, the specific implementation method of step S5 includes the following steps: S5.1. Based on and , establish the loss function between the auxiliary model B and the output layer of the basic model A, and the calculation formula is: ; based on and , construct the output value The corresponding probability value The loss function of the base model A , the calculation formula is: ; based on and , establish the relationship loss function between the auxiliary model B and the output layer of the basic model A , the calculation formula is: , in, for The weight coefficient of for The weight coefficient of S5.2. Establish the relationship loss function between the output layer of the reinforcement model C and the auxiliary model B , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function of Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.3. Establishing the relationship loss function between the output layer of the enhanced model C and the base model A , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function is Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.4. Define the overall loss function corresponding to the basic model A, auxiliary model B, and enhanced model C , the calculation formula is: ; S5.5. Using the data set for pothole recognition obtained in step S3, the basic model, auxiliary model and enhanced model are trained by error back propagation based on the chain rule. When the overall loss function obtained in step S4.7 is When is minimum, the corresponding model parameters are optimal, and the trained basic model A is a deep learning model for road pothole recognition.
[0027] S6. Build a digital twin model based on the deep learning model for pothole recognition; The specific implementation method of step S6 includes the following steps: S6.1. Export the deep learning model for pothole recognition obtained in step S5 as a deployment file for independent services, including one of a pt file, a pth file, a json file, and a txt file; S6.2. Construct a road information model; S6.2.1. Collect information on the location, length, width, and layout of roads; S6.2.2. The data obtained in step S6.2.1 is stored in a database, and a basic three-dimensional model of the road is constructed, wherein the basic three-dimensional model of the road includes geometric information of the road; S6.2.3. Based on step S6.2.2, material properties, design specifications, and historical maintenance data are added to the basic three-dimensional model of the road to obtain a road information model; S6.2.4. Export the road information model to a format file supported by the web application, including a json file or a js file; S6.3. In the digital twin platform, import the deployment file of the deep learning model for pothole recognition obtained in step S5.1, and then import the road information model format file obtained in step S5.2; S6.4. Use Python to build an API interface between image data and a deep learning model for pothole recognition, for intelligent pothole recognition; Use Python to build an API interface between a deep learning model for pothole recognition and a road information model, to complete the construction of a digital twin model; S7. Based on the digital twin model obtained in step S6, a traffic safety driving assessment method based on the impact of road potholes is established.
[0028] Furthermore, the specific implementation method of step S7 includes the following steps: S7.1. Use a 3D structured light camera to collect road surface data and upload it to the digital twin model established in step S6. Then use the deep learning model for pothole recognition deployed in the system to intelligently identify pothole areas in the road image, and select all images containing potholes to obtain pothole data. S7.2. Based on the images containing road potholes obtained in step S6.7, identify the number of road potholes in each image and calculate the total number of potholes NM ; S7.3. Construct a method for randomly selecting the number of pits and obtain the number of randomly selected pits The calculation formula is: ; in, The confidence level is selected z The score is obtained by looking up the z-value table; is an estimate of the proportion of units in a population that have a particular characteristic; is an acceptable error; S7.4. By number of pits Randomly draw pits and ditches, and after the drawing is completed, measure the size information of the pits and ditches on site and calculate the volume of the pits; Take the first pothole on the first road as an example and measure the length of the pothole Then divide the pit into 10 equal parts along its length, and measure the width of the section at each equal section. and depth ,in, Refers to the width of the ijth section of the first pit; Refers to the depth of the ijth section of the first pit, ij=1,2,3,…,9; Then calculate the pit volume V 1. The calculation formula is: ; Calculate all the first road The pit volume of the pit , , …, , then calculate the mean pothole volume of the first road : ; S7.5. Orientation NP roads, collect road images, and calculate all NP The average pothole volume corresponding to the road, , , …, ; Then calculate the average pothole volume corresponding to all roads , the calculation formula is: ; S7.6. Set traffic safety driving evaluation indicators, including the number of accidents, number of injured persons, number of vehicle damages, driving speed reduction value, and number of severe vehicle jolts; Get NP The number of accidents corresponding to the roads is recorded as: ; Get NP The number of injured people corresponding to the roads is recorded as: ; Get NP The number of vehicle damages corresponding to the roads is recorded as: ; Get NP The driving speed reduction values corresponding to the roads are recorded as follows: ; Get NP The number of severe vehicle bumps corresponding to the roads is recorded as: ; Then calculate the average number of severe bumps of the vehicle and record it as ; Then calculate the average value of each traffic safety driving evaluation index ZB: ; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB. That is, the average values of the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively ; S7.7. Based on the traffic safety driving evaluation index of step S7.6, the correlation relationship between the pothole and the traffic safety driving evaluation index is established as follows: Establish the correlation between potholes and traffic safety driving evaluation index ZB, ; in, jj for NP One of the roads, Indicates jjThe average volume of potholes corresponding to the roads; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB; that is, the correlation between potholes and the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively , ; S7.8. Set critical values for the degree of correlation between potholes and the number of accidents, the number of injured persons, the number of vehicle damages, the value of reduced driving speed, and the number of severe vehicle jolts, which are respectively: ; S7.9. Compare the correlation between potholes and the number of accidents, number of injured persons, number of vehicle damages, speed reduction, and number of severe vehicle jolts to the critical value. ; make comparisons; when , , , , When any one of the following is true, it means that potholes have an impact on traffic safety.
[0029] The key points and intended protection points of the present invention are: (1) A method for identifying similar diseases; (2) A method for constructing the output layer loss function of the basic model, auxiliary model and enhanced model.
[0030] (3) A method for evaluating traffic safety under the influence of potholes on the road.
[0031] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0032] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A traffic safety driving assessment method based on digital twins for the impact of road potholes, characterized in that: The steps include: S1. Using a vehicle-mounted three-dimensional structured light camera to collect a road image, and then performing image pixel update processing on the collected road image to obtain a processed road image; S2. performing comparative processing on the processed road surface image obtained in step S1, and extracting all road surface images containing road potholes and road surface images of the disease type most similar to the image features of road potholes; S3. Based on all the road images obtained in step S2, a data set for road pothole identification is constructed; S4. Construct a deep learning model for pothole recognition, including a basic model, an auxiliary model, and an enhanced model; S5. constructing a loss function between the output layers of the basic model, the auxiliary model, and the enhanced model, and then constructing an overall loss function between the basic model, the auxiliary model, and the enhanced model, and training the constructed basic model, the auxiliary model, and the enhanced model using the data set for road pothole recognition obtained in step S3, and obtaining the trained basic model as a deep learning model for road pothole recognition; S6. Build a digital twin model based on the deep learning model for pothole recognition; S7. Based on the digital twin model obtained in step S6, a traffic safety driving assessment method based on the impact of road potholes is established.
2. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1. Use a vehicle-mounted 3D structured light camera to collect road images, and control the vehicle speed within 70km / h during the collection process; S1.
2. compressing the collected road surface image using Huffman coding to obtain a compressed road surface image, and then performing plane projection to obtain a projected road surface image P; S1.
3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , NU is the gray level number, NU =1,2,3,…,16; S1.
4. Divide the projected road surface image P obtained in step S1.2 into N equal parts, and record the image matrix after equal division as Pnum, where num is the number of the image matrix after equal division, num=1,2,…,N; S1.
5. Count each grayscale interval in the equally divided image The frequency of the corresponding pixel ; S1.
6. Calculate each gray level in Pnum The corresponding gray frequency cumulative value , the calculation formula is: ; in, no is less than or equal to NU The gray level number, no =0,1,2,…, NU ; S1.
7. Construct a pixel value update method, the calculation formula is: ; in, is the minimum non-zero value among all frequency accumulation values, The image after equal division P The total number of pixels corresponding to num, is the maximum value of the gray level, Grayscale The corresponding pixel update value; S1.8 Perform pixel value update processing on all equally divided images to obtain processed road surface images.
3. A method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 1 or 2, characterized in that: The specific implementation method of step S2 includes the following steps: S2.
1. Similar defects of road potholes include longitudinal cracks, transverse cracks, cracking, rutting, bulging, spalling, oiling, subsidence, and undercutting; S2.
2. Manually select the processed road surface images obtained in step S1, select all road surface images including potholes and similar diseases, and select the corresponding disease areas, and mark the road surface potholes, longitudinal cracks, transverse cracks, cracks, rutting, bumps, peeling, oil spills, subsidence, and edge gnawing areas in the road surface images in turn. PHO 1. PHO 2. PHO 3. PHO 4. PHO 5. PHO 6. PHO 7. PHO 8. PHO 9. PHO 10; S2.
3. Divide the pixel grayscale value 0~255 into 16 grayscale intervals, denoted as , NU =1,2,3,…,16; S2.
4. Count the number of pixels corresponding to each gray level in the potholes Then, the number of pixels corresponding to each gray level of similar diseases of potholes such as longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oiling, subsidence, and edge biting is counted. , , , , , , , , ; S2.
5. Standardize the number of pixels corresponding to each gray level to obtain the standard number of pixels, where the standard number of pixels corresponding to the potholes is , the calculation formula is: ; The standard number of pixels corresponding to longitudinal cracks, transverse cracks, cracks, rutting, bulging, peeling, oil spilling, subsidence, and edge biting is , , , , , , , , ; S2.
6. As a benchmark, the relationship function between road potholes and other diseases is established in turn, and the calculation formula is: ; Among them, ii represents the serial number of other similar diseases, ii=2,3,4,5…10, and we get , , , , , , , , Respectively represent the relationship functions between road potholes and longitudinal cracks, transverse cracks, cracks, rutting, bumps, spalling, oiling, subsidence, and undercutting; S2.
7. Comparison , , , , , , , , , and sort them in order from large to small, select the disease types corresponding to the top 3 relationship functions as the disease types most similar to the image features of road potholes, and then select all road surface images containing road potholes and the disease types most similar to the image features of road potholes for use.
4. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 3 is characterized in that: The specific implementation method of step S3 includes the following steps: S3.
1. By manually labeling, the pothole area and the other three diseases are selected from all the road images containing potholes and the disease types with the most similar image features to the potholes obtained in step S2, and the potholes are labeled as Pit, and the other three diseases are labeled as other1, other2, and other3 respectively, to obtain a data set for pothole recognition; S3.
2. The dataset for pothole identification is divided into a training set and a test set in a ratio of 8:
2.
5. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 4 is characterized in that: The specific implementation method of step S4 includes the following steps: S4.
1. Constructing a basic model for pothole identification: S4.1.
1. The basic model A consists of 1 input layer, 14 convolutional layers, 14 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: S4.1.
2. Input the training set obtained in step S3 into the basic model A, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the basic model A, and obtain the trained basic model A; The calculation formula of the loss function of the basic model A is: ; Among them, La is the loss function of the basic model A; Na is the total number of data samples in the basic model A; For the ia The true labels of samples, Based on the prediction of model A ia The probability that a sample is class 1; S4.1.
3. The node output values of the four output units corresponding to the basic model A are a 1. a 2. a 3. a 4. Perform initial conversion on the node output values of the four output units to obtain the output values The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as ; S4.1.
4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the basic model A. After the training is completed, the output values of the four types of diseases corresponding to the basic model A are obtained, which are recorded as , , , ; S4.
2. Constructing an auxiliary model for pothole identification: S4.2.
1. Auxiliary model B consists of 1 input layer, 18 convolutional layers, 18 activation layers, 5 pooling layers, 4 concatenation layers, 4 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: S4.2.
2. Input the training set obtained in step S3 into the auxiliary model B, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the auxiliary model B, and obtain the trained auxiliary model B; The calculation formula of the loss function of auxiliary model B is: ; Where Lb is the loss function of auxiliary model B; Nb is the total number of data samples in auxiliary model B; For the ib The true labels of samples, For auxiliary model B, predict ib The probability that a sample is class 1; S4.2.
3. The node output values of the four output units corresponding to auxiliary model B are b 1. b 2. b 3. b 4. Perform initial conversion on the node output values of the four output units to obtain the output values b i The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value b i The corresponding enhancement probability value The calculation formula is: ; S4.2.
4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the auxiliary model B. After the training is completed, the disease output value corresponding to the auxiliary model B is obtained as , among the 4 nodes in the output layer, the actual disease corresponding to each node is marked as , i =1,2,3,4; S4.
3. Building an enhanced model for pothole identification: S4.3.
1. The enhanced model C consists of 1 input layer, 22 convolutional layers, 22 activation layers, 6 pooling layers, 5 concatenation layers, 5 convolutional upsampling layers, and 1 output layer. The specific network structure is the following structure connected in sequence: S4.3.
2. Input the training set obtained in step S3 into the enhanced model C, calculate the weights and biases of the neural network units through the error back propagation algorithm, complete the training of the enhanced model C, and obtain the trained enhanced model C; The loss function of the reinforcement model C is calculated as: ; Where Lc is the loss function of the enhanced model C; Nc is the total number of data samples in the enhanced model C; For the ic The true labels of samples, To strengthen the model C to predict the ic The probability that a sample is class 1; S4.3.
3. The node output values of the four output units corresponding to the enhanced model C are , the node output values of the four output units are initially converted to obtain The corresponding probability value The calculation formula is: ; Then the node output values of the four output units are further optimized and converted to obtain the output value The corresponding enhancement probability value The calculation formula is: ; in, G is the enhancement coefficient, and then the actual disease corresponding to each node is marked as ; S4.3.
4. Adoption The training set obtained in step S3 is combined with the error chain derivation rule to train the enhanced model C. After the training is completed, the output values of the four types of diseases corresponding to the enhanced model C are obtained as follows: .
6. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 5 is characterized in that: The specific implementation method of step S5 includes the following steps: S5.
1. Based on and , establish the loss function between the auxiliary model B and the output layer of the basic model A, and the calculation formula is: ; based on and , construct the output value The corresponding probability value The loss function of the base model A , the calculation formula is: ; based on and , establish the relationship loss function between the auxiliary model B and the output layer of the basic model A , the calculation formula is: , in, for The weight coefficient of for The weight coefficient of S5.
2. Establish the relationship loss function between the output layer of the reinforcement model C and the auxiliary model B , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function of Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.
3. Establishing the relationship loss function between the output layer of the enhanced model C and the base model A , the expression is: ; in, for The weight coefficient of for The weight coefficient of Based on and The loss function is Based on and The loss function of The calculation formula is: ; The calculation formula is: ; S5.
4. Define the overall loss function corresponding to the basic model A, auxiliary model B, and enhanced model C , the calculation formula is: ; S5.
5. Using the data set for pothole recognition obtained in step S3, the basic model, auxiliary model and enhanced model are trained by error back propagation based on the chain rule. When the overall loss function obtained in step S4.7 is When is minimum, the corresponding model parameters are optimal, and the trained basic model A is a deep learning model for road pothole recognition.
7. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 6, characterized in that: The specific implementation method of step S6 includes the following steps: S6.
1. Export the deep learning model for pothole recognition obtained in step S5 as a deployment file for independent services, including one of a pt file, a pth file, a json file, and a txt file; S6.
2. Construct a road information model; S6.2.
1. Collect information on the location, length, width, and layout of roads; S6.2.
2. The data obtained in step S6.2.1 is stored in a database, and a basic three-dimensional model of the road is constructed, wherein the basic three-dimensional model of the road includes geometric information of the road; S6.2.
3. Based on step S6.2.2, material properties, design specifications, and historical maintenance data are added to the basic three-dimensional model of the road to obtain a road information model; S6.2.
4. Export the road information model to a format file supported by the web application, including a json file or a js file; S6.
3. In the digital twin platform, import the deployment file of the deep learning model for pothole recognition obtained in step S5.1, and then import the road information model format file obtained in step S5.2; S6.
4. Use Python to construct an API interface between image data and the deep learning model for pothole recognition, which is used for intelligent pothole recognition. Use Python to construct an API interface between the deep learning model for pothole recognition and the road information model to complete the construction of the digital twin model.
8. The method for evaluating the impact of potholes on road surfaces based on digital twins according to claim 7, characterized in that: The specific implementation method of step S7 includes the following steps: S7.
1. Use a 3D structured light camera to collect road surface data and upload it to the digital twin model established in step S6. Then use the deep learning model for pothole recognition deployed in the system to intelligently identify pothole areas in the road image, and select all images containing potholes to obtain pothole data. S7.
2. Based on the images containing road potholes obtained in step S6.7, identify the number of road potholes in each image and calculate the total number of potholes NM ; S7.
3. Construct a method for randomly selecting the number of pits and obtain the number of randomly selected pits The calculation formula is: ; in, The confidence level is selected z The score is obtained by looking up the z-value table; is an estimate of the proportion of units in a population that have a particular characteristic; is an acceptable error; S7.
4. By number of pits Randomly draw pits and ditches, and after the drawing is completed, measure the size information of the pits and ditches on site and calculate the volume of the pits; Take the first pothole on the first road as an example and measure the length of the pothole Then divide the pit into 10 equal parts along its length, and measure the width of the section at each equal section. and depth ,in, Refers to the width of the ijth section of the first pit; Refers to the depth of the ijth section of the first pit, ij=1,2,3,…,9; Then calculate the pit volume V 1. The calculation formula is: ; Calculate all the first road The pit volume of the pit , , …, , then calculate the mean pothole volume of the first road : ; S7.
5. Orientation NP roads, collect road images, and calculate all NP The average pothole volume corresponding to the road, , , …, ; Then calculate the average pothole volume corresponding to all roads , the calculation formula is: ; S7.
6. Set traffic safety driving evaluation indicators, including the number of accidents, number of injured persons, number of vehicle damages, driving speed reduction value, and number of severe vehicle jolts; Get NP The number of accidents corresponding to the roads is recorded as: ; Get NP The number of injured people corresponding to the roads is recorded as: ; Get NP The number of vehicle damages corresponding to the roads is recorded as: ; Get NP The driving speed reduction values corresponding to the roads are recorded as follows: ; Get NP The number of severe vehicle bumps corresponding to the roads is recorded as: ; Then calculate the average number of severe bumps of the vehicle and record it as ; Then calculate the average value of each traffic safety driving evaluation index ZB: ; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB. That is, the average values of the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively ; S7.
7. Based on the traffic safety driving evaluation index of step S7.6, the correlation relationship between the pothole and the traffic safety driving evaluation index is established as follows: Establish the correlation between potholes and traffic safety driving evaluation index ZB, ; in, jj for NP One of the roads, Indicates jj The average volume of potholes corresponding to the roads; ZB represents the traffic safety driving evaluation index, including the number of accidents SGS, the number of injured people SSR, the number of vehicle damages CSS, the driving speed reduction value SJD, and the number of severe vehicle bumps DB; that is, the correlation between potholes and the number of accidents, the number of injured people, the number of vehicle damages, the driving speed reduction value, and the number of severe vehicle bumps are respectively , ; S7.
8. Set critical values for the degree of correlation between potholes and the number of accidents, the number of injured persons, the number of vehicle damages, the value of reduced driving speed, and the number of severe vehicle jolts, which are respectively: ; S7.
9. Compare the correlation between potholes and the number of accidents, number of injured persons, number of vehicle damages, speed reduction, and number of severe vehicle jolts to the critical value. ; make comparisons; when , , , , When any one of the following is true, it means that potholes have an impact on traffic safety.
Citation Information
Patent Citations
Urban bridge group state evaluation method based on axle coupling and digital twinning
CN114444983A
Road surface pit slot identification and evaluation method, electronic equipment and storage medium
CN115424232A
Road pit slot identification method
CN117809282A
Three-dimensional information detection device
CN216012115U
Method for constructing recognition model of moisture damage of asphalt pavement and method and system for recognizing moisture damage of asphalt pavement
US20220276182A1
Cited By
Lightweight processing method for digital twin model
CN119556571A
