Road guardrail state detection method based on artificial intelligence

Abstract

The invention belongs to the technical field of road guardrail state detection, and particularly relates to a road guardrail state detection method based on artificial intelligence. Comprising the following steps: collecting guardrail state information by adopting machine vision, an ultrasonic sensor and a laser radar; performing intelligent enhancement processing on image data by using an improved U-Net network architecture, improving adaptive filtering processing distance data, and improving clustering analysis to separate guardrail point cloud from surrounding environment; introducing knowledge graph enhanced fusion to construct a multi-modal fusion neural network, and outputting a guardrail state feature vector; and finally, determining the final state of the guardrail and sending alarm information by utilizing an advanced state evaluation algorithm and combining fuzzy logic and an expert system. The guardrail state can be accurately and rapidly detected, and the road traffic safety level is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of road guardrail status detection, and in particular relates to a road guardrail status detection method based on artificial intelligence. Background Art

[0002] Road guardrails are important facilities to ensure road traffic safety, and their status directly affects driving safety. However, guardrails may be damaged, deformed, and other problems during long-term use, and need to be inspected and maintained in a timely manner. The existing guardrail status detection method mainly relies on manual inspection, which is not only inefficient, but also prone to missed detection and false detection. With the rapid development of artificial intelligence technology, it has become a trend to apply it to road guardrail status detection. At present, some artificial intelligence-based detection methods have achieved certain results, but there are still some shortcomings. For example, in terms of data acquisition, a single sensor may not be able to fully and accurately obtain the status information of the guardrail; in terms of data processing, traditional image processing and analysis algorithms may be difficult to effectively cope with complex environments and changeable guardrail states; in terms of status assessment, there is a lack of effective methods to integrate multiple features, resulting in inaccurate and unreliable evaluation results. Therefore, there is a need for an artificial intelligence-based road guardrail status detection method that can comprehensively utilize data collected by multiple sensors and adopt advanced image processing and analysis technologies to achieve accurate and rapid detection of guardrail status, timely discover potential problems of guardrails, and improve road traffic safety. Summary of the invention

[0003] In view of the above-mentioned technical problems, the present invention proposes a road guardrail status detection method based on artificial intelligence, which is simple in method, easy to operate and can effectively solve the problems.

[0004] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0005] S1. First, use machine vision, ultrasonic sensors and lidar to collect comprehensive road guardrail status information;

[0006] S2. Secondly, the improved U-Net network architecture is used to perform intelligent enhancement processing on the collected machine vision image data; the improved adaptive filtering is used to process the distance data detected by the ultrasonic sensor; the improved clustering analysis is performed on the three-dimensional point cloud data collected by the lidar to quickly separate the point cloud of the guardrail from the surrounding environment, thereby improving the efficiency of subsequent processing;

[0007] S3, then introduce knowledge graph enhanced fusion to construct a multimodal fusion neural network, input the preprocessed image data, distance data and three-dimensional point cloud data into the network, and output a comprehensive guardrail state feature vector;

[0008] S4. Finally, the fused feature vector is analyzed using an advanced state assessment algorithm to determine the state of the guardrail. An assessment method combining fuzzy logic and expert system is introduced to perform fuzzy reasoning and decision-making based on various indicators in the fused feature vector to determine the final state of the guardrail and issue corresponding alarm information in a timely manner.

[0009] In step S3, the knowledge graph enhancement fusion is introduced to construct a multimodal fusion neural network, and the specific operation of realizing the comprehensive output of the comprehensive guardrail state feature vector is as follows:

[0010] S31. First, construct a guardrail knowledge graph G = (V, E), where V is a node set and E is an edge set. For image data, a convolutional neural network is used for feature extraction. For distance data, a fully connected network is used for feature extraction. For three-dimensional point cloud data, a point cloud processing network is used for feature extraction.

[0011] S32, then define the fusion function, the fused comprehensive feature vector is where F merge is the fusion operation function, G represents the knowledge graph, They are feature extraction for image, distance and point cloud data enhancement;

[0012] S33, then perform a fusion operation, using a weighted summation method, and the fused feature vector is where ω I ,ω D ,ω P is the weight coefficient, whose weight is set by the information in the knowledge graph;

[0013] S34. Finally, after feature fusion guided by the knowledge graph, a comprehensive guardrail status feature vector is obtained for subsequent guardrail status judgment and analysis.

[0014] Preferably, the intelligent enhancement processing of the collected machine vision image data using the improved U-Net network architecture in step S2 is implemented as follows:

[0015] S211, first input image is I(x,y), the output feature map of the first layer of the encoder is F l E , the lth output feature map of the decoder is F l D , the encoder calculation formula is in represents the lth convolution operation of the encoder, and the calculation formula of the decoder is in represents the l-th convolution operation of the decoder, Represents the concatenation operation of feature maps;

[0016] S212, then multi-scale feature fusion is performed and the attention mechanism is introduced. In the loss function, the loss function is composed of the mean square loss function Structural similarity index loss function L ssim =1-SSIM(I enhanced , I ideal ), perceptual loss function Adversarial loss function L adv =log(1-D(I enhanced ))+log(D(I ideal )), where II enhanced I ideal are the original image, the enhanced image, and the ideal enhanced image, respectively. Φ represents the features extracted by the pre-trained neural network. D is the discriminator network used to distinguish the enhanced image from the ideal enhanced image. The total loss function L = αL mse +βL ssim +γL perceptual +δL adv , where α, β, γ, δ are weight coefficients;

[0017] S213. Finally, after improvement, the collected image is input into the network and the enhanced image is output.

[0018] Preferably, the improved adaptive filtering in step S2 is implemented to process the distance data detected by the ultrasonic sensor as follows:

[0019] S221, the first is the structure design of the filter, using a finite impulse response filter structure, its output is Where d(ti) is the distance data sequence, t represents time, N is the length of the filter, ω i (t) is the coefficient of the filter at time t;

[0020] S222, secondly, based on the gradient descent method, the momentum term and the learning rate adjustment strategy are combined to update the filter coefficients where μ is the learning rate, is the loss function L with respect to coefficient ω i The gradient of (t), α is the momentum coefficient; the learning rate is adjusted according to the exponentially decaying learning rate strategy μ(t) = μ0γ t , where μ0 is the initial learning rate and γ is the decay coefficient;

[0021] S223. Finally, after the improvement, the new data is input into the filter and the output of the filter is calculated.

[0022] Preferably, the improved cluster analysis in step S2 is implemented to quickly separate the point cloud of the guardrail from the surrounding environment as follows:

[0023] S231, first perform feature extraction, for each point p i , calculate its normal vector n i and curvature c i , the similarity measure between defined points considers the similarity of spatial distance and local features, and uses the comprehensive similarity measure formula s(p i ,p j )=αd(p i ,p j )+βcosθ i,j |+γc i -c j |, where is the spatial distance, θ i,j It's point p i and p j The angle between the normal vectors, α, β, γ are weight coefficients;

[0024] S232, then using a density-based clustering algorithm, select an unmarked point as a seed point, determine its neighborhood points based on a similarity metric, and if the number of neighborhood points is greater than a threshold, mark these points as the same cluster, and continue to expand the cluster, repeating the steps until all stores are marked;

[0025] S233. Finally, according to the shape and height range prior features of the guardrail, the clustering results are screened to separate the point cloud belonging to the guardrail.

[0026] Preferably, the evaluation method combining fuzzy logic and expert system introduced in step S4 is implemented as follows:

[0027] S41. First, establish a fuzzy logic system, determine the input variables used to evaluate the guardrail status and define fuzzy sets for each input variable, determine the output variable as the guardrail status and define its fuzzy set, and determine the fuzzy rule R based on expert experience and knowledge. i :IFC img iA i ANDD dist iB i ANDI pc iC i THENS fence iD i , where A i , B i , C i , D i are the fuzzy sets of the corresponding variables, C img , D dist , Ipc , S fence They are image feature index, distance feature index and point cloud feature index;

[0028] S42, then fuzzify the input variables, convert the specific input values ​​into corresponding fuzzy set membership, perform fuzzy reasoning, and then defuzzify the demerits into specific road guardrail status values;

[0029] S43, next, establish an expert knowledge database to store various rules, experiences and cases about the guardrail status judgment, which is used to supplement and correct the results of the fuzzy logic system;

[0030] S44. Finally, the final state of the guardrail is determined based on the comprehensive results of fuzzy logic and expert system.

[0031] Compared with the prior art, the advantages and positive effects of the present invention are that it comprehensively uses a variety of sensors to collect data and comprehensively and accurately obtain guardrail status information; adopts improved algorithms and technologies to improve data processing efficiency and accuracy; introduces knowledge graphs and fuzzy logic to more accurately judge the guardrail status; has a high degree of automation, reduces manual intervention, reduces costs, and improves the level of road traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is a schematic diagram of the implementation process of the present invention;

[0034] Figure 2 It is a schematic diagram of the structure of a multimodal neural network; DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0037] In order to improve the accuracy and efficiency of road guardrail status detection, timely discover potential problems of guardrails, and ensure road traffic safety, this embodiment adopts a road guardrail status detection method based on artificial intelligence. The implementation process of the present invention is as follows Figure 1 shown.

[0038] First, data collection is carried out. A machine vision system equipped with a high-resolution camera is installed on the inspection vehicle to collect images of road guardrails at a certain frequency to ensure clear and comprehensive guardrail image information. Multiple ultrasonic sensors are installed on the inspection vehicle, evenly distributed around the vehicle, to detect the distance data between the guardrail and the sensor in real time, covering all parts of the guardrail. At the same time, a high-precision laser radar is installed to collect 3D point cloud data of the guardrail and obtain the 3D shape and spatial position information of the guardrail.

[0039] In order to improve the image quality and highlight the guardrail details and potential problem areas, the present invention uses the improved U-Net network architecture to perform intelligent enhancement processing on the machine vision image data. First, the input image is I(x, y), and the output feature map of the first layer of the encoder is F l E , the lth output feature map of the decoder is F l D , the encoder calculation formula is in represents the lth convolution operation of the encoder, and the calculation formula of the decoder is in represents the l-th convolution operation of the decoder, Represents the concatenation operation of the feature map; then multi-scale feature fusion is performed and the attention mechanism is introduced. In the loss function, the loss function is composed of the mean square loss function Structural similarity index loss function L ssim =1-SSIM(I enhanced ,I ideal ), perceptual loss function Adversarial loss function L adv =log(1-D(I enhanced ))+log(D(I ideal )), where II enhanced I ideal are the original image, the enhanced image, and the ideal enhanced image, respectively. Φ represents the features extracted by the pre-trained neural network. D is the discriminator network used to distinguish the enhanced image from the ideal enhanced image. The total loss function L = αL mse +βL ssim +γL perceptual +δL adv, where α, β, γ, δ are weight coefficients; finally, the collected images are input into the network after improvement and the enhanced images are output. This processing method can integrate multi-scale features and attention mechanisms, automatically learn the best enhancement method, improve the clarity, contrast and accuracy of the image, and provide more reliable information for subsequent guardrail status detection.

[0040] Next, the purpose of improving the adaptive filtering to process the distance data of the ultrasonic sensor is to remove noise interference and improve the accuracy of the distance data. The first step is to design the structure of the filter. The finite impulse response filter structure is adopted, and its output is Where d(ti) is the distance data sequence, t represents time, N is the length of the filter, ω i (t) is the coefficient of the filter at time t; secondly, based on the gradient descent method, the momentum term and the learning rate adjustment strategy are combined to update the filter coefficients where μ is the learning rate, is the loss function L with respect to coefficient ω i The gradient of (t), α is the momentum coefficient; the learning rate is adjusted according to the exponentially decaying learning rate strategy μ(t) = μ0γ t , where μ0 is the initial learning rate and γ is the attenuation coefficient; finally, after improvement, the new data is input into the filter and the output of the filter is calculated.

[0041] In order to accurately extract the point cloud data of the guardrail so as to more accurately analyze the state of the guardrail, the present invention uses improved cluster analysis to quickly separate the point cloud of the guardrail from the surrounding environment. First, feature extraction is performed. For each point p i , calculate its normal vector n i and curvature c i , the similarity measure between defined points considers the similarity of spatial distance and local features, and uses the comprehensive similarity measure formula s(p i ,p j )=αd(p i ,p j )+βcosθ i,j |+γc i -c j |, where is the spatial distance, θ i,j It's point p i and p jThe angle between the normal vectors of the guardrail, α, β, γ are weight coefficients; then a density-based clustering algorithm is used to select an unmarked point as a seed point, and its neighborhood points are determined based on the similarity metric. If the number of neighborhood points is greater than the threshold, these points are marked as the same cluster, and the cluster is continued to be expanded, and the steps are repeated until all stores are marked; finally, according to the prior features of the guardrail shape and height range, the clustering results are screened to separate the point cloud belonging to the guardrail. This measure can combine the prior features of the guardrail, such as shape, height range, etc., to quickly and accurately separate the guardrail point cloud, improve the efficiency and accuracy of subsequent processing, and reduce misjudgments and missed judgments.

[0042] Considering that the multimodal fusion neural network may have defects such as insufficient fusion of different modal data and difficulty in effectively utilizing prior knowledge, the advantage of introducing knowledge graph enhanced fusion is that it can better guide feature extraction and fusion, enhance key information fusion, and enable the network to more accurately output the comprehensive guardrail state feature vector. The structure of the multimodal fusion neural network is as follows: Figure 2 As shown, first, a guardrail knowledge graph G = (V, E) is constructed, where V is a node set and E is an edge set. For image data, a convolutional neural network is used for feature extraction, for distance data, a fully connected network is used for feature extraction, and for three-dimensional point cloud data, a point cloud processing network is used for feature extraction; in the state detection of road guardrails in this embodiment, for image data collected by machine vision, a convolutional neural network uses a convolution kernel to slide on the image, multiply and sum with the elements of the local area of ​​the image, and extracts more abstract features as the network deepens. Texture, shape and other features are extracted from the guardrail image to provide a basis for subsequent judgment of the guardrail state. For the distance data of the ultrasonic sensor, the fully connected network takes it as input, the neurons are fully connected and processed through weight matrix linear transformation and activation function, and features such as the change trend of the distance between the guardrail and the sensor are extracted to assist in judging whether the guardrail has displacement. For the three-dimensional point cloud data collected by the lidar, the point cloud processing network first calculates the normal vector and curvature of each point, and then integrates the spatial information. By analyzing the spatial distribution characteristics of the guardrail point cloud, it is judged whether the integrity and shape of the guardrail have changed.

[0043] Next, define the fusion function. The fused comprehensive feature vector is: where F merge is the fusion operation function, G represents the knowledge graph, They are the feature extraction of image, distance and point cloud data enhancement respectively; then the fusion operation is performed, using the weighted summation method, and the fused feature vector is where ω I ,ω D ,ω Pis the weight coefficient, and its weight is set by the information in the knowledge graph, according to the connection strength between nodes and the importance of edges. In the knowledge graph, there are nodes A and B, and the edge weight between them is ω AB In the guardrail status detection, the data related to A has a strong influence on the final result, so the corresponding weight given to the data during fusion is large; finally, after the feature fusion guided by the knowledge graph, a comprehensive guardrail status feature vector is obtained for subsequent guardrail status judgment and analysis.

[0044] Finally, the present invention considers introducing an evaluation method combining fuzzy logic and expert system, and performs fuzzy reasoning and decision-making based on various indicators in the fusion feature vector to determine the final state of the guardrail. First, a fuzzy logic system is established, the input variables used to evaluate the guardrail state are determined, and a fuzzy set is defined for each input variable. The output variable is determined as the guardrail state and its fuzzy set is defined. The fuzzy rule R is determined based on expert experience and knowledge. i :IFC img iA i ANDD dist iB i ANDI pc iC i THENS fence iD i , where A i , B i , C i , D i are the fuzzy sets of the corresponding variables, C img , D dist , I pc , S fenceThey are image feature index, distance feature index and point cloud feature index respectively; then fuzzify the input variables, convert the specific input values ​​into the corresponding fuzzy set membership, perform fuzzy reasoning, and then defuzzify the demerits into specific road guardrail status values. The implementation method of fuzzy rules is to first determine fuzzy rules based on expert experience and knowledge. These rules are based on the fuzzy sets of input variables including image feature index, distance feature index and point cloud feature index and the fuzzy sets of output variables including guardrail status. Then fuzzify the input variables and convert the specific values ​​into corresponding fuzzy set membership. Then perform fuzzy reasoning based on fuzzy rules and comprehensively consider the membership of each input variable to obtain the reasoning result. Finally, defuzzify and convert the reasoning result into a specific road guardrail status value to determine the final state of the guardrail; then establish an expert knowledge database to store various rules, experiences and cases about guardrail status judgment, which is used to supplement and correct the results of the fuzzy logic system; finally, determine the final state of the guardrail based on the comprehensive results of fuzzy logic and expert system. This state judgment and analysis method can fully consider the uncertainty and ambiguity of various indicators in the fusion feature vector, and make more flexible reasoning and decision-making. Combining the knowledge and experience of the expert system can improve the accuracy and reliability of the assessment and more accurately determine the final state of the guardrail.

[0045] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A road guardrail status detection method based on artificial intelligence, characterized in that: The following steps are involved: S1. First, use machine vision, ultrasonic sensors and lidar to collect comprehensive road guardrail status information; S2. Secondly, the improved U-Net network architecture is used to perform intelligent enhancement processing on the collected machine vision image data; Improved adaptive filtering is used to process the distance data detected by the ultrasonic sensor; Improved cluster analysis of the 3D point cloud data collected by the LiDAR can quickly separate the point cloud of the guardrail from the surrounding environment, improving the efficiency of subsequent processing; S3, then introduce knowledge graph enhanced fusion to construct a multimodal fusion neural network, input the preprocessed image data, distance data and three-dimensional point cloud data into the network, and output a comprehensive guardrail state feature vector; S4. Finally, the fused feature vector is analyzed using an advanced state assessment algorithm to determine the state of the guardrail. An assessment method combining fuzzy logic and expert system is introduced to perform fuzzy reasoning and decision-making based on various indicators in the fused feature vector to determine the final state of the guardrail and issue corresponding alarm information in a timely manner. In step S3, the knowledge graph enhancement fusion is introduced to construct a multimodal fusion neural network, and the specific operation of realizing the comprehensive output of the comprehensive guardrail state feature vector is as follows: S31. First, construct a guardrail knowledge graph G = (V, E), where V is a node set and E is an edge set. For image data, a convolutional neural network is used for feature extraction. For distance data, a fully connected network is used for feature extraction. For three-dimensional point cloud data, a point cloud processing network is used for feature extraction. S32, then define the fusion function, the fused comprehensive feature vector is where F merge is the fusion operation function, G represents the knowledge graph, They are feature extraction for image, distance and point cloud data enhancement; S33, then perform a fusion operation, using a weighted summation method, and the fused feature vector is where ω I ,ω D ,ω P is the weight coefficient, whose weight is set by the information in the knowledge graph; S34. Finally, after feature fusion guided by the knowledge graph, a comprehensive guardrail status feature vector is obtained for subsequent guardrail status judgment and analysis.

2. The method for detecting road guardrail status based on artificial intelligence according to claim 1, characterized in that: The implementation of intelligent enhancement processing of the collected machine vision image data using the improved U-Net network architecture in step S2 is: S211, first input image is I(x,y), the output feature map of the first layer of the encoder is F l E , the lth output feature map of the decoder is F l D , the encoder calculation formula is in represents the lth convolution operation of the encoder, and the calculation formula of the decoder is in represents the l-th convolution operation of the decoder, Represents the concatenation operation of feature maps; S212, then multi-scale feature fusion is performed and the attention mechanism is introduced. In the loss function, the loss function is composed of the mean square loss function Structural similarity index loss function L ssim =1-SSIM(I enhanced , I ideal ), perceptual loss function Adversarial loss function L adv =log(1-D(I enhanced ))+log(D(I ideal )), where II enhanced I ideal are the original image, the enhanced image, and the ideal enhanced image, respectively. Φ represents the features extracted by the pre-trained neural network. D is the discriminator network used to distinguish the enhanced image from the ideal enhanced image. The total loss function L = αL mse +βL ssim +γL perceptual +δL adv , where α, β, γ, δ are weight coefficients; S213. Finally, after improvement, the collected image is input into the network and the enhanced image is output.

3. The road guardrail state detection method based on artificial intelligence according to claim 1 is characterized in that: The improved adaptive filtering in step S2 is implemented by processing the distance data detected by the ultrasonic sensor as follows: S221, the first is the structure design of the filter, using a finite impulse response filter structure, its output is Where d(ti) is the distance data sequence, t represents time, N is the length of the filter, ω i (t) is the coefficient of the filter at time t; S222, secondly, based on the gradient descent method, the momentum term and the learning rate adjustment strategy are combined to update the filter coefficients where μ is the learning rate, is the loss function L with respect to coefficient ω i The gradient of (t), α is the momentum coefficient; The learning rate is adjusted according to the exponentially decaying learning rate strategy μ(t)=μ0γ t , where μ0 is the initial learning rate and γ is the decay coefficient; S223. Finally, after the improvement, the new data is input into the filter and the output of the filter is calculated.

4. The road guardrail state detection method based on artificial intelligence according to claim 1 is characterized in that: The improved cluster analysis in step S2 is implemented to quickly separate the point cloud of the guardrail from the surrounding environment as follows: S231, first perform feature extraction, for each point p i , calculate its normal vector n i and curvature c i , the similarity measure between the defined points takes into account the similarity of spatial distance and local features, and the comprehensive similarity measure formula is s(p i ,p j )=αd(p i ,p j )+βcosθ i,j |+γc i -c j |, where is the spatial distance, θ i,j It's point p i and p j The angle between the normal vectors, α, β, γ are weight coefficients; S232, then using a density-based clustering algorithm, select an unmarked point as a seed point, determine its neighborhood points based on a similarity metric, and if the number of neighborhood points is greater than a threshold, mark these points as the same cluster, and continue to expand the cluster, repeating the steps until all stores are marked; S233. Finally, according to the prior features of the shape and height range of the guardrail, the clustering results are screened to separate the point cloud belonging to the guardrail.

5. The road guardrail state detection method based on artificial intelligence according to claim 1 is characterized in that: The evaluation method combining fuzzy logic and expert system introduced in step S4 is implemented as follows: S41. First, establish a fuzzy logic system, determine the input variables used to evaluate the guardrail status and define fuzzy sets for each input variable, determine the output variable as the guardrail status and define its fuzzy set, and determine the fuzzy rule R based on expert experience and knowledge. i :IFC img iA i ANDD dist iB i ANDI pc iC i THENS fence iD i , where A i , B i , C i , D i are the fuzzy sets of the corresponding variables, C img , D dist , I pc , S fence They are image feature index, distance feature index and point cloud feature index; S42, then fuzzify the input variables, convert the specific input values ​​into corresponding fuzzy set membership, perform fuzzy reasoning, and then defuzzify the demerits into specific road guardrail status values; S43, next, establish an expert knowledge database to store various rules, experiences and cases about the guardrail status judgment, which is used to supplement and correct the results of the fuzzy logic system; S44. Finally, the final state of the guardrail is determined based on the comprehensive results of fuzzy logic and expert system.

Images