Air-ground integrated intelligent patrol inspection system for expressway

Abstract

The invention discloses an expressway air-ground integrated intelligent tour inspection system, and relates to the technical field of expressway tour inspection. The inspection management platform is in communication connection with a data acquisition module, an environment perception analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module and an inspection early warning management module, and the data acquisition module is used for collecting and preprocessing various inspection data required by highway inspection. Through cooperative work of air patrol of the unmanned aerial vehicle and the ground sensor, large-area patrol data can be rapidly obtained, all-directional and real-time monitoring of the expressway is achieved, the patrol efficiency is remarkably improved, a large-area patrol task can be completed in a short time, potential road diseases can be found in time, and the safety of the expressway is improved. Timely and accurate information support is provided for road maintenance and management departments, and safe and smooth operation of the expressway is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of highway inspection, and particularly relates to an integrated intelligent inspection and patrol system for highways integrating air and ground. Background Art

[0002] As an important part of the modern transportation network, highways undertake a large number of passenger and freight transportation tasks and play a key supporting role in economic development. Their safe and efficient operation is directly related to the travel safety of the people and the stable development of the social economy. However, highway inspection work faces many challenges. The traditional highway inspection methods mainly rely on manual inspection and ground patrol, which usually have problems such as limited inspection scope, low efficiency, and lagged information transmission. With the rapid development of technologies such as the Internet of Things, artificial intelligence, and drones, new solutions have been provided for highway inspection work.

[0003] During the highway inspection process, affected by factors such as light, weather, and shadow occlusion, it is easy to misjudge road debris and temporary construction signs as diseases, which in turn leads to the problem of reduced recognition accuracy of real disease targets. Therefore, an integrated intelligent inspection and patrol system for highways integrating air and ground is proposed to improve the recognition accuracy of highway inspection. Summary of the Invention

[0004] The purpose of the present invention is to provide an integrated intelligent inspection and patrol system for highways integrating air and ground to solve the problems raised in the above background art.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is:

[0006] An integrated intelligent inspection and patrol system for highways integrating air and ground, including an inspection management platform, which is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a road surface disease dynamic recognition module, and an inspection early warning management module;

[0007] The data acquisition module is used to collect and preprocess various inspection data required for highway inspection, including image data of drones, ground sensor data, real-time monitoring videos, environmental sensor data, as well as traffic flow and road condition information;

[0008] The environmental perception and analysis module is used to analyze the preprocessed inspection data and remove interference information;

[0009] The target detection module is used to perform target detection on the image inspection data of environmental perception and analysis by using deep learning algorithms, identify the diseases and non-disease targets on the road surface, improve the accuracy and robustness of target detection, and reduce the occurrence of misjudgment and missed judgment;

[0010] The disease target classification module is used to extract features and classify the detected targets, and combine classification algorithms to distinguish the specific categories of real disease targets, improving the recognition accuracy;

[0011] The road surface disease dynamic recognition module is used to establish a dynamic disease recognition model by using machine learning technology, compare it with historical data, identify the evolution trend of diseases, improve the accuracy and timeliness of disease recognition, and enhance the reliability of disease recognition;

[0012] The inspection and warning management module is used to perform real-time analysis on inspection data and warn of existing diseases.

[0013] A further improvement in the technical solution of the present invention lies in that: the data acquisition module specifically includes:

[0014] Determine the target area for highway inspection and patrol, and conduct demand planning for aerial and ground inspection and patrol to obtain various inspection data including drone image data, ground sensor data, surveillance videos, environmental sensor data, as well as traffic flow and road condition information;

[0015] According to the inspection requirements and target area range of the highway, plan the flight route, flight altitude and shooting angle parameters of the drone to ensure that the target area of the inspection and patrol can be fully covered, and automatically fly according to the preset flight route. Use the equipped high-definition camera and infrared camera equipment to photograph the highway and its surrounding environment to obtain high-resolution image data, and then wirelessly transmit the photographed image data to the storage server of the data acquisition module;

[0016] Deploy various types of ground sensors along the highway, including meteorological sensors (monitoring meteorological parameters such as temperature, humidity, wind speed, and wind direction), temperature and humidity sensors (monitoring road surface temperature and humidity), and vibration sensors (monitoring the vibration conditions of structures such as bridges and tunnels), etc. The ground sensors collect various data and transmit the data to the storage server of the data acquisition module through a wireless communication network;

[0017] Install high-definition surveillance cameras at important sections, intersections and service areas of the highway to conduct all-round monitoring of the highway, and transmit the video stream data collected by the surveillance cameras to the storage server of the data acquisition module through the network;

[0018] Set environmental sensors including light sensors, rainfall sensors and snowfall sensors in the inspection target area to monitor environmental information such as light intensity, rainfall and snowfall during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through a communication network to provide data support for subsequent environmental perception analysis;

[0019] By traffic flow monitoring devices (inductive coils, video monitors, etc.) installed at key positions on highways, the flow and speed information of vehicles is monitored, and the data is transmitted to the storage server of the data acquisition module. Road condition monitoring devices (road surface flatness detectors, road surface damage detectors, etc.) are used to detect the flatness, damage degree and other conditions of the road, and the relevant data is transmitted to the storage server of the data acquisition module;

[0020] Preprocess various inspection data of the target area obtained, including data cleaning and data format conversion, and perform encoding processing on the preprocessed data to assign a unique identifier to each type of data;

[0021] Fuse the preprocessed multi-source data of drone image data, ground sensor data, surveillance videos, environmental sensor data, and traffic flow and road condition information to form an inspection data set.

[0022] A further improvement of the technical solution of the present invention is that: the environmental perception analysis module includes a light and weather correction unit and a shadow occlusion processing unit;

[0023] The light and weather correction unit is used to correct the influence of light changes and weather factors on the preprocessed image inspection data, adjust the brightness and contrast parameters of the image, and reduce the influence of light and weather changes on target recognition;

[0024] The shadow occlusion processing unit is used to process the shadow occlusion areas in the image inspection data, reduce the influence of shadows on target recognition, and remove or weaken the influence of shadow occlusion areas.

[0025] A further improvement of the technical solution of the present invention is that: the environmental perception analysis module specifically includes:

[0026] Extract the preprocessed inspection data from the data acquisition module, and analyze the preprocessed image inspection data through the light and weather correction unit and the shadow occlusion processing unit;

[0027] Divide the image into multiple regions through the light and weather correction unit, calculate the brightness values of each pixel point in the image, calculate the light intensity of each region respectively, determine the overall light intensity of the image to identify the light differences in different regions of the image, judge the weather conditions according to the color characteristics and texture characteristics of the image, and perform gamma correction on the image according to the light intensity and weather factors, adjust the brightness and contrast of the image, and enhance the dark or bright details of the image by adjusting the gamma value;

[0028] The image is converted from the RGB color space to the Lab color space by the shadow occlusion processing unit, and the shadow region in the image is identified by using image segmentation and edge detection algorithms, and the edge information in the image is detected to further confirm the position and shape of the shadow region;

[0029] Image inpainting processing is performed on the detected shadow region, and the image inpainting algorithm is used to fill the pixel values of the shadow region. The image inpainting algorithm can generate pixel values consistent with the surrounding environment according to the information of the surrounding pixels, thereby removing the influence of the shadow, and color correction is performed on the shadow region to adjust the color of the shadow region to match the surrounding environment. By adjusting the color parameters, the influence of the shadow on target recognition can be weakened. Furthermore, light compensation is performed on the shadow region to increase the brightness of the shadow region to make it consistent with the lighting conditions of the surrounding environment.

[0030] A further improvement of the technical solution of the present invention lies in that: the target detection module specifically includes:

[0031] Receive the image inspection data after illumination and weather correction and shadow occlusion processing from the environmental perception analysis module, extract the historical inspection data therefrom, and mark the positions and categories of the disease and non-disease targets therein. Integrate the marked historical inspection data and divide it into a training set and a test set. The training set is used to train the target recognition model, and the test set is used to evaluate the performance of the model;

[0032] Use the training set data combined with the convolutional neural network model to train the target recognition model using the disease and non-disease target data therein, and use the test set data to evaluate the trained target recognition model, calculate indicators such as the accuracy rate, recall rate, and F1 value of the model to evaluate the performance of the model. According to the evaluation results, further adjust the model parameters and optimization strategies to improve the accuracy and robustness of the model;

[0033] Input the preprocessed real-time inspection data into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease category labels. Furthermore, according to the generated bounding boxes and disease category labels, classify and locate the detected targets, distinguish disease targets from non-disease targets, and determine the position of the target in the image through the bounding boxes.

[0034] A further improvement of the technical solution of the present invention lies in that: the process of classifying and locating the detected targets is as follows:

[0035] The pre - processed real - time inspection image is passed through multiple convolutional layers in the target recognition model. Each convolutional layer contains multiple convolutional kernels. The convolutional kernels slide on the image and perform convolutional operations with local regions of the image to extract local features of the image through convolutional operations. A 3x3 convolutional kernel slides on the image and performs dot - product operations with each 3x3 region in the image to obtain a new feature map. After the convolutional layer, an average pooling operation of the pooling layer is adopted to reduce the size of the feature map to reduce the dimension of the features and retain key features;

[0036] The target recognition model generates candidate regions through a region proposal network. The region proposal network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes, and aspect ratios of the image. Then, the non - maximum suppression algorithm is used to screen the generated candidate regions, removing redundant or unreasonable regions. Among them, the non - maximum suppression algorithm removes candidate regions with a high degree of overlap according to the confidence score of the candidate regions and retains the candidate region with the highest confidence;

[0037] The feature map corresponding to the screened candidate regions is fed into the fully - connected layer to extract semantic features. The softmax function is used to classify the extracted semantic features and output the classification results in the form of a probability vector to determine the probability that each candidate region belongs to a disease target or a non - disease target. And the coordinates of the bounding box of each candidate region are predicted through a regression algorithm. Then, by calculating the difference between the predicted bounding box and the ground - truth bounding box, the mean - squared error loss is used to optimize the regression result of the bounding box. Among them, the coordinates of the bounding box are represented by the offset relative to the candidate region, including the center - point coordinates, width, and height;

[0038] For the classification and bounding - box regression results of multiple candidate regions, if multiple candidate regions correspond to the same target, they are merged into one target. The non - maximum suppression algorithm is used to further screen and merge the classification results and the bounding - box regression results to retain the most reliable object detection results;

[0039] The target recognition model outputs the position, category, and confidence of the detected targets in the image. Each detection result includes a bounding box and a disease - category label. Among them, the bounding box is used to determine the position of the target in the image, and the disease - category label is used to distinguish disease targets from non - disease targets.

[0040] A further improvement of the technical solution of the present invention lies in that: the disease target classification module specifically includes:

[0041] Based on the real disease targets output by the target detection module, feature analysis is performed on them, and the shape features, texture features, and color features of the real disease targets are extracted. Then, a weighted - average feature fusion method is adopted to combine different types of features into a feature vector;

[0042] According to the characteristics and classification requirements of disease targets, the classification algorithm of the support vector machine is trained using the feature dataset of known disease targets to obtain a disease classification model;

[0043] The extracted and fused feature vectors are input into the trained disease classification model. The disease classification model calculates the probability of the true disease category to which each target belongs based on the feature vectors and outputs the label of the true disease category to which each target belongs.

[0044] A further improvement of the technical solution of the present invention lies in: the road surface disease dynamic recognition module specifically includes;

[0045] Obtain the latest disease target data from the target detection module and the disease target classification module, including the type and location information of the disease, as well as the corresponding image data, and collect the disease data accumulated during previous inspections, including the development process and treatment records of the disease. Integrate the historical data with the real-time inspection data to form a complete time series dataset;

[0046] Extract the features related to disease evolution from the integrated data, including the shape features, texture features, color features of the disease, and the disease location. Then integrate the extracted features related to disease evolution to obtain a feature dataset, and divide it into a training set and a test set;

[0047] Use the training set data to train the time series analysis model to construct a dynamic disease recognition model. During the training process, the time series analysis model learns the relationship between disease features and evolution trends, adjusts the model parameters, minimizes the prediction error, and uses the test set to test and evaluate the established dynamic disease recognition model, and then optimize the model to improve its generalization ability and accuracy;

[0048] Input the real-time inspection data into the trained dynamic disease recognition model, compare the newly detected disease features with the disease features in the historical data, analyze the similarity and difference between the features through the similarity calculation method, calculate the disease evolution tendency index, and then analyze the evolution process of historical diseases to identify the evolution trend and severity of new diseases.

[0049] A further improvement of the technical solution of the present invention lies in: the inspection warning management module includes a warning prompt unit and an inspection disposal unit;

[0050] The warning prompt unit is used to give a warning prompt for the identified real disease target and send out a warning message, including the real disease target category, disease location, disease evolution trend and warning time;

[0051] The inspection disposal unit is used to plan, allocate, execute and track inspection tasks, including setting inspection routes, times, personnel, etc.

[0052] A further improvement of the technical solution of the present invention lies in that: the inspection and early warning management module specifically includes:

[0053] The early warning prompt unit receives the inspection analysis results output by the target detection module, the disease target classification module and the road disease dynamic recognition module, including the type, location, severity and evolution trend information of the disease;

[0054] According to the preset early warning threshold, it judges whether an early warning needs to be issued. When the disease evolution tendency index exceeds the preset early warning threshold, the early warning condition is triggered, and then an early warning prompt including detailed disease information is generated, including: disease target category, disease location, disease evolution trend and early warning time, and then the early warning information is sent to relevant personnel through multiple channels such as text messages, emails, and in-system notifications to ensure that the information can be conveyed to inspection personnel and management personnel in a timely manner;

[0055] The inspection and disposal unit plans inspection tasks according to the early warning information and the inspection plan, comprehensively considers the location, severity of the disease and the distribution factors of inspection personnel, arranges the inspection route, gives priority to dealing with high-risk diseases, and assigns the inspection tasks to specific inspection personnel, clarifying the responsible person and completion time of the task. The inspection personnel go to the disease location for on-site inspection and treatment according to the assigned tasks, and record the treatment situation and results during the execution process, including the repair situation of the disease and the measures taken, etc.;

[0056] The execution situation of the inspection task is tracked in real time to ensure that the task is completed on time. The task progress is monitored through the system to timely discover and solve problems that occur during the execution process. After the inspection personnel complete the task, the treatment results are fed back to the system.

[0057] Due to the adoption of the above technical solution, the technical progress achieved by the present invention compared with the prior art is:

[0058] 1. The present invention provides an integrated intelligent inspection and patrol system for highway air and ground. Through the collaborative work of aerial inspections by drones and ground sensors, it can quickly obtain large-area inspection data, realize all-round and real-time monitoring of highways, significantly improve the inspection efficiency, complete large-area inspection tasks in a short time, timely discover potential road disease problems, and provide timely and accurate information support for road maintenance and management departments to ensure the safe and smooth operation of highways.

[0059] 2. The present invention provides an integrated intelligent inspection and patrol system for highway air and ground. Through the collaborative work of the environmental perception and analysis module, the target detection module, and the disease target classification module, it can effectively improve the accuracy of disease recognition, reduce the occurrence of misjudgment and missed judgment, and can also achieve accurate recognition and classification of disease targets, and accurately judge the type, location, and severity of diseases. Combined with the real-time data transmission and processing mechanism, it ensures the immediate feedback of disease information. At the same time, the system can also adjust the inspection strategy according to the disease evolution trend, realize more targeted inspection operations, and further improve the maintenance efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0061] Figure 1 It is a schematic diagram of the system module process of the present invention;

[0062] Figure 2 It is a schematic diagram of the working process of the pavement disease dynamic recognition module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0064] Example 1, as Figure 1 shown, the present invention provides an integrated intelligent inspection and patrol system for highway air and ground, including an inspection management platform, which is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a pavement disease dynamic recognition module, and an inspection warning management module;

[0065] The data acquisition module is used to collect and preprocess various types of inspection data required for highway inspections, including image data of drones, ground sensor data, real-time monitoring videos, environmental sensor data, as well as traffic flow and road condition information. It determines the target areas for highway inspections and conducts demand planning for aerial and ground inspections to obtain various types of inspection data, including image data of drones, ground sensor data, monitoring videos, environmental sensor data, as well as traffic flow and road condition information. According to the inspection requirements and the scope of the target area of the highway, it plans the flight route, flight altitude, and shooting angle parameters of the drone to ensure that the target area of the inspection can be fully covered and the drone can fly automatically along the preset flight route. It uses the equipped high-definition camera and infrared camera devices to take pictures of the highway and its surrounding environment to obtain high-resolution image data, and then wirelessly transmits the captured image data to the storage server of the data acquisition module. Deploy various types of ground sensors along the highway, including meteorological sensors (monitoring meteorological parameters such as temperature, humidity, wind speed, and wind direction), temperature and humidity sensors (monitoring the temperature and humidity of the road surface), and vibration sensors (monitoring the vibration conditions of structures such as bridges and tunnels), etc. The ground sensors collect various types of data and transmit the data to the storage server of the data acquisition module through a wireless communication network. Install high-definition surveillance cameras at important sections, intersections, and service areas of the highway to conduct all-round monitoring of the highway, and transmit the video stream data collected by the surveillance cameras to the storage server of the data acquisition module through the network. Set environmental sensors including light sensors, rainfall sensors, and snowfall sensors in the inspection target area to monitor environmental information such as light intensity, rainfall, and snowfall during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through a communication network to provide data support for subsequent environmental perception analysis. Through traffic flow monitoring devices (inductive coils, video monitors, etc.) set at key positions on the highway, monitor the traffic flow and speed information of vehicles and transmit the data to the storage server of the data acquisition module. Use road condition monitoring devices (road surface roughness detectors, road surface damage detectors, etc.) to detect the conditions of the road surface such as roughness and damage degree, and transmit the relevant data to the storage server of the data acquisition module. Preprocess various types of inspection data of the obtained target area, including data cleaning and data format conversion. Through data cleaning, conduct preliminary screening of the collected data, remove invalid data or outliers caused by sensor failures, communication interferences, etc. For some missing data, reasonably complete it according to the characteristics and correlations of the data to ensure the integrity and continuity of the data. Convert data of different types and from different sources into a unified format for subsequent data processing and analysis, and conduct coding processing on the preprocessed data to assign a unique identifier to each type of data.The pre-processed UAV image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information are integrated to form an inspection data set;

[0066] The environment perception analysis module is used to analyze the pre-processed inspection data and remove interference information. The environment perception analysis module includes a lighting and weather correction unit and a shadow occlusion processing unit. The lighting and weather correction unit is used to correct the influence of lighting changes and weather factors on the pre-processed image inspection data, adjust the brightness and contrast parameters of the image, reduce the influence of lighting and weather changes on target recognition, and improve the adaptability of the system. The shadow occlusion processing unit is used to process the shadow occlusion area in the image inspection data, reduce the influence of shadows on target recognition, remove or weaken the influence of shadow occlusion areas, and improve the accuracy and completeness of target recognition. The data acquisition module extracts the pre-processed inspection data, analyzes the pre-processed image inspection data through the illumination and weather correction unit and the shadow occlusion processing unit, divides the image into multiple areas through the illumination and weather correction unit, calculates the brightness value of each pixel in the image, calculates the illumination intensity of each area respectively, determines the overall illumination intensity of the image, identifies the illumination difference in different areas of the image, and judges the weather conditions based on the color and texture features of the image. On cloudy days, the image color is usually darker and the color saturation is lower. On rainy days, the image may have a water mist effect and the color is blue or gray. Under severe weather conditions, the texture characteristics of the image are The image features will be affected. In rainy or foggy days, the edges of the image will become blurred and the details will be lost. According to the light intensity and weather factors, the image is gamma corrected to adjust the brightness and contrast of the image so that the image can maintain a good visual effect under different lighting conditions. By adjusting the gamma value, the dark details or bright details of the image are enhanced. The image is converted from the RGB color space to the Lab color space through the shadow occlusion processing unit. In the Lab color space, the color features of the shadow area are made more obvious, which is convenient for shadow detection. The image segmentation and edge detection algorithms are used to identify the shadow area in the image and detect the edge information in the image. The first step is to confirm the position and shape of the shadow area, perform image restoration processing on the detected shadow area, and use an image restoration algorithm to fill the pixel value of the shadow area. The image restoration algorithm can generate pixel values ​​consistent with the surrounding environment based on the information of the surrounding pixels, thereby removing the influence of the shadow, and perform color correction on the shadow area, adjusting the color of the shadow area to match the surrounding environment. By adjusting the color parameters, the influence of the shadow on target recognition can be weakened, and then perform illumination compensation on the shadow area, increase the brightness of the shadow area, and make it consistent with the lighting conditions of the surrounding environment. Through illumination compensation, the visibility of the shadow area is improved, and the interference of the shadow on target recognition is reduced;

[0067] The target detection module is used to perform target detection on the image inspection data of environmental perception analysis using deep learning algorithms, identify the pavement diseases and non-disease targets, improve the accuracy and robustness of target detection, reduce the occurrence of misjudgment and missed judgment, receive the image inspection data after illumination and weather correction and shadow occlusion processing from the environmental perception analysis module, extract historical inspection data from it, and mark the positions and categories of the disease and non-disease targets. Integrate the marked historical inspection data and divide it into a training set and a test set. The training set is used to train the target recognition model, and the test set is used to evaluate the performance of the model. Use the training set data combined with the convolutional neural network model to train the target recognition model using the disease and non-disease target data in it. During the training process, the model learns how to identify the marked targets, generate corresponding bounding boxes and class labels, and optimize the model by adjusting model parameters, using data augmentation techniques, introducing regularization methods, etc., to improve its generalization ability and accuracy. And use the test set data to evaluate the trained target recognition model, calculate indicators such as the accuracy rate, recall rate, and F1 value of the model, evaluate the performance of the model, and further adjust the model parameters and optimization strategies according to the evaluation results to improve the accuracy and robustness of the model. Input the preprocessed real-time inspection data into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease class labels, and then classifies and locates the detected targets according to the generated bounding boxes and disease class labels, distinguishes the disease targets from the non-disease targets, and determines the position of the targets in the image through the bounding boxes;

[0068] Among them, the process of classifying and locating the detected targets is as follows:

[0069] The preprocessed real-time inspection image is passed through multiple convolutional layers in the object recognition model. Each convolutional layer contains multiple convolutional kernels. The convolutional kernels slide over the image and perform convolutional operations with local regions of the image to extract local features of the image. A 3x3 convolutional kernel slides over the image and performs a dot product operation with each 3x3 region in the image to obtain a new feature map. After the convolutional layer, an average pooling operation of the pooling layer is used to reduce the size of the feature map to reduce the dimension of the features and retain key features. The object recognition model generates candidate regions through a region proposal network. The region proposal network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes, and aspect ratios of the image. Then, the non-maximum suppression algorithm is used to screen the generated candidate regions, removing redundant or unreasonable regions. Among them, the non-maximum suppression algorithm removes candidate regions with a high degree of overlap based on the confidence score of the candidate regions and retains the candidate region with the highest confidence. The feature map corresponding to the screened candidate regions is sent to the fully connected layer to extract semantic features. The softmax function is used to classify the extracted semantic features and output the classification results in the form of a probability vector to determine the probability that each candidate region belongs to a disease target or a non-disease target. And the coordinates of the bounding box of each candidate region are predicted through a regression algorithm. Then, by calculating the difference between the predicted bounding box and the true bounding box, the mean squared error loss is used to optimize the regression result of the bounding box. Among them, the coordinates of the bounding box are represented by the offset relative to the candidate region, including the center point coordinates, width, and height. For the classification and bounding box regression results of multiple candidate regions, if multiple candidate regions correspond to the same target, they are merged into one target. The non-maximum suppression algorithm is used to further screen and merge the classification results and the bounding box regression results to retain the most reliable object detection results. The object recognition model outputs the position, category, and confidence of the detected objects in the image. Each detection result includes a bounding box and a disease category label. Among them, the bounding box is used to determine the position of the target in the image, and the disease category label is used to distinguish between disease targets and non-disease targets;

[0070] The disease target classification module is used to extract features and classify the detected targets, and combine classification algorithms to distinguish the specific categories of real disease targets, improve the recognition accuracy. Based on the real disease targets output by the target detection module, analyze their features, extract the shape features, texture features and color features of the real disease targets, and then adopt a weighted average feature fusion method to combine different types of features into a feature vector. Among them, analyze the shape information of the target, including area, perimeter and aspect ratio, use the texture analysis method of gray level co-occurrence matrix to extract the texture features of the target, and calculate the color features of the target through color histogram. For example, some disease targets (such as oil stains) may have specific colors, which are significantly different from the surrounding road surface colors. For example, the area and perimeter of cracks are usually small, but the aspect ratio may be large, the area and perimeter of potholes are relatively large and the shape is irregular. The texture at the crack is usually rough, and the contrast and entropy values of the gray level co-occurrence matrix may be high, while the texture of the normal road surface is relatively smooth, and the contrast and entropy values are low. Some disease targets (such as oil stains) may have specific colors, which are significantly different from the surrounding road surface colors. According to the characteristics of disease targets and classification requirements, use the feature dataset of known disease targets to train the classification algorithm of support vector machine to obtain a disease classification model. The model learns the differences between the feature vectors of each category, constructs a classification decision boundary, and realizes the classification of newly detected disease targets. During the training process, adjust the parameters of the support vector machine (C parameter, kernel function, etc.) to optimize the model performance. Optimize the classification model by methods such as increasing the dataset size, using data augmentation techniques (rotation, scaling, flipping, etc.) and adjusting model parameters to improve the classification accuracy and the generalization ability of the model. Input the extracted and fused feature vectors into the trained disease classification model. The disease classification model calculates the probability of each target belonging to the real disease category according to the feature vector and outputs the label of the real disease category to which each target belongs;

[0071] The road surface disease dynamic recognition module is used to establish a dynamic disease recognition model by using machine learning technology, compare with historical data, identify the evolution trend of diseases, improve the accuracy and timeliness of disease recognition, and enhance the reliability of disease recognition;

[0072] The inspection and warning management module is used to analyze the inspection data in real time, warn of existing diseases, send an alarm to the staff, and prompt potential road disease problems, so as to ensure that the inspection personnel can discover problems in time and take measures to effectively reduce road safety hazards.

[0073] Embodiment 2, as Figure 1 、 Figure 2 shown, on the basis of Embodiment 1, the present invention provides a technical solution: Preferably, the road surface disease dynamic recognition module specifically includes;

[0074] Obtain the latest disease target data from the object detection module and the disease target classification module, including the type and location information of the diseases, as well as the corresponding image data, and collect the disease data accumulated during previous inspections, including the development history and treatment records of the diseases. Integrate the historical data with the real-time inspection data to form a complete time series dataset. Extract the features related to disease evolution from the integrated data, including the shape features, texture features, color features of the diseases, and the disease locations. Furthermore, integrate the extracted features related to disease evolution to obtain a feature dataset, and divide it into a training set and a test set. Use the training set data to train the time series analysis model to construct a dynamic disease recognition model. During the training process, the time series analysis model learns the relationship between disease features and evolution trends, adjusts the model parameters, minimizes the prediction error, and uses the test set to test and evaluate the established dynamic disease recognition model. Then, optimize the model to improve its generalization ability and accuracy. Input the real-time inspection data into the trained dynamic disease recognition model, compare the newly detected disease features with the disease features in the historical data, analyze the similarity and difference between the features through the similarity calculation method, calculate the disease evolution tendency index, and then analyze the evolution process of historical diseases, identify the evolution trend and severity of new diseases;

[0075] Among them, the calculation process of the disease evolution tendency index is as follows:

[0076] Obtain the feature vector of the newly detected disease and the historical disease feature vector , for each historical data point j, calculate the sum of the squares of the weighted Euclidean distances between the newly detected disease features and the disease features in the historical data, and then take the square root of the sum of the squares of the weighted Euclidean distances to obtain the weighted Euclidean distance. Perform similarity calculation based on the exponential operation method. Multiply the weighted Euclidean distance by the negative value of the scaling parameter in the similarity calculation and perform the exponential operation to analyze the distribution of the disease features in the historical data over time. Sum the similarity calculation results for all historical data points j, and divide the summation result by the number of historical data points to obtain the average value. Then, subtract this average value from 1 to obtain the similarity summary part. Combine the parameter affected by the scaling trend and the slope calculation logic of the similarity time series regression to obtain the logical function part. Then, multiply the logical function part by the similarity summary part to obtain the final disease evolution tendency index;

[0077] The expression of the disease evolution tendency index is:

[0078]

[0079]

[0080] In the formula, is the disease evolution trend index, is the newly detected disease feature vector, , is the historical disease feature vector, , is the feature weight, is the scaling parameter in the similarity calculation, is the slope obtained from the regression of the similarity time series, is to map the slope to the scaling parameter of the logistic function, is the number of historical data points, is the number of disease features, is the th disease feature, is the index of the historical data point, is the th disease feature of the historical data point, is the weighted Euclidean distance, ranges from (0, 1), and a higher value indicates a more severe evolution trend and severity;

[0081] The inspection and early warning management module includes an early warning prompt unit and an inspection and handling unit. The early warning prompt unit is used to give early warning prompts for the identified real disease targets and send out early warning information, including the real disease target category, disease location, disease evolution trend, and early warning time. The inspection and handling unit is used to plan, allocate, execute, and track inspection tasks, including setting inspection routes, times, personnel, etc., improving the organization and efficiency of inspection work, ensuring that inspection tasks can be completed comprehensively and orderly, avoiding omissions and repetitions. The early warning prompt unit receives the inspection analysis results output from the target detection module, disease target classification module, and road surface disease dynamic identification module, including disease type, location, severity, and evolution trend information. According to the preset early warning threshold, it judges whether an early warning needs to be issued. When the disease evolution trend index exceeds the preset early warning threshold, the early warning condition is triggered, and then an early warning prompt containing detailed disease information is generated, including: disease target category, disease location, disease evolution trend, and early warning time. Among them, the disease target category clearly indicates the type of disease, such as cracks, potholes, oil stains, etc., so that the staff can quickly understand the nature of the disease. The disease location provides the specific location information of the disease on the highway, such as section number, mileage number, etc., which is convenient for inspection personnel to accurately locate. The disease evolution trend shows the development trend of the disease, such as the expansion speed of cracks, the deepening degree of potholes, etc., which helps the staff evaluate the urgency of the disease. The early warning time records the time when the early warning is issued, so as to trace and count early warning events, and then the early warning information is sent to relevant personnel through multiple channels such as text messages, emails, and system notifications, ensuring that the information can be conveyed to inspection personnel and management personnel in a timely manner. The inspection and handling unit plans inspection tasks according to the early warning information and inspection plan, comprehensively considers factors such as the location and severity of the disease and the distribution of inspection personnel, arranges inspection routes, gives priority to handling high-risk diseases, and assigns inspection tasks to specific inspection personnel, clarifying the responsible person and completion time of the task. The inspection personnel go to the disease location for on-site inspection and handling according to the assigned tasks, record the handling situation and results during the execution process, including the repair situation of the disease and the measures taken, etc., and track the execution situation of the inspection tasks in real time to ensure that the tasks are completed on time. Through the system to monitor the task progress, problems occurring during the execution process are discovered and solved in a timely manner. After the inspection personnel complete the tasks, the handling results are fed back to the system.

[0082] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. A highway air-ground integrated intelligent inspection system, including an inspection management platform, characterized in that: The inspection management platform is communicatively connected with a data acquisition module, an environmental perception analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module and an inspection warning management module; The data acquisition module is used to collect and pre-process various inspection data required for highway inspection; The environmental perception analysis module is used to analyze the pre-processed inspection data and remove interference information; The target detection module is used to perform target detection on the image inspection data of environmental perception analysis using a deep learning algorithm to identify diseased and non-disease targets on the road surface; The disease target classification module is used to extract features and classify detected targets, and distinguish the specific categories of real disease targets in combination with classification algorithms; The pavement disease dynamic identification module is used to establish a dynamic disease identification model using machine learning technology, compare it with historical data, and identify the evolution trend of the disease; The inspection and early warning management module is used to analyze the inspection data in real time and to warn of existing diseases.

2. According to claim 1, the highway air-ground integrated intelligent inspection and patrol system is characterized by: The data acquisition module specifically includes: Determine the target areas for highway inspections and conduct demand planning for air and ground inspections to obtain various types of inspection data including drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information; According to the inspection requirements of the highway and the scope of the target area, the flight route, flight altitude and shooting angle parameters of the drone are planned, and the drone automatically flies according to the preset flight route to obtain image data, and then the captured image data is wirelessly transmitted to the storage server of the data acquisition module; Various types of ground sensors are deployed along the highway, including meteorological sensors, temperature and humidity sensors, and vibration sensors. The ground sensors collect various types of data and transmit the data to the storage server of the data acquisition module through a wireless communication network; Install high-definition surveillance cameras at important sections, intersections and service areas of the expressway, and transmit the video stream data collected by the surveillance cameras to the storage server of the data acquisition module through the network; Environmental sensors including light sensors, rain sensors and snow sensors are set in the inspection target area to monitor the light intensity, rainfall and snowfall environmental information during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through the communication network; By installing traffic flow monitoring equipment at key locations on the expressway, the traffic flow and speed information of vehicles are monitored and the data is transmitted to the storage server of the data acquisition module. The road condition monitoring equipment is used to detect the flatness and damage of the road and the relevant data is transmitted to the storage server of the data acquisition module. Preprocess the various inspection data of the target area, including data cleaning and data format conversion, and encode the preprocessed data to assign a unique identifier to each type of data; The pre-processed drone image data, ground sensor data, surveillance video, environmental sensor data, and multi-source data of traffic flow and road condition information are fused to form an inspection data set.

3. According to claim 2, the highway air-ground integrated intelligent inspection and patrol system is characterized by: The environmental perception analysis module includes a lighting and weather correction unit and a shadow occlusion processing unit; The illumination and weather correction unit is used to correct the influence of illumination changes and weather factors on the pre-processed image inspection data; The shadow occlusion processing unit is used to process the shadow occlusion area in the image inspection data to remove or weaken the influence of the shadow occlusion area.

4. According to claim 3, the highway air-ground integrated intelligent inspection system is characterized by: The environmental perception analysis module specifically includes: Extract the pre-processed inspection data from the data acquisition module, and analyze the pre-processed image inspection data through the illumination and weather correction unit and the shadow occlusion processing unit; The image is divided into multiple areas through the illumination and weather correction unit, the brightness value of each pixel in the image is calculated, the illumination intensity of each area is calculated respectively, and the overall illumination intensity of the image is determined to identify the illumination difference between different areas in the image, and the weather conditions are judged according to the color and texture characteristics of the image. The image is gamma corrected according to the illumination intensity and weather factors to adjust the brightness and contrast of the image; The shadow occlusion processing unit converts the image from the RGB color space to the Lab color space, and uses the image segmentation and edge detection algorithms to identify the shadow area in the image, detect the edge information in the image, and further confirm the position and shape of the shadow area; The detected shadow area is processed by image restoration, and the pixel values ​​of the shadow area are filled with the image restoration algorithm. The shadow area is color corrected and the color of the shadow area is adjusted to match the surrounding environment. Then, the shadow area is compensated for illumination and the brightness of the shadow area is increased to make it consistent with the lighting conditions of the surrounding environment.

5. According to claim 4, the highway air-ground integrated intelligent inspection and patrol system is characterized by: The target detection module specifically includes: Receive image inspection data after illumination and weather correction and shadow occlusion processing from the environmental perception analysis module, extract historical inspection data from it, mark the locations and categories of diseased and non-disease targets, integrate the marked historical inspection data, and divide it into training set and test set; Use the training set data combined with the convolutional neural network model, use the diseased and non-disease target data in it to train the target recognition model, and use the test set data to evaluate the trained target recognition model and the performance of the model. According to the evaluation results, further adjust the model parameters and optimization strategy; The preprocessed real-time inspection data is input into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease category labels. Then, based on the generated bounding boxes and disease category labels, the detected targets are classified and located to distinguish between diseased targets and non-disease targets.

6. The highway air-ground integrated intelligent inspection system according to claim 5 is characterized by: The process of classifying and locating the detected targets is as follows: The pre-processed real-time inspection image is input to the multi-layer convolution layer in the target recognition model. Each convolution layer contains multiple convolution kernels. The convolution kernel slides on the image and performs convolution operation with the local area of ​​the image. The local features of the image are extracted through convolution operation. A 3x3 convolution kernel slides on the image and performs dot product operation with each 3x3 area in the image to obtain a new feature map. After the convolution layer, the average pooling operation of the pooling layer is used to reduce the size of the feature map to reduce the dimension of the feature. The target recognition model generates candidate regions through the region candidate network. The region candidate network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes and aspect ratios of the image. Then, the non-maximum suppression algorithm is used to screen the generated candidate regions to remove redundant or unreasonable regions. The non-maximum suppression algorithm removes candidate regions with high overlap according to the confidence scores of the candidate regions and retains the candidate regions with the highest confidence. The feature map corresponding to the screened candidate area is sent to the fully connected layer to extract semantic features. The extracted semantic features are classified using the softmax function, and the classification results are output in the form of a probability vector to determine the probability that each candidate area belongs to a diseased target or a non-disease target. The bounding box coordinates of each candidate area are predicted by the regression algorithm, and then the difference between the predicted bounding box and the true bounding box is calculated. The regression result of the bounding box is optimized using the mean square error loss, where the coordinates of the bounding box are expressed as an offset relative to the candidate area, including the center point coordinates, width, and height; For the classification and bounding box regression results of multiple candidate regions, if multiple candidate regions correspond to the same target, they are merged into one target, and the classification results and bounding box regression results are further screened and merged using the non-maximum suppression algorithm; The target recognition model outputs the location, category, and confidence of the target detected in the image. Each detection result includes a bounding box and a disease category label. The bounding box is used to determine the location of the target in the image, and the disease category label is used to distinguish diseased targets from non-diseased targets.

7. The highway air-ground integrated intelligent inspection system according to claim 6 is characterized by: The disease target classification module specifically includes: Based on the real disease target output by the target detection module, feature analysis is performed to extract the shape features, texture features and color features of the real disease target, and then the weighted average feature fusion method is used to combine different types of features into a feature vector; According to the characteristics and classification requirements of the disease targets, the classification algorithm of the support vector machine is trained using the feature data set of the known disease targets to obtain the disease classification model; The extracted and fused feature vectors are input into the trained disease classification model. The disease classification model calculates the probability of the true disease category to which each target belongs based on the feature vectors, and outputs the true disease category label to which each target belongs.

8. The highway air-ground integrated intelligent inspection system according to claim 7 is characterized by: The pavement disease dynamic identification module specifically includes: Obtain the latest disease target data from the target detection module and the disease target classification module, including the type and location information of the disease, as well as the corresponding image data, and collect the disease data accumulated during the previous inspection process, including the development history and treatment records of the disease, and integrate the historical data with the real-time inspection data to form a complete time series data set; Extracting features related to the evolution of the disease from the integrated data, including shape features, texture features, color features, and disease locations of the disease, and then integrating the extracted features related to the evolution of the disease to obtain a feature data set, and dividing it into a training set and a test set; Use the training set data to train the time series analysis model, build a dynamic disease recognition model, and use the test set to test and evaluate the established dynamic disease recognition model, and then optimize the model; The real-time inspection data is input into the trained dynamic disease recognition model, and the newly detected disease features are compared with the disease features in the historical data. The similarities and differences between the features are analyzed through the similarity calculation method, and the disease evolution trend index is calculated. The evolution process of historical diseases is analyzed, and the evolution trend and severity of new diseases are identified.

9. The highway air-ground integrated intelligent inspection system according to claim 8 is characterized by: The inspection warning management module includes an early warning prompt unit and an inspection disposal unit; The early warning prompt unit is used to provide early warning prompts for the identified real disease targets and issue early warning information, including the real disease target category, disease location, disease evolution trend and early warning time; The inspection handling unit is used to plan, allocate, execute and track inspection tasks.

10. The highway air-ground integrated intelligent inspection system according to claim 9 is characterized by: The inspection and early warning management module specifically includes: The early warning prompt unit receives the inspection and analysis results output by the target detection module, the disease target classification module and the pavement disease dynamic identification module, including the type, location, severity and evolution trend information of the disease; According to the preset warning threshold, it is determined whether a warning needs to be issued. When the disease evolution trend index exceeds the preset warning threshold, the warning condition is triggered, and then a warning prompt containing detailed disease information is generated, including: disease target category, disease location, disease evolution trend and warning time, and then the warning information is issued to relevant personnel through multiple channels such as SMS, email, and system notification; The inspection and disposal unit plans the inspection tasks according to the early warning information and inspection plan, arranges the inspection routes based on the location, severity and distribution of the disease, and assigns the inspection tasks to specific inspection personnel, clearly defines the person responsible for the task and the completion time. The inspection personnel go to the disease location for on-site inspection and treatment according to the assigned tasks, and record the treatment status and results during the execution process; Track the execution of inspection tasks in real time, monitor the progress of tasks through the system, discover and solve problems that arise during the execution process, and after the inspectors complete the tasks, they will feedback the processing results to the system.

Images