An integrated air-ground intelligent inspection system for highways

Abstract

The present invention discloses an integrated air-ground intelligent highway inspection and patrol system, which relates to the field of highway inspection technology and includes an inspection and patrol management platform. The inspection and patrol management platform is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module, and an inspection and early warning management module. The data acquisition module is used to collect and pre-process various types of inspection data required for highway inspections. Through the coordinated operation of aerial inspections by drones and ground sensors, the present invention can quickly acquire inspection data over a large area, achieving all-round, real-time monitoring of highways, significantly improving inspection efficiency, and being able to complete large-scale inspection tasks in a short period of time, promptly identifying potential road disease problems, providing timely and accurate information support to road maintenance and management departments, and ensuring the safe and smooth operation of highways.

Description

Technical Field

[0001] The present invention relates to the field of highway inspection technology, and in particular to an air-ground integrated intelligent inspection system for highways. Background Art

[0002] As an important part of the modern transportation network, expressways undertake a large amount of passenger and freight transportation tasks and play a key supporting role in economic development. Their safe and efficient operation is directly related to the travel safety of the people and the stable development of the social economy. However, expressway inspection work faces many challenges. Traditional expressway inspection methods mainly rely on manual inspections and ground patrols. This method usually has problems such as limited inspection range, low efficiency, and delayed information transmission. With the rapid development of technologies such as the Internet of Things, artificial intelligence, and drones, new solutions have been provided for expressway inspection work.

[0003] During highway inspections, road debris and temporary construction signs are easily misidentified as defects due to interference from factors such as lighting, weather, and shadows, which in turn reduces the recognition accuracy of real defect targets. To this end, a highway air-ground integrated intelligent inspection system is proposed to improve the recognition accuracy of highway inspections. Summary of the Invention

[0004] The purpose of the present invention is to provide an integrated intelligent inspection system for highways to solve the problems raised in the above-mentioned background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] An integrated highway air-ground intelligent inspection and patrol system includes an inspection and patrol management platform, which is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module, and an inspection and early warning management module;

[0007] The data acquisition module is used to collect and pre-process various types of inspection data required for highway inspections, including drone image data, ground sensor data, real-time monitoring video, environmental sensor data, and traffic flow and road condition information;

[0008] The environmental perception analysis module is used to analyze the pre-processed inspection data and remove interference information;

[0009] The target detection module is used to perform target detection on the image inspection data of environmental perception analysis using a deep learning algorithm, identify diseased and non-disease targets on the road surface, improve the accuracy and robustness of target detection, and reduce the occurrence of misjudgments and missed judgments;

[0010] The disease target classification module is used to extract features and classify detected targets, and distinguish the specific categories of real disease targets in combination with classification algorithms to improve recognition accuracy;

[0011] The pavement disease dynamic identification module is used to establish a dynamic disease identification model using machine learning technology, compare it with historical data, identify the evolution trend of diseases, improve the accuracy and timeliness of disease identification, and enhance the reliability of disease identification;

[0012] The inspection and early warning management module is used to analyze the inspection data in real time and to warn of existing diseases.

[0013] A further improvement of the technical solution of the present invention is that the data acquisition module specifically includes:

[0014] Identify target areas for highway inspections and conduct demand planning for air and ground inspections to obtain various inspection data, including drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information;

[0015] According to the inspection requirements of the expressway and the scope of the target area, the drone's flight route, flight altitude and shooting angle parameters are planned to ensure that the target area can be fully covered. The drone automatically flies according to the preset flight route, using the high-definition camera and infrared camera equipment on board to shoot the expressway and its surrounding environment, obtaining high-resolution image data, and then wirelessly transmitting the captured image data to the storage server of the data acquisition module;

[0016] Various types of ground sensors are deployed along the highway, including meteorological sensors (monitoring temperature, humidity, wind speed, wind direction and other meteorological parameters), temperature and humidity sensors (monitoring road surface temperature and humidity), and vibration sensors (monitoring the vibration of structures such as bridges and tunnels). The ground sensors collect various types of data and transmit the data to the storage server of the data acquisition module via wireless communication networks;

[0017] Install high-definition surveillance cameras at important sections, intersections, and service areas of the expressway to conduct all-round monitoring of the expressway, and transmit the video stream data collected by the surveillance cameras to the storage server of the data acquisition module through the network;

[0018] Environmental sensors, including light sensors, rain sensors, and snow sensors, are installed in the inspection target area to monitor environmental information such as light intensity, rainfall, and snowfall during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through the communication network, providing data support for subsequent environmental perception analysis.

[0019] Traffic flow monitoring equipment (induction coils, video monitors, etc.) installed at key locations on the highway monitors vehicle flow and speed information, and transmits the data to the storage server of the data acquisition module. Road condition monitoring equipment (pavement smoothness detectors, pavement damage detectors, etc.) is used to detect the smoothness, damage level, and other conditions of the road, and transmits the relevant data to the storage server of the data acquisition module.

[0020] Pre-process the various inspection data acquired in the target area, including data cleaning and data format conversion, and encode the pre-processed data to assign a unique identifier to each type of data;

[0021] The pre-processed drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information multi-source data are fused to form an inspection data set.

[0022] A further improvement of the technical solution of the present invention is that: the environment perception analysis module includes a lighting and weather correction unit and a shadow occlusion processing unit;

[0023] The illumination and weather correction unit is used to correct the influence of illumination changes and weather factors on the pre-processed image inspection data, adjust the brightness and contrast parameters of the image, and reduce the influence of illumination and weather changes on target recognition;

[0024] The shadow occlusion processing unit is used to process the shadow occlusion area in the image inspection data, reduce the influence of the shadow on the target recognition, and remove or weaken the influence of the shadow occlusion area.

[0025] A further improvement of the technical solution of the present invention is that the environmental perception analysis module specifically includes:

[0026] Extract the pre-processed inspection data from the data acquisition module, and analyze the pre-processed image inspection data through the illumination and weather correction unit and the shadow occlusion processing unit;

[0027] The lighting and weather correction unit divides the image into multiple regions, calculates the brightness value of each pixel in the image, calculates the lighting intensity of each region separately, and determines the overall lighting intensity of the image. This allows identification of lighting differences between different regions in the image, and determines weather conditions based on the image's color and texture features. Gamma correction is then performed on the image based on the lighting intensity and weather factors, adjusting the image's brightness and contrast. By adjusting the gamma value, details in dark or bright areas of the image can be enhanced.

[0028] The shadow occlusion processing unit converts the image from RGB color space to Lab color space, and uses image segmentation and edge detection algorithms to identify the shadow area in the image, detect the edge information in the image, and further confirm the position and shape of the shadow area;

[0029] The detected shadow area is subjected to image restoration processing, and the pixel values ​​of the shadow area are filled with the image restoration algorithm. The image restoration algorithm can generate pixel values ​​consistent with the surrounding environment based on the information of the surrounding pixels, thereby removing the influence of the shadow, and performing color correction on the shadow area, adjusting the color of the shadow area to match the surrounding environment. By adjusting the color parameters, the influence of the shadow on target recognition can be weakened, and then the shadow area is subjected to illumination compensation, and the brightness of the shadow area is increased to make it consistent with the lighting conditions of the surrounding environment.

[0030] A further improvement of the technical solution of the present invention is that the target detection module specifically includes:

[0031] The environmental perception and analysis module receives image inspection data that has been corrected for lighting and weather, as well as shadows, and extracts historical inspection data. The data is then labeled with the locations and categories of diseased and non-disease targets. The labeled historical inspection data is then integrated and divided into a training set and a test set. The training set is used to train the target recognition model, and the test set is used to evaluate the model's performance.

[0032] The training set data is combined with a convolutional neural network model, and the diseased and non-disease target data are used to train the target recognition model. The trained target recognition model is then evaluated using the test set data. The model's performance is evaluated by calculating metrics such as accuracy, recall, and F1 value. Based on the evaluation results, the model parameters and optimization strategies are further adjusted to improve the model's accuracy and robustness.

[0033] The preprocessed real-time inspection data is input into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease category labels. Then, based on the generated bounding boxes and disease category labels, the detected targets are classified and located, diseased targets are distinguished from non-diseased targets, and the position of the targets in the image is determined by the bounding boxes.

[0034] A further improvement of the technical solution of the present invention is that the process of classifying and locating the detected targets is as follows:

[0035] The pre-processed real-time inspection image is input and passed through the multi-layer convolutional layers in the target recognition model. Each convolutional layer contains multiple convolution kernels, which slide across the image and perform convolution operations with local areas of the image. The convolution operation extracts local features of the image. A 3x3 convolution kernel slides across the image and performs dot product operations with each 3x3 area in the image to obtain a new feature map. The average pooling operation of the pooling layer after the convolution layer is used to reduce the size of the feature map to reduce the feature dimension and retain key features.

[0036] The target recognition model generates candidate regions through a region proposal network. The region proposal network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes, and aspect ratios of the image. The generated candidate regions are then screened using a non-maximum suppression algorithm to remove redundant or unreasonable regions. The non-maximum suppression algorithm removes candidate regions with high overlap based on their confidence scores and retains the candidate regions with the highest confidence.

[0037] The feature maps corresponding to the filtered candidate regions are fed into the fully connected layer to extract semantic features. The extracted semantic features are classified using the softmax function, and the classification results are output as probability vectors. The probability of each candidate region belonging to a diseased target or a non-diseased target is determined, and the bounding box coordinates of each candidate region are predicted using a regression algorithm. The difference between the predicted bounding box and the true bounding box is then calculated, and the mean square error loss is used to optimize the bounding box regression results. The coordinates of the bounding box are expressed as an offset relative to the candidate region, including the center point coordinates, width, and height.

[0038] For the classification and bounding box regression results of multiple candidate regions, if multiple candidate regions correspond to the same target, they are merged into one target, and the non-maximum suppression algorithm is used to further filter and merge the classification results and bounding box regression results to retain the most reliable target detection results;

[0039] The target recognition model outputs the location, category, and confidence of the target detected in the image. Each detection result includes a bounding box and a disease category label. The bounding box is used to determine the location of the target in the image, and the disease category label is used to distinguish diseased targets from non-diseased targets.

[0040] A further improvement of the technical solution of the present invention is that the disease target classification module specifically includes:

[0041] Based on the real diseased target output by the target detection module, feature analysis is performed to extract the shape, texture and color features of the real diseased target. Then, a weighted average feature fusion method is used to combine different types of features into a feature vector.

[0042] According to the characteristics of the disease target and the classification requirements, the support vector machine classification algorithm is trained using the feature data set of the known disease target to obtain the disease classification model;

[0043] The extracted and fused feature vectors are input into the trained disease classification model. The disease classification model calculates the probability of the true disease category of each target based on the feature vector and outputs the true disease category label of each target.

[0044] A further improvement of the technical solution of the present invention is that: the pavement disease dynamic identification module specifically includes:

[0045] The latest disease target data, including disease type and location information and corresponding image data, is obtained from the target detection module and the disease target classification module. The disease data accumulated during previous inspections, including the disease development history and treatment records, is collected. The historical data is integrated with the real-time inspection data to form a complete time series data set.

[0046] Extract features related to disease evolution from the integrated data, including shape, texture, color, and location of the disease. Then, integrate the extracted features related to disease evolution to obtain a feature dataset, which is divided into a training set and a test set.

[0047] The training set data is used to train the time series analysis model and construct a dynamic disease recognition model. During the training process, the time series analysis model learns the relationship between disease characteristics and evolution trends, adjusts model parameters, and minimizes prediction errors. The established dynamic disease recognition model is then tested and evaluated using the test set, thereby optimizing the model and improving its generalization ability and accuracy.

[0048] The real-time inspection data is input into the trained dynamic disease recognition model, and the newly detected disease features are compared with the disease features in the historical data. The similarities and differences between the features are analyzed through the similarity calculation method, and the disease evolution trend index is calculated. Then, the evolution process of historical diseases is analyzed, and the evolution trend and severity of new diseases are identified.

[0049] A further improvement of the technical solution of the present invention is that: the inspection warning management module includes an early warning prompt unit and an inspection disposal unit;

[0050] The early warning prompt unit is used to provide early warning prompts for the identified real disease targets and issue early warning information, including the real disease target category, disease location, disease evolution trend and warning time;

[0051] The inspection handling unit is used to plan, allocate, execute and track inspection tasks, including setting inspection routes, time, personnel, etc.

[0052] A further improvement of the technical solution of the present invention is that the inspection and early warning management module specifically includes:

[0053] The early warning prompt unit receives the inspection and analysis results output by the target detection module, the disease target classification module and the pavement disease dynamic identification module, including the type, location, severity and evolution trend of the disease;

[0054] Based on the preset warning threshold, the system determines whether a warning is necessary. When the disease evolution trend index exceeds the preset warning threshold, the warning condition is triggered, and a warning prompt containing detailed disease information is generated, including: disease target category, disease location, disease evolution trend and warning time. The warning information is then released to relevant personnel through multiple channels such as SMS, email, and system notifications to ensure that the information can be conveyed to inspection personnel and management personnel in a timely manner.

[0055] The inspection and disposal unit plans inspection tasks based on early warning information and inspection plans. It arranges inspection routes based on the location, severity, and distribution of inspection personnel, prioritizing high-risk diseases. It also assigns inspection tasks to specific inspection personnel, clearly identifying the responsible individuals and completion deadlines. Inspection personnel, based on assigned tasks, conduct on-site inspections and address the disease locations. During the execution process, they record the treatment status and results, including the repair status of the disease and the measures taken.

[0056] Track the execution of inspection tasks in real time to ensure that tasks are completed on time. Monitor the progress of tasks through the system to promptly discover and resolve problems that arise during the execution process. After the inspection personnel complete the task, they will feedback the processing results to the system.

[0057] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art:

[0058] 1. The present invention provides an integrated air-ground intelligent inspection system for highways. Through the coordinated work of aerial inspections by drones and ground sensors, it can quickly obtain inspection data over a large area, realizing all-round and real-time monitoring of highways. This significantly improves inspection efficiency, enables the completion of large-area inspection tasks in a short period of time, and promptly discovers potential road damage problems. It provides timely and accurate information support to road maintenance and management departments, thus ensuring the safe and smooth operation of highways.

[0059] 2. The present invention provides an integrated air-ground intelligent inspection and patrol system for highways. Through the collaborative work of the environmental perception and analysis module, the target detection module and the disease target classification module, it can effectively improve the accuracy of disease identification, reduce the occurrence of misjudgment and missed judgment, and realize the precise identification and classification of disease targets, and accurately judge the type, location and severity of the disease. Combined with the real-time data transmission and processing mechanism, it ensures the immediate feedback of disease information. At the same time, the system can also adjust the inspection strategy according to the evolution trend of the disease, realize more targeted inspection operations, and further improve maintenance efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0061] Figure 1 This is a schematic diagram of the system module flow of the present invention;

[0062] Figure 2 Schematic diagram of the working process of the pavement disease dynamic identification module of the present invention. DETAILED DESCRIPTION

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] Example 1, as Figure 1 As shown, the present invention provides an integrated air-ground intelligent inspection and patrol system for highways, including an inspection and patrol management platform, which is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module, and an inspection and early warning management module;

[0065] The data acquisition module is used to collect and pre-process various inspection data required for highway inspections, including drone image data, ground sensor data, real-time monitoring video, environmental sensor data, as well as traffic flow and road condition information, determine the target area for highway inspections, and carry out demand planning for air-ground inspections to obtain various inspection data including drone image data, ground sensor data, monitoring video, environmental sensor data, as well as traffic flow and road condition information. According to the inspection needs of the highway and the scope of the target area, the flight route, flight altitude and shooting angle parameters of the drone are planned to ensure that the target area for inspection can be fully covered and the drone can automatically fly according to the preset flight route. The high-definition camera and infrared camera equipment carried by the vehicle are used to shoot the highway and its surrounding environment to obtain high-resolution image data, which is then wirelessly transmitted to the storage server of the data acquisition module. Various types of ground sensors are deployed along the highway, including meteorological sensors (monitoring meteorological parameters such as temperature, humidity, wind speed, and wind direction), temperature and humidity sensors (monitoring road surface temperature and humidity), and vibration sensors (monitoring the vibration of structures such as bridges and tunnels). The ground sensors collect various types of data and transmit the data to the storage server of the data acquisition module through a wireless communication network. High-definition surveillance cameras are installed at important sections, intersections, and service areas of the highway to conduct comprehensive monitoring of the highway. The system monitors the road conditions and transmits the video stream data collected by the surveillance camera to the storage server of the data acquisition module through the network. Environmental sensors including light sensors, rain sensors and snow sensors are set in the inspection target area to monitor the light intensity, rainfall and snowfall environmental information during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through the communication network to provide data support for subsequent environmental perception analysis. Traffic flow monitoring equipment (induction coils, video monitors, etc.) set at key locations on the highway monitors the flow and speed information of vehicles and transmits the data to the storage server of the data acquisition module. The road condition monitoring equipment (pavement flatness detector) is used to monitor the road conditions and the road surface roughness. , road damage detector, etc.) to detect the flatness, degree of damage and other conditions of the road, and transmit the relevant data to the storage server of the data acquisition module, and pre-process the various inspection data of the target area obtained, including data cleaning and data format conversion. Through data cleaning, the collected data is preliminarily screened to remove invalid data or abnormal values ​​caused by sensor failure, communication interference, etc. For some missing data, reasonable supplement is made according to the characteristics and relevance of the data to ensure the integrity and continuity of the data, and data of different types and sources are converted into a unified format to facilitate subsequent data processing and analysis, and the pre-processed data is encoded to give each type of data a unique identifier.The pre-processed drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information are integrated to form an inspection data set;

[0066] The environmental perception analysis module is used to analyze the pre-processed inspection data and remove interference information. The environmental perception analysis module includes a lighting and weather correction unit and a shadow occlusion processing unit. The lighting and weather correction unit is used to correct the influence of lighting changes and weather factors on the pre-processed image inspection data, adjust the brightness and contrast parameters of the image, reduce the influence of lighting and weather changes on target recognition, and improve the adaptability of the system. The shadow occlusion processing unit is used to process the shadow occlusion area in the image inspection data, reduce the influence of shadows on target recognition, remove or weaken the influence of shadow occlusion areas, and improve the accuracy and completeness of target recognition. The data acquisition module extracts the pre-processed inspection data, analyzes the pre-processed image inspection data through the illumination and weather correction unit and the shadow occlusion processing unit, divides the image into multiple areas through the illumination and weather correction unit, calculates the brightness value of each pixel in the image, calculates the illumination intensity of each area separately, determines the overall illumination intensity of the image, and identifies the illumination difference between different areas in the image. The weather condition is judged based on the color and texture feature analysis of the image. On cloudy days, the image color is usually darker and the color saturation is lower. On rainy days, the image may have a water mist effect and the color is blue or gray. Under severe weather conditions, the texture characteristics of the image are The image features will be affected. In rainy or foggy days, the edges of the image will become blurred and the details will be lost. According to the light intensity and weather factors, the image is gamma corrected to adjust the brightness and contrast of the image so that the image can maintain a good visual effect under different lighting conditions. By adjusting the gamma value, the dark details or bright details of the image are enhanced. The image is converted from RGB color space to Lab color space through the shadow occlusion processing unit. In Lab color space, the color features of the shadow area are made more obvious, which is convenient for shadow detection. The image segmentation and edge detection algorithms are used to identify the shadow area in the image and detect the edge information in the image. The first step is to confirm the position and shape of the shadow area, perform image restoration processing on the detected shadow area, and use the image restoration algorithm to fill the pixel value of the shadow area. The image restoration algorithm can generate pixel values ​​consistent with the surrounding environment based on the information of the surrounding pixels, thereby removing the influence of the shadow, and perform color correction on the shadow area, adjusting the color of the shadow area to match the surrounding environment. By adjusting the color parameters, the influence of the shadow on target recognition can be weakened, and then perform illumination compensation on the shadow area, increasing the brightness of the shadow area to make it consistent with the lighting conditions of the surrounding environment. Through illumination compensation, the visibility of the shadow area is improved and the interference of the shadow on target recognition is reduced;

[0067] The target detection module is used to use deep learning algorithms to perform target detection on the image inspection data of environmental perception analysis, identify diseased and non-disease targets on the road, improve the accuracy and robustness of target detection, and reduce the occurrence of misjudgment and missed judgment. It receives the image inspection data after illumination and weather correction and shadow occlusion processing from the environmental perception analysis module, extracts historical inspection data from it, and marks the location and category of diseased and non-disease targets, integrates the marked historical inspection data, and divides it into training set and test set. The training set is used to train the target recognition model, and the test set is used to evaluate the performance of the model. The training set data is combined with the convolutional neural network model, and the target recognition model is trained using the diseased and non-disease target data. During the training process, the model learns how to identify the marked targets and generates corresponding bounding boxes and category labels. The model is optimized by adjusting model parameters, using data enhancement technology, introducing regularization methods, etc. to improve its generalization ability and accuracy. The trained target recognition model is evaluated using the test set data, and the model's accuracy, recall rate, and F1 are calculated. The performance of the model is evaluated using indicators such as the accuracy and robustness of the model. Based on the evaluation results, the model parameters and optimization strategies are further adjusted to improve the accuracy and robustness of the model. The preprocessed real-time inspection data is input into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease category labels. Based on the generated bounding boxes and disease category labels, the detected targets are classified and located, and diseased targets are distinguished from non-disease targets. The position of the target in the image is determined by the bounding box.

[0068] The process of classifying and locating the detected targets is as follows:

[0069] The input preprocessed real-time inspection image passes through the multi-layer convolution layer in the target recognition model. Each convolution layer contains multiple convolution kernels. The convolution kernel slides on the image and performs convolution operation with the local area of ​​the image. The local features of the image are extracted by convolution operation. A 3x3 convolution kernel is used to slide on the image and perform dot product operation with each 3x3 area in the image to obtain a new feature map. The average pooling operation of the pooling layer is used after the convolution layer to reduce the size of the feature map to reduce the dimension of the feature and retain the key features. The target recognition model generates candidate regions through the region candidate network. The region candidate network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes and aspect ratios of the image, and then uses the non-maximum suppression algorithm to screen the generated candidate regions to remove redundant or unreasonable regions. Among them, the non-maximum suppression algorithm removes candidate regions with higher overlap according to the confidence score of the candidate region and retains the candidate region with the highest confidence. The feature map corresponding to the screened candidate region is sent to the fully connected layer to extract semantic features. The extracted semantic features are classified using the softmax function, and the classification results are output in the form of a probability vector to determine the probability that each candidate region belongs to a diseased target or a non-diseased target. The bounding box coordinates of each candidate region are predicted by the regression algorithm, and then the difference between the predicted bounding box and the true bounding box is calculated. The regression results of the bounding box are optimized using the mean square error loss, where the coordinates of the bounding box are expressed as an offset relative to the candidate region, including the center point coordinates, width, and height. For the classification and bounding box regression results of multiple candidate regions, if multiple candidate regions correspond to the same target, they are merged into one target. The non-maximum suppression algorithm is used to further filter and merge the classification results and bounding box regression results to retain the most reliable target detection results. The target recognition model outputs the position, category, and confidence of the target detected in the image. Each detection result includes a bounding box and a disease category label, where the bounding box is used to determine the position of the target in the image, and the disease category label is used to distinguish between diseased targets and non-diseased targets.

[0070] The defect target classification module is used to extract and classify the detected targets, and combine the classification algorithm to distinguish the specific categories of real defect targets to improve the recognition accuracy. Based on the real defect targets output by the target detection module, feature analysis is performed on them to extract the shape features, texture features and color features of the real defect targets. Then, the feature fusion method of weighted average is used to combine different types of features into a feature vector, wherein the shape information of the target is analyzed, including area, perimeter and aspect ratio. The texture analysis method of gray-level co-occurrence matrix is ​​used to extract the texture features of the target, and the color features of the target are calculated by color histogram. For example, some defect targets (such as oil stains) may have specific colors that are significantly different from the color of the surrounding road surface. For example, the area and perimeter of cracks are usually small, but the aspect ratio may be large. The area and perimeter of potholes are relatively large and the shape is irregular. The texture at the cracks is usually rough. The contrast and entropy values ​​of the gray-level co-occurrence matrix may be high, while the normal The texture of the road surface is relatively smooth, with low contrast and entropy values. Some disease targets (such as oil stains) may have a specific color that is significantly different from the color of the surrounding road surface. Based on the characteristics of the disease targets and classification requirements, the feature data set of known disease targets is used to train the support vector machine classification algorithm to obtain a disease classification model. The model learns the differences between the feature vectors of each category, constructs a classification decision boundary, and classifies newly detected disease targets. During the training process, the parameters of the support vector machine (C parameter, kernel function, etc.) are adjusted to optimize the model performance. The classification model is optimized by increasing the dataset size, using data enhancement techniques (rotation, scaling, flipping, etc.), and adjusting model parameters to improve classification accuracy and model generalization ability. The extracted and fused feature vectors are input into the trained disease classification model. The disease classification model calculates the probability of the true disease category of each target based on the feature vector and outputs the true disease category label of each target.

[0071] The pavement disease dynamic identification module is used to establish a dynamic disease identification model using machine learning technology, compare it with historical data, identify the evolution trend of diseases, improve the accuracy and timeliness of disease identification, and enhance the reliability of disease identification;

[0072] The inspection and early warning management module is used to conduct real-time analysis of inspection data, warn of existing defects, and send alarms to staff to remind them of potential road defects, ensuring that inspectors can detect problems in a timely manner and take measures to effectively reduce road safety hazards.

[0073] Example 2, as Figure 1 、 Figure 2 As shown, based on Example 1, the present invention provides a technical solution: preferably, the pavement disease dynamic identification module specifically includes:

[0074] The latest disease target data, including the type and location information of the disease and the corresponding image data, are obtained from the target detection module and the disease target classification module. The disease data accumulated during the previous inspection process, including the development history and treatment records of the disease, are collected. The historical data is integrated with the real-time inspection data to form a complete time series data set. Features related to the evolution of the disease are extracted from the integrated data, including the shape features, texture features, color features, and location of the disease. The extracted features related to the evolution of the disease are then integrated to obtain a feature data set, which is divided into a training set and a test set. The training set data is used to train the time series analysis model. , build a dynamic disease recognition model. During the training process, the time series analysis model learns the relationship between disease characteristics and evolution trends, adjusts model parameters, minimizes prediction errors, and uses a test set to test and evaluate the established dynamic disease recognition model, thereby optimizing the model to improve its generalization ability and accuracy. The real-time inspection data is input into the trained dynamic disease recognition model, and the newly detected disease characteristics are compared with the disease characteristics in the historical data. The similarity and difference between the characteristics are analyzed through the similarity calculation method, and the disease evolution trend index is calculated. Then, the evolution process of historical diseases is analyzed, and the evolution trend and severity of new diseases are identified;

[0075] The calculation process of the disease evolution trend index is as follows:

[0076] Get the newly detected disease feature vector and historical disease feature vector For each historical data point j, the sum of the squares of the weighted Euclidean distances between the newly detected disease features and the disease features in the historical data is calculated, and then the square root of the sum of the squares of the weighted Euclidean distances is taken to obtain the weighted Euclidean distance. Similarity calculation is performed based on the exponential operation method. The weighted Euclidean distance is multiplied by the negative value of the scaling parameter in the similarity calculation and an exponential operation is performed. The distribution of disease features in the historical data over time is analyzed. The similarity calculation results are summed for all historical data points j, and the sum is divided by the number of historical data points to obtain the average value. The average value is then subtracted from 1 to obtain the similarity summary part. The logistic function part is calculated by combining the parameters affecting the scaling trend and the slope of the similarity time series regression. The logistic function part and the similarity summary part are multiplied to obtain the final disease evolution trend index.

[0077] The expression of disease evolution trend index is:

[0078]

[0079]

[0080] Where, is the disease evolution trend index, is the newly detected disease feature vector, , is the historical disease feature vector, , is the feature weight, is the scaling parameter in similarity calculation, is the slope obtained from the similarity time series regression, To set the slope Scaling parameter mapped to the logistic function, is the number of historical data points, is the number of disease characteristics, For the Disease characteristics, is the index of the historical data point, For the Disease characteristics of historical data points, is the weighted Euclidean distance, The value range of is (0, 1), and higher values ​​indicate more serious evolution trends and severity;

[0081] The inspection warning management module includes an early warning prompt unit and an inspection disposal unit. The early warning prompt unit is used to provide early warning prompts for the identified real disease targets and issue early warning information, including the real disease target category, disease location, disease evolution trend and warning time. The inspection disposal unit is used to plan, allocate, execute and track inspection tasks, including setting inspection routes, time, personnel, etc., to improve the organization and efficiency of inspection work, ensure that inspection tasks can be completed comprehensively and orderly, and avoid omissions and duplications. The early warning prompt unit receives information from the target detection module, disease target classification module and road disease The inspection and analysis results output by the dynamic damage identification module include the type, location, severity and evolution trend of the damage. Based on the preset warning threshold, it is determined whether an early warning is needed. When the disease evolution trend index exceeds the preset warning threshold, the warning condition is triggered, and then an early warning prompt containing detailed disease information is generated, including: disease target category, disease location, disease evolution trend and warning time. Among them, the disease target category clearly indicates the type of disease, such as cracks, potholes, oil stains, etc., so that staff can quickly understand the nature of the disease. The disease location provides the specific location of the disease on the highway. Location information, such as section number and stake number, facilitates accurate positioning by inspectors. Defect evolution trends display development trends, such as crack expansion rate and pothole deepening, helping staff assess the urgency of the defect. Warning time records the time when warnings are issued, enabling the tracing and statistics of warning events. Warning information is then distributed to relevant personnel through multiple channels, such as text messages, emails, and in-system notifications, ensuring timely delivery to inspectors and management personnel. The inspection and disposal unit plans inspection tasks based on warning information and inspection plans. It arranges inspection routes based on the location, severity, and distribution of inspectors, prioritizing high-risk defects. Inspection tasks are then assigned to specific inspectors, with clear responsibilities and completion deadlines. Based on the assigned tasks, inspectors conduct on-site inspections and address the defect locations. During the execution process, the inspection status and results, including repair status and measures taken, are recorded. Inspection task execution is tracked in real time to ensure timely completion. The system monitors task progress to promptly identify and resolve problems that arise during execution. Upon completion of the task, inspectors provide feedback to the system on the results of the inspection.

[0082] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A highway air-ground integrated intelligent inspection and patrol system, including an inspection and patrol management platform, characterized by: The inspection management platform is communicatively connected to a data acquisition module, an environmental perception and analysis module, a target detection module, a disease target classification module, a pavement disease dynamic identification module, and an inspection warning management module; The data acquisition module is used to collect and pre-process various types of inspection data required for highway inspection; The environmental perception analysis module is used to analyze the pre-processed inspection data and remove interference information; The target detection module is used to perform target detection on the image inspection data obtained through environmental perception analysis using a deep learning algorithm, and identify diseased and non-disease targets on the road surface; The disease target classification module is used to extract features and classify detected targets, and distinguish the specific categories of real disease targets in combination with classification algorithms; The pavement disease dynamic identification module is used to establish a dynamic disease identification model using machine learning technology, compare it with historical data, and identify the evolution trend of the disease; the pavement disease dynamic identification module specifically includes: The latest disease target data, including disease type and location information and corresponding image data, is obtained from the target detection module and the disease target classification module. The disease data accumulated during previous inspections, including the disease development history and treatment records, is collected. The historical data is integrated with the real-time inspection data to form a complete time series data set. Extract features related to disease evolution from the integrated data, including shape, texture, color, and location of the disease. Then, integrate the extracted features related to disease evolution to obtain a feature dataset, which is divided into a training set and a test set. Use the training set data to train the time series analysis model, build a dynamic disease recognition model, and use the test set to test and evaluate the established dynamic disease recognition model, and then optimize the model; Input real-time inspection data into a trained dynamic disease recognition model, compare the newly detected disease features with those in historical data, analyze the similarities and differences between the features using similarity calculation methods, calculate the disease evolution trend index, and then analyze the evolution process of historical diseases to identify the evolution trend and severity of new diseases; Among them, the expression of disease evolution trend index is: ； ； Where, is the disease evolution trend index, is the newly detected disease feature vector, , is the historical disease feature vector, , is the feature weight, is the scaling parameter in similarity calculation, is the slope obtained from the similarity time series regression, To set the slope Scaling parameter mapped to the logistic function, is the number of historical data points, is the number of disease characteristics, For the Disease characteristics, is the index of the historical data point, For the Disease characteristics of historical data points, is the weighted Euclidean distance, The value range of is (0, 1), and higher values ​​indicate more serious evolution trends and severity; The inspection and early warning management module is used to analyze the inspection data in real time and warn of existing diseases; The environmental perception analysis module specifically includes: converting the image from RGB color space to Lab color space through a shadow occlusion processing unit, and using image segmentation and edge detection algorithms to identify the shadow area in the image, performing image restoration processing on the detected shadow area, and then performing lighting compensation on the shadow area to increase the brightness of the shadow area so that it is consistent with the lighting conditions of the surrounding environment.

2. The highway air-ground integrated intelligent inspection and patrol system according to claim 1 is characterized by: The data acquisition module specifically includes: Identify target areas for highway inspections and conduct demand planning for air and ground inspections to obtain various inspection data, including drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information; According to the inspection requirements of the highway and the scope of the target area, the flight route, flight altitude and shooting angle parameters of the drone are planned, and the drone automatically flies according to the preset flight route to obtain image data, and then wirelessly transmits the captured image data to the storage server of the data acquisition module; Various types of ground sensors are deployed along the highway, including meteorological sensors, temperature and humidity sensors, and vibration sensors. The ground sensors collect various types of data and transmit the data to the storage server of the data acquisition module through a wireless communication network; Install high-definition surveillance cameras at important sections, intersections, and service areas of the highway, and transmit the video stream data collected by the surveillance cameras to the storage server of the data acquisition module through the network; Environmental sensors including light sensors, rain sensors and snow sensors are set up in the inspection target area to monitor the light intensity, rainfall and snowfall environmental information during the inspection process. The environmental sensors transmit the collected data to the storage server of the data acquisition module through the communication network; Traffic flow monitoring equipment installed at key locations on the highway monitors vehicle flow and speed information, and transmits the data to the storage server of the data acquisition module. Road condition monitoring equipment is used to detect the flatness and damage of the road, and the relevant data is transmitted to the storage server of the data acquisition module. Pre-process the various inspection data acquired in the target area, including data cleaning and data format conversion, and encode the pre-processed data to assign a unique identifier to each type of data; The pre-processed drone image data, ground sensor data, surveillance video, environmental sensor data, and traffic flow and road condition information multi-source data are fused to form an inspection data set.

3. The highway air-ground integrated intelligent inspection and patrol system according to claim 2 is characterized by: The environmental perception analysis module includes a lighting and weather correction unit and a shadow occlusion processing unit; The illumination and weather correction unit is used to correct the effects of illumination changes and weather factors on the pre-processed image inspection data; The shadow occlusion processing unit is used to process the shadow occlusion area in the image inspection data to remove or reduce the influence of the shadow occlusion area.

4. The highway air-ground integrated intelligent inspection and patrol system according to claim 3 is characterized by: The environmental perception analysis module specifically includes: Extract the pre-processed inspection data from the data acquisition module, and analyze the pre-processed image inspection data through the illumination and weather correction unit and the shadow occlusion processing unit; The image is divided into multiple areas through the illumination and weather correction unit, the brightness value of each pixel in the image is calculated, the illumination intensity of each area is calculated separately, and the overall illumination intensity of the image is determined to identify the illumination differences between different areas in the image. The weather conditions are judged based on the color and texture characteristics of the image. The image is gamma corrected and the brightness and contrast of the image are adjusted according to the illumination intensity and weather factors. The edge information in the image is detected to further confirm the position and shape of the shadow area. The pixel values ​​of the shadow area are filled using the image restoration algorithm, and the shadow area is color corrected to adjust the color of the shadow area to match the surrounding environment.

5. The highway air-ground integrated intelligent inspection and patrol system according to claim 4 is characterized by: The target detection module specifically includes: The environmental perception and analysis module receives image inspection data that has been corrected for lighting and weather, as well as shadows, and extracts historical inspection data. The data is then labeled with the locations and categories of diseased and non-disease objects. The labeled historical inspection data is then integrated and divided into training and test sets. The training set data is combined with the convolutional neural network model, and the diseased and non-disease target data are used to train the target recognition model. The trained target recognition model is then evaluated using the test set data to evaluate the model performance. Based on the evaluation results, the model parameters and optimization strategy are further adjusted. The preprocessed real-time inspection data is input into the trained target recognition model. The target recognition model detects the targets in the image and generates bounding boxes and disease category labels. Then, based on the generated bounding boxes and disease category labels, the detected targets are classified and located, distinguishing between diseased targets and non-diseased targets.

6. The highway air-ground integrated intelligent inspection and patrol system according to claim 5 is characterized by: The process of classifying and locating the detected targets is as follows: The pre-processed real-time inspection image is input and passed through the multi-layer convolutional layers in the target recognition model. Each convolutional layer contains multiple convolution kernels, which slide across the image and perform convolution operations with local areas of the image. The convolution operation extracts local features of the image. A 3x3 convolution kernel slides across the image and performs dot product operations with each 3x3 area in the image to obtain a new feature map. The average pooling operation in the pooling layer after the convolution layer is used to reduce the size of the feature map and reduce the feature dimension. The target recognition model generates candidate regions through a region proposal network. The region proposal network generates multiple candidate boxes on the feature map. The candidate boxes cover different positions, sizes, and aspect ratios of the image. The generated candidate regions are then screened using a non-maximum suppression algorithm to remove redundant or unreasonable regions. The non-maximum suppression algorithm removes candidate regions with high overlap based on their confidence scores and retains the candidate regions with the highest confidence. The feature maps corresponding to the filtered candidate regions are fed into the fully connected layer to extract semantic features. The extracted semantic features are classified using the softmax function, and the classification results are output as probability vectors. The probability of each candidate region belonging to a diseased target or a non-diseased target is determined, and the bounding box coordinates of each candidate region are predicted using a regression algorithm. The difference between the predicted bounding box and the true bounding box is then calculated, and the mean square error loss is used to optimize the bounding box regression results. The coordinates of the bounding box are expressed as an offset relative to the candidate region, including the center point coordinates, width, and height. For the classification and bounding box regression results of multiple candidate regions, if multiple candidate regions correspond to the same object, they are merged into one object, and the non-maximum suppression algorithm is used to further filter and merge the classification results and bounding box regression results; The target recognition model outputs the location, category, and confidence of the target detected in the image. Each detection result includes a bounding box and a disease category label. The bounding box is used to determine the location of the target in the image, and the disease category label is used to distinguish diseased targets from non-diseased targets.

7. The highway air-ground integrated intelligent inspection and patrol system according to claim 6 is characterized by: The disease target classification module specifically includes: Based on the real diseased target output by the target detection module, feature analysis is performed to extract the shape, texture and color features of the real diseased target. Then, a weighted average feature fusion method is used to combine different types of features into a feature vector. According to the characteristics of the disease target and the classification requirements, the feature data set of the known disease target is used to train the support vector machine classification algorithm to obtain the disease classification model; The extracted and fused feature vectors are input into the trained disease classification model. The disease classification model calculates the probability of the true disease category of each target based on the feature vector and outputs the true disease category label of each target.

8. The highway air-ground integrated intelligent inspection and patrol system according to claim 7 is characterized by: The inspection warning management module includes an early warning prompt unit and an inspection disposal unit; The early warning prompt unit is used to provide early warning prompts for the identified real disease targets and issue early warning information, including the real disease target category, disease location, disease evolution trend and warning time; The inspection handling unit is used to plan, allocate, execute and track inspection tasks.

9. The highway air-ground integrated intelligent inspection and patrol system according to claim 8 is characterized by: The inspection and early warning management module specifically includes: The early warning prompt unit receives the inspection and analysis results output by the target detection module, the disease target classification module and the pavement disease dynamic identification module, including the type, location, severity and evolution trend of the disease; Based on the preset warning threshold, it is determined whether an early warning is needed. When the disease evolution trend index exceeds the preset warning threshold, the warning condition is triggered, and an early warning prompt containing detailed disease information is generated, including: disease target category, disease location, disease evolution trend and warning time. The early warning information is then released to relevant personnel through multiple channels such as SMS, email, and system notifications; The inspection and disposal unit plans inspection tasks based on early warning information and inspection plans. Taking into account the location, severity, and distribution of inspection personnel, it arranges inspection routes and assigns inspection tasks to specific inspection personnel. It also specifies the responsible persons and completion deadlines for the tasks. Inspection personnel, based on the assigned tasks, conduct on-site inspections and treatment at the locations of the diseases, and record the treatment status and results during the execution process. Track the execution of inspection tasks in real time, monitor the progress of tasks through the system, discover and solve problems that arise during the execution process, and after the inspection personnel complete the task, they will feedback the processing results to the system.

Images