Road disease detection method and system based on autonomous agent

Abstract

The invention relates to the technical field of data processing, in particular to a road disease detection method and system based on an autonomous agent. The detection method comprises the following steps: constructing and deploying an autonomous agent, controlling a ground penetrating radar unit GPR to obtain road internal disease data based on a large language model of the autonomous agent, and obtaining road apparent disease data through an image acquisition unit; the autonomous agent obtains and preprocesses road internal disease data and road apparent disease data, then calls a pre-trained semi-supervised detection algorithm for processing by using a plurality of sub-agents, and performs multi-agent decision fusion on decision results of the sub-agents through weighting to determine an internal disease detection result. And types and positions of the road internal diseases and the road apparent diseases are obtained. Through cooperative application of the main learning intelligent agent technology and road disease detection, the detection performance is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a road damage detection method and system based on an autonomous intelligent agent. Background Art

[0002] With the rapid growth of traffic volume, the increase in overweight vehicles, coupled with the influence of extreme weather conditions, the performance of highway pavement and roadbed is vulnerable to damage, such as cracks, looseness, collapse and voids. These defects will directly affect driving safety and increase the risk of traffic accidents. Therefore, timely detection and repair of road defects is one of the important means to ensure road safety and improve traffic efficiency.

[0003] Traditional road defect detection mainly relies on manual experience exploration. This method is not only inefficient and costly, but also easily interfered by human factors, making it difficult to meet the current needs of large-scale road detection. With the development of information technologies such as computer vision, some research on automated road defect detection has achieved certain results, and supervised detection algorithms based on computer vision have been widely used in road disease detection. However, such methods face challenges such as difficulty in data acquisition, insufficient sample size, and high data annotation costs, which limit the development and popularization of intelligent detection equipment. Especially in the detection of internal road diseases, although ground penetrating radar (GPR) technology has been applied to actual road surface detection, the lack of consistent training data makes it difficult for the detection model to achieve high-precision predictions in real detection scenarios. Summary of the invention

[0004] The embodiments of the present application provide a road damage detection method and system based on an autonomous intelligent agent, which realizes intelligent detection of multi-source data integration, can complete the comprehensive collection and detection of road damage, and has the ability to autonomously learn and update new samples. It can not only reduce the cost of data annotation, but also improve the accuracy of road damage detection, providing strong support for efficient highway maintenance management.

[0005] In a first aspect, an embodiment of the present application provides a road disease detection method based on an autonomous intelligent agent, comprising: The data acquisition step is to build an autonomous agent through the AutoStudio tool and deploy it in a computing unit of a central control module. The autonomous agent is used to receive detection instructions from a remote terminal and run according to a preset workflow. Based on the large language model of the autonomous agent, i.e., the LLM model, the ground penetrating radar unit GPR is controlled to acquire road internal disease data according to a planned path, and road surface disease data is acquired through an image acquisition unit. The road internal disease data includes B-scan images, C-scan images, and D-scan images, and the road surface disease data includes visible light images of surface diseases and infrared images of surface diseases. In the internal disease detection step, after the autonomous intelligent agent obtains and pre-processes the road internal disease data, it uses multiple sub-agents to call the pre-trained semi-supervised detection algorithm through the API port to process the B-scan image, C-scan image, and D-scan image, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the internal disease detection results, and obtain the type and location of the road internal disease; In the step of apparent disease detection, the autonomous intelligent agent obtains and pre-processes the road apparent disease data, and then uses two sub-agents to call the pre-trained semi-supervised detection algorithm through the API port to process the apparent disease visible light image and the apparent disease infrared image, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the apparent disease detection results, and obtain the type and location of the road apparent disease; Among them, the brain of the autonomous intelligent agent uses the LLM model to decompose the detection target based on the preset workflow, and calls ChatGPT4 for decision reasoning according to the workflow.

[0006] In some embodiments, the training process of the semi-supervised detection algorithm specifically includes: In the training data acquisition step, a cycle adaptor is used to obtain labeled images and unlabeled images corresponding to the image type to be processed in the initial data set in each training cycle; wherein, the initial data set is pre-labeled by using the labelimg tool to collect typical disease radar images, visible light images, and infrared images to construct an internal disease initial data set and an external disease initial data set.

[0007] Data preprocessing step: CutMix and Mosaic enhancement are performed on the labeled images, and CutMix, Mosaic and strong enhancement are performed on the unlabeled images. The strong enhancement includes random flipping, large-scale jittering, grayscale processing and Gaussian blurring to simulate various changes and disturbances that images may encounter in practical applications, so as to improve the robustness and generalization ability of the model. In the teacher model training step, unlabeled data is input into the dense detector of the teacher model to generate prediction results as pseudo labels. The pseudo labels are divided into reliable pseudo labels and uncertain pseudo labels based on a confidence threshold by a pseudo label distributor. The prediction results include bounding boxes and classification scores. ; In the student model collaboration step, the dense detector of the student model obtains the unlabeled image and its reliability pseudo-label and uncertain pseudo-label, performs training based on the reliable pseudo-label of the unlabeled image and the labeled image, performs loss calculation, performs training based on the uncertain pseudo-label of the unlabeled image, and after training, uses the EMA algorithm to update the parameters of the dense detector of the student model to the dense detector of the teacher model, and the uncertain pseudo-label is pre-processed by the soft label loss function; The model update iteration step repeats the teacher model training step and the student model collaboration step, and uses the student model that meets the iteration end condition as the detection model.

[0008] In some of the embodiments, the dense detector uses a deep residual network ResNet-50 as a backbone network, combined with a feature pyramid network FPN to extract features of different scales, and its output includes a classification branch, a regression branch, and a target branch to predict the target category of each anchor box in the image, the bounding box coordinate offset of the anchor box, and evaluate whether each anchor box contains the target.

[0009] In some embodiments, the pseudo-label assigner preconfigures two confidence thresholds: , , If the classification score , then the pseudo-label is a reliable pseudo-label and can be directly used for the training of the student model. The classification loss, regression loss and target loss are used for loss calculation. The classification loss is exemplified by but not limited to the cross entropy loss, the regression loss is exemplified by but not limited to the mean square error and the mean absolute error, and the target loss is exemplified by but not limited to the intersection-over-union loss; If the classification score , then discard the pseudo label; If the classification score , then the pseudo-label is an uncertain pseudo-label, and the target loss is used for loss calculation.

[0010] In a second aspect, an embodiment of the present application provides a road disease detection system based on an autonomous intelligent agent, including a vehicle body and a central control module and a perception module arranged on the vehicle body, wherein the central control module is used to control the movement of the vehicle body and provide data processing resources, and the perception module is used to perceive the external environment and obtain disease data; Walking equipment is installed on both sides of the vehicle body, which is used to drive the road disease detection and serve as a carrier for the integration of various components; the walking equipment is specifically a pair of rollers, which are driven to rotate by a motor; The central control module is arranged inside the vehicle body, connected to the perception module and connected to the remote terminal via a communication module, so as to realize data processing, control instruction generation and task management of the perception module; the communication module includes a WIFI module or an Ethernet module; The central control module includes a computing unit, a 4G module and an extended storage module, wherein the 4G module and the extended storage module are connected to the computing unit in communication, an autonomous agent is deployed on the computing unit and a semi-supervised detection algorithm is deployed based on Torch and Onnx frameworks, wherein the autonomous agent is used to control the vehicle body to move according to control instructions and implement the road disease detection method based on the autonomous agent as described in the first aspect above. The semi-supervised detection algorithm uses the TensorRT engine for inference acceleration.

[0011] In some embodiments, the perception module includes: The RTK positioning unit has a mobile station mounted on the vehicle body and wirelessly connected to the central control module; the base station of the RTK positioning unit is set at a fixed position, and the mobile station and the base station communicate via a cellular network to perform differential data transmission, and the output data is transmitted via the NMEA 0183 protocol, and the output data includes global positioning system fixed data GGA, ground speed information VTG, etc.

[0012] An image acquisition unit, including a visible light camera and an infrared camera, is fixed to the vehicle body through a camera bracket and is communicatively connected to the central control module. The image acquisition unit is used to acquire visible light images of road surface defects and infrared images of road surface defects and perform registration and fusion, thereby combining the two images to provide more comprehensive defect detection information for detection; A laser ranging radar is installed on the vehicle body and is communicatively connected to the central control module, and is used to generate 360-degree planar point cloud data to achieve environmental perception, so as to support obstacle avoidance, turning and construction of environmental maps during walking; The depth camera is located at the front of the vehicle and is connected to the central control module for capturing the 3D point cloud data and RGB images in front of the vehicle, detecting and identifying obstacles in front of the vehicle, so as to support real-time adjustment of the driving path to avoid obstacles and improve driving safety. The ground penetrating radar unit GPR is arranged on the vehicle body and is communicatively connected to the central control module to form an air coupling system with the road surface.

[0013] In some of the embodiments, the computing unit is pre-configured with an automatic inspection task, which includes: an inspection target point list, an inspection area boundary, a local inspection frequency, or a collection data type.

[0014] In some of the embodiments, the ground penetrating radar assembly GPR includes a main controller, a transmitting antenna, a transmitter, a receiving antenna and a receiver, and the set frequency range is 2.5 GHz to 10 GHz.

[0015] In some of the embodiments, a battery module is also disposed in the vehicle body, electrically connected to the motor and the sensing module to provide power therefor.

[0016] In some embodiments, an inertial measurement unit (IMU) and a GPS sensor are also provided in the vehicle body and are electrically connected to the central control module. The central control module uses an attitude fusion algorithm to fuse the acceleration and angular velocity of the inertial measurement unit (IMU) with the position information of the RTK positioning unit to obtain a stable attitude estimation for providing auxiliary positioning and navigation.

[0017] Compared with the related art, the embodiment of the present application constructs the autonomous intelligent body to conduct autonomous learning and detection of road surface diseases and internal diseases, mutually enhance and verify each other, improve detection performance, and synergistically apply autonomous learning intelligent agent technology and road disease detection, and realize technical improvements that integrate data collection, autonomous learning of disease characteristics and disease identification functions, thereby effectively addressing the problem of inconsistent road internal disease data, improving detection efficiency and accuracy, and reducing the intellectual cost and time cost of detection.

[0018] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a road damage detection method according to an embodiment of the present application; Figure 2 is a flowchart of a training process of a semi-supervised detection algorithm according to an embodiment of the present application; Figure 3 is a schematic diagram of the principle of the training process of the semi-supervised detection algorithm according to an embodiment of the present application; Figure 4 is a structural diagram of a road disease detection system according to an embodiment of the present application; Figure 5 It is a side structural diagram of a road damage detection system according to an embodiment of the present application.

[0020] In the figure: 1. Ground Penetrating Radar Unit GPR; 2. Walking Equipment; 3. Depth Camera; 4. Laser Range Finder Radar; 5. Camera Bracket; 6. RTK positioning unit; 7. Towing hole; 8. Battery module. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can be applied to other similar scenarios based on these drawings without any creative work.

[0023] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices.

[0024] This embodiment provides a road damage detection method based on an autonomous intelligent agent. Figure 1 is a flow chart of a road damage detection method according to an embodiment of the present application. Figure 1 As shown, the process includes the following steps: Data acquisition step S1, constructing an autonomous agent through the AutoStudio tool and deploying it in a computing unit of a central control module, the autonomous agent is used to receive detection instructions from a remote terminal and run according to a preset workflow, and based on the large language model of the autonomous agent, i.e., the LLM model, controls the ground penetrating radar unit GPR to obtain road internal disease data according to the planned path, and obtains road surface disease data through an image acquisition unit, the road internal disease data includes B-scan images, C-scan images, and D-scan images, and the road surface disease data includes visible light images of surface diseases and infrared images of surface diseases; In the internal disease detection step S2, after the autonomous intelligent agent obtains and pre-processes the road internal disease data, it uses multiple sub-agents to call the pre-trained semi-supervised detection algorithm through the API port to process the B-scan image, C-scan image, and D-scan image, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the internal disease detection results, and the type and location of the road internal disease are obtained, thereby improving the accuracy and reliability of the detection; In the apparent disease detection step S3, after the autonomous intelligent agent obtains and pre-processes the road apparent disease data, two sub-agents are used to call the pre-trained semi-supervised detection algorithm through the API port to process the apparent disease visible light image and the apparent disease infrared image, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the apparent disease detection results, and the type and location of the road apparent disease are obtained, thereby improving the accuracy and reliability of the detection; Among them, the brain of the autonomous intelligent agent uses the LLM model to decompose the detection target based on the preset workflow, and calls the large language model, such as ChatGPT4, according to the workflow to perform decision reasoning. The combination of the large language model and advanced decision reasoning algorithm enables the intelligent agent to handle complex problems more intelligently and improve the scientificity and rationality of decision-making.

[0025] The internal disease detection results and the apparent disease detection results are written into the knowledge base of the autonomous agent and put into training to optimize parameters to achieve autonomous learning, so that the agent can continuously accumulate experience and knowledge and improve detection performance and accuracy. At the same time, by continuously optimizing parameters, it can adapt to changes in different scenarios and needs.

[0026] In the above embodiments, the autonomous intelligent agent of the embodiments of the present application pre-selects and configures its brain, skills, roles, and workflows, and can meet the needs of different scenarios by configuring the brain, skills, roles, and workflows of the autonomous intelligent agent, thereby improving practicality and applicability. Among them, the brain of the autonomous intelligent agent selects the LLM model by limiting api_key and base_url, uses the LLM model to understand the workflow to decompose the detection target, and calls ChatGPT4 for decision-making and reasoning according to the workflow; the skill encapsulates the algorithm through the Flask tool and obtains the API interface, and realizes the detection function by calling the API interface. The algorithm includes semi-supervised detection algorithm and open source tools such as gapier, which give the autonomous intelligent agent the identity of ground penetrating radar signal analysis expert and apparent disease image analysis expert, and upload expert knowledge bases in the field of road internal disease radar map and apparent disease image analysis respectively to enrich the domain indication of the intelligent agent, and configure functional descriptions for it, such as "You are an expert in ground penetrating radar signal analysis, and you understand and are familiar with the interface definition and design of various types of Action APIs. Your goal is to automatically call relevant tools and functions according to the designed workflow to complete the detection and positioning of internal road diseases, and realize signal feedback and parameter adjustment", and the same is true for apparent disease image analysis; this enables the intelligent agent to understand and process image data more accurately, improving the accuracy and efficiency of detection.

[0027] Taking the road internal disease radar identification process as an example, the workflow of the autonomous agent is set to include the following steps: S101: Read the road section information and initial decision weight setting input by the user and decompose them, and call the write function to record the road section situation; among them, the B-scan image is suitable for identifying vertical anomalies, such as crack depth and vertical extension of large cavities, the C-scan image is suitable for identifying extensive planar defects and uniformity problems, such as uneven moisture distribution or weak road base, and the D-scan image is suitable for identifying overall structural continuity and long-distance changes. For road disease identification tasks with vertical anomalies as the main detection target, the weight of the B-scan image should be relatively high. For road damage identification tasks with plane defects and uniformity problems as the main detection targets, the weight of the C-scan image should be relatively high. Similarly, for road damage identification tasks with overall structural continuity and long-distance changes as the main detection targets, the weight of the D-scan image should be relatively high. When the workflow is used for apparent damage detection, initial decision weights can also be configured for apparent damage visible light images and apparent damage infrared images. The weights are updated during the working process to make them more in line with actual road conditions and detection requirements. Through iterative optimization, the weights are gradually balanced to achieve the best detection effect.

[0028] S102: Apache Kafka is used to receive real-time data streams to be processed, such as GPR data from ground penetrating radar units, including B-scan images, C-scan images, and D-scan images. IS functions, IFFT functions, and BRF functions are called to process the data streams to extract and analyze information in radar images. RabbitMQ middleware buffers data to ensure stable transmission and efficient processing of data streams.

[0029] S103: Separate the data stream according to the slice type, corresponding to the above example is B-scan image, C-scan image, D-scan image, and send the separated data stream to the corresponding sub-agent. Each sub-agent calls the detect function to detect the disease, and uses the pre-trained semi-supervised detection algorithm and expert knowledge base to analyze and judge to obtain the result.

[0030] S104: After reading the preset weights and calling the Result function to weight the detection results of each slice to perform multi-agent decision fusion to obtain the internal disease detection results, the Write function is called to write the detection results and location information into text.

[0031] The workflow of the autonomous agent in the above embodiment also includes: S105: The detection result is transmitted in reverse, and the Return function is called to write the internal disease detection result into the data set of the semi-supervised detection algorithm, and the weight of the semi-supervised detection algorithm is re-learned and updated.

[0032] When the workflow of the autonomous agent in the above embodiment is applied to the analysis of road surface damage images, the input of the autonomous agent and the semi-supervised detection algorithm is adjusted to the road surface damage data.

[0033] Figure 2 Flowchart of the training process of the semi-supervised detection algorithm. Figure 3 This is a schematic diagram of the training process principle of the semi-supervised detection algorithm. Figure 2-Figure 3 As shown in Figure 2, the training process of the semi-supervised detection algorithm specifically includes: In the training data acquisition step S201, the Epoch Adaptor is used to obtain the labeled images and unlabeled images corresponding to the image type to be processed in the initial data set in each training cycle; wherein, the initial data set is pre-labeled by the labelimg tool for the collected typical disease radar images, visible light images and infrared images, and the internal disease initial data set and the external disease initial data set are constructed. When processing the C-scan image detection task, the labeled C-scan images are obtained in the internal disease initial data set, and the other images are similarly not listed one by one. The Epoch Adaptor applies domain adaptation and distribution adaptation strategies on unlabeled images to create a more stable and efficient training environment for semi-supervised training.

[0034] Data preprocessing step S202, performing CutMix enhancement and Mosaic enhancement processing on the marked image, and performing CutMix enhancement, Mosaic enhancement and strong enhancement processing on the unmarked image, wherein the strong enhancement processing includes random flipping operation, large-scale jitter operation, grayscale processing operation, and Gaussian blur operation to simulate various changes and disturbances that the image may encounter in practical applications, so as to improve the robustness and generalization ability of the model; In the teacher model training step S203, the unlabeled data is input into the dense detector of the teacher model to generate prediction results as pseudo labels. The pseudo labels are divided into reliable pseudo labels and uncertain pseudo labels based on a confidence threshold by a pseudo label distributor. The prediction results include bounding boxes and classification scores. ; The student model cooperates with step S204, the dense detector of the student model obtains the unlabeled image and its reliability pseudo-label and uncertain pseudo-label, performs training based on the reliable pseudo-label of the unlabeled image and the labeled image, and performs loss calculation, and performs training based on the uncertain pseudo-label of the unlabeled image. After training, the EMA algorithm is used to update the parameters of the dense detector of the student model to the dense detector of the teacher model, and the uncertain pseudo-label is pre-processed by the soft label loss function; the soft label loss function allows the model to be more flexible in dealing with uncertainty, rather than simply treating it as a correct or incorrect classification. The soft label loss function usually considers the probability distribution of the teacher model output, rather than just focusing on the category with the highest probability. This helps the student model to be more robust when dealing with complex scenes and uncertainties.

[0035] The model updating iteration step S205 repeats the teacher model training step S203 and the student model collaboration step S204, and uses the student model that meets the iteration end condition as the detection model.

[0036] The above-mentioned embodiment utilizes the dense detector of the teacher model and the dense detector of the student model to realize the application of semi-supervised learning and self-training, wherein the teacher model is used to generate pseudo labels, and these pseudo labels are subsequently used to guide the training of the student model. The student model is trained under the guidance of the teacher model, and the teacher-student mutual learning architecture and pseudo-label generation technology are used to combine labeled and unlabeled data to improve the performance of the detection model.

[0037] In the above embodiment, CutMix enhancement generates new training samples by cutting and mixing two images during training, and introduces local occlusion and random background to improve the generalization ability of the model. The working principle is as follows: , in the formula , , It means randomly selecting the labels corresponding to two images A and B and the mixed image C. λ Indicates the proportion of the cut area to the new image; Mosaic enhancement stitches four images together at random positions and proportions to form a new training image, and maps the labels of each image (such as bounding boxes and categories) to the new stitched image to improve the performance of the detector at different scales and backgrounds.

[0038] In some of the embodiments, the dense detector uses a deep residual network ResNet-50 as a backbone network, combined with a feature pyramid network FPN to extract features of different scales, and its output includes a classification branch, a regression branch, and a target branch to predict the target category of each anchor box in the image, the bounding box coordinate offset of the anchor box, and evaluate whether each anchor box contains the target. At the same time, a dense sampling strategy is adopted in the input and output stages for higher resolution and denser processing to improve the detection density and detail processing capabilities of the model.

[0039] In some embodiments, the pseudo-label assigner preconfigures two confidence thresholds: , , confidence threshold The specific value of can be flexibly adjusted to optimize the pseudo-label usage strategy.

[0040] If the classification score , then the pseudo-label is a reliable pseudo-label and can be directly used for the training of the student model. The classification loss, regression loss and target loss are used for loss calculation. The classification loss is exemplified by but not limited to the cross entropy loss, the regression loss is exemplified by but not limited to the mean square error and the mean absolute error, and the target loss is exemplified by but not limited to the intersection-over-union loss; If the classification score , then discard the pseudo label; If the classification score , then the pseudo-label is an uncertain pseudo-label, and the target loss is used for loss calculation, such as Figure 3 As shown, represents the classification loss of reliable pseudo labels, represents the regression loss of reliable pseudo-labels, represents the targetness loss of pseudo labels.

[0041] also, Figure 3 middle, represents the classification loss of the labeled image, represents the regression loss of labeling avatars, represents the target loss of the labeled image, represents the domain adaptation loss.

[0042] In order to reduce the difference in feature distribution between labeled and unlabeled data, Epoch Adaptor also introduces domain adaptation loss , ,in, It is the output of the domain classifier, D represents the data type, D=0 is used to represent labeled data, and D=1 is used to represent unlabeled data.

[0043] Based on the above embodiments, the present application constructs the autonomous intelligent body to conduct autonomous learning and detection of road surface diseases and internal diseases, enhance each other, verify each other, improve detection performance, and synergize the application of autonomous learning intelligent agent technology and road disease detection, and realize technical improvements that integrate data collection, autonomous learning of disease characteristics and disease identification functions, thereby effectively addressing the problem of inconsistent road internal disease data, improving detection efficiency and accuracy, and reducing the intellectual cost and time cost of detection.

[0044] In other embodiments, considering that the data acquired by the ground penetrating radar contains information such as the location and nature of internal road defects, as well as a large amount of noise interference, such as environmental noise, multipath effects, surface reflections, system noise, etc., in order to extract effective underground information, filtering algorithms are used to remove and gain adjustment is performed to optimize the resolution and accuracy of imaging.

[0045] For the road internal disease data, Interference Suppression is first used, and the power limit threshold is set to 10 dB to ensure that the signal components exceeding this power level will be considered as interference and suppressed, reducing the interference of non-target signals and making the target signal clearer.

[0046] Furthermore, IFFT (Inverse Fast Fourier Transform) is used to convert the frequency domain signal back to the time domain signal, and the Tukey window function is used to smooth the frequency domain signal, thereby effectively reducing spectrum leakage. The gradient parameter of the Tukey window is set to 0.6, and the bandwidth of the frequency domain signal is limited. Only signals with specific frequencies of 30 MHz-3030 MHz are processed. In this way, the radar signal processed reduces the edge effect of frequency filtering on the time domain signal and improves the time-frequency resolution of the signal.

[0047] Then, a background removal filter is used to calculate the background data within 10 meters using the Sliding Window Mean and remove 100% of the estimated background signal, effectively removing the strong background signal caused by surface reflection or large-scale underground structure to avoid masking smaller internal disease features.

[0048] Finally, the gain adjustment is performed to amplify the signal by 6 dB, enhancing the amplitude of the signal and making the weaker signal components more obvious. After this processing, the radar signal of the road internal disease is more obvious and easier to detect.

[0049] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. For example, steps S2 and S3 can be executed sequentially.

[0050] This embodiment also provides a road damage detection system based on an autonomous intelligent agent. Figure 4-Figure 5 is a structural diagram of a road disease detection system based on an autonomous agent according to an embodiment of the present application. Figure 4-Figure 5 As shown, the system includes: a vehicle body and a central control module and a perception module arranged on the vehicle body, wherein the central control module is used to control the movement of the vehicle body and provide data processing resources, and the perception module is used to perceive the external environment and obtain disease data; Traveling devices 2 are installed on both sides of the vehicle body, which are used to drive the road disease detection and serve as a carrier for the integration of various components; the vehicle body is made of hard aluminum alloy, and the traveling device 2 is specifically a pair of rollers, which are driven by a motor to rotate. Preferably, the rollers are Mecanum wheels to achieve omnidirectional movement and support the efficient detection capability of the detection vehicle body in complex road environments; Figure 5 As shown, a towing hole 7 is also provided at the rear of the vehicle body for connecting a towing vehicle via a rope so that the vehicle body can be used for conventional ground penetrating radar, supporting the vehicle body to be used in two modes: towing and automatic inspection.

[0051] The central control module is arranged inside the vehicle body, connected to the perception module and connected to the remote terminal via a communication module, so as to realize data processing, control instruction generation and task management of the perception module; the communication module includes a WIFI module or an Ethernet module; Among them, the central control module includes a computing unit, a 4G module and an extended storage module. The 4G module and the extended storage module are communicatively connected to the computing unit. An autonomous agent is deployed on the computing unit and a semi-supervised detection algorithm is deployed based on the Torch and Onnx frameworks. The autonomous agent is used to control the movement of the vehicle according to control instructions and implement the road disease detection method based on the autonomous agent as described in the above embodiment. Among them, the semi-supervised detection algorithm uses the TensorRT engine for reasoning acceleration. Optionally, the computing unit adopts the NVIDIA Jetson AGX Orin 64GB high-performance edge AI computing platform, equipped with a 2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores and a 12-core Arm Cortex-A78AEv8.2 64-bit CPU, which can provide up to 275TOPS of Al performance and process high-resolution data from multiple sensors at the same time. The power can be flexibly adjusted between 15 and 60 watts. The Jetson AGX Orin 64GB core board is responsible for receiving and processing the data from the sensors of the perception module. It has built-in 64GB LPDDR5 memory and 64GB eMMC storage device for storing and running multiple programs. The computing unit is connected to the remote terminal through a WIFI module or an Ethernet module to realize remote control and operation of the computing unit by the remote terminal. The 4G module adopts Air724UG The 4G communication module provides network support for the computing unit, and the WIFI module uses the TL8188FCB-IPX module whose main controller is RTL8188FTV-VC.

[0052] Combine the following Figure 4-Figure 5 The various components of the road damage detection system based on autonomous intelligent agents are introduced in detail: The perception module includes: an RTK positioning unit 6, an image acquisition unit, a laser ranging radar 4, a depth camera 3 and a ground penetrating radar unit GPR1.

[0053] The RTK positioning unit 6 has a mobile station mounted on the vehicle body and wirelessly connected to the central control module; the base station of the RTK positioning unit 6 is set at a fixed position, and the mobile station and the base station communicate via a cellular network for differential data transmission, and the output data is transmitted via the NMEA 0183 protocol, and the output data includes the fixed data GGA of the global positioning system, the ground speed information VTG, etc. The computing unit reads the data through the serial port, and uses the nmea_navsat_driver node of the ROS middleware to parse and convert it into the nav_msgs / Odometry message type in ROS. Optionally, the RTK positioning module uses the Trimble R10 high-precision GNSS receiver, which supports multiple positioning systems such as GPS, GLONASS, Galileo, BeiDou, etc., to achieve centimeter-level precision positioning in complex environments. When the vehicle body is in the state of traction and loading, the RTK positioning unit 6 is used for positioning feedback, and the monitoring results are monitored through the remote terminal.

[0054] An image acquisition unit, including a visible light camera and an infrared camera, is fixed to the vehicle body through a camera bracket 5 and is communicatively connected to the central control module. The image acquisition unit is used to acquire visible light images of road surface diseases and infrared images of road surface diseases and to perform registration and fusion, thereby combining the two images to provide more comprehensive disease detection information for detection; optionally, the visible light camera uses GoPro HERO5 BLACK to acquire visible light images of road surface diseases with a resolution of 4000 x 3000, and the infrared camera uses FLIR A65 infrared camera to acquire infrared images of road surface diseases with a spectral range of 7.5-13.5 µm and a resolution of 640x 512; The laser ranging radar 4 is arranged on the vehicle body and is communicatively connected to the central control module, and is used to realize 360-degree all-round scanning and ranging within a range of 12 meters at a measurement frequency of 8000 times / second to generate 360-degree plane point cloud data, realize environmental perception, and support obstacle avoidance, turning and construction of environmental maps during walking; optionally, the laser ranging radar 4 model of the present application adopts Silan RPLIDAR A1.

[0055] The depth camera 3 is arranged at the front side of the vehicle body and is communicatively connected to the central control module, and is used to capture the front three-dimensional point cloud data and RGB images, detect and identify the obstacles in front, so as to support real-time adjustment of the driving path for obstacle avoidance and improve walking safety. Optionally, the depth camera 3 model of the present application adopts Intel's D435 series, which has a depth image resolution of 1280×720 and an RGB image resolution of 1920×1080, supports large-scale high-quality depth data capture and full HD video acquisition, and helps the detection equipment to perceive the depth information of the front environment in real time; The ground penetrating radar unit GPR1 is arranged on the vehicle body and is communicatively connected to the central control module, so as to form an air coupling system with the road surface, realize non-destructive detection of the inside of the road through the reflection difference of electromagnetic waves in different media, integrate the high-speed ground penetrating radar into the protective shell, and have a built-in antenna electronic unit with pulse width and antenna matching to realize integrated design; the ground penetrating radar unit GPR1 is equipped with an interchangeable 12V lithium-ion battery, a 12V DC adapter and a 1.6GHz atomic processor, the lithium-ion battery supports at least 8 hours of continuous operation, and the DC adapter supports hot plugging; the 160MHz and 670MHz dual-frequency antennas of the ground penetrating radar unit GPR1 can be used for fast synchronous high-resolution data acquisition of shallow and deep layers, and multiple antenna setting parameter files can be stored according to the detection conditions and antenna configuration, and the number is not limited.

[0056] In some of the embodiments, the computing unit is pre-configured with an automatic inspection task, which includes: an inspection target point list, an inspection area boundary, a local inspection frequency, or a collection data type.

[0057] In the above embodiment, the walking robot system is developed based on the Linux operating system and the ROS (Robot Operating System) architecture, and the decision-making control of the autonomous intelligent body is used to automatically inspect, and the next target point is dynamically selected according to the inspection target point list, and the dynamic window method DWA is used to calculate the driving area in real time to generate the inspection path according to the current speed, target speed and obstacle position, and the Dijkstra global path planning algorithm is further used to calculate the optimal path from the current position to the target position on the constructed map. During the inspection process, the Teb Local Planner local path planning tool is used to dynamically adjust the motion path of the vehicle body according to the obstacles detected by the sensor to ensure safe driving; the task execution order is also adjusted according to the data changes of the sensor, and the message_filters time synchronization tool in ROS can ensure the synchronization of the timestamps of all sensor data.

[0058] In the above embodiment, the planar point cloud data and the three-dimensional point cloud data are processed by the pointcloud_to_laserscan and pointcloud tools in ROS to generate fused point cloud data for building an inspection map and real-time positioning based on the SLAM algorithm, preferably using the RTAB-Map (Real-Time Appearance-Based Mapping) SLAM algorithm.

[0059] In another embodiment, the central control module of the embodiment of the present application monitors the status of each node in real time, including sensor status and path execution status, and triggers corresponding emergency responses when abnormalities are found, such as re-planning the path, stopping the equipment, etc., and returning to the charging station when the battery is low. The task status is updated and shared through the ROS service and parameter server.

[0060] In some embodiments, the ground penetrating radar unit GPR1 includes a main controller, a transmitting antenna, a transmitter, a receiving antenna and a receiver, and the frequency range is set to 2.5 GHz to 10 GHz. The main controller is used to receive the start-up instruction of the computing unit and control the transmitter to output a high-frequency oscillating current to the transmitting antenna, and then the transmitting antenna converts the received high-frequency current into electromagnetic pulses and sends them out. These pulses will propagate downward and enter the underground. When the electromagnetic pulse encounters the interface of different underground materials (such as the boundary between soil and rock, or buried objects), due to the different electromagnetic properties of these materials (such as different dielectric constants and conductivity), part of the electromagnetic waves will be reflected at the interface, and part will continue to propagate. The reflected electromagnetic pulse signal is received again by the receiving antenna and then converted into an electrical signal for the receiver to process. The time delay and intensity of the received signal can reflect the depth and characteristics of the underground target, so that single-point sampling is completed and a radar echo A-scan image reflecting the time domain information inside the current road is obtained.

[0061] Furthermore, by moving the antenna multiple times in a grid pattern within an area and recording data, the reflected signals of each detection point within an area can be collected, and the data of multiple detection points can be integrated to create a three-dimensional image of the underground target. The three-dimensional image of the underground target includes a B-scan image, a C-scan image, and a D-scan image, which are used to characterize the internal disease conditions of the road, thereby providing more comprehensive and intuitive underground information and being able to display complex underground structures. Among them: B-scan is a vertical cross-sectional image along the radar scanning path, showing the change of the reflected signal between the antenna and the underground target when the ground penetrating radar moves along a linear track, that is, it shows the vertical reflection under the radar path, with the radar's moving direction (such as distance) on the horizontal axis and time or depth on the vertical axis; C-scan is a horizontal profile image at a specific depth in the direction of radar travel, showing the distribution of reflected signal intensity of the ground penetrating radar within a certain depth range. It is similar to a horizontal slice and is composed of multiple B-scans, which are accumulated and constructed at different plane positions. D-scan is a vertical profile image at a certain depth, showing the signal changes at different depths along the perpendicular to the radar scanning path. The shape fluctuations of internal road defects under these scanning methods will show abnormalities. Combining these scanning methods can provide a comprehensive understanding of the characteristics of internal road defects.

[0062] In some embodiments, a battery module 8 is also provided in the vehicle body, electrically connected to the motor and the sensing module to supply power to them. Optionally, the battery module adopts a 7.4V 2200Ah high-rate lithium battery; Figure 4 As shown, an interface protection cover is provided at the battery module.

[0063] In some embodiments, an inertial measurement unit IMU and a GPS sensor are also provided in the vehicle body and are electrically connected to the central control module. The central control module uses an attitude fusion algorithm to fuse the acceleration and angular velocity of the inertial measurement unit IMU with the position information of the RTK positioning unit 6 to obtain a stable attitude estimation for providing auxiliary positioning and navigation.

[0064] This application integrates multiple sensing technologies (such as GPR, infrared cameras, depth cameras, etc.) and combines autonomous learning capabilities to efficiently identify and classify road defects, thereby improving the accuracy and efficiency of detection. In addition, the system can learn new disease characteristics during the detection process without the need for additional complex labeling work, and continuously optimize the detection algorithm to adapt to changing road conditions. Compared with traditional manual detection methods, this intelligent detection equipment can significantly reduce labor costs and time costs. The automated operation of the equipment reduces dependence on professionals, only requires the input of simple instructions, simplifies the operation process, and makes road maintenance more economical and efficient. At the same time, the design of the present invention takes into account road conditions of different types and levels. The weights can be initialized according to actual conditions during the instruction input stage, and it has good adaptability. The open architecture of the system also supports the further integration and upgrading of future technologies, enhancing the long-term use value of the equipment.

[0065] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A road damage detection method based on autonomous intelligent agent, It is characterized in that include: The data acquisition step includes building and deploying an autonomous intelligent agent, controlling a ground penetrating radar unit GPR based on a large language model of the autonomous intelligent agent to acquire road internal disease data, and acquiring road surface disease data through an image acquisition unit, wherein the road internal disease data includes B-scan images, C-scan images, and D-scan images, and the road surface disease data includes visible light images of surface diseases and infrared images of surface diseases; In the internal disease detection step, after the autonomous intelligent agent obtains and pre-processes the road internal disease data, it uses multiple sub-agents to call the pre-trained semi-supervised detection algorithm to process the B-scan image, C-scan image, and D-scan image, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the internal disease detection results, and obtain the type and location of the road internal disease; In the apparent disease detection step, after the autonomous intelligent agent obtains and pre-processes the road apparent disease data, two sub-agents are used to call the pre-trained semi-supervised detection algorithm to process the apparent disease visible light image and the apparent disease infrared image respectively, and the decision results of the sub-agents are weighted to perform multi-agent decision fusion to determine the apparent disease detection results, and obtain the type and location of the road apparent disease; Among them, the brain of the autonomous intelligent agent uses a large language model to decompose the detection target based on the preset workflow, and performs decision reasoning according to the workflow.

2. The road damage detection method based on autonomous agent according to claim 1, It is characterized in that The training process of the semi-supervised detection algorithm specifically includes: A training data acquisition step, using a cycle adaptor to acquire labeled images and unlabeled images corresponding to the image type to be processed in the initial data set in each training cycle; Data preprocessing step: CutMix enhancement and Mosaic enhancement are performed on the labeled images, and CutMix enhancement, Mosaic enhancement and strong enhancement are performed on the unlabeled images; In the teacher model training step, unlabeled data is input into the dense detector of the teacher model to generate prediction results as pseudo labels. The pseudo labels are divided into reliable pseudo labels and uncertain pseudo labels based on a confidence threshold by a pseudo label distributor. The prediction results include bounding boxes and classification scores. ; In the student model collaboration step, the dense detector of the student model obtains the unlabeled image and its reliability pseudo-label and uncertain pseudo-label, performs training based on the reliable pseudo-label of the unlabeled image and the labeled image, performs loss calculation, performs training based on the uncertain pseudo-label of the unlabeled image, and after training, uses the EMA algorithm to update the parameters of the dense detector of the student model to the dense detector of the teacher model, and the uncertain pseudo-label is pre-processed by the soft label loss function; The model update iteration step repeats the teacher model training step and the student model collaboration step, and uses the student model that meets the iteration end condition as the detection model.

3. The road damage detection method based on autonomous agent according to claim 2, It is characterized in that The dense detector adopts the deep residual network ResNet-50 as the backbone network, combined with the feature pyramid network FPN to extract features of different scales. Its output includes classification branch, regression branch and target branch to predict the target category of each anchor box in the image, the bounding box coordinate offset of the anchor box and evaluate whether each anchor box contains the target.

4. The road damage detection method based on autonomous agent according to claim 2, It is characterized in that The pseudo-label assigner preconfigures two confidence thresholds , , If the classification score , then the pseudo-label is a reliable pseudo-label, and the loss is calculated using classification loss, regression loss and target loss; If the classification score , then discard the pseudo label; If the classification score , then the pseudo-label is an uncertain pseudo-label, and the target loss is used for loss calculation.

5. A road disease detection system based on an autonomous intelligent agent, comprising a vehicle body and a central control module and a perception module arranged on the vehicle body. It is characterized in that Walking equipment is installed on both sides of the vehicle body; The central control module is arranged inside the vehicle body, connected to the sensing module and connected to the remote terminal via a communication module; Among them, the central control module includes a computing unit, a 4G module and an extended storage module, the 4G module and the extended storage module are communicatively connected to the computing unit, an autonomous agent and a semi-supervised detection algorithm are deployed on the computing unit, and the autonomous agent is used to control the movement of the vehicle according to control instructions and implement the road disease detection method based on the autonomous agent as described in any one of claims 1 to 4.

6. The road damage detection system based on autonomous agent according to claim 5, It is characterized in that The perception module comprises: An RTK positioning unit, whose rover is mounted on the vehicle body and is wirelessly connected to the central control module; An image acquisition unit, including a visible light camera and an infrared camera, is fixed to the vehicle body through a camera bracket and is communicatively connected to the central control module. The image acquisition unit is used to acquire visible light images of road surface defects and infrared images of road surface defects and perform registration and fusion; A laser ranging radar is arranged on the vehicle body and is communicatively connected to the central control module to generate 360-degree plane point cloud data; A depth camera is disposed on the front side of the vehicle body and is communicatively connected to the central control module, and is used to capture the front three-dimensional point cloud data and RGB images; The ground penetrating radar unit GPR is arranged on the vehicle body and is communicatively connected to the central control module.

7. The road damage detection system based on autonomous agent according to claim 6, It is characterized in that The computing unit is pre-configured with an automatic inspection task, which includes: an inspection target point list, an inspection area boundary, a local inspection frequency or a collection data type.

8. The road damage detection system based on autonomous agent according to claim 6, It is characterized in that The ground penetrating radar unit GPR includes a main controller, a transmitting antenna, a transmitter, a receiving antenna and a receiver, and the set frequency range is 2.5 GHz to 10 GHz.

9. The road damage detection system based on autonomous agent according to claim 5 or 6, It is characterized in that A battery module is also disposed in the vehicle body and electrically connected to the sensing module to supply power thereto.

10. The road damage detection system based on autonomous agent according to claim 5 or 6, It is characterized in that The vehicle body is also provided with an inertial measurement unit IMU and a GPS sensor, which are electrically connected to the central control module.

Images