A road defect intelligent detection system and method based on mobile collection and cloud deployment

Abstract

This invention provides a road defect intelligent detection system and method based on mobile data acquisition and cloud deployment, addressing the problems of low efficiency, incomplete detection, and high manpower and material costs associated with traditional road defect detection. The system includes: a mobile data acquisition vehicle equipped with a high-definition camera to collect and push road surface video streams in real time; a deep learning recognition end that uses a locally deployed YOLO11 target detection model to pull video streams for real-time inference and accurately label road defect categories; and a defect detection platform relying on a cloud server platform to achieve real-time live streaming and data storage of defect detection images, as well as real-time positioning of the mobile acquisition end. This invention adopts a combined local inference and cloud deployment model, balancing detection efficiency, accuracy, and real-time performance. It utilizes a miniaturized intelligent vehicle as the data acquisition carrier to reduce detection costs and improve deployment flexibility, achieving intelligent, real-time, and traceable road defect detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, Internet of Things and embedded technology, and specifically to a road defect intelligent detection system and method based on mobile acquisition and cloud deployment. Background Technology

[0002] Long-term vehicle traffic and increased loads can easily lead to road surface defects such as cracks, potholes, bumps, and ruts, affecting driving stability and posing traffic safety hazards. Currently, road defect detection mainly relies on two methods: traditional manual inspection and professional inspection vehicle inspection. Traditional manual inspection is inefficient, incomplete, and resource-intensive, while professional inspection vehicles are expensive and have complex operating procedures, making them difficult to widely apply in ordinary highways and rural roads, and thus have poor applicability.

[0003] With the rapid development of deep learning, IoT, embedded systems, and sensor technologies, the field of road defect detection is gradually evolving towards intelligence and automation. However, existing detection solutions still have significant shortcomings. Most solutions focus only on optimizing pure software algorithms, failing to achieve deep coupling between hardware acquisition devices, algorithm inference, and cloud-based business platforms, thus failing to form a complete technical closed loop of acquisition-inference-display-storage-location. Some solutions with hardware acquisition capabilities lack efficient mobile acquisition carrier design and fail to achieve associated storage of detection data and location information, making it difficult to meet the actual needs of road defect location, remote monitoring, and subsequent maintenance traceability. At the same time, existing solutions mostly adopt local data processing modes, failing to achieve remote real-time viewing, live display, and multi-terminal collaborative management of detection results, thereby restricting the improvement of detection efficiency and management convenience.

[0004] Therefore, in response to the problems of high cost, poor accessibility, low system coupling and remote management capability in existing road defect detection solutions, it is urgent to solve the technical challenge of constructing a road defect detection system that uses a mobile acquisition carrier to acquire real-time road surface data, combines intelligent algorithm inference, and realizes real-time live display of defects, data storage and location information association management through a cloud platform. This also provides a new development path for intelligent road defect detection. Summary of the Invention

[0005] The purpose of this invention is to propose a road defect intelligent detection system and method based on mobile acquisition and cloud deployment. Specifically, it uses an intelligent vehicle to collect road surface image data in real time and uploads it to a deep learning recognition terminal for detection and recognition. Relying on a cloud business platform, it realizes inspection management, live defect display, data storage and location information association storage with defect categories, map display, etc., so as to achieve efficient and accurate detection and location of road defects.

[0006] To achieve the above objectives, this invention provides a road defect intelligent detection system based on mobile data acquisition and cloud deployment. The system design is as follows: The mobile data acquisition vehicle, as the core component for road data collection, includes an embedded main control board, an image acquisition device, an obstacle avoidance module, a positioning and navigation module, a drive module, a wireless communication module, and a power supply module; each module is connected to the embedded main control board. The embedded main control board is responsible for overall data processing, command scheduling, and communication management, ensuring the coordinated operation of all modules within the vehicle. The image acquisition device is responsible for capturing real-time road images. The obstacle avoidance module enables the vehicle to automatically avoid obstacles. The positioning and navigation module can achieve real-time location positioning of the acquisition terminal in open outdoor spaces and most indoor environments. The drive module controls the vehicle's motion. The wireless communication module enables interaction between the acquisition terminal and other devices. The power supply module adopts a dual-power independent power supply scheme.

[0007] The deep learning recognition terminal is used to detect and identify road surface defects. The hardware carrier device has inference computing and communication functions. It realizes the relay between the acquisition terminal and the defect detection platform, and undertakes tasks such as pulling camera video streams, identifying and annotating road defects in video frames, and pushing the annotated stream to the platform in real time.

[0008] The defect detection platform, deployed on a cloud server, is the core of the entire system's management and interaction. It includes a task management module, a data receiving and storage module, a streaming media service module, a real-time map module, and a user interaction module. This platform is mainly used to receive task instructions from users and send them to the deep learning recognition end. At the same time, it stores video streams and receives defect frame images and related information, and provides users with services such as low-latency live video streaming and real-time map display.

[0009] Preferably, the image acquisition device is a Module 3 high-definition camera, which is placed on an adjustable bracket at the front of the vehicle and can be flexibly adjusted in height to facilitate adjustment of the image angle according to actual shooting needs.

[0010] Preferably, the obstacle avoidance module includes an ultrasonic sensor and an infrared sensor, which are installed at the front and rear of the vehicle, respectively, to detect the distance between the vehicle and obstacles in front and behind. When the distance in front is less than the safe distance, an emergency reverse is triggered. When the distance behind is less than the safe distance during reverse, the vehicle stops all movement until it receives the next start command.

[0011] Preferably, the embedded main control board is a Raspberry Pi 5; the positioning and navigation module includes GPS and IMU, and uses information fusion correction to output a smooth, drift-free navigation solution; the drive module receives control signals from the Raspberry Pi 5 and can accurately control the speed and direction of the motor, and the user can remotely control the car through the control page on the mobile phone.

[0012] To support the implementation of the aforementioned detection system, this invention also provides a road defect intelligent detection method based on mobile data acquisition and cloud deployment, comprising the following steps: S1. The mobile data acquisition vehicle's camera automatically starts upon power-on, pushing real-time video streams. After powering on, the mobile data acquisition vehicle moves along the road under user control, automatically avoiding obstacles if detected. Users access the defect detection platform through a client browser, add a new inspection task, and select whether to record. Upon receiving the request, the platform backend generates a unique task identifier for the task. After the task starts, the platform backend calls the control interface of the deep learning recognition terminal to issue task execution instructions.

[0013] S2. After receiving the instruction, the deep learning recognition terminal pulls the camera video stream and loads the inference model to perform frame-by-frame inference. When the recognition terminal detects a defective frame, it marks the defective frame and uploads the specific defective frame, its location information, and the defect type and quantity to the platform.

[0014] S3, the deep learning recognition end, pushes the video stream with superimposed labeled boxes to the platform's backend service in real time. Users can then open the defect detection platform's live stream to watch the labeled detection video in real time.

[0015] S4. When the user selects to record in the task instructions, the deep learning recognition terminal will call the recording interface and trigger the recording operation when pushing the video stream. After the video stream is pushed out, the platform will trigger a stop recording operation and complete the video data archiving.

[0016] S5. During the inspection process, users can monitor the current location of the target vehicle through the platform's real-time map function. After the vehicle completes the inspection of the designated section, the user shuts it down, and the camera stops working. The deep learning recognition end stops pulling the camera video stream and sends a "task completed" status message to the platform, which then updates the task status. Users can subsequently track the inspection information through the platform.

[0017] Preferably, in step S1, the embedded main control board pushes the video stream captured by the camera. Specifically, the embedded main control board acquires the real-time image from the camera, then transmits the original image to FFmpeg for encoding and encapsulates it into a streaming media format. The video stream is then pushed to the streaming server via a real-time streaming protocol. The remote streaming device is connected to the local area network where the streaming server is located via a virtual private network.

[0018] In step S1, the platform issues a task execution instruction. Specifically, it sends a data transmission request to the deep learning recognition terminal. The request field includes a task identifier and a recording flag. The task identifier is a unique identifier that is generated. When the recording flag is set to 1, it indicates that recording has started. When it is set to 0, it indicates that recording has not started.

[0019] As a preferred option, step S2 specifically involves: S2-1, Task-Driven. After the deep learning recognition service starts, it provides a preset network interface. Upon receiving a network request from the defect detection platform, it starts a thread to run the core processing logic.

[0020] S2-2. Pulling Stream and Reconnection. The pull stream address is the streaming media address. The deep learning recognition end pulls the video stream from the push stream server via the virtual private network. When reading frame data from the video stream fails, resources will be automatically released and a reconnection attempt will be made until the connection is successful or the task termination flag is set.

[0021] S2-3, Model Inference. The recognition end calls the optimal weight model file obtained from training and performs inference frame by frame on the pulled video stream. Inference is accelerated using a graphics processing unit (GPU) and parameters such as image size and confidence threshold are preset.

[0022] S2-4, Defect Labeling. When a road defect is detected in a frame, the recognition device directly draws the detection bounding box and category label on the original video frame, generating an overlay video stream.

[0023] S2-5. Uploading Defect Frames and Related Information. When a defect frame is detected, it is cached within a preset time window, and the defect category and quantity are recorded. For each frame within the window, a quality score is calculated by combining sharpness, average detection confidence, and defect location centrality. Similarity is compared using the correlation of inter-frame color histograms, and the frame with the higher quality score among similar frames is retained. After the preset time period is reached, the frame with the highest quality score is selected from the window. The platform is logged in via network request, and the labeled defect frame is uploaded to the platform. Simultaneously, a request is sent to the mobile acquisition vehicle control terminal to obtain the real-time latitude and longitude data of the current vehicle. The upload task of the defect frame, its defect category and quantity, and the current integrated navigation positioning coordinates is executed, and then the time window is cleared.

[0024] S2-6. Image Management. The platform records image association information in the database and displays all defective images and their related information in a list format on the front-end page.

[0025] As a preferred option, step S3 specifically involves: the deep learning recognition end writing the superimposed video stream into FFmpeg and pushing it to the cloud streaming media server; the platform front-end pulling the live stream from the streaming media server to achieve real-time low-latency playback.

[0026] As a preferred option, step S4 specifically involves: S4-1. When the recording flag is set to 1 in the task instruction issued by the platform, the deep learning recognition terminal triggers recording by sending a network request to the streaming media server. In the specific implementation, a retry mechanism is set up to prevent interface call failure due to network jitter.

[0027] S4-2. After recording is triggered, the streaming media server begins writing the received video data to a local file. After the video stream ends, the platform sends a network request to the streaming media server to stop recording. Upon receiving the instruction, the streaming media server encapsulates the buffered data into a complete MP4 file.

[0028] Preferably, in step S5, the real-time map function is implemented by passing in the Key and security key to complete the map initialization operation. During the inspection, the platform sends a network request to the mobile data collection vehicle to obtain the combined navigation and positioning information of GPS and IMU. The front end uses the latitude and longitude as the center coordinates of the point markers and overlays them on the electronic map layer to complete the rendering of the point markers.

[0029] Compared with the prior art, the beneficial effects of the above technical solution adopted by the present invention are as follows: By constructing a complete technical closed loop of mobile data acquisition, local detection, and cloud management, the system combines mobile data acquisition vehicles, deep learning recognition terminals, and cloud-based defect detection platforms to address the technical pain points of low coordination and hardware-software disconnect in existing detection solutions. This enables integrated operation of road data acquisition, defect detection, real-time live display, real-time inspection map display, task management, video and image data storage, and metadata association management, significantly improving system coordination and operational efficiency.

[0030] This system achieves low-cost, intelligent, real-time, and traceable road defect detection. It utilizes a low-cost embedded main control board and ordinary laptops, significantly reducing investment in detection equipment. Deep learning models combined with GPU acceleration improve detection accuracy and efficiency. Low-latency live streaming of detection footage is achieved using streaming media technologies (such as RTMP protocol and ZLMediaKit streaming media server) and relevant streaming parameter settings. Simultaneously, it uses spatiotemporal features to deduplicate and prioritize uploaded defect frames, automatically removing blurry, low-confidence, and redundant frames. This reduces the amount of uploaded data while ensuring the quality of key frames. These frames are then linked and stored with metadata such as IMU / GPS positioning, task information, defect category, and quantity, providing accurate data support for road maintenance and enabling queryable and traceable defect information. Attached Figure Description

[0031] Figure 1 This is the main interactive flowchart of the road defect detection system of the present invention.

[0032] Figure 2 This is a schematic diagram of the mobile data acquisition vehicle structure of the present invention.

[0033] Figure 3 This is a graph showing the changes in performance metrics during the training process of the deep learning model of this invention.

[0034] Figure 4This is a precision-recall curve of the deep learning model of this invention.

[0035] Figure 5 This is a confidence curve of the deep learning model of this invention.

[0036] Figure 6 Precision-recall curves of the deep learning model after adding the P6 detection layer in this invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Example

[0038] This embodiment provides a road defect intelligent detection system and method based on mobile data acquisition and cloud deployment. Its core interactive process is as follows: Figure 1 As shown. The system is physically divided into three core parts: a mobile data acquisition vehicle, a YOLO11 identification system, and a defect detection platform. The specific system deployment plan is as follows: The mobile data acquisition vehicle, as a core component for road surface data collection, such as... Figure 2 As shown, the system includes an embedded main control board 1, an image acquisition device 2, obstacle avoidance modules 3 and 4, positioning and navigation modules 5 and 6, a drive module 7, a wireless communication module, and a power supply module 8. Each module is connected to the embedded main control board 1. The embedded main control board 1 uses a Raspberry Pi 5 as its core controller, undertaking the overall data processing, command scheduling, and communication management to ensure the coordinated operation of all modules of the vehicle. The image acquisition device 2 uses a Module 3 high-definition camera, which is connected to the Raspberry Pi 5 via a CSI interface to acquire real-time images of the road surface. The obstacle avoidance modules 3 and 4 enable the vehicle to automatically avoid obstacles, including an HC-SR04 ultrasonic sensor 3 and an infrared sensor 4. The IMU / GPS integrated navigation and positioning modules 5 and 6 can achieve real-time positioning of the acquisition end in open outdoor spaces and most indoor environments, using the Zhongke Micro ATGM336H-5N. GPS positioning module 5 and CMP10AIMU inertial navigation module 6; drive module 7 uses L298N motor drive module to control 4 520 DC geared motors to realize the motion control of the car; wireless communication module is the WiFi module on the Raspberry Pi 5 board to realize the interaction between the acquisition end and other devices; power module 8 adopts a dual power supply independent power supply scheme. The YOLO11 recognition terminal is used to detect and identify road surface defects. It utilizes a laptop equipped with an NVIDIA GeForce RTX 3050 Ti Laptop GPU as the hardware platform, acting as an intermediary between the acquisition terminal and the defect detection platform. Its tasks include pulling camera video streams, identifying and annotating road defects in video frames, and pushing the annotated streams to the platform in real time. The software environment includes: CUDA 11.8, cuDNN 90100, PyTorch 2.7.1+cu118, OpenCV 4.13.0, Python 3.11.0, and Ultralytics YOLO11. The defect detection platform is deployed on a Tencent Cloud Light Application Server (4 cores, 4GB RAM, 3Mbps bandwidth in this embodiment), running Ubuntu Server 22.04 LTS 64bit. The technology stack used is Spring Boot + MySQL + JavaScript + HTML + CSS. It is the core of the entire system's management and interaction, including a task management module, a data receiving and storage module, a streaming media service module, a real-time map module, and a user interaction module. This platform primarily receives task instructions from users and sends them to the deep learning recognition end. It also stores video streams and receives defect frame images and related information, providing users with low-latency live video streaming and real-time map display services.

[0039] The Module 3 high-definition camera is placed on an adjustable bracket at the front of the vehicle, which can be flexibly adjusted up and down to allow for adjustment of the image angle according to actual shooting needs.

[0040] The ultrasonic sensor 3 and the infrared sensor 4 are respectively installed at the front and rear of the vehicle to detect the distance to obstacles in front and behind. When the distance in front is less than the safe distance, an emergency reverse is triggered. When the distance behind is less than the safe distance during reverse, the vehicle stops all movement until it receives the next start command.

[0041] The GPS positioning module 5 provides latitude and longitude information, while the IMU inertial navigation module 6 acquires acceleration, angular velocity, and Euler angles. These are fused using a Kalman filter. The IMU outputs high-frequency acceleration and angular velocity, which are integrated to obtain short-term pose; GPS provides low-frequency absolute position correction. The fusion algorithm constructs a state vector, uses the IMU's dynamic model for prediction, and then updates it using GPS observations, thereby outputting a smooth, drift-free navigation solution.

[0042] The drive module 7 receives control signals from the Raspberry Pi 5 and can precisely control the speed and direction of the motor. Users can remotely control the car through the control page on their mobile phones.

[0043] Based on the above solutions, this embodiment designs an intelligent road defect detection system based on mobile data acquisition and cloud deployment. The detailed implementation steps of this system are as follows: S1. The mobile data acquisition vehicle's camera automatically starts upon power-on, pushing real-time video streams via the RTMP protocol. After powering on, the mobile data acquisition vehicle moves along the road under user control, automatically avoiding obstacles if detected. Users access the defect detection platform through a client browser, add a new inspection task, and select whether to record. Upon receiving the request, the platform backend generates a unique task identifier for the task. After the task starts, the platform backend calls the control interface of the YOLO11 recognition terminal via the HTTP protocol to issue task execution instructions.

[0044] In step S1, the Raspberry Pi 5 pushes the video stream captured by the camera via FFmpeg and RTMP protocols. Specifically, the Raspberry Pi acquires the real-time camera feed (720×480 resolution, 30 frames per second, BGR24 pixel format) and then transmits the raw feed to FFmpeg for encoding. FFmpeg begins encoding and streaming, involving key parameter settings including: setting the encoder preset to ultrafast for maximum encoding speed and reduced latency; setting the tuning parameter to zerolatency for zero-latency optimization in live streaming scenarios; setting the pixel format (pix_fmt) to yuv420p for YUV420p output; setting the video bitrate (b:v) to 2000 kbps to balance image quality and smoothness; and setting the RTMP send buffer (rtmp_buffer) to 100 milliseconds to further reduce streaming latency. FFmpeg encodes the original video into an H.264 video stream, encapsulates it into FLV format, and pushes the video stream to the local server via the RTMP protocol; a virtual private network is established between the local server and the remote streaming device, so that the streaming device and the local server are in the same virtual local area network.

[0045] In step S1, the platform issues a task execution instruction. The specific method is as follows: a POST request is sent to the YOLO11 recognition terminal. The request path is / control_task, and the request body is in JSON format, containing the fields task_id and record. The task_id value is the generated task identifier, and the record field is set to 1 to indicate that recording has started, and set to 0 to indicate that recording has not started.

[0046] After receiving the instruction, the S2 and YOLO11 recognition terminals retrieve the camera video stream via the HTTP-FLV protocol and load the inference model for frame-by-frame inference. When the recognition terminal detects a defective frame, it marks the defective frame and uploads the specific defective frame, its location information, and the defect type and quantity to the platform.

[0047] In step S2, the training process and result analysis of the inference model deployed on the YOLO11 recognition terminal are as follows: The training process of the YOLO model is as follows: (1) The initial weight file yolo11s.pt, which is based on the YOLO11 object detection architecture and pre-trained on the COCO dataset, is selected as the backbone network for model initialization.

[0048] (2) Construct a multi-source road defect image dataset. In order to cover a variety of road defect types and ensure sample diversity, this embodiment obtains road surface images and corresponding annotation information from four public datasets: RDD2022_China_MotorBike, UAV Dataset for Automated Road Surface Degradation Detection in Real-World Conditions, BumpDataset_AlexOnly_withoutNight_1280x720, and Annotated Potholes Image Dataset. The defect types include cracks, potholes, repairs, ruts, bumps, etc.

[0049] After aggregation and standardization of annotation formats, a total of 4723 valid road surface images were obtained, covering seven types of road surface defects: longitudinal cracks, transverse cracks, network cracks, potholes, ruts, bumps, and repairs. The aggregated dataset was randomly divided into training, validation, and test sets at a ratio of 70%, 20%, and 10%, respectively. The annotation files underwent coordinate system normalization and YOLO format conversion preprocessing to adapt to the input requirements of the YOLO11 detection model.

[0050] (3) Execute the training command in the server environment (GPU model: NVIDIA RTX A6000, driver version: 550.127.05, CUDA version: 12.4) and configure the following key parameters to achieve model optimization: Basic configuration includes 150 training epochs, 640 input resolution (imgsz), 32 batch size, single GPU, 8 data loading processes (workers), and 20 early stopping patience. Data augmentation strategies include setting hue, saturation, and brightness offset parameters to 0.015, 0.7, and 0.4 respectively; random rotation angle range of ±10 degrees; random translation ratio of 0.1; random scaling ratio of 0.5; random cropping angle of ±2 degrees; perspective transformation coefficient of 0.001; horizontal flip probability of 0.5; mosaic splicing probability of 0.8; mixed sample probability of 0.2; and instance copy-paste probability of 0.3. The learning rate strategy includes setting the initial learning rate lr0 to 0.001, using cosine annealing decay, setting the final learning rate factor lrf to 0.01, setting the warmup epochs to 3, and setting the weight decay coefficient to 0.0005.

[0051] In this embodiment, a variety of comprehensive evaluation metrics are used to fully reflect the model's accuracy, robustness, and response efficiency, mainly including loss, precision, recall, F1 score, average precision (mAP50), and mAP50-95.

[0052] like Figure 3-5 As shown, based on the above model training process, the training results of the YOLO model are as follows: Figure 3 It can be seen that during training, the model's loss on both the training and validation sets steadily decreased, with no obvious signs of overfitting; parameters such as precision, recall, and mAP50 gradually increased, with the final mAP50 approaching 0.9 and mAP50-95 approaching 0.6. Figure 4 It can be seen that the model achieves an overall performance of mAP50 = 0.890 on the validation set. From... Figure 5 It can be seen that when the confidence threshold is set to 0.418, the model's overall F1 score for all categories reaches a maximum of 0.86, indicating good performance. Finally, the trained and converged YOLO11 inference model (optimal weight file best.pt) is deployed on the YOLO11 recognition terminal to perform road defect detection and recognition tasks; this model has out-of-the-box characteristics and can run continuously without repeated training.

[0053] Step S2 is as follows: S2-1, Task-Driven. After the YOLO11 identification service starts, it provides a preset network interface. Upon receiving an HTTP request (containing task_id and record flags) from the defect detection platform, it starts a thread to run the core processing logic.

[0054] S2-2. Streaming and Reconnection. The streaming address is an HTTP-FLV format streaming media address. The YOLO11 identification terminal pulls the video stream from the local server through the virtual LAN, realizing cross-network remote streaming. When reading frame data from the video stream fails, resources will be automatically released and a reconnection attempt will be made until the connection is successful or the task termination flag is set.

[0055] S2-3, Model Inference. The recognition end calls the trained optimal weight model file best.pt and performs inference frame by frame on the pulled road video stream. Inference is accelerated using a graphics processing unit (GPU) and parameters such as image size and confidence threshold are preset.

[0056] S2-4, Defect Labeling. When a road defect is detected in a frame, the recognition device calls YOLO's built-in method to directly draw the detection bounding box and category label (including defect category and confidence level) on the original video frame, generating an overlay video stream.

[0057] S2-5. Uploading Defect Frames and Related Information. When a defect frame is detected, it is cached within a preset time window, and the defect category and quantity are recorded. For each frame within the window, a quality score is calculated by combining sharpness, average detection confidence, and defect location centrality. Similarity is compared using the correlation of inter-frame color histograms, and the frame with the higher quality score among similar frames is retained. After the preset time period, the frame with the highest quality score is selected from the window. The platform is logged in via an HTTP POST request, and the labeled defect frame is uploaded to the platform. Simultaneously, a request is sent to the HTTP GET interface of the mobile acquisition vehicle control terminal to obtain the real-time latitude and longitude data of the current vehicle. The upload task of the defect frame, its defect category and quantity, and the current integrated navigation positioning coordinates is executed, and then the time window is cleared.

[0058] S2-6. Image Management. The platform records image association information in a MySQL database and displays all defective images and their related information in a list format on the front-end page.

[0059] The S3 and YOLO11 recognition terminals push the video stream with superimposed labeled boxes to the platform's backend service (streaming media server) in real time via FFmpeg. Users can then view the labeled inspection video in real time by opening the defect detection platform's live stream.

[0060] Step S3 is as follows: The S3-1 and YOLO11 recognition terminals write the overlaid video stream to FFmpeg in BGR24 raw pixel format via standard input; the encoding parameters specify H.264 encoding, YUV420P pixel format, ultra-low latency optimization, and constant quality factor CRF=23; and push the stream to the ZLMediaKit streaming media server.

[0061] S3-2. The platform front-end integrates flv.js to pull HTTP-FLV live streams from the streaming media server, achieving real-time low-latency playback with an end-to-end latency of approximately 1 second.

[0062] S4. When the user selects to record in the task instructions, the YOLO11 recognition terminal will call the recording interface and trigger the recording operation when pushing the video stream. After the video stream is pushed out, the platform will trigger a stop recording operation and complete the video data archiving.

[0063] Step S4 is as follows: S4-1. When the record flag is set to 1 in the task instruction issued by the platform, the YOLO11 recognition terminal triggers recording by calling the ZLMediaKit HTTP API interface. In the specific implementation, a retry mechanism is set up to prevent interface call failure due to network jitter.

[0064] S4-2. After recording is triggered, ZLMediaKit begins writing the received video data to a local file. After the video stream ends, the platform calls the ZLMediaKit interface to stop recording. Upon receiving the instruction, ZLMediaKit encapsulates the buffered data into a complete MP4 file. The platform's front-end page supports online video playback.

[0065] S5. During the inspection process, users can monitor the current location of the target vehicle through the platform's real-time map function. After the vehicle completes the inspection of the designated section, the user shuts it down, and the camera stops working. The YOLO11 recognition terminal stops pulling the camera video stream and sends a "task completed" status message to the platform's preset notification interface via HTTP POST. The platform updates the task status upon receiving the message. Users can then track the inspection information through the platform.

[0066] In step S5, the specific implementation method of the real-time map function is as follows: The map initialization operation is completed by passing in the Key and the security key, and the map is specifically connected to Gaode Map. During the inspection process, the platform obtains the combined navigation and positioning information of GPS and IMU by calling the HTTPGET interface of the mobile data collection vehicle control terminal. The front end uses the latitude and longitude as the center point coordinates of the point markers and overlays them on the electronic map layer to complete the rendering of the point markers. Example

[0067] The difference between this embodiment and Embodiment 1 is that the YOLO11 model training uses the initial weight file yolo11s.pt, which is based on the YOLO11 object detection architecture and pre-trained on the COCO dataset, as the backbone network for model initialization, and adopts the model configuration file improved for the road defect detection task.

[0068] The proposed solution has three modifications to the official yolo11.yaml file: First, to adapt to road scenarios, the number of detection categories has been reduced from 80 to 7; second, a P6 layer (downsampling by 64 times) has been added at the end of the backbone network; and third, the original 3-scale FPN has been expanded to 4-scale at the neck, with an additional P6 detection branch. Overall, the depth and width coefficients at the s-scale of the original solution are retained, and all modules (Conv, C3k2, SPPF, C2PSA) are consistent with the original solution.

[0069] This approach enhances multi-scale sensing capabilities, helping to capture larger targets or those with a wider imaging range. For example... Figure 6 As shown, the mAP50 of potholes increased from 0.709 in the original scheme to 0.734, and the mAP50 of ruts also increased by 0.002. The mAP50 of other types showed slight fluctuations in accuracy, achieving targeted optimization.

[0070] The intelligent road defect detection system and method based on mobile data acquisition and cloud deployment provided by this invention can be widely applied to defect inspection in scenarios such as urban roads, highways, industrial parks, and rural roads. It utilizes a low-cost embedded main control board and a regular laptop computer, significantly reducing investment in detection equipment. Through a three-tiered architecture of mobile data acquisition, local recognition and processing, and cloud management, it effectively integrates hardware and software. Simultaneously, it provides functions such as task management, real-time streaming, video and image data archiving, and real-time mapping, offering a one-stop solution for road maintenance and possessing high industrial practical value.

[0071] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A road defect intelligent detection system based on mobile data acquisition and cloud deployment, characterized in that: The mobile data acquisition vehicle, as the core component for road data acquisition, includes an embedded main control board, an image acquisition device, an obstacle avoidance module, a positioning and navigation module, a drive module, a wireless communication module, and a power supply module. Each module is connected to the embedded main control board. The embedded main control board is responsible for overall data processing, command scheduling, and communication management, ensuring the coordinated operation of all modules within the vehicle. The image acquisition device is responsible for acquiring real-time road images. The obstacle avoidance module enables the vehicle to automatically avoid obstacles. The positioning and navigation module can achieve real-time location positioning of the acquisition terminal in open outdoor spaces and most indoor environments. The drive module controls the vehicle's motion. The wireless communication module enables interaction between the acquisition terminal and other devices. The power supply module adopts a dual-power independent power supply scheme. The deep learning recognition terminal is used to detect and identify road surface defects. The hardware carrier device has inference computing and communication functions. It realizes the relay between the acquisition terminal and the defect detection platform, and undertakes tasks such as pulling camera video streams, identifying and annotating road defects in video frames, and pushing the annotated video stream to the platform in real time. The defect detection platform, deployed on a cloud server, is the core of the entire system's management and interaction. It includes a task management module, a data receiving and storage module, a streaming media service module, a real-time map module, and a user interaction module. This platform is mainly used to receive task instructions from users and send them to the deep learning recognition end. At the same time, it stores video streams and receives defect frame images and related information, and provides users with services such as low-latency live video streaming and real-time map display.

2. The intelligent road defect detection system based on mobile data acquisition and cloud deployment according to claim 1, characterized in that, The image acquisition device is placed on an adjustable bracket at the front of the trolley, which can be flexibly adjusted up and down to facilitate adjusting the image angle according to actual shooting needs.

3. The intelligent road defect detection system based on mobile data acquisition and cloud deployment according to claim 1, characterized in that, The obstacle avoidance modules are installed at the front and rear of the vehicle to detect the distance between the vehicle and obstacles in front and behind. When the distance in front is less than the safe distance, an emergency reverse is triggered. When the distance behind is less than the safe distance while reversing, the vehicle stops all movement until it receives the next start command.

4. The intelligent road defect detection system based on mobile data acquisition and cloud deployment according to claim 1, characterized in that, The positioning and navigation module uses information fusion correction to output a smooth, drift-free navigation solution; the drive module receives control signals from the embedded main control board and can precisely control the speed and direction of the motor, allowing users to remotely control the vehicle through the control page on their mobile phones.

5. A road defect intelligent detection method based on mobile data acquisition and cloud deployment, characterized in that, The system according to any one of claims 1-4 includes the following steps: S1. The mobile data acquisition vehicle's camera automatically starts upon power-on, pushing real-time video streams. After powering on, the mobile data acquisition vehicle moves along the road under user control. If an obstacle is detected along the way, it will automatically avoid it. Users access the defect detection platform through a client browser, add a new inspection task, and select whether to record. After receiving the request, the platform backend generates a unique task identifier for the task. After the task starts, the platform backend calls the control interface of the deep learning recognition terminal to issue task execution instructions. S2. After receiving the instruction, the deep learning recognition end pulls the camera video stream and loads the inference model to perform frame-by-frame inference. When the recognition end detects a defective frame, it marks the defective frame and uploads the specific defective frame, the frame's location information, and the defect type and quantity to the platform. S3, the deep learning recognition end pushes the video stream with superimposed labeled boxes to the platform's backend service in real time; users can open the defect detection platform's live broadcast screen to watch the labeled detection video in real time; S4. When the user selects to record in the task instruction, the deep learning recognition terminal will call the recording interface when pushing the video stream, triggering the recording operation; after the video stream is pushed out, the platform will trigger the stop recording operation to complete the video data archiving. S5. During the inspection process, users can monitor the current location information of the target vehicle through the platform's real-time map function; after the vehicle completes the inspection of the designated road section, the user turns off the vehicle and the camera stops working; the deep learning recognition end stops pulling the camera video stream and sends a "task completed" status message to the platform, which updates the task status upon receiving it; users can then trace the inspection information through the platform.

6. The intelligent road defect detection method based on mobile data acquisition and cloud deployment according to claim 5, characterized in that, Step S1 specifically involves the embedded main control board pushing the video stream captured by the camera. The specific method is as follows: the embedded main control board acquires the real-time image from the camera, then transmits the raw image to FFmpeg for encoding and encapsulates it into a streaming media format. The video stream is then pushed to the streaming server via a real-time streaming protocol. The remote streaming device is connected to the local area network where the streaming server is located via a virtual private network. The platform issues task execution instructions by sending a data transmission request to the deep learning recognition terminal. The request field includes a task identifier and a recording flag. The task identifier is a unique identifier that is generated. The recording flag is set to 1 to indicate that recording has started, and set to 0 to indicate that recording has not started.

7. The intelligent road defect detection method based on mobile data acquisition and cloud deployment according to claim 5, characterized in that, Step S2 is as follows: S2-1, Task-driven: After the deep learning recognition service starts, it provides a preset network interface; when it receives a network request from the defect detection platform, it starts a thread to run the core processing logic. S2-2, Pulling and Reconnecting: The pull address is the streaming media address. The deep learning recognition end pulls the video stream from the push end server via the virtual private network. When reading frame data from the video stream fails, resources will be automatically released and a reconnection attempt will be made until the connection is successful or the task termination flag is set. S2-3, Model Inference: The recognition end calls the optimal weight model file obtained from training and performs inference frame by frame on the pulled video stream; the inference is accelerated by the graphics processing unit (GPU) and parameters such as image size and confidence threshold are preset; S2-4, Defect Labeling: When a road defect is detected in a frame, the recognition end directly draws the detection bounding box and category label on the original video frame to generate an overlay video stream; S2-5. Uploading Defect Frames and Related Information: When a defect frame is detected, it is cached within a preset time window, and the defect type and quantity are recorded. For each frame of image within the window, a quality score is calculated by combining sharpness, average detection confidence, and defect location centrality. Similarity is compared by the correlation of color histograms between frames, and the frames with higher quality scores among similar frames are retained. After the preset time period is reached, select the frame with the highest quality score from the window; log in to the platform via network request and upload the marked defective frame to the platform; Simultaneously, a request is sent to the control terminal of the mobile data acquisition vehicle to obtain the real-time latitude and longitude data of the vehicle. Execute the task of uploading the defect frame, the defect type and quantity of the frame, and the current integrated navigation positioning coordinates, and then clear the time window; S2-6. Image Management: The platform records image association information in the database and displays all defective images and their related information in a list format on the front-end page.

8. The intelligent road defect detection method based on mobile data acquisition and cloud deployment according to claim 5, characterized in that, Step S3 specifically involves: the deep learning recognition end writing the superimposed video stream into FFmpeg and pushing it to the cloud streaming media server; the platform front-end pulling the live stream from the streaming media server to achieve real-time low-latency playback.

9. The intelligent road defect detection method based on mobile data acquisition and cloud deployment according to claim 5, characterized in that, Step S4 is as follows: S4-1. When the recording flag is set to 1 in the task instruction issued by the platform, the deep learning recognition end triggers recording by sending a network request to the streaming media server. In the specific implementation, a retry mechanism is set to prevent network jitter from causing the interface call to fail. S4-2. After recording is triggered, the streaming media server begins writing the received video data to the local file; After the video stream ends, the platform sends a network request to the streaming media server to stop recording the stream; After receiving the instruction, the streaming media server encapsulates the buffered data into a complete MP4 file.

10. The intelligent road defect detection method based on mobile data acquisition and cloud deployment according to claim 5, characterized in that, Step S5 is as follows: The specific implementation method of the real-time map function is to pass in the Key and the security key to complete the map initialization operation; during the inspection, the platform sends a network request to the mobile data collection vehicle to obtain the combined navigation and positioning information of GPS and IMU. The front end uses the latitude and longitude as the center point coordinates of the point markers and overlays them on the electronic map layer to complete the rendering of the point markers.

Images