Mobile road property facility detection tracking method based on deep learning

Through deep learning technology and edge computing equipment, real-time detection and tracking of road production facilities is solved, and the problems of damage and loss of road production facilities are realized, and automated monitoring and early warning of road traffic safety is realized to ensure the normal operation of road traffic functions.

CN118942048APending Publication Date: 2024-11-12CHINA RAILWAY SEVENTH GRP CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410780143.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Due to traffic accidents, damage to residents along the route and theft of criminals, road production facilities are often damaged and lost, resulting in the impact of road traffic functions. Timely detection and early warning are needed to ensure normal road driving.

Method used

Using a mobile road production facility detection and tracking method based on deep learning, the road production facility image is extracted, and the data set is constructed. The YOLOV8 target detection network is used to identify and track the facilities, establish a dictionary model, and match the query in real time to detect damage. With the loss situation and real-time detection by edge computing devices.

Benefits of technology

Timely detection and alarm of road production facilities has been achieved, the degree of automation of inspection has been improved, road traffic safety has been ensured, and traffic accidents have been reduced due to the loss of facilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942048A_ABST
    Figure CN118942048A_ABST
Patent Text Reader

Abstract

The invention discloses a mobile road property facility detection tracking method based on deep learning, and the method comprises the steps: employing mobile road property facility detection equipment (hereinafter referred to as mobile equipment) which is installed in a center console region below a front windshield of a vehicle, and can shoot various signs, guardrails and other road property facilities of a road along with the driving of the vehicle; carrying out image segmentation on the collected video of the road property facility, and labeling a target picture by using a labeling model in combination with a road property facility feature picture collected on a network to construct a data set; a YOLOV8 deep learning model is adopted for training, so that the mobile equipment can identify and track various road property facilities; in addition, a mobile positioning device is adopted to carry out GPS positioning on roadside road property facilities, and a standard dictionary of position information of the road property facilities is constructed; in the driving process of the automobile, the mobile equipment continuously identifies and positions various road property facilities, and detection and alarm on damage and loss conditions of the road property facilities are realized by matching and querying dictionary information in real time. The method is of great significance for monitoring the intact degree of the road property facilities of roads and bridges, is very effective for timely detecting whether the road property facilities are lost or not and improving the automation degree of road property facility detection, and can improve the detection efficiency and the road safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention proposes a mobile road property facility detection and tracking method based on deep learning, which belongs to the field of intelligent detection technology. Background Art

[0002] National and provincial roads contain numerous roadside equipment and facilities, including traffic signs, markings, and guardrails, ensuring the normal function of roads. However, these facilities often suffer damage or loss due to traffic accidents, vandalism by residents along the roads, and theft by criminals. Therefore, timely monitoring of the integrity of these facilities is crucial, providing timely warnings and replacements when lost, to ensure the normal function of roads. To promptly detect the loss of roadside facilities and improve the automation of facility detection, this paper develops a deep learning method for facility detection and identification. This method uses on-board detection equipment to capture and monitor the integrity of these facilities in real time. While the vehicle is in motion, this method detects and alerts users of damaged or missing facilities through real-time matching and querying dictionary information. Summary of the Invention

[0003] This invention provides a deep learning-based mobile road property facility detection and tracking method and system to address the frequent damage and loss of road property equipment and facilities due to traffic accidents, vandalism by residents along the road, and theft by criminals. It is necessary to promptly detect the integrity of road property facilities and issue early warnings and replacements in the event of loss to ensure the normal function of roads: The present invention proposes a method for detecting and tracking mobile road facilities based on deep learning, which includes: S1. Obtaining video data of road facilities of a road to be tested, and extracting images of the road facilities from the recorded video data; S2. Constructing a data set from the road and facility images, labeling the road and facility images (specific labels include: instruction signs, warning signs, signal lights, caution signs, prohibition signs, and auxiliary signs), and determining the categories of the current road and facility; S3. Identify multiple target road facilities from the road image based on the YOLO V8 target detection network, then track and identify the multiple target road facilities based on the target tracking algorithm, and determine status information of the multiple target road facilities based on the identified road facilities and the corresponding road facility coordinates; S4. Dictionary modeling is performed based on the intact status and location information of the target road property facilities. During vehicle operation, the dictionary information is matched and queried in real time to detect and alarm the damage and loss of road property facilities.

[0004] According to a specific implementation, in the above-mentioned mobile road property facility detection and tracking method based on deep learning, a deep learning network module is used to label the dataset images using a feature annotation method.

[0005] According to a specific embodiment, in the above-mentioned deep learning-based mobile road property facility detection and tracking method, the real-time tracking target detection model is: YOLOV8 deep learning network model. The model deployment effect is achieved by adjusting the parameters and training the network model.

[0006] According to a specific implementation, in the above-mentioned mobile road property facility detection and tracking method based on deep learning, the target real-time tracking detection model is constructed by the following method, including: constructing a target data set, pre-training a data set based on the road property facility pictures generated by the target road and the road property facility pictures collected from the network to obtain a pre-trained network model; predicting the road image data set outside the database to obtain a recognition effect diagram of the road property facilities.

[0007] According to a specific embodiment, in the above-mentioned deep learning-based mobile road property facility detection and tracking method, the target tracking algorithm is used to track and identify multiple target road property facilities, including: based on the YOLOV8 model, target recognition is performed according to the appearance features of the target road property facilities; and the appearance features of the target road property facilities are extracted using the Feature Extractor network.

[0008] According to a specific embodiment, the deep learning-based mobile road property facility detection and tracking method analyzes whether the road property facilities on the current road are missing or intact based on their GPS positioning information and classification information. This includes storing a location information dictionary composed of the characteristic information and location information of each road property facility along the route into a real-time tracking and recognition model, and performing information matching during real-time target detection to determine whether the road property facilities are missing.

[0009] Another aspect of the present invention provides an edge computing device comprising: a portable Jetson Nano GPU 4GB processor, a GPS locator, a display screen, and a control keyboard. The storage device is built into the portable Jetson Nano GPU 4GB processor, and the GPS locator and the Jetson Nano GPU 4GB processor are connected. The storage device is used to store a computer program, which includes program instructions for executing the above-mentioned real-time road and property facility status detection method based on edge computing.

[0010] Another aspect of the present invention provides a real-time road property and facility detection system based on deep learning, the system comprising: a USB camera device for video acquisition, for acquiring detection video stream data of the current road and transmitting it to a computing device that deploys a deep learning model; a Jetson Nano GPU 4GB computing device, for acquiring real-time video stream data, and using the above-mentioned mobile road property and facility detection and tracking method based on deep learning to analyze the integrity of the current road property and facilities during normal road operation.

[0011] According to a specific embodiment, in the above-mentioned real-time road and property facility detection system based on deep learning calculation, the computing device for deploying deep learning is a Jetson Nano GPU 4GB computing development board; the system also includes: a GPS locator; the GPS locator is connected to the deep model computing device by a data cable, and is used to transmit the location data of the current road and property facilities to the edge computing device.

[0012] Compared with the existing technology, the present invention has the following beneficial effects: the present invention constructs an on-board road property facility detection hardware model based on the Jetson Nano GPU 4GB development board and camera equipment, deploys the above-mentioned algorithm into the model, collects and organizes image information of roadside road property facilities to be detected, and detects and tracks and identifies road property facilities in real time. Afterwards, dictionary modeling is performed for intact road property facilities along the road. Finally, while the vehicle is running, the dictionary information is matched and queried in real time to detect and alarm for damage and loss of road property facilities. The present invention can promptly detect whether road property facilities are lost, thereby improving the degree of automation of road property facility detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a method for detecting and tracking mobile roadside facilities based on deep learning in one embodiment; Figure 2 This is a road property facility labeling interface in one embodiment; Figure 3 This is a target detection effect diagram in one embodiment; Figure 4 This is a target tracking effect diagram in one embodiment; Figure 5 A block diagram of the computing device structure in one embodiment. DETAILED DESCRIPTION

[0014] The present invention will be described in further detail below in conjunction with specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments, as all technologies implemented based on the present invention fall within the scope of the present invention.

[0015] Example 1 Figure 1 The present invention illustrates a method for detecting and tracking mobile roadside facilities based on deep learning, which is applied to a portable Jetson Nano GPU 4GB computing device. The method includes: S1. Obtain video stream data of current road facilities, and extract road facility images from the video stream data of the target road; S2. Extract images containing road facilities from the road video, construct a data set based on a network road facility feature map, label the images in the data set to construct a labeled data set, and determine the data set required for input into the training model; S3. Identify multiple target road facilities from the road image based on a real-time target tracking detection network, and track and identify the multiple target road facilities based on the YOLOv8 target tracking algorithm according to the identified road facility coordinates, and determine the integrity information and location information of the multiple target road facilities; S4. Dictionary modeling is performed for intact road facilities along the road. During vehicle operation, the dictionary information is matched and queried in real time to detect and alarm for damage or loss of road facilities.

[0016] In this embodiment, the specific location information of the target road facilities is obtained based on the GPS locator by analyzing the characteristic map of road facilities obtained, and the target detection network based on real-time detection, tracking and identification identifies multiple target road facilities appearing in the real-time detection video. The target road facilities are tracked and identified based on the built-in target tracking algorithm to determine the characteristics and specific location information of the target road facilities. Then, based on the modeling dictionary of intact road facilities along the road, the dictionary information is matched and queried in real time while the vehicle is running, to detect and alarm the damage and loss of road facilities. The present invention tracks and identifies road facilities through a labeling feature algorithm, a target detection algorithm and a target tracking algorithm, and based on the modeling dictionary of intact road facilities, to detect and alarm the damage and loss of road facilities through real-time matching and querying dictionary information while the vehicle is running. The present invention provides an automated method for the detection of road traffic safety facilities, which can prevent traffic safety hazards caused by the loss of safety signs.

[0017] Example 2 In one possible implementation, in step S2 of the aforementioned real-time road and property status detection method based on a 4GB Jetson Nano GPU, a traffic sign detection dataset consisting of 2,202 images was constructed. Some images were selected from the China Traffic Sign Detection Database, and some were captured on test roads. According to my country's GB 51038-2015 "Specifications for the Installation of Traffic Signs and Markings on Urban Roads," traffic signs are classified into six categories based on their characteristics: direction signs, warning signs, signal lights, caution signs, prohibition signs, and auxiliary signs. The dataset was then manually annotated using a feature annotation module, circling the location of each traffic sign in each image and outputting a label file for the YOLO V8 model. Ultimately, each image received a label file containing the image dimensions and the coordinates of each traffic sign.

[0018] In one possible implementation, in S3 of the above-mentioned real-time road property and facility status detection method based on edge computing, the target detection model is based on the YOLOV8 multi-target detection algorithm and is used to identify six types of road property and facility targets. During the training process, the constructed road property and facility detection dataset is used for pre-training, and then the annotated data of the road property and facility images collected from the actual roadside are used for verification to achieve higher detection accuracy, such as Figure 4 As shown, it can be seen that the example of the present invention can perform clear and accurate target detection and anchoring.

[0019] In this embodiment, the YOLOV8 multi-target detection algorithm is used to accurately detect the positions and categories of all road property facilities in the image, and a feature tracking algorithm is used to improve the accuracy of road property facility target tracking.

[0020] In one possible implementation, the aforementioned edge computing-based real-time road and infrastructure status detection method utilizes the latest YOLO V8 model, a more lightweight and faster model with a built-in tracking algorithm, to enable real-time inference on the edge device's Jetson Nano GPU 4GB computing platform. This platform-based approach enables single-frame image detection in under 3ms, meeting the requirements for real-time detection and tracking.

[0021] In one possible implementation, in S3 of the above-mentioned real-time road property facility status detection method based on edge computing, the target tracking algorithm is used to track and identify multiple target road property facilities, including: using the built-in tracking algorithm of YOLOV8 to re-identify according to the appearance of the target image. When extracting the appearance features of road property facilities, YOLOV8 uses a deep convolutional neural network for efficient feature extraction, and gradually refines the complex features in the image through structures such as convolution layers, normalization layers, and activation functions, and integrates feature maps of different scales with the help of technologies such as feature pyramid networks to enhance the detection capability of targets of various sizes. In addition, YOLOV8 integrates a target tracking algorithm to detect and track video stream targets. It tracks targets in continuous frames through feature matching and motion prediction technology to provide an efficient and accurate target detection and tracking system.

[0022] In this embodiment, the target tracking algorithm is used to track the road property facilities, so that each road property facility has a unique ID tag corresponding to it after it appears, thereby effectively collecting statistics on the status information of the road property facilities.

[0023] In summary, the method provided by the example of the present invention has the following advantages: 1. Adjusting the network parameters for road property facility target detection and tracking to make it suitable for deployment and calculation at the edge, and real-time identification and tracking of road property facilities while ensuring high accuracy; 2. Automatically labeling feature images during initialization to segment the training set and verification set, dividing the data set, while improving accuracy and avoiding tedious manual labeling, saving time and labor costs; 3. Being able to more flexibly and reliably count the status of various road property facilities within the detection time interval, judge the current status of traffic road property facilities, and effectively communicate to the cloud, so as to facilitate further unified processing of the entire road network traffic road property facilities, so as to ensure the good condition of traffic road property facilities and reduce traffic accidents caused by the loss of road property facilities.

[0024] Example 3 Another aspect of the present invention, as Figure 5 As shown, an edge computing device is also provided, comprising a processor, a GPS locator, a display screen, a control keyboard, and a memory, wherein the GPS locator, the display screen, the control keyboard, and the memory are interconnected via the network interface and the Jetson Nano GPU 4GB. The memory is used to store a computer program, and the processor invokes the program instructions to execute the above-mentioned real-time road and property facility status detection method based on edge computing.

[0025] In one possible implementation, the edge computing device uses a Jetson Nano GPU 4GB edge computing development board. The Jetson Orin Nano 4GB features a Jetson Nano GPU 4GB and a dual-core CPU processor, offering high computing power, low power consumption, low cost, and ease of deployment.

[0026] Another aspect of the present invention provides a real-time road and infrastructure status monitoring system based on edge computing. The system comprises the aforementioned edge computing device, a camera, a screen, a GPS module, a keyboard, and a mounting clamp. During use, the system is secured to a moving vehicle using the mounting clamp. The system is then powered by the vehicle and can operate autonomously for extended periods of time.

[0027] The system provided in this example fully deploys the algorithm on edge computing devices close to the front-end acquisition equipment. Compared with deployment on cloud servers and host computers, it has lower transmission costs, lower computing latency, and less computing and storage pressure on cloud servers.

[0028] Specifically, once the system is correctly installed and initialized, it can perform real-time detection and tracking of road infrastructure, collect statistics on its status, and transmit these results via the communication module. During initialization, semantic segmentation is performed on images of road infrastructure. During detection, real-time frame-by-frame video analysis is performed, using a modified YOLO V8 model for infrastructure object detection. This is followed by object tracking, analysis, and statistics based on a real-time matching query dictionary.

[0029] In one possible implementation, the detection system further includes: connecting a GPS locator. The built-in GPS locator is communicatively connected to the edge computing device to transmit the location information of the road property facility currently identified by the edge computing device.

[0030] Another aspect of the present invention provides a computer storage medium, which stores program instructions. When the program instructions are executed by at least one processor, they are used to implement the above-mentioned real-time road and property facility status detection based on edge computing, and are also used to receive current road and property facility identification and detection effect data transmitted by the edge computing device.

[0031] In an embodiment of the present invention, the processor may be an integrated circuit chip having a signal processing capability.

[0032] It should be understood that the system disclosed herein can be implemented in other ways. For example, the module division described is merely a logical functional division, and actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, the communication connections between modules may be through interfaces, indirect coupling or communication connections between servers or units, and may be electrical or otherwise.

[0033] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each module may exist physically separately, or two or more modules may be integrated into a single processing unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0034] The integrated unit, if implemented as a software functional module and sold or used as a separate product, can be stored on a computer-readable storage medium. Accordingly, the technical content of the present invention, or the main contribution to advanced technology, can exist in whole or in part in the form of a software product. This software product is encoded on a storage medium and contains an instruction set that enables a computer device (such as a personal computer, server or network device) to perform part or all of the steps of the method proposed in the present invention. The above-mentioned storage medium may be a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, etc., that is, any medium that can store program code.

[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A mobile road property facility detection and tracking method based on deep learning, characterized in that: The method comprises: Acquire road property and facility video data of the road to be tested, and extract road property and facility images from the recorded video data; Constructing a data set from the road property and facility images, labeling the features of the road property and facility images (specific labels are: instruction signs, warning signs, signal lights, attention signs, prohibition signs, auxiliary signs), and determining the feature categories of the current road property and facility; Based on the YOLOV8 target detection network, multiple target road facilities are identified from the road image, and then the multiple target road facilities are tracked and identified based on the target tracking algorithm, and the status information of the multiple target road facilities is determined according to the identified road facilities and the corresponding road facility coordinates; Dictionary modeling is performed based on the intact status information and location information of the target road property facilities. When the vehicle is running, the damage and loss of road property facilities can be detected and alarmed through real-time matching and query of dictionary information.

2. According to the method for detecting and tracking mobile road facilities based on deep learning in claim 1, it is characterized in that: In the above-mentioned mobile road property facility detection and tracking method based on deep learning, a deep learning network module is used to label the data set images using a feature annotation method.

3. According to claim 2, a method for detecting and tracking mobile road facilities based on deep learning, characterized in that: According to a specific implementation, in the above-mentioned mobile road property facility detection and tracking method based on deep learning, the target real-time tracking detection model is constructed by the following method, including: constructing a target data set, constructing a data set based on the road property facility pictures generated by the target road and the road property facility pictures collected from the network for pre-training, and obtaining a pre-trained network model; predicting the road image data set outside the database to obtain a recognition effect diagram of the road property facilities.

4. According to claim 3, a method for detecting and tracking mobile road facilities based on deep learning is characterized in that: According to a specific implementation, in the above-mentioned mobile road property facility detection and tracking method based on deep learning, the real-time tracking target detection model is: YOLOV8 deep learning network model. The effect of model deployment is achieved by adjusting parameters and training the network model.

5. According to claim 4, a method for detecting and tracking mobile road facilities based on deep learning is characterized in that: The target real-time tracking detection model is constructed by the following method, including: constructing a target data set, pre-training a data set based on road property facility pictures generated by the target road and road property facility pictures collected from the network to obtain a pre-trained network model; predicting a road image data set outside the database to obtain a recognition effect diagram of the road property facilities.

6. According to claim 5, a method for detecting and tracking mobile road facilities based on deep learning is characterized in that: The target tracking algorithm is used to track and identify the plurality of target road facilities, including: based on the YOLOV8 model, target identification is performed according to the appearance features of the target road facilities; and the appearance features of the target road facilities are extracted using a Feature Extractor network.

7. A method for detecting and tracking mobile road facilities based on deep learning according to claim 6, characterized in that: According to the GPS positioning information and classification information of the road facilities, analyze whether the road facilities on the current road are lost or intact. This includes: saving the location information dictionary composed of the characteristic information of each road facility along the route and the location information of each road facility to the real-time tracking and recognition model, and performing information matching during real-time target detection to determine whether the road facilities are lost.

8. The method for detecting and tracking mobile road facilities based on deep learning according to claim 7, characterized in that: The storage device is built into a portable Jetson Nano GPU 4GB processor, and the GPS locator is connected to the Jetson Nano GPU 4GB processor. The storage device is used to store a computer program, and the computer program includes program instructions to execute the above-mentioned real-time road property facility status detection method based on edge computing.

9. A method for detecting and tracking mobile road facilities based on deep learning according to claim 8, characterized in that: The system includes: a USB camera device for video acquisition, which is used to collect detection video stream data of the current road and transmit it to a computing device that deploys a deep learning model; a Jetson Nano GPU 4GB computing device, which is used to obtain real-time video stream data and use the above-mentioned deep learning-based mobile road property facility detection and tracking method to analyze the integrity of the current road property facilities during normal road operation.

10. According to the method of mobile road property facility detection and tracking based on deep learning in claim 9, the above algorithm is deployed in the vehicle-mounted road property facility detection hardware model based on the Jetson Nano GPU 4GB development board and camera equipment, and the image information of the road property facilities on the road side to be detected is collected and sorted, and the road property facilities are detected and tracked in real time. After that, dictionary modeling is performed for the intact road property facilities along the road. Finally, when the vehicle is running, the detection and alarm of damage and loss of road property facilities are realized by real-time matching and querying dictionary information.

Citation Information

Cited By

  • Landslide early warning method based on polymorphic target tracking intelligent identification

    CN119942730A