Device and method for detecting road defects based on deep learning using cctv, and computer program executing the same

Abstract

A road defect detection device according to the present invention comprises: a real data collection unit that collects actual CCTV data from a fixed CCTV installed on a road; a virtual data generation unit that generates virtual CCTV data by inserting an image of a defect object into the actual CCTV data; a road defect information acquisition unit that acquires road defect information using a road defect detection model constructed by learning the actual CCTV data and the virtual CCTV data; a traffic volume information acquisition unit that acquires road traffic volume information based on the actual CCTV data; and a restoration priority determination unit that determines a restoration priority based on the road defect information and the traffic volume information. According to the present invention, by automatically detecting road defects based on road images acquired from pre-installed traffic information CCTVs, manpower, time, and costs are saved, and the nationwide expansion of road defect detection becomes possible. Furthermore, by performing road vulnerability analysis using an artificial intelligence model based on a Convolutional Neural Network (CNN), preemptive maintenance of roads nationwide becomes possible before an accident occurs.

Description

Technology Field

[0001] The present invention relates to a road defect detection device, a method, and a computer program for executing the same, which can detect defects occurring on a road using a CNN (Convolutional Neural Network)-based artificial intelligence model. Background Technology

[0002] Recently, potholes on roads have emerged as a social issue. Potholes are deep depressions in the road surface that significantly hinder vehicle traffic and threaten safety. They are caused by various factors, including climate change, poor quality asphalt concrete, and the use of de-icing agents. The problem of potholes extends beyond simple road damage to become a serious social issue directly linked to traffic safety, urgently requiring measures to address it.

[0003] The existing method for detecting road defects, including potholes, primarily relies on receiving public complaints, followed by a procedure where a 담당자 (person in charge) conducts an on-site investigation. However, this approach has limitations; it makes it difficult to respond proactively to accidents caused by road defects, consumes excessive manpower, time, and costs, and results in economic losses due to the disruption of traffic system operations.

[0004] To address these issues, some organizations are introducing technology that detects road defects by equipping vehicles with cameras. However, this method also faces practical limitations in its nationwide expansion due to issues such as the high costs associated with securing vehicles and installing cameras, the manpower burden required for operation, and the occurrence of detection blind spots.

[0005] Specifically, even if a nationwide survey is conducted using a road defect detection method based on vehicle cameras, it is impossible to resolve numerous defects simultaneously, and determining their priorities is by no means an easy task. Furthermore, taking preemptive action on road defects with low urgency lowers confidence in infrastructure and results in wasting the budget for accident compensation. Additionally, since road defect detection using vehicle cameras is applied only within the jurisdiction, regional disparities based on budget are inevitable. Prior art literature

[0006] Republic of Korea Published Patent Application No. 10-2019-0073144 (Published June 26, 2019) The problem to be solved

[0007] The present invention has been devised in consideration of the aforementioned problems, and the objective of the present invention is to provide an automated road defect detection method and device capable of saving manpower, costs, and time. Furthermore, the objective of the present invention is to provide a road defect detection device, a method, and a computer program for executing the same, which can automatically select roads requiring preemptive maintenance among the detected road defects. means of solving the problem

[0008] A road defect detection device according to the present invention comprises: a real data collection unit that collects actual CCTV data captured by a fixed CCTV; a virtual data generation unit that generates virtual CCTV data by inserting an image of a defect object into the actual CCTV data; a road defect information acquisition unit that acquires road defect information using a road defect detection model constructed by learning the actual CCTV data and the virtual CCTV data; a traffic volume information acquisition unit that acquires road traffic volume information based on the actual CCTV data; and a recovery priority determination unit that determines a recovery priority based on the road defect information and the traffic volume information.

[0009] In addition, the above road defect detection model can be constructed by learning from data obtained by preprocessing and augmenting the actual CCTV data and the virtual CCTV data.

[0010] In addition, the virtual CCTV data can be obtained by inserting an image of a defective object into the actual CCTV data, wherein at least one of the number, size, depth, shape, and angle of the defective object is different.

[0011] In addition, the traffic volume information acquisition unit performs image labeling to detect vehicles on the road, extracts a region of interest, calculates the Intersection Over Union (IOU) between the bounding box of the previous frame and the bounding box of the current frame, and excludes duplicate objects determined using the IOU from counting to acquire traffic volume information on the road.

[0012] In addition, the above defective object image may be a pot hole image.

[0013] In addition, the above recovery priority can be determined based on the priority calculated by [Equation 1] described below.

[0014] In addition, the road defect information includes at least one of the length, width, depth, shape, and number of potholes existing on the road, and the value of weight α of [Equation 1] described below is determined based on at least one of the length, width, depth, shape, and number of potholes, and the value of weight β can be determined based on the average value of at least one of the size, weight, wheel diameter, and speed of vehicles passing on the road.

[0015] In addition, the actual data collection unit can collect the actual CCTV data using an open API provided by a public institution related to road traffic.

[0016] Additionally, it may further include a communication unit that transmits at least one of the GPS coordinates of the fixed CCTV, the road defect information, the traffic volume information, and the recovery priority to an external server.

[0017] Meanwhile, a road defect detection method according to the present invention for achieving the above objective comprises: a step of collecting actual CCTV data captured by a fixed CCTV; a step of generating virtual CCTV data by inserting an image of a defect object into the actual CCTV data; a step of generating a road defect detection model by learning the actual CCTV data and the virtual CCTV data; a step of obtaining road defect information using the road defect detection model; a step of generating traffic volume information based on the actual CCTV data; and a step of determining a restoration priority based on the road defect information and the traffic volume information.

[0018] And, it may further include the step of preprocessing and augmenting the actual CCTV data and the virtual CCTV data.

[0019] In addition, the step of generating the virtual CCTV data may insert an image of a defective object into the actual CCTV data in which at least one of the number, size, depth, shape, and angle of the defective object is different.

[0020] And, further comprising the step of collecting the above-mentioned defective object image; wherein the above-mentioned defective object image may be a pot hole image.

[0021] In addition, the above recovery priority can be determined based on the priority calculated by [Mathematical Formula 1] described below.

[0022] And, the above road defect information includes at least one of the length, width, depth, shape, and number of potholes existing on the road, and the value of weight α of [Equation 1] described below is determined based on at least one of the length, width, depth, shape, and number of potholes, and the value of weight β can be determined based on the average value of at least one of the size, weight, wheel diameter, and speed of vehicles passing on the road.

[0023] In addition, the actual CCTV data mentioned above can be collected using open APIs provided by public institutions related to road traffic.

[0024] And, the method may further include the step of obtaining the GPS coordinates of the fixed CCTV installed on the road where the road defect occurred; and the step of transmitting at least one of the GPS coordinates, the road defect information, the traffic volume information, and the recovery priority to an external server.

[0025] Meanwhile, the computer program according to the present invention may be stored on a computer-readable recording medium to execute each step of the above-described road defect detection method on a computer. Effects of the invention

[0026] According to the present invention, road defects are automatically detected based on road images acquired from pre-installed traffic information CCTVs, thereby saving manpower, time, and costs, and enabling the nationwide expansion of road defect detection. Furthermore, by performing road vulnerability analysis using a CNN (Convolutional Neural Network)-based artificial intelligence model, preemptive maintenance of roads nationwide becomes possible before accidents occur. Brief explanation of the drawing

[0027] Figure 1 is a diagram comparing a road defect detection method according to the present invention with a conventional method. FIG. 2 is a block diagram of a road defect detection device according to the present invention. Figure 3 is a flowchart of a road defect detection method according to the present invention. Figure 4 is a diagram illustrating a method of configuring a data set for learning in a road defect detection method according to the present invention. Figure 5 is a diagram illustrating the method of constructing a road defect detection model in the road defect detection method according to the present invention. Figures 6 and 7 are drawings illustrating the performance of a road defect detection model in a road defect detection method according to the present invention. Figure 8 is data showing the performance of the road defect detection model of the road defect detection method according to the present invention. FIG. 9 is a diagram illustrating a vehicle tracking method of a road defect detection method according to the present invention. FIG. 10 is a diagram illustrating a method for determining the recovery priority of a road defect detection method according to the present invention. Specific details for implementing the invention

[0028] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0029] The present invention will be described in detail below with reference to the attached drawings.

[0030] Figure 1 is a diagram comparing a road defect detection method according to the present invention with a conventional method.

[0031] The left figure of Fig. 1 illustrates a conventional road defect detection method using a vehicle-mounted camera. As illustrated, the conventional method allows for the detection of potholes and the like at the location where a vehicle passes. However, since the conventional method identifies potholes on the road only at the moment a vehicle passes, periodic vehicle operation is essential for continuous road defect monitoring. This increases manpower, time, and costs, resulting in the consumption of a significant portion of the national budget.

[0032] The right figure of FIG. 1 illustrates a road defect detection method using a fixed CCTV camera according to the present invention. Unlike conventional methods, using a fixed CCTV camera enables continuous monitoring of road defects without vehicle operation, thereby preventing the waste of manpower, time, and costs. Furthermore, the road defect detection method according to the present invention not only detects road defects such as potholes but also accurately identifies the volume of vehicle traffic in real time in the area where the fixed CCTV camera is installed. By using a fixed CCTV camera, the volume of vehicle traffic can be identified on any road nationwide, including expressways and national roads, at the interval of CCTV installation. Therefore, by determining the priority of road repair targets based on real-time road defect information and vehicle traffic volume information obtained as in the present invention, significant reductions in manpower and costs in road maintenance can be achieved.

[0033] In addition, in the case of conventional methods, it is difficult to determine the exact traffic volume at the location of the pothole because a vehicle equipped with a camera is used. In the case of highways, while it is possible to estimate the traffic volume at a given point to some extent based on the number of vehicles passing through toll gates, it is virtually impossible to obtain the exact traffic volume at a specific point because vehicles are dispersed or concentrated through junctions or intersections. In particular, it becomes even more difficult to accurately determine the traffic volume at a specific point on provincial roads rather than highways. Therefore, the technical effects of the present invention mentioned above cannot be expected with the conventional method using a vehicle equipped with a camera.

[0034] FIG. 2 is a block diagram of a road defect detection device according to the present invention.

[0035] The road defect detection device (100) according to the present invention includes a control unit (110), an actual data collection unit (120), a virtual data generation unit (130), a road defect information acquisition unit (140), a traffic volume information acquisition unit (150), a recovery priority determination unit (160), a communication unit (170), an input unit (180), and a storage unit (190).

[0036] Two or more of these configurations may be implemented as a single composite configuration to perform multiple functions integrally, or a single configuration may be separated into two or more configurations to allow each configuration to perform subdivided functions. Additionally, other configurations not shown in FIG. 2 (e.g., I / O ports, displays, etc.) may be additionally included as needed.

[0037] The control unit (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device.

[0038] The control unit (110) can read a computer program stored in the storage unit (190) and enable the road defect information acquisition unit (140) and / or traffic volume information acquisition unit (150) to perform data processing for machine learning to generate an AI model.

[0039] In another embodiment, the control unit (110) can perform operations for neural network learning on its own. The control unit (110) can perform operations for neural network learning, such as processing input data for learning in deep learning, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation.

[0040] Additionally, at least one of the CPU, GPGPU, and TPU of the control unit (110) can process the learning of the network function. For example, the CPU and GPGPU can together process the learning of the network function and data classification using the network function. In one embodiment, processors of a plurality of computing devices can be used together to process the learning of the network function and data classification using the network function. The network function can be used interchangeably with an artificial neural network and a neural network. The network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of one or more neural networks. A model (inference model) may include a network function. The model may include one or more network functions, in which case the output of the model may be an ensemble of the outputs of one or more network functions.

[0041] A computer program for detecting road defects according to the present invention includes one or more instructions that cause a control unit (110) to perform a method according to various embodiments of the present invention when loaded from a storage unit (190). The control unit (110) can perform a method according to various embodiments of the present invention by executing one or more instructions.

[0042] The actual data collection unit (120) collects actual CCTV data from a fixed CCTV installed on the road. The fixed CCTV may be installed along the roadside, or fixed to a structure of a toll gate or an installation crossing the road. The CCTV can be implemented as a camera of various shapes, functions, and types, such as a PTZ (pan / tilt / zoom) camera, a box camera, a speed dome camera, a bi-spectrum camera, a tri-spectrum camera, a bullet camera, a panoramic camera, an infrared (IR) camera, and is not limited to a specific camera.

[0043] The actual data collection unit (120) may receive images directly from a fixed CCTV, but may also collect actual CCTV data using an open API provided by a road traffic related database. In the following description, "image" can be interpreted to encompass both still images and video / video images.

[0044] The virtual data generation unit (130) generates virtual CCTV data by inserting a defective object image into the actual CCTV data collected by the actual data collection unit (120). To do this, the virtual data generation unit (130) can collect defective object images from various sources. Various sources may be image provision platforms, databases, etc. that are accessible on the internet. At this time, the defective object image may be a pothole image, but is not limited thereto, and may be an image of a hazardous object on the road created by damage, erosion, cracks, ruptures, uplifts, etc. of the road, such as a crack.

[0045] Specifically, the virtual data generation unit (130) receives actual CCTV data and a defective object image, and generates a road defect simulation scenario by inserting the defective object image into the actual CCTV data. That is, virtual CCTV data is obtained by inserting a defective object image into the actual CCTV data in which at least one of the number, size, depth, shape, and angle of the defective object is different. Various image synthesis techniques can be used for inserting the defective object image into the actual CCTV image. For example, the defective object image can be overlaid on the actual CCTV data using a layer-based editing tool or synthesized using a masking technique. In addition, the defective object image and the actual CCTV data can be synthesized using deep learning-based image synthesis techniques such as GAN (Generative Adversarial network) and NeRF (Neural Radiance Fields).

[0046] The road defect information acquisition unit (140) acquires road defect information using a road defect detection model. The road defect information may be information regarding the presence of defects such as potholes on the road, the number, size, depth, shape, angle, etc. of potholes.

[0047] A road defect detection model can be constructed by learning actual CCTV data collected from an actual data collection unit (120) and virtual CCTV data generated from a virtual data generation unit (130). At this time, the actual CCTV data and virtual CCTV data can be preprocessed and augmented before learning to increase learning efficiency. The construction of the road defect detection model will be explained in more detail below.

[0048] The traffic volume information acquisition unit (150) acquires traffic volume information using actual CCTV data acquired from a fixed CCTV. The traffic volume information is information regarding the volume of vehicle traffic on the road and may include periodic traffic volume (hourly, daily, weekly, monthly, and yearly). In the present invention, since image data on the road is extracted using a fixed CCTV, real-time traffic volume monitoring is possible on the road at the location where the fixed CCTV is installed.

[0049] Specifically, the traffic volume information acquisition unit (150) performs image labeling for detecting vehicles on the road, extracts a Region of Interest (ROI), calculates the Intersection Over Union (IOU) between the bounding box of the previous frame and the bounding box of the current frame, and obtains traffic volume information by excluding duplicate objects determined using the IOU from counting. This will be explained in detail below.

[0050] The restoration priority determination unit (160) receives road defect information and traffic volume information from the road defect information acquisition unit (140) and the traffic volume information acquisition unit (150), respectively. Then, it determines the restoration priority based on the road defect information and traffic volume information. The restoration priority refers to the urgency of road restoration within a unit area, and the unit area may be a regional scope such as the entire country, a province, a city, or a county.

[0051] The recovery priority determination unit (160) can calculate the priority in various ways and determine the recovery priority based thereon. Various ways may include, for example, using a specific formula, using a learned model, or applying the judgment results of an expert (or organization).

[0052] The communication unit (170) has the function of transmitting and receiving data such as actual CCTV data, GPS coordinates of a fixed CCTV, road defect information, traffic volume information, and recovery priority. The communication unit (170) can transmit and receive data to and from external devices such as CCTVs, external computing devices, servers, etc., via a wired communication network or a wireless communication network. That is, when the object defect detection device (100) according to the present invention is installed inside a CCTV or in close proximity to a CCTV, it can be connected to the CCTV via a wired communication network or a short-range wireless communication network. Conversely, when the object defect detection device (100) according to the present invention and the CCTV are placed at a distance from each other, they can be connected to each other via a wireless communication network.

[0053] Wired communication networks may include, but are not limited to, wired communication networks using wired cables such as HDMI (high-definition multimedia interface) and USB (universal serial bus). Wireless communication networks may include short-range wireless communication networks such as Z-wave, Zigbee, Wi-Fi, Bluetooth, LTE-M, LoRa (long range), Narrowband Internet of Things (NB-IoT), and Infrared Data Association (IrDA). In addition, wireless communication networks may include, but are not limited to, 2G mobile communication networks such as Wireless LAN (WLAN), WiBro (Wireless Broadband), Wi-Fi (wireless fidelity), WiMax (world interoperability for microwave access), GSM (global system for mobile communication) or CDMA (code division multiple access), 3G mobile communication networks such as WCDMA (wideband code division multiple access) or CDMA2000, 3.5G mobile communication networks such as HSDPA (high speed downlink packet access) or HSUPA (high speed uplink packet access), and 4G, 5G, and 6G mobile communication networks such as LTE (long term evolution) networks or LTE-Advanced networks.

[0054] The input unit (180) is configured for inputting user commands to control road defect detection, inputting various parameters for machine learning, inputting user values ​​for determining road defect and traffic volume information and recovery priority, etc.

[0055] The storage unit (190) stores various information, data, and programs necessary for executing the present invention, including information on multiple CCTVs installed on a road (location information, model information, etc.), actual CCTV data, virtual CCTV data, road defect information, traffic volume information, and recovery priority information, as well as a pre-learned road defect information acquisition model, a pre-learned traffic volume information acquisition model, and a computer program that executes a road defect detection method. The storage unit (190) may be implemented as a storage medium of at least one type among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk, but is not limited thereto.

[0056] Figure 3 is a flowchart of a road defect detection method according to the present invention.

[0057] First, actual CCTV data captured by a fixed CCTV is collected (S210). The actual CCTV data can be collected using an open API provided by a public institution related to road traffic. In another embodiment, the actual CCTV data can be collected directly from the fixed CCTV via wireless communication.

[0058] Subsequently, virtual CCTV data is generated by inserting a defective object image into the collected actual CCTV data (S220). To generate virtual CCTV data, a step of collecting a defective object image may be further included, and the defective object image may be a pothole image. In the step of generating virtual CCTV data, the number of data is amplified by inserting the defective object image, which differs in at least one of the number, size, depth, shape, and angle of the defective object, into the actual CCTV data.

[0059] Subsequently, a road defect detection model is created by training actual CCTV data and virtual CCTV data (S230). At this time, to optimize and improve learning efficiency, additional steps of preprocessing and augmenting actual CCTV data and virtual CCTV data may be performed.

[0060] Afterwards, road defect information is obtained using a road defect detection model (S240), and traffic volume information is obtained based on actual CCTV data (S250).

[0061] Finally, a restoration priority is determined based on road defect information and traffic volume information (S260).

[0062] Meanwhile, although not illustrated in FIG. 3, the road defect detection method according to the present invention may further include the step of obtaining the GPS coordinates of the fixed CCTV installed on the road where the road defect occurred, and transmitting at least one of the said GPS coordinates, road defect information, traffic volume information, and recovery priority to an external server.

[0063] Below, each step of the road defect detection method described above is explained in detail along with drawings.

[0064] 1. Generation of road defect simulation scenarios and configuration of datasets (S210 and S220)

[0065] Figure 4 is a diagram illustrating a method of configuring a data set for learning in a road defect detection method according to the present invention.

[0066] The present invention utilizes a CNN (Convolutional Neural Network)-based artificial intelligence model for road defect detection. Therefore, a sufficient amount of data set is required for training the artificial intelligence model.

[0067] To construct a dataset, the present invention collects actual CCTV data captured from CCTVs. The actual CCTV data can be collected using a CCTV open API (Application Programming Interface). For example, as actual CCTV data, real-time images (videos) captured by CCTVs installed on a national road network can be used through an API provided by the National Traffic Information Center, etc.

[0068] In another embodiment, actual CCTV data may be received directly via wireless communication from CCTVs installed in the road network.

[0069] Since actual CCTV data alone is insufficient for training an artificial intelligence model, it is necessary to amplify the amount of training data. In particular, the proportion of actual CCTV data containing defective objects is significantly low. To address this, the present invention generates virtual data necessary for training. First, multiple images of defective objects are collected from accessible sites. The multiple images of defective objects may be images of potholes. That is, images of potholes with varying sizes, shapes, lengths, widths, depths, angles, etc., may be collected.

[0070] When a sufficient amount of images of defective objects are collected, multiple virtual data are generated by combining them with actual CCTV images. For example, a first virtual data is generated by synthesizing a pothole image onto a road area within an actual CCTV image. Then, a second virtual data is generated by synthesizing a pothole image used above onto a road area within another actual CCTV image after modifying its angle. Again, a third virtual data is generated by synthesizing two pothole images different from those used above onto a road area within yet another actual CCTV image. In this way, the quantity (number) of road images containing potholes can be amplified to approximately 2,000 or more, and the accuracy and reliability of the artificial intelligence model can be improved.

[0071] 2. Establishment of Road Defect Detection Model (S230)

[0072] If a sufficient amount of data set for learning is secured, a road defect detection model is constructed based on it. Figure 5 is a diagram illustrating the method of constructing a road defect detection model in the road defect detection method according to the present invention.

[0073] Real and virtual CCTV data undergo an image labeling process to improve object detection rates. Image labeling generally includes various methods such as image classification, object detection, semantic segmentation, keypoint labeling, and image transcription, and the labeling type can be selected depending on the application field of artificial intelligence.

[0074] Image labeling for actual CCTV data and virtual CCTV data can be performed through automated AI solutions or based on user selection signals through the input.

[0075] Once image labeling is performed, augmentation can be carried out on actual and virtual CCTV data. This involves performing operations such as vertical and horizontal movement, rotation, color conversion, noise insertion, cropping, enlargement, and reduction on the actual and virtual CCTV images to amplify the number of visually distinct images.

[0076] Once image labeling and data augmentation are performed on actual and virtual CCTV data, artificial intelligence learning is carried out. While road defect detection models can be built using various object detection algorithms, using the YOLO (You Only Look Once) algorithm allows for high performance and fast prediction speeds.

[0077] The YOLO algorithm can detect objects at very high speeds by processing the object detection task using a single neural network. The YOLO algorithm divides the original images of a dataset, which includes both real and virtual CCTV data, into grids of equal size and detects objects within each grid cell. It subdivides the image into multiple grids and generates information about the detected objects for each grid cell. Subsequently, the bounding box of the object's location, the object's class, and a confidence score are output. Here, the object location information is expressed as a relative position within the grid cell and is converted to the full image. The object class indicates the type of object recognized within the image, while the confidence score represents the probability of the object's actual existence. Because the YOLO algorithm processes the image only once, it is very fast and suitable for real-time pothole detection. Furthermore, it operates at multiple scales to detect both small objects (potholes) and large objects (potholes, vehicles, structures), and can acquire information regarding the width, depth, length, shape, form, and angle of potholes.

[0078] Meanwhile, road defect detection models can utilize versions of the YOLO algorithm such as YOLOv8 or higher. YOLOv8 incorporates a new architecture and training techniques, enabling it to provide high detection performance even in complex scenes with small objects like potholes and high traffic volume.

[0079] FIGS. 6 to 8 are drawings illustrating the performance of a road defect detection model in a road defect detection method according to the present invention.

[0080] The graph on the left in Figure 6 is the Precision-Recall Curve when only raw data obtained through the Open API is used, and it can be seen that the mean Average Precision (mAP) is only 0.440 when only raw data is used. In addition, the graph curve shows a tendency for precision to decrease as it gradually descends.

[0081] The graph on the right in Figure 6 shows the precision-recall curve after applying the data augmentation technique, and it can be seen that the average precision has been significantly improved to 0.770. In addition, compared to the case where only raw data was used, the precision-recall curve was maintained in a much higher region, demonstrating better model performance.

[0082] The left figure of Fig. 7 shows the result of a model trained using only raw data obtained through an open API detecting a pothole. Although a pothole was detected by a bounding box, the confidence score was only 0.3, which is a very low value.

[0083] The right figure of Figure 7 shows the results detected by the model after applying data augmentation. It can be seen that the pothole detection performance has improved and the confidence level has increased to 0.7–0.9. In addition, it can be seen that the model’s recognition ability has improved, capable of detecting even vehicles on the road (car 0.9).

[0084] Figure 8 shows the test results of defect detection performed in different road environments. The original image is an actual CCTV image, and the output image represents the results detected by the model. The detection probability refers to the probability value of the model detecting the size, location, depth, length, shape, etc., of a pothole, and the results of the five tests showed a detection probability of approximately 86.7%.

[0085] Intersection over Union (IoU) indicates how accurately the model predicted the size, location, depth, length, shape, etc. of potholes, and it was confirmed that it had an average accuracy of 62.3%.

[0086] 3. Obtain road defect information (S240)

[0087] The defect object detection method according to the present invention increases the diversity of training data by augmenting actual CCTV data and virtual CCTV data. The defect object detection model built by training this increases the detection probability regarding the size, location, depth, length, shape, etc. of potholes (increase by 88.2%) and improves detection accuracy (increase by 2.6 times), as shown in FIGS. 6 to 8, thereby reliably detecting potholes in various environments and obtaining road defect information.

[0088] 4. Obtain traffic volume information (S250)

[0089] The road defect detection method according to the present invention acquires traffic volume information along with road defect information. At this time, the order of acquisition of each piece of information is not important. That is, the acquisition of road defect information may precede the acquisition of traffic volume information, or conversely, the acquisition of traffic volume information may precede the acquisition of road defect information, and the road defect information and traffic volume information may be acquired simultaneously.

[0090] FIG. 9 is a diagram illustrating a vehicle tracking method of a road defect detection method according to the present invention.

[0091] First, image labeling is performed to detect vehicles on the road, and then a Region of Interest (ROI) is extracted. Here, the ROI may be a vehicle. The number of ROIs captured per hour in the corresponding CCTV footage may represent the traffic volume of the road where the CCTV footage was recorded.

[0092] At this time, the Intersection Over Union (IOU) between the bounding box of the previous frame and the bounding box of the current frame of the CCTV video is calculated, and duplicate objects identified using the IOU can be excluded from the count. This is intended to prevent duplicate counting of vehicles. The traffic volume information obtained by the above method can be stored together with location information or transmitted to an external server.

[0093] Meanwhile, in other embodiments, traffic volume information may be received through an open API. However, since the traffic volume information received through the open API is calculated based on highway toll gates, it is difficult to accurately determine the traffic volume at a specific location. In contrast, the present invention enables the acquisition of accurate traffic volume information anywhere in the country, excluding highways, as long as CCTVs can be installed.

[0094] 5. Determination of recovery priority (S260)

[0095] FIG. 10 is a diagram illustrating a method for determining the recovery priority of a road defect detection method according to the present invention.

[0096] GPS location information of CCTVs, road defect information such as the number, depth, width, length, and shape of potholes, and traffic volume information are used to determine restoration priorities through various methods.

[0097] In one embodiment, the determination of the restoration priority can be performed through a pre-established ranking model. The ranking model takes the road defect index and the traffic volume index as input variables and the road restoration priority as an output variable. In this case, factors such as climate, road type, accident rate, type, size, weight, wheel diameter, and speed of passing vehicles may be added as input variables.

[0098] The construction of a ranking model can be achieved by collecting road defect data, traffic volume data, and additional variables, cleaning and normalizing the data, and then performing training using a supervised learning-based model of regression or classification.

[0099] Once the ranking model is established, road defect information and traffic volume information are converted into a road defect index and a traffic volume index, respectively, and each converted index is input into the ranking model to predict a score of 0 to 1 or classify into grades 1 to 5.

[0100] In another embodiment, the recovery priority determination can be determined based on the priority calculated based on [Equation 1] below.

[0101]

[0102] Here, P R is the priority, α and β are weights (constant values), I D is the road defect index, I T represents the traffic volume index. The road defect index and the traffic volume index can be obtained using road defect information and traffic volume information, respectively, and the weights can be set automatically or by user input.

[0103] In particular, the value of the weight α for the road defect index can be determined based on at least one of the length, width, depth, shape, and number of potholes. For example, the value of the weight α can be set to increase in proportion to the difference between the length, width, depth, and number of potholes and a reference value.

[0104] In another embodiment, the value of the weight β for the traffic volume index may be determined based on at least one of the type, size, weight, wheel diameter, and speed of vehicles passing through a specific road. For example, even if the traffic volume of a specific road is low, if most of the vehicles passing through the road are trucks weighing 1 ton or more, the value of the weight β may be set high to increase the recovery priority.

[0105] In particular, the length, width, and depth of a pothole may have different levels of risk depending on the diameter of the wheel passing over it. By adjusting the values ​​of weights α and β to reflect this, the present invention can determine the recovery priority more reliably.

[0106] The road defect detection method according to the present invention may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or through an application store, or directly distributed between two user devices (e.g., smartphones) (e.g., download or upload). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0107] Meanwhile, the road defect detection method according to the present invention may be performed by a computer, a computing device, etc. Additionally, the road defect detection method according to the present invention may be implemented as a computer program. That is, the computer program according to the present invention is intended to execute various embodiments of the above-described method on a computer and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0108] Furthermore, although the invention has been described above with reference to embodiments, this is merely illustrative and does not limit the invention. Those skilled in the art will understand that various modifications and applications not exemplified above are possible within the scope of the essential characteristics of the embodiments. For example, each component specifically shown in the embodiments may be modified and implemented. Differences related to such modifications and applications should be interpreted as being included within the scope of the invention as defined in the appended claims. Explanation of the symbols

[0109] 100: Road defect detection device 110: Control unit 120: Actual Data Collection Unit 130: Virtual Data Generation Unit 140: Road defect information acquisition unit 150: Traffic volume information acquisition unit 160: Recovery Priority Determination Unit 170: Communications Department 180: Input section 190: Storage section

Claims

Claim 1 A road defect detection device comprising: a real data collection unit for collecting actual CCTV data captured from a fixed CCTV; a virtual data generation unit for generating virtual CCTV data by inserting an image of a defect object into the actual CCTV data; a road defect information acquisition unit for acquiring road defect information using a road defect detection model built by learning the actual CCTV data and the virtual CCTV data; a traffic volume information acquisition unit for acquiring road traffic volume information based on the actual CCTV data; and a recovery priority determination unit for determining a recovery priority based on the road defect information and the traffic volume information. Claim 2 A road defect detection device according to claim 1, wherein the road defect detection model is constructed by learning data obtained by preprocessing and augmenting the actual CCTV data and the virtual CCTV data. Claim 3 A road defect detection device according to claim 1, wherein the virtual CCTV data is obtained by inserting a defect object image, in which at least one of the number, size, depth, shape, and angle of the defect object is different, into the actual CCTV data. Claim 4 A road defect detection device according to claim 1, wherein the traffic volume information acquisition unit performs image labeling to detect vehicles on the road, extracts a region of interest, calculates the Intersection Over Union (IOU) between the bounding box of the previous frame and the bounding box of the current frame, and acquires traffic volume information on the road by excluding duplicate objects determined using the IOU from counting. Claim 5 A road defect detection device according to claim 1, wherein the defect object image is a pothole image. Claim 6 A road defect detection device according to claim 1, wherein the above-mentioned recovery priority is determined based on the priority calculated by the following formula. (P R : Priority, α, β: Weights (constant values), I D : Road Defect Index, I T : Traffic volume index) Claim 7 A road defect detection device according to claim 6, wherein the road defect information includes at least one of the length, width, depth, shape, and number of potholes existing on the road, and the value of the weight α is determined based on at least one of the length, width, depth, shape, and number of potholes. Claim 8 A road defect detection device according to claim 6, wherein the value of the weighting factor β is determined based on the average value of at least one of the size, weight, wheel diameter, and speed of a vehicle passing on the road. Claim 9 A road defect detection device according to claim 1, wherein the actual data collection unit collects the actual CCTV data using an open API provided by a public institution related to road traffic. Claim 10 A road defect detection device according to claim 1, further comprising a communication unit that transmits at least one of the GPS coordinates of the fixed CCTV, the road defect information, the traffic volume information, and the recovery priority to an external server. Claim 11 A road defect detection method comprising: a step of collecting actual CCTV data captured by a fixed CCTV; a step of generating virtual CCTV data by inserting an image of a defect object into the actual CCTV data; a step of generating a road defect detection model by learning the actual CCTV data and the virtual CCTV data; a step of obtaining road defect information using the road defect detection model; a step of generating traffic volume information based on the actual CCTV data; and a step of determining a restoration priority based on the road defect information and the traffic volume information. Claim 12 A road defect detection method according to claim 11, further comprising the step of preprocessing and augmenting the actual CCTV data and the virtual CCTV data. Claim 13 A road defect detection method according to claim 11, wherein the step of generating the virtual CCTV data involves inserting an image of a defect object, in which at least one of the number, size, depth, shape, and angle of the defect object is different, into the actual CCTV data. Claim 14 A road defect detection method according to claim 11, further comprising the step of collecting the defect object image; wherein the defect object image is a pothole image. Claim 15 A road defect detection method according to claim 11, wherein the above-mentioned recovery priority is determined based on the priority calculated by the following formula. (P R : Priority, α, β: Weights (constant values), I D : Road Defect Index, I T : Traffic volume index) Claim 16 A road defect detection method according to claim 15, wherein the road defect information includes at least one of the length, width, depth, shape, and number of potholes existing on the road, and the value of the weight α is determined based on at least one of the length, width, depth, shape, and number of potholes. Claim 17 A road defect detection method according to claim 15, wherein the value of the weighting factor β is determined based on the average value of at least one of the size, weight, wheel diameter, and speed of a vehicle passing over the road. Claim 18 In paragraph 11, the above-mentioned actual CCTV data is collected using an open API provided by a public institution related to road traffic, a road defect detection method. Claim 19 A road defect detection method according to claim 11, further comprising: a step of obtaining GPS coordinates of a fixed CCTV installed on the road where the road defect occurred; and a step of transmitting at least one of the GPS coordinates, the road defect information, the traffic volume information, and the recovery priority to an external server. Claim 20 A computer program stored on a computer-readable recording medium to execute each step of the method described in any one of claims 11 through 19 on a computer.

Images