Method for detecting damaged road facilities and road facility management system using same

Abstract

The present invention relates to a method for detecting damaged road facilities. According to one embodiment of the present invention, the method comprises the steps of: obtaining a road image; detecting an object area of interest from the road image; identifying a degree of damage to the object of interest from the detected object area; and displaying the degree of damage to the object of interest.

Description

Method for detecting damaged road facilities and a road facility management system using the same

[0001] The present invention relates to a method for detecting damage to road markings and road facilities, a device equipped therewith, and a management system using the same.

[0002] Road facilities include road markings and traffic facilities.

[0003] Road markings are lines, symbols, and letters that appear on road surfaces, such as roads and parking lots. They serve to guide and instruct drivers and pedestrians on road usage methods, precautions, and rules. Examples include crosswalks, lane markings and lane lines, stop lines, and no-passing and restricted zone markings.

[0004] Road traffic facilities include traffic lights, bollards, guide signs, and guardrails. Along with road markings, they refer to all facilities and devices used to support and manage a smooth traffic system. Road markings and traffic facilities enhance road safety, smooth traffic flow, and enhance the efficiency of the transportation system.

[0005] If road markings providing road information become blurred or road facilities are damaged, preventing road users from receiving accurate guidance, this can impair the judgment of both drivers and pedestrians, potentially leading to misdirection or even accidents. Therefore, maintaining and repairing road markings and facilities is crucial for the safety of both drivers and pedestrians. Therefore, ongoing monitoring and maintenance, along with prompt repairs, are essential.

[0006] However, currently, the only methods for identifying areas requiring preventive road maintenance are city and county governments, such as suggestions from private users, patrols, and complaints. This means that continuous monitoring is necessary for direct human detection. This wastes significant manpower and hinders prompt repairs when problems arise, creating a need for more efficient management methods.

[0007] The purpose of the present invention is to address the aforementioned problems by automating the identification and maintenance of damage to road markings and facilities, previously requiring manual reporting and patrol. The present invention proposes a more efficient road facility management method, device, and system by accurately and quickly assessing the extent of road facility damage and determining the timing of repairs through various video and image data.

[0008] However, the problems to be solved in the present invention are not limited to the above-described contents.

[0009] In order to solve the above-described problem, a method for detecting damaged road facilities proposed in one aspect of the present invention comprises the steps of: acquiring a road image; detecting an object area of ​​interest from the road image; determining the degree of damage to the object of interest from the detected object area; and displaying the degree of damage to the object of interest.

[0010] According to one embodiment, the step of acquiring the road image may be acquiring the image from one or more of satellite photos, CCTV, and black boxes.

[0011] According to one embodiment, the step of acquiring the road image may include acquiring a plurality of images captured by different means or locations.

[0012] According to one embodiment, the step of determining the degree of damage to the object of interest may include comparing the degrees of damage to the same object of interest from the plurality of images to determine the degree of damage.

[0013]

[0014] According to one embodiment, the method further comprises: a step of determining the degree of damage; and a step of displaying the degree of damage; and a step of storing the degree of damage of the object of interest between the steps; and the step of displaying the degree of damage may display the address data of the object of interest together with the degree of damage of the object of interest that was last stored.

[0015] According to one embodiment, the step of determining the degree of damage may utilize one or more pieces of information from among color, inclination, and appearance of the object of interest.

[0016] In one embodiment, the step of indicating the degree of damage may include indicating the degree of damage of the object of interest on the digital twinned road image.

[0017] In one embodiment, the digital twinned road image may be displayed including historical information regarding the maintenance of the object of interest.

[0018]

[0019] In another aspect of the present invention, a detection device for damaged road facilities is proposed, comprising: an acquisition unit for obtaining a road image in which road facilities are photographed; a processor unit for detecting an object related to the road facilities and determining the degree of damage using the image obtained from the acquisition unit; and a display unit for receiving information from the processor unit and displaying the degree of damage to the damaged road facilities; and using a detection method according to one embodiment of the present invention.

[0020]

[0021] In another aspect of the present invention, a system for detecting and maintaining damaged road facilities is proposed, including: an acquisition unit for obtaining a road image in which road facilities are photographed; a processor unit for detecting an object related to the road facility and determining the degree of damage using the image obtained from the acquisition unit; and a display unit for receiving information from the processor unit and displaying the degree of damage to the damaged road facility; and using a detection method for road facilities according to an embodiment of the present invention, the display unit displays road facilities requiring maintenance by grade on a digital twin virtual road image, and a user determines a maintenance cycle of the road facility based on the information displayed on the display unit.

[0022] According to one embodiment of the present invention, in a conventional method of managing road facilities by directly reporting or monitoring by a person, there is an effect of reducing manpower consumption by automatically detecting damaged areas in various video data regardless of the type of video or image.

[0023] Additionally, by providing information on the current state of damage, users can know the current status of repairs, and users who need road repairs can select areas where repairs should be prioritized based on the urgent need for repairs.

[0024] However, the effects of the present invention are not limited to the effects described above, but include all effects naturally implemented due to the various configurations proposed in the present invention.

[0025] FIG. 1 is a flowchart showing each step of a method for detecting damaged road facilities according to one embodiment of the present invention.

[0026] FIG. 2 is a structural diagram showing the relationship between each component of a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention.

[0027] FIG. 3 is a structural diagram showing an example of a road facility requiring maintenance displayed on a display unit of a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention.

[0028] FIGS. 4 and 5 are images of information on damaged road facilities displayed on a display unit of a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention.

[0029] FIGS. 6 and 7 are images showing that the accuracy values ​​for detecting an object of interest and determining the degree of damage vary depending on how data is trained in different ways in a method for detecting damaged road facilities according to one embodiment of the present invention.

[0030]

[0031] The embodiments of the present invention are provided for the purpose of illustrating the technical concept of the present invention. The scope of the rights of the present invention is not limited to the embodiments presented below or the specific descriptions of these embodiments.

[0032] All technical and scientific terms used in this invention, unless otherwise defined, have the meanings commonly understood by those skilled in the art to which this invention pertains. All terms used in this invention have been selected for the purpose of more clearly explaining the invention and are not intended to limit the scope of the rights provided for in this invention.

[0033] Expressions such as “comprising,” “having,” and the like used in the present invention should be understood as open-ended terms that imply the possibility of including other embodiments, unless otherwise stated in the phrase or sentence in which the expression is included.

[0034] The singular expressions described in the present invention may include plural meanings unless otherwise stated, and the same applies to the singular expressions described in the claims.

[0035] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. Furthermore, in the description of the embodiments below, duplicate descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, this does not mean that such components are not included in any embodiment.

[0036] Previously, maintenance of pavement markings and road facilities was conducted through monitoring, either through user requests for repairs or through direct site visits by managers. Without these requests, repairs to damaged areas were sometimes delayed. Furthermore, monitoring through manager visits required significant manpower, resulting in significant time-consuming maintenance of pavement markings and road facilities.

[0037] However, a method for detecting damage to road markings and road facilities according to one embodiment of the present invention, and a device including the same, can detect and repair damaged areas early, even without a user requesting repairs. Furthermore, monitoring by an administrator is not required, and using one embodiment of the present invention, an administrator can quickly and easily detect areas requiring repairs and determine the timing of maintenance.

[0038] FIG. 1 is a flowchart illustrating each step of a method for detecting damaged road facilities according to one embodiment of the present invention. Hereinafter, with reference to FIG. 1, the configuration of each step of the method for detecting damaged road facilities proposed in one embodiment of the present invention will be described in detail.

[0039] In order to solve the above-described problem, a method for detecting damaged road facilities proposed in one aspect of the present invention includes the steps of acquiring a road image (S10); detecting an object area of ​​interest from the road image (S20); determining the degree of damage to the object of interest from the detected object area (S30); and displaying the degree of damage to the object of interest (S40).

[0040] In the present invention, the object of interest may refer to one or more road facilities. The road facilities may include 2D signs such as lane markings or direction indicators, as well as 3D structures such as traffic lights, bollards, guide signs, and guardrails. The object of interest may be one type of object, or in another example, may include multiple types of objects.

[0041] The object of interest may be a plurality of types of objects. In this case, the step of detecting the area of ​​interest for the object of interest may include detecting each object of interest separately. The step of detecting the area of ​​interest for the object of interest may first perform a task of classifying a group of object of interest candidates from the road image based on information input in advance. In this case, the group of object of interest candidates may be expressed as a region. The area of ​​interest for the object of interest may be further specified as multiple road images are input.

[0042] According to one embodiment, the step of acquiring the road image may be acquiring the image from one or more of satellite photos, CCTV, and black boxes.

[0043] The above road image may be a road image containing various objects of interest acquired through various means. The objects of interest may be multiple objects of different types contained within the same image. For example, the first object of interest may be a traffic light, and the second object of interest may be a bollard.

[0044] In the past, the inventors conducted research on detecting objects of interest and assessing the extent of damage based on aerial photography. However, this research proved insufficient for assessing the condition of three-dimensional structures such as bollards and traffic signs. Therefore, the present invention does not specifically limit the method of acquiring road images, but encompasses the concept of acquiring all types of images suitable for 3D structure recognition and damage detection, including not only satellite photos but also ground-based images such as CCTV and black boxes.

[0045] At this time, the step of detecting the region of interest may include a process of detecting a second object of interest after detecting the first object of interest. At this time, a preprocessing task of first selecting a group of object candidates from the entire image may be performed, and then a process of detecting the first object of interest may be performed. At this time, in the process of detecting the first object of interest, objects in the image other than the objects selected as the first object of interest from among the obtained group of object candidates may be classified as reserved objects. Thereafter, in the process of detecting the second object of interest, a process of determining the second object of interest from among the reserved objects may be performed, rather than extracting objects again from the entire image. Thereafter, the regions of interest may be detected from each of the first object of interest and the second object of interest.

[0046] The above road image may, in one example, be obtained for the purpose of assessing the extent of damage to road facilities. However, in other examples, it may be an image obtained for other purposes, such as a CCTV for crime prevention, a black box for traffic accident monitoring, or an image submitted by a user to report damage. The image submitted by the user may be provided by an object capable of taking pictures, such as a smartphone or digital camera.

[0047] In the present invention, the road image is not limited to a satellite photograph taken from a particularly high altitude, and may be an image included in an image taken from a similar height to or even below a road facility.

[0048] One or more of the steps of detecting the above-mentioned area of ​​interest and the steps of determining the degree of damage may be performed based on data learned in advance using artificial intelligence technology.

[0049] That is, a step of learning data in advance based on multiple images may be included at least before the step of detecting the object area of ​​interest and the step of determining the degree of damage.

[0050] The step of learning the above data may include labeling various information for the purpose of detecting object regions and determining the degree of damage. The present invention does not specifically limit the algorithm or method used for the learning. However, as an example, the pre-learning step may be performed by injecting image information on various road facilities using one or more object detection models such as the YOLO model, faster R-CNN, and SSD. In another example, the learning step may be omitted, in which case a step of detecting object regions of interest or determining the degree of damage may be performed using an image processing method.

[0051] At this time, the step of detecting the object area of ​​interest and / or the step of determining the degree of damage may be to detect the object area of ​​interest or determine the degree of damage by comparing the image obtained based on previously learned data. The artificial intelligence technology may relate to an image processing method.

[0052] The steps of detecting the region of interest and determining the extent of damage may be performed simultaneously in one example, but may be performed sequentially in another example. That is, in one example, the steps may be performed to detect and identify damaged objects of interest at once from the beginning. In another example, the detection of all objects of interest, such as bollards and traffic lights, may be performed first, and each may be classified separately from the image. Then, the task of identifying damaged objects of interest for bollards may be performed, and the task of identifying damaged objects of interest for traffic lights may be performed subsequently.

[0053]

[0054] The obtained image may be labeled and converted into data through an image learning process prior to the step of detecting the object area of ​​interest and / or the step of determining the degree of damage. The labeling may include information about the type of object, information about its shape, information about the time the video or image was captured, information about its location, and information about the degree of damage.

[0055] According to one embodiment, the step of acquiring the road image may include acquiring a plurality of images captured by different means or locations.

[0056] The above multiple images may not have been captured by the same means. That is, images captured by a black box and images captured by CCTV may each be used as road images.

[0057] According to one embodiment, the step of determining the degree of damage to the object of interest may include comparing the degrees of damage to the same object of interest from the plurality of images to determine the degree of damage.

[0058] The multiple images described above may be images with different altitudes or orientations. In this case, the degree of damage to the object of interest can be determined by combining the multiple images. To achieve this, it may be necessary to recognize the altitude or orientation at which the multiple images were captured and correct any image distortion accordingly.

[0059]

[0060] The above method may be performed multiple times for the same object of interest, with each cycle. In this case, if the area of ​​the object of interest A is initially recognized and the extent of damage is recognized through the detection of the first cycle, then in the detection of the second cycle (e.g., 3 or 6 months after the initial detection), the extent of damage to the object of interest A can be determined by partially correcting the extent of damage to the object of interest A based on the extent of damage initially identified.

[0061] For example, let's assume that in the first cycle, the damage level of object of interest A is detected as medium out of a total of 5 levels (high-medium-high-medium-medium-low-low), and then information on the maintenance history is additionally input. In this case, the information on the maintenance history may be about the history of the on-site maintenance staff repairing the damaged object of interest A. In this case, the damage level identified in the detection stage of the second cycle can be determined by placing more weight on information that the damage level in the first cycle is maintained at a similar level (medium) or is mild (medium-low or low).

[0062]

[0063] In the step of detecting the above-mentioned area of ​​interest, the acquired image may be compared and analyzed based on the image of the road and the location information of road facilities input in advance, and the shooting location, direction, altitude, etc. of the acquired image may be identified, and then the degree of distortion of the image may be identified and then the object of interest may be detected by correcting it.

[0064] According to the above embodiment, there is an advantage in that damage to road facilities, which can only be detected when viewed from a specific direction, can be easily detected through a combination of multiple different images of the object of interest. At this time, among the multiple different images of the object of interest, a first image that is captured so that the target object of interest is most easily recognized can be selected and used as a reference image of the object of interest. The first image may be different images captured by different means or from different directions for each object of interest. That is, the first image of a traffic light may be an image captured by CCTV, and the first image of a bollard may be an image captured by another black box. In this way, a primary direction in which detection and damage severity are easily possible can be determined for each object of interest. At this time, the first image may be determined as an image captured from the primary direction. At least one of the steps of detecting the area of ​​the object of interest and the steps of identifying the area of ​​damage may be determined based on the primary direction, thereby weighting the images captured from the primary direction and collecting information.

[0065] For example, the step of detecting the object area of ​​interest may apply a region detection model. The region detection model may utilize data learned through data preprocessing that takes into account differences in light according to shadows or areas, and occlusion effects due to obstacles. The data preprocessing method may include, for example, means such as brightness adjustment and blindless adjustment. Through this, errors that occur in object recognition due to shadows, differences in light, and obstacles can be significantly resolved. The present invention does not specifically limit the means used for such data preprocessing, and for example, one or more of histogram equalization, relief mapping, the retinex algorithm, and deep learning-based methods may be used.

[0066]

[0067] In one embodiment, the specific shape and degree of damage of the object of interest can be determined by basically assuming the area and degree of damage of the object of interest based on the image of the object of interest exposed to the first image of the specific object of interest, and supplementing the first image using second and third images that are different from the other first images. In this case, if a difference greater than a threshold value occurs between the area and damage information of the object of interest obtained from the first image and the average value of the area and damage information of the object of interest obtained from the other images, a process of changing the first image of the object of interest to a different image can be included.

[0068] The above object of interest may be classified into a first object and a second object. In this case, the first object may refer to an object of interest for which new information different from existing information has been obtained when compared to an image obtained in the past, and the second object may refer to an object of interest for which the same information as the existing information has been obtained. The first object may be an object for which any one of color, slope, and appearance values ​​has been changed. The first object may be a traffic light, bollard, or median strip that was not present in the area in the past but has been newly installed. The first object and the second object may be designated as the first object when the difference between the image obtained in the past and the second object exceeds a certain threshold, and as the second object when the difference is less than the threshold. The past may refer to a past time set at a certain time interval. For example, among a plurality of acquired images, an image obtained in the past 6 months ago may be compared with a current image to determine whether a threshold value is exceeded.

[0069] At this time, in the step of determining the degree of damage, a step of determining the degree of damage may be performed by selecting only the first object. In other words, the step of determining the degree of damage may not be performed for objects of interest with images identical to the previous image, thereby enabling more efficient operation than determining the degree of damage for all objects of interest.

[0070] All of the distinguished information of the first object and the second object may be stored in a storage unit and then retrieved together with information about the distinction time.

[0071] According to one embodiment, the method further comprises: a step of determining the degree of damage; and a step of displaying the degree of damage; and a step of storing the degree of damage of the object of interest between the steps; and the step of displaying the degree of damage may display the address data of the object of interest together with the degree of damage of the object of interest that was last stored.

[0072] By displaying the address data of the object of interest, one or more of the address data of the object of interest (e.g., location information) and information about maintenance (e.g., information about the construction date and construction company) can be simultaneously delivered to a user planning maintenance, along with the extent of damage to the object.

[0073] By including the above-mentioned saving step, the time and cycle of occurrence of the damage can be digitized and the time when maintenance is required can be predicted.

[0074] According to one embodiment, the step of determining the degree of damage may utilize one or more pieces of information from among color, inclination, and appearance of the object of interest.

[0075] For example, road markings can be identified through changes in color values. Falling bollards or traffic lights can be identified through their inclination, and damage to median strips can be identified through comparison with images obtained based on external information.

[0076] For example, the step of determining the degree of damage may include obtaining coordinate values ​​corresponding to an object of interest area of ​​the road surface marking, detecting an object classified as a damaged road facility from multiple images using a damage detection model, calculating coordinate values ​​corresponding to the detected object, and determining the damaged object of interest and determining the degree of damage by comparing the two coordinate values.

[0077] In one embodiment, the step of indicating the degree of damage may include indicating the degree of damage of the object of interest on the digital twinned road image.

[0078] In the above-described display step, the degree of damage to the object of interest may be displayed numerically and / or image-wise on the virtual road image. The degree of damage may be displayed in at least two levels. The image may be displayed in different colors depending on the degree of damage.

[0079] The step of displaying the above may be performed by processing an image of an object area of ​​interest obtained from the road image to create a virtual object image, which may be expressed on the digital twin road image. In this case, the virtual object image may be an image processed by combining images of collected traffic facilities from various angles.

[0080] The step of displaying the above may be to display the degree of damage to the object of interest on a plan image provided based on a geographic information system (GIS).

[0081] In one embodiment, the digital twin road image may be displayed including historical information regarding the maintenance of the object of interest.

[0082] The image displayed in the step indicated above displays the most recent maintenance information for each image of the object of interest, thereby enabling identification of the breakage cycle or maintenance cycle for each object of interest, and selection of areas requiring frequent maintenance.

[0083] The step of indicating above may select objects of interest that require priority maintenance and provide an indication of the need for maintenance for a specific object of interest. At this time, the selection of objects of interest may be performed based on the location of the object of interest. For example, if the object of interest is located in a children's protection zone or an elderly protection zone, the indication of need for maintenance may be displayed at a higher level than the degree of damage to the object of interest. For example, the indication of need for maintenance may be displayed based on the product of the damage degree of the object of interest and the importance (risk of accident) of the location where the object of interest is located, respectively, numbered.

[0084]

[0085] According to another embodiment proposed in the present invention, a computer program stored in a medium can be implemented in combination with hardware to execute the above-described method for detecting damaged road facilities.

[0086] In another aspect of the present invention, a detection device for damaged road facilities is proposed, comprising: an acquisition unit (100) for obtaining a road image in which road facilities are photographed; a processor unit (200) for detecting an object related to the road facilities and determining the degree of damage using the image obtained from the acquisition unit; and a display unit (300) for receiving information from the processor unit and displaying the degree of damage to the damaged road facilities; and using a detection method according to one embodiment of the present invention.

[0087]

[0088] FIG. 2 is a structural diagram showing the relationship between each component of a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention.

[0089] FIG. 3 is a structural diagram showing an example of a road facility requiring maintenance displayed on a display unit of a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention.

[0090] Hereinafter, a system for detecting and maintaining damaged road facilities according to another embodiment of the present invention will be described in detail with reference to FIGS. 2 and 3.

[0091] In another aspect of the present invention, a system for detecting and maintaining damaged road facilities is proposed, including: an acquisition unit for obtaining a road image in which road facilities are photographed; a processor unit for detecting an object related to the road facility and determining the degree of damage using the image obtained from the acquisition unit; and a display unit for receiving information from the processor unit and displaying the degree of damage to the damaged road facility; and using a detection method for road facilities according to an embodiment of the present invention, the display unit displays road facilities requiring maintenance by grade on a digital twin virtual road image, and a user determines whether or not to repair the road facility based on the information displayed on the display unit.

[0092] The above-mentioned processor unit may pre-store images of the shape or installation structure of each road facility and compare them with the road image obtained through the acquisition unit to determine the extent of damage. In this case, the extent of damage may be determined by calculating the pixel distance between the pre-stored image and the acquired road image, which represents differences in color, slope, shape, etc., and then multiplying this by a factor corresponding to the actual size to determine the extent of damage to the road facility.

[0093] The display unit may provide both 2D and 3D images of the road, and the user may select the 2D and 3D images as needed. The display unit may express the degree of damage to the road facility as an image on the road image.

[0094] The display unit may be provided on a user terminal. The user terminal may be connected to the processor unit via a wireless, wired, or wired / wireless combination network. Depending on the embodiment, the display unit may be a screen implemented in an application or a service platform implemented on the web. The user can use the user terminal to check damage information and history information on damaged road facilities, and based on this, determine whether to perform maintenance on the road facilities. Alternatively, the maintenance cycle may be shortened or lengthened depending on the region, distinguishing between areas with frequent and rare damage.

[0095] The above-mentioned system for detecting and maintaining damaged road facilities may go beyond simply detecting damaged road facilities and may include determining whether the road facilities require repair. Furthermore, it may additionally include determining a regular maintenance cycle.

[0096] Prior to determining whether or not to repair the above road facility, the processor unit may first recommend objects requiring repair.

[0097] The above display may allow both the user requesting the repair and the user performing the repair to enter and edit information. This allows a construction worker visiting the site to assess the extent of damage to the road facility, as reported through image analysis, and then directly input corrections. Furthermore, the user can directly input corrections to the extent of damage to the road facility after repair. This on-site feedback and corrections are treated as weighted data and can be used as additional learning data for detecting areas of interest or assessing the extent of damage.

[0098]

[0099] <Example>

[0100] In order to confirm the effectiveness of the present invention, the inventors applied a method for detecting damaged road facilities according to one embodiment of the present invention and a system for detecting and maintaining damaged road facilities using the same to road facilities in Cheonan City and confirmed how effectively maintenance of road facilities is possible.

[0101] The inventors trained image data using a deep learning method based on a large number of secured road images.

[0102] The data was trained using the YOLO application model, and at this time, YOLOv7 and YOLOv8 versions released between 2012 and 2013 were used. Among these two versions of the algorithm, the average precision (Mean Average Precision) was significantly improved when YOLOv8 was used (mAP 93%) compared to YOLOv7 (mAP 87.1%), confirming that object detection and damage degree identification are possible.

[0103] During this process, data augmentation was used to improve accuracy. For example, data augmentation was performed by flipping, rotating, or cropping image data before input. Data augmentation was found to improve accuracy by up to 5% compared to training without it.

[0104] The inventors and Cheonan City completed the present invention after judging that the average precision value derived after using data augmentation and learning using the YOLOv8 version is sufficient for practical application in the field.

[0105] At this time, the average precision (mAP) value was calculated based on the ratio of the intersection of the union between the predicted bounding box and the actual bounding box for a specific object, and the average precision (mAP) value was calculated as the average for all objects of interest.

[0106] The inventors were able to confirm that the detection accuracy results vary significantly depending on the type of each object of interest (bollard, sign, traffic light, crosswalk, and speed bump) and the method of configuring the learning data set in the preprocessing step.

[0107] Based on this, the inventors of the present invention configured the learning data set in five different ways (train, validation, test set 8:1:1, Epoch = 100, Batch size = 32) and analyzed the experimental results for each object of interest (bollard, sign, traffic light, crosswalk, and speed bump) for each method.

[0108] The road images obtained at this time had a significantly unbalanced dataset for each object of interest. By optimizing different hyperparameters within the images, the inventors were able to identify an optimized learning data construction method for each type of object of interest. In another example, they were able to discover a learning data construction method that, on average, successfully identified all objects of interest.

[0109] That is, when a bollard, a crosswalk, and a speed bump are included together in a single image, Example 1 is a case where the images secured for each of the bollard, crosswalk, and speed bump are each processed as three images, each of which is designated as Image 1 to Image 3. On the other hand, Example 2 is a case where multiple objects of interest are processed separately using a single image containing multiple objects of interest.

[0110] On the other hand, Example 3 is a case where the number of data containing unbalanced objects is normalized and used, and Examples 4 and 5 are cases where data with an ambiguous degree of corruption is excluded and only data with a clear degree of corruption is handled.

[0111] That is, Examples 1 to 3 classified the damage degree into three classes: high, medium, and low, while Examples 4 and 5 classified the damage degree into only two classes: high and low.

[0112]

[0113] How to organize learning data (classify classes as upper-middle-low or upper-low)Example 1 Divide the learning data set into each image and use it - the number of data is unbalanced, so each image is separated (learn the degree of damage as upper-middle-low data)Example 2 When treating multiple objects of interest as one image - use it by organizing it into a single file (learn the degree of damage as upper-middle-low data)Example 3 Normalize the number of data by class (learn the degree of damage as upper-middle-low data)Example 4 Change the image with the degree of damage 'medium' to 'high' and classify it into two classes (learn the degree of damage as upper-low data)Example 5 Excluding the image with the degree of damage 'medium', classify it into two classes (learn the degree of damage as upper-low data)

[0114] For example, as shown in FIGS. 6 and 7, Example 2 showed the highest accuracy when a crosswalk was selected as an object of interest, and in other examples, Example 2 showed the most accurate detection (compared to Examples 4 and 5) for crosswalk images that were overfitted (Examples 4 and 5) or not found. These experimental results can be confirmed in FIGS. 6 and 7. In addition, Example 2 showed the highest mean accuracy (mAP) value, which means that the scores of various elements were measured to be evenly high on average regardless of the type of object of interest.

[0115] On the other hand, when other objects are used as objects of interest, other embodiments showed higher accuracy.

[0116]

[0117] Based on this data, a system was developed using a digital twin approach, as shown in Figure 3, to display 3D and 2D images similar to actual road conditions and indicate the extent of damage to road facilities. Damaged road facilities were then displayed as images. This system enabled the transmission of on-site maintenance instructions to external contractors.

[0118] At this time, the company preliminarily confirmed images of damaged road facilities, as shown in Figures 4 and 5, and enabled on-site visits. Furthermore, a function was built to photograph the extent of damage to actual road facilities during on-site visits and upload these images back to the system. The inventors confirmed that the system of the present invention continuously learns information through continuous on-site feedback, thereby improving the accuracy of object detection and damage assessment.

[0119] Furthermore, the system was designed to handle information on children's and senior citizen protection zones separately, as shown in the right image of Figure 3. The system displays the urgency of repairs based on the extent of damage to road facilities, categorized into five levels. The urgency of repairs in children's and senior citizen protection zones is weighted more heavily than the extent of damage to similar facilities in other zones. In determining these weights, factors such as the frequency of accidents in specific street areas are taken into account, in addition to the children's and senior citizen protection zones.

[0120]

[0121] The above description is merely an illustrative example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

Claims

1. Step of acquiring road images; A step of detecting an area of ​​interest object from the above road image; A step of determining the degree of damage to the object of interest from the detected object area; and Including a step of indicating the degree of damage to the object of interest; Method for detecting damaged road facilities.

2. In paragraph 1, The steps of obtaining the above road image are: Obtaining images from one or more of the following means: satellite photos, CCTV, and black boxes; Method for detecting damaged road facilities.

3. In paragraph 2, The step of acquiring the above road image comprises acquiring a plurality of images captured by different means or locations, The step of determining the degree of damage to the object of interest includes determining the degree of damage by comparing the degrees of damage to the same object of interest from the plurality of images. Method for detecting damaged road facilities.

4. In paragraph 1, The step of determining the degree of damage; and the step of indicating the degree of damage; between them, further comprising a step of storing the degree of damage to the object of interest; The step of indicating the degree of damage is to indicate the degree of damage of the last saved object of interest, along with at least one of address data and information about maintenance history of the object of interest. Method for detecting damaged road facilities.

5. In paragraph 1, The steps to determine the extent of the above damage are: Using one or more pieces of information from the color, slope and shape of the object of interest, Method for detecting damaged road facilities.

6. In paragraph 1, The steps to indicate the degree of damage are: Including displaying the extent of damage to the object of interest on the digital twinned road image, Method for detecting damaged road facilities.

7. In paragraph 6, The above digital twin road image is displayed with history information regarding the maintenance of the object of interest. Method for detecting damaged road facilities.

8. An acquisition unit for obtaining a road image in which road facilities are photographed; A processor unit that detects objects related to road facilities and determines the degree of damage using the images obtained from the above acquisition unit; and Including a display unit that receives information from the processor unit and displays the degree of damage to damaged road facilities; Using the detection method of Article 1, A device for detecting damaged road facilities.

9. An acquisition unit for obtaining a road image in which road facilities are photographed; A processor unit that detects objects related to road facilities and determines the degree of damage using the images obtained from the above acquisition unit; and Including a display unit that receives information from the processor unit and displays the degree of damage to damaged road facilities; Using the detection method of road facilities in Article 1, The above display section indicates road facilities requiring maintenance by grade on the road image. The user determines the maintenance cycle of road facilities based on the information displayed on the above display. Detection and maintenance system for damaged road facilities.

Images