Traffic marking performance decay monitoring method and system based on deep learning

Abstract

The invention relates to the technical field of image processing, in particular to a traffic marking performance decay monitoring method and system based on deep learning, and compared with a traditional method, the traffic marking performance decay monitoring method and system have the obvious comprehensive detection capability advantage. According to the deep learning method, multiple aspects of characteristics, including shape, color distribution, texture change and the like, of the traffic marking image can be analyzed at the same time, so that the performance decay condition of the marking is comprehensively judged. In the aspect of accuracy, deep learning can learn marking performance decay characteristic modes of different degrees through a large amount of data training, and judgment can be more accurately carried out during detection. The traditional visual detection is difficult to ensure high precision due to subjective factors. In the aspect of efficiency, once a model is trained in traffic marking performance decay detection based on deep learning, a large number of traffic marking images can be rapidly processed, and the method is suitable for large-scale road network detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a method and system for monitoring the performance decay of traffic markings based on deep learning. Background Art

[0002] With the development of intelligent transportation systems, the recognition and detection of traffic markings have become an important part of ensuring traffic safety and improving traffic efficiency.

[0003] However, traditional traffic marking detection methods often rely on specific image processing technologies and feature extraction algorithms, and these methods are often difficult to maintain stable performance when facing complex and changeable road environments. In recent years, deep learning technology has made remarkable progress in the field of image recognition, and the deep learning-based method has become an ideal choice for solving this problem due to its advantages in image recognition and processing. Summary of the Invention

[0004] To achieve the above object, this application provides the following technical solutions:

[0005] According to the first aspect of the present invention, the present invention claims protection for a method for monitoring the performance decay of traffic markings based on deep learning, including:

[0006] Collecting target marking image samples, and training a first target marking monitoring model and a second target marking monitoring model based on the target marking image samples;

[0007] Obtaining the marking image to be monitored, and inputting the marking image to be monitored into the first target marking monitoring model and the second target marking monitoring model respectively to obtain a first marking monitoring result and a second marking monitoring result;

[0008] Integrating the first marking monitoring result and the second marking monitoring result to obtain the performance decay result of the marking image to be monitored.

[0009] Further, the collecting of the target marking image samples further includes:

[0010] Obtaining the scene and data source type of the original marking sample, and obtaining candidate marking image samples based on the scene and data source type;

[0011] Performing image preprocessing on the candidate marking image samples to obtain target marking image samples;

[0012] Setting marking annotation criteria, and annotating the target marking image samples according to the marking annotation criteria.

[0013] Further, the training of the first target marking monitoring model and the second target marking monitoring model based on the marking image samples further includes:

[0014] Extract the marking line features from the marked marking line image samples to obtain the marking line features of the marking line image samples;

[0015] Configure a marking line attenuation evaluation element set and a marking line attenuation cause set;

[0016] Adopt a CNN model as the first candidate marking line monitoring model and LSTM models and GRU as the second candidate marking line monitoring models;

[0017] Train the first candidate marking line monitoring model and the second candidate marking line monitoring model according to the marking line features, the marking line attenuation evaluation element set and the marking line attenuation cause set of the marking line image samples;

[0018] Evaluate and optimize the first candidate marking line monitoring model and the second candidate marking line monitoring model to obtain a first target marking line monitoring model and a second target marking line monitoring model.

[0019] Further, the step of obtaining the marking line image to be monitored and inputting the marking line image to be monitored into the first target marking line monitoring model and the second target marking line monitoring model respectively to obtain a first marking line monitoring result and a second marking line monitoring result further includes:

[0020] Input the marking line image to be monitored into the first target marking line monitoring model respectively to obtain the real-time marking line state and the marking line attenuation factors of the marking line image to be monitored;

[0021] Input the marking line image to be monitored into the second target marking line monitoring model respectively to obtain the marking line attenuation prediction result of the marking line image to be monitored.

[0022] Further, the step of integrating the first marking line monitoring result and the second marking line monitoring result to obtain the performance decay result of the marking line image to be monitored further includes:

[0023] Input the real-time marking line state, the marking line attenuation factors and the marking line attenuation prediction result of the marking line image to be monitored into a first string, a second string and a third string respectively;

[0024] Connect the first string, the second string and the third string, and use the string set of the connection result as the performance decay result of the marking line image to be monitored.

[0025] According to the second aspect of the present invention, the present invention claims to protect a monitoring system for the performance decay of traffic marking lines based on deep learning, including:

[0026] A training module, which collects target marking line image samples and trains a first target marking line monitoring model and a second target marking line monitoring model according to the target marking line image samples;

[0027] A monitoring module that acquires an image of a marking line to be monitored and inputs the image of the marking line to be monitored into the first target marking line monitoring model and the second target marking line monitoring model respectively to obtain a first marking line monitoring result and a second marking line monitoring result;

[0028] A result output module that integrates the first marking line monitoring result and the second marking line monitoring result to obtain a performance decay result of the image of the marking line to be monitored.

[0029] Furthermore, the training module further includes:

[0030] Acquire the scenario and data source type of the original marking line sample, and obtain candidate marking line image samples according to the scenario and data source type;

[0031] Perform image preprocessing on the candidate marking line image samples to obtain target marking line image samples;

[0032] Set marking line annotation criteria, and annotate the target marking line image samples according to the marking line annotation criteria.

[0033] Furthermore, the training module further includes:

[0034] Extract marking line features from the annotated marking line image samples to obtain the marking line features of the marking line image samples;

[0035] Configure a marking line decay evaluation factor set and a marking line decay cause set;

[0036] Adopt a CNN model as the first candidate marking line monitoring model and an LSTM model and a GRU as the second candidate marking line monitoring model;

[0037] Train the first candidate marking line monitoring model and the second candidate marking line monitoring model according to the marking line features, the marking line decay evaluation factor set and the marking line decay cause set of the marking line image samples;

[0038] Perform model evaluation and optimization on the first candidate marking line monitoring model and the second candidate marking line monitoring model to obtain a first target marking line monitoring model and a second target marking line monitoring model.

[0039] Furthermore, the monitoring module further includes:

[0040] Input the image of the marking line to be monitored into the first target marking line monitoring model respectively to obtain the real-time marking line state and marking line decay factors of the image of the marking line to be monitored;

[0041] Input the image of the marking line to be monitored into the second target marking line monitoring model respectively to obtain a marking line decay prediction result of the image of the marking line to be monitored.

[0042] Further, the result output module further includes:

[0043] Input the real-time marking state, marking attenuation factor, and marking attenuation prediction result of the marking image to be monitored into the first string, the second string, and the third string respectively;

[0044] Connect the first string, the second string, and the third string, and use the string set of the connection result as the performance decay result of the marking image to be monitored.

[0045] This application relates to the technical field of image processing, and in particular, to a method and system for monitoring the performance decay of traffic markings based on deep learning, which has obvious comprehensive detection ability advantages compared with traditional methods. The deep learning method can simultaneously analyze multiple aspects of features of traffic marking images, including shape, color distribution, texture changes, etc., so as to comprehensively judge the performance decay of the markings. In terms of accuracy, through a large amount of data training, deep learning can learn the feature patterns of different degrees of marking performance decay, and can make more accurate judgments during detection. However, it is difficult to ensure high precision for traditional visual inspection due to subjective factors. In terms of efficiency, once the model for detecting the performance decay of traffic markings based on deep learning is trained, it can quickly process a large number of traffic marking images, which is suitable for large-scale road network detection. Description of the Drawings

[0046] Figure 1 It is a working flowchart of a method for monitoring the performance decay of traffic markings based on deep learning, which is requested to be protected by the embodiments of this application;

[0047] Figure 2 It is a second working flowchart of a method for monitoring the performance decay of traffic markings based on deep learning, which is requested to be protected by the embodiments of this application;

[0048] Figure 3 It is a third working flowchart of a method for monitoring the performance decay of traffic markings based on deep learning, which is requested to be protected by the embodiments of this application;

[0049] Figure 4 It is a structural module diagram of a system for monitoring the performance decay of traffic markings based on deep learning, which is requested to be protected by the embodiments of this application. Detailed Embodiments

[0050] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0051] The terms "first", "second", and "third" in this application are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative positional relationship, movement conditions, etc. between components in a specific posture (as shown in the drawings). If the specific posture changes, then the directional indications will also change accordingly. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0052] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0053] The purpose of detecting the performance decay of traffic markings is to timely discover changes in the performance of markings so as to take corresponding maintenance measures. In actual traffic management, once the performance decay of traffic markings can be detected more accurately and efficiently, the markings can be repaired and maintained in a timely manner. This helps to improve the level of road traffic safety. Accurate markings can provide clear road guidance for drivers and reduce traffic accidents caused by unclear markings. The detection of traffic marking performance decay based on deep learning applies more optimized detection methods, which can reduce the input of human and material resources. The traffic management department can use limited resources to cover more road areas, better perform traffic management functions, thereby realizing preventive management of traffic facilities and ensuring the continuous safe and efficient operation of the traffic network.

[0054] According to the first embodiment of the present invention, the present invention claims protection for a method for monitoring the performance decay of traffic markings based on deep learning. Referring to Figure 1 , including:

[0055] Collecting target marking image samples, and training a first target marking monitoring model and a second target marking monitoring model based on the target marking image samples;

[0056] Obtain the marking image to be monitored, and input the marking image to be monitored into the first target marking monitoring model and the second target marking monitoring model respectively to obtain the first marking monitoring result and the second marking monitoring result;

[0057] Integrate the first marking monitoring result and the second marking monitoring result to obtain the performance decay result of the marking image to be monitored.

[0058] Deep learning is a subfield of machine learning. It automatically learns the feature representations in data by constructing neural network models with many layers. These neural networks mimic the neuron structure of the human brain and consist of multiple hidden layers. For example, a convolutional neural network (CNN) includes convolutional layers, pooling layers, and fully connected layers, etc. The convolutional layer is mainly responsible for extracting local features in the image. For example, in traffic marking monitoring, it can capture features such as the shape of the marking, the thickness of the line, and the angle. Taking a simple two-lane traffic marking image as an example, the convolutional layer can identify the triangular shape feature of the arrow marking and the line feature of the straight lane line. The role of the pooling layer is to reduce the dimension and the amount of data, while retaining important feature information. This helps to improve the computational efficiency of the model and prevent overfitting. For example, max pooling in traffic marking images can simplify the data by selecting the maximum value in a local area while retaining the significant marking features. The fully connected layer is used to summarize the previously extracted features and finally output the results related to the task. For example, in traffic marking performance decay detection, the fully connected layer may output judgment results such as whether there is decay in the marking and the degree level of the decay.

[0059] The performance decay monitoring and detection of traffic markings is mainly based on image recognition technology. An in-vehicle camera or a monitoring camera fixed near the road acquires the image data of traffic markings. Then, a deep learning model is used to analyze and process these images. Since different traffic markings (such as lane lines, stop lines, guiding arrows, etc.) have their own characteristics in terms of shape, color, position, etc., the deep learning model learns through a large amount of marking image data, so as to accurately identify various marking types.

[0060] Further, referring to Figure 2 , the acquisition of the target marking image sample further includes:

[0061] Obtain the scene and data source type of the original marking sample, and obtain candidate marking image samples according to the scene and data source type;

[0062] Perform image preprocessing on the candidate marking image samples to obtain target marking image samples;

[0063] Set the marking annotation criterion, and annotate the target marking image samples according to the marking annotation criterion.

[0064] Among them, in this embodiment, the scenarios of the original marking samples at least include traffic marking images in different weather conditions, different road types, different traffic flows, etc., and different periods (from newly painted markings to markings with different degrees of wear), which can improve the generalization ability of the model.

[0065] For example, collect marking images from urban roads, highways, and mountain roads, including image data under weather conditions such as sunny days, cloudy days, rainy days, and snowy days, and also include images in different states from intact markings to severely worn markings. In this way, when the model encounters various unknown situations in actual applications, it can make more accurate judgments.

[0066] The data source types of the original marking samples at least include data sources that fuse multi-source image data and data sources that combine non-image data with technologies;

[0067] For the data source that fuses multi-source image data, in addition to the traffic marking image data obtained by in-vehicle cameras and road monitoring cameras, image data from other sources can also be fused. For example, use the aerial view images of traffic markings taken by drones. These images can provide more macroscopic information on the layout and regional conditions of traffic markings. When fused with the images of ground cameras, it is possible to understand the situation of traffic markings more comprehensively. For example, at some large transportation hubs or complex road intersections, the images taken by drones can help determine whether the layout of traffic markings in the overall area is reasonable and whether some markings have been neglected in management and maintenance due to the construction of other facilities (such as isolation belts, etc.), and then combine the ground camera images to accurately judge the performance decay of the markings.

[0068] Fuse optical images with other types of images (such as infrared images). In some special cases, the optical images of traffic markings may be affected by weather (such as rainy or foggy weather) or the environment (such as strong light reflection interference, etc.). Infrared images can penetrate some interference sources to obtain information about the markings. Fusing optical images and infrared images can improve the accuracy of traffic marking detection in complex environments. For example, on a foggy day, the traffic markings in the optical image may become blurred, but by virtue of the sensitivity of infrared images to temperature differences, the position and shape of the markings can be more clearly identified, thereby more accurately judging their performance status.

[0069] For data sources that combine non-image data with technologies, including the combination of traffic flow data: Traffic flow data is closely related to the performance decay of traffic markings. If the data of the traffic marking monitoring system can be combined with that of the traffic flow monitoring system, more evidence can be provided for the performance decay of the markings. For example, when the traffic flow on a road suddenly increases and lasts for a period of time, it can be inferred that the wear pressure on the traffic markings is also increasing, and the performance of the traffic markings on this section of the road needs to be focused on. At the same time, if the traffic markings show wear or other performance decay phenomena, it may also affect the distribution of traffic flow. When the lane lines are severely worn, vehicles may drive in disorder, resulting in a disordered traffic flow distribution.

[0070] Combined with Geographic Information System (GIS) technology: GIS technology can provide information such as the geographical location of the road and the surrounding environment. Combining traffic marking monitoring with GIS technology can more deeply analyze the influencing factors of traffic marking performance decay and predict the decay trend. For example, using the terrain data in GIS, if a road is located in a section with a large slope, the wear of the markings by vehicles during driving may be aggravated, and this information on the influence of the terrain on the marking wear can help more accurately monitor and analyze the performance decay of the markings.

[0071] The image preprocessing at least includes: image cleaning, image annotation, and image enhancement;

[0072] In the image cleaning and image annotation, invalid images in the dataset (such as images with severely deformed markings due to extremely tilted shooting angles and unable to be used for analysis) are cleaned up and accurately annotated.

[0073] The annotation content should include the type, location of the traffic markings, and the annotation of whether there is performance decay (if it is determined that there is, information such as the approximate type and degree of decay can also be annotated). Accurate annotation can enable the deep learning model to learn the correct relationship between features and categories, thereby improving the detection accuracy.

[0074] For example, in the lane line image dataset, accurately marking information such as the boundaries and worn areas of each lane line can enable the model to better distinguish the normal and decayed states of the lane lines.

[0075] The image enhancement includes transformations such as image rotation, translation, and scaling: By rotating the original traffic marking image by a certain angle, translating it to a certain position, or scaling it, the diversity of the dataset can be increased. For example, when monitoring traffic markings at intersections, due to different driving angles of vehicles, there are angular differences in the captured marking images. The dataset after rotating and other operations on the images can enable the model to better adapt to the changes in the actual scenario. In addition, the scaling operation simulates the change in the size of the markings in the image caused by different distances, making the model more accurate in detecting markings at different distances during actual monitoring.

[0076] The image enhancement also includes brightness and contrast adjustment: Changing the brightness and contrast of the traffic marking image also belongs to the means of data enhancement. Under different weather and lighting conditions, there are significant differences in the brightness and contrast of the markings in the image. By adjusting the brightness and contrast of the image, the model can learn the characteristics of the markings under various lighting conditions. For example, the image is darker on cloudy days and brighter at noon under strong sunlight. In order to prevent the model from only being able to correctly detect the markings under a certain lighting condition, the brightness and contrast of the image need to be adjusted so that the model can adapt to the recognition of markings under different lighting conditions, thereby improving the detection accuracy.

[0077] Setting the marking annotation criteria and annotating the target marking image samples according to the marking annotation criteria also includes:

[0078] After obtaining the data, these data need to be annotated so that the model can learn the characteristics of the performance decay of the markings. The annotation criteria should be refined and clear. The annotation of the performance decay of the markings can be classified and annotated from the appearance aspect, such as different degrees of wear, discoloration, and peeling of the markings; for reflective markings, the change level of their reflective performance can also be annotated. The annotators need to undergo systematic training and operate according to the unified annotation criteria. For example, for the wear degree of the markings, it can be divided into different levels such as slight wear (the marking thickness reduction does not exceed 10%), moderate wear (10%-30%), and severe wear (more than 30%) for annotation. And in order to improve the annotation quality, multiple people can be used to annotate the same set of data, and then the annotation results are reviewed and unified to ensure the accuracy of the annotated data, thereby providing a reliable data source for training an accurate deep learning model.

[0079] Further, referring to Figure 3 , training the first target marking monitoring model and the second target marking monitoring model according to the marking image samples also includes:

[0080] Performing marking feature extraction on the annotated marking image samples to obtain the marking features of the marking image samples;

[0081] Configuring a marking decay evaluation element set and a marking decay cause set;

[0082] The CNN model is used as the first candidate marking line monitoring model, and the LSTM model and GRU are used as the second candidate marking line monitoring models;

[0083] Train the first candidate marking line monitoring model and the second candidate marking line monitoring model according to the marking line features, marking line attenuation evaluation factor set, and marking line attenuation cause set of the said marking line image samples;

[0084] Perform model evaluation and optimization on the first candidate marking line monitoring model and the second candidate marking line monitoring model to obtain the first target marking line monitoring model and the second target marking line monitoring model.

[0085] Among them, in this embodiment, perform marking line feature extraction on the marked marking line image samples to obtain the marking line features of the marking line image samples, including:

[0086] Appearance features: Scratches, damages, etc. may occur on the marking line. For example, during vehicle driving, the tires may scrape the marking line, or factors such as road construction and foreign object impact may cause the surface of the marking line to be uneven or have cracks. After long-term use, there may also be a phenomenon of overall peeling of the marking line, resulting in partial loss of the marking line. The surface is no longer smooth and flat, and defects such as bubbles or impurities may occur. These defects may be due to construction process problems or environmental factors during use.

[0087] Color features: The originally distinct color will become dull. For example, white marking lines may turn yellow, and yellow marking lines may become lighter in color. This is mainly due to the change of pigments in the marking line paint caused by long-term sunlight exposure, rain, and chemical erosion. The decline in color will affect the visibility of the marking line. Especially under low light conditions, drivers may have difficulty accurately distinguishing different types of marking lines.

[0088] Reflectivity features: For reflective marking lines, their retroreflective coefficient will decrease over time and with usage. For example, the wear and shedding of glass beads or the damage of the surface coating of the marking line will cause the reflective effect to deteriorate. Under low visibility conditions such as at night or in rainy and foggy weather, drivers rely on the reflected light of the marking line to identify road information. If the reflectivity decreases, it will seriously affect traffic safety.

[0089] Size and shape features: The dimensions such as the width, length, and angle of the marking line may change. For example, due to road surface deformation (such as thermal expansion and contraction, rutting, etc.), the marking line will be stretched or compressed, resulting in a narrower or wider width, and the solid segment length and interval of the dotted line may also not meet the design requirements.

[0090] Special-shaped marking lines, such as arrows, characters, etc. may be deformed, thus affecting the accuracy of their indication.

[0091] Encroachment feature: In the actual road traffic scenario, the environment around the traffic markings may have an impact on them. For example, the building materials piled up during road construction may encroach on the area of the markings. The deep learning model analyzes the relative position relationship between the traffic markings and the surrounding objects in the image. When it is found that other objects occupy the area that the markings should have, it can be judged that the markings have been encroached, which also represents the decay of the marking performance to a certain extent because the normal guiding function of the markings is damaged.

[0092] Configuring the marking attenuation evaluation factor set and the marking attenuation cause set also includes:

[0093] The marking attenuation evaluation factor set includes: marking thickness, retroreflective coefficient of reflective markings, and marking integrity;

[0094] The marking thickness is an important indicator reflecting the quality and performance decay degree of traffic markings. There is a certain thickness standard for normally applied traffic markings. For example, the thickness of hot-melt markings generally requires to be between 1.5 - 2.5 mm. If the measured marking thickness is lower than this standard, it indicates that the markings are worn, and as the thickness continuously decreases, its performance is also continuously decaying. The thickness can be detected using a special marking thickness measuring instrument, and multiple measurement points are selected on the road for measurement by sampling, and then the performance decay and the decay degree of the markings are evaluated according to the measurement results.

[0095] The retroreflective coefficient of reflective markings directly determines the visibility of the markings at night, which is crucial for ensuring night driving safety. A newly applied qualified reflective marking has a relatively high retroreflective coefficient and can effectively reflect the vehicle lights, enabling the driver to clearly identify the markings. However, over time and due to factors such as wear, the retroreflective coefficient will decrease. When the retroreflective coefficient is lower than the specified minimum value (such as the retroreflective coefficient of white reflective markings is lower than 150 mcd·m-2·lx-1), it indicates that the reflective performance of the markings has severely decayed and needs to be repaired or re-applied. The retroreflective coefficient can be measured using a retroreflective coefficient measuring instrument, and generally, standard measurement points are selected for testing at night or in a low-light environment.

[0096] Marking integrity includes the continuity of the markings, whether there is peeling and defect, etc. Through visual inspection or image analysis means, if it is found that there are partial missing parts, breakpoints or large-scale peeling phenomena in the markings, it indicates that the integrity of the markings is damaged and its performance has decayed. For example, if there is a long section missing on the lane line, it will affect the normal driving guidance of vehicles, which is an obvious manifestation of the marking performance decay. Generally, regular road inspections or the use of monitoring image analysis can be adopted to evaluate the integrity of the markings.

[0097] The set of reasons for the attenuation of the marking line also includes: natural environment reasons, traffic environment reasons, and engineering environment reasons;

[0098] The natural environment reasons include: climate reasons and temperature reasons;

[0099] Climate factors have a significant impact on the performance decay of traffic markings. For example, long-term sunlight exposure can gradually age the marking materials. The ultraviolet part of sunlight has relatively high energy, which can break the chemical bonds in the marking materials, resulting in changes in the material properties on the surface of the markings. For example, the colors of some markings will gradually fade. Rainwater is also an important factor. When there is more precipitation, the scouring effect of rainwater may cause the glass beads on the surface of the markings to fall off or the coating of the markings to become thinner. In some areas with heavy rainfall, this phenomenon is more obvious. If acid rain is encountered, the chemical corrosiveness of acid rain will also cause the marking materials to be damaged faster. In addition, the freeze-thaw cycle in cold weather also has a destructive effect on the markings. When water freezes in the pores or microcracks of the markings, the expansion force of the ice will make these tiny defects larger, thereby causing the peeling of the markings or generating more cracks, reducing the integrity and reflective performance of the markings.

[0100] The influence of temperature changes on traffic markings cannot be underestimated. In a high-temperature environment, the traffic marking paint may become soft and deformed. For example, the viscosity of hot-melt marking paint will decrease at high temperatures, which may cause the surface of the markings to be uneven or flow, thus affecting the appearance and its due functions of the markings. At the same time, a high-temperature environment will also accelerate the aging process of the marking materials. When the temperature is too low, some marking materials may become brittle and easy to break. The elasticity of the materials will decrease at low temperatures, making the markings prone to cracks when being run over by vehicles or subjected to other external forces, reducing their service life and performance.

[0101] The traffic environment reasons include: traffic flow and vehicle condition;

[0102] The size of the traffic flow is directly related to the wear degree of the traffic markings. On sections with a large traffic flow, such as the main roads in the city center or the busy sections of highways, a large number of vehicles frequently run over the traffic markings, which will accelerate the wear speed of the markings. For example, in terms of the number of vehicles passing through per day, the marking wear situation on a section with a daily traffic volume of tens of thousands of vehicles is much more serious than that on a section with a daily traffic volume of only thousands of vehicles. This is because the vehicle tires continuously rub and squeeze against the surface of the markings, and more frictional forces act on the markings, gradually peeling off the materials on the surface of the markings, making the marking thickness thinner. For reflective markings, this kind of wear will also cause the reflective glass beads to fall off, resulting in a decline in the reflective performance.

[0103] Different types of vehicles have different impacts on traffic markings. The wear caused by large and heavy vehicles is much greater than that by small vehicles. Due to their large load capacity, the tires of heavy trucks exert greater pressure on the road surface and markings, pressing deeper into the surface of the markings, resulting in more serious deformation and wear of the markings. Moreover, considering the vehicle driving speed, vehicles traveling at high speeds have a greater impact on the markings. When a vehicle passes at high speed, the instantaneous impact force between the wheels and the markings makes the markings more likely to crack or shift, accelerating the performance decay of the markings over time.

[0104] The said engineering environmental reasons include: material reasons, construction reasons;

[0105] Different types of marking materials (such as hot-melt type, water-based, solvent-based, etc.) have different characteristics such as durability and compressive resistance. Hot-melt markings usually have good durability, but if the raw material ratio is improper during construction, such as a low resin content, it will affect its adhesiveness, resulting in the markings being easily peeled off and causing performance decay. Although water-based markings are environmentally friendly, their water resistance may be poor. If the material production process is not up to standard, their service life in a humid environment will be greatly shortened, and they are easily washed away or faded.

[0106] Environmental conditions such as temperature and humidity during construction, as well as the operation standardization degree of construction workers, will all affect the quality of traffic markings. If the temperature is too low during construction, the hot-melt marking paint may not be fully melted, resulting in an insecure bond between the markings and the road surface; when constructing in wet weather conditions, water will mix into the marking paint, affecting the performance of the paint, such as possibly causing the surface of the markings to be uneven or blistering, which will further affect the performance and service life of the markings and accelerate their performance decay.

[0107] The use of the CNN model as the first candidate marking monitoring model and the LSTM model and GRU as the second candidate marking monitoring models specifically includes:

[0108] Since the detection of traffic marking decay involves the judgment of multiple aspects of characteristics of the markings, such as shape, color, and reflective performance (which can be indirectly inferred through appearance and comparison with the surrounding environment, etc.), a convolutional neural network (CNN) architecture is usually selected. CNN is a deep learning network structure widely used in traffic marking monitoring. It itself has a special structure of convolutional layers, which can automatically learn the characteristics of traffic marking images without a large amount of manual feature extraction work. For example, for the characteristics of traffic markings such as shape and lines mentioned above, CNN can extract them through the convolution operation of the convolutional kernel sliding on the image.

[0109] Due to the local connection and weight sharing characteristics of CNN, the number of network parameters is greatly reduced, and the computational complexity is lowered. For example, when dealing with a large-scale traffic marking image dataset, compared with traditional fully connected neural networks, CNN can train and infer more quickly and efficiently. Taking a traffic marking image dataset containing different sections of multiple roads in a city as an example, using CNN can quickly obtain accurate results, while other networks may face difficulties in training or overfitting problems due to a huge number of parameters.

[0110] The specific applications in marking monitoring are as follows:

[0111] Feature extraction: The convolutional layer of CNN can extract various features of traffic markings, such as the shape, color, etc. features mentioned above, as the basis for subsequent classification and detection performance decay. For example, in the task of detecting whether a lane line is blurred, the convolutional layer can extract the edge and color features of the lane line to judge the change in its clarity.

[0112] Object detection: CNN can also be used to detect whether there are other abnormal objects on traffic markings, such as a stone occupying the lane line area. Through the object detection algorithm of CNN, the positions and categories of these abnormal objects can be located and identified, so as to judge whether the traffic marking is in a state of performance decay and whether the function is damaged due to external occupation.

[0113] Structure improvement and advantages: Long short-term memory network (LSTM) and gated recurrent unit (GRU) are improved versions of RNN. LSTM can better control the transmission and retention of information by introducing special structures such as input gate, forget gate, and output gate, effectively solving the problems of gradient disappearance and gradient explosion of RNN. GRU is a relatively more concise improved structure, which combines some gating operations and reduces the complexity of the model.

[0114] Applications in marking monitoring: In the monitoring of traffic marking performance decay, assume that we want to analyze the wear trend of traffic markings over a long period of time. LSTM or GRU can predict the future wear situation based on the data of the wear degree of traffic markings detected each time before. For example, if the wear speed of a lane line is accelerating for several consecutive weeks, then the model can predict whether this lane line will be completely worn out and unable to be used normally in a short time in the future, and notify the relevant department for maintenance in advance.

[0115] The model evaluation and optimization of the first candidate marking monitoring model and the second candidate marking monitoring model to obtain the first target marking monitoring model and the second target marking monitoring model further includes:

[0116] Evaluation Index Setting and Its Importance Ranking: To evaluate the performance of a traffic marking performance decay detection model based on deep learning, appropriate evaluation indexes need to be set. Commonly used evaluation indexes include accuracy, recall rate, F1-score, etc. Accuracy represents the proportion of tickets accurately predicted by the model. For traffic marking performance decay detection, this is related to whether the detection results are reliable; the recall rate reflects the proportion of truly decayed markings that the model can identify among the actual decayed markings, which is very important for avoiding missing the detection of decayed markings; the F1-score is an index that comprehensively considers accuracy and recall rate and can more comprehensively evaluate the performance of the model. In practical applications, the importance of these indexes can be ranked according to specific requirements. For example, in scenarios with extremely high requirements for traffic marking safety, the recall rate may be relatively more important to ensure that as many decayed markings as possible can be discovered; while in some scenarios that require precise management and minimize misjudgments, more attention should be paid to accuracy.

[0117] To improve the performance of the model, a series of optimization measures usually need to be taken and the model needs to be iteratively updated. Through such continuous optimization and iteration processes, the model can continuously adapt to different traffic marking performance decay detection requirements.

[0118] Model Structure Optimization: For deep learning models such as convolutional neural networks (CNNs), the number of network layers and the number of neurons in each layer can be adjusted according to the characteristics of the traffic marking monitoring task. If the model is too simple, it may not be able to extract complex enough marking features. If the number of model layers is too large, there may be a risk of overfitting. For example, in the traffic marking performance decay detection task, if it is found that the detection of some slightly worn markings is inaccurate, the number of convolutional layers used to extract more subtle features in the neural network can be increased, or the number of neurons in the fully connected layer can be adjusted to obtain better output results.

[0119] Select Appropriate Model Components: When constructing a deep learning model, appropriate components are selected according to the task requirements. For example, when using a recurrent neural network (RNN) to process the time series data of traffic markings, whether to choose a long short-term memory network (LSTM) or a gated recurrent unit (GRU) depends on the specific data characteristics and task requirements. If the time dependence of the data is very strong and long-term memory is required, LSTM may be more suitable; if more emphasis is placed on the simplicity and computational speed of the model, GRU can be considered first. In addition, some proven effective convolutional structures (such as the residual block structure in ResNet, etc.) can be selected in the convolutional neural network to improve the performance and stability of the model.

[0120] Model Parameter Adjustment and Optimization: Hyperparameter Tuning: Hyperparameters include learning rate, batch size, number of iterations, etc. A reasonable learning rate can control the learning speed of the model. If the learning rate is too large, the model may skip the optimal solution during training; if the learning rate is too small, the training process will be very slow. The batch size determines the amount of data used in one training. An appropriate batch size can improve the training efficiency of the model. The number of iterations needs to be determined according to the convergence of the model.

[0121] Weight Initialization: Appropriate weight initialization methods can help the model converge faster and improve accuracy. For example, common random initialization (such as the Xavier initialization method, etc.) can make the weights in the neural network randomly take values within a reasonable range, avoiding problems such as gradient disappearance or gradient explosion, thereby improving the accuracy of the model's learning of traffic marking image features.

[0122] Real-time Detection and Deployment: Deploy the trained model to the actual environment for real-time detection of the performance decay of traffic markings. Technologies such as edge computing and cloud computing can be combined to achieve real-time data processing and fast response. At the same time, establish a feedback mechanism to continuously update and optimize the model to adapt to changing road conditions and traffic situations.

[0123] Furthermore, the step of obtaining the to-be-monitored marking image and inputting the to-be-monitored marking image into the first target marking monitoring model and the second target marking monitoring model respectively to obtain the first marking monitoring result and the second marking monitoring result further includes:

[0124] Input the to-be-monitored marking image into the first target marking monitoring model respectively to obtain the real-time marking state and marking decay factors of the to-be-monitored marking image;

[0125] Input the to-be-monitored marking image into the second target marking monitoring model respectively to obtain the marking decay prediction result of the to-be-monitored marking image.

[0126] Furthermore, the step of integrating the first marking monitoring result and the second marking monitoring result to obtain the performance decay result of the to-be-monitored marking image further includes:

[0127] Input the real-time marking state, marking decay factors, and marking decay prediction result of the to-be-monitored marking image into the first string, the second string, and the third string respectively;

[0128] Connect the first string, the second string, and the third string, and use the set of strings obtained by the connection as the performance decay result of the to-be-monitored marking image.

[0129] According to the second embodiment of the present invention, the present invention claims protection for a monitoring system for the performance decay of traffic markings based on deep learning. Referring to Figure 4 , it includes:

[0130] A training module that collects target marking image samples and trains a first target marking monitoring model and a second target marking monitoring model based on the target marking image samples;

[0131] A monitoring module that acquires an image of the marking to be monitored and inputs the image of the marking to be monitored into the first target marking monitoring model and the second target marking monitoring model respectively to obtain a first marking monitoring result and a second marking monitoring result;

[0132] A result output module that integrates the first marking monitoring result and the second marking monitoring result to obtain the performance decay result of the image of the marking to be monitored.

[0133] Furthermore, the training module further includes:

[0134] Obtain the scene and data source type of the original marking sample, and obtain candidate marking image samples based on the scene and data source type;

[0135] Perform image preprocessing on the candidate marking image samples to obtain target marking image samples;

[0136] Set marking annotation criteria, and annotate the target marking image samples according to the marking annotation criteria.

[0137] Furthermore, the training module further includes:

[0138] Extract marking features from the annotated marking image samples to obtain the marking features of the marking image samples;

[0139] Configure a marking decay evaluation element set and a marking decay reason set;

[0140] Adopt a CNN model as the first candidate marking monitoring model and an LSTM model and a GRU as the second candidate marking monitoring model;

[0141] Train the first candidate marking monitoring model and the second candidate marking monitoring model based on the marking features, marking decay evaluation element set and marking decay reason set of the marking image samples;

[0142] Perform model evaluation and optimization on the first candidate marking monitoring model and the second candidate marking monitoring model to obtain a first target marking monitoring model and a second target marking monitoring model.

[0143] Furthermore, the monitoring module further includes:

[0144] Input the to-be-monitored marking images into the first target marking monitoring model respectively to obtain the real-time marking state and marking attenuation factors of the to-be-monitored marking images;

[0145] Input the to-be-monitored marking images into the second target marking monitoring model respectively to obtain the marking attenuation prediction results of the to-be-monitored marking images.

[0146] Further, the result output module further includes:

[0147] Input the real-time marking state, marking attenuation factors and marking attenuation prediction results of the to-be-monitored marking images into the first string, the second string and the third string respectively;

[0148] Connect the first string, the second string and the third string, and use the string set of the connection result as the performance decay result of the to-be-monitored marking image.

[0149] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0150] In addition, each functional unit in various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only the implementation manner of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, shall be included in the patent protection scope of the present application by the same token.

[0151] The specific implementation manners of the invention have been described in detail above, but it is only an example, and the present application is not limited to the specific implementation manners described above. For those skilled in the art, any equivalent modification or substitution to the invention is also within the scope of the present application. Therefore, all equal transformations, modifications and improvements made without departing from the spirit and principle scope of the present application should be covered within the scope of the present application.

Claims

1. A method for monitoring traffic marking performance decay based on deep learning, characterized in that: include: Collect target marking image samples, and train a first target marking monitoring model and a second target marking monitoring model according to the target marking image samples; Acquire a line marking image to be monitored, and input the line marking image to be monitored into the first target line marking monitoring model and the second target line marking monitoring model respectively to obtain a first line marking monitoring result and a second line marking monitoring result; The first reticle monitoring result and the second reticle monitoring result are integrated to obtain a performance decay result of the reticle image to be monitored.

2. The method for monitoring traffic marking performance decay based on deep learning according to claim 1, characterized in that: The collecting of target mark line image samples also includes: Obtaining the scene and data source type of the original marking line sample, and obtaining the candidate marking line image sample according to the scene and data source type; Performing image preprocessing on the candidate marking line image samples to obtain target marking line image samples; A marking line marking criterion is set, and the target marking line image sample is marked according to the marking line marking criterion.

3. The method for monitoring traffic marking performance decay based on deep learning according to claim 1, characterized in that: The step of obtaining a first target marking monitoring model and a second target marking monitoring model based on the marking image samples training also includes: Extracting marking features from the marked marking image samples to obtain marking features of the marking image samples; Configure the marking attenuation assessment element set and marking attenuation cause set; The CNN model is used as the first candidate marking monitoring model, and the LSTM model and GRU are used as the second candidate marking monitoring model; Training the first candidate marking monitoring model and the second candidate marking monitoring model according to the marking features, marking attenuation assessment factor set and marking attenuation cause set of the marking image sample; Model evaluation and optimization are performed on the first candidate marking monitoring model and the second candidate marking monitoring model to obtain a first target marking monitoring model and a second target marking monitoring model.

4. The method for monitoring traffic marking performance decay based on deep learning according to claim 1, characterized in that: The step of acquiring the image of the marking line to be monitored and inputting the image of the marking line to be monitored into the first target marking line monitoring model and the second target marking line monitoring model to obtain the first marking line monitoring result and the second marking line monitoring result further includes: Inputting the line marking images to be monitored into the first target line marking monitoring model respectively to obtain the real-time line marking state and line marking attenuation factor of the line marking images to be monitored; The target line images to be monitored are respectively input into the second target line monitoring model to obtain the line attenuation prediction result of the target line images to be monitored.

5. The method for monitoring traffic marking performance decay based on deep learning according to claim 4, characterized in that: The step of integrating the first reticle monitoring result and the second reticle monitoring result to obtain the performance decay result of the reticle image to be monitored further includes: The real-time marking state, marking attenuation factor and marking attenuation prediction result of the marking image to be monitored are input into the first character string, the second character string and the third character string respectively; The first character string, the second character string and the third character string are connected, and a character string set of the connection result is used as the performance decay result of the reticle image to be monitored.

6. A traffic marking performance decay monitoring system based on deep learning, characterized in that: include: A training module collects target marking image samples, and obtains a first target marking monitoring model and a second target marking monitoring model based on the target marking image samples; A monitoring module, which obtains a line marking image to be monitored, and inputs the line marking image to be monitored into the first target line marking monitoring model and the second target line marking monitoring model to obtain a first line marking monitoring result and a second line marking monitoring result; The result output module integrates the first marking line monitoring result and the second marking line monitoring result to obtain the performance decay result of the marking line image to be monitored.

7. The traffic marking performance decay monitoring system based on deep learning according to claim 6, characterized in that: The training module further includes: Obtaining the scene and data source type of the original marking line sample, and obtaining the candidate marking line image sample according to the scene and data source type; Performing image preprocessing on the candidate marking line image samples to obtain target marking line image samples; A marking line marking criterion is set, and the target marking line image sample is marked according to the marking line marking criterion.

8. The traffic marking performance decay monitoring system based on deep learning according to claim 6, characterized in that: The training module further includes: Extracting marking features from the marked marking image samples to obtain marking features of the marking image samples; Configure the marking attenuation assessment element set and marking attenuation cause set; The CNN model is used as the first candidate marking monitoring model, and the LSTM model and GRU are used as the second candidate marking monitoring model; Training the first candidate marking monitoring model and the second candidate marking monitoring model according to the marking features, marking attenuation assessment factor set and marking attenuation cause set of the marking image sample; Model evaluation and optimization are performed on the first candidate marking monitoring model and the second candidate marking monitoring model to obtain a first target marking monitoring model and a second target marking monitoring model.

9. The traffic marking performance decay monitoring system based on deep learning according to claim 6, characterized in that: The monitoring module further includes: Inputting the line marking images to be monitored into the first target line marking monitoring model respectively to obtain the real-time line marking state and line marking attenuation factor of the line marking images to be monitored; The target line images to be monitored are respectively input into the second target line monitoring model to obtain the line attenuation prediction result of the target line images to be monitored.

10. The traffic marking performance decay monitoring system based on deep learning according to claim 9, characterized in that: The result output module further includes: The real-time marking state, marking attenuation factor and marking attenuation prediction result of the marking image to be monitored are input into the first character string, the second character string and the third character string respectively; The first character string, the second character string and the third character string are connected, and a character string set of the connection result is used as the performance decay result of the reticle image to be monitored.

Images