Road damage automatic detection and repair method based on generative adversarial network

Abstract

The invention discloses an automatic road damage detection and repair method based on a generative adversarial network. The method comprises the following steps: S1, acquiring a road image data set; s2, performing image preprocessing on the acquired road image data set; s3, constructing a generative adversarial network model based on a TensorFlow framework; s4, performing damage detection on the preprocessed road image data set by using a discriminator part in the interpretability enhanced generative adversarial network model, and identifying various damage types in the road image data set; s5, positioning the detected damaged area through an image segmentation technology, and generating a mask image of the damaged area; s6, a specific repairing scheme is formulated; s7, automatically executing a repair task; and S8, optimizing parameters of the generative adversarial network model. According to the invention, the defects in the prior art are effectively overcome, and efficient, accurate and intelligent road maintenance is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of road detection, and in particular to a method for automatically detecting and repairing road damage based on a generative adversarial network. Background Art

[0002] With the acceleration of urbanization and the continuous growth of traffic demand, the maintenance and management of roads as an important part of transportation infrastructure has become particularly critical. However, with the increase of usage time, various forms of damage will inevitably appear on the road surface, such as potholes, cracks, settlement, etc. These damages will not only affect the comfort of vehicle driving, but also may pose a serious threat to traffic safety. Therefore, the detection and repair of road damage has always been an important topic in infrastructure maintenance.

[0003] Existing road damage detection methods mainly rely on manual inspections or traditional image processing techniques. Although manual inspections are intuitive, they are inefficient, labor-intensive, and easily affected by human factors, resulting in inconsistent detection results. Traditional image processing methods usually rely on manual feature extraction or predefined rules, which show significant limitations when faced with complex and changing road environments. Due to the various forms of road damage and the influence of external factors such as lighting, weather conditions, and vehicle driving, these traditional methods have low detection accuracy and robustness. Especially when the environmental conditions are complex or the damage forms are diverse, traditional methods often find it difficult to meet the high-precision detection requirements.

[0004] In addition, existing road damage repair methods usually require a lot of manual participation, and the repair process is cumbersome and inefficient. Traditional repair methods are difficult to achieve real-time and high efficiency, and the repair effect is easily affected by human factors, resulting in uneven repair quality. In addition, the existing technology lacks an intelligent repair plan generation mechanism, making it difficult to automate the repair decision process, further limiting the efficiency and effectiveness of road maintenance work.

[0005] At present, in the field of road damage detection and repair, some studies have begun to explore the application of machine learning and deep learning technologies, especially the use of deep learning models for automated damage detection. However, these methods still have many problems in practical applications. For example, existing deep learning models usually require a large amount of labeled data for training, and the labeling process of road damage data itself is very time-consuming and error-prone. In addition, the black box characteristics of these models make their decision-making process difficult to explain, which to some extent limits the application of these methods in practical engineering. Especially when facing high-risk repair decisions, the lack of explainability may lead to distrust of the model output results, thereby affecting the implementation of repair decisions.

[0006] In response to the above problems, Generative Adversarial Networks (GANs), as a new type of deep learning model, have gained widespread attention and application in the field of computer vision in recent years. GAN can generate realistic images through the adversarial training mechanism between the generator and the discriminator, and can adapt to different environments and damage types to a certain extent. However, the road damage detection and repair technology based on GAN is still in the preliminary research stage and has not yet formed a systematic solution. Existing research mainly focuses on the image generation ability of the GAN model, and lacks research on the interpretation mechanism of the generated images and the generation of intelligent repair solutions. In addition, how to combine the generation ability of GAN with the detection, positioning, and repair decision-making of road damage to achieve automated and intelligent road maintenance is still a technical problem that needs to be solved urgently.

[0007] In summary, the existing technologies in the field of road damage detection and repair have the following main defects: first, traditional methods are difficult to cope with complex and changeable road environments, and the detection accuracy and robustness are low; second, the existing repair methods are overly dependent on manual work and difficult to achieve automation and intelligence; third, the detection methods based on deep learning lack interpretability, which affects the credibility and scalability of the model in practical applications. Therefore, there is an urgent need for an automated road damage detection and repair method based on a generative adversarial network to solve these problems in the existing technology. Summary of the invention

[0008] One purpose of the present invention is to propose a method for automatic detection and repair of road damage based on a generative adversarial network, which effectively solves the deficiencies in the prior art and achieves efficient, accurate and intelligent road maintenance.

[0009] According to an embodiment of the present invention, a method for automatic detection and repair of road damage based on a generative adversarial network comprises the following steps:

[0010] S1, obtaining a road image dataset in real time through a camera or other image acquisition device;

[0011] S2, performing image preprocessing on the collected road image dataset, including denoising, enhancement and histogram equalization processing;

[0012] S3. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model consists of three parts: generator, discriminator, and explanation module.

[0013] S4. Using the discriminator part of the generative adversarial network model with enhanced interpretability, damage detection is performed on the preprocessed road image dataset to identify various types of damage in the road image dataset, including potholes, cracks, and settlements. Combined with the explanation information generated by the explanation module, the specific location of the damaged area and its judgment reasons are output;

[0014] S5. Use image segmentation technology to locate the detected damaged area, generate a mask map of the damaged area, and combine GPS positioning technology to mark the specific geographic coordinates of the damaged location;

[0015] S6. Generate a repaired road image based on the detected damaged area using the generator part of the generative adversarial network, and formulate a specific repair plan based on the information output by the interpretation module, including the required materials, repair methods and construction steps, and provide the basis and explanation for generating the repair plan;

[0016] S7, transmitting the generated repair plan and its explanation information to the automated construction equipment to automatically perform the repair task;

[0017] S8. Utilize the self-learning ability of the generative adversarial network to adapt to different road environments and damage types during the continuous detection and repair process, and optimize the parameters of the generative adversarial network model.

[0018] Optionally, S1 includes the following sub-steps:

[0019] S11, through the camera or other image acquisition equipment deployed on the vehicle or fixed position, with a fixed sampling frequency f s Continuously collect road images to form an image sequence:

[0020] I={I1,I2,,I n};

[0021] Among them, I i Indicates that at time t i The i-th road image collected, n is the total number of images collected;

[0022] S12, for each collected image I i , record the corresponding timestamp t i and the geographic coordinates of the image acquisition device (x i ,y i ), where x i and i They are respectively the image acquisition devices at the acquisition time t i longitude and latitude;

[0023] S13, compare the acquired image sequence I with the corresponding timestamp sequence T = {t1, t2,, t n} and the geographic coordinate sequence G = {(x1,y1),(x2,y2),,(x n ,y n )} to generate a road image dataset with time and space annotations:

[0024] D={(I1,t1,(x1,y1)),(I2,t2,(x2,y2)),,(I n ,t n ,(x n ,y n ))}.

[0025] Optionally, S3 includes the following sub-steps:

[0026] S31. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model includes a generator G(z) and a discriminator D disc (D i ) and interpretation module E(D disc (D i ))Three parts:

[0027] The generator G(z) takes the noise vector z as input and generates the repaired road image G(z) through a multi-layer convolutional neural network, where z follows a Gaussian distribution z~N(0,1);

[0028] G(z)=Deconv(W G ReLU(BN(W z ·z+b z )));

[0029] Among them, W G is the weight matrix of the generator, Deconv represents the deconvolution operation, ReLU is the activation function, BN represents the batch normalization operation, W z and b z are the weight and bias of the input noise vector respectively;

[0030] Discriminator D disc (D i ) Take a single image D in the road image dataset i As input, the convolutional neural network outputs the discriminant probability, which is used to distinguish whether the input image is from the real dataset D i ~p data (D i ) or the image D generated by the generator i ~p G (D i );

[0031] Explanation module E(D disc (D i )) and the discriminator D disc (D i ) integration, by analyzing the activation pattern of the convolutional layer inside the discriminator Act l (D disc (D i)), generating explanation information E(D) about why the GAN model identified a certain area as a damaged area disc (D i ), the interpretation information includes parameters such as the activation layer position of the damaged area and the activation intensity of the damage type, and introduces an adaptive interpretation mechanism to dynamically adjust the interpretation strategy of the interpretation module according to different environments and damage types. E ;

[0032]

[0033] in, is the gradient of the discriminator at layer l, α l To explain the module's adaptive coefficient, Act l (D disc (D i )) is the activation value of the lth layer, S E To explain the strategy;

[0034] S32. Train the generative adversarial network model:

[0035] Using the loss function L D and L G For the discriminator D disc (D i ) and the generator G(z) for adversarial training, the loss function of the discriminator is defined as:

[0036]

[0037] The loss function of the generator is defined as:

[0038]

[0039] Among them, λ is a hyperparameter that weighs the generation loss and restoration accuracy, and D GT is the real road image, MSE represents the mean square error between the generated image and the real image;

[0040] Explanation module E(D disc (D i )) is updated after each discriminator update to reflect the latest discrimination results and decision basis, and dynamically optimizes the explanation strategy S through the adaptive explanation mechanism E ;

[0041]

[0042] Among them, L E is the loss function of the explanation module, η is the learning rate, and are the interpretation strategies at the tth and t+1th iterations respectively;

[0043] S33, after training is completed, the adversarial network model {G(z),D disc (D i ),E(D disc (D i ))}Save as the final detection and repair model.

[0044] Optionally, the interpretation module E(D disc (D i The explanation information generation step in )) includes the following sub-steps:

[0045] In the discriminator D disc (D i ) Inside, perform activation analysis on each convolutional layer l and extract the activation value Act of the layer l (D disc (D i ));

[0046] Calculate the gradient of each convolutional layer The gradient represents the sensitivity of the discriminator to the activation value, which is calculated by calculating the discriminator output D disc (D i ) relative to the activation value Act l (D disc (D i ))'s partial derivative is obtained:

[0047]

[0048] Introducing adaptive coefficient α l Adjust the contribution of each layer's activation value to the final interpretation information, and dynamically adjust the interpretation strategy of the interpretation module based on damage type and environmental factors;

[0049] Explanation information E(D disc (D i )) is summarized to generate interpretation information including the activation layer position of the damaged area and the activation intensity parameters of the damage type, and the interpretation information is passed to the repair plan generation module.

[0050] Optionally, S4 includes the following sub-steps:

[0051] S41. Discriminator D in Generative Adversarial Network Model with Interpretability Enhancement disc (D i ), for the preprocessed road image dataset D′={D′1,D′2,…,D′ n}Analyze each image one by one to identify the single image D′ in the road image dataset i Whether the damaged area is included;

[0052] S42, for each image D′ i When performing damage detection, the discriminator D disc (D i ) Output damage probability P damage (D′ i ), the damage probability is obtained by convolution processing of the discriminator on the image, and combined with the interpretation module E(D disc (D i )) generates interpretation information to determine the specific location of the damaged area Loc(D′) i ) and its damage type Type(D′ i ):

[0053] P damage (D′ i )=Sigmoid(D disc (D i ));

[0054]

[0055] Among them, P damage (D′ i ) is the damage probability, which means that for image D i For example, the probability of being detected as damaged is directly related to whether the image contains road damage. A high P damage The value means that the model has a higher confidence that there is damage in the image, Sigmoid represents the Sigmoid activation function, and Loc(D′ i ) indicates the specific location of the damaged area, marking the i In the above example, the precise geographic location of the damaged area, Region is the candidate damaged area, the argmax operation is used to select the location of the damaged area with the largest activation intensity, and the Classify operation is the classification process for determining the damage type, combining the activation intensity and gradient information generated by the interpretation module;

[0056] S43, output the damage detection result, including the specific location of the damaged area Loc (D′ i ), damage type Type(D′ i ) and its judgment reason R(D′ i ):

[0057] R(D′ i )=(Loc(D′ i ),Type(D′ i ),E(D disc (D i )));

[0058] Among them, the reason for judgment R(D′ i) integrates the spatial location of the damaged area, type classification and interpretation information of the interpretation modules.

[0059] Optionally, S5 includes the following sub-steps:

[0060] S51, detect the damaged area Loc (D′ i ) Use the convolutional neural network based segmentation algorithm to segment the image and remove the damaged area from the image D′ i Extract it and generate the mask image M(D′) of the damaged area i ):

[0061] M(D′ i )=Segment(Loc(D′ i ),D′ i );

[0062] Among them, Segment represents the image segmentation operation, M(D′ i ) is the generated damaged area mask image;

[0063] S52, generate the mask image M(D′ i ) Calculate the bounding box B(D′) of the damaged area i ) and the center point coordinates C(D′ i ) to determine the damaged area;

[0064] S53, combining GPS positioning technology to obtain image D′ i The collection time t′ i The corresponding geographic coordinates (x′ i ,y′ i ), the coordinates of the center point of the damaged area C(D′ i ) and geographic coordinates (x′ i ,y′ i ) to mark the specific geographic coordinates G(D′) of the damaged location i ):

[0065] G(D′ i )=(x′ i +Δx(C(D′ i )),y′ i +Δy(C(D′ i )));

[0066] Among them, Δx(C(D′ i ) and Δy(C(D′ i )) are the lateral and longitudinal offsets of the center point of the damaged area relative to the position of the image acquisition device, G(D′ i ) are the specific geographic coordinates of the marked damage location.

[0067] Optionally, S6 includes the following sub-steps:

[0068] S61, using the generator G(z) in the generative adversarial network, input the mask image M(D′) of the detected damaged area i ), and the original image D′ i Combined together, the restored image D is generated by the generator repair (D′ i ):

[0069] D repair (D′ i )=Deconv(W G ReLU(BN(W M ·M(D′ i )+W D ·D′ i +b G )));

[0070] Among them, W G is the weight matrix of the generator, W M and W D are the weight matrices of the mask image and the original image, respectively, G is the bias vector, BN represents batch normalization operation, ReLU is the activation function, and Deconv represents deconvolution operation;

[0071] S62, combined with the damaged area information E(D′) output by the interpretation module i ) and the generated restored image D repair (D′ i ), formulate a specific repair plan P(D′) i ), the repair plan includes the required materials, repair methods and construction steps:

[0072]

[0073] Among them, Mat k represents the kth type of repair material, Action k represents the kth construction step, is the weight matrix of the repair solution, is the bias vector of the repair solution, Conv represents the convolution operation;

[0074] S63, repair plan P(D′ i ) provides the basis and explanation for the generation, combining the output of the generator G(z) and the explanation information of the explanation module to describe the specific reasons and expected effects of each step in the repair plan:

[0075]

[0076] Among them, β m To explain the adaptive coefficient of the module, is the gradient information of the mth step in the repair scheme, γ m is the influence coefficient of the construction step, A collection of steps for a repair solution.

[0077] The beneficial effects of the present invention are:

[0078] (1) The present invention uses a generative adversarial network model built on the TensorFlow framework. The discriminator can accurately identify various types of road damage, such as potholes, cracks, and subsidence in complex and changeable road environments. The generative adversarial network model uses the generator to generate repaired images. Through adversarial training of the discriminator, the model can maintain stable detection performance when facing different lighting, weather conditions, and road conditions. In addition, the integration of the discriminator and the interpretation module can output detailed interpretation information, increase the credibility of the detection results, and significantly improve the accuracy and robustness of damage detection.

[0079] (2) The present invention utilizes the generator part of the generative adversarial network to generate a repaired road image based on the detected damaged area, and combines the detailed interpretation information generated by the interpretation module to formulate a specific repair plan, which not only includes the required materials, repair methods and construction steps, but also automatically optimizes the algorithm to ensure the efficiency of the repair process and the consistency of the effect. The generation of the repair plan is based on the actual situation and environmental parameters of the damaged area. Through the multi-layer convolutional neural network and the information feedback of the interpretation module, the repair decision formed has a high level of intelligence, can be adjusted and optimized in real time, reduces the need for manual intervention, and improves the repair efficiency.

[0080] (3) The present invention combines image segmentation technology with GPS positioning technology to accurately locate the detected damaged area, generate a mask map of the damaged area and mark its specific geographic coordinates, which not only ensures the spatial accuracy of damage detection, but also provides reliable data support for subsequent repair work. During the repair process, using the generated repair plan, the system can automatically perform the repair task, and conduct real-time monitoring and adjustment through the feedback mechanism to ensure that the repair quality meets the expected standards, realizing fully automated management from detection to repair, and significantly improving the efficiency of road maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0082] Figure 1A flowchart of a method for automatic detection and repair of road damage based on a generative adversarial network proposed by the present invention;

[0083] Figure 2 This is a structural flow chart of building a generative adversarial network model based on the TensorFlow framework in a method for automatic road damage detection and repair based on a generative adversarial network proposed in the present invention. DETAILED DESCRIPTION

[0084] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0085] refer to Figure 1-2 , a method for automatic detection and repair of road damage based on generative adversarial network, comprising the following steps:

[0086] S1, obtaining a road image dataset in real time through a camera or other image acquisition device;

[0087] S2, performing image preprocessing on the collected road image dataset, including denoising, enhancement and histogram equalization processing;

[0088] S3. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model consists of three parts: generator, discriminator, and explanation module.

[0089] S4. Using the discriminator part of the generative adversarial network model with enhanced interpretability, damage detection is performed on the preprocessed road image dataset to identify various types of damage in the road image dataset, including potholes, cracks, and settlements. Combined with the explanation information generated by the explanation module, the specific location of the damaged area and its judgment reasons are output;

[0090] S5. Use image segmentation technology to locate the detected damaged area, generate a mask map of the damaged area, and combine GPS positioning technology to mark the specific geographic coordinates of the damaged location;

[0091] S6. Generate a repaired road image based on the detected damaged area using the generator part of the generative adversarial network, and formulate a specific repair plan based on the information output by the interpretation module, including the required materials, repair methods and construction steps, and provide the basis and explanation for generating the repair plan;

[0092] S7, transmitting the generated repair plan and its explanation information to the automated construction equipment to automatically perform the repair task;

[0093] S8. Utilize the self-learning ability of the generative adversarial network to adapt to different road environments and damage types during the continuous detection and repair process, and optimize the parameters of the generative adversarial network model.

[0094] In this implementation, S1 includes the following sub-steps:

[0095] S11, through the camera or other image acquisition equipment deployed on the vehicle or fixed position, with a fixed sampling frequency f s Continuously collect road images to form an image sequence:

[0096] I={I1,I2,,I n};

[0097] Among them, I i Indicates that at time t i The i-th road image collected, n is the total number of images collected;

[0098] S12, for each collected image I i , record the corresponding timestamp t i and the geographic coordinates of the image acquisition device (x i ,y i ), where x i and i They are respectively the image acquisition devices at the acquisition time t i longitude and latitude;

[0099] S13, compare the acquired image sequence I with the corresponding timestamp sequence T = {t1, t2,, t n} and the geographic coordinate sequence G = {(x1,y1),(x2,y2),,(x n ,y n )} to generate a road image dataset with time and space annotations:

[0100] D={(I1,t1,(x1,y1)),(I2,t2,(x2,y2)),,(I n ,t n ,(x n ,y n ))}.

[0101] In this implementation, S3 includes the following sub-steps:

[0102] S31. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model includes a generator G(z) and a discriminator D disc (D i ) and interpretation module E(D disc (D i ))Three parts:

[0103] The generator G(z) takes the noise vector z as input and generates the repaired road image G(z) through a multi-layer convolutional neural network, where z follows a Gaussian distribution z~N(0,1);

[0104] G(z)=Deconv(W G ReLU(BN(W z ·z+b z )));

[0105] Among them, W G is the weight matrix of the generator, Deconv represents the deconvolution operation, ReLU is the activation function, BN represents the batch normalization operation, W z and b z are the weight and bias of the input noise vector respectively;

[0106] Discriminator D disc (D i ) Take a single image D in the road image dataset i As input, the convolutional neural network outputs the discriminant probability, which is used to distinguish whether the input image is from the real dataset D i ~p data (D i ) or the image D generated by the generator i ~p G (D i );

[0107] Explanation module E(D disc (D i )) and the discriminator D disc (D i ) integration, by analyzing the activation pattern of the convolutional layer inside the discriminator Act l (D disc (D i )), generating explanation information E(D) about why the GAN model identified a certain area as a damaged area disc (D i ), the interpretation information includes parameters such as the activation layer position of the damaged area and the activation intensity of the damage type, and introduces an adaptive interpretation mechanism to dynamically adjust the interpretation strategy of the interpretation module according to different environments and damage types. E ;

[0108]

[0109] in, is the gradient of the discriminator at layer l, α l To explain the module's adaptive coefficient, Act l (D disc(D i )) is the activation value of the lth layer, S E To explain the strategy;

[0110] S32. Train the generative adversarial network model:

[0111] Using the loss function L D and L G For the discriminator D disc (D i ) and the generator G(z) for adversarial training, the loss function of the discriminator is defined as:

[0112]

[0113] The loss function of the generator is defined as:

[0114]

[0115] Among them, λ is a hyperparameter that weighs the generation loss and restoration accuracy, and D GT is the real road image, MSE represents the mean square error between the generated image and the real image;

[0116] Explanation module E(D disc (D i )) is updated after each discriminator update to reflect the latest discrimination results and decision basis, and dynamically optimizes the explanation strategy S through the adaptive explanation mechanism E ;

[0117]

[0118] Among them, L E is the loss function of the explanation module, η is the learning rate, and are the interpretation strategies at the tth and t+1th iterations respectively;

[0119] S33, after training is completed, the adversarial network model {G(z),D disc (D i ),E(D disc (D i ))}Save as the final detection and repair model.

[0120] In this embodiment, the interpretation module E(D disc (D i The explanation information generation step in )) includes the following sub-steps:

[0121] In the discriminator D disc (D i ) Inside, perform activation analysis on each convolutional layer l and extract the activation value Act of the layerl (D disc (D i ));

[0122] Calculate the gradient of each convolutional layer The gradient represents the sensitivity of the discriminator to the activation value, which is calculated by calculating the discriminator output D disc (D i ) relative to the activation value Act l (D disc (D i ))'s partial derivative is obtained:

[0123]

[0124] Introducing adaptive coefficient α l Adjust the contribution of each layer's activation value to the final interpretation information, and dynamically adjust the interpretation strategy of the interpretation module based on damage type and environmental factors;

[0125] Explanation information E(D disc (D i )) is summarized to generate interpretation information including the activation layer position of the damaged area and the activation intensity parameters of the damage type, and the interpretation information is passed to the repair plan generation module.

[0126] In this implementation, S4 includes the following sub-steps:

[0127] S41. Discriminator D in Generative Adversarial Network Model with Interpretability Enhancement disc (D i ), for the preprocessed road image dataset D′={D′1,D′2,…,D′ n}Analyze each image one by one to identify the single image D′ in the road image dataset i Whether the damaged area is included;

[0128] S42, for each image D′ i When performing damage detection, the discriminator D disc (D i ) Output damage probability P damage (D′ i ), the damage probability is obtained by convolution processing of the discriminator on the image, and combined with the interpretation module E(D disc (D i )) generates interpretation information to determine the specific location of the damaged area Loc(D′) i ) and its damage type Type(D′ i ):

[0129] P damage (D′ i )=Sigmoid(Ddisc (D i ));

[0130]

[0131] Among them, P damage (D′ i ) is the damage probability, which means that for image D′ i For example, the probability of being detected as damaged is directly related to whether the image contains road damage. A high P damage The value means that the model has a higher confidence that there is damage in the image, Sigmoid represents the Sigmoid activation function, and Loc(D′ i ) indicates the specific location of the damaged area, marking the i In the above example, the precise geographic location of the damaged area, Region is the candidate damaged area, the argmax operation is used to select the location of the damaged area with the largest activation intensity, and the Classify operation is the classification process for determining the damage type, combining the activation intensity and gradient information generated by the interpretation module;

[0132] S43, output the damage detection result, including the specific location of the damaged area Loc (D′ i ), damage type Type(D′ i ) and its judgment reason R(D′ i ):

[0133] R(D′ i )=(Loc(D′ i ),Type(D′ i ),E(D disc (D i )));

[0134] Among them, the reason for judgment R(D′ i ) integrates the spatial location of the damaged area, type classification and interpretation information of the interpretation modules.

[0135] In this implementation, S5 includes the following sub-steps:

[0136] S51, detect the damaged area Loc (D′ i ) Use the convolutional neural network based segmentation algorithm to segment the image and remove the damaged area from the image D′ i Extract it and generate the mask image M(D′) of the damaged area i ):

[0137] M(D′ i )=Segment(Loc(D′ i ),D′ i );

[0138] Among them, Segment represents the image segmentation operation, M(D′ i ) is the generated damaged area mask image;

[0139] S52, generate the mask image M(D′ i ) Calculate the bounding box B(D′) of the damaged area i ) and the center point coordinates C(D′ i ) to determine the damaged area;

[0140] S53, combining GPS positioning technology to obtain image D′ i The collection time t′ i The corresponding geographic coordinates (x′ i ,y′ i ), the coordinates of the center point of the damaged area C(D′ i ) and geographic coordinates (x′ i ,y′ i ) to mark the specific geographic coordinates G(D′) of the damaged location i ):

[0141] G(D′ i )=(x′ i +Δx(C(D′ i )),y′ i +Δy(C(D′ i )));

[0142] Among them, Δx(C(D′ i ) and Δy(C(D′ i )) are the lateral and longitudinal offsets of the center point of the damaged area relative to the position of the image acquisition device, G(D′ i ) are the specific geographic coordinates of the marked damage location.

[0143] In this implementation, S6 includes the following sub-steps:

[0144] S61, using the generator G(z) in the generative adversarial network, input the mask image M(D′) of the detected damaged area i ), and the original image D′ i Combined together, the restored image D is generated by the generator repair (D′ i ):

[0145] D repair (D′ i )=Deconv(W G ReLU(BN(W M ·M(D′ i )+WD ·D′ i +b G )));

[0146] Among them, W G is the weight matrix of the generator, W M and W D are the weight matrices of the mask image and the original image, respectively, G is the bias vector, BN represents batch normalization operation, ReLU is the activation function, and Deconv represents deconvolution operation;

[0147] S62, combined with the damaged area information E(D′) output by the interpretation module i ) and the generated restored image D repair (D′ i ), formulate a specific repair plan P(D′) i ), the repair plan includes the required materials, repair methods and construction steps:

[0148]

[0149] Among them, Mat k represents the kth type of repair material, Action k represents the kth construction step, is the weight matrix of the repair solution, is the bias vector of the repair solution, Conv represents the convolution operation;

[0150] S63, repair plan P(D′ i ) provides the basis and explanation for the generation, combining the output of the generator G(z) and the explanation information of the explanation module to describe the specific reasons and expected effects of each step in the repair plan:

[0151]

[0152] Among them, β m To explain the adaptive coefficient of the module, is the gradient information of the mth step in the repair scheme, γ m is the influence coefficient of the construction step, A collection of steps for a repair solution.

[0153] Embodiment 1:

[0154] In Example 1, the present invention was applied to a road maintenance project in May 2024. The municipal engineering department selected East Ring Road, a main road in the eastern part of the city, as a test scenario for the method of the present invention. The road is about 15 kilometers long, with an average daily traffic volume of more than 100,000 times. Due to the frequent passage of heavy-loaded vehicles for a long time, many potholes, cracks and settlement problems have appeared on the road surface.

[0155] At 8 a.m. on May 10, 2024, the municipal engineering department dispatched three inspection vehicles equipped with high-precision cameras and GPS positioning systems to conduct a comprehensive inspection of Donghuan Road. The vehicles traveled at a constant speed of 20 kilometers per hour and collected road images at a rate of 2 frames per second. During the entire inspection process, a total of approximately 540,000 road images were collected. The data was uploaded to the cloud processing center in real time. During the upload process, the system automatically denoised, enhanced, and histogram-equalized the images to ensure that the image quality was sufficient for high-precision damage detection.

[0156] In the cloud processing center, the system analyzes the uploaded road image data set one by one based on the generative adversarial network model. At 2 pm on May 10, 2024, the system detected obvious cracks and potholes at the 7th kilometer of the East Ring Road. The specific geographic coordinates of the damaged area were 31.2304 degrees latitude and 121.4737 degrees longitude. Through discriminator analysis, the crack width of the damaged area is about 3 cm, the length is about 1.5 meters, the pothole depth is about 5 cm, and the area is about 0.8 square meters. The system generates a detailed damage report based on the detection results, and marks the specific location and type of the damaged area. The report lists the detection time, the geographic coordinates of the damaged area, the type of damage, and the specific degree of damage.

[0157] At 9:00 a.m. on May 11, 2024, the municipal engineering department received a repair plan generated by the system. The plan generated an image of the repaired road through the generator part of the generative adversarial network, and combined with the damage type and environmental factors, proposed suggestions for the use of repair materials, repair methods and specific construction steps. Specifically, the system recommends the use of C30 concrete to fill potholes, and recommends the use of hot asphalt mixture to repair cracked areas. It also recommends 24-hour maintenance of the repair area after construction is completed. The system also generated a detailed description of the repair steps, including the ratio of materials, temperature control of construction, the use of construction machinery, and the inspection process after construction is completed.

[0158] In order to verify the effectiveness of the method of the present invention, the municipal engineering department conducted comparative tests on the East Ring Road using the method of the present invention and the traditional manual method. First, in the damaged area at the 7th kilometer, the construction team carried out construction according to the repair plan generated by the system, and the entire repair process was completed at 3 pm on May 12, 2024. During the construction process, the system monitored the progress of the repair in real time, and conducted a secondary test after the repair was completed to confirm that the repair effect was basically consistent with the repaired image generated by the system. The test data after the repair showed that the flatness error of the potholes and cracks was within 1 mm, which met the national road maintenance standards.

[0159] At the same time, the municipal engineering department used traditional methods to inspect and repair the 14th kilometer of the East Ring Road. The manual inspection team detected the damage in the area at 3 pm on May 10, 2024, and the inspection process took about 1 hour and 15 minutes. After detecting the damage, the construction team started the repair work at 9 am on May 13, 2024 and completed it at 5 pm on the same day. However, the post-repair inspection showed that the flatness error of the area was about 3 mm, and there were 2 cracks that still had obvious traces after the repair, requiring secondary repair.

[0160] By comparing the data, we can see that the method of the present invention is significantly superior to the traditional method in both detection efficiency and repair quality. In terms of detection efficiency, the traditional method takes about 1 hour and 15 minutes, while the method of the present invention completes the same range of detection within 2 seconds; in terms of repair quality, the flatness error of the method of the present invention is within 1 mm, which meets the standard, while the flatness error of the traditional method reaches 3 mm and requires secondary repair. In addition, the method of the present invention does not require manual participation in the generation and adjustment of the repair plan during the repair process. The entire process is highly automated, effectively reducing errors caused by human factors.

[0161] The results of this test show that the method of the present invention can not only greatly improve the efficiency of road damage detection and repair, but also significantly improve the quality of repair, ensuring the safety and durability of the repaired road. This achievement has laid a solid foundation for the application of the method of the present invention in larger-scale road maintenance in the future.

[0162] The present invention uses a generative adversarial network model built on the TensorFlow framework. The discriminator can accurately identify various types of road damage, such as potholes, cracks, and subsidence, in complex and changeable road environments. The generative adversarial network model uses the generator to generate repaired images. Through adversarial training of the discriminator, the model can maintain stable detection performance when facing different lighting, weather conditions, and road conditions. In addition, the integration of the discriminator and the interpretation module can output detailed interpretation information, increase the credibility of the detection results, and significantly improve the accuracy and robustness of damage detection.

[0163] The present invention utilizes the generator part in the generative adversarial network to generate a repaired road image based on the detected damaged area, and combines the detailed interpretation information generated by the interpretation module to formulate a specific repair plan, which not only includes the required materials, repair methods and construction steps, but also automatically optimizes the algorithm to ensure the efficiency of the repair process and the consistency of the effect. The generation of the repair plan is based on the actual situation and environmental parameters of the damaged area. Through the information feedback of the multi-layer convolutional neural network and the interpretation module, the repair decision formed has a high level of intelligence, can be adjusted and optimized in real time, reduces the need for manual intervention, and improves the repair efficiency.

[0164] The present invention combines image segmentation technology with GPS positioning technology to accurately locate the detected damaged area, generate a mask map of the damaged area and mark its specific geographic coordinates, which not only ensures the spatial accuracy of damage detection, but also provides reliable data support for subsequent repair work. During the repair process, using the generated repair plan, the system can automatically perform the repair task, and conduct real-time monitoring and adjustment through the feedback mechanism to ensure that the repair quality meets the expected standards, realizing fully automated management from detection to repair, and significantly improving the efficiency of road maintenance.

[0165] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for automatic detection and repair of road damage based on generative adversarial networks, characterized in that: The steps include: S1, obtaining a road image dataset in real time through a camera or other image acquisition device; S2, performing image preprocessing on the collected road image dataset, including denoising, enhancement and histogram equalization processing; S3. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model consists of three parts: generator, discriminator, and explanation module. S4. Using the discriminator part of the generative adversarial network model with enhanced interpretability, damage detection is performed on the preprocessed road image dataset to identify various types of damage in the road image dataset, including potholes, cracks, and settlements. Combined with the explanation information generated by the explanation module, the specific location of the damaged area and its judgment reasons are output; S5. Use image segmentation technology to locate the detected damaged area, generate a mask map of the damaged area, and combine GPS positioning technology to mark the specific geographic coordinates of the damaged location; S6. Generate a repaired road image based on the detected damaged area using the generator part of the generative adversarial network, and formulate a specific repair plan based on the information output by the interpretation module, including the required materials, repair methods and construction steps, and provide the basis and explanation for generating the repair plan; S7, transmitting the generated repair plan and its explanation information to the automated construction equipment to automatically perform the repair task; S8. Utilize the self-learning ability of the generative adversarial network to adapt to different road environments and damage types during the continuous detection and repair process, and optimize the parameters of the generative adversarial network model.

2. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 1, characterized in that: The S1 comprises the following sub-steps: S11, through the camera or other image acquisition equipment deployed on the vehicle or fixed position, with a fixed sampling frequency f s Continuously collect road images to form an image sequence: I={I1,I2,,I n }; Among them, I i Indicates that at time t i The i-th road image collected, n is the total number of images collected; S12, for each collected image I i , record the corresponding timestamp t i and the geographic coordinates of the image acquisition device (x i ,y i ), where x i and i They are respectively the image acquisition devices at the acquisition time t i longitude and latitude; S13, compare the acquired image sequence I with the corresponding timestamp sequence T = {t1, t2,, t n } and the geographic coordinate sequence G = {(x1,y1),(x2,y2),,(x n ,y n )} to generate a road image dataset with time and space annotations: D={(I1,t1,(x1,y1)),(I2,t2,(x2,y2)),,(I n ,t n ,(x n ,y n ))}。 3. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 1, characterized in that: The S3 comprises the following sub-steps: S31. Build a generative adversarial network model based on the TensorFlow framework. The generative adversarial network model includes a generator G(z) and a discriminator D disc (D i ) and interpretation module E(D disc (D i ))Three parts: The generator G(z) takes the noise vector z as input and generates the repaired road image G(z) through a multi-layer convolutional neural network, where z follows a Gaussian distribution z~N(0,1); G(z)=Deconv(W G ·ReLU(BN(W z ·z+b z ))); Among them, W G is the weight matrix of the generator, Deconv represents the deconvolution operation, ReLU is the activation function, BN represents the batch normalization operation, W z and b z are the weight and bias of the input noise vector respectively; Discriminator D disc (D i ) Take a single image D in the road image dataset i As input, the convolutional neural network outputs the discriminant probability, which is used to distinguish whether the input image is from the real dataset D i ~p data 9D i ) or the image D generated by the generator i ~p G (D i ); Explanation module E(D disc (D i )) and the discriminator D disc (D i ) integration, by analyzing the activation pattern of the convolutional layer inside the discriminator Act l (D disc (D i )), generating explanation information E(D) about why the GAN model identified a certain area as a damaged area disc (D i ), the interpretation information includes parameters such as the activation layer position of the damaged area and the activation intensity of the damage type, and introduces an adaptive interpretation mechanism to dynamically adjust the interpretation strategy of the interpretation module according to different environments and damage types. E ; in, is the gradient of the discriminator at layer l, α l To explain the module's adaptive coefficient, Act l (D disc (D i )) is the activation value of the lth layer, S E To explain the strategy; S32. Train the generative adversarial network model: Using the loss function L D and L G For the discriminator D disc (D i ) and the generator G(z) for adversarial training, the loss function of the discriminator is defined as: The loss function of the generator is defined as: Among them, λ is a hyperparameter that weighs the generation loss and restoration accuracy, and D GT is the real road image, MSE represents the mean square error between the generated image and the real image; Explanation module E(D disc (D i )) is updated after each discriminator update to reflect the latest discrimination results and decision basis, and dynamically optimizes the explanation strategy S through the adaptive explanation mechanism E ; Among them, L E is the loss function of the explanation module, η is the learning rate, and are the interpretation strategies at the tth and t+1th iterations respectively; S33, after training is completed, the adversarial network model {G(z),D disc (D i ),E(D disc (D i ))}Save as the final detection and repair model.

4. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 3 is characterized in that: The interpretation module E(D disc (D i The explanation information generation step in )) includes the following sub-steps: In the discriminator D disc (D i ) Inside, perform activation analysis on each convolutional layer l and extract the activation value Act of the layer l (D disc (D i )); Calculate the gradient of each convolutional layer The gradient represents the sensitivity of the discriminator to the activation value, which is calculated by calculating the discriminator output D disc (D i ) relative to the activation value Act l (D disc (D i ))'s partial derivative is obtained: Introducing adaptive coefficient α l Adjust the contribution of each layer's activation value to the final interpretation information, and dynamically adjust the interpretation strategy of the interpretation module based on damage type and environmental factors; Explanation information E(D disc (D i )) is summarized to generate interpretation information including the activation layer position of the damaged area and the activation intensity parameters of the damage type, and the interpretation information is passed to the repair plan generation module.

5. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 1, characterized in that: The S4 comprises the following sub-steps: S41. Discriminator D in Generative Adversarial Network Model with Interpretability Enhancement disc (D i ), for the preprocessed road image dataset D′={D′1,D′2,…,D′ n }Analyze each image one by one to identify the single image D′ in the road image dataset i Whether the damaged area is included; S42, for each image D′ i When performing damage detection, the discriminator D disc (D i ) Output damage probability P damage (D′ i ), the damage probability is obtained by convolution processing of the discriminator on the image, and combined with the interpretation module E(D disc (D i )) generates interpretation information to determine the specific location of the damaged area Loc(D′) i ) and its damage type Type(D′ i ): P damage (D′ i )=Sigmoid(D disc (D i )); Among them, P damage (D′ i ) is the damage probability, which means that for image D′ i For example, the probability of being detected as damaged is directly related to whether the image contains road damage. A high P damage The value means that the model has a higher confidence that there is damage in the image, Sigmoid represents the Sigmoid activation function, and Loc(D′ i ) indicates the specific location of the damaged area, marking the i In the above example, the precise geographic location of the damaged area, Region is the candidate damaged area, the argmax operation is used to select the location of the damaged area with the largest activation intensity, and the Classify operation is the classification process for determining the damage type, combining the activation intensity and gradient information generated by the interpretation module; S43, output the damage detection result, including the specific location of the damaged area Loc (D′ i ), damage type Type(D′ i ) and its judgment reason R(D′ i ): R(D′ i )=(Loc(D′ i ),Type(D′ i ),E(D disc (D i ))); Among them, the reason for judgment R(D′ i ) integrates the spatial location of the damaged area, type classification and interpretation information of the interpretation modules.

6. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 1, characterized in that: The S5 comprises the following sub-steps: S51, detect the damaged area Loc (D′ i ) Use the convolutional neural network based segmentation algorithm to segment the image and remove the damaged area from the image D′ i Extract it and generate the mask image M(D′) of the damaged area i ): M(D′ i )=Segment(Loc(D′ i ),D′ i ); Among them, Segment represents the image segmentation operation, M(D′ i ) is the generated damaged area mask image; S52, generate the mask image M(D′ i ) Calculate the bounding box B(D′) of the damaged area i ) and the center point coordinates C(D′ i ) to determine the damaged area; S53, combining GPS positioning technology to obtain image D′ i The collection time t′ i The corresponding geographic coordinates (x′ i ,y′ i ), the coordinates of the center point of the damaged area C(D′ i ) and geographic coordinates (x′ i ,y′ i ) to mark the specific geographic coordinates G(D′) of the damaged location i ): G(D′ i )=(x′ i +Δx(C(D′ i )),y′ i +Δy(C(D′ i ))); Among them, Δx(C(D′ i ) and Δy(C(D′ i )) are the lateral and longitudinal offsets of the center point of the damaged area relative to the position of the image acquisition device, G(D′ i ) are the specific geographic coordinates of the marked damage location.

7. The method for automatic detection and repair of road damage based on generative adversarial networks according to claim 1, characterized in that: The S6 comprises the following sub-steps: S61, using the generator G(z) in the generative adversarial network, input the mask image M(D′) of the detected damaged area i ), and the original image D′ i Combined together, the restored image D is generated by the generator repair (D′ i ): D repair (D′ i )=Deconv(W G ·ReLU(BN(W M ·M(D′ i )+W D ·D′ i +b G ))); Among them, W G is the weight matrix of the generator, W M and W D are the weight matrices of the mask image and the original image, respectively, G is the bias vector, BN represents batch normalization operation, ReLU is the activation function, and Deconv represents deconvolution operation; S62, combined with the damaged area information E(D′) output by the interpretation module i ) and the generated restored image D repair (D′ i ), formulate a specific repair plan P(D′) i ), the repair plan includes the required materials, repair methods and construction steps: Among them, Mat k represents the kth type of repair material, Action k represents the kth construction step, is the weight matrix of the repair solution, is the bias vector of the repair solution, Conv represents the convolution operation; S63, repair solution P(D′ i ) provides the basis and explanation for the generation, combining the output of the generator G(z) and the explanation information of the explanation module to describe the specific reasons and expected effects of each step in the repair plan: Among them, β m To explain the adaptive coefficient of the module, is the gradient information of the mth step in the repair scheme, γ m is the influence coefficient of the construction step, A collection of steps for a repair solution.

Images