A highway crack image segmentation method based on deep reinforcement learning

Abstract

This invention discloses a highway crack image segmentation method based on deep reinforcement learning. 1) The highway crack image is preprocessed to improve its contrast; 2) A deep learning-based segmentation network algorithm is used to obtain the segmentation result of the highway crack image; 3) The segmentation result output by the segmentation network is set as the initial state of the DoubleDQN algorithm; a pixel value range [100, 255] is set as the action space of the agent in deep reinforcement learning, which also serves as the reference point for adjusting the pixel values ​​of the segmentation result; 4) The coarse segmentation result of the crack image is iteratively optimized using the DoubleDQN algorithm; 5) After obtaining the optimal segmentation result, the segmentation performance of the algorithm is evaluated using the evaluation metrics IoU, Recall, and F1 score. This invention optimizes the segmentation result. Compared with the segmentation result obtained by simply using deep learning methods, the crack segmentation result after iterative optimization using the method of this invention is more refined, which greatly improves the accuracy of highway health status assessment.

Description

Technical Field

[0001] This invention relates to the fields of image processing and intelligent algorithm technology, specifically to a method for segmenting highway crack images based on deep reinforcement learning. Background Technology

[0002] Highways are a crucial part of national infrastructure. However, due to factors such as overloading, aging, and thermal expansion, road surfaces often develop cracks, posing significant safety hazards to the transportation industry. The segmentation and identification of highway cracks presents the following challenges:

[0003] 1) There is a lack of highway crack datasets, especially high-quality labeled data.

[0004] 2) Highway pavement has complex textures, and the cracks have low contrast with the pavement background. The resulting highway cracks are diverse in shape and size, which poses a challenge to crack segmentation.

[0005] Traditional image processing-based segmentation methods are easily affected by noise and background. Deep learning-based segmentation methods perform better than traditional methods, but due to the final output of the network, the results cannot completely converge to the target value, and often need to be adjusted by thresholding.

[0006] Deep reinforcement learning is an algorithm that combines deep learning and reinforcement learning. It constructs a Markov decision process, allowing the agent to continuously interact with the environment to guide it in making actions that maximize benefits under different states, thereby obtaining the optimal decision. This invention employs a deep reinforcement learning algorithm to iteratively optimize the segmentation results of highway cracks by adjusting the threshold in the segmentation results. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a highway crack image segmentation method based on deep reinforcement learning. This method refines the segmentation results of highway cracks and continuously iterates and optimizes the results using a deep reinforcement learning algorithm, achieving better segmentation results than methods based solely on deep learning. The deep reinforcement learning method constructs a Markov decision process for the current segmentation result. Through each iteration, it uses the IoU value of the segmentation result to guide the agent in optimizing the segmentation result, ultimately achieving better performance in terms of IoU metric and F1 score.

[0008] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0009] A method for segmenting highway crack images based on deep reinforcement learning includes the following steps:

[0010] Step 1: Preprocess the labeled highway crack images. Considering the low contrast and complex texture of highway images, the ACE algorithm is used for preprocessing to improve the contrast between the cracks and the background. For the preprocessed images, the mean and variance of the entire dataset are calculated, and the dataset is standardized using the mean and variance.

[0011] Step 2: Build a suitable deep learning model to train the preprocessed highway crack image. After training, use the model to perform forward inference on the highway crack image to obtain a rough crack segmentation result.

[0012] Step 3: Construct a Markov decision process, which is the state value, action value, and reward value required in deep reinforcement learning;

[0013] Step 4: Use appropriate deep learning networks as the value network and target network in deep reinforcement learning; take the state values ​​as the input of the deep Q network, and take the Q values ​​corresponding to all thresholds in the threshold action set A as the output; randomly draw some samples from the experience pool, calculate the parameters of the value network through the update formula of the Double DQN algorithm, and update the parameters of the value network to the target Q network at the appropriate time;

[0014] Step 5: During the iteration of Double DQN, the experience pool is continuously filled with sample data explored by the agent; deep reinforcement learning will randomly draw some samples from the experience pool and calculate the target Q value using the update formula of the Double DQN algorithm:

[0015] Q target =R t+1 +γQ(S t+1 ,argmaxQ(S t+1 ,a t+1 ;w);w') (1)

[0016] Among them, S t+1 R represents the state after an action is performed. t+1 Let represent the reward for performing action 'a', w represent the parameters of the estimation network, w' represent the parameters of the target network, γ represent the reward discount factor, and t represent time 'a'. t+1 This represents the action at time t+1;

[0017] Step 6: Calculate the loss value of the deep Q network output; this loss is obtained based on the mean square error between the target Q value obtained in Step 5 and the Q value output by the deep Q network; at the same time, the weight parameters in the deep Q network are randomly initialized using a truncated normal distribution, and the final output layer of the network uses Softmax;

[0018]

[0019] Where B represents the number of samples in one iteration, and i represents the sample number.

[0020] Furthermore, the Markov decision process constructed in step 3, i.e., the state values, action values, and reward values ​​required in deep reinforcement learning, are as follows:

[0021] Step 3-1: Set the segmentation result at time T as the state value of deep reinforcement learning, while the initial state of the algorithm is the image segmentation result output by the segmentation network model in the first stage, which is also the input of the Double DQN network;

[0022] Step 3-2: Set the action space of the agent, which is also the threshold set A that needs to be adjusted for image segmentation. Select the corresponding action according to the ε-greedy strategy: Select an integer from the action set A as the benchmark for adjusting the segmentation result. If the selected action is a... i Then the pixel values ​​in the segmented image will be in [a i -5,a i The pixel value of +5 is set to 255;

[0023] Step 3-3: Set the IoU difference before and after taking the action as the reward for the environment in deep reinforcement learning; if the IoU value of the segmentation map after taking the action is greater than the IoU before taking the action, the reward is set to 1; if the IoU of the segmentation map before and after taking the action is the same, the reward is set to 0; if the IoU value of the segmentation map after taking the action is less than the IoU before taking the action, the reward is set to -1.

[0024] Furthermore, the experience pool in step 4 consists of samples obtained by the agent of the Double DQN algorithm interacting with the environment at each time step. These samples contain four parts: the state S at time t. t Action a is taken, reward R is given by the environment, and state S is formed after taking action a. t+1 .

[0025] Furthermore, the reward in the Markov decision process used is formulated as follows:

[0026]

[0027] Furthermore, the constructed deep Q-network is a classification network, where the number of neurons in the last layer equals the size of the action space. As a network model for the action value function, the deep Q-network uses weight parameters w, and Q(s,a;w) simulates the action value function Q*(s,a), where s represents the current state and a represents the current action.

[0028] Q * (s,a)=Q(s,a;w) (4).

[0029] The Adam optimizer is used for gradient updates during the training of the deep Q network. After the deep Q network is trained, the segmentation performance of the algorithm is evaluated using the segmentation algorithm evaluation metrics IoU and F1-score.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] This invention greatly optimizes the segmentation results. Compared with the segmentation results obtained by simply using deep learning methods, the crack segmentation results after iterative optimization using the deep reinforcement learning method of this invention are more refined, which greatly improves the accuracy of highway health status assessment. Attached Figure Description

[0032] Figure 1 This is a diagram of the overall architecture of the two-stage segmentation in this invention.

[0033] Figure 2 The images show a before-and-after comparison of images preprocessed using the ACE algorithm: from left to right, the original image, the image after ACE processing, and the ground truth of the original image.

[0034] Figure 3 The improvement in IoU on the test set after using the method provided by this invention: the horizontal axis represents the sample number, and the vertical axis represents the difference in IoU between the segmentation results using DoubleDQN and those not using DoubleDQN.

[0035] Figure 4 The confusion matrix calculated on the test set for this invention is shown on the left: the left side represents the method used in deep learning, and the right side represents the method of this invention.

[0036] Figure 5 Visualization of the segmentation results in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0038] This invention provides a method for segmenting highway crack images based on deep reinforcement learning. This method uses Double DQN (Depth Reinforcement Learning) to iteratively optimize the coarse segmentation results of the neural network, thus refining the segmentation results and improving the segmentation accuracy of crack regions. Figure 1 As shown, the present invention consists of two processing stages: the first stage uses a deep learning-based segmentation algorithm to obtain a coarse segmentation result image of a highway crack image; the second stage uses the Double DQN algorithm to iteratively optimize the coarse segmentation result image of the neural network.

[0039] Step 1: Preprocess the labeled highway crack images. Considering the low contrast and complex texture of highway images, the ACE algorithm is used for preprocessing to improve the contrast between the cracks and the background. For the preprocessed images, the mean and variance of the entire dataset are calculated, and the dataset is standardized using the mean and variance.

[0040] The ACE algorithm corrects the final pixel value by calculating the relative brightness relationship between the target point and its surrounding pixels using difference, resulting in a good enhancement effect. Assuming I is a single-channel image normalized to [0,1], the calculation formula for the ACE algorithm is:

[0041]

[0042] Where Ω represents {y∈Ω,y≠x}, d(x,y) represents the distance metric function, which can be directly taken as Euclidean distance. Where I(x) represents the pixel value at the current position in the image, I(y) represents the pixel values ​​at other positions in the image, Ω represents the entire pixel set of the image, and S(*) represents the slope function, which can be taken as:

[0043] S(x)=max(min(αx,1.0),-1.0) (6)

[0044] Here, α is a control parameter, and a larger value indicates a more significant enhancement of details. Next, normalizing R(x) yields the final result, as shown in formula (7). Here, R(x) represents the pixel value in the image after processing by the ACE algorithm.

[0045]

[0046] Step 2: Construct a suitable deep learning model to train on the preprocessed highway crack images. After training, use the model to perform forward inference on the highway crack images to obtain coarse crack segmentation results. Here, the AttentionUNet segmentation network model is used as an example.

[0047] Step 3: Construct the state values, action values, and reward values ​​required for deep reinforcement learning.

[0048] The segmentation result at time T is set as the state value for deep reinforcement learning, while the initial state of the algorithm is the image segmentation result output by AttentionUNet. This state is also the input to the Double DQN network.

[0049] The action space of the agent is set to [100-255], resulting in 156 actions. These actions represent the pixel values ​​in the segmentation image that the agent can choose, meaning the action space consists of discrete pixel values. The action space of [100-156] also represents the threshold set A that needs to be adjusted for segmenting the image. Based on the ε-greedy strategy, appropriate actions are selected to iteratively optimize the coarse segmentation result of the highway crack image: an integer is selected from the action set A as the benchmark for adjusting the segmentation result. If the selected action is a... i Then the pixel values ​​in the segmented image will be in [a i -5,a i The pixel value of +5 is set to 255.

[0050] The difference in IoU before and after taking an action is used as the reward for the environment in deep reinforcement learning. If the IoU of the segmentation map after taking the action is greater than the IoU before taking the action, the reward is set to 1; if the IoU of the segmentation map before and after taking the action is the same, the reward is set to 0; if the IoU of the segmentation map after taking the action is less than the IoU before taking the action, the reward is set to -1. The formula for the reward is as follows:

[0051]

[0052] Step 4: Use ResNet18 as the value network and target network in the deep reinforcement learning algorithm. The state values ​​(initial state is the output of Step 2) are used as input to the deep Q-network, and the Q-values ​​corresponding to all thresholds in the threshold action set A are used as output. A subset of samples is randomly drawn from the experience pool, and the parameters of the value network are calculated using the update formula of the Double DQN algorithm. The parameters of the value network are then updated to the target Q-network at appropriate times.

[0053] Using ResNet18, also known as a deep Q-network in deep reinforcement learning, as the network model for the action value function, with network weights w, and Q(s,a;w) simulating the action value function Q*(s,a), where s represents the current state and a represents the current action, i.e., w = w * ...

[0054] Q * (s,a)=Q(s,a;w) (4)

[0055] The aforementioned experience pool consists of samples obtained by the agent of the Double DQN algorithm interacting with the environment at each time step. These samples comprise four parts: the state S at time t. t Action a is taken, reward R is given by the environment, and state S is formed after taking action a. t+1 .

[0056] Step 5: During the iteration of Double DQN, the experience pool is continuously filled with sample data explored by the agent. Deep reinforcement learning will randomly select a portion of samples from the experience pool and calculate the target Q-value using the update formula of the Double DQN algorithm:

[0057] Q target =R t+1 +γQ(S t+1 ,argmaxQ(S t+1 ,a;w);w') (1)

[0058] Among them, S t+1 R represents the state after an action is performed. t+1 Let represent the reward for performing action 'a', w represent the parameters of the estimation network, and w' represent the parameters of the target network. γ represents the reward discount factor; t represents time 'a'. t+1 This represents the action at time t+1.

[0059] Step 6: Calculate the loss value of the Deep Q-Network output. This loss is obtained based on the mean squared error between the target Q-value obtained in Step 5 and the Q-value output by the Deep Q-Network; simultaneously, the weight parameters in the Deep Q-Network are randomly initialized using a truncated normal distribution, and the final output layer of the network uses Softmax. The Adam optimizer is used for gradient updates during the training of the Deep Q-Network. In the loss formula, B represents the number of samples in one iteration, and i represents the sample number.

[0060]

[0061] Example

[0062] In a specific implementation example of this invention, the CFD (Crack Forest Dataset) dataset is used as the research object. In the first stage, the AttentionUNet network model is used as an image segmentation algorithm to obtain coarse segmentation results of crack images in the CFD dataset; in the second stage, the Double DQN algorithm is used to iteratively optimize the coarse segmentation results of the AttentionUNet network, and the deep Q network used in this stage is the ResNet18 network model.

[0063] 1. Highway crack images from the CFD dataset are preprocessed, standardized using a uniform mean and standard deviation, and the ACE algorithm is used to improve image contrast. The specific effects before and after preprocessing can be seen in [link to preprocessing documentation]. Figure 2 .

[0064] 2. Train AttentionUNet using the preprocessed crack image data, and then use the trained model to obtain the coarse segmentation results of the crack images in the dataset.

[0065] 3. As described in step 2, the segmentation result image is taken as the state. The image resolution in the CFD dataset is 448*448, so the dimension of the state is 448*448*1.

[0066] 4. In this example, the action space is [100-255]. This action space is also the threshold set A that needs to be adjusted for segmenting the image. According to the ε-greedy strategy, the corresponding action is selected: an integer is selected in the action set A as the basis for adjusting the segmentation result. If the selected action is ai, then the pixels with pixel values ​​in [ai-5, ai+5] in the segmentation image are set to 255.

[0067] 5. In the DoubleDQN algorithm, the agent first continuously explores the environment and adds the samples encountered during the exploration to the experience pool. When the experience pool is full, the deep Q network is trained.

[0068] The training parameters for the deep Q-network are set as follows:

[0069] The epoch size is set to 15, and the number of steps to perform the action on each crack image segmentation result in each epoch is set to 1000. The batch size for randomly selecting samples from the experience pool is set to 16.

[0070] The discount factor γ is set to 0.9, the size of the experience replay set replay_size is set to 10000, the learning rate of the deep Q network is set to 0.001, and the greedy policy for selecting actions is ∈ = 0.85;

[0071] Results on the CFD test set using deep reinforcement learning and without deep reinforcement learning, such as Figure 4 As shown in the confusion matrix, the number of crack pixels identified without reinforcement learning is 73,145, while the number of crack pixels increases to 77,812 after using deep reinforcement learning.

[0072] This invention divides the segmentation task of highway crack images into two stages. The first stage uses a deep learning-based segmentation algorithm (such as UNet or AttentionUNet) to obtain coarse segmentation results. The second stage uses a deep reinforcement learning Double DQN algorithm to iteratively optimize the coarse segmentation results from the neural network, essentially refining the segmentation results and further improving their accuracy. Experiments on the CFD dataset show improved IoU values ​​for the segmentation results of crack images on the test set. Figure 3 As can be seen, the horizontal axis represents the sample number, and the vertical axis represents the IoU difference between the segmentation results using Double DQN and those not using Double DQN. The visualized segmentation results are as follows: Figure 5As shown, the segmentation results are more accurate after using DoubleDQN.

[0073] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A highway crack image segmentation method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Preprocess the labeled highway crack images. Considering the low contrast and complex texture of highway images, the ACE algorithm is used for preprocessing to improve the contrast between the cracks and the background. For the preprocessed images, the mean and variance of the entire dataset are calculated, and the dataset is standardized using the mean and variance. Step 2: Build a suitable deep learning model to train the preprocessed highway crack image. After training, use the model to perform forward inference on the highway crack image to obtain a rough crack segmentation result. Step 3: Construct the Markov decision process, which consists of the state values, action values, and reward values ​​required in deep reinforcement learning. The specific steps are as follows: Step 3-1: Set the segmentation result at time T as the state value of deep reinforcement learning, while the initial state of the algorithm is the image segmentation result output by the segmentation network model in the first stage, which is also the input of the Double DQN network; Step 3-2: Set the action space of the agent, which is also the threshold set A that needs to be adjusted for image segmentation. Select the corresponding action according to the ε-greedy strategy: Select an integer from the action set A as the benchmark for adjusting the segmentation result. If the selected action is a i Then the pixel values ​​in the segmented image will be in [a i -5, a i The pixel value of +5 is set to 255; Step 3-3: Set the IoU difference before and after taking the action as the reward for the environment in deep reinforcement learning; if the IoU value of the segmentation map after taking the action is greater than the IoU before taking the action, the reward is set to 1; if the IoU of the segmentation map before and after taking the action is the same, the reward is set to 0; if the IoU value of the segmentation map after taking the action is less than the IoU before taking the action, the reward is set to -1. Step 4: Use appropriate deep learning networks as the value network and target Q network in deep reinforcement learning; use the state values ​​as input to the value network, and the Q values ​​corresponding to all thresholds in the threshold action set A as output; randomly sample some samples from the experience pool, calculate the parameters of the value network using the update formula of the Double DQN algorithm, and update the parameters of the value network to the target Q network at appropriate times; the experience pool here consists of samples obtained by the agent of the Double DQN algorithm interacting with the environment at each time step, and the samples contain 4 parts: the state S at time t. t Action a is taken, reward R is given by the environment, and state S is formed after taking action a. t+1 ; Step 5: During the iteration of Double DQN, the experience pool is continuously filled with sample data explored by the agent; deep reinforcement learning will randomly draw some samples from the experience pool and calculate the target Q value using the update formula of the Double DQN algorithm: ； (1) wherein S t+1 represents a state after performing an action, R t+1 represents a reward of performing an action a, w represents a parameter of an estimation value network, w' represents a parameter of a target value network; γ represents a discount factor of a reward, t represents a time point, represents an action at time t+1; Step 6: Calculate the loss value of the deep Q network output; this loss is obtained based on the mean square error between the target Q value obtained in Step 5 and the Q value output by the deep Q network; at the same time, the weight parameters in the deep Q network are randomly initialized using a truncated normal distribution, and the final output layer of the network uses Softmax; ； (2) Where B represents the number of samples in one iteration, and i represents the sample number.

2. The highway crack image segmentation method based on deep reinforcement learning according to claim 1, characterized in that, The reward formula for the constructed Markov decision process is as follows: ； (3)。 3. The method for highway crack image segmentation based on deep reinforcement learning according to claim 1, characterized in that, The constructed deep Q-network is a classification network where the number of neurons in the last layer equals the size of the action space. As a network model for the action value function, the deep Q-network uses w as its weight parameter and Q(s, a;w) to simulate the action value function Q*(s,a), where s represents the current state and a represents the current action. ； (4)。

Images