Bridge maintenance strategy intelligent decision-making method and system based on deep reinforcement learning

Abstract

The application provides a bridge maintenance strategy intelligent decision-making method and system based on deep reinforcement learning, relates to the technical field of artificial intelligence, and comprises the following steps: constructing an experience replay buffer of historical maintenance records; establishing a double-layer strategy network, wherein a main strategy network generates a preliminary maintenance action, a shadow strategy network deduces a future trajectory and calculates a worst-case reward value through Monte Carlo tree search, and the preliminary maintenance action is replaced conservatively when the worst-case reward value is lower than a safety boundary; performing maintenance to obtain a real reward; for sparse reward samples, searching for similar historical cases to construct a reference trajectory and calculating direction consistency as an auxiliary reward; and updating network parameters by using the real reward and the auxiliary reward.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to a bridge maintenance strategy intelligent decision-making method and system based on deep reinforcement learning. BACKGROUND

[0002] Although the existing bridge maintenance decision-making method based on machine learning can learn the maintenance rules from historical data, it generally faces the dilemma of sparse reward feedback. The long-term effect of bridge maintenance behavior often needs a long time to appear, and it is difficult to obtain clear reward signals in the short term, which leads to slow convergence or even failure of the learning process. At the same time, the existing method lacks a forward-looking evaluation mechanism for the future impact of maintenance actions, and cannot predict the long-term risks caused by current decisions, which may lead to aggressive decisions when facing maintenance of critical structural parts, threatening the safety of bridge operation. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a bridge maintenance strategy intelligent decision-making method and system based on deep reinforcement learning, which can solve the problems in the prior art.

[0004] In a first aspect of the embodiments of the present application, a bridge maintenance strategy intelligent decision-making method based on deep reinforcement learning is provided, comprising:

[0005] Obtaining the historical maintenance records of the bridge, and constructing an experience replay buffer;

[0006] Establishing a double-layer policy network including a main policy network and a shadow policy network, wherein the main policy network generates a preliminary maintenance action according to the current bridge state monitoring data, and the shadow policy network deduces multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value, and generates a conservative maintenance action to replace the output when the worst-case cumulative reward value is below a safety boundary;

[0007] Executing the output maintenance action, obtaining the post-maintenance state monitoring data and the real reward, and storing them in the experience replay buffer;

[0008] Sampling training samples from the experience replay buffer, and for samples with sparse real rewards, searching for a reference case similar to the sample state monitoring data in the historical maintenance records, taking the state change path of the reference case as a reference trajectory, and calculating the consistency score between the state change direction of the sample action and the direction of the reference trajectory as an immediate auxiliary reward;

[0009] Updating the parameters of the double-layer policy network using the real reward and the immediate auxiliary reward.

[0010] In an optional implementation, the step of obtaining the historical maintenance records of the bridge and constructing the experience replay buffer comprises:

[0011] The historical maintenance record includes state monitoring data at a historical maintenance time, maintenance actions at the historical maintenance time, and a state change path after maintenance;

[0012] The state monitoring data in the historical maintenance record is subjected to multi-time scale decomposition to form a multi-dimensional state representation vector;

[0013] The maintenance actions in the historical maintenance record are classified and coded according to the action sites and intervention intensities to construct a maintenance action coding dictionary;

[0014] The state change path after maintenance in the historical maintenance record is subjected to time sequence segmentation to extract evolution sequences of a pre-maintenance state, an immediate post-maintenance state, and a long-term post-maintenance state, and to calculate a change trajectory vector of the state evolution sequences in the multi-dimensional state representation space;

[0015] The multi-dimensional state representation vector, the maintenance action coding in the maintenance action coding dictionary, and the change trajectory vector are combined to form a historical sample, which is stored in an experience replay buffer.

[0016] In an optional embodiment, a double-layer strategy network including a main strategy network and a shadow strategy network is established, the main strategy network generates a preliminary maintenance action according to current bridge state monitoring data, and the shadow strategy network deduces multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value, and a conservative maintenance action is generated to replace the output when the worst-case cumulative reward value is lower than a safety boundary, and the steps include:

[0017] The main strategy network is established, current bridge state monitoring data is converted into a state feature vector, and a preliminary maintenance action is output according to the state feature vector;

[0018] The shadow strategy network is established, a state transition prediction function is trained based on the state change path in the historical maintenance record, and a first-step predicted state after execution of the preliminary maintenance action is calculated through the state transition prediction function according to the current state monitoring data and the preliminary maintenance action;

[0019] Starting from the first-step predicted state, subsequent maintenance actions are iteratively selected through Monte Carlo tree search, and corresponding subsequent-step predicted states are calculated by calling the state transition prediction function, thereby constructing multiple future evolution trajectories extending from the first-step predicted state to a terminal state;

[0020] For each future evolution trajectory, the cumulative reward value is obtained by accumulating the immediate reward values of the first-step predicted state and each subsequent-step predicted state node in the trajectory;

[0021] extracting a minimum value of all trajectory cumulative reward values as a worst-case cumulative reward value, and outputting a conservative maintenance action to replace the preliminary maintenance action according to a safe trajectory whose cumulative reward value is higher than the safety boundary when the worst-case cumulative reward value is lower than the safety boundary.

[0022] In an optional implementation, the step of constructing a plurality of future evolution trajectories extending from the first-step predicted state to the terminal state by iteratively selecting a subsequent maintenance action and calling the state transition prediction function to calculate a corresponding subsequent-step predicted state from the first-step predicted state through a Monte Carlo tree search comprises:

[0023] taking the first-step predicted state as a root node of the Monte Carlo tree, and initializing a search tree;

[0024] at a current leaf node of the search tree, extracting a historical case set similar to a state of the current leaf node according to a state change path in the historical maintenance record, counting execution frequencies and corresponding state improvement amplitudes of each maintenance action in the historical case set, and calculating a selection priority score;

[0025] selecting a subsequent maintenance action to be expanded according to the selection priority score and an exploration coefficient, calling the state transition prediction function to calculate a subsequent-step predicted state after the subsequent maintenance action is executed, adding the subsequent-step predicted state as a new child node to the search tree, deriving, for the new child node, to a preset search depth or a terminal state through the state transition prediction function to obtain a complete evolution path and calculate a derivation reward value, and updating the derivation reward value along the evolution path to each node on a path of the root node to update a visit frequency and an average reward value of each node;

[0026] repeating the selection-expansion-derivation-backtracking process until a preset iteration number is reached, and extracting a plurality of different branch paths starting from the root node as the plurality of future evolution trajectories.

[0027] In an optional implementation, the step of executing the output maintenance action, obtaining post-maintenance state monitoring data and a real reward, and storing the post-maintenance state monitoring data and the real reward in an experience replay buffer comprises:

[0028] performing a bridge maintenance operation according to the output maintenance action; and collecting post-maintenance state monitoring data after the maintenance action is completed;

[0029] calculating a state improvement amount between the post-maintenance state monitoring data and bridge state monitoring data before the maintenance action is performed, and calculating a real reward according to the state improvement amount and a resource consumption amount of the maintenance action;

[0030] forming an experience sample comprising the bridge state monitoring data before the maintenance action is performed, the maintenance action, the post-maintenance state monitoring data, and the real reward, and storing the experience sample in the experience replay buffer.

[0031] In an optional embodiment, the step of sampling a training sample from the experience replay buffer, for the sample with sparse real reward, retrieving a reference case similar to the sample state monitoring data in the historical maintenance record, taking the state change path of the reference case as a reference trajectory, and calculating a direction consistency score of the state change direction of the sample action with the direction of the reference trajectory as an instant auxiliary reward comprises:

[0032] sampling a training sample from the experience replay buffer, the training sample comprising sample pre-maintenance state monitoring data, sample post-maintenance state monitoring data, and sample real reward;

[0033] calculating a sparsity index of the sample real reward, and determining the sample with sparse real reward when the sparsity index exceeds a preset sparsity threshold; for the sample with sparse real reward, calculating the similarity between the sample pre-maintenance state monitoring data and the state monitoring data of each historical case in the historical maintenance record, and extracting a plurality of historical cases with the highest similarity as reference cases;

[0034] obtaining a state change path of the reference case, the state change path comprising a pre-maintenance state vector and a post-maintenance state vector; calculating a reference state change vector based on the pre-maintenance state vector and the post-maintenance state vector, and constructing a reference trajectory by weighted averaging the reference state change vectors of the plurality of reference cases;

[0035] calculating the difference between the sample post-maintenance state monitoring data and the sample pre-maintenance state monitoring data as a sample state change vector;

[0036] calculating the direction cosine value between the reference trajectory and the sample state change vector as a direction consistency score, and normalizing the direction consistency score as an instant auxiliary reward.

[0037] In an optional embodiment, the step of updating the parameters of the double-layer policy network using the real reward and the instant auxiliary reward comprises:

[0038] for the sample with sparse real reward in the training sample, combining the corresponding instant auxiliary reward and the real reward to form a comprehensive reward signal; for the sample with non-sparse real reward in the training sample, directly using the real reward as the comprehensive reward signal;

[0039] inputting the sample pre-maintenance state monitoring data of the training sample into the main policy network to obtain a predicted maintenance action output by the main policy network; calculating a policy loss function value between the predicted maintenance action and the sample maintenance action actually performed in the training sample, and weighting the policy loss function value according to the comprehensive reward signal;

[0040] The sample pre-inspection state monitoring data of the training sample is input into the shadow policy network with the sample inspection action to obtain a predicted post-inspection state; and a state transition prediction loss function value between the predicted post-inspection state and actual post-inspection state monitoring data of the sample in the training sample is calculated;

[0041] The state transition prediction loss function value of the shadow policy network is combined with the weighted policy loss function value of the main policy network to form a total loss function; and the main policy network parameters and the shadow policy network parameters are updated according to the total loss function through a gradient back propagation algorithm.

[0042] In a second aspect, a bridge inspection strategy intelligent decision system based on deep reinforcement learning is provided, comprising:

[0043] A first unit is configured to obtain historical inspection records of a bridge and construct an experience replay buffer;

[0044] A second unit is configured to establish a double-layer policy network comprising a main policy network and a shadow policy network, wherein the main policy network generates a preliminary inspection action according to current bridge state monitoring data, the shadow policy network deduces multiple future evolution trajectories of the preliminary inspection action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value, and a conservative inspection action is generated to replace the output when the worst-case cumulative reward value is lower than a safety boundary;

[0045] A third unit is configured to execute the output inspection action, obtain post-inspection state monitoring data and a real reward, and store them in the experience replay buffer;

[0046] A fourth unit is configured to sample training samples from the experience replay buffer, retrieve reference cases similar to sample state monitoring data in the historical inspection records for samples with sparse real rewards, take the state change path of the reference cases as a reference trajectory, and calculate a consistency score between the state change direction of the sample action and the direction of the reference trajectory as an instant auxiliary reward;

[0047] A fifth unit is configured to update the double-layer policy network parameters using the real reward and the instant auxiliary reward.

[0048] In a third aspect, a computer readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0049] The double-layer policy network architecture balances the decision safety and learning efficiency. The double-layer cooperative mechanism retains the autonomous exploration ability of deep reinforcement learning, and avoids blind aggressive decision-making through prospective risk assessment, ensuring that the inspection strategy meets the safety constraint requirements while pursuing performance optimization.

[0050] The auxiliary reward mechanism guided by historical cases effectively alleviates the reward sparsity problem. This method converts historical experience into real-time learning signals, accelerates network convergence while ensuring that decisions comply with engineering practice rules, can fully utilize accumulated maintenance knowledge, and improves learning efficiency and decision quality. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a bridge maintenance strategy intelligent decision method based on deep reinforcement learning according to an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. The following specific embodiments can be combined with each other, and the same or similar concepts or processes will not be described in detail in some embodiments.

[0053] Figure 1 A flowchart of a bridge maintenance strategy intelligent decision method based on deep reinforcement learning according to the present application is shown in FIG. 1, as shown in the figure, the method comprises: Figure 1

[0054] Obtaining historical maintenance records of the bridge and constructing an experience replay buffer;

[0055] Establishing a double-layer policy network including a main policy network and a shadow policy network, the main policy network generates a preliminary maintenance action according to current bridge state monitoring data, the shadow policy network deduces multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value, and generates a conservative maintenance action to replace the output when the worst-case cumulative reward value is lower than a safety boundary;

[0056] Executing the output maintenance action, obtaining post-maintenance state monitoring data and real reward, and storing them in the experience replay buffer;

[0057] Sampling training samples from the experience replay buffer, for samples with sparse real rewards, searching for a reference case similar to the sample state monitoring data in the historical maintenance records, taking the state change path of the reference case as a reference trajectory, and calculating a consistency score between the state change direction of the sample action and the direction of the reference trajectory as an immediate auxiliary reward;

[0058] Updating the double-layer policy network parameters using the real reward and the immediate auxiliary reward.

[0059] In an optional embodiment, the step of obtaining historical maintenance records of the bridge and constructing an experience replay buffer comprises:

[0060] The historical maintenance records include state monitoring data at the historical maintenance time, maintenance actions at the historical maintenance time, and post-maintenance state change paths.​

[0061] The state monitoring data in the historical maintenance record is decomposed into multiple time scales to form a multi-dimensional state representation vector;

[0062] The maintenance actions in the historical maintenance record are classified and coded according to the action site and intervention intensity to construct a maintenance action coding dictionary;

[0063] The maintenance state change path in the historical maintenance record is time-sequentially segmented, and the evolution sequence of the pre-maintenance state, the immediate post-maintenance state and the long-term post-maintenance state is extracted, and the change trajectory vector of the state evolution sequence in the multi-dimensional state representation space is calculated;

[0064] The multi-dimensional state representation vector, the maintenance action coding in the maintenance action coding dictionary and the change trajectory vector are combined to form a historical sample, which is stored in an experience replay buffer.

[0065] For example, the historical maintenance record is derived from the historical database of the bridge operation management system, covering the complete archives of all completed maintenance operations in the past few years. Each historical maintenance record consists of three core data fields: the state monitoring data field at the historical maintenance time records the monitoring parameters of the bridge structure before the maintenance implementation, including the strain sensor collected main beam strain value, the vibration sensor collected acceleration response, the displacement sensor collected deflection deformation, the crack detector collected crack width and depth, the temperature sensor collected environmental temperature and structure temperature, all sensor data sampling frequency is 1 time per hour, the original data is stored in time sequence form; the maintenance action field at the historical maintenance time records the specific implementation content of this maintenance operation, including the action site identification such as the main beam of the first span and the first section, the support type, the bridge deck pavement area, the expansion joint number, and the intervention intensity description such as local repair, overall replacement, reinforcement treatment, cleaning and maintenance; the post-maintenance state change path field records the evolution process of the bridge state after the maintenance is completed, including the immediate post-maintenance state monitoring data, the short-term post-maintenance state monitoring data within 1 month, the long-term post-maintenance state monitoring data within 3 months to 1 year, forming a complete state evolution sequence from pre-maintenance to long-term post-maintenance.

[0066] Multi-timescale decomposition of condition monitoring data was performed using wavelet transform for feature extraction. Condition monitoring data from the seven days preceding each maintenance time in historical maintenance records were extracted, forming a time series containing 168 sampling points. The wavelet transform employed the Daubechies wavelet basis function, with a decomposition level of five. The first level extracted high-frequency components with a period of 1 to 2 hours, reflecting the bridge's instantaneous response to traffic loads; the second level extracted second-high frequency components with a period of 2 to 4 hours, reflecting the dynamic characteristics of local structures; the third level extracted mid-frequency components with a period of 4 to 8 hours, reflecting intra-diurnal variations in structural stiffness; the fourth level extracted second-low frequency components with a period of 8 to 16 hours, reflecting the cumulative effects of temperature; and the fifth level extracted low-frequency components with a period exceeding 16 hours, reflecting the slow drift trend of the overall structural performance. The wavelet coefficients obtained from each level were energy normalized, scaling the sum of squares of the coefficients to the 0-1 range. The normalized coefficients from the five levels were then concatenated sequentially to form a multidimensional state representation vector. Data from five types of sensors—strain, vibration, displacement, crack, and temperature—are decomposed into multiple time scales. Each type of sensor data generates a 5-dimensional feature vector, and the five types of sensors together constitute a 25-dimensional multi-dimensional state representation vector.

[0067] The classification and coding of maintenance actions adopts a hierarchical coding scheme, constructing a maintenance action coding dictionary. The coding of the affected parts is divided according to the spatial order of the bridge structure from top to bottom and from left to right. The bridge deck system includes bridge deck pavement, expansion joints, and drainage facilities, with a coding range of 100 to 199; the main beam system includes steel beams, concrete beams, and diaphragms, with a coding range of 200 to 299; the support system includes bearings, piers, and foundations, with a coding range of 300 to 399; and the auxiliary system includes guardrails, lighting, and monitoring equipment, with a coding range of 400 to 499. The intervention intensity coding reflects the degree of impact of maintenance operations on the structural condition, divided into four levels: cleaning and maintenance operations such as rust removal, cleaning, and painting, with an intensity code of 1; local repair operations such as crack grouting, concrete repair, and steel plate bonding, with an intensity code of 2; overall replacement operations such as bearing replacement, expansion joint replacement, and bridge deck repaving, with an intensity code of 3; and reinforcement operations such as adding prestress, bonding carbon fiber, and adding supports, with an intensity code of 4. A complete maintenance action code consists of a code for the affected location and a code for the intensity of intervention, represented by a 4-digit number. The first three digits represent the affected location, and the fourth digit represents the intensity of intervention. For example, the code for partial repair of the second span of the main beam is 2012, and the code for complete replacement of the support is 3013. The maintenance action code dictionary is stored in key-value pair format, where the key is the text description of the maintenance action, and the value is the corresponding 4-digit number code.

[0068] The temporal segmentation extraction and trajectory vector calculation of the post-maintenance status change path are performed on the status evolution sequence in historical maintenance records. The pre-maintenance status is defined as the time average of the multi-dimensional status representation vectors obtained from the status monitoring data within 7 days prior to maintenance, decomposed over multiple time scales, forming a 25-dimensional pre-maintenance status vector. The immediate post-maintenance status is defined as the multi-dimensional status representation vector obtained from the first complete collection of status monitoring data within 24 hours after maintenance, decomposed over multiple time scales, forming a 25-dimensional immediate post-maintenance status vector. The long-term post-maintenance status is defined as the time average of the multi-dimensional status representation vectors obtained from the status monitoring data from day 90 to day 120 after maintenance, decomposed over multiple time scales, forming a 25-dimensional long-term post-maintenance status vector. The status evolution sequence consists of the pre-maintenance status vector, the immediate post-maintenance status vector, and the long-term post-maintenance status vector arranged in chronological order, totaling 75 dimensions. The change trajectory vector is obtained by calculating the direction and amplitude of the state evolution sequence in the multidimensional state representation space. Specifically, it is the difference between the long-term state vector after maintenance and the state vector before maintenance, reflecting the long-term improvement effect of maintenance on the bridge's condition, forming a 25-dimensional change trajectory vector.

[0069] The assembly of historical samples and the construction of the experience replay buffer integrate the above processing results into structured data. Each historical maintenance record corresponds to one historical sample, which contains four fields: a multidimensional state representation vector field storing the state vector before maintenance (length 25); a maintenance action code field storing a 4-digit code obtained from the maintenance action code dictionary (length 1); a change trajectory vector field storing the direction and magnitude of the state change after maintenance (length 25); and a timestamp field storing the year, month, day, hour, minute, and second of the maintenance. Historical samples are stored in the experience replay buffer in timestamp order. The buffer is implemented using a circular queue data structure with a queue capacity of 10,000 samples. When a new sample is added and the queue exceeds its capacity, the oldest sample is automatically removed. The experience replay buffer supports a fast retrieval function based on state similarity. By establishing a K-dimensional tree index structure on the multidimensional state representation vector, the query complexity is logarithmic. During retrieval, the Euclidean distance between the query state vector and the state vectors of each historical sample in the buffer is calculated, and the historical samples with the smallest distance are returned as a set of similar cases.

[0070] This implementation achieves a structured representation of historical maintenance experience through multi-timescale decomposition and hierarchical coding. The change trajectory vector captures the long-term effects of maintenance operations, and the experience replay buffer supports efficient historical case retrieval and knowledge reuse, providing a reliable foundation of historical experience.

[0071] In an alternative embodiment, a double-layer strategy network comprising a main strategy network and a shadow strategy network is established, the main strategy network generates a preliminary maintenance action according to current bridge state monitoring data, the shadow strategy network deduces multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value, and generates a conservative maintenance action to replace the output when the worst-case cumulative reward value is below a safety boundary, the step of outputting the conservative maintenance action comprises:

[0072] The main strategy network is established, the current bridge state monitoring data is converted into a state feature vector, and a preliminary maintenance action is output according to the state feature vector;

[0073] The shadow strategy network is established, a state transition prediction function is trained based on the state change path in the historical maintenance record, and a first-step predicted state after execution of the preliminary maintenance action is calculated through the state transition prediction function according to the current state monitoring data and the preliminary maintenance action;

[0074] Through Monte Carlo tree search, a subsequent maintenance action is iteratively selected from the first-step predicted state, and a corresponding subsequent-step predicted state is calculated by calling the state transition prediction function, thereby constructing multiple future evolution trajectories extending from the first-step predicted state to a terminal state;

[0075] For each future evolution trajectory, the immediate reward values of the first-step predicted state and each subsequent-step predicted state node in the trajectory are accumulated to obtain a cumulative reward value;

[0076] The minimum value of all trajectory cumulative reward values is extracted as a worst-case cumulative reward value, and a conservative maintenance action is generated according to a safe trajectory with a cumulative reward value higher than a safety boundary to replace the preliminary maintenance action for output when the worst-case cumulative reward value is below the safety boundary.

[0077] Exemplarily, the establishment of the main policy network adopts a deep neural network architecture to realize the mapping from state to action. The network input layer receives the current bridge state monitoring data, and the state monitoring data contains a 25-dimensional multi-dimensional state representation vector, which is derived from the feature splicing results of strain, vibration, displacement, crack, and temperature 5 types of sensor data after multi-time scale decomposition. The input layer normalizes the 25-dimensional vector, scaling the numerical value of each dimension to the interval of -1 to +1. The normalization parameters are determined based on the statistical distribution of the state monitoring data in the historical maintenance records, specifically, each dimension is subtracted by the mean and divided by the standard deviation. The network hidden layer adopts a 3-layer fully connected structure, the first hidden layer contains 128 neurons, the second hidden layer contains 64 neurons, and the third hidden layer contains 32 neurons. The activation function of each hidden layer is ReLU function, which prevents gradient disappearance and accelerates convergence. The number of neurons in the network output layer is equal to the total number of action types in the maintenance action coding dictionary, and the output layer activation function is Softmax function, which converts the output of each neuron to an action selection probability distribution. The process of the main policy network outputting the preliminary maintenance action according to the state feature vector is as follows: input the normalized 25-dimensional state feature vector into the network, pass it through the non-linear transformation of the 3-layer hidden layer, and the output layer generates the selection probability of each maintenance action. The action with the maximum selection probability is selected as the preliminary maintenance action. The network parameter initialization adopts the Xavier initialization method, and the weight matrix elements are sampled from a uniform distribution with a mean of 0 and a variance related to the input and output dimensions of the layer. The bias vector is initialized to 0.

[0078] The establishment of the shadow policy network is centered on the construction and training of the state transition prediction function. The state transition prediction function receives the combination of the current state monitoring data and the maintenance action as input, and outputs the predicted state vector after executing the maintenance action. The data basis of the function comes from the state change path in the historical maintenance record, and each training sample contains three fields: the pre-maintenance state vector, the maintenance action code, and the long-term post-maintenance state vector. The pre-maintenance state vector is a 25-dimensional state representation vector, the maintenance action code is a 4-digit number, and the long-term post-maintenance state vector is also a 25-dimensional vector. The state transition prediction function uses the Long Short-Term Memory Network (LSTM) to realize the time series mapping relationship. The input layer concatenates the 25-dimensional state vector and the maintenance action vector. The 4-digit code of the maintenance action needs to be converted into a one-hot encoding vector. The length of the one-hot encoding vector is equal to the total number of action types in the maintenance action code dictionary. Assuming that the total number of action types is 200, the one-hot encoding corresponding to the maintenance action code 2012 is a 200-dimensional vector with the 2012th position being 1 and the rest being 0. The dimension of the concatenated input vector is 225. The LSTM layer adopts a double-layer structure. The first layer of LSTM unit receives a 225-dimensional input vector. It contains three gating mechanisms: input gate, forget gate, and output gate. The number of hidden units is 64. The calculation process of the input gate is as follows: the input vector is multiplied by the input gate weight matrix, the multiplication result is added to the input gate bias vector, and the sum is processed by the Sigmoid activation function. The final output value is between 0 and 1, which controls the retention proportion of the input information. The calculation process of the forget gate is as follows: the input vector is multiplied by the forget gate weight matrix, the multiplication result is added to the forget gate bias vector, and the sum is processed by the Sigmoid activation function. The final output value is between 0 and 1, which controls the forgetting proportion of historical memory. The calculation process of the candidate memory value is as follows: the input vector is multiplied by the candidate weight matrix, the multiplication result is added to the candidate bias vector, and the sum is processed by the Tanh activation function. The final output value is between -1 and +1, which represents the memory content to be updated. The memory cell value at the current time is calculated as follows: the multiplication of the forget gate output value and the memory cell value at the previous time is added to the product of the input gate output value and the candidate memory value, and the sum is obtained as the memory cell value at the current time. The calculation process of the output gate is as follows: the input vector is multiplied by the output gate weight matrix, the multiplication result is added to the output gate bias vector, and the sum is processed by the Sigmoid activation function. The final output value is between 0 and 1, which controls the output proportion of the memory cell. The hidden state at the current time is calculated as follows: the current memory cell value is processed by the Tanh activation function, and the result is multiplied by the output gate output value to obtain a 64-dimensional hidden state vector. The second layer of LSTM unit receives the 64-dimensional hidden state vector of the first layer as input, and the internal calculation process is the same as that of the first layer. It outputs a 64-dimensional hidden state vector.The output layer is a fully connected layer, which receives the 64-dimensional hidden state vector of the second LSTM layer, multiplies it with a weight matrix and adds a bias vector to produce a 25-dimensional predicted state vector.

[0079] The training process of the state transition prediction function adjusts the weight matrices and bias vectors of the LSTM layers through the backpropagation algorithm. The training objective is to minimize the mean squared error between the predicted state vector and the true long-term state vector after maintenance. The error is calculated by squaring the difference between the 25-dimensional values of the predicted state vector and the corresponding values of the true state vector, summing the 25 squared differences, and dividing the sum by 25 to obtain the mean squared error. Training data is batch sampled from the experience replay buffer, with each batch containing 32 historical maintenance record samples. The training process uses the Adam optimizer to update parameters, with a learning rate of 0.001, momentum parameters beta1 set to 0.9, momentum parameters beta2 set to 0.999, and numerical stability parameter epsilon set to 0.00000001. The weight update method is as follows: calculate the gradient of the loss function with respect to each layer's weights, propagate the gradient from the output layer to the input layer using the backpropagation through time algorithm, and maintain the first and second moment estimates of each weight using the Adam optimizer. The update calculation of the first moment estimate is as follows: t = beta1 * m t-1 + (1 - beta1) * g t , where m t represents the current first moment estimate, m t-1 represents the previous first moment estimate, and g t represents the current gradient. The update calculation of the second moment estimate is as follows: t = beta2 * v t-1 + (1 - beta2) * g t ², where v t represents the current second moment estimate, v t-1 represents the previous second moment estimate, and g t ² represents the square of the current gradient. The bias-corrected first moment calculation is as follows: t m t / (1 - beta1 t ), where t represents the current iteration number. The bias-corrected second moment calculation is as follows: t v -8 / (1 - beta2 -8 ), where t represents the current iteration number. The weight update amount calculation is as follows: multiply the learning rate by the bias-corrected first moment, divide the product by the square root of the bias-corrected second moment plus epsilon, and set epsilon to 1 x 10The weight update amount is obtained. The new weight is equal to the old weight minus the weight update amount. The training round number is set to 100 rounds, and each round traverses all historical samples in the experience replay buffer, and the weight is updated in batches of 32 samples. After training, the state transition prediction function weight matrix and bias vector are saved and used for subsequent prediction calculations.

[0080] The process of calculating the first step predicted state from the current state monitoring data and the preliminary maintenance action by the shadow policy network is as follows: the 25-dimensional vector of the current state and the 4-digit code of the preliminary maintenance action are converted into one-hot encoding and spliced into a 225-dimensional input vector, which is input into the trained state transition prediction function. The input vector is calculated by the input gate, the forget gate, and the output gate of the first layer LSTM to generate a 64-dimensional hidden state. This hidden state is input into the second layer LSTM for repeated gate calculation to generate a 64-dimensional hidden state. The second layer hidden state is multiplied by the weight matrix and added to the bias vector after input into the full connection layer. The output layer produces a 25-dimensional first step predicted state vector, and each dimension of the vector represents the expected value of each monitoring parameter of the bridge after performing the preliminary maintenance action.

[0081] Monte Carlo tree search constructs future evolution trajectories from the first step predicted state. The search tree takes the first step predicted state as the root node, which stores a 25-dimensional state vector, a node access count initialized to 0, and a node cumulative reward value initialized to 0. The tree is expanded by iteratively selecting subsequent maintenance actions, and the selection strategy is based on the upper confidence bound algorithm (UCB). The selection score of each sub-action contains two parts: the average reward term and the exploration reward term. The average reward term is calculated by dividing the cumulative reward value of the child node by the access count of the child node. The exploration reward term is calculated in three steps: first, calculate the natural logarithm of the parent node access count, second, divide the first step result by the child node access count, and third, take the square root of the second step result and multiply it by the exploration coefficient 1.414.

[0082] In the current leaf node state, all feasible maintenance actions are traversed, the selection score of each action is calculated, and the action with the highest selection score is selected as the action to be expanded. The state transition prediction function is called, and the 25-dimensional state vector of the current leaf node and the one-hot encoding of the action to be expanded are spliced into a 225-dimensional vector. This vector is input into the first layer LSTM gate calculation, the second layer LSTM gate calculation, and the full connection layer linear transformation in turn, and a 25-dimensional vector of the subsequent step predicted state is output. This vector is added to the search tree as a new child node. The search depth is limited to 5 steps, i.e., a maximum of 5 subsequent steps are expanded from the first step predicted state. When the search depth limit is reached or the change amplitude of all dimensions of the predicted state is less than the threshold value 0.01, it is determined as a termination state. The search iteration number is set to 100 times, and each iteration includes selection, expansion, deduction, and backtracking. After 100 iterations are completed, all complete paths from the root node constitute multiple future evolution trajectories.

[0083] The instant reward value is composed of a state improvement reward and a risk penalty. The state improvement reward measures the performance improvement of the predicted state relative to the initial state, and is calculated by summing the differences between the values of each dimension of the predicted state vector and the corresponding dimension of the initial state vector. A positive difference indicates a positive reward for state improvement, while a negative difference indicates a negative reward for state deterioration. The risk penalty is calculated based on the safety margin of the predicted state. The safety margin is defined as the distance between each dimension of the predicted state vector and the safety threshold. The safety threshold is set according to bridge design specifications and operational experience. The safety threshold for the strain dimension is 80% of the design strain limit, the safety threshold for the vibration dimension is 70% of the allowed acceleration, the safety threshold for the displacement dimension is 75% of the allowable deflection, the safety threshold for the crack dimension is 60% of the dangerous crack width, and the safety threshold for the temperature dimension is 90% of the material tolerance temperature. When any dimension of the predicted state exceeds the corresponding safety threshold, the risk penalty value is set to -100, otherwise the risk penalty value is 0. The instant reward value is equal to the state improvement reward plus the risk penalty. The cumulative reward value of each future evolution trajectory is the sum of the instant reward values of the first predicted state node and all subsequent predicted state nodes in the trajectory.

[0084] The extraction of the worst-case cumulative reward value and the generation of the conservative maintenance action are based on safety boundary judgment. The safety boundary is set to 0, indicating that the cumulative reward value must be non-negative to ensure the safety of the maintenance strategy. All future evolution trajectories constructed by traversing the Monte Carlo tree search are traversed, and the cumulative reward value of each trajectory is extracted. The minimum value among all cumulative reward values is the worst-case cumulative reward value. When the worst-case cumulative reward value is lower than the safety boundary of 0, the preliminary maintenance action is determined to have potential risks and needs to be replaced by a conservative maintenance action. The conservative maintenance action is generated from the safe trajectories whose cumulative reward values are higher than the safety boundary. The safe trajectory set includes all evolution trajectories whose cumulative reward values are greater than or equal to 0. The trajectory with the maximum cumulative reward value is selected from the safe trajectory set, and the first step action of this trajectory is the conservative maintenance action. The conservative maintenance action replaces the preliminary maintenance action and serves as the final output of the double-layer strategy network, with an output format of a 4-digit code.

[0085] This embodiment realizes decision safety guarantee through the cooperation of the double-layer network. The main strategy network quickly generates candidate schemes, the shadow strategy network uses the state transition prediction function and Monte Carlo tree search to deduce future risks, the worst-case evaluation mechanism automatically triggers conservative replacement, and safety hazards caused by aggressive decisions are avoided.

[0086] In an alternative embodiment, the step of constructing multiple future evolution trajectories extending from the first predicted state to the termination state by starting from the first predicted state through Monte Carlo tree search, iteratively selecting subsequent maintenance actions, and calculating the corresponding subsequent predicted states by calling the state transition prediction function, comprises:

[0087] initializing a search tree with the first-step predicted state as a root node of the Monte Carlo tree;

[0088] At a current leaf node of the search tree, extracting a historical case set similar to the current leaf node state according to a state change path in the historical maintenance record, counting the execution frequency and the corresponding state improvement amplitude of each maintenance action in the historical case set, and calculating a selection priority score;

[0089] According to the selection priority score and an exploration coefficient, selecting a subsequent maintenance action to be expanded, calling the state transition prediction function to calculate a subsequent-step predicted state after the subsequent maintenance action is performed, adding the subsequent-step predicted state as a new child node to the search tree, and deriving the new child node to a preset search depth or a terminal state through the state transition prediction function to obtain a complete evolution path and calculate a derivation reward value; and updating the derivation reward value along the evolution path to each node on the root node path to update the visit times and the average reward value of each node.

[0090] Repeating the selection, expansion, rollback, and derivation process until a preset iteration number is reached, and extracting multiple different branch paths starting from the root node as multiple future evolution trajectories.

[0091] Illustratively, the Monte Carlo tree search establishes a search tree structure with the first-step predicted state as a root node, and the root node data structure includes five core fields of a state vector field, a visit times field, a cumulative reward value field, a parent node pointer field, and a child node list field. The state vector field stores a 25-dimensional floating-point number array, the visit times field is initialized to an integer 0, the cumulative reward value field is initialized to a floating-point number 0.0, the parent node pointer field is set to a null pointer for the root node, and the child node list field is initialized to an empty list. The search tree is stored in the memory in a tree-linked list structure, each node occupies about 320 bytes of memory, including 25-dimensional state vectors occupying 100 bytes, visit times occupying 4 bytes, cumulative reward values occupying 8 bytes, parent node pointers occupying 8 bytes, child node list pointers occupying 8 bytes, action identifiers occupying 4 bytes, node creation timestamps occupying 8 bytes, and reserved fields occupying 180 bytes. The upper limit of the total number of nodes of the search tree is set to 10,000 nodes, corresponding to a memory occupation of about 3.2 megabytes. When the number of nodes approaches the upper limit, a pruning strategy for the oldest unvisited nodes is triggered, and the pruning strategy calculates the idle duration according to the node creation timestamp and the most recent access timestamp. Nodes and their sub-trees with an idle duration exceeding 300 seconds are removed to release memory.

[0092] The historical maintenance record database is stored in a relational database and includes three main tables, i.e., a pre-maintenance state table, a maintenance action table and a post-maintenance state table. The pre-maintenance state table includes a record identifier, a timestamp and 25-dimension fields of a state vector. The maintenance action table includes a record identifier, an action code, an action type and an execution duration. The post-maintenance state table includes a record identifier, a timestamp and 25-dimension fields of a state vector. The state similarity is calculated by using the Euclidean distance. The Euclidean distance between a current leaf node state vector and a pre-maintenance state vector in a historical record is calculated in the following manner: the square sum of the difference between corresponding dimensions of the two vectors is taken and then the square root is taken. The historical records with a Euclidean distance less than a threshold value of 3.5 are included in a similar case set. The threshold value 3.5 is determined based on the average value of the standard deviations of the state vectors in the historical data and is approximately equal to 2.5 times the single-dimension standard deviation 1.4. The size of the similar case set is limited to a maximum of 200 records. When the records meeting the distance threshold value exceed 200, the first 200 records are selected in the order of distance from small to large. The execution frequency of each maintenance action is counted for the similar case set. The execution frequency is calculated as the number of times that the action appears in the similar case set. For example, if the action code 2012 appears 35 times in the 200 records of the similar case set, the frequency is 35. The state improvement amplitude corresponding to each action is also counted. The state improvement amplitude is calculated as the average value of the difference between the dimensions of the post-maintenance state vector and the pre-maintenance state vector. A positive difference indicates state improvement, and a negative difference indicates state deterioration. The arithmetic average of all the state improvement amplitudes of the same action in the similar case set is taken as the average improvement amplitude of the action.

[0093] The selection priority score is composed of a frequency item and an improvement item. The frequency item is calculated as the action execution frequency divided by the total number of records in the similar case set. The ratio represents the historical adoption rate of the action and has a value range of 0 to 1. The improvement item is the average improvement amplitude of the action and has a value range of -25 to +25. A negative value indicates historical average deterioration, and a positive value indicates historical average improvement. The selection priority score = frequency item x 0.3 + (improvement item / 50) x 0.7. The normalization coefficient 50 scales the improvement item to the interval of -0.5 to +0.5. The sum of the two weight coefficients is 1 to ensure the consistency of the score scale. The exploration coefficient introduces an exploration reward for unvisited actions, The exploration coefficient is set to 1.414, and the number of visits to the child node is incremented by 1 to avoid division by zero errors. For an unexpanded action, the number of visits to the child node is considered to be 0, and therefore, The action selection strategy to be expanded is: calculate the selection priority score + exploration reward of all actionable actions, select the action with the highest total score as the action to be expanded, and randomly select one of them when multiple actions have the same total score. The set of actionable actions is extracted from the maintenance action coding dictionary, which contains 200 types of maintenance actions. The inactionable actions are filtered according to the safety constraints of the current state, for example, when the strain dimension value exceeds the threshold, the load increasing action is prohibited, and when the crack dimension value exceeds the threshold, the reinforcement action is forced to be selected as the priority.

[0094] The state transition prediction function is called to calculate the predicted state of the subsequent step after the execution of the action to be expanded. The calling process is: the 25-dimensional state vector of the current leaf node and the action to be expanded code are converted into one-hot encoding and spliced into a 225-dimensional input vector, which is input into the trained LSTM network for forward propagation calculation. The first layer of the LSTM network receives the 225-dimensional input vector, and generates a 64-dimensional hidden state vector through the input gate, the forget gate and the output gate. The second layer receives the 64-dimensional hidden state vector of the first layer and continues to calculate the 64-dimensional hidden state vector through the gate. The output layer full connection layer receives the second layer hidden state vector, multiplies it with the 64x25 weight matrix, and adds the 25-dimensional bias vector to obtain the 25-dimensional predicted state vector. The value of each dimension of the predicted state vector represents the expected value of the bridge state after executing the action to be expanded. This vector is used as the state vector field of the new child node. The access count of the new child node is initialized to 0, the cumulative reward value is initialized to 0.0, the parent node pointer points to the current leaf node, and the action identifier records the action code to be expanded. The new child node is added to the child node list of the current leaf node to complete the tree structure expansion.

[0095] The deduction process is started immediately after the new child node is created, which iteratively calls the state transition prediction function to extend the evolution path from the state of the new child node. The deduction depth is limited to 5 steps, i.e. a 6-node path is formed by extending 5 subsequent nodes from the new child node. The action selection at each step of the deduction process adopts a greedy strategy, randomly selecting an action from the set of available actions, calling the state transition prediction function to calculate the next state vector corresponding to the action, and continuing the deduction with the next state vector as a temporary node. The temporary node is not added to the search tree structure and is only used for recording the deduction path. There are two termination conditions for the deduction process: first, when the number of deduction steps reaches the depth limit of 5 steps, the process is terminated; second, when the change amplitude of all dimensions of the predicted state vector is less than the threshold value of 0.01, it is determined that the state is stable and the process is terminated in advance. The change amplitude is calculated as the absolute value of the difference between the current step state vector and the previous step state vector. The deduction path includes the new child node and all temporary nodes in the deduction process, and the number of path nodes ranges from 2 to 6. The deduction reward value is calculated for each node state vector in the path, and the immediate reward value of the node is = state improvement reward - risk penalty. The state improvement reward is the sum of the difference between the state vector of the node and the initial state vector of the new child node in each dimension, and the risk penalty is 100 if any dimension of the state vector of the node exceeds the safety threshold, otherwise it is 0. The path deduction reward value is the sum of the immediate reward values of all nodes in the path, and the sum ranges from -500 to +125. A negative value indicates that the deduction path leads to state deterioration or triggers a safety risk, and a positive value indicates that the deduction path brings state improvement without safety risk.

[0096] The deduction reward value is updated back to the root node along the deduction path, and the backtracking process starts from the last temporary node of the deduction path and traverses in reverse to the new child node, and then continues to backtrack to the root node along the search tree structure. During the backtracking process, two updates are performed on each search tree node passed by the path, the first update is node access count + 1, and the second update is node cumulative reward value + deduction reward value. The average reward value of the node = node cumulative reward value ÷ node access count, which is used to calculate the improvement item of the selection priority score in the subsequent iteration. The backtracking update ensures that all nodes on the path from the root node to the new child node in the search tree reflect the statistical information of the deduction result. The access count records the frequency of selection of the node, and the cumulative reward value and the average reward value record the average income of the deduction path passing through the node.

[0097] The number of iterations is set to 100 times, and each iteration performs four stages of selection, expansion, deduction and backtracking. The selection stage selects a leaf node from the leaf node set of the current search tree as the starting point of expansion, and the leaf node is defined as a node with an empty child node list. The leaf node selection strategy preferentially selects the leaf node with the least access times, and when multiple leaf nodes have the same access times, the leaf node with the highest average reward value is selected, and if they are still the same, a random selection is made. The expansion stage performs the state similarity case extraction, priority score calculation, action selection, state transition prediction, and new child node creation processes described above on the selected leaf node. The deduction stage performs the iteration deduction and reward value calculation processes described above on the new child node. The backtracking stage performs the path backtracking and updating processes described above. After 100 iterations, the search tree contains the root node and the multi-layer child node structure generated by 100 expansions. Starting from the root node, a depth-first traversal is performed along the child node list, and the complete path from the root node to each leaf node is extracted. Each path represents a future evolution trajectory that extends from the first predicted state to the terminal state. The total number of trajectories is equal to the total number of leaf nodes in the search tree, ranging from 10 to 200 trajectories, and the length of the trajectory ranges from 2 to 6 nodes.

[0098] This embodiment constructs multiple evolution trajectories through Monte Carlo tree search, balances utilization and exploration based on the cooperative selection mechanism of historical case statistics and state transition prediction, and guarantees the reliability of statistical information through the push-back mechanism, providing diversified future scenario support for decision-making.

[0099] In an optional embodiment, the step of performing the output maintenance action, obtaining the post-maintenance state monitoring data and the true reward, and storing them in the experience replay buffer includes:

[0100] Performing a bridge maintenance operation according to the output maintenance action; after the maintenance action is completed, collecting post-maintenance state monitoring data;

[0101] Calculating the state improvement between the post-maintenance state monitoring data and the bridge state monitoring data before the maintenance action is performed, and calculating the true reward according to the state improvement and the resource consumption of the maintenance action;

[0102] The bridge state monitoring data before the maintenance action is performed, the maintenance action, the post-maintenance state monitoring data and the true reward are combined to form an experience sample, which is stored in the experience replay buffer.

[0103] Exemplarily, the maintenance action execution resolves the specific operation instruction from the maintenance action code dictionary according to the 4-digit code output by the double-layer strategy network. The code dictionary is stored in a hierarchical structure, the first digit represents the maintenance category, and the values 1 to 9 correspond to the nine categories of preventive maintenance, local reinforcement, crack repair, support replacement, expansion joint adjustment, drainage system dredging, anti-corrosion coating, load restriction, comprehensive maintenance respectively. The second digit represents the operation site, and the values 0 to 9 correspond to the ten sites of main girder, bridge pier, abutment, support, expansion joint, guardrail, bridge deck pavement, drainage facility, auxiliary facility, and full-bridge range respectively. The third digit represents the operation intensity, and the values 0 to 9 correspond to 10 levels from light intervention to heavy repair, level 0 is daily inspection without substantial intervention, and level 9 is demolition and reconstruction level intervention. The fourth digit represents the operation timing, and the values 0 to 9 represent the execution order of the action in the multi-action combination scheme or the identification of the independent action. The parsing module receives the 4-digit code and queries the dictionary database, which is stored in a key-value pair structure, the key is a 4-digit code string, and the value is a structure containing seven fields of operation type, operation site, operation intensity, operation timing, resource demand, estimated duration, and safety requirement. The operation instruction obtained by parsing is distributed to the on-site execution unit through the operation management system, and the operation management system verifies the compliance of the operation instruction, and the verification items include whether the maintenance site is currently in an operable state, whether the required resources can be allocated, whether the weather conditions meet the requirements of outdoor operation, and whether the traffic control has been approved. After verification, the operation work order is generated, which contains eight mandatory fields of operation number, operation content description, resource list, personnel configuration, start time, estimated completion time, safety measures, and quality acceptance standard. The operation work order is issued to the on-site work team, which prepares equipment and materials, arranges the operation area, implements traffic control, and performs specific operation operations according to the work order.

[0104] The judgment of the completion of the maintenance action execution is based on the quality acceptance standard in the operation work order. The acceptance standard sets different indicators for different maintenance categories, the preventive maintenance category action acceptance indicator is that the target site cleanliness meets the standard and no new damage is added, the local reinforcement category action acceptance indicator is that the reinforcement material strength detection is qualified and the connection with the original structure is reliable, the crack repair category action acceptance indicator is that the repair material is filled and the surface is smooth without hollowing, the support replacement category action acceptance indicator is that the new support installation position is accurate and the pre-pressure reaches the design value, and the expansion joint adjustment category action acceptance indicator is that the joint width meets the design requirements and the force transmission device is intact. The acceptance process consists of three levels of self-inspection by the work team, supervision sampling inspection, and owner final inspection, with a self-inspection qualified rate of 100%, a sampling inspection qualified rate of more than 95%, and a final inspection that requires rectification before re-inspection. After acceptance, the operation management system updates the operation work order status to completed, records the actual completion time and resource consumption, and triggers the post-maintenance state monitoring data collection process.

[0105] The post-maintenance state monitoring data collection is started within 24 hours after the maintenance action is accepted, and the delay time allows the structural response affected by the maintenance to be fully released and reach a stable state. The collection equipment is the same sensor network as the pre-maintenance state monitoring, strain sensors are arranged on the upper and lower flanges of the mid-span and support point sections of the main beam, the sampling frequency is 100 Hz, the collection time is 10 minutes, and the data processing extracts three characteristic values of mean, standard deviation and peak value. Vibration sensors are arranged at the mid-span and quarter point positions of the main beam, the sampling frequency is 200 Hz, the collection time is 15 minutes, and the data processing extracts three characteristic values of main frequency, acceleration root mean square and vibration attenuation coefficient. Displacement sensors are arranged at the mid-span and cantilever end of the main beam, the sampling frequency is 50 Hz, the collection time is 5 minutes, and the data processing extracts two characteristic values of maximum displacement and displacement rate. Crack monitoring is achieved by shooting the target crack area with a high-resolution camera, the image resolution is 0.01 mm per pixel, the shooting angle is perpendicular to the crack direction, and the image processing extracts three characteristic values of crack width, crack length and crack direction angle. Temperature sensors are arranged on the bridge deck pavement and the interior of the main beam, the sampling frequency is 0.1 Hz, the collection time is 1 hour, and the data processing extracts two characteristic values of average temperature and temperature gradient. The remaining sensors are arranged on the support, expansion joint, pier and guardrail of the auxiliary structure, the sampling frequency is 10-100 Hz, the collection time is 5-10 minutes, and the data processing extracts 12 characteristic values of displacement, angle, etc. The original data of various sensors are pre-processed by the on-site data collection unit, transmitted to the central server through the wireless network, the transmission protocol uses MQTT, the message quality level is set to QoS2 to ensure that the data is not lost and not repeated, and the transmission encryption uses AES256 algorithm. After receiving the data, the central server performs outlier detection, the outlier judgment standard is 3 times the standard deviation range of the same dimension statistical distribution of the pre-maintenance state monitoring data, and the data beyond the range is marked as abnormal and triggers the manual review process. After the outliers are removed or confirmed, the characteristic values of each sensor are spliced into a 25-dimensional state vector in a pre-defined order, the value range of each dimension of the vector is consistent with the pre-maintenance state vector, and the numerical accuracy is retained to two decimal places.

[0106] The state improvement amount calculation is performed on the difference between the post-maintenance state vector and the pre-maintenance state vector in each dimension. The pre-maintenance state vector is denoted as , the post-maintenance state vector is denoted as , both are 25-dimensional floating point vectors, and the i-th dimension is denoted as and , where i takes values from 1 to 25. The difference calculation method is to subtract the corresponding dimension value of the post-maintenance state vector from the dimension value of the pre-maintenance state vector, and the state improvement difference ΔS(i) of the i-th dimension is calculated as follows:

[0107] ;

[0108] The difference value is positive, indicating that the dimension state is improved, the difference value is negative, indicating that the dimension state is deteriorated, and the difference value is zero, indicating that the dimension has no change. The total state improvement amount is obtained by summing the difference values of the 25 dimensions, and the total state improvement amount AS_total is calculated according to the following formula:

[0109] ;

[0110] The total state improvement amount ranges from -125 to +125, and a negative value indicates that the overall state is deteriorated due to the maintenance action, and a positive value indicates that the overall state is improved due to the maintenance action.

[0111] The improvement amount weight coefficient is set for the key dimensions. The first to third dimensions of the state vector correspond to the three characteristic values extracted by the strain sensor, and the weight coefficients of the three dimensions are set to 2.0. The fourth to sixth dimensions of the state vector correspond to the three characteristic values extracted by the vibration sensor, and the weight coefficients of the three dimensions are set to 1.5. The seventh to eighth dimensions of the state vector correspond to the two characteristic values extracted by the displacement sensor, and the weight coefficients of the two dimensions are set to 1.8. The ninth to eleventh dimensions of the state vector correspond to the three characteristic values extracted by the crack monitoring, and the weight coefficients of the three dimensions are set to 2.5. The twelfth to thirteenth dimensions of the state vector correspond to the two characteristic values extracted by the temperature sensor, and the weight coefficients of the two dimensions are set to 1.0. The weight coefficients of the remaining 12 dimensions of the state vector, i.e., the fourteenth to twenty-fifth dimensions, are all 1.0. The weight coefficient of the ith dimension is denoted as w(i). The weighted state improvement amount AS_weighted is calculated according to the following formula:

[0112] ;

[0113] The weighted state improvement amount ranges from -312.5 to +312.5, and this value is used as the benefit item for calculating the true reward.

[0114] The maintenance action resource consumption is extracted from the actual record of the work order, including five items of labor cost, material cost, equipment rental cost, traffic control cost, and construction period loss. Labor cost = number of workers x actual working hours x labor unit price, and the labor unit price is set to 120 yuan per person-hour. The material cost is summarized from the material requisition list, including the purchase price and transportation cost of reinforcing materials, repair materials, and replacement parts. The equipment rental cost = number of rented equipment x rental days x daily rental. The traffic control cost includes traffic guide personnel wages, temporary signboard rental fees, and delayed vehicle compensation fees, which are calculated according to the control time and control range. The construction period loss is calculated according to the economic loss caused by the decrease of the traffic capacity of the bridge due to maintenance, and the conversion coefficient is 800 yuan for every 10% decrease of the hourly traffic capacity. The sum of the five costs is the total resource consumption, which is mapped to the interval of 0 to 1 by the maximum and minimum value normalization method, and the normalization formula is (current consumption - historical minimum consumption) ÷ (historical maximum consumption - historical minimum consumption). The normalized value is used as the cost item for real reward calculation.

[0115] The real reward = weighted state improvement - normalized resource consumption, with a value range of -362.5 to +307.5. A positive real reward indicates that the maintenance action benefit is greater than the cost, and a negative real reward indicates that the maintenance action cost is greater than the benefit. After the real reward is calculated, it is combined with the pre-maintenance state vector, the maintenance action code, and the post-maintenance state vector to form an experience sample. The experience sample is stored in a structured data format, with nine fields including sample identification, timestamp, pre-maintenance state vector 25-dimensional float number array, maintenance action code 4-bit string, post-maintenance state vector 25-dimensional float number array, real reward float number, resource consumption float number, work order number string, and acceptance result Boolean value. The sample identification is generated using UUID to ensure global uniqueness, and the timestamp records the time when the maintenance is completed, accurate to seconds.

[0116] The experience replay buffer is implemented using a circular queue data structure, with a queue capacity of 50,000 experience samples. The buffer is stored in a memory-mapped file, with a file path configured as experience_buffer.dat in the data directory. Each sample occupies a storage space of about 320 bytes, and the total file size is about 16 megabytes. The sample insertion operation is to append to the queue tail, and when the queue reaches the upper limit of the capacity, the oldest sample at the queue head is overwritten to realize circular use. The insertion operation is thread-safe, and the queue data structure is protected by read-write locks. When writing, a write lock is added to block all read operations, and when reading, a read lock is added to allow concurrent reading. The buffer provides a batch sampling interface, which supports filtering samples by time range, reward value range, and action category, and the sampling batch size can be configured in the range of 16 to 128, with a default value of 32.

[0117] The embodiment establishes a complete closed loop from maintenance action execution to experience sample generation, and the experience replay buffer provides high-quality training data support for continuous optimization of the policy network.

[0118] In an optional embodiment, the step of sampling a training sample from the experience replay buffer, for a sample with sparse real reward, retrieving a reference case similar to the sample state monitoring data in the historical maintenance record, taking the state change path of the reference case as a reference trajectory, and calculating a direction consistency score of the state change direction of the sample action and the reference trajectory as an instant auxiliary reward comprises:

[0119] sampling a training sample from the experience replay buffer, the training sample comprising sample pre-maintenance state monitoring data, sample post-maintenance state monitoring data, and sample real reward;

[0120] calculating a sparsity index of the sample real reward, and determining that the sample is a sample with sparse real reward when the sparsity index exceeds a preset sparsity threshold; for the sample with sparse real reward, calculating the similarity of the sample pre-maintenance state monitoring data and the state monitoring data of each historical case in the historical maintenance record, and extracting a number of historical cases with the highest similarity as reference cases;

[0121] obtaining a state change path of the reference case, the state change path comprising a pre-maintenance state vector and a post-maintenance state vector; calculating a reference state change vector based on the pre-maintenance state vector and the post-maintenance state vector, and constructing a reference trajectory by weighted averaging the reference state change vectors of a plurality of reference cases;

[0122] calculating the difference between the sample post-maintenance state monitoring data and the sample pre-maintenance state monitoring data as a sample state change vector;

[0123] calculating the direction cosine value between the reference trajectory and the sample state change vector as a direction consistency score, and normalizing the direction consistency score as an instant auxiliary reward.

[0124] Exemplarily, the training sample sampling is achieved by batch extraction from the experience replay buffer through a random sampling mechanism. The buffer maintains a circular queue of 50,000 experience samples, and the sampling module sets the batch size to 32. A uniform random number generator is used to generate 32 non-repeating index values, with the index range being 0 to the current capacity of the queue minus 1. The random number generator uses the Mason rotation algorithm to ensure the period length, and the seed value is initialized based on the system timestamp. The corresponding samples are read from the queue according to the index values, and each sample contains nine fields. The training process extracts four core fields: the pre-inspection state monitoring data field, the sample inspection action encoding field, the post-inspection state monitoring data field, and the sample true reward field. The pre-inspection state monitoring data is a 25-dimensional floating-point number vector, and the post-inspection state monitoring data is also a 25-dimensional floating-point number vector. The sample inspection action encoding is a 4-digit string, and the sample true reward is a floating-point value ranging from -362.5 to +307.5. The 32 samples obtained by sampling form a training batch, and the batch data is stored in a structured array in memory. Each element of the array contains the above four fields, and the array occupies about 10 kilobytes of memory.

[0125] The sparsity index calculation is based on the statistical distribution characteristics of the true reward values in the training batch. The average of the absolute values of the true reward values of the 32 samples in the batch is calculated as the reward amplitude mean, and the calculation method is to sum the absolute values of the 32 true reward values and divide by 32. The number of samples with true reward absolute value less than 20% of the reward amplitude mean is counted, and this number is divided by the batch size 32 to obtain the sparsity index, which takes a value in the range of 0 to 1. The sparsity threshold is preset to 0.5, and when the sparsity index exceeds 0.5, it is determined that the current batch has a true reward sparsity phenomenon. The true reward absolute value is checked sample by sample to see if it is less than 20% of the reward amplitude mean. The samples that meet the condition are marked as true reward sparse samples. The marking process adds a sparse flag field in the training batch array, and the field type is Boolean. The sparse sample flag is set to true, and the non-sparse sample flag is set to false. The sparse sample set extracts all samples with a sparse flag of true from the training batch, and the set size ranges from 0 to 32.

[0126] The historical maintenance record database contains complete archives of all past completed maintenance, and the database table structure contains four main fields of historical record identification field, pre-maintenance state vector field, maintenance action coding field, and post-maintenance long-term state vector field. The pre-maintenance state vector field stores a 25-dimensional floating-point number array, and the post-maintenance long-term state vector field also stores a 25-dimensional floating-point number array. The maintenance action coding field is a 4-digit string. Similarity calculation is carried out for each sparse sample of pre-maintenance state monitoring data and the pre-maintenance state vector of each historical case in the historical record. The similarity measure adopts inverse normalization processing of Euclidean distance, the distance calculation is the square root of the sum of the square differences of the corresponding dimensions of the sample state vector and the historical state vector, the distance value is denoted as d, the similarity value is calculated as 1 / (d+0.1), and the distance value is added by 0.1 to avoid division by zero error, and the similarity value is in the range of 0.1 to 10. All cases in the historical maintenance record database are traversed, and the similarity value is calculated for each case. The similarity value and the case identification are stored in a temporary sorting list in the form of key-value pairs. The sorting list is sorted in descending order of similarity value, and the top 10 cases after sorting are extracted as the reference case set. The number of reference cases is set to 10, and when the total number of historical records is less than 10, the actual available number is taken. The reference case set contains 10 case identifications and corresponding similarity values.

[0127] The state change path is extracted from the reference case set. The pre-maintenance state vector and the post-maintenance long-term state vector are queried from the historical maintenance record database according to the case identification, and both vectors are 25-dimensional floating-point number arrays. The reference state change vector is calculated as the post-maintenance long-term state vector minus the pre-maintenance state vector. The subtraction operation is to subtract each dimension one by one to produce a 25-dimensional difference vector. A positive value in each dimension of the difference vector indicates that the state is improved, and a negative value indicates that the state is deteriorated. Ten reference cases produce ten reference state change vectors. Weighted average calculation is based on the similarity value of each reference case as a weight coefficient. The weight of the kth reference state change vector is calculated as the similarity value of the case divided by the sum of the similarity values of the ten cases. The weight value is in the range of 0 to 1, and the sum of the ten weights is equal to 1. The i-th component of the reference trajectory is calculated as the sum of the i-th components of the ten reference state change vectors multiplied by the corresponding weights, i is from 1 to 25, and a 25-dimensional reference trajectory vector is generated. The reference trajectory vector represents the typical state change direction of the maintenance action in the similar historical scenario.

[0128] The sample state change vector is calculated based on the pre-inspection state data and the post-inspection state data of the sparse sample. The post-inspection state monitoring data of the sample is subtracted from the pre-inspection state monitoring data of the sample, and the subtraction is a subtraction of corresponding dimensions one by one, to generate a 25-dimensional sample state change vector. The vector represents the direction and amplitude of the state change of the current sample after the actual execution of the inspection action. The direction consistency score is obtained by calculating the direction cosine value of the reference trajectory vector and the sample state change vector. The inner product of the two vectors is calculated as the sum of the products of the corresponding dimensions, that is, the multiplication of the first dimension component is added to the multiplication of the second dimension component, and then the multiplication of the 25th dimension component is added, and the inner product value is denoted as p. The length of the reference trajectory vector is calculated as the square root of the sum of the squares of each dimension, and the length is denoted as m1. The length of the sample state change vector is calculated as the square root of the sum of the squares of each dimension, and the length is denoted as m2. The direction cosine value is calculated as p / (m1*m2). The direction cosine value ranges from -1 to +1. A value of +1 indicates complete consistency in direction, a value of -1 indicates complete opposite direction, and a value of 0 indicates orthogonal independence. The normalization processing maps the direction cosine value from the interval of -1 to +1 to the interval of 0 to 1, and the mapping calculation is (direction cosine value+1) / 2. The normalized value is used as the immediate auxiliary reward, and the immediate auxiliary reward ranges from 0 to 1.

[0129] The embodiment establishes an auxiliary reward triggering mechanism through sparse degree adaptive identification, provides reliable reference trajectories through historical similar case retrieval, and quantifies the rationality of the direction consistency score, thereby supplementing the immediate feedback signal for the reward sparse scene, accelerating the convergence of the strategy network, and improving the training stability.

[0130] In an optional embodiment, the step of updating the double-layer strategy network parameters by using the real reward and the immediate auxiliary reward comprises:

[0131] For the real reward sparse samples in the training samples, the corresponding immediate auxiliary reward and the real reward are combined to form a comprehensive reward signal; for the real reward non-sparse samples in the training samples, the real reward is directly used as the comprehensive reward signal;

[0132] The pre-inspection state monitoring data of the training sample is input into the main strategy network to obtain the predicted inspection action output by the main strategy network; a strategy loss function value between the predicted inspection action and the actual executed sample inspection action in the training sample is calculated, and the strategy loss function value is weighted according to the comprehensive reward signal;

[0133] The pre-inspection state monitoring data of the training sample and the sample inspection action are input into the shadow strategy network to obtain the predicted post-inspection state; a state transition prediction loss function value between the predicted post-inspection state and the actual post-inspection state monitoring data of the sample in the training sample is calculated;

[0134] The weighted policy loss function value of the main policy network and the state transition prediction loss function value of the shadow policy network are combined to form a total loss function; the main policy network parameters and the shadow policy network parameters are updated according to the total loss function through a gradient back propagation algorithm.

[0135] For example, the construction of the comprehensive reward signal is based on the classification processing of the sparse flag field of each sample in the training batch. The training batch contains 32 samples, each carrying a sparse flag Boolean value, a real reward floating point value, and an instant auxiliary reward floating point value. Each sample in the training batch is traversed, and the sparse flag field is read. The sample with a true sparse flag is determined as a real reward sparse sample, and the sample with a false sparse flag is determined as a real reward non-sparse sample. For the real reward sparse sample, the real reward value and the instant auxiliary reward value of the sample are read, and the comprehensive reward signal = real reward value × α + instant auxiliary reward value × β. The sum of the weighting coefficients α and β is equal to 1, the value range of α is 0.2 to 0.4, the value range of β is 0.6 to 0.8, and the default configuration α is 0.3 to ensure that the auxiliary reward dominates the learning signal in the sparse scene. For the real reward non-sparse sample, the comprehensive reward signal is directly assigned as the real reward value of the sample, without introducing the instant auxiliary reward. The comprehensive reward signal is stored in the newly added field of the training batch array, and the field type is floating point. The value range of the comprehensive reward of the sparse sample is 0 to 1, and the value range of the comprehensive reward of the non-sparse sample is -362.5 to +307.5. After the traversal is completed, the 32 samples in the training batch all have the comprehensive reward signal field, which is used for subsequent loss function calculation and gradient update.

[0136] The main policy network receives the pre-inspection state monitoring data of the sample as the input tensor. The main policy network structure is a four-layer fully connected neural network, the input layer node number is 25 corresponding to the state monitoring data dimension, the first hidden layer node number is 128, the second hidden layer node number is 64, and the output layer node number is 200 corresponding to the number of optional inspection action encodings. The activation function uses the ReLU function, and the output layer activation function uses the Softmax function to generate the action probability distribution. The pre-inspection state monitoring data of the 32 samples in the training batch form the input tensor, and the tensor shape is 32 rows and 25 columns, and the data type is 32-bit floating point number. The input tensor is sent to the main policy network for forward propagation, the first hidden layer output tensor shape is 32 rows and 128 columns, the second hidden layer output tensor shape is 32 rows and 64 columns, and the output layer output tensor shape is 32 rows and 200 columns. Each row of the output layer is a 200-dimensional probability vector, and each element of the vector has a value range of 0 to 1 and a sum equal to 1. The index corresponding to the element with the maximum probability is taken as the predicted inspection action encoding, and the index range 0 to 199 is mapped to a four-digit string, and the mapping rule is that the index value plus 1000 is converted to a string form. 32 predicted inspection action encodings are generated from 32 samples, and the encodings are stored in the predicted action array.

[0137] The policy loss function value is calculated based on the cross-entropy metric of the predicted maintenance action and the actual sample maintenance action performed in the training sample. Each sample in the training batch carries a sample maintenance action field, which is a four-digit string. The string is converted to an integer and subtracted by 1000 to obtain the action index, which ranges from 0 to 199. The action index is converted to a one-hot encoding vector, with a length of 200. The action index corresponds to the position element of 1, and the remaining position elements are 0. The action one-hot encoding vectors of 32 samples form a label tensor, with a shape of 32 rows and 200 columns. The cross-entropy loss is calculated between the probability tensor output by the main policy network and the label tensor. The calculation method is to multiply each element of the label tensor by the negative value of the logarithm of the corresponding element of the probability tensor, and then sum the element-wise product to obtain the cross-entropy value of each sample. The cross-entropy value is stored in the loss vector, which has a length of 32. The weighting of the comprehensive reward signal to the policy loss function value is achieved by multiplying each sample. The negative value of the k-th element of the loss vector multiplied by the k-th element of the comprehensive reward signal gives the weighted loss value, and the negative value conversion ensures that high reward samples produce low loss contributions. The average of the 32 weighted loss values gives the weighted policy loss function value of the main policy network, which is a scalar floating-point number with a value range of 0 to positive infinity.

[0138] The shadow policy network receives the sample maintenance pre-state monitoring data and the sample maintenance action as a spliced input tensor. The shadow policy network structure is a three-layer fully connected neural network, with 225 input layer nodes, of which the first 25 dimensions are state monitoring data and the last 200 dimensions are one-hot encoding of maintenance actions. The first hidden layer has 128 nodes, the second hidden layer has 64 nodes, and the output layer has 25 nodes corresponding to the dimension of the predicted post-maintenance state monitoring data. The activation function uses the ReLU function, and the output layer directly outputs linear values without activation function. The pre-maintenance state monitoring data of 32 samples in the training batch is spliced with the maintenance action one-hot encoding to form an input tensor, with a shape of 32 rows and 225 columns. The input tensor is fed into the shadow policy network for forward propagation, with the first hidden layer output tensor shape of 32 rows and 128 columns, the second hidden layer output tensor shape of 32 rows and 64 columns, and the output layer output tensor shape of 32 rows and 25 columns. Each row of the output tensor is a 25-dimensional predicted post-maintenance state vector, with each element being a floating-point number with the same value range as the actual state monitoring data.

[0139] The state transition prediction loss function value is calculated based on the mean square error of the predicted post-maintenance state and the actual post-maintenance state monitoring data of the training samples. Each sample in the training batch carries a post-maintenance state monitoring data field, which is a 25-dimensional floating-point number vector. The post-maintenance state monitoring data of 32 samples form a true value tensor with a shape of 32 rows and 25 columns. The mean square error loss is calculated between the predicted tensor output by the shadow policy network and the true value tensor. The calculation method is to square the difference between the corresponding elements of the predicted tensor and the true value tensor, sum the element-wise squares, and then sum the 25 dimensions to obtain the error sum of squares for each sample. The error sum of squares is divided by 25 to obtain the mean square error value for each sample, which is stored in the loss vector with a length of 32. The average of the 32 mean square error values is the state transition prediction loss function value of the shadow policy network, which is a scalar floating-point number with a value range of 0 to positive infinity.

[0140] The total loss function is composed of the linear combination of the weighted policy loss function value of the main policy network and the state transition prediction loss function value of the shadow policy network. Total loss function = weighted policy loss function value x γ + state transition prediction loss function value x δ. The sum of combination coefficients γ and δ is always equal to 1, γ has a value range of 0.5 to 0.7, δ has a value range of 0.3 to 0.5, and the default configuration γ is 0.6 to ensure that policy optimization dominates the training process. The total loss function is a scalar floating-point number with a value range of 0 to positive infinity, which is used as the target loss for gradient backpropagation.

[0141] The gradient backpropagation algorithm calculates the partial derivatives of the total loss function with respect to the parameters of the main policy network and the shadow policy network. The parameters of the main policy network include the weight matrices and bias vectors of the four fully connected layers, with shapes of 25 rows x 128 columns, 128 rows x 64 columns, 64 rows x 200 columns, and lengths of 128, 64, and 200, respectively. The parameters of the shadow policy network include the weight matrices and bias vectors of the three fully connected layers, with shapes of 225 rows x 128 columns, 128 rows x 64 columns, 64 rows x 25 columns, and lengths of 128, 64, and 25, respectively. Backpropagation starts with the total loss function and calculates the gradient of each layer's parameters using the chain rule. Gradient calculation is completed using an automatic differentiation framework, which maintains a computation graph inside the framework to record the intermediate variables and operations during forward propagation. The computation graph is traversed in reverse to accumulate gradient contributions. The gradient tensors of the main policy network and the shadow policy network have the same shape as the corresponding parameter tensors, and the gradient values are floating-point numbers with a value range of negative infinity to positive infinity.

[0142] The parameter update adopts an adaptive moment estimation optimizer, the learning rate is set to 0.0001, the first-order moment estimation decay coefficient is set to 0.9, the second-order moment estimation decay coefficient is set to 0.999, and the numerical stability term is set to 0.00000001. The optimizer maintains the first-order moment vector and the second-order moment vector of each parameter, and the moment vectors are initialized as zero vectors. When updating the parameters, the gradient tensor is read, the first-order moment vector is updated as the old first-order moment vector x 0.9 + the gradient tensor x 0.1, and the second-order moment vector is updated as the old second-order moment vector x 0.999 + the square of the gradient tensor x 0.001. The first-order moment of the bias correction is calculated as the first-order moment vector ÷ (1-0.9 t ), the second-order moment of the bias correction is calculated as the second-order moment vector ÷ (1-0.9990 t ), and t is the iteration number. The parameter update amount is calculated as the learning rate 0.0001 x the first-order moment of the bias correction ÷ (the square root of the second-order moment of the bias correction + the numerical stability term 0.00000001), and the new parameter value is the old parameter value - the parameter update amount. The weight matrix and the bias vector of all the main strategy network and the shadow strategy network are updated according to the above rules, and the updated parameters are stored to overwrite the original parameters. After the parameter update of a single training batch is completed, the iteration number counter is incremented by 1, and the iteration number is used for bias correction calculation. The training process continues to execute the loop of sampling, forward propagation, loss calculation, back propagation, and parameter update, and the loop number is set to 50,000 times. Network parameter snapshots are saved to persistent storage every 1,000 times.

[0143] The embodiment balances real feedback and auxiliary guidance through a sparse adaptive reward combination mechanism, cooperatively optimizes the double-layer network to consider policy selection and state prediction, and ensures training stability and convergence efficiency through weighted loss and gradient update, thereby significantly improving the decision quality in a complex maintenance scene.

[0144] In a second aspect, a bridge maintenance strategy intelligent decision system based on deep reinforcement learning is provided, comprising:

[0145] A first unit is configured to acquire historical maintenance records of a bridge and construct an experience replay buffer;

[0146] A second unit is configured to establish a double-layer strategy network including a main strategy network and a shadow strategy network. The main strategy network generates a preliminary maintenance action according to current bridge state monitoring data. The shadow strategy network deduces multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculates a worst-case cumulative reward value. When the worst-case cumulative reward value is lower than a safety boundary, a conservative maintenance action is generated to replace the output;

[0147] A third unit is configured to execute the output maintenance action, acquire post-maintenance state monitoring data and real rewards, and store them in the experience replay buffer;

[0148] The fourth unit uses the training samples sampled from the experience replay buffer, and searches for a reference case similar to the sample state monitoring data in the historical maintenance record, takes the state change path of the reference case as a reference trajectory, and calculates a state change direction consistency score of the sample action with the direction of the reference trajectory as an instant auxiliary reward.

[0149] The fifth unit updates the double-layer policy network parameters by using the real reward and the instant auxiliary reward.

[0150] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions. When the computer program instructions are executed by a processor, the method described above is implemented.

Claims

1. A bridge maintenance strategy intelligent decision-making method based on deep reinforcement learning, characterized in that, The method comprises the following steps: acquiring historical maintenance records of the bridge to construct an experience replay buffer; establishing a double-layer strategy network comprising a main strategy network and a shadow strategy network, the main strategy network generating a preliminary maintenance action according to current bridge state monitoring data, the shadow strategy network deducing multiple future evolution trajectories of the preliminary maintenance action in a virtual environment through Monte Carlo tree search and calculating a worst-case cumulative reward value, and generating a conservative maintenance action to replace the output when the worst-case cumulative reward value is lower than a safety boundary, specifically comprising: establishing the main strategy network, converting the current bridge state monitoring data into a state feature vector, and outputting the preliminary maintenance action according to the state feature vector; establishing the shadow strategy network, training a state transition prediction function based on the state change path in the historical maintenance records, calculating the first-step predicted state after the execution of the preliminary maintenance action through the state transition prediction function according to the current state monitoring data and the preliminary maintenance action; starting from the first-step predicted state, iteratively selecting subsequent maintenance actions and calling the state transition prediction function to calculate the corresponding subsequent-step predicted states, and constructing multiple future evolution trajectories extending from the first-step predicted state to the terminal state; for each future evolution trajectory, accumulating the immediate reward values of the first-step predicted state and each subsequent-step predicted state node in the trajectory to obtain a cumulative reward value; extracting the minimum value of all trajectory cumulative reward values as the worst-case cumulative reward value, and generating the conservative maintenance action according to the safe trajectory with a cumulative reward value higher than the safety boundary to replace the preliminary maintenance action and then outputting the conservative maintenance action when the worst-case cumulative reward value is lower than the safety boundary; executing the output maintenance action, acquiring post-maintenance state monitoring data and a real reward, and storing them in the experience replay buffer; sampling training samples from the experience replay buffer, searching for a reference case similar to the sample state monitoring data in the historical maintenance records, taking the state change path of the reference case as a reference trajectory, and calculating the direction consistency score of the state change direction of the sample action with the direction of the reference trajectory as an immediate auxiliary reward; updating the double-layer strategy network parameters using the real reward and the immediate auxiliary reward.

2. The method of claim 1, wherein, The step of acquiring the historical maintenance records of the bridge to construct the experience replay buffer comprises: the historical maintenance records comprising state monitoring data at a historical maintenance time, a maintenance action at the historical maintenance time, and a post-maintenance state change path; performing multi-time-scale decomposition on the state monitoring data in the historical maintenance records to form a multi-dimensional state representation vector; classifying and coding the maintenance action in the historical maintenance records according to the action site and intervention intensity to construct a maintenance action coding dictionary; performing time sequence segmentation on the post-maintenance state change path in the historical maintenance records, extracting the evolution sequence of the pre-maintenance state, the immediate post-maintenance state, and the long-term post-maintenance state, and calculating the change trajectory vector of the state evolution sequence in the multi-dimensional state representation space; grouping the multi-dimensional state representation vector, the maintenance action coding in the maintenance action coding dictionary, and the change trajectory vector into a historical sample, and storing it in the experience replay buffer.

3. The method of claim 1, wherein, The step of constructing multiple future evolution trajectories extending from the first-step predicted state to the terminal state by starting from the first-step predicted state through Monte Carlo tree search, iteratively selecting subsequent maintenance actions and calling the state transition prediction function to calculate the corresponding subsequent-step predicted state comprises: Taking the first-step predicted state as the root node of the Monte Carlo tree, initializing the search tree; At the current leaf node of the search tree, extracting a historical case set similar to the current leaf node state according to the state change path in the historical maintenance record, counting the execution frequency and the corresponding state improvement amplitude of each maintenance action in the historical case set, and calculating the selection priority score; According to the selection priority score and the exploration coefficient, the subsequent maintenance action to be expanded is selected, the state transition prediction function is called to calculate the subsequent-step predicted state after the subsequent maintenance action is executed, and the subsequent-step predicted state is added to the search tree as a new child node; for the new child node, the state transition prediction function is used to deduce to the preset search depth or the terminal state, to obtain a complete evolution path and calculate a deduction reward value; the deduction reward value is updated along the evolution path to each node on the root node path, and the access frequency and the average reward value of each node are updated; Repeating the selection, expansion, deduction and updating process until a preset iteration number is reached, and extracting multiple different branch paths starting from the root node as multiple future evolution trajectories.

4. The method of claim 1, wherein, The step of executing the output maintenance action, obtaining the post-maintenance state monitoring data and the true reward, and storing them in the experience replay buffer comprises: Performing a bridge maintenance operation according to the output maintenance action; after the maintenance action is completed, collecting post-maintenance state monitoring data; Calculating the state improvement amount between the post-maintenance state monitoring data and the bridge state monitoring data before the maintenance action is performed, and calculating the true reward according to the state improvement amount and the resource consumption amount of the maintenance action; Storing the bridge state monitoring data before the maintenance action, the maintenance action, the post-maintenance state monitoring data and the true reward in the experience replay buffer to form an experience sample.

5. The method of claim 1, wherein, The step of sampling training samples from the experience replay buffer and calculating the consistency score of the state change direction of the sample action with the direction of the reference trajectory as the immediate auxiliary reward for the sample with sparse true reward comprises: Randomly sampling training samples from the experience replay buffer, wherein the training samples include sample pre-maintenance state monitoring data, sample post-maintenance state monitoring data and sample true reward; Calculating the sparsity index of the sample true reward, and determining that the sample has a sparse true reward when the sparsity index exceeds a preset sparsity threshold; for the sample with a sparse true reward, calculating the similarity between the sample pre-maintenance state monitoring data and the state monitoring data of each historical case in the historical maintenance record, and extracting a number of historical cases with the highest similarity as reference cases; Obtaining a state change path of a reference case, the state change path including a pre-inspection state vector and a post-inspection state vector; calculating a reference state change vector based on the pre-inspection state vector and the post-inspection state vector, and weighting and averaging the reference state change vectors of multiple reference cases to form a reference trajectory; Calculating a difference between the sample post-inspection state monitoring data and the sample pre-inspection state monitoring data as a sample state change vector; Calculating a direction cosine value between the reference trajectory and the sample state change vector as a direction consistency score, and normalizing the direction consistency score as an instant auxiliary reward.

6. The method of claim 5, wherein, The steps of updating the parameters of the double-layer policy network using the real reward and the instant auxiliary reward include: For the real reward sparse samples in the training samples, combining the corresponding instant auxiliary reward and the real reward to form a comprehensive reward signal; for the real reward non-sparse samples in the training samples, directly using the real reward as the comprehensive reward signal; Inputting the sample pre-inspection state monitoring data of the training sample into the main policy network to obtain a predicted inspection action output by the main policy network; calculating a policy loss function value between the predicted inspection action and the sample inspection action actually performed in the training sample, and weighting the policy loss function value according to the comprehensive reward signal; Inputting the sample pre-inspection state monitoring data and the sample inspection action of the training sample into the shadow policy network to obtain a predicted post-inspection state; calculating a state transition prediction loss function value between the predicted post-inspection state and the actual sample post-inspection state monitoring data in the training sample; Combining the weighted policy loss function value of the main policy network and the state transition prediction loss function value of the shadow policy network to form a total loss function; updating the parameters of the main policy network and the parameters of the shadow policy network according to the total loss function through a gradient back propagation algorithm.

7. A bridge inspection strategy intelligent decision system based on deep reinforcement learning, for implementing the method of any one of the preceding claims 1-6, characterized in that, It includes: A first unit configured to obtain historical inspection records of a bridge and construct an experience replay buffer; A second unit configured to establish a double-layer policy network including a main policy network and a shadow policy network, the main policy network being configured to generate a preliminary inspection action according to current bridge state monitoring data, and the shadow policy network being configured to deduce multiple future evolution trajectories of the preliminary inspection action in a virtual environment and calculate a worst-case cumulative reward value through Monte Carlo tree search, and generate a conservative inspection action to replace the output when the worst-case cumulative reward value is lower than a safety boundary; A third unit configured to execute the output inspection action, obtain post-inspection state monitoring data and a real reward, and store the post-inspection state monitoring data and the real reward in the experience replay buffer; A fourth unit configured to sample training samples from the experience replay buffer, retrieve reference cases similar to the sample state monitoring data in the historical inspection records for the real reward sparse samples, use the state change path of the reference cases as a reference trajectory, and calculate a direction consistency score between the state change direction of the sample action and the reference trajectory as an instant auxiliary reward; A fifth unit configured to update the parameters of the double-layer policy network using the real reward and the instant auxiliary reward.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 6. The computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 6.

Images