Self-adaptive control method for tunnel construction environment in cold region

Abstract

The invention provides a cold region tunnel construction environment adaptive control method, and relates to the technical field of tunnel construction, and the method comprises the steps: obtaining multi-dimensional monitoring data, employing a multi-layer neural network and combining an attention mechanism to extract features, and outputting a freezing and thawing working condition type and a dynamic importance weight; constructing a combination state based on the working condition identifier and the construction record, and generating a control strategy by using reinforcement learning; adjusting a control parameter according to the dynamic weight to generate an instruction set; executing the instruction and analyzing feedback data to adjust system parameters. The control strategy can be adaptively adjusted according to the dynamic change of the environment, and the construction safety and efficiency of the tunnel in the cold region are improved.

Description

Technical Field

[0001] This invention relates to tunnel construction technology, and more particularly to an adaptive control method for tunnel construction environments in cold regions. Background Technology

[0002] Tunnel construction in cold regions is a crucial aspect of civil engineering, as its construction environment is significantly affected by factors such as temperature variations and freeze-thaw cycles. Existing environmental monitoring methods for tunnel construction in cold regions typically employ fixed parameters or preset thresholds for environmental control, lacking adaptive learning capabilities. These methods cannot dynamically adjust control strategies based on different freeze-thaw conditions and construction stages, resulting in low control accuracy, slow response times, and potentially exacerbating freeze-thaw disaster risks due to improper control. Summary of the Invention

[0003] This invention provides an adaptive control method for tunnel construction in cold regions, which can solve the problems in the prior art.

[0004] A first aspect of the present invention provides an adaptive control method for tunnel construction environments in cold regions, comprising:

[0005] Acquire multidimensional monitoring data of the tunnel construction environment in cold regions; perform multi-layer neural network feature extraction on the multidimensional monitoring data, calculate the contribution of each feature component to the judgment of freeze-thaw conditions based on the attention mechanism, and perform weighted fusion to output the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source.

[0006] Based on the freeze-thaw condition category identifier and construction procedure record data, a combined state is constructed, and environmental control strategy parameters are generated through a policy network using a reinforcement learning agent.

[0007] The dynamic importance weights are used as control modulation coefficients to adjust the environmental control strategy parameters, thereby generating a comprehensive control instruction set.

[0008] The integrated control instruction set is executed and feedback data is collected; the feedback data is matched with the historical working condition evolution trajectory for similarity, and trend features are extracted as feedforward signals to adjust the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent.

[0009] The steps of performing multi-layer neural network feature extraction on the multi-dimensional monitoring data, calculating the contribution of each feature component to the freeze-thaw condition discrimination based on the attention mechanism, performing weighted fusion, and outputting the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source include:

[0010] A primary spatial feature map is obtained by extracting spatial dimension features from the multidimensional monitoring data through a convolutional layer, and a spatiotemporal fusion feature vector is obtained by extracting temporal dimension features from the primary spatial feature map through a recurrent layer.

[0011] Based on the spatiotemporal fusion feature vector, a query vector, a key vector, and a value vector are generated respectively. An attention score matrix is ​​obtained by performing a dot product operation between the query vector and the key vector.

[0012] The attention score matrix is ​​normalized to obtain a normalized attention weight matrix. Each element in the normalized attention weight matrix represents the contribution of the corresponding feature component to the freeze-thaw condition discrimination.

[0013] The normalized attention weight matrix and the value vector are weighted and summed to obtain a weighted fusion feature vector. The weighted fusion feature vector is input into the classifier to output the freeze-thaw condition category identifier. The average weight of the normalized attention weight matrix in the corresponding dimension of each monitoring data source is extracted as the dynamic importance weight of each monitoring data source.

[0014] The steps of constructing a combined state based on the freeze-thaw condition category identifier and construction procedure record data, and generating environmental control policy parameters using a reinforcement learning agent through a policy network, include:

[0015] The freeze-thaw condition category identifier is converted into a unique thermal coding vector; the construction process record data is subjected to temporal feature extraction to obtain a process progress vector and a process duration vector, which are then concatenated with the unique thermal coding vector to obtain a combined state vector;

[0016] The combined state vector is input into the policy network of the reinforcement learning agent, and the policy network performs a nonlinear mapping on the combined state vector through a multi-layer fully connected transformation.

[0017] The output of the strategy network includes a mean vector and a variance vector. The mean vector represents the expected value of the target control parameter of each environmental control device, and the variance vector represents the uncertainty of the target control parameter of each environmental control device. A parameter distribution function is constructed based on the mean vector and the variance vector, and environmental control strategy parameters are sampled from the parameter distribution function.

[0018] The steps of constructing a parameter distribution function based on the mean vector and the variance vector, and sampling environmental control strategy parameters from the parameter distribution function include:

[0019] The variance vector is subjected to a non-negative constraint transformation to obtain a constraint variance vector. The constraint variance vector is then compared element-by-element with a preset lower bound of variance. When an element in the constraint variance vector is less than the corresponding lower bound of variance, the element is replaced with the lower bound of variance value to obtain a corrected variance vector.

[0020] Obtain the physical boundaries of the control parameters of each environmental control device; construct a truncated normal distribution function as the parameter distribution function based on the mean vector, the corrected variance vector, and the physical boundaries of the control parameters.

[0021] The truncated normal distribution function is sampled by reparameterization transformation, which converts the sampling operation into a combination operation of deterministic function and independent noise variable. Candidate control parameter samples are generated from the truncated normal distribution function, and the environmental control strategy parameters are obtained by boundary truncation of the candidate control parameter samples.

[0022] The step of adjusting the environmental control strategy parameters using the dynamic importance weights as control modulation coefficients to generate a comprehensive control instruction set includes:

[0023] Obtain the time series of dynamic importance weights for each monitoring data source, and calculate the fluctuation variance and decay rate for each data source; when the fluctuation variance of a data source is greater than a preset fluctuation threshold, the data source is marked as environmental source uncertainty; when the fluctuation variance of a data source is not greater than a preset fluctuation threshold and the decay rate is greater than a preset decay threshold, the data source is marked as equipment source uncertainty.

[0024] Establish a dependency matrix between monitoring data sources and environmental control equipment, and obtain the variance vector corresponding to each environmental control equipment;

[0025] For each control device, the weighted sum of the uncertain data sources of the environment sources it depends on is calculated according to the dependency matrix to obtain the data transmission uncertainty component; the data transmission uncertainty component is obtained by subtracting the device variance vector from the device variance vector.

[0026] For data sources marked as device-source uncertainty, their weights are used as suppression coefficients to reduce the control parameters of control devices that depend on these data sources; for data sources marked as environmental-source uncertainty, their inverse weights are used as compensation coefficients to amplify the control parameters; for control devices whose inherent uncertainty components exceed the threshold, their control parameters are reverted to historical safe parameters; and the control parameters of each control device after differential modulation are encapsulated into a comprehensive control instruction set.

[0027] The steps of executing the integrated control instruction set and collecting feedback data, matching the feedback data with the historical operating condition evolution trajectory, extracting trend features as feedforward signals, and adjusting the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent include:

[0028] The integrated control instruction set is sent to each environmental control device for execution, and environmental monitoring data is continuously collected and combined into a feedback data sequence.

[0029] The feedback data sequence is dynamically time-warped and matched with the historical working condition evolution trajectory in the historical database. Historical trajectories with similarity scores exceeding a preset score threshold are selected as a set of similar working conditions. The subsequent evolution path of each historical trajectory in the set of similar working conditions after the matching time is extracted. The subsequent evolution path is statistically analyzed to obtain the probability distribution of the environmental state evolution direction, which is encoded as a trend feature vector.

[0030] Obtain the parameter matrix used to calculate attention weights in the attention mechanism; use the trend feature vector as a feedforward signal, and adjust the weight parameters of the monitoring data source corresponding to the evolution direction whose probability value exceeds a preset probability threshold in the parameter matrix according to the probability value of each environmental state evolution direction; update the adjusted parameter matrix to the attention mechanism.

[0031] The association rules between the control decision sequence and the final control effect evaluation index are extracted from the set of similar operating conditions, and the set of decision features that lead to positive control effect and the set of decision features that lead to negative control effect are identified.

[0032] Obtain the reward function of the reinforcement learning agent, wherein the reward function includes reward weight parameters and penalty weight parameters corresponding to the decision features;

[0033] The corresponding reward weight parameter is increased based on the decision feature set of the positive control effect, and the corresponding penalty weight parameter is increased based on the decision feature set of the negative control effect, generating adjusted reward function parameters; the adjusted reward function parameters are then updated to the reinforcement learning agent.

[0034] A second aspect of the present invention provides an electronic device, comprising:

[0035] processor;

[0036] Memory used to store processor-executable instructions;

[0037] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0038] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0039] This invention innovatively uses dynamic importance weights as control modulation coefficients to adjust environmental control strategy parameters, enabling the control strategy to adaptively adjust according to the importance of different monitoring data sources, thus enhancing its adaptability to complex environmental changes in cold regions. By matching feedback data with historical operating condition evolution trajectories and extracting trend features as feedforward signals, dynamic adjustment of the attention mechanism contribution calculation rules and the reinforcement learning agent reward function is achieved, forming a closed-loop self-optimization mechanism. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the adaptive control method for tunnel construction environment in cold regions according to an embodiment of the present invention.

[0041] Figure 2 This is a flowchart illustrating the generation of environmental control policy parameters based on a reinforcement learning agent policy network. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0044] Figure 1 This is a flowchart illustrating the adaptive control method for tunnel construction environment in cold regions according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0045] Acquire multidimensional monitoring data of the tunnel construction environment in cold regions; perform multi-layer neural network feature extraction on the multidimensional monitoring data, calculate the contribution of each feature component to the judgment of freeze-thaw conditions based on the attention mechanism, and perform weighted fusion to output the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source.

[0046] Based on the freeze-thaw condition category identifier and construction procedure record data, a combined state is constructed, and environmental control strategy parameters are generated through a policy network using a reinforcement learning agent.

[0047] The dynamic importance weights are used as control modulation coefficients to adjust the environmental control strategy parameters, thereby generating a comprehensive control instruction set.

[0048] The integrated control instruction set is executed and feedback data is collected; the feedback data is matched with the historical working condition evolution trajectory for similarity, and trend features are extracted as feedforward signals to adjust the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent.

[0049] In one optional implementation, the steps of performing multi-layer neural network feature extraction on the multi-dimensional monitoring data, calculating the contribution of each feature component to the freeze-thaw condition discrimination based on an attention mechanism, performing weighted fusion, and outputting the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source include:

[0050] A primary spatial feature map is obtained by extracting spatial dimension features from the multidimensional monitoring data through a convolutional layer, and a spatiotemporal fusion feature vector is obtained by extracting temporal dimension features from the primary spatial feature map through a recurrent layer.

[0051] Based on the spatiotemporal fusion feature vector, a query vector, a key vector, and a value vector are generated respectively. An attention score matrix is ​​obtained by performing a dot product operation between the query vector and the key vector.

[0052] The attention score matrix is ​​normalized to obtain a normalized attention weight matrix. Each element in the normalized attention weight matrix represents the contribution of the corresponding feature component to the freeze-thaw condition discrimination.

[0053] The normalized attention weight matrix and the value vector are weighted and summed to obtain a weighted fusion feature vector. The weighted fusion feature vector is input into the classifier to output the freeze-thaw condition category identifier. The average weight of the normalized attention weight matrix in the corresponding dimension of each monitoring data source is extracted as the dynamic importance weight of each monitoring data source.

[0054] For example, the acquisition of multidimensional monitoring data is accomplished through a sensor array deployed at the tunnel construction site. Sensor types include temperature sensors, humidity sensors, surrounding rock stress sensors, frozen soil depth detectors, and wind speed sensors. Each sensor is arranged in a spatial grid with a grid spacing of 5 to 10 meters, and the sampling frequency is uniformly set to once per minute. The collected raw data undergoes outlier removal and missing value imputation to form multidimensional monitoring data. Data dimensions include spatial location coordinates, sensor type identifiers, timestamps, and corresponding physical measurement values. The multidimensional monitoring data is organized in tensor form. The first dimension of the tensor represents the spatial location index, the second dimension represents the sensor type, and the third dimension represents the time series window. The window length is set to 60 sampling points, corresponding to one hour of historical data. The tensor values ​​are 32-bit floating-point numbers to balance computational precision and storage overhead.

[0055] The convolutional layer performs spatial dimension feature extraction on the multidimensional monitoring data tensor. The kernel size is set to 3×3, the stride is set to 1, and zero padding is used to ensure that the output feature map is consistent with the input spatial size. The convolutional layer contains 64 convolutional kernels. Each convolutional kernel performs sliding computation on the spatial neighborhood of the input tensor, weighting and accumulating the sensor data of the local region before applying a nonlinear activation function. The activation function uses a modified linear unit to introduce nonlinear expressive power. The primary spatial feature map output by the convolutional layer retains spatial location information and maps the original sensor data to an abstract spatial feature representation. The feature map has 64 channels, corresponding to the output of 64 convolutional kernels. The primary spatial feature map is then input into the recurrent layer for temporal dimension feature extraction. The recurrent layer uses a long short-term memory (LSTM) structure with 128 units and a time step size corresponding to the time series window length of the input tensor, which is 60 time steps. At each time step, the recurrent layer receives the feature slices corresponding to the time in the primary spatial feature map and updates the hidden state and cell state through a gating mechanism. The hidden state dimension is set to 128 dimensions. After the recurrent layer processes the entire time series, the hidden state of the last time step is taken as the spatiotemporal fusion feature vector. This vector integrates information from both the spatial and temporal dimensions and has a length of 128.

[0056] The spatiotemporal fusion feature vector generates query vector, key vector, and value vector through three independent linear transformation layers. The weight matrix of each linear transformation layer has a dimension of 128×128, and the bias vector has a dimension of 128. The query vector, key vector, and value vector all have a dimension of 128, representing the mapping of the spatiotemporal fusion features to the vector space required for attention computation. When performing a dot product operation between the query vector and the key vector, the key vector is first transposed to obtain a 128×1 column vector. Then, the inner product of the query vector and the transposed key vector is calculated. The inner product result is a scalar value representing the relevance strength between the query and the key. Since the spatiotemporal fusion feature vector contains 128 feature components, the dot product operation actually matches the query and key corresponding to each feature component one by one, generating 128 relevance scores to form an attention score matrix. The attention score matrix has a dimension of 128×128, where the element in the i-th row and j-th column represents the attention score of the i-th feature component on the j-th feature component.

[0057] The attention score matrix is ​​normalized using row-wise exponential normalization. For each row, the exponential function value is calculated for all elements, and then divided by the sum of all exponential values ​​in that row, ensuring that the sum of elements in each row is 1. A temperature coefficient is introduced to scale the attention score during normalization. This temperature coefficient is set to the reciprocal of the square root of the feature dimension, approximately 0.088. The scaling operation is performed by multiplying each element of the attention score matrix by the temperature coefficient, and then the scaled score is subjected to exponential normalization again. The values ​​of each element in the normalized attention weight matrix range from 0 to 1; a larger element value indicates a higher contribution of the corresponding feature component to the freeze-thaw condition judgment. When the normalized attention weight matrix is ​​weighted and summed with the value vector, matrix multiplication is performed. The value vector has a dimension of 128×1, and the result is a 128-dimensional weighted fused feature vector. The weighted fused feature vector integrates the information of each feature component weighted according to its contribution.

[0058] Weighted fusion feature vectors are input into a classifier to classify freeze-thaw conditions. The classifier consists of two fully connected layers. The first layer maps the 128-dimensional vector to a 64-dimensional intermediate representation and applies a modified linear unit activation. The second layer maps the 64-dimensional intermediate representation to the output dimension corresponding to the number of freeze-thaw condition categories. Freeze-thaw condition categories are defined as four states: complete freezing, partial freezing, alternating freeze-thaw, and complete thawing. The output dimension is set to 4. The last layer of the classifier applies an exponential normalization function to convert the output into a probability distribution, selecting the category index with the highest probability as the freeze-thaw condition category identifier. The weights of the normalized attention weight matrix on the corresponding dimensions of each monitoring data source are extracted through dimension aggregation. Specifically, a mapping relationship is established between the monitoring data source and the feature components. For example, the first 32 feature components correspond to temperature sensor data, the 33rd to 64th feature components correspond to humidity sensor data, and subsequent feature components correspond to other sensor types. The aggregated weight value for the corresponding monitoring data source is obtained by summing or averaging every 32 columns of the normalized attention weight matrix. This aggregated weight value serves as the dynamic importance weight for each monitoring data source. The values ​​of dynamic importance weights reflect the degree of contribution of each monitoring data source to the judgment of freeze-thaw conditions under the current spatiotemporal fusion characteristics. Data sources with larger weight values ​​indicate that the monitoring information they provide has a more significant impact on the judgment results.

[0059] This invention automatically learns the contribution of each monitoring data source to the judgment of freeze-thaw conditions through an attention mechanism, avoiding the subjectivity of manually setting weights and improving the accuracy of condition identification.

[0060] In one optional implementation, the step of constructing a combined state based on the freeze-thaw condition category identifier and construction procedure record data, and generating environmental control policy parameters using a reinforcement learning agent through a policy network, includes:

[0061] The freeze-thaw condition category identifier is converted into a unique thermal coding vector; the construction process record data is subjected to temporal feature extraction to obtain a process progress vector and a process duration vector, which are then concatenated with the unique thermal coding vector to obtain a combined state vector;

[0062] The combined state vector is input into the policy network of the reinforcement learning agent, and the policy network performs a nonlinear mapping on the combined state vector through a multi-layer fully connected transformation.

[0063] The output of the strategy network includes a mean vector and a variance vector. The mean vector represents the expected value of the target control parameter of each environmental control device, and the variance vector represents the uncertainty of the target control parameter of each environmental control device. A parameter distribution function is constructed based on the mean vector and the variance vector, and environmental control strategy parameters are sampled from the parameter distribution function.

[0064] Combination Figure 2 This explanation uses a flowchart of the environmental control policy parameters generated by a reinforcement learning agent policy network. For example, the process of converting freeze-thaw condition category identifiers into one-hot encoded vectors is implemented through a predefined category mapping table. This table assigns indices 0 to 3 to the four condition states: fully frozen, partially frozen, alternating freeze-thaw, and fully thawed. The one-hot encoded vector has a length of 4, with only one element valued at 1 and the rest at 0. The position of the element with the value 1 is determined by the index of the freeze-thaw condition category identifier. For example, when the freeze-thaw condition category identifier is alternating freeze-thaw, it corresponds to index 2, and the one-hot encoded vector is represented as a 4-dimensional vector with positions 0, 0, 1, and 0 for positions 0 to 3. The one-hot encoding uses 32-bit floating-point storage, and the generated vector is stored in a memory buffer for subsequent concatenation operations.

[0065] Construction process record data originates from the process management system at the construction site. Data fields include process name, process start timestamp, process end timestamp, process progress percentage, and process status identifier. The process record data is arranged in ascending order of timestamps to form a time-series sequence, with the sequence length corresponding to the number of completed or ongoing processes within the construction cycle. Time-series feature extraction encodes the time and progress dimensions of the process record data. The process progress vector is constructed by extracting the progress percentage value of each process; the vector length equals the number of processes, and each element's value ranges from 0 to 100, representing the completion degree of the corresponding process. The process duration vector is obtained by calculating the difference between the end timestamp and the start timestamp of each process, with the difference in hours and rounded to one decimal place; the vector length is also equal to the number of processes. When a process is not yet completed, the end timestamp is replaced with the current system timestamp to calculate the real-time duration. The process progress vector and process duration vector undergo normalization processing. Each element of the progress vector is divided by 100 to map to the 0-1 range, and each element of the duration vector is divided by a preset maximum duration threshold of 240 hours (10 days), also mapping to the 0-1 range. Elements exceeding the threshold are truncated to 1.

[0066] The combined state vector is formed by concatenating the one-hot encoded vector, the process progress vector, and the process duration vector. The concatenation operation arranges the vector elements sequentially according to their dimensions. Assuming the construction cycle includes 5 processes, the process progress vector has a length of 5, the process duration vector has a length of 5, and the one-hot encoded vector has a length of 4, the length of the concatenated combined state vector is 14 (4+5+5). The first 4 elements of the combined state vector correspond to the one-hot encoded data for the freeze-thaw condition, the 5th to 9th elements correspond to the process progress, and the 10th to 14th elements correspond to the process duration. After concatenation, the combined state vector is stored as a contiguous block of memory, with the data type being a 32-bit floating-point array. The array pointer is passed to the input interface of the policy network.

[0067] The policy network of the reinforcement learning agent adopts a multi-layer fully connected neural network architecture with three layers. The first layer has an input dimension equal to the length of the combined state vector (14) and an output dimension of 64. The second layer has an input dimension of 64 and an output dimension of 32. The third layer has an input dimension of 32 and an output dimension equal to twice the number of environmental control devices. The environmental control devices include heating, ventilation, and dehumidification equipment (3 types in total). The output dimension is set to 6 (3×2), with the first three outputs corresponding to the mean vector and the last three to the variance vector. The fully connected transformation at each layer involves matrix multiplication of the weight matrix with the input vector. After adding the bias vector to the multiplication result, an activation function is applied. The first two layers use modified linear units (MLUs) as activation functions, while the third layer does not apply an activation function to maintain the unrestricted range of the mean vector. The weight matrix and bias vector parameters are obtained through offline training. The training data comes from historical construction environment monitoring data and control decision records. The training algorithm uses the policy gradient method to iteratively update the network parameters. The number of iterations is set to 5000, the learning rate is set to 0.001, and the batch size is set to 64. During training, the reward function is designed based on control effect feedback. The reward signal contains three components: a negative deviation of the ambient temperature from the target temperature, a negative normalized value of energy consumption cost, and a positive reward for operational stability. Operational stability is calculated by quantizing the inverse of the variance of ambient temperature fluctuation within a continuous time window; the smaller the variance, the greater the stability value. The three components are weighted and summed using weighting coefficients of 0.5, 0.3, and 0.2. The state space is defined by a combined state vector, and the action space is a continuous space of parameter adjustments for each control device. Training data is extracted from historical records using state-action-reward triples. Each triple corresponds to a control decision and its subsequent environmental response data. The total number of triples is no less than 10,000 to ensure sufficient training samples.

[0068] The mean vector output by the strategy network represents the expected value of the target control parameters for each environmental control device. The vector length is 3. The first element corresponds to the target heating power of the heating device, the second to the target wind speed of the ventilation device, and the third to the target dehumidification capacity of the dehumidification device. The range of values ​​for each element in the mean vector is determined based on the physical constraints of the equipment: heating power ranges from 0 to 100 kW, wind speed from 0 to 10 m / s, and dehumidification capacity from 0 to 50 liters / hour. The variance vector represents the uncertainty of the target control parameters for each environmental control device. The vector length is also 3, and each element is a non-negative real number; a larger value indicates higher uncertainty for the corresponding control parameter. An exponential function transformation is applied after the variance vector is output to ensure all elements are positive. The transformed variance element values ​​are set to range from 0.01 to 10, with a lower limit of 0.01 to prevent excessively small variance from causing sampling degradation and an upper limit of 10 to prevent excessively large variance from causing drastic fluctuations in control parameters.

[0069] The parameter distribution function is constructed using a Gaussian distribution model. The mean parameter is taken from the corresponding element of the mean vector, and the variance parameter is taken from the corresponding element of the variance vector. For the i-th environmental control device, the distribution function of its control parameters is defined as a Gaussian distribution where the mean is equal to the i-th element of the mean vector and the variance is equal to the i-th element of the variance vector. The sampling operation generates random numbers following a standard Gaussian distribution, multiplies these random numbers by the square root of the corresponding element of the variance vector, and adds the product to the corresponding element of the mean vector to obtain sampled values ​​conforming to the parameter distribution function. If a sampled value exceeds the physical constraints of the device, boundary truncation is performed. When the sampled value is less than the minimum control parameter of the corresponding device, it is truncated to the minimum value; when the sampled value is greater than the maximum control parameter, it is truncated to the maximum value. The truncation operation ensures that all elements of the environmental control strategy parameters are within the feasible control range of each device, avoiding the issuance of control commands beyond the physical capabilities of the devices. The sampled environmental control strategy parameters are vectors of length 3, where each element corresponds to a specific control parameter value for each environmental control device. The sampling process introduces randomness to enable strategy exploration, while constraints based on mean and variance prevent sampled values ​​from deviating from a reasonable range.

[0070] This invention combines multi-dimensional state information from freeze-thaw conditions and construction procedures, and uses a strategy network to automatically generate control parameters that take into account uncertainties, thereby avoiding the limitations of fixed parameter settings and improving the adaptability of environmental control to dynamic conditions.

[0071] In one optional implementation, the step of constructing a parameter distribution function based on the mean vector and the variance vector, and sampling environmental control strategy parameters from the parameter distribution function includes:

[0072] The variance vector is subjected to a non-negative constraint transformation to obtain a constraint variance vector. The constraint variance vector is then compared element-by-element with a preset lower bound of variance. When an element in the constraint variance vector is less than the corresponding lower bound of variance, the element is replaced with the lower bound of variance value to obtain a corrected variance vector.

[0073] Obtain the physical boundaries of the control parameters of each environmental control device; construct a truncated normal distribution function as the parameter distribution function based on the mean vector, the corrected variance vector, and the physical boundaries of the control parameters.

[0074] The truncated normal distribution function is sampled by reparameterization transformation, which converts the sampling operation into a combination operation of deterministic function and independent noise variable. Candidate control parameter samples are generated from the truncated normal distribution function, and the environmental control strategy parameters are obtained by boundary truncation of the candidate control parameter samples.

[0075] For example, the nonnegative constraint transformation of the variance vector is achieved by applying an exponential function to each element of the vector. The exponential function maps any real number to a positive real number, ensuring that all elements in the constraint variance vector are strictly greater than zero. The variance vector comes from the output of the policy network, and its elements contain negative or near-zero values. After the exponential transformation, the i-th element of the constraint variance vector is equal to the i-th element of the variance vector raised to the power of the natural constant, which is approximately 2.718. After the constraint variance vector is generated, it is compared element-wise with a preset lower bound of variance. The lower bound of variance is set to a vector containing the same number of elements as the number of environmental control devices, with each element having a default value of 0.01. This lower bound prevents the variance from being too small, causing the parameter distribution to degenerate into a deterministic distribution. The comparison operation iterates through each element of the constraint variance vector. When the value of the i-th element is less than the i-th element of the lower bound of variance vector, that element in the constraint variance vector is replaced with the lower bound of variance value; otherwise, the original value remains unchanged. After completing the element-by-element comparison and replacement, the corrected variance vector is obtained. All elements of the corrected variance vector are not less than the corresponding lower bound of variance, while the original values ​​of elements in the constrained variance vector that are greater than the lower bound are retained.

[0076] The physical boundaries of the control parameters for each environmental control device are obtained from the device configuration database. Each physical boundary consists of two values: a lower bound and an upper bound, representing the minimum and maximum acceptable control parameter values ​​for the device, respectively. For heating equipment, the lower bound is 0 kW and the upper bound is 100 kW; for ventilation equipment, the lower bound is 0 m / s and the upper bound is 10 m / s; and for dehumidification equipment, the lower bound is 0 L / h and the upper bound is 50 L / h. The physical boundary data is organized into two vectors: the lower bound vector contains the lower bound values ​​for each device, and the upper bound vector contains the upper bound values ​​for each device. The length of both vectors is equal to the number of environmental control devices, which is 3.

[0077] The truncated normal distribution function is constructed based on the mean vector, the corrected variance vector, and the physical boundaries of the control parameters. The distribution function truncates and renormalizes the standard normal distribution within a specified interval. For the i-th environmental control device, its corresponding truncated normal distribution function is defined as a normal distribution within the interval from the i-th element of the lower bound vector to the i-th element of the upper bound vector. The mean parameter of the normal distribution is taken from the i-th element of the mean vector, and the variance parameter is taken from the i-th element of the corrected variance vector. The probability density function of the truncated normal distribution is expressed as:

[0078] Where x is the value of the control parameter, representing the specific value sampled from the truncated normal distribution; μ is the mean parameter, taken from the corresponding element of the mean vector, representing the center position of the untruncated normal distribution; σ is the standard deviation parameter, equal to the square root of the corresponding element of the corrected variance vector, representing the dispersion of the distribution; a is the lower bound parameter, taken from the corresponding element of the lower bound vector, representing the minimum value of the physical boundary of the control parameter; b is the upper bound parameter, taken from the corresponding element of the upper bound vector, representing the maximum value of the physical boundary of the control parameter. Let be the probability density function of the standard normal distribution, defined as z = (x - μ) / σ; Let be the cumulative distribution function of the standard normal distribution, representing the integral probability of the standard normal distribution from negative infinity to a certain value. The normalization factor in the denominator ensures that the truncated distribution integrals to 1 over the interval [a, b].

[0079] The reparameterization transformation converts the sampling operation of the truncated normal distribution function. The sampling process is divided into two stages: generating standard noise and deterministic transformation. When generating standard noise, random numbers u are drawn from a standard uniform distribution, with values ​​ranging from 0 to 1. The deterministic transformation maps the random numbers u to sampled values ​​of the truncated normal distribution. This mapping is achieved through the inverse function of the cumulative distribution function. For the i-th environmental control device, the mapping calculation involves first calculating the cumulative probability of the standard normal distribution at the standardized value corresponding to the lower bound. The standardized value equals the lower bound minus the mean, divided by the square root of the corrected variance; this cumulative probability is denoted as the lower bound cumulative probability. Similarly, the upper bound cumulative probability is calculated. Subtracting the lower bound cumulative probability from the upper bound cumulative probability yields the effective probability interval width. The random number u is multiplied by the effective probability interval width and then added to the lower bound cumulative probability to obtain the target cumulative probability value. The target cumulative probability value is converted to a standardized sample value through the inverse function of the standard normal cumulative distribution function. The standardized sample value is multiplied by the square root of the corrected variance and then added to the mean to obtain the candidate control parameter sample.

[0080] Candidate control parameter samples may slightly exceed physical boundaries due to numerical calculation errors or approximate algorithms. Boundary truncation applies mandatory constraints to the candidate samples. For the i-th element of a candidate control parameter sample, if its value is less than the i-th element of the lower bound vector, that element's value is set as the lower bound value; if its value is greater than the i-th element of the upper bound vector, that element's value is set as the upper bound value; if its value is between the lower and upper bounds, it remains unchanged. After truncation, the resulting environmental control strategy parameter vector has a length equal to the number of environmental control devices, and each element strictly satisfies the physical boundary constraints of its corresponding device.

[0081] The implementation of the reparameterization transformation relies on the numerical calculation of the cumulative distribution function of the standard normal distribution and its inverse function. The cumulative distribution function is calculated using a numerical approximation algorithm of the error function, with an approximation precision set to 6 decimal places. The inverse function is implemented through iterative solution or lookup table interpolation. The iterative solution uses Newton's method, with the initial value set as the target cumulative probability value minus 0.5 multiplied by 6. The iteration terminates when the absolute value of the difference between two consecutive iterations is less than 0.0001 or the number of iterations exceeds 20. The lookup table interpolation method pre-calculates the standardized sample values ​​corresponding to the cumulative probability values ​​from 0.0001 to 0.9999 with a step size of 0.0001. During the lookup, the target cumulative probability value is subjected to nearest neighbor search or linear interpolation, and the interpolation weight is determined according to the distance ratio between the target value and the adjacent table entries.

[0082] This invention combines truncating the normal distribution function with reparameterization transformation to ensure that the sampled values ​​conform to physical constraints while maintaining the differentiability of the sampling process, supporting gradient backpropagation optimization of policy network parameters, and improving the adaptability of the control policy to device boundary conditions.

[0083] In one optional implementation, the step of adjusting the environmental control strategy parameters using the dynamic importance weights as control modulation coefficients to generate a comprehensive control instruction set includes:

[0084] Obtain the time series of dynamic importance weights for each monitoring data source, and calculate the fluctuation variance and decay rate for each data source; when the fluctuation variance of a data source is greater than a preset fluctuation threshold, the data source is marked as environmental source uncertainty; when the fluctuation variance of a data source is not greater than a preset fluctuation threshold and the decay rate is greater than a preset decay threshold, the data source is marked as equipment source uncertainty.

[0085] Establish a dependency matrix between monitoring data sources and environmental control equipment, and obtain the variance vector corresponding to each environmental control equipment;

[0086] For each control device, the weighted sum of the uncertain data sources of the environment sources it depends on is calculated according to the dependency matrix to obtain the data transmission uncertainty component; the data transmission uncertainty component is obtained by subtracting the device variance vector from the device variance vector.

[0087] For data sources marked as device-source uncertainty, their weights are used as suppression coefficients to reduce the control parameters of control devices that depend on these data sources; for data sources marked as environmental-source uncertainty, their inverse weights are used as compensation coefficients to amplify the control parameters; for control devices whose inherent uncertainty components exceed the threshold, their control parameters are reverted to historical safe parameters; and the control parameters of each control device after differential modulation are encapsulated into a comprehensive control instruction set.

[0088] For example, the dynamic importance weight time series of each monitoring data source is obtained from the output history of the attention mechanism module. The time series length is set to the most recent 30 consecutive sampling periods, corresponding to a 30-minute weight change trajectory. The weight time series is organized as a two-dimensional array, with row indices corresponding to the monitoring data source type and column indices corresponding to timestamps. The array elements are the dynamic importance weight values ​​at the corresponding time. The monitoring data sources include five types: temperature sensors, humidity sensors, surrounding rock stress sensors, frozen soil depth detectors, and wind speed sensors. The time series array has 5 rows. The variance of each data source is calculated by performing variance statistics on its weight time series. The variance value is equal to the sum of the squares of the differences between each element of the time series and the series mean, divided by the difference between the series length and 1. The calculation result is rounded to 4 decimal places. The rate of decline is calculated by performing linear regression fitting on the weight time series to obtain the slope coefficient. A negative slope coefficient indicates that the weight is declining. The absolute value of the slope coefficient is the rate of decline. The calculation result is also rounded to 4 decimal places.

[0089] The preset fluctuation threshold is set to 0.05, which represents the upper limit of the standard deviation of the weight time series fluctuation. Exceeding this value indicates that the data source is affected by external environmental factors, causing drastic fluctuations in the weight. The preset decay threshold is set to 0.003, which represents the upper limit of the average rate of decrease of the weight time series per unit time. Exceeding this value indicates that the data source quality or equipment performance is continuously deteriorating. A dual-discrimination logic is executed for each monitored data source. When the data source's fluctuation variance is greater than 0.05, the data source is marked as environmental source uncertainty, and the marking information is stored as a Boolean flag and associated with the data source index. When the data source's fluctuation variance is not greater than 0.05 and the rate of decrease is greater than 0.003, the data source is marked as equipment source uncertainty, and the marking operation also sets a Boolean flag. Data sources with fluctuation variance not greater than 0.05 and a rate of decrease not greater than 0.003 are not marked, indicating that the data source is in a normal stable state. The marking results are organized as a flag vector of length 5. The i-th element of the vector corresponds to the marking state of the i-th data source, and the values ​​include three enumerated values: environmental source uncertainty, equipment source uncertainty, and normal state.

[0090] The dependency matrix between monitoring data sources and environmental control equipment is established through expert knowledge rules or historical data mining. The matrix has a 5x3 dimension, with row indices corresponding to 5 types of monitoring data sources and column indices corresponding to 3 types of environmental control equipment: heating equipment, ventilation equipment, and dehumidification equipment. Matrix elements are either 0 or 1. A value of 1 indicates a dependency between the monitoring data source in the corresponding row and the control equipment in the corresponding column; a value of 0 indicates no dependency. For example, temperature sensor data is dependent on heating equipment, so the element in the first row and first column of the matrix is ​​1; humidity sensor data is dependent on dehumidification equipment, so the element in the second row and third column of the matrix is ​​1; and surrounding rock stress sensor data is dependent on ventilation equipment, so the element in the third row and second column of the matrix is ​​1. The dependency matrix is ​​stored in a sparse matrix format to save memory. After being loaded into memory, the matrix participates in subsequent weighted calculations. The variance vector corresponding to each environmental control equipment is obtained from the output of the policy network. The variance vector has a length of 3, and each element corresponds to the uncertainty value of the target control parameter of each control equipment. The value has undergone the non-negative constraint transformation and lower bound correction processing described above.

[0091] When calculating the weighted sum of the environmental uncertainty data sources that each control device depends on, the process iterates through all rows of the corresponding column in the dependency matrix, identifying the set of row indices where the matrix element value is 1 and the corresponding data source is marked as environmental uncertainty. For each data source in the set, the weight value of the latest time step of its dynamic importance weight time series is extracted, and all weight values ​​are summed to obtain the data transmission uncertainty component of that control device. For example, if a heating device depends on a temperature sensor, and the temperature sensor is marked as environmental uncertainty with a latest weight value of 0.72, then the data transmission uncertainty component of the heating device is 0.72. If a control device depends on multiple environmental uncertainty data sources, the summation operation is performed on the weight values ​​of all dependent data sources. The policy-inherent uncertainty component is calculated by subtracting the data transmission uncertainty component from the corresponding element of the device variance vector. If the subtraction result is negative, it is truncated to 0 to ensure that the policy-inherent uncertainty component is a non-negative real number. The policy-inherent uncertainty component represents the control parameter uncertainty generated by the policy network itself after deducting the influence of environmental uncertainty.

[0092] For data sources marked as device-source uncertainty, their latest weight values ​​are extracted as suppression coefficients, with values ​​ranging from 0 to 1. The set of control devices dependent on this data source is identified using a dependency matrix. A reduction modulation operation is performed on the control parameters of each control device in the set; the modulated control parameters are equal to the original control parameters multiplied by the suppression coefficient. For example, if a humidity sensor is marked as a device-source uncertainty with a weight of 0.65, and a dehumidifier depends on the humidity sensor, the original control parameter of the dehumidifier is 32 liters per hour; the reduced control parameter is 32 × 0.65 = 20.8 liters per hour. For data sources marked as environmental-source uncertainty, their latest weight values ​​are extracted, and their reciprocals are calculated as compensation coefficients. These compensation coefficients are greater than 1 to achieve a control parameter amplification effect. An amplification modulation operation is performed on the control devices dependent on this data source; the modulated control parameters are equal to the original control parameters multiplied by the compensation coefficient. For example, if a temperature sensor is labeled as an environmental source of uncertainty with a weight of 0.72, the compensation coefficient is 1 ÷ 0.72 ≈ 1.39. The original control parameter of the heating equipment is 45.93 kW, and the amplified control parameter is 45.93 × 1.39 ≈ 63.84 kW. When calculating the compensation coefficient, a lower limit protection is applied to the weight value. When the weight value is less than 0.1, it is replaced with 0.1 before calculating the reciprocal, preventing the compensation coefficient from becoming too large due to an excessively small weight value, which could cause abnormal control parameters.

[0093] The threshold judgment for policy-inherent uncertainty components is achieved by comparing the policy-inherent uncertainty components of each control device with a preset inherent uncertainty threshold. The threshold is set to 0.15, representing the upper limit of the policy network's own uncertainty tolerance. When the policy-inherent uncertainty component of a control device exceeds 0.15, a mechanism for backtracking control parameters to historical safety parameters is triggered. Historical safety parameters are retrieved from the parameter history database. The query criteria are historical records where the freeze-thaw condition category identifier is consistent with the current condition and the construction process progress is close to the current progress. "Close to the current progress" is defined as the absolute value of the progress difference being less than 5 percentage points. The query results are sorted in descending order of timestamp, and the control parameter of the most recent record is selected as the historical safety parameter. The backtracking operation replaces the current control parameter with the historical safety parameter. After replacement, the control parameter needs to undergo physical boundary checks again. If the parameter value exceeds the device's physical boundary, boundary truncation is performed. The backtracking mechanism only applies to devices whose policy-inherent uncertainty exceeds the threshold; other devices retain their modulated control parameters.

[0094] The integrated control instruction set encapsulation organizes the control parameters of each control device, after differential modulation, into structured data packets. Each data packet contains an instruction identifier, a timestamp, a device type array, a control parameter array, and a checksum field. The device type array is 3 bytes long, with each element representing the type identifier for heating, ventilation, and dehumidification equipment, respectively. The control parameter array is also 3 bytes long, with each element corresponding to the modulated control parameter value for each device. The checksum is calculated by summing all elements of the control parameter array and taking the modulo 256; it is used for integrity verification during communication transmission. The data packets are serialized into a byte stream format with a total length of 64 bytes, using little-endian byte order encoding. The data packet header contains a 4-byte magic number identifier with a fixed value of 0x5A5A5A5A, indicating a valid instruction packet.

[0095] This invention implements differentiated modulation strategies by distinguishing between environmental and equipment uncertainties, avoiding overly conservative or aggressive control caused by uniform adjustment. At the same time, it introduces an inherent uncertainty decomposition mechanism to improve the adaptability and robustness to complex multi-source uncertainties.

[0096] In one optional implementation, the steps of executing the integrated control instruction set and collecting feedback data, performing similarity matching between the feedback data and historical operating condition evolution trajectories, extracting trend features as feedforward signals, and adjusting the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent include:

[0097] The integrated control instruction set is sent to each environmental control device for execution, and environmental monitoring data is continuously collected and combined into a feedback data sequence.

[0098] The feedback data sequence is dynamically time-warped and matched with the historical working condition evolution trajectory in the historical database. Historical trajectories with similarity scores exceeding a preset score threshold are selected as a set of similar working conditions. The subsequent evolution path of each historical trajectory in the set of similar working conditions after the matching time is extracted. The subsequent evolution path is statistically analyzed to obtain the probability distribution of the environmental state evolution direction, which is encoded as a trend feature vector.

[0099] Obtain the parameter matrix used to calculate attention weights in the attention mechanism; use the trend feature vector as a feedforward signal, and adjust the weight parameters of the monitoring data source corresponding to the evolution direction whose probability value exceeds a preset probability threshold in the parameter matrix according to the probability value of each environmental state evolution direction; update the adjusted parameter matrix to the attention mechanism.

[0100] For example, the integrated control command set is sent to each environmental control device for execution via the fieldbus protocol. Upon receiving the control parameters, the heating device adjusts its heating power output; the ventilation device adjusts its fan speed to control airflow; and the dehumidification device adjusts its compressor operating frequency to control dehumidification capacity. The command issuance cycle is set to 60 seconds. Within each cycle, the control master station sends a complete integrated control command set data packet to the bus once, with a data packet transmission delay not exceeding 200 milliseconds. After executing the control commands, the devices enter a stable operating state. The stabilization period is set to two-thirds of the control cycle, i.e., 40 seconds. During this period, the control devices maintain constant parameter output to avoid system oscillations caused by frequent adjustments.

[0101] Environmental monitoring data is acquired through a distributed sensor network, which includes five types of devices: temperature sensors, humidity sensors, surrounding rock stress sensors, permafrost depth detectors, and wind speed sensors. The sampling interval for all sensors is uniformly set to 10 seconds. The temperature sensor uses a digital thermistor solution, with a measurement range of -40℃ to +60℃ and a resolution of 0.1℃. The sensor connects to the data acquisition module via a four-wire interface. The humidity sensor uses a capacitive humidity sensing element, measuring relative humidity from 0% to 100%, with a resolution of 0.5%, and outputting a 4 to 20 mA current loop signal. The surrounding rock stress sensor uses a vibrating wire stress gauge, measuring from 0 to 5 MPa, with an accuracy class of 0.5. The sensor is installed at key stress concentration points on the tunnel arch and sidewalls. The permafrost depth detector uses the ground-penetrating radar reflected wave time-delay ranging principle, detecting depths from 0 to 6 meters with a depth resolution of 0.05 meters. The device transmits the detection results via a serial communication interface. The wind speed sensor uses a hot-wire anemometer with a measurement range of 0 to 20 meters per second and a resolution of 0.01 meters per second. The sensor is installed at the center line of the ventilation duct.

[0102] The feedback data sequence is organized as a time series matrix. The matrix row index corresponds to the sampling time, the column index corresponds to the five types of monitoring data sources, and the matrix elements are the monitoring values ​​of the corresponding data source at the corresponding time. The continuous acquisition duration is set to one control cycle, i.e., 60 seconds, with a sampling interval of 10 seconds. The feedback data sequence contains 6 consecutive sampling points. The feedback data sequence is cached in memory as a 6-row, 5-column floating-point array, with the array occupying a total memory footprint of 240 bytes. After the array is filled, a similarity matching process with the historical working condition evolution trajectory is triggered. The historical working condition evolution trajectory is stored in a time series database. The database adopts a sharded index structure, establishing secondary indexes according to the construction process progress and freeze-thaw condition category. Each index partition stores the historical monitoring data time series for the corresponding working condition.

[0103] The Dynamic Time Warping (VTW) matching algorithm performs flexible alignment between the feedback data sequence and historical trajectories. The algorithm calculates a cumulative distance matrix between the feedback data sequence and each historical trajectory. The number of rows in the matrix is ​​the sum of the length of the feedback sequence and the value 1 (7), and the number of columns is the sum of the length of the historical trajectory and the value 1. The cumulative distance is calculated recursively. The cumulative distance at the current position is the sum of the Euclidean distance at the current position and the minimum of three candidate cumulative distances. These three candidate cumulative distances are from the left-hand adjacent position, the upper-hand adjacent position, and the upper-left diagonal adjacent position, respectively. The recursive calculation starts from the upper-left corner of the matrix and ends at the lower-right corner. The cumulative distance at the ending position is the VTW distance between the two sequences. The similarity score is calculated by taking the negative of the VTW distance, applying an exponential transformation, and normalizing it to the 0-1 range. The smaller the distance value, the closer the similarity score is to 1. The preset score threshold is set to 0.75, meaning that the historical trajectory must match the current feedback data with a degree of at least 75% to be included in the similarity case set. The historical database stores approximately 5,000 historical trajectories. The time cost of traversing all trajectories to perform similarity calculations is approximately 8 seconds. To reduce computational latency, an index filtering strategy is adopted, which only performs matching on historical trajectories within the current work condition category and progress range. After filtering, the number of candidate trajectories is reduced to approximately 200, and the matching time cost is reduced to 0.3 seconds.

[0104] After extracting the set of similar operating conditions, the subsequent evolution path of each historical trajectory in the set after the matching time is obtained. The matching time is determined by path backtracking using a dynamic time warping algorithm. The backtracking path records the alignment relationship between each sampling point of the feedback data sequence and each sampling point of the historical trajectory. The moment when the last sampling point of the feedback sequence aligns with the historical trajectory is the matching time. The subsequent evolution path is defined as a monitoring data sequence of 20 consecutive sampling points after the matching time, with a time span of 200 seconds. The subsequent evolution path is also organized into a 20-row, 5-column matrix format, where the matrix elements are the monitoring values ​​of the corresponding data source at the corresponding time. The direction of environmental state evolution is determined by calculating the sign of the linear regression slope of each column of the subsequent evolution path. A positive slope indicates that the corresponding data source value is evolving in an upward direction, a negative slope indicates a downward direction, and a slope with an absolute value less than 0.01 is considered a stable evolution direction. The five types of monitoring data sources correspond to five evolution direction dimensions, each dimension containing three values: upward, downward, and stable. The combination of the five-dimensional evolution direction is encoded as an evolution state label.

[0105] The probability distribution of environmental state evolution direction is obtained by performing frequency statistics on the evolution state labels of all historical trajectories in the similar working condition set. Assuming the similar working condition set contains 12 historical trajectories, statistics show that in the temperature dimension, 7 trajectories are increasing, 3 are stable, and 2 are decreasing. The probability of temperature increase is calculated as 7 ÷ 12 ≈ 0.583, the probability of temperature stability is calculated as 3 ÷ 12 = 0.25, and the probability of temperature decrease is calculated as 2 ÷ 12 ≈ 0.167. In the humidity dimension, 9 trajectories are decreasing, 2 are stable, and 1 is increasing. The probability of humidity decrease is calculated as 9 ÷ 1 = 0.75. In the surrounding rock stress dimension, 8 trajectories are increasing and 4 are stable. The probability of stress increase is calculated as 8 ÷ 12 ≈ 0.667. In the frozen soil depth dimension, 10 trajectories are decreasing and 2 are stable. The probability of depth decrease is calculated as 10 ÷ 12 ≈ 0.833. In the wind speed dimension, 6 trajectories show an upward trend, 5 show a stable trend, and 1 shows a downward trend. The probability of an upward wind speed is calculated as 6 ÷ 12 = 0.5. The probability distribution calculation results are rounded to 3 decimal places and organized into a 5-row, 3-column probability matrix. The row indices correspond to the five types of monitoring data sources, and the column indices correspond to the three evolution directions: upward, stable, and downward.

[0106] The trend feature vector encoding is implemented by flattening the probability matrix into a one-dimensional vector with a length of 15. The elements are in the following order: probability of temperature increase, probability of temperature stabilization, probability of temperature decrease, probability of humidity increase, probability of humidity stabilization, probability of humidity decrease, probability of surrounding rock stress increase, probability of surrounding rock stress stabilization, probability of surrounding rock stress decrease, probability of permafrost depth increase, probability of permafrost depth stabilization, probability of permafrost depth decrease, probability of wind speed increase, probability of wind speed stabilization, and probability of wind speed decrease. The vector elements are 32-bit floating-point numbers, and the vector occupies a total of 60 bytes of memory. The trend feature vector is passed as a feedforward signal to the attention mechanism module. The feedforward signal interface is defined as a function call, with the trend feature vector pointer as the function input parameter and the adjustment operation execution status code as the function return value.

[0107] The parameter matrix used to calculate attention weights in the attention mechanism is loaded from the model parameter storage area. The parameter matrix has a dimension of 5 rows and 128 columns. The row index corresponds to the five types of monitoring data sources, and the column index corresponds to the query vector dimension of the attention mechanism. The initial values ​​of the parameter matrix elements are obtained through pre-training, and the parameter values ​​follow a normal distribution with a mean of 0 and a standard deviation of 0.02. The probability values ​​of the evolution direction of each environmental state in the trend feature vector are compared with a preset probability threshold. The probability threshold is set to 0.6, indicating that the certainty of the evolution direction must reach more than 60% before parameter adjustment is triggered. Adjustment is not triggered if the probability of temperature increase (0.583) does not exceed the threshold; the probability of humidity decrease (0.75) exceeds the threshold and triggers an increase in the humidity sensor weight parameters; the probability of surrounding rock stress increase (0.667) exceeds the threshold and triggers an increase in the surrounding rock stress sensor weight parameters; the probability of permafrost depth decrease (0.833) exceeds the threshold and triggers an increase in the permafrost depth detector weight parameters; and the probability of wind speed increase (0.5) does not trigger an adjustment.

[0108] Weight parameter enhancement is achieved by applying a scaling factor to all elements in the corresponding row of the parameter matrix. The scaling factor is calculated as the sum of the value 1 and the difference, where the difference is half the result of subtracting the threshold from the probability value. For the humidity sensor, the scaling factor is calculated as 1 + ((0.75 - 0.6) ÷ 2) = 1.075. All 128 elements in the second row of the parameter matrix corresponding to the humidity sensor are scaled by 1.075 to complete the enhancement adjustment. For the surrounding rock stress sensor, the scaling factor is calculated as 1 + ((0.667 - 0.6) ÷ 2) ≈ 1.034. Elements in the third row of the parameter matrix are scaled by 1.034. For the frozen soil depth detector, the scaling factor is calculated as 1 + ((0.833 - 0.6) ÷ 2) ≈ 1.117. Elements in the fourth row of the parameter matrix are scaled by 1.117. After adjustment, the element values ​​in the parameter matrix must undergo a range check. Elements with absolute values ​​exceeding 10 are truncated to ±10 to prevent overflow. The adjusted parameter matrix is ​​updated to the model parameter storage area of ​​the attention mechanism via atomic write operations. Locking is used during the update operation to ensure data consistency during concurrent access by multiple threads, with the lock holding time not exceeding 5 milliseconds. After the parameter matrix update is complete, the attention mechanism's weight recalculation process is triggered. This recalculation takes effect during the state encoding phase of the next control cycle, achieving feedforward adjustment based on historical operating condition evolution trends. The similar operating condition set, after completing the attention mechanism parameter adjustment, has its data cached and retained until the end of the current control cycle for use in subsequent reward function association rule mining processes. The two adjustment processes are executed sequentially without interference.

[0109] This invention extracts the probability distribution of environmental state evolution trends by matching historical operating conditions similarity, and feeds the trend information forward to the adjustment of attention mechanism parameters, enabling the system to predict the direction of environmental changes before control execution, thereby improving the control strategy's forward adaptability to operating condition evolution and decision stability.

[0110] In an optional implementation, the method further includes:

[0111] The association rules between the control decision sequence and the final control effect evaluation index are extracted from the set of similar operating conditions, and the set of decision features that lead to positive control effect and the set of decision features that lead to negative control effect are identified.

[0112] Obtain the reward function of the reinforcement learning agent, wherein the reward function includes reward weight parameters and penalty weight parameters corresponding to the decision features;

[0113] The corresponding reward weight parameter is increased based on the decision feature set of the positive control effect, and the corresponding penalty weight parameter is increased based on the decision feature set of the negative control effect, generating adjusted reward function parameters; the adjusted reward function parameters are then updated to the reinforcement learning agent.

[0114] For example, extracting the association rules between control decision sequences and final control effect evaluation indicators from a set of similar operating conditions requires first pairing and organizing the control decision sequences and effect evaluation indicators for each historical trajectory in the set of similar operating conditions. This process starts after the attention mechanism parameters are adjusted, reusing the cached similar operating condition set data to avoid repeatedly performing similarity matching calculations. Each historical trajectory record in the set of similar operating conditions contains three types of core data: a snapshot of environmental monitoring data at the start of the operating condition, the control decision sequence executed within the control cycle, and the final control effect evaluation indicator after the control execution is completed. The environmental monitoring data snapshot is represented by a five-dimensional vector, with the dimensions corresponding to the values ​​of five types of monitoring data sources: temperature, humidity, surrounding rock stress, frost depth, and wind speed. The data type is a 32-bit floating-point number, and the value range is determined according to the sensor measurement range. The control decision sequence is represented by discrete action codes. The action space is defined as the Cartesian product of three dimensions: heating power adjustment, ventilation speed adjustment, and dehumidification intensity adjustment. Each dimension includes three values: decrease, maintain, and increase. The three-dimensional action space contains a total of 27 discrete action combinations, and the action codes use integer indices from 0 to 26. The control effect evaluation index is calculated from the environmental monitoring data collected continuously after the control is implemented. The evaluation index includes five sub-indicators: temperature deviation, humidity deviation, stress fluctuation amplitude, permafrost depth change rate, and energy consumption efficiency. Each sub-indicator is mapped to the interval of 0 to 1 through a preset normalization function. The comprehensive evaluation index is the weighted average of the five sub-indicators, with weight coefficients set to 0.3, 0.25, 0.2, 0.15, and 0.1, respectively. The closer the comprehensive evaluation index value is to 1, the better the control effect.

[0115] Association rule mining employs a frequent pattern mining algorithm. The algorithm takes as input the paired data of control decision sequences and comprehensive evaluation indicators for all historical trajectories in a set of similar operating conditions, and outputs a set of association rules satisfying minimum support and minimum confidence. The control decision sequences need to be converted to a transaction database format to adapt to the frequent pattern mining algorithm. Each historical trajectory corresponds to a transaction record in the transaction database, containing two fields: a set of decision feature items and an effect label. Decision feature items are generated by decoupling the three-dimensional actions of the control decision sequence into single-dimensional decision features. For example, decreasing, maintaining, and increasing heating power adjustment are encoded as decision feature items H-, H0, and H+, respectively; ventilation speed adjustment is encoded as V-, V0, and V+; and dehumidification intensity adjustment is encoded as D-, D0, and D+. The set of decision feature items corresponding to a single control decision contains three elements. If the control decision sequence of a historical trajectory contains decisions for 5 consecutive control cycles, then the transaction record corresponding to that trajectory contains 15 decision feature items. The effect label is generated by comparing the comprehensive evaluation index with a preset threshold. The threshold is set to 0.7. A comprehensive evaluation index greater than or equal to 0.7 is marked as a positive effect label P, and a comprehensive evaluation index less than 0.7 is marked as a negative effect label N.

[0116] The frequent pattern mining algorithm first counts the frequency of occurrence of decision feature sets in the transaction database and calculates the support of each decision feature set. Support is defined as the ratio of the number of transactions containing that feature set to the total number of transactions. A minimum support threshold of 0.2 is set, meaning that a decision feature set must appear in at least 20% of the historical trajectories to be considered a frequent pattern. Assuming the similar work condition set contains 50 historical trajectories, and decision feature sets H+ and V+ appear simultaneously in 32 trajectories, the support is calculated as 32 ÷ 50 = 0.64, exceeding the minimum support threshold and thus considered a frequent itemset. The algorithm generates frequent itemsets of increasing length through a layer-by-layer search strategy. Starting with single-element frequent itemsets, it generates candidate two-element itemsets through combination operations. After counting the support of the candidate itemsets, it selects the frequent two-element itemsets, repeating this process until no new frequent itemsets can be generated. After frequent itemsets are generated, the algorithm calculates association rules for each frequent itemset. The association rule format is an antecedent feature item set leading to a consequent effect label. The rule confidence is defined as the number of transactions containing both the antecedent feature item set and the consequent effect label divided by the number of transactions containing only the antecedent feature item set. The minimum confidence threshold is set to 0.75, meaning that the conditional probability of an association rule must reach at least 75% to be adopted.

[0117] Taking the decision feature set H+ and V+ as an example, statistics show that out of 32 trajectories containing this feature set, 28 are labeled with a positive effect tag P. The confidence score of the association rule H+, V+ leading to P is calculated to be 28 ÷ 32 ≈ 0.875, exceeding the minimum confidence threshold, and this rule is added to the association rule set. Similarly, statistics show that out of 18 trajectories containing the decision feature set H- and D-, 15 are labeled with a negative effect tag N. The confidence score of the association rule H-, D- leading to N is calculated to be 15 ÷ 18 ≈ 0.833, and this rule is added to the association rule set. After the association rule mining is completed, the rule set is divided into two categories according to the consequent effect tag: the antecedent feature set corresponding to the rule with a positive consequent effect tag P is identified as the decision feature set of positive control effect, and the antecedent feature set corresponding to the rule with a negative consequent effect tag N is identified as the decision feature set of negative control effect. Examples of positive decision feature sets include combinations of features such as H+, V+, H+, D0, V+, and D+, while examples of negative decision feature sets include combinations of features such as H-, D-, H-, V-, V-, and D-. The decision feature sets are stored using a hash table data structure, where the key is a string representation of the decision feature combination, and the value is the confidence score of the corresponding association rule. The hash table occupies approximately 2 kilobytes of memory.

[0118] Obtaining the reward function for a reinforcement learning agent requires loading reward function configuration data from the agent's model parameter storage area. This configuration data is organized in key-value pair format and includes two types of parameters: reward weight parameters and penalty weight parameters corresponding to the decision features. The reward function design employs a composite structure, comprising two levels: a basic reward component and a weight modulation component. The basic reward component calculates an immediate reward signal based on changes in environmental conditions, including five sub-components: reduction in temperature deviation, reduction in humidity deviation, suppression of stress fluctuations, reduction in energy consumption costs, and improvement in operational stability. The weight coefficients for each sub-component are 0.3, 0.25, 0.2, 0.15, and 0.1, respectively. The weight modulation component scales the basic reward component using the reward weight parameters and penalty weight parameters corresponding to the decision features, achieving reward signal modulation based on historical experience. The decision features define reward and penalty weights at the granularity of single-dimensional decision feature terms. Each of the nine single-dimensional decision feature terms (H-, H0, H+, V-, V0, V+, D-, D0, D+) corresponds to a pair of reward and penalty weight parameters. The reward weight parameter is defined as the positive reward coefficient obtained when the agent executes the corresponding decision feature and the environmental state improves, while the penalty weight parameter is defined as the negative penalty coefficient suffered when the agent executes the corresponding decision feature and the environmental state deteriorates. Both the reward and penalty weight parameters are initially set to 1.0, and the parameter data type is 32-bit floating-point numbers. The parameter values ​​are limited to between 0.1 and 5.0 to prevent the reward signal from being excessively sparse or excessively dense. The reward function is called after each action executed by the agent. The function input parameters include the code of the currently executed decision action, the environmental state vector before the action, and the environmental state vector after the action. The function first calculates the values ​​of each sub-component of the basic reward component and sums them with weights to obtain the basic reward value. Then, it decouples the three-dimensional decision action into three single-dimensional decision feature items. Based on the sign of the basic reward value, it selects the reward weight parameter or penalty weight parameter of the corresponding decision feature item as the modulation coefficient. Finally, the reward signal is equal to the product of the basic reward value and the modulation coefficient.

[0119] To increase the reward weight parameter of the decision feature set based on the positive control effect, it is necessary to traverse all combinations of decision feature items in the positive decision feature set and apply an increase adjustment to the reward weight parameter of the single-dimensional decision feature item contained in the combination. The increase adjustment adopts an incremental update strategy. The increment value is calculated as the product of the current reward weight parameter value and the adjustment coefficient. The adjustment coefficient is the difference between the confidence of the corresponding association rule and the baseline confidence, which is set to 0.75. Taking the decision feature item H+ as an example, if the confidence of the positive association rule containing H+ is 0.875, the adjustment coefficient is calculated as 0.875-0.75=0.125. The current reward weight parameter value of H+ is 1.0, the increment value is calculated as 1.0×0.125=0.125, and the adjusted reward weight parameter value is updated to 1.0+0.125=1.125. If the same decision feature item appears in multiple positive association rules, the increment value corresponding to all rules is added to the reward weight parameter of that feature item. After the parameters are updated, a value range check is performed. If the reward weight parameter value exceeds 5.0, it is truncated to 5.0; if it is below 0.1, it is truncated to 0.1.

[0120] Based on the decision feature set representing the negative control effect, the corresponding penalty weight parameter is increased. The adjustment process is similar to that of the reward weight parameter adjustment. It iterates through all combinations of decision feature items in the negative decision feature set, applying an increase adjustment to the penalty weight parameter of each single-dimensional decision feature item contained within the combination. The incremental value is also calculated by multiplying the current penalty weight parameter value by the adjustment coefficient, where the adjustment coefficient is the difference between the confidence level of the corresponding association rule and the baseline confidence level. Taking decision feature item H- as an example, if the confidence level of the negative association rule containing H- is 0.833, the adjustment coefficient is calculated as 0.833 - 0.75 = 0.083. The current penalty weight parameter value of H- is 1.0, and the incremental value is calculated as 1.0 × 0.083 = 0.083. The adjusted penalty weight parameter value is then updated to 1.0 + 0.083 = 1.083. Similarly, the incremental value is accumulated for recurring decision feature items, and a numerical range check and truncation from 0.1 to 5.0 are performed. After the reward weight parameters and penalty weight parameters are adjusted, the adjusted reward function parameter configuration data is generated. The data format is consistent with the original configuration data, still using key-value pair format, where the key is the decision feature item identifier and the value is the adjusted parameter value.

[0121] Updating the adjusted reward function parameters to the reinforcement learning agent requires parameter persistence and version management operations. Parameter persistence uses atomic write operations to write the adjusted parameter configuration data to the agent's model parameter storage area. The storage area is implemented using a shared memory segment, supporting concurrent access by multiple processes. Atomic write operations ensure data consistency by acquiring a mutex lock, with the lock holding time not exceeding 3 milliseconds. Parameter version management is achieved by maintaining parameter update timestamps and version numbers. The version number increments by 1 with each parameter update, and the timestamp is recorded as a millisecond-level timestamp of the current system time. At the beginning of the next control cycle, the agent checks the parameter version number. If the version number has changed, it reloads the reward function parameter configuration from the storage area, ensuring that the agent always uses the latest version of the reward function parameters to execute decisions. Parameter updates trigger the agent's policy network gradient recalculation process. Gradient recalculation is performed asynchronously in the background, without blocking the decision generation of the current control cycle. The recalculation takes effect in subsequent control cycles after completion. Assuming the set of similar working conditions contains 50 historical trajectories, association rule mining identifies 8 positive association rules and 5 negative association rules. The positive rules involve 6 single-dimensional decision feature items, and the negative rules involve 4 single-dimensional decision feature items. After parameter adjustment, the average increase in reward weight parameters among the 10 decision feature items is 0.15, and the average increase in penalty weight parameters is 0.12. In the subsequent 100 control cycles, the agent's frequency of selecting positive decision features increased from 42% to 61%, and the frequency of selecting negative decision features decreased from 18% to 9%. The average value of the comprehensive control effect evaluation index increased from 0.68 to 0.76.

[0122] This invention mines the correlation rules between decisions and results in historical working conditions, and injects successful experiences and lessons learned into the reward function of the reinforcement learning agent in the form of reward weights and penalty weights. This guides the agent to prioritize positive decision features with high confidence and avoid negative decision features, thereby improving the agent's decision quality and learning efficiency, and reducing exploration costs and control risks.

[0123] A second aspect of the present invention provides an electronic device, comprising:

[0124] processor;

[0125] Memory used to store processor-executable instructions;

[0126] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0127] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0128] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive control method for tunnel construction environment in cold regions, characterized in that, include: Obtain multi-dimensional monitoring data on the tunnel construction environment in cold regions; Multi-layer neural network feature extraction is performed on the multi-dimensional monitoring data. The contribution of each feature component to the freeze-thaw condition judgment is calculated based on the attention mechanism and then weighted and fused to output the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source. Based on the freeze-thaw condition category identifier and construction procedure record data, a combined state is constructed, and environmental control strategy parameters are generated through a policy network using a reinforcement learning agent. The dynamic importance weights are used as control modulation coefficients to adjust the environmental control strategy parameters, thereby generating a comprehensive control instruction set. The integrated control instruction set is executed and feedback data is collected; the feedback data is matched with the historical working condition evolution trajectory for similarity, and trend features are extracted as feedforward signals to adjust the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent.

2. The method according to claim 1, characterized in that, The steps of performing multi-layer neural network feature extraction on the multi-dimensional monitoring data, calculating the contribution of each feature component to the freeze-thaw condition discrimination based on the attention mechanism, performing weighted fusion, and outputting the freeze-thaw condition category identifier and the dynamic importance weight of each monitoring data source include: A primary spatial feature map is obtained by extracting spatial dimension features from the multidimensional monitoring data through a convolutional layer, and a spatiotemporal fusion feature vector is obtained by extracting temporal dimension features from the primary spatial feature map through a recurrent layer. Based on the spatiotemporal fusion feature vector, a query vector, a key vector, and a value vector are generated respectively. An attention score matrix is ​​obtained by performing a dot product operation between the query vector and the key vector. The attention score matrix is ​​normalized to obtain a normalized attention weight matrix. Each element in the normalized attention weight matrix represents the contribution of the corresponding feature component to the freeze-thaw condition discrimination. The normalized attention weight matrix and the value vector are weighted and summed to obtain a weighted fusion feature vector. The weighted fusion feature vector is input into the classifier to output the freeze-thaw condition category identifier. The average weight of the normalized attention weight matrix in the corresponding dimension of each monitoring data source is extracted as the dynamic importance weight of each monitoring data source.

3. The method according to claim 1, characterized in that, The steps of constructing a combined state based on the freeze-thaw condition category identifier and construction procedure record data, and generating environmental control policy parameters using a reinforcement learning agent through a policy network, include: The freeze-thaw condition category identifier is converted into a unique thermal coding vector; the construction process record data is subjected to temporal feature extraction to obtain a process progress vector and a process duration vector, which are then concatenated with the unique thermal coding vector to obtain a combined state vector; The combined state vector is input into the policy network of the reinforcement learning agent, and the policy network performs a nonlinear mapping on the combined state vector through a multi-layer fully connected transformation. The output of the strategy network includes a mean vector and a variance vector. The mean vector represents the expected value of the target control parameter of each environmental control device, and the variance vector represents the uncertainty of the target control parameter of each environmental control device. A parameter distribution function is constructed based on the mean vector and the variance vector, and environmental control strategy parameters are sampled from the parameter distribution function.

4. The method according to claim 3, characterized in that, The steps of constructing a parameter distribution function based on the mean vector and the variance vector, and sampling environmental control strategy parameters from the parameter distribution function include: The variance vector is subjected to a non-negative constraint transformation to obtain a constraint variance vector. The constraint variance vector is then compared element-by-element with a preset lower bound of variance. When an element in the constraint variance vector is less than the corresponding lower bound of variance, the element is replaced with the lower bound of variance value to obtain a corrected variance vector. Obtain the physical boundaries of the control parameters of each environmental control device; construct a truncated normal distribution function as the parameter distribution function based on the mean vector, the corrected variance vector, and the physical boundaries of the control parameters. The truncated normal distribution function is sampled by reparameterization transformation, which converts the sampling operation into a combination operation of deterministic function and independent noise variable. Candidate control parameter samples are generated from the truncated normal distribution function, and the environmental control strategy parameters are obtained by boundary truncation of the candidate control parameter samples.

5. The method according to claim 1, characterized in that, The step of adjusting the environmental control strategy parameters using the dynamic importance weights as control modulation coefficients to generate a comprehensive control instruction set includes: Obtain the time series of dynamic importance weights for each monitoring data source, and calculate the fluctuation variance and decay rate for each data source; when the fluctuation variance of a data source is greater than a preset fluctuation threshold, the data source is marked as environmental source uncertainty; when the fluctuation variance of a data source is not greater than a preset fluctuation threshold and the decay rate is greater than a preset decay threshold, the data source is marked as equipment source uncertainty. Establish a dependency matrix between monitoring data sources and environmental control equipment, and obtain the variance vector corresponding to each environmental control equipment; For each control device, the weighted sum of the uncertain data sources of the environment sources it depends on is calculated according to the dependency matrix to obtain the data transmission uncertainty component; the data transmission uncertainty component is obtained by subtracting the device variance vector from the device variance vector. For data sources marked as device-source uncertainty, their weights are used as suppression coefficients to reduce the control parameters of control devices that depend on these data sources; for data sources marked as environmental-source uncertainty, their inverse weights are used as compensation coefficients to amplify the control parameters; for control devices whose inherent uncertainty components exceed the threshold, their control parameters are reverted to historical safe parameters; and the control parameters of each control device after differential modulation are encapsulated into a comprehensive control instruction set.

6. The method according to claim 1, characterized in that, The steps of executing the integrated control instruction set and collecting feedback data, matching the feedback data with the historical operating condition evolution trajectory, extracting trend features as feedforward signals, and adjusting the contribution calculation rules of the attention mechanism and the reward function of the reinforcement learning agent include: The integrated control instruction set is sent to each environmental control device for execution, and environmental monitoring data is continuously collected and combined into a feedback data sequence. The feedback data sequence is dynamically time-warped and matched with the historical working condition evolution trajectory in the historical database. Historical trajectories with similarity scores exceeding a preset score threshold are selected as a set of similar working conditions. The subsequent evolution path of each historical trajectory in the set of similar working conditions after the matching time is extracted. The subsequent evolution path is statistically analyzed to obtain the probability distribution of the environmental state evolution direction, which is encoded as a trend feature vector. Obtain the parameter matrix used to calculate attention weights in the attention mechanism; use the trend feature vector as a feedforward signal, and adjust the weight parameters of the monitoring data source corresponding to the evolution direction whose probability value exceeds a preset probability threshold in the parameter matrix according to the probability value of each environmental state evolution direction; update the adjusted parameter matrix to the attention mechanism.

7. The method according to claim 6, characterized in that, The method further includes: The association rules between the control decision sequence and the final control effect evaluation index are extracted from the set of similar operating conditions, and the set of decision features that lead to positive control effect and the set of decision features that lead to negative control effect are identified. Obtain the reward function of the reinforcement learning agent, wherein the reward function includes reward weight parameters and penalty weight parameters corresponding to the decision features; The corresponding reward weight parameter is increased based on the decision feature set of the positive control effect, and the corresponding penalty weight parameter is increased based on the decision feature set of the negative control effect, generating adjusted reward function parameters; the adjusted reward function parameters are then updated to the reinforcement learning agent.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images