Pipe gallery assembly device autonomous feedback control method based on reinforcement learning and big data

Abstract

This invention provides an autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data, belonging to the field of feedback control technology for utility tunnel assembly devices. It includes preliminary technical groundwork through big data collection and analysis and environmental modeling, followed by training the optimal control strategy using the reinforcement learning PPO algorithm to achieve autonomous perception, real-time correction, and precise assembly during the assembly process. Finally, it uses full-data feedback from on-site measurements to continuously upgrade the model, forming an intelligent control closed loop for utility tunnel assembly devices that adapts to complex engineering conditions and can be continuously optimized. This invention, employing the aforementioned autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data, significantly reduces the intensity of manual operation and human error, improves assembly accuracy and construction efficiency, and also possesses strong adaptability to various working conditions and lifelong evolution capabilities.

Description

Technical Field

[0001] This invention relates to the field of feedback control technology for utility tunnel assembly devices, and in particular to an autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data. Background Technology

[0002] In the construction of urban underground utility tunnels, the precise assembly of tunnel components is a crucial step in ensuring project quality and construction efficiency. In traditional prefabricated construction of utility tunnels, the hoisting, alignment, and assembly of components rely heavily on manual operation. This not only leads to low construction efficiency but also makes it difficult to guarantee assembly accuracy. Especially when facing complex and variable construction conditions, such as different types of utility tunnel components, different foundation pit construction environments, and climatic conditions, the limitations of traditional methods become increasingly apparent.

[0003] Traditional assembly processes require extensive manual adjustments and corrections, which not only increases the labor intensity of workers but also easily leads to assembly errors due to human factors, affecting project quality. Due to the limitations of manual operation, traditional methods struggle to achieve high-precision assembly, especially in micron-level adjustments, failing to meet design standards and thus limiting the overall quality of prefabricated utility tunnel construction. Traditional control models are often static and unchanging, making it difficult to adapt to complex conditions such as different utility tunnel component types, foundation pit construction environments, and climatic conditions. This necessitates readjusting control strategies for different projects or different stages of the same project, increasing construction difficulty and cost.

[0004] To solve the above problems, a new autonomous feedback control method for utility tunnel assembly devices is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to provide an autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data, which can significantly reduce the intensity of manual operation and human error, improve assembly accuracy and construction efficiency, and also have strong adaptability to working conditions and lifelong evolution capability.

[0006] To achieve the above objectives, this invention provides an autonomous feedback control method for a utility tunnel assembly device based on reinforcement learning and big data, comprising the following steps: S1. Big Data Acquisition and Preprocessing: Collect historical assembly data, real-time sensing data, and component parameter data to form a raw dataset. Perform format verification and filtering, 3σ criterion outlier removal, linear interpolation to complete missing data, and Min-Max normalization on the raw dataset. Divide the preprocessed data into an offline training set and an online real-time set, and upload them to the cloud big data platform and transmit them to the edge controller, respectively. S2. Reinforcement Learning Environment Modeling: The problem of precise control of pipe gallery assembly is transformed into a Markov decision process. A 10-dimensional state space is constructed, which includes four core dimensions: positioning, attitude, force, and environment. A 6-dimensional motion space is constructed to match the mechanical motion capability of the multi-dimensional attitude adjustment execution module. A multi-objective weighted reward function is designed with the goals of prioritizing accuracy, taking into account efficiency, and ensuring safety. Historical standardized datasets are input into the environment model for simulation and interactive verification. After successful verification, the model is deployed to the edge controller in a lightweight manner. S3. Model training based on PPO algorithm: A PPO algorithm model framework including policy network and value network is built on a cloud big data platform. A deep neural network is used to construct the network structure. The standardized offline training set of S1 is input into the environment model of S2 for offline batch pre-training. The model is optimized by GAE advantage function, PPO cut-off objective function and value network loss function. The converged pre-trained model is lightweighted and deployed to the edge controller. Online fine-tuning is performed using online real-time assembly data. S4. Online autonomous decision-making and feedback control: The prefabricated components of the pipe gallery are hoisted to the initial position above the assembly reference point through the gantry crane base module. The full-domain perception and acquisition module collects data in real time and after the standardization preprocessing of S1, it is input to the PPO optimal control model of the edge controller. The model infers and generates 6-dimensional motion control commands and sends them to the multi-dimensional posture adjustment execution module. After the action is executed, new state data is collected and the overall assembly error is calculated. S5. Model Iteration and Big Data Update: The cloud-based big data platform receives and stores the assembly data uploaded by S4. After joint classification and labeling by humans and intelligence, a valid dataset is formed. When there are enough new valid assembly samples to support the data, the new dataset is merged with the original training set to incrementally train the PPO model. The iteratively optimized model is then lightweighted and distributed to all field edge controllers. At the same time, the new dataset is integrated into the cloud-based big data sample library.

[0007] Preferably, real-time sensing data is collected by the global sensing acquisition module, including positioning sensing data, attitude sensing data, force sensing data, and environmental sensing data. The sampling frequency of the global sensing acquisition module is set synchronously, and all sensors are networked through an industrial Ethernet network and the data is uniformly aggregated into the edge gateway.

[0008] Preferably, the 3σ criterion expression is: ; in, This refers to single-frame sampled data from the sensor. The average value of 100 frames of sliding window data; This represents the standard deviation of the corresponding data; The Min-Max normalization expression is: ; in, For the collected state characteristic data; The data is after normalization; This is the lower limit of the sensor's measurement range; This represents the upper limit of the sensor's measurement range.

[0009] Preferably, the 10-dimensional state-space expression is: ; in, This refers to the three-axis positioning error; For attitude tilt angle; For clamping tension; Ambient temperature; Wind speed; Dust concentration; The 6-dimensional action space expression is: ; in, This refers to the three-axis fine-tuning amount; For attitude fine-tuning; Set the clamping force value.

[0010] The preferred formula for the multi-objective weighted reward function is: ; in, This represents the current assembly error; This represents the maximum permissible assembly error. This represents the current force value. Standard assembly time; This refers to the actual time spent. The safe stress threshold; All are weighting coefficients; the reward interval R∈[-10,10].

[0011] Preferably, the deep neural network structure is a 10-dimensional state space, 3 hidden layers, and a 6-dimensional action space, and the network parameters are initialized using a random normal distribution.

[0012] Preferred, GAE dominance function The formula is: ; in, Discount factor; For smoothing coefficients; For time-series difference residuals; PPO shearing objective function The formula is: ; in, The expected value of the objective function for PPO shearing; The probability ratio between the old and new strategies; The shear coefficient; Value network loss function The expression is: ; in, Let $\mathbf{ ... For cumulative discount rewards; For the value network to the current state The estimated value.

[0013] Preferably, the formula for calculating the overall assembly error is: ; in, , , This represents the displacement along the X, Y, and Z axes; , For the attitude tilt angle changes along the α and β axes; The time delay of closed-loop control satisfies: ; in, For the total closed-loop delay constraint. For data acquisition and preprocessing latency; For model inference and decision-making delay; To control the delay between issuing commands and executing mechanical actions.

[0014] Preferably, the method further includes establishing a coordinate origin based on the standard assembly reference point of the utility tunnel foundation pit. The project uses a right-hand rectangular coordinate system, with the X-axis representing the longitudinal extension of the utility tunnel, the Y-axis representing the transverse span of the utility tunnel, and the Z-axis representing the vertical hoisting and lifting direction. The positioning, data acquisition, and control commands of all modules of the utility tunnel assembly device are based solely on this coordinate system.

[0015] Preferably, the utility tunnel assembly device includes a gantry crane base module, a multi-dimensional attitude adjustment execution module, a global perception and acquisition module, and a control and data processing module. The control and data processing module includes an edge controller and a cloud big data platform. The edge controller and the cloud big data platform achieve high-speed data transmission through 5G communication.

[0016] Therefore, the present invention adopts the above-mentioned autonomous feedback control method for pipe gallery assembly devices based on reinforcement learning and big data, and the technical effects are as follows: 1. Achieve intelligent and autonomous control of the entire assembly process: Based on the closed-loop architecture of "hardware full-domain perception - edge real-time decision-making - execution of precise actions", the system deeply integrates big data preprocessing, reinforcement learning decision-making and real-time feedback control. The entire process does not require manual posture adjustment and correction, and realizes the autonomous operation of the entire process of pipe gallery components from hoisting and alignment to assembly, which greatly reduces the intensity of manual operation and human error.

[0017] 2. Significantly improve assembly accuracy and construction efficiency: Through 100Hz full-domain synchronous perception, ≤100ms closed-loop control delay, and micron-level multi-axis servo attitude adjustment, the overall assembly error is controlled within 2mm, meeting the high-precision engineering requirements of prefabricated construction of utility tunnels; at the same time, the reinforcement learning strategy with "precision first, efficiency second" as the optimization goal can shorten the assembly time while ensuring accuracy, improving construction efficiency by more than 50% compared with the traditional manual assembly mode.

[0018] 3. Possesses strong adaptability to working conditions and lifelong evolution capability: Based on the continuous expansion of the big data sample library and the incremental iterative optimization of the PPO model, the method can adapt to complex working conditions such as different types of pipe gallery components, foundation pit construction environment, and climate conditions. It breaks the limitation of the traditional control model being "fixed and unchanging" and realizes the lifelong evolution of the control strategy, continuously improving accuracy and adaptability as construction samples accumulate. Attached Figure Description

[0019] Figure 1 This is a flowchart of the autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to the present invention. Figure 2 This is a schematic diagram of the overall architecture of the intelligent control closed-loop system for the utility tunnel assembly device of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0022] Example 1 This invention provides an autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data. Addressing the pain point of precise assembly in the prefabricated construction of urban underground integrated utility tunnels, it lays the groundwork through big data collection and analysis and environmental modeling. Then, relying on the reinforcement learning PPO algorithm to train the optimal control strategy, it achieves autonomous perception, real-time correction, and precise assembly during the assembly process. Finally, it continuously upgrades the model by feeding back full data from on-site measurements, forming an intelligent control closed loop for utility tunnel assembly devices that adapts to complex engineering conditions and can be continuously optimized. This solves the industry problems of excessive manual intervention, low precision, and poor adaptability in traditional assembly control, significantly improving the efficiency and quality of prefabricated utility tunnel construction.

[0023] The standard assembly reference point of the utility tunnel foundation pit is used as the origin of the coordinate system, and is set as follows: A dedicated right-handed rectangular coordinate system was established for the project, providing a unique and unified spatial reference for the spatial positioning, data acquisition, and control command issuance of all modules. This eliminates spatial positioning deviations in multi-module collaborative operations at the source, ensuring spatial consistency between data and commands. In this coordinate system, the X-axis represents the longitudinal extension direction of the pipe gallery, i.e., the length of the foundation pit; the Y-axis represents the transverse span direction of the pipe gallery, i.e., the width of the foundation pit; and the Z-axis represents the vertical hoisting and lifting direction, i.e., the direction perpendicular to the ground. The positioning, data acquisition, and control commands of all modules strictly adhere to this coordinate system as the sole reference, effectively eliminating spatial positioning deviations and ensuring the accuracy of each stage of the project.

[0024] The gantry crane base module is positioned at the bottom support level in the spatial layout. It is symmetrically arranged along the Y-axis on both sides of the foundation pit, providing full coverage of the assembly section in the X-axis direction, and a Z-axis height range of 0-8m. Specifically, the columns stand on the ground on both sides of the foundation pit (Y=±5m), and the main beam spans the foundation pit (Y=-5m~+5m), with a net height of 6m above the ground. The module consists of gantry columns, a longitudinal traveling mechanism, a transverse sliding platform, and a lifting winch. The traveling mechanism engages with high-precision guide rails pre-embedded in the foundation pit, enabling smooth movement along the X-axis; the sliding platform and main beam are connected by high-precision linear guide rails, achieving micron-level translation along the Y-axis; the winch is fixed to the bottom of the sliding platform, and the end effector is vertically suspended by high-strength steel cables, ensuring the vertical transmission of lifting force; power cables and control buses are laid along dedicated cable trays on the columns and orderly connected to the control cabinet, avoiding faults caused by messy cables and ensuring the stability of power supply and communication.

[0025] The multi-dimensional attitude adjustment execution module is located at the end of the spatial execution position, suspended 2-5m below the main beam along the Z-axis. During initial alignment, it is directly above the reference origin O (X=0, Y=0, Z=3m), covering an operating space of X±2m, Y±1m, and Z±1m, fully adaptable to the attitude adjustment requirements of the pipe gallery components. This module consists of a three-dimensional leveling platform, electric clamping claws, tilt adjustment cylinders, and a buffer damper. The leveling platform is connected to the end of the steel cable via a ball joint, accommodating small attitude adjustments during hoisting and avoiding interference from the cable tension. The clamping claws are rigidly fixed to the platform bottom surface with high-strength bolts, ensuring clamping stability. The tilt adjustment cylinder is embedded inside the platform and linked with the platform legs to achieve precise adjustment of the α and β dual-axis tilt angles. The servo driver is directly connected to the edge controller, achieving closed-loop precise control of three-axis displacement and attitude tilt angle with zero command response delay.

[0026] The global sensing and acquisition module is responsible for spatial three-dimensional monitoring tasks, and its three-dimensional location distribution is as follows: Positioning and sensing: LiDAR and total station are deployed at both ends of the main beam (Y=±5m, Z=7m) and the foundation pit reference point (Z=0.5m) to ensure positioning accuracy through multi-point positioning.

[0027] Attitude perception: A six-axis IMU and tilt sensor are mounted at the center of the attitude adjustment platform (Z=3m) to perceive the attitude changes of the platform in real time.

[0028] Force sensing: The tension sensor is installed at the cable joint, and the pressure sensor is embedded inside the clamping jaws to accurately measure the force during the hoisting process.

[0029] Environmental sensing: Temperature, humidity and wind speed sensors are installed on the top of the column (Z=7.5m) to monitor environmental parameters at the construction site.

[0030] All sensors are networked via an industrial Ethernet network, with data uniformly fed into the edge gateway. Simultaneously, the sampling frequency is synchronously set to 100Hz to ensure data real-time performance and synchronization, providing a reliable basis for subsequent data processing and analysis.

[0031] The control and data processing module is the core of the entire project's spatial management. The edge control cabinet is installed on the side of the gantry column (Z=1.2m), a location convenient for maintenance personnel to operate and maintain. The cloud-based big data platform is deployed in the on-site control room, away from the vibration interference of the foundation pit, ensuring the stability of data processing. The edge layer and the cloud achieve high-speed data transmission via 5G communication. Offline training data is uploaded to the cloud for storage and analysis to uncover patterns and value behind the data; online control commands are issued locally in real time, realizing a collaborative working mode of "cloud training and edge decision-making," improving the system's response speed and decision accuracy.

[0032] like Figures 1-2As shown, the specific autonomous feedback control method for the utility tunnel assembly device includes the following steps: S1. Big data collection and preprocessing, specifically covering historical assembly data, real-time sensing data, and component parameter data; Real-time sensing data (positioning, attitude, force, environment) is collected through the full-domain sensing acquisition module. Historical assembly data (assembly records, error data, and control strategies for different working conditions and different components) are retrieved through the cloud database. Parameter data of prefabricated pipe gallery components (dimensions, weight, assembly reference points) are imported through the engineering design system to form a multi-source, full-dimensional raw dataset. The data format is unified to the engineering standard format for easy subsequent processing.

[0033] The original dataset underwent format validation and data filtering to remove invalid data entries with format errors or missing values ​​exceeding 50%, retaining only valid original data. The 3σ criterion was used to detect and remove outliers from single-dimensional sensor data, eliminating abnormal data caused by sensor vibration, dust, and electromagnetic interference, thus ensuring data authenticity.

[0034] ; in, This refers to single-frame sampled data from the sensor. The average value of 100 frames of sliding window data; This represents the standard deviation of the corresponding data.

[0035] Data that satisfies this formula is considered abnormal and is directly removed from the dataset; missing data is filled using linear interpolation with a completion rate of ≤5% of the total data volume to avoid data distortion caused by over-filling.

[0036] Min-Max normalization is performed on all state feature data to eliminate the dimensional differences between different data, so that the data are uniformly mapped to the [0,1] interval, thereby improving the training efficiency and convergence speed of the reinforcement learning model.

[0037] ; in, The data is after normalization; This is the lower limit of the sensor's measurement range; This represents the upper limit of the sensor's measurement range.

[0038] The preprocessed datasets are categorized into "offline training set" and "online real-time set". The offline training set is uploaded to the cloud big data platform for model training, while the online real-time set is transmitted to the edge controller for online decision-making. At the same time, all preprocessed data is backed up to the cloud to achieve full lifecycle storage of data.

[0039] This step enables unified acquisition, quality verification, cleaning and noise reduction, and standardization of multi-source data for pipe gallery assembly, outputting a high-quality, highly consistent, dimensionless standardized dataset. This lays the data foundation for subsequent reinforcement learning environment modeling and model training, while providing clean real-time perception data for online real-time control, ensuring the accuracy of decision-making.

[0040] S2. Reinforcement learning environment modeling: transforming the engineering problem into an algorithm problem, clarifying the reinforcement learning decision rules and optimization objectives; based on the standardized dataset of S1, transforming the precise control problem of pipe gallery assembly into a reinforcement learning standard Markov decision process (MDP), clarifying the three core elements of state space, action space, and reward function, and formulating the decision rules and optimization objectives for model training.

[0041] The problem of precise control of utility tunnel assembly is abstracted as a Markov decision process of "agent-environment" interaction. The agent is the control and data processing module of the utility tunnel assembly device, and the environment is the operation scenario of the utility tunnel assembly. The agent makes action decisions by perceiving the state of the environment, the environment provides feedback to the agent's actions, and the agent adjusts its subsequent decisions based on the feedback.

[0042] Feature parameters that have a core impact on assembly control decisions are selected, and a high-dimensional state space is constructed, covering four core dimensions: positioning, attitude, force, and environment, to comprehensively reflect the real-time status of assembly operations.

[0043] ; in, This refers to the three-axis positioning error; For attitude tilt angle; For clamping tension; Ambient temperature; Wind speed; This refers to the dust concentration.

[0044] Three-axis positioning error refers to the three-axis deviation between the actual position of the component and the assembly reference position, which is monitored and calculated in real time by LiDAR / total station. Attitude tilt angle refers to the three-axis tilt angle deviation between the actual attitude of the component and the attitude of the assembly reference, which is monitored in real time by a six-axis IMU / tilt sensor. Clamping force refers to the actual clamping force of the clamping jaws on the component, which is monitored in real time by a pressure sensor. Ambient temperature refers to the ambient temperature at the assembly site, which is monitored in real time by a temperature and humidity sensor, affecting the accuracy of equipment operation and material properties. Wind speed refers to the ambient wind speed above the foundation pit, which is monitored in real time by a wind speed sensor, affecting the stability of hoisting and attitude adjustment. Dust concentration refers to the dust concentration at the assembly site, which is monitored in real time by a dust sensor, affecting the sensor's monitoring accuracy. The spatial dimension is a 10-dimensional state space, comprehensively reflecting the real-time status of the pipe gallery assembly and providing complete environmental information for intelligent agent decision-making.

[0045] By combining the mechanical motion capabilities of the multi-dimensional attitude adjustment execution module (three-axis fine adjustment, two-axis attitude adjustment, and clamping force adjustment), a motion space that precisely matches the motion of the execution end is constructed. The step size and value range of the motion parameters are strictly adapted to the mechanical performance of the equipment to avoid failures caused by exceeding the motion range of the equipment.

[0046] ; in, This refers to the three-axis fine-tuning amount, which is the micro-displacement adjustment amount of the multi-dimensional attitude adjustment execution module in the X, Y, and Z axes. The step size is set according to the servo drive accuracy of the equipment. The attitude fine-tuning amount refers to the attitude tilt angle adjustment amount of the multi-dimensional attitude adjustment execution module on the α and β axes. The γ axis is the rotation axis, and no adjustment is required for the tube gallery assembly. The clamping force setting value is the clamping force setting command issued by the intelligent agent to the electric gripper, taking into account both the stability of clamping and the damage prevention requirements of the components; the motion dimension is a 6-dimensional motion space, which is precisely matched with the mechanical motion capability of the multi-dimensional posture adjustment execution module, ensuring that the decision-making action can be accurately realized by the execution end.

[0047] Based on the engineering optimization goals of the utility tunnel assembly, prioritizing accuracy while considering efficiency and ensuring safety, a multi-objective weighted reward function is designed. This function transforms the three core engineering indicators—assembly error, assembly time, and stress safety—into reward / penalty signals that can be recognized by the reinforcement learning algorithm, thus clarifying the optimization direction of the intelligent agent.

[0048] ; in, The current assembly error, the maximum permissible assembly error If the value exceeds this value, the assembly is considered to have failed. Standard assembly time; This refers to the actual time spent. The safe stress threshold; All are weighting coefficients; the reward interval R∈[-10,10], negative values ​​are penalties triggered when assembly errors are large, time is too long, or force exceeds the threshold, and positive values ​​are rewards triggered when assembly errors are small, time is short, and force is stable.

[0049] Historical standardized datasets are input into the constructed environment model for simulated interactive verification to check the rationality and suitability of the state space, action space, and reward function. If the simulation results deviate from the actual engineering results by more than 10%, the model parameters are readjusted until the model highly matches the actual engineering results. The verified reinforcement learning environment model is uploaded to a cloud-based big data platform for PPO model training in S3. Simultaneously, the model is lightweighted and deployed to an edge controller for online real-time decision-making in S4.

[0050] S3. Model training based on the PPO algorithm to train the optimal control strategy and provide a decision-making brain for online control; the Proximal Policy Optimization (PPO) algorithm is used for offline pre-training and online fine-tuning of the model to generate the optimal control strategy adapted to different working conditions of the pipe gallery assembly.

[0051] The PPO algorithm model framework, including the policy network and the value network, is built on a cloud big data platform. The network structure is constructed using a deep neural network (DNN). The number of network layers and neurons is set according to the dimensions of the state space and action space (10-dimensional state space - 3 hidden layers - 6-dimensional action space). The network parameters are initialized using a random normal distribution.

[0052] The standardized offline training set of S1 is input into the environment model of S2 to perform offline batch pre-training of the PPO model. The agent continuously performs the interactive loop of "state perception-action decision-reward feedback-policy update" in the environment model. The efficiency of policy update is optimized by estimating the advantage function of GAE, the training instability caused by excessive policy update amplitude is avoided by using the PPO pruning objective function, and the accuracy of value estimation is optimized by using the value network loss function.

[0053] The GAE dominance function is: ; in, This is a discount factor used to balance current rewards and future rewards. A value close to 1 indicates a greater emphasis on future rewards, which aligns with the long-term optimization goals of utility tunnel assembly. For smoothing coefficients; The temporal difference residual is the difference between the immediate reward at the current moment and the estimated value of the future state, reflecting the degree of advantage of the current action. The traditional advantage function is improved to achieve unbiased estimation of advantage and reduction of variance, thereby improving the efficiency and stability of policy updates and accelerating model training convergence.

[0054] The objective function for PPO shearing is: ; in, The expected value of the objective function for PPO shearing; The probability ratio of the new strategy to the old strategy reflects the degree of improvement of the new strategy compared to the old strategy; The shearing coefficient limits the probability ratio between the old and new policies, preventing training crashes caused by excessively large policy update magnitudes. By limiting the magnitude of policy updates through a shearing mechanism, the unstable and divergent training problems of traditional policy gradient algorithms are solved, achieving smooth policy updates and improving the robustness of model training.

[0055] The value network loss function is expressed as follows: ; in, Let $\mathbf{ ... For cumulative discount rewards; For the value network to the current state The value estimate is obtained by optimizing the value estimation capability of the value network through mean squared error loss, enabling the value network to more accurately estimate the true value of each state and providing a reliable value reference for updating the policy network.

[0056] The converged pre-trained model is lightweighted (model pruning, quantization, knowledge distillation) to remove redundant network nodes, reduce the computational load of the model, adapt the model to the computing power and storage resources of the edge controller, and ensure that the model inference accuracy loss is ≤5%.

[0057] Deploying lightweight models to edge controllers and using small-batch data from real-time online assembly for online fine-tuning of the models enables them to quickly adapt to actual on-site conditions (such as different foundation pit environments and different component types), thereby improving the on-site adaptability of the strategy.

[0058] The offline pre-trained and online fine-tuned models are used as the optimal control strategy and deployed to the core inference module of the edge controller for S4's online real-time decision-making. At the same time, the model weights and training records are backed up to the cloud to provide a foundation for subsequent model iterations and updates.

[0059] S4. Online autonomous decision-making and feedback control: Through real-time model reasoning of the edge controller, precise control commands are generated to drive the multi-dimensional attitude adjustment execution module to complete the autonomous correction, precise attitude adjustment and alignment assembly of the pipe gallery components; After execution, the assembly status, error, reward, time consumption and other data are synchronously uploaded to S5 to complete the closed-loop feedback of data.

[0060] The utility tunnel assembly device uses a gantry crane base module to hoist prefabricated components to an initial position above the assembly reference point (X=0, Y=0, Z=3m). The global perception and acquisition module begins to collect positioning, attitude, force, and environmental data in real time. The edge controller performs S1 standardization preprocessing on the real-time data to generate online real-time status data.

[0061] The preprocessed real-time state data (10-dimensional state space) is input into the PPO optimal control model deployed in the edge controller as input data for model inference. Data transmission and processing latency is ≤20ms to ensure real-time performance. The edge controller performs rapid model inference on the input state data to generate 6-dimensional motion control commands (three-axis fine-tuning, two-axis attitude adjustment, and clamping force setpoint). Inference latency is ≤30ms to ensure real-time decision-making. The edge controller sends the generated control commands to the servo drivers of the multi-dimensional attitude adjustment execution module, driving each execution component to complete the corresponding mechanical actions, realizing the micro-adjustment and alignment of the components. Execution latency is ≤50ms to ensure timely command execution.

[0062] After the action is completed, the global perception and acquisition module collects the new status data of the component in real time. After preprocessing, the edge controller calculates the current comprehensive assembly error according to the comprehensive assembly error formula and determines whether the assembly is qualified.

[0063] ; For acceptable error, If E≤2mm, the assembly is considered qualified and the component is immediately locked to complete the assembly. Otherwise, continue the autonomous correction cycle of "perception-decision-execution" with a maximum of ≤50 iterations to avoid equipment failures and construction delays caused by infinite correction.

[0064] If the overall assembly error E > 2mm (the non-conforming error threshold), the new status data is input into the model again to start the next round of the "perception-decision-execution-feedback" cycle until the error is acceptable; if E ≤ 2mm (the acceptable error threshold), the assembly is determined to be complete, the electric clamping claw is driven to lock the components, and the assembly operation is completed; if the error still fails to meet the standard after 50 consecutive iterations, a correction failure warning is triggered, and the operation is suspended.

[0065] ; in, For the total closed-loop delay constraint. For data acquisition and preprocessing latency; For model inference and decision-making delay; To control the delay between the issuance of control commands and the execution of mechanical actions, the total delay of the entire closed-loop control is strictly constrained to ensure real-time control, avoid error accumulation caused by excessive delay, ensure timely correction during assembly, and improve assembly accuracy.

[0066] After standardizing all data from this assembly operation (status data, action commands, error data, time consumption data, and reward data), the data is synchronously uploaded to the cloud big data platform for S5 model iteration and updates, thus completing the data loop.

[0067] S5. Model iteration and big data updates enable lifelong optimization of models and data, improving the continuous adaptability of solutions. The cloud-based big data platform receives real-time on-site measured assembly operation data uploaded by S4, including status data, action instructions, error data, time consumption data, reward data, equipment status data, etc., and stores the data in a unified manner according to engineering standard formats to achieve full lifecycle management of the data.

[0068] The received field data is classified and labeled using a combination of manual and intelligent methods. It is classified according to the type of pipe gallery components, construction conditions, and assembly results. Core data (such as abnormal working condition data and high-precision assembly data) are highlighted and labeled to form an effective dataset after labeling, thus avoiding the impact of invalid data on model training.

[0069] The number of newly added valid assembly samples is counted (a single complete assembly operation record is counted as 1 valid sample). Based on the model update trigger judgment formula, it is determined whether the threshold for model iteration update has been reached. If it has been reached, the update is triggered; otherwise, the samples continue to be accumulated. ; in, To increase the number of valid assembly samples; To minimize the update sample threshold, Through engineering testing and setting, we ensure that incremental training has sufficient sample support to improve the model optimization effect.

[0070] The newly added effective labeled dataset was merged with the existing offline training set to incrementally train the S3 PPO pre-trained model. The same algorithm and hyperparameters as the original model were used, with only minor adjustments to the model weights, without resetting the model parameters. This ensured the continuity and stability of the control strategy. After training, the model convergence was verified to ensure that the optimized model's accuracy was superior to the original model. The iteratively optimized model was then lightweighted and distributed to all field edge controllers via a 5G communication module, enabling remote batch updates and deployment of the model. Simultaneously, the optimized model weights and the new sample library were backed up to provide a foundation for subsequent iterations. The newly added effective labeled dataset was integrated into the S1 cloud-based big data sample library, categorized and stored according to working conditions and component types, enabling continuous expansion of the sample library. This provides richer and more diverse sample data for subsequent model training, improving the model's generalization ability.

[0071] Therefore, this invention adopts the above-mentioned autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data. Through a closed-loop architecture of "hardware full-domain perception - edge real-time decision-making - execution of precise actions", it integrates big data preprocessing, reinforcement learning decision-making and real-time feedback control to achieve intelligent autonomous assembly of utility tunnel components throughout the entire process, which greatly improves assembly accuracy and construction efficiency. At the same time, it has strong adaptability to working conditions and lifelong evolution capability. Through incremental optimization of the PPO model and continuous expansion of the big data sample library, it breaks through the limitations of the solidification of traditional control models and significantly reduces manual intervention and operational errors.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An autonomous feedback control method for utility tunnel assembly devices based on reinforcement learning and big data, characterized in that, Includes the following steps: S1. Big Data Acquisition and Preprocessing: Collect historical assembly data, real-time sensing data, and component parameter data to form a raw dataset. Perform format verification and filtering, 3σ criterion outlier removal, linear interpolation to complete missing data, and Min-Max normalization on the raw dataset. Divide the preprocessed data into an offline training set and an online real-time set, and upload them to the cloud big data platform and transmit them to the edge controller, respectively. S2. Reinforcement Learning Environment Modeling: The problem of precise control of pipe gallery assembly is transformed into a Markov decision process. A 10-dimensional state space is constructed, which includes four core dimensions: positioning, attitude, force, and environment. A 6-dimensional motion space is constructed to match the mechanical motion capability of the multi-dimensional attitude adjustment execution module. A multi-objective weighted reward function is designed with the goals of prioritizing accuracy, taking into account efficiency, and ensuring safety. Historical standardized datasets are input into the environment model for simulation and interactive verification. After successful verification, the model is deployed to the edge controller in a lightweight manner. S3. Model training based on PPO algorithm: A PPO algorithm model framework including policy network and value network is built on a cloud big data platform. A deep neural network is used to construct the network structure. The standardized offline training set of S1 is input into the environment model of S2 for offline batch pre-training. The model is optimized by GAE advantage function, PPO cut-off objective function and value network loss function. The converged pre-trained model is lightweighted and deployed to the edge controller. Online fine-tuning is performed using online real-time assembly data. S4. Online autonomous decision-making and feedback control: The prefabricated components of the pipe gallery are hoisted to the initial position above the assembly reference point through the gantry crane base module. The full-domain perception and acquisition module collects data in real time and after the standardization preprocessing of S1, it is input to the PPO optimal control model of the edge controller. The model infers and generates 6-dimensional motion control commands and sends them to the multi-dimensional posture adjustment execution module. After the action is executed, new state data is collected and the overall assembly error is calculated. S5. Model Iteration and Big Data Update: The cloud-based big data platform receives and stores the assembly data uploaded by S4. After joint classification and labeling by humans and intelligence, a valid dataset is formed. When there are enough new valid assembly samples to support the data, the new dataset is merged with the original training set to incrementally train the PPO model. The iteratively optimized model is then lightweighted and distributed to all field edge controllers. At the same time, the new dataset is integrated into the cloud-based big data sample library. 2.The pipeline assembly device autonomous feedback control method based on reinforcement learning and big data according to claim 1, wherein, Real-time sensing data is collected by the global sensing acquisition module, including positioning sensing data, attitude sensing data, force sensing data, and environmental sensing data. The sampling frequency of the global sensing acquisition module is set synchronously, and all sensors are networked through an industrial Ethernet network and the data is uniformly aggregated into the edge gateway. 3.The pipeline assembly device autonomous feedback control method based on reinforcement learning and big data according to claim 1, characterized in that, The expression for the 3σ criterion is: ； in, This refers to single-frame sampled data from the sensor. The average value of 100 frames of sliding window data; This represents the standard deviation of the corresponding data; The Min-Max normalization expression is: ； in, For the collected state characteristic data; The data is after normalization; This is the lower limit of the sensor's measurement range; This represents the upper limit of the sensor's measurement range.

4. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, The 10-dimensional state-space expression is: ； in, This refers to the three-axis positioning error; For attitude tilt angle; For clamping tension; Ambient temperature; Wind speed; Dust concentration; The 6-dimensional action space expression is: ； in, This refers to the three-axis fine-tuning amount; For attitude fine-tuning; Set the clamping force value.

5. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, The formula for the multi-objective weighted reward function is: ； in, This represents the current assembly error; This represents the maximum permissible assembly error. This represents the current force value. Standard assembly time; This refers to the actual time spent. The safe stress threshold; All are weighting coefficients; the reward interval R∈[-10,10].

6. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, The network structure of a deep neural network is a 10-dimensional state space, 3 hidden layers, and a 6-dimensional action space. The network parameters are initialized using a random normal distribution.

7. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, GAE Advantage Function The formula is: ； in, Discount factor; For smoothing coefficients; For time-series difference residuals; PPO shearing objective function The formula is: ； in, The expected value of the objective function for PPO shearing; The probability ratio between the old and new strategies; The shear coefficient; Value network loss function The expression is: ； in, Let $\mathbf{ ... For cumulative discount rewards; For the value network to the current state The estimated value.

8. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, Current assembly overall error The calculation formula is: ； in, , , This represents the displacement along the X, Y, and Z axes; , For the attitude tilt angle changes along the α and β axes; The time delay of closed-loop control satisfies: ； in, For the total closed-loop delay constraint. For data acquisition and preprocessing latency; For model inference and decision-making delay; To control the delay between issuing commands and executing mechanical actions.

9. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, The method also includes establishing the standard assembly reference point of the utility tunnel foundation pit as the origin of the coordinate system. The project uses a right-hand rectangular coordinate system, with the X-axis representing the longitudinal extension of the utility tunnel, the Y-axis representing the transverse span of the utility tunnel, and the Z-axis representing the vertical hoisting and lifting direction. The positioning, data acquisition, and control commands of all modules of the utility tunnel assembly device are based solely on this coordinate system.

10. The autonomous feedback control method for pipe gallery assembly device based on reinforcement learning and big data according to claim 1, characterized in that, The utility tunnel assembly device includes a gantry crane base module, a multi-dimensional attitude adjustment execution module, a global perception and acquisition module, and a control and data processing module. The control and data processing module includes an edge controller and a cloud big data platform. The edge controller and the cloud big data platform achieve high-speed data transmission through 5G communication.

Images