Predictive beam field beam moving track optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of intelligent construction and trajectory optimization control, and discloses a prediction beam field beam moving trajectory optimization method based on deep reinforcement learning. According to the method, a camera monitoring image and building information model (BIM) structure data are fused, a beam body position, an equipment state and pedestal distribution characteristics are extracted, and a high-dimensional state vector is constructed; a beam field space is modeled into a graph structure, and a graph attention network and a double-Q network are combined to achieve track value evaluation and optimal path selection; before track generation, a disturbance weight map is formed based on a converter attention prediction network, and dynamic covering is performed on high-risk nodes to ensure track accessibility and safety; and the generated trajectory instruction is issued in real time through the edge computing node and is subjected to three-dimensional simulation verification and feedback correction in the digital twin platform. According to the method, through fusion of visual perception, reinforcement learning and twin simulation, self-learning and closed-loop optimization of a beam moving path are realized, and automation, safety and scheduling efficiency of beam field operation can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent construction and trajectory optimization control, specifically to a method for optimizing beam movement trajectories in a pre-defined beam yard based on deep reinforcement learning. Background Technology

[0002] With the expansion of infrastructure construction, pre-positioned bridge beam yards have become an important part of engineering construction. To improve construction efficiency and safety, the industry is gradually introducing information management methods, using BIM models, IoT sensors, and digital twin platforms to achieve visualized monitoring of beams, lifting equipment, and tracks. However, in existing systems, the scheduling and trajectory planning of beam movement operations in beam yards still rely mainly on manual experience, lacking intelligent decision support.

[0003] Currently, most common path planning methods are based on static algorithms or rule templates, with algorithms matching paths based on preset working conditions. These methods can complete basic path calculations in single-device, small-scale scenarios, but they have significant shortcomings in multi-device collaboration or dynamic construction environments. On the one hand, the algorithms cannot perceive equipment status and on-site disturbances in real time, leading to delayed or invalid planning results; on the other hand, the path optimization objective is singular, making it difficult to take into account multi-dimensional constraints such as safety distance, equipment utilization, and construction timeliness.

[0004] Furthermore, existing systems lack the ability to fuse and model multi-source sensing data. The spatial relationships between various lifting devices, beams, and track nodes in the beam yard are complex, making it difficult for traditional algorithms to maintain global optimality and local safety in path planning within dynamic topologies. Due to this limitation, manual intervention is still required on-site to avoid equipment conflicts, impacting scheduling efficiency and safety. Therefore, we propose a deep reinforcement learning-based method for optimizing beam movement trajectories in the beam yard. Summary of the Invention

[0005] The purpose of this invention is to provide a method for optimizing the trajectory of beam moving in a beam yard based on deep reinforcement learning, in order to solve the problems of path planning relying on human experience, difficulty in coping with dynamic environmental changes, lack of real-time perception and simulation verification, and insufficient trajectory execution security in the existing beam moving process.

[0006] To achieve the above-mentioned objectives, the present invention employs the following technical solution: a method for optimizing beam movement trajectories in a predicted beam field based on deep reinforcement learning, comprising the following steps: S1: The camera surveillance images and building information model (BIM) structural data are fused and identified to obtain information on beam location, equipment status and pedestal distribution. The pre-trained residual network is used to extract features from the beams, cranes, pedestals and work area objects contained in the camera surveillance images to obtain multi-scale image features that reflect beam location, equipment status and scene semantics. These features are then stitched with structured data to construct a high-dimensional state vector. S2: Model the beam yard space as a graph structure, where graph nodes represent reachable locations of equipment and graph edges represent passageways. Construct a deep reinforcement learning trajectory optimization model that combines a graph attention network and a double Q network structure. The deep reinforcement learning trajectory optimization model takes the graph structure and the high-dimensional state vector as input to learn the optimal beam moving path strategy between reachable nodes of equipment in the beam yard. The generated trajectory optimization results are evaluated according to a preset reward function to guide strategy updates. S3: Before the deep reinforcement learning trajectory optimization model generates trajectory actions, the action space of the deep reinforcement learning trajectory optimization model is masked based on the perturbation weight graph composed of node risk scores output by the transformer attention prediction network. When the perturbation weight of a certain path node in the graph structure calculated by the transformer attention prediction network is higher than a preset threshold, the action corresponding to the path node is removed from the set of optional actions of the deep reinforcement learning trajectory optimization model to ensure the reachability and safety of the generated trajectory. S4: The optimal strategy output by the deep reinforcement learning trajectory optimization model after occlusion processing generates the corresponding beam-moving path sequence. The beam-moving path is composed of nodes in the graph structure, reflecting the feasible movement route of the beam-moving device. The beam-moving path is converted into a trajectory instruction containing device type, path nodes and timestamp, and sent to the edge control terminal to drive the beam-moving device to execute. S5: In the digital twin platform, the trajectory simulation module is called. The beam-moving path is mapped to a three-dimensional animation trajectory for conflict pre-simulation. Based on the equipment operation feedback information and simulation analysis results collected by the edge control terminal, the trajectory planning parameters and occlusion threshold are corrected to achieve closed-loop optimization of the strategy.

[0007] Preferably, the high-dimensional state vector is composed of an image feature vector and a location information vector extracted from Building Information Modeling (BIM) structural data, concatenated along the channel dimension. The image feature vector is extracted by a pre-trained 50-layer residual neural network in multi-scale channels, with an output feature dimension of 256–512. The location information vector includes the three-dimensional coordinates of the beam center point, the target platform number, and the current pose parameters of the operating equipment. After encoding, it forms a 32–64-dimensional structural vector. The total dimension of the concatenated high-dimensional state vector is approximately 512–1024, used to characterize the spatial topology and operating status of the beam yard.

[0008] Preferably, the perturbation weight map is generated by a transformer attention prediction network, and the output node weights are used to dynamically mask actions on impassable paths before policy execution, forming a corresponding action masking mechanism. The action masking mechanism is used to limit the selectable range of action sets based on node risk scores during the execution phase, thereby achieving masking control of high-risk paths.

[0009] Preferably, the deep reinforcement learning trajectory optimization model adopts a dual-Q network structure, including a main Q network and a target Q network, wherein the parameters of the target Q network are updated in accordance with the parameters of the main Q network through a soft update method.

[0010] Preferably, in the process of constructing the deep reinforcement learning trajectory optimization model, graph nodes are encoded into high-dimensional state vectors. The feature vectors include topological location information, perturbation prediction scores, and image detection results. After being encoded by a graph attention network, they are input into a double-Q network for decision calculation. The deep reinforcement learning trajectory optimization model evaluates the trajectory optimization results according to a preset reward function to guide the model's policy update and path selection.

[0011] Preferably, the reward function is obtained by linearly weighting the travel distance penalty, path congestion penalty, timeout penalty, and equipment idle rate reward with adjustable weights. The travel distance penalty is used to limit the trajectory path length, the path congestion penalty reflects the node passage risk or congestion level, the timeout penalty is used to constrain the task execution time, and the equipment idle rate reward is used to improve equipment utilization efficiency. The weight parameters can be adaptively adjusted according to simulation or field operation feedback to achieve a dynamic balance between safety and operation efficiency.

[0012] Preferably, the action masking mechanism dynamically removes path actions with risk scores greater than a set threshold based on node blocking scores in the perturbation weight graph generated by the transformer attention prediction network, so as to ensure the physical feasibility of the trajectory action set.

[0013] Preferably, the trajectory instruction includes a device ID, a path number sequence, target node coordinates, and an execution timestamp. The instruction format is compatible with the gantry crane control interface and the beam-moving vehicle control protocol, and is sent to the edge device via a 5G network.

[0014] Preferably, the trajectory simulation module constructs a three-dimensional simulation scene based on a digital twin platform, and the path execution is visualized in the virtual beam yard in the form of animation, supporting path conflict detection, target switching and manual review.

[0015] Preferably, the deep reinforcement learning trajectory optimization model is deployed on an edge computing node. During the training phase, the model completes offline learning on a historical task trajectory set. During the execution phase, it receives real-time on-site status data and completes closed-loop operations of trajectory generation, occlusion processing, device driving, and feedback correction.

[0016] Compared with existing technologies, the deep reinforcement learning-based method for optimizing beam movement trajectories in a pre-defined beam field, which employs the above-mentioned technical solution, has the following beneficial effects: I. By integrating video surveillance images with Building Information Modeling (BIM) structural data, a unified perception system for multi-source heterogeneous information is established. Residual neural networks are used to extract multi-scale features of beams, lifting equipment, platforms, and work areas. These features are then jointly modeled with spatial coordinates, equipment numbers, and topological relationships from the BIM, constructing a high-dimensional vector space reflecting the dynamic state of the beam yard. This enables the system to identify beam positions and equipment operating status in real time, fundamentally improving the accuracy and timeliness of the work environment perception. Second, by abstracting the beam field space into a graph structure and introducing a joint decision-making mechanism of graph attention network and double Q network, the trajectory planning process no longer relies on fixed rules or human experience. This structure can adaptively learn in a dynamic environment with multiple nodes and multiple paths, and achieves dynamic optimization of the global path through value function iteration and node attention weighting, effectively avoiding path redundancy, local optima and resource conflicts in traditional algorithms, and improving the global convergence and execution stability of trajectory planning; Third, this invention introduces a disturbance prediction module based on a transformer structure before trajectory generation to quantify and score node risks and generate a disturbance weight map, performing dynamic masking on high-risk nodes. This mechanism can mask passageways affected by obstacles, congestion, or abnormal loads in real time, ensuring that the trajectory is physically reachable and safety is controllable, achieving dual protection of safety and efficiency.

[0017] Fourth, this invention combines a digital twin platform to perform three-dimensional simulation verification of the generated trajectory, and uses a virtual beam yard environment to visualize and detect trajectory conflicts, energy consumption distribution, and equipment coordination. After the simulation results are fed back to the learning module, the reward function parameters and occlusion threshold can be automatically corrected, realizing closed-loop optimization of planning, execution, and verification, thereby enabling the model to have self-correction and self-evolution capabilities.

[0018] Fifth, this invention adopts an edge computing deployment mode in its system architecture. Offline reinforcement learning is performed during the training phase, while real-time access to on-site monitoring data is conducted during the execution phase, ensuring that decision-making latency is lower than engineering scheduling requirements. Trajectory command issuance and equipment feedback collection are achieved through a 5G network, supporting parallel control of multiple devices and cross-section collaborative scheduling, significantly improving the system's response speed and scalability. Attached Figure Description

[0019] Figure 1 The overall method flowchart of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] like Figure 1As shown, the method for optimizing the trajectory of beam movement in a beam yard based on deep reinforcement learning uses the fused data of the beam yard's Building Information Model (BIM) structural data and video surveillance as input data sources to construct the basis for beam movement state perception and path decision-making. The method includes the following steps: S1: The camera surveillance images and building information model (BIM) structural data are fused and identified to obtain information on beam location, equipment status and pedestal distribution. The pre-trained residual network (ResNet-50) is used to extract features from the beams, cranes, pedestals and work area objects in the camera surveillance images to obtain multi-scale image features that reflect beam location, equipment status and scene semantics. These features are then concatenated with structured data to construct a high-dimensional state vector. The video surveillance images come from a multi-channel high-definition industrial camera system deployed on-site at the beam yard. This system includes two types of cameras: top-view cameras and side-view cameras.

[0022] The overhead camera is installed above the hoisting rail to capture real-time panoramic views of the beam storage yard, transportation channels, and crane operation area; Side-view cameras are installed near the hoisting robotic arm and pedestal to monitor the relative motion of the beam and equipment.

[0023] The acquired video stream is processed by edge computing nodes for frame extraction and illumination normalization to form a time-synchronized image sequence input.

[0024] Building Information Modeling (BIM) originates from the three-dimensional digital model files created during the beam yard design phase. BIM includes beam numbering, platform coordinates, equipment layout, and access path topology information.

[0025] After Building Information Modeling (BIM) data is imported into the system through the IFC interface, it undergoes spatial subdivision and coordinate alignment, and external parameter calibration with the camera's perspective, enabling the two types of data to be integrated under a unified coordinate system.

[0026] This step belongs to the state awareness and feature construction stage, where the system acquires frame sequences from the surveillance camera. (Preferred resolution: 1280×720, frame rate: 10~30fps), and export structured data tables from Building Information Modeling (BIM). ,in, Here are the coordinates of the center point of the beam. Number the target platform. This represents the location vector of the operating equipment. The image is obtained by extracting multi-scale features using a residual network model (ResNet-50) and then reducing the dimensionality using global average pooling (GAP).

[0027] Target pedestal number Location of working equipment composition Building Information Modeling (BIM) data normalization mapping is .

[0028] The two are concatenated along the channels to form a high-dimensional state vector:

[0029] To ensure numerical stability, the image channels are normalized using mean / variance, and the coordinates are standardized to zero mean at the scene scale. The state vector update frequency is consistent with the video sampling frequency to achieve real-time response to on-site dynamics.

[0030] S2: Model the beam yard space as a graph structure, where graph nodes represent reachable locations of equipment and graph edges represent passageways. Construct a deep reinforcement learning trajectory optimization model that combines a graph attention network (GAT) and a double Q network structure. The deep reinforcement learning trajectory optimization model takes the graph structure and the high-dimensional state vector as input to learn the optimal beam moving path strategy between reachable nodes of equipment in the beam yard.

[0031] Abstract the beam field as a directed graph. ,node To reach the work point / platform / transfer point, the side For the passage path, edge weight This represents the travel cost (a weighted sum of energy consumption, time, and security risks). Node characteristics are initialized as follows: A graph attention network (GAT) is used to aggregate attention in the neighborhood:

[0032] in For trainable mapping matrix, For attention vectors, " is for concatenation. The encoded node is embedded into the double-Q network for action value evaluation, and the output is used for subsequent policy selection.

[0033] S3: Before the deep reinforcement learning trajectory optimization model generates trajectory actions, the action space of the deep reinforcement learning trajectory optimization model is masked based on the perturbation weight graph composed of node risk scores output by the transformer attention prediction network. A certain path node in the graph structure (from the previous...) When the perturbation weight calculated by the transformer attention prediction network (the node risk score obtained by calculating the node state sequence) is higher than a preset threshold, the action corresponding to the path node is removed from the set of optional actions of the deep reinforcement learning trajectory optimization model to ensure the reachability and safety of the generated trajectory. Construct a perturbation prediction network based on a transformer to achieve near-normal perturbation prediction. Step node state sequence The output is the perturbation score vector. Employing multi-head attention:

[0034] Set threshold ,when When the action is removed from the selectable set, the occluded action set is obtained. To adapt to the uncertainties of the project, It can be adaptively fine-tuned according to equipment load / environment level (e.g., increase or decrease by 0.05).

[0035] S4: The optimal strategy output by the deep reinforcement learning trajectory optimization model after occlusion processing generates the corresponding beam-moving path sequence. The path is composed of nodes in the graph structure, reflecting the feasible movement route of the beam-moving device. The path is converted into a trajectory instruction containing device type, path nodes and timestamp, and sent to the edge control terminal to drive the beam-moving device to execute. Strategy Adoption or Sampling at Select an action to generate a path sequence. Encapsulate the path into an instruction frame:

[0036] and with / The format is transmitted to the edge controller via 5G or Industrial Ethernet; the controller will... It is mapped to the device's underlying control commands (speed, acceleration, docking point, etc.) and has packet loss retransmission and timeout rollback mechanisms to ensure continuous execution.

[0037] S5: In the digital twin platform, the trajectory simulation module is called, and the beam-moving path is mapped to a three-dimensional animation trajectory for conflict pre-simulation; and based on the equipment operation feedback information and simulation analysis results collected by the edge control terminal, the trajectory planning parameters and occlusion threshold are corrected to achieve closed-loop optimization of the strategy; Will The simulation is replayed in the twin engine, performing collision detection (node / edge occupancy conflicts, timing conflicts, device interlock conflicts). Simulation feedback (time consumption, congestion rate, conflict count) is used to correct trajectory planning parameters and occlusion thresholds, realizing a feedback loop between simulation verification, training updates, and deployment execution; when the conflict exceeds the set threshold, local replanning is automatically triggered.

[0038] The high-dimensional state vector is composed of an image feature vector and a location information vector extracted from Building Information Modeling (BIM) structural data, concatenated along the channel dimension. The image feature vector is extracted from multi-scale channels by a pre-trained 50-layer residual neural network (ResNet-50), with an output feature dimension of 256–512. The location information vector includes the three-dimensional coordinates of the beam center point, the target platform number, and the current pose parameters of the operating equipment. After encoding, it forms a 32–64-dimensional structural vector. The total dimension of the concatenated high-dimensional state vector is approximately 512–1024, used to characterize the spatial topology and operating status of the beam yard.

[0039] In this embodiment, The preferred dimension is 256–512. The preferred dimensions are 32-64, after splicing The dimensions are approximately 512~1024. The meaning of the state vector field is as follows: Semantic features obtained by combining the output layer (C5) of the ResNet-50 backbone network with global average pooling (GAP) characterize the edges and textures of the beam / spreader / track. :Include , , Fields such as these are used to constrain spatial topology and target location information.

[0040] To facilitate engineering reproduction, Given in the global coordinate system of Building Information Modeling (BIM) (unit: m). Obtain the current center coordinates of the device or the pose vector of key joints.

[0041] The perturbation weight graph is generated by a Transformer attention prediction network. The output node weights are used to dynamically mask actions on impassable paths before policy execution, forming a corresponding action masking mechanism. This action masking mechanism is used to limit the selectable range of action sets based on node risk scores during the execution phase, thereby achieving masking control of high-risk paths.

[0042] This invention predicts the disturbance risk of beam field nodes using a Transformer attention network. The input is the sequence of node states from the previous k steps. The state vector of each node From the state vector in step S1 The mapping is obtained.

[0043] The Transformer network employs a multi-head attention mechanism, and its calculation formula is as follows:

[0044] In the formula: , , These are the query matrix, key matrix, and value matrix, respectively. : Key vector dimension; : Node perturbation weight vector; Total number of nodes.

[0045] Perturbation weights of network output It reflects the risk value of the node due to factors such as obstacles, vibration, load fluctuations, or ground friction.

[0046] when ( To set a threshold (preferably 0.6–0.8), the corresponding path node is marked as impassable, and occlusion is performed in the action space. The set of actions that can be executed after occlusion is as follows:

[0047] This mechanism ensures that the model explores only reachable and safe paths in dynamic environments, significantly reducing the proportion of invalid actions during training and improving beam-moving safety.

[0048] The deep reinforcement learning trajectory optimization model adopts a dual Q-network structure, including a main Q-network and a target Q-network. The parameters of the target Q-network are updated in a soft update manner, following the parameters of the main Q-network. The update formula is as follows:

[0049] In the formula: Main Q network parameters; The target Q network parameters; This is the soft update coefficient (usually taken as 0.001 to 0.01), used to control the synchronization rate; This indicates a parameter assignment operation.

[0050] By using the soft update method described above, overfitting and oscillations of the reinforcement learning model can be avoided during parameter iteration, allowing the target network to maintain stability during parameter updates, thereby improving the convergence and reliability of trajectory optimization results.

[0051] The main Q-network (online Q-network) is responsible for outputting real-time action policies during training. The target Q-network is used to provide a reference target in Q-value updates, and its update formula is as follows:

[0052] The target Q value; The reward is given immediately at the current moment. This is the discount factor (usually taken as 0.9 to 0.99). The next state Q value is calculated for the target network; Operators that indicate the selection of the optimal action.

[0053] Once the target network output stabilizes, the main Q-network updates its parameters using gradient descent.

[0054] In the formula: The learning rate; Let be the loss function of the Q-network.

[0055] By introducing a dual-Q network structure: the main network is responsible for policy exploration, and the target network provides reference signals. The parameter synchronization adopts a soft update mechanism, which avoids instability caused by large parameter updates. Compared with single-Q networks or hard update mode, the training convergence speed is significantly improved, and the trajectory planning is smoother.

[0056] This structure can effectively reduce training fluctuations and improve the stability and safety of trajectory generation in beam yard trajectory optimization.

[0057] In the process of constructing the deep reinforcement learning trajectory optimization model, graph nodes are encoded into high-dimensional state vectors (i.e., multimodal feature vectors that fuse image features and Building Information Modeling (BIM) structural data). The feature vectors include topological location information, disturbance prediction scores, and image detection results. After being encoded by a graph attention network, they are input into a dual-Q network for decision calculation. The deep reinforcement learning trajectory optimization model evaluates the trajectory optimization results according to a preset reward function to guide the model's strategy update and path selection.

[0058] Multimodal node features are defined as follows:

[0059] in For topological / geometric location features, For perturbation scoring, Image detection results (category and bounding box center). Aggregated by a Graph Attention Network (GAT). Then, enter the Q network. The value of each candidate action is calculated to achieve a joint measurement of safety and efficiency.

[0060] The reward function is obtained by linearly weighting the travel distance penalty, path congestion penalty, timeout penalty, and equipment idle rate reward with adjustable weights. The travel distance penalty is used to limit the trajectory path length, the path congestion penalty reflects the node passage risk or congestion degree, the timeout penalty is used to constrain the task execution time, and the equipment idle rate reward is used to improve equipment utilization efficiency. The weight parameters can be adaptively adjusted according to simulation or field operation feedback to achieve a dynamic balance between safety and operation efficiency. The reward function includes the following components: travel distance penalty, path congestion penalty, timeout penalty, and device idle rate reward. Its comprehensive scoring function is as follows:

[0061] in , , , These are adjustable weight parameters.

[0062] Set up a comprehensive reward:

[0063] in: (Distance normalization penalty) (Blocking probability penalty) (overtime penalty) (Idle rate reward); This can be determined through simulation mesh search. Adaptive weights can be introduced to match field preferences. .

[0064] The action masking mechanism dynamically removes path actions with risk scores greater than a set threshold based on node blocking scores in the perturbation weight graph generated by the transformer attention prediction network, in order to ensure the physical feasibility of the trajectory action set.

[0065] The blocking score can be defined as:

[0066] in, A weighted index of energy consumption, slope, and historical congestion. For empirical coefficients, when ( The preferred value is 0.5–0.7, which corresponds to occlusion of the action. This implementation ensures that the action set is physically feasible and reduces invalid exploration.

[0067] The trajectory command includes device ID, path number sequence, target node coordinates and execution timestamp. The command format supports adaptation with gantry crane control interface and beam moving vehicle control protocol, and is sent to edge device via 5G network.

[0068] In this embodiment, in order to achieve controllable execution of the beam-moving device, the system converts the path sequence output by the deep reinforcement learning model into a structured trajectory instruction frame.

[0069] The trajectory instruction includes the following field definitions: , , , in, This represents a unique device identifier, and its type is a 32-bit unsigned integer. ), used to identify specific beam-moving equipment or gantry cranes; The path number sequence is an integer array. ), used to describe the execution order of each node in the trajectory; The target node coordinates are a three-dimensional floating-point array. ), indicating that the target point is in three , , Spatial coordinates in the direction; This is a timestamp field, of type 64-bit integer ( ), used to record the time or scheduling cycle index when the instruction was generated.

[0070] In an optional embodiment, to achieve motion curve control and smooth acceleration and deceleration, the following fields can be extended based on the above: , .

[0071] in, This represents the velocity distribution curve for each segment of the trajectory. Both represent acceleration curves and are floating-point arrays used to achieve dynamic trajectory smoothing during execution. Trajectory instructions are executed according to... or The command is encapsulated in a specific format and transmitted to the edge control terminal via a 5G network or industrial Ethernet. The edge control terminal then parses the command and, based on... Call the corresponding device control channel to The nodes shown are executed sequentially until the target coordinates are reached. The beam-moving operation is shown.

[0072] It should be noted that the above field type annotations are only used to describe the data structure definition and to explain the technical implementation method, and do not limit the specific programming language or communication protocol format.

[0073] The trajectory simulation module constructs a three-dimensional simulation scene based on a digital twin platform. The path execution is visualized in the virtual beam yard in the form of animation, and supports path conflict detection, target switching and manual review.

[0074] The trajectory simulation module in this embodiment is built on a digital twin platform and is used to perform 3D simulation and visualization verification of the beam-moving operation process in the beam yard. The main function of this module is to map the path planning results generated by the deep reinforcement learning model into the virtual space, present the beam-moving process in the form of animation, and realize path conflict detection, target switching and manual review functions in the simulation environment, thereby verifying the feasibility and safety of the trajectory in advance before the model is deployed.

[0075] The simulation module is based on Building Information Modeling (BIM) and imports the geometric parameters of beams, pedestals, tracks, hoists, and operating vehicles from the engineering database. Through a data interface, it maps the actual beam yard's dimensional information, coordinate references, and equipment kinematic characteristics (including travel limits, speed limits, and safety distances) into the virtual scene.

[0076] To ensure consistency between the simulation scenario and the actual environment, the system performs coordinate alignment and unit conversion during the model import phase, unifying the Building Information Model (BIM) coordinate system with the simulation coordinate system. The digital twin platform periodically receives status data from on-site sensors, such as hoist posture, platform occupancy information, and equipment operating position, and displays this data dynamically and synchronously in the virtual scene, achieving virtual-real linkage. When the deep reinforcement learning model outputs a trajectory instruction set... Then, the simulation module imports the path node data into the visualization rendering engine. The system calculates the 3D coordinates, velocity, and attitude changes of each node sequentially according to the time series, and generates a continuous path curve based on the interpolation algorithm.

[0077] The path execution process is presented in animation, with the beam moving smoothly along the planned trajectory, and the movements of the lifting equipment and track rendered synchronously. The system provides a free view and preset view switching function, enabling multiple observation modes such as global bird's-eye view, local follow, and path comparison. Key nodes in the animation process (such as lifting, acceleration, deceleration, and placement) are displayed with markers and text labels, making it easy for operators to understand the trajectory logic and equipment operating status.

[0078] The above settings enable dynamic reproduction of the beam yard operation process, allowing users to intuitively observe the beam moving path, equipment movement rhythm, and potential risk areas in a virtual space.

[0079] The simulation module incorporates a path conflict detection algorithm to analyze motion interference between different devices, beams, or lifting devices. The system establishes an envelope model of the devices and uses 3D bounding boxes and time-annotated path segments for collision detection. When the minimum spatial distance between two devices is less than a set safety threshold (preferably 0.6–1.0 m), the system immediately identifies it as a potential conflict and displays a red path or warning light effect.

[0080] In addition, the system also supports time conflict detection. This is achieved by comparing the action time intervals of the devices. , If two trajectories occupy the same platform or adjacent track areas within the same interval, a conflict report is automatically generated. Conflict information is stored in the simulation log, providing a data foundation for subsequent trajectory optimization and training sample expansion.

[0081] This detection mechanism can identify trajectory intersections, operational congestion, and path competition issues before the model is actually deployed, thereby adjusting planning parameters in advance and significantly reducing the risk of on-site scheduling conflicts.

[0082] To cope with task adjustments or emergencies at the construction site, the simulation module supports target switching and local replanning functions.

[0083] When the user selects a new target platform or modifies the beam number in the simulation interface, the system automatically calculates the shortest safe path from the current state to the new target point and generates a new, unexecuted path without changing the already executed trajectory segments. The new path is immediately updated and displayed in the animation after generation, and its color is distinguishable from the old trajectory for easy identification.

[0084] This function can update the path without interrupting the simulation process, enabling real-time connection between simulation verification and parameter adjustment, thereby ensuring the continuity and security of path planning results under multi-task scheduling conditions.

[0085] The trajectory simulation module provides a manual review interface, allowing designers or operators to manually verify the rationality of the path during the simulation process. The system supports interactive functions such as pause, playback, single-step playback, node editing, and path annotation. Users can measure the distance, angle, or speed of the beam position at any time and manually adjust the node positions on the interface.

[0086] If manual review reveals path deviations or execution risks, the system automatically records the differences before and after modification, generates a "review record file," and uploads it to the model management module for use as a training or validation sample for reinforcement learning. This combination of manual review and algorithmic analysis achieves a dual verification mechanism for the path planning scheme.

[0087] The review module can also output a "path risk report", which includes KPI indicators such as the number of conflicts, minimum spacing, running time and number of task switching, providing data support for the evaluation of construction plans.

[0088] Through the above design, the trajectory simulation module of this embodiment realizes the three-dimensional dynamic display and safety verification of the beam-moving path on the digital twin platform. This not only significantly improves the interpretability and auditability of trajectory planning, but also prevents operational conflicts and scheduling errors in the virtual environment. Compared with traditional two-dimensional path verification methods, this module has advantages such as strong interactivity, high simulation accuracy, and fast review efficiency, providing a safe and reliable technical foundation for subsequent model training, parameter adjustment, and on-site deployment.

[0089] The method is deployed on edge computing nodes. During the training phase, the model learns offline on the historical task trajectory set. During the execution phase, the on-site status data is received in real time to complete the closed-loop operation of trajectory generation, occlusion processing, device driving and feedback correction.

[0090] The trajectory optimization method in this embodiment adopts a distributed deployment approach in its system architecture, where the core model is deployed on edge computing nodes, enabling layered processing of offline model training and online execution. This design can improve model update efficiency and operational safety while ensuring that decision latency meets on-site requirements.

[0091] During the training phase, the system utilizes historical task trajectory data, sensor records, and virtual samples generated by the simulation platform to establish a dataset. The dataset includes state vectors, action commands, immediate reward values, and execution results. The state vectors, composed of video features and Building Information Modeling (BIM) structural data, describe the spatial state of the beams, lifting equipment, and tracks. Action commands correspond to the equipment's displacement, velocity, or attitude control parameters, and reward values ​​are calculated based on beam-moving efficiency, safety distance, and energy consumption levels.

[0092] Training is performed in a cloud or data center with high computing power. The model employs a Double-Q network structure (DoubleDQN) combined with a Graph Attention Network (GAT) to achieve dynamic interaction modeling between multiple devices. The model parameters are optimized using the following loss function:

[0093] in, For instant rewards, As a discount factor, For the target network, The main network is used. After training, the output parameter files and configuration files are packaged into a unified model version (e.g., model_v1.0) and distributed to edge computing nodes via an encrypted channel. Each version is verified and signed to ensure the integrity and reliability of parameter transmission.

[0094] The offline training phase also includes a model validation stage. The system uses a digital twin platform for virtual simulation to verify the stability and security of the model's output trajectory. When the average reward converges and the number of trajectory conflicts is less than a preset threshold, the model is deemed to meet the deployment conditions. In this way, the system ensures that the model executed at the edge has sufficient feasibility and robustness.

[0095] During the execution phase, edge computing nodes are deployed at the control terminals on the beam yard site. Equipped with GPU or AI acceleration modules, they can complete data parsing and inference calculations within millisecond latency. The nodes receive on-site monitoring images, equipment sensor data, and incremental Building Information Modeling (BIM) information in real time via a high-speed industrial network.

[0096] The system first constructs the current state vector. ,in This represents the feature vector extracted from the video surveillance image. This represents the spatial structural features extracted from Building Information Modeling (BIM). The state vector is input into the deep reinforcement learning model, which outputs a set of actions based on the current policy. .

[0097] To ensure the safety and executability of the trajectory, the system introduces a masking mechanism to filter high-risk actions and retain the set of actions assessed as safe by the disturbance prediction module. The dual-Q network then calculates the value function for each candidate action, selects the action corresponding to the largest Q value, and generates a trajectory instruction sequence.

[0098] The generated trajectory instruction set includes device number, path node sequence, target coordinates, and timestamp information, which are encapsulated into a trajectory instruction set. The instruction includes fields such as device number, path node sequence, target coordinates, and timestamp. The instruction set is sent to the equipment control system via 5G or industrial Ethernet, and is parsed and executed by the PLC or servo control module to achieve automated scheduling of equipment such as gantry cranes and beam movers.

[0099] During trajectory execution, the system continuously collects execution feedback data, including indicators such as position error, motion latency, equipment load, and energy consumption. Edge nodes compare the real-time feedback with the model prediction results. When the deviation exceeds a threshold, a local replanning mechanism is triggered to recalculate the remaining nodes of the current path and generate alternative instructions.

[0100] Meanwhile, the system dynamically adjusts the model parameters based on key performance indicators (KPIs): If path conflicts or excessive energy consumption occur, increase the safety weight factor. Reduce the speed priority weight; If the execution delay is significant, increase the weight of the efficiency component in the action reward. ; Masking threshold Perform adaptive correction:

[0101] in, For learning rate, This is the current deviation value.

[0102] The corrected parameters take effect immediately without requiring model reloading. Execution records and correction logs are periodically uploaded to a cloud server for offline retraining and model distillation, enabling long-term self-learning and performance iteration.

[0103] This layered deployment approach allows for a reasonable separation between model training and on-site execution. The cloud handles centralized training and periodic updates of model parameters, while edge nodes are responsible for trajectory inference and real-time generation of equipment control commands. These two components form a closed loop through data feedback and parameter update mechanisms, enabling the model to maintain stable planning accuracy and response speed under real-world conditions.

[0104] In typical engineering applications, the average decision delay of edge nodes is controlled at around 60 ms, and the trajectory positioning error is less than 0.1 m, which can meet the timeliness and safety requirements of beam moving operations in beam yards.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning, characterized in that, Includes the following steps: S1: The camera surveillance images and building information model (BIM) structural data are fused and identified to obtain information on beam location, equipment status and pedestal distribution. The pre-trained residual network is used to extract features from the beams, cranes, pedestals and work area objects contained in the camera surveillance images to obtain multi-scale image features that reflect beam location, equipment status and scene semantics. These features are then stitched with structured data to construct a high-dimensional state vector. S2: Model the beam yard space as a graph structure, where graph nodes represent reachable locations of equipment and graph edges represent passageways. Construct a deep reinforcement learning trajectory optimization model that combines a graph attention network and a double Q network structure. The deep reinforcement learning trajectory optimization model takes the graph structure and the high-dimensional state vector as input to learn the optimal beam moving path strategy between reachable nodes of equipment in the beam yard. The generated trajectory optimization results are evaluated according to a preset reward function to guide strategy updates. S3: Before the deep reinforcement learning trajectory optimization model generates trajectory actions, the action space of the deep reinforcement learning trajectory optimization model is masked based on the perturbation weight graph composed of node risk scores output by the transformer attention prediction network. When the perturbation weight of a certain path node in the graph structure calculated by the transformer attention prediction network is higher than a preset threshold, the action corresponding to the path node is removed from the set of optional actions of the deep reinforcement learning trajectory optimization model to ensure the reachability and safety of the generated trajectory. S4: The optimal strategy output by the deep reinforcement learning trajectory optimization model after occlusion processing generates the corresponding beam-moving path sequence. The beam-moving path is composed of nodes in the graph structure, reflecting the feasible movement route of the beam-moving device. The beam-moving path is converted into a trajectory instruction containing device type, path nodes and timestamp, and sent to the edge control terminal to drive the beam-moving device to execute. S5: In the digital twin platform, the trajectory simulation module is called. The beam-moving path is mapped to a three-dimensional animation trajectory for conflict pre-simulation. Based on the equipment operation feedback information and simulation analysis results collected by the edge control terminal, the trajectory planning parameters and occlusion threshold are corrected to achieve closed-loop optimization of the strategy.

2. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The high-dimensional state vector is composed of an image feature vector and a location information vector extracted from Building Information Modeling (BIM) structural data, concatenated along the channel dimension. The image feature vector is extracted by a pre-trained 50-layer residual neural network in multi-scale channels, with an output feature dimension of 256–512. The location information vector includes the three-dimensional coordinates of the beam center point, the target platform number, and the current pose parameters of the operating equipment. After encoding, it forms a 32–64-dimensional structural vector. The total dimension of the concatenated high-dimensional state vector is approximately 512–1024, used to characterize the spatial topology and operating status of the beam yard.

3. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The perturbation weight graph is generated by the transformer attention prediction network. The output node weights are used to dynamically mask actions on impassable paths before policy execution, forming a corresponding action masking mechanism. The action masking mechanism is used to limit the selectable range of action sets based on node risk scores during the execution phase, thereby achieving the shielding control of high-risk paths.

4. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning trajectory optimization model adopts a dual-Q network structure, including a main Q network and a target Q network. The parameters of the target Q network are updated in accordance with the parameters of the main Q network through a soft update method.

5. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 2, characterized in that: In the process of constructing the deep reinforcement learning trajectory optimization model, graph nodes are encoded into high-dimensional state vectors. The feature vectors include topological location information, perturbation prediction scores, and image detection results. After being encoded by a graph attention network, they are input into a double-Q network for decision calculation. The deep reinforcement learning trajectory optimization model evaluates the trajectory optimization results according to a preset reward function to guide the model's policy update and path selection.

6. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 5, characterized in that: The reward function is obtained by linearly weighting the travel distance penalty, path congestion penalty, timeout penalty, and equipment idle rate reward with adjustable weights. The travel distance penalty is used to limit the trajectory path length, the path congestion penalty reflects the node passage risk or congestion level, the timeout penalty is used to constrain the task execution time, and the equipment idle rate reward is used to improve equipment utilization efficiency. The weight parameters can be adaptively adjusted according to simulation or field operation feedback to achieve a dynamic balance between safety and operation efficiency.

7. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 3, characterized in that: The action masking mechanism dynamically removes path actions with risk scores greater than a set threshold based on node blocking scores in the perturbation weight graph generated by the transformer attention prediction network, in order to ensure the physical feasibility of the trajectory action set.

8. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The trajectory command includes device ID, path number sequence, target node coordinates and execution timestamp. The command format supports compatibility with gantry crane control interface and beam moving vehicle control protocol, and is sent to edge device via 5G network.

9. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The trajectory simulation module constructs a three-dimensional simulation scene based on a digital twin platform. The path execution is visualized in the virtual beam yard in the form of animation, and supports path conflict detection, target switching and manual review.

10. The method for optimizing beam movement trajectory in a predicted beam field based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning trajectory optimization model is deployed on edge computing nodes. During the training phase, the model completes offline learning on the historical task trajectory set. During the execution phase, it receives on-site status data in real time and completes closed-loop operations of trajectory generation, occlusion processing, device driving, and feedback correction.

Images