A tower crane remote control method based on multi-modal fusion and dynamic twin

Abstract

The application provides a tower crane remote control method based on multi-modal fusion and dynamic twinning, belonging to the field of building construction. Its technical scheme is: the method generates a virtual environment model through multi-modal sensor modeling, and constructs a personalized interaction parameter set and a communication baseline; on this basis, an execution and a return package are generated and fed back to a digital twin; combined with a multi-source real-time data set formed by a state monitoring sensor, a dynamic risk prediction is carried out by fusing a physical model and federated learning, a hierarchical intervention strategy is generated, and disposal such as speed limiting, hovering or stopping is executed, and meanwhile, data and instructions are tamper-proofed and stored as evidence. The beneficial effects of the application are: through multi-modal interaction personalization, communication link differentiation, trajectory planning intelligentization, risk prediction foresight and tamper-proofing, the application realizes the transformation of tower crane remote operation from "passive protection" to "active prediction and intelligent intervention", and overall improves the operation efficiency and safety guarantee level under complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of building construction, and more particularly to a method for remote control of tower cranes based on multimodal fusion and dynamic twins. Background Technology

[0002] While tower crane remote control technology has developed in the modern construction field, it still faces numerous bottlenecks. In terms of communication, the complex construction site environment, numerous metal structures, and frequent electromagnetic interference disrupt communication signals. This results in an interruption rate exceeding 12% when using a single communication link (such as 5G), severely impacting the reliable transmission of remote control commands. Regarding operation interaction, traditional control methods based on 2D interfaces and video feedback fail to provide operators with intuitive spatial awareness, and response delays exceed 500ms in emergency situations, making it difficult to meet the demands for efficient and safe operation. In terms of safety protection, existing threshold alarm mechanisms can only respond passively after a risk occurs, failing to anticipate dynamic risks such as load swaying and tower crane structural fatigue. Therefore, this invention provides a tower crane remote control method based on multimodal fusion and dynamic twinning. Summary of the Invention

[0003] The purpose of this invention is to provide a tower crane remote control method based on multimodal fusion and dynamic twins to improve the operation efficiency and safety level under complex working conditions.

[0004] This invention is achieved through the following measures:

[0005] A method for remote control of tower cranes based on multimodal fusion and dynamic twins includes the following steps:

[0006] S1. A virtual environment model is generated through multimodal sensor modeling, and a personalized multimodal interaction parameter set and communication baseline are constructed to provide stable and low-latency assurance for subsequent tower crane task planning and remote control. The modeling process is carried out before tower crane operation. The multimodal sensors include fixed lidar, vision sensors, millimeter-wave radar, and inertial units and short-range ranging sensors at the hook end. The multimodal sensors model the construction site environment and finally obtain a three-dimensional environment model containing high-precision point cloud maps and semantic annotations, which serves as the basic environmental baseline for the digital twin.

[0007] S2. Generate, execute, and send back packets based on the virtual environment model, personalized multimodal interaction parameter set, and communication baseline;

[0008] S3. Based on the execution and feedback packets and the data collected and preprocessed by the state monitoring sensors, a multi-source real-time dataset is formed. Combined with the physical model and federated learning model, dynamic risk prediction is performed, a graded intervention strategy is generated and controlled disposal is executed, and the data and instructions are protected against tampering and stored.

[0009] S1 includes the following steps:

[0010] S11. Collect raw data through a multimodal sensor and perform time synchronization and spatial calibration on the raw data. After preliminary denoising and spatiotemporal alignment, a multimodal dataset is obtained. The multimodal dataset contains a unified spatiotemporal sequence of laser points, image features, radar targets, IMU attitude and near-range ranging data.

[0011] S12. The multimodal dataset is fused from multiple sources to generate a high-precision point cloud map and semantic operation scenario, and synchronized to the digital twin to form a virtual environment model corresponding to the actual construction site. The semantic operation scenario refers to adding semantic labels to the point cloud map, such as obstacles, restricted areas, pedestrian passages, height-restricted areas, tower cranes, storage areas, etc. These semantic labels provide support for subsequent path planning, safety envelope generation and risk prediction.

[0012] S13. In the virtual environment model, the sensitivity, threshold and mapping relationship of gestures and voice are calibrated, and a personalized multimodal interaction parameter set is formed by combining the operator's EEG characteristics. The personalized multimodal interaction parameter set includes individualized parameters and permission policies, which are used for subsequent task input and control superposition.

[0013] S14. Based on the virtual environment model and the personalized multimodal interaction parameter set, 5G, Wi-Fi, data transmission radio, and low-speed links are aggregated. Combining packet replication, forward error correction, and QoS prediction, mapping is completed according to data flow level to obtain a communication baseline that supports differentiated data flows. Specifically, local fast-loop control remains at the edge and does not rely on the public network; remote supervision commands use reinforced links; video uses adaptive bitrate; and telemetry and evidence storage use low-speed reliable links, providing stable, low-latency, and available communication guarantees for subsequent task planning and remote control execution.

[0014] S2 includes the following steps:

[0015] S21. Based on the virtual environment model, combined with the personalized multimodal interaction parameter set, the instruction structuring is completed to obtain the structured task description package;

[0016] Specifically, the process involves parsing and converting the operator's natural interactive input (voice, gestures, EEG intentions, etc.) into a structured task description package. This description package is encapsulated using a unified data structure, facilitating subsequent path planning, risk assessment, and task scheduling by the digital twin.

[0017] The structured task description package includes at least the following fields:

[0018] Target pose sequence: Represents a series of key spatial positions and attitudes that the tower crane hook or suspended object needs to reach, represented by three-dimensional coordinates and orientation angles. Each point can be accompanied by a timestamp or sequence number.

[0019] Target object attributes: Used to describe the basic characteristics of the object being lifted, including size, weight, material type, and whether it has special constraints (such as fragile, long, or requiring horizontal holding).

[0020] Priority: Indicates the relative importance of the task in the current job plan, expressed as an integer or level to support multi-task scheduling and conflict resolution.

[0021] Deadline requirement: This indicates the expected time window for the task to be completed, including the start time, end time, or duration requirement, and is used as a time constraint in the scheduling strategy.

[0022] Work Area / Restricted Area Reference: By indexing semantic regions in the virtual environment model, the target work area that the hoisted object should enter is clearly specified, as well as restricted areas, height-restricted areas, etc., that are prohibited from being passed through or approached.

[0023] Allowed control mode set: List the control modes applicable to this task, such as "fully automatic planning and execution", "semi-automatic execution + operator fine-tuning", and "purely manual auxiliary control", so that the edge control node can switch in different scenarios.

[0024] Security policy reference: Points to the set of security envelope parameters within the digital twin, including minimum safe distance, buffer coefficient, and rate limit parameters, which are used for invocation during the path planning and execution phases.

[0025] Furthermore, the structured task description package is also bound to a personalized multimodal interaction parameter set. Specifically, this includes:

[0026] Operator ID: A unique identifier that identifies the operator from whom the task originated;

[0027] Permission level: Calculated by the permission policy in the personalized parameter set, used to restrict whether high-risk actions (such as near-obstacle operation, high-speed hoisting) are allowed in the task description package.

[0028] Interactive parameter references: These are associated with the operator's individualized sensitivity, thresholds, and mapping relationships, and are used to parse the operator's real-time fine-tuning instructions during task execution.

[0029] S22. Align the real-time working condition data continuously collected and updated by the multimodal sensors in step S11 to the digital twin during the operation with the virtual environment model to form the current working condition situation including dynamic obstacles and safety constraint sets.

[0030] S23. After receiving the structured task description package and the current working condition, the digital twin performs constrained path planning and obstacle avoidance calculation based on the point cloud map and semantic operation scenario, and performs dynamic constraint correction in combination with the load swing model and wind load influence, thereby generating a reference trajectory package for the tower crane edge control node to call and execute. The reference trajectory package includes a time-parameterized reference trajectory, a safety envelope, a control mode, and a swing suppression strategy reference trajectory package.

[0031] S23 includes the following steps:

[0032] S231. The tower crane edge control node receives the reference trajectory packet and tracks the time-parameterized reference trajectory in the local high-speed closed-loop control to obtain trajectory tracking status information, while performing swing suppression and emergency stop protection.

[0033] S232. During execution, the tower crane edge control node analyzes the operator's gesture input based on the trajectory tracking status information and the personalized multimodal interaction parameter set, and generates corresponding fine-tuning amounts. Then, the fine-tuning amounts are superimposed within the safety envelope constraint range corresponding to the reference trajectory package to form a control instruction set after superimposed gesture fine-tuning, thereby improving operational flexibility while ensuring operational safety.

[0034] S233. At the data transmission level, based on the communication baseline, the control command set, path command, real-time video stream, and telemetry data are mapped to the corresponding link channels respectively, completing multi-link aggregation, packet replication, and error correction processing to form a set of real-time data streams after multi-link transmission, so as to achieve low-latency and high-reliability transmission. The path command comes from the reference trajectory packet; the real-time video stream comes from real-time acquisition devices such as vision sensors at the construction site, including vision sensors, panoramic / zoom cameras, etc.; the telemetry data comes from the operation monitoring and status monitoring sensors of the tower crane body and is collected in real time by the edge control node.

[0035] S234. The tower crane control method renders a three-dimensional visualization scene based on the real-time data stream set in the operator interface, and provides safety prompts related to the operating status. Finally, it generates an execution and feedback package, which includes trajectory tracking error, equipment operating status, telemetry data, video stream, and alarm prompts.

[0036] The tower crane control method includes tower crane edge control nodes, communication baselines, multimodal interaction modules, and operator interfaces.

[0037] S24. The tower crane control method performs high-speed closed-loop tracking of the reference trajectory package through the tower crane edge control node and executes swing suppression and emergency stop protection. It combines personalized multimodal interactive parameter set to parse operator gestures and generate superimposed control command set, completes multi-link transmission through communication baseline, and renders a three-dimensional visualization scene on the operator terminal interface and generates execution and feedback packets to the digital twin. The execution and feedback packets are fed back to the digital twin for continuous updating of the operation status and supporting necessary online replanning.

[0038] S3 includes the following steps:

[0039] S31. Real-time status data related to the tower crane is collected through condition monitoring sensors, and time alignment and preprocessing are performed to form a multi-source real-time dataset (including tower crane structural stress, fatigue, and environmental parameters) for dynamic risk prediction. The real-time status data includes tower crane structural stress, fatigue, and environmental parameters. The condition monitoring sensors include structural monitoring sensors for measuring tower crane structural stress and fatigue, and environmental monitoring sensors for measuring environmental parameters. The structural monitoring sensors include stress sensors, strain gauges, accelerometers, vibration sensors, and fatigue monitoring modules. The environmental monitoring sensors include anemometers, temperature and humidity sensors, and environmental weather stations.

[0040] S32. Based on multi-source real-time datasets, dynamic risk prediction is performed by combining physical models and federated learning models. When structural anomalies or load risks are predicted, virtual pre-simulation is performed in the digital twin to derive corresponding hierarchical intervention strategies. The hierarchical intervention strategies include controlling the speed limit, hovering, or stopping the tower crane.

[0041] S33. Control the tower crane according to the graded intervention strategy (speed limit, hovering or stopping); at the same time, generate hashes periodically for multi-source sensor data and control commands and store them in the anti-tampering evidence storage module, and establish an index binding with the original video and sensor data to support post-event evidence collection. Evidence and records of the original video and sensor data can be quickly retrieved when an accident occurs for operation safety protection and responsibility tracing.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention not only achieves breakthroughs in the real-time performance, reliability, and safety of remote tower crane control, but also realizes the transformation of remote tower crane operation from "passive protection" to "active prediction and intelligent intervention" through multimodal interactive personalization, communication link differentiation, intelligent trajectory planning, forward-looking risk prediction, and tamper-proof evidence storage, thereby improving the overall efficiency and safety level of operation under complex working conditions. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0045] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0046] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0048] Example 1

[0049] A method for remote control of tower cranes based on multimodal fusion and dynamic twins includes the following steps:

[0050] S1. A virtual environment model is generated through multimodal sensor modeling, and a personalized multimodal interaction parameter set and communication baseline are constructed to provide stable and low-latency assurance for subsequent tower crane task planning and remote control. The modeling process is carried out before tower crane operation. The multimodal sensors include fixed lidar, vision sensors, millimeter-wave radar, and inertial units and short-range ranging sensors at the hook end. The multimodal sensors model the construction site environment and finally obtain a three-dimensional environment model containing high-precision point cloud maps and semantic annotations, which serves as the basic environmental baseline for the digital twin.

[0051] S1 includes the following steps:

[0052] S11. Collect raw data through multimodal sensors and perform time synchronization and spatial calibration on the raw data. After preliminary denoising and spatiotemporal alignment, a multimodal dataset is obtained. The multimodal dataset contains a unified spatiotemporal sequence of laser points, image features, radar targets, IMU attitude and near-range ranging data.

[0053] S12. Multimodal datasets are fused from multiple sources to generate high-precision point cloud maps and semantic operation scenarios, and synchronized to a digital twin to form a virtual environment model corresponding to the actual construction site. Semantic operation scenarios refer to adding semantic labels to the point cloud map, such as obstacles, restricted areas, pedestrian passages, height-restricted areas, tower cranes, storage areas, etc. These semantic labels provide support for subsequent path planning, safety envelope generation, and risk prediction.

[0054] S13. In the virtual environment model, complete the calibration of the sensitivity, threshold and mapping relationship of gestures and speech, and combine the operator's EEG characteristics to form a personalized multimodal interaction parameter set. The personalized multimodal interaction parameter set includes individualized parameters and permission policies, which are used for subsequent task input and control superposition.

[0055] S14. Based on the virtual environment model and personalized multimodal interaction parameter set, 5G, Wi-Fi, data transmission radio and low-speed links are aggregated. Combining packet replication, forward error correction and QoS prediction, mapping is completed according to data flow level to obtain a communication baseline that supports differentiated data flows. Specifically, local fast loop control stays at the edge and does not rely on the public network; remote supervision commands go through the hardened link; video goes through the adaptive bitrate; telemetry and evidence storage go through the low-speed reliable link, providing stable, low-latency and availability communication guarantees for subsequent task planning and remote control execution.

[0056] S2, Based on virtual environment model, personalized multimodal interaction parameter set and communication baseline, generate, execute and return packets;

[0057] S2 includes the following steps:

[0058] S21. Based on the virtual environment model, combined with the personalized multimodal interaction parameter set, the instruction structuring is completed to obtain the structured task description package;

[0059] S22. Align the real-time working condition data continuously collected and updated by the multimodal sensors in step S11 to the digital twin during the operation with the virtual environment model to form the current working condition situation including dynamic obstacles and safety constraint sets.

[0060] S23. After receiving the structured task description package and the current working situation, the digital twin performs constrained path planning and obstacle avoidance calculation based on the point cloud map and semantic operation scenario, and performs dynamic constraint correction in combination with the load swing model and wind load influence, thereby generating a reference trajectory package for the tower crane edge control node to call and execute. The reference trajectory package includes a time parameterized reference trajectory, safety envelope, control mode and swing suppression strategy reference trajectory package.

[0061] S23 includes the following steps:

[0062] S231. The tower crane edge control node receives the reference trajectory packet and tracks the time-parameterized reference trajectory in the local high-speed closed-loop control to obtain trajectory tracking status information, while performing swing suppression and emergency stop protection.

[0063] S232. During execution, the tower crane edge control node analyzes the operator's gesture input based on trajectory tracking status information and combined with a personalized multimodal interaction parameter set, and generates corresponding fine-tuning amounts. Then, the fine-tuning amounts are superimposed within the safety envelope constraint range corresponding to the reference trajectory package to form a control instruction set after superimposed gesture fine-tuning, thereby improving operational flexibility while ensuring operational safety.

[0064] S233. At the data transmission level, based on the communication baseline, the control command set, path command, real-time video stream, and telemetry data are mapped to the corresponding link channels respectively. This completes multi-link aggregation, packet replication, and error correction processing, forming a set of real-time data streams after multi-link transmission. This achieves low-latency and high-reliability transmission. The path command originates from the reference trajectory packet; the real-time video stream comes from real-time acquisition devices such as vision sensors and panoramic / zoom cameras at the construction site; and the telemetry data comes from the tower crane's operation monitoring and status monitoring sensors, which are collected in real-time by the edge control nodes.

[0065] S234. The tower crane control method renders a three-dimensional visualization scene based on a real-time data stream set in the operator interface, and provides safety prompts related to the operating status. Finally, it generates an execution and feedback packet, which includes trajectory tracking error, equipment operating status, telemetry data, video stream, and alarm prompts.

[0066] The tower crane control method includes tower crane edge control nodes, communication baselines, multimodal interaction modules, and operator interfaces.

[0067] S24. The tower crane control method performs high-speed closed-loop tracking of the reference trajectory package through the tower crane edge control node and executes swing suppression and emergency stop protection. It combines personalized multimodal interactive parameter set to parse operator gestures and generate superimposed control instruction set. Multi-link transmission is completed through the communication baseline. The operator interface renders a three-dimensional visualization scene and generates execution and feedback packets to the digital twin. The execution and feedback packets are fed back to the digital twin for continuous updating of the operation status and supporting necessary online replanning.

[0068] S3. Based on the execution and feedback packets and the data collected and preprocessed by the status monitoring sensors, a multi-source real-time dataset is formed. Combined with the physical model and federated learning model, dynamic risk prediction is performed, a graded intervention strategy is generated and controlled disposal is executed, and the data and instructions are protected against tampering and stored.

[0069] S3 includes the following steps:

[0070] S31. Real-time status data related to the tower crane is collected through condition monitoring sensors, and time alignment and preprocessing are performed to form a multi-source real-time dataset (including tower crane structural stress, fatigue, and environmental parameters) for dynamic risk prediction. The real-time status data includes tower crane structural stress, fatigue, and environmental parameters. The condition monitoring sensors include structural monitoring sensors for measuring tower crane structural stress and fatigue, and environmental monitoring sensors for measuring environmental parameters. Structural monitoring sensors include stress sensors, strain gauges, accelerometers, vibration sensors, and fatigue monitoring modules. Environmental monitoring sensors include anemometers, temperature and humidity sensors, and environmental weather stations.

[0071] S32. Based on multi-source real-time datasets, dynamic risk prediction is performed by combining physical models and federated learning models. When structural anomalies or load risks are predicted, virtual pre-simulation is performed in the digital twin to derive corresponding graded intervention strategies. The graded intervention strategies include controlling the speed limit, hovering, or stopping the tower crane.

[0072] S33. Control the tower crane according to the graded intervention strategy (speed limit, hovering or stopping); at the same time, generate hashes periodically for multi-source sensor data and control commands and store them in the anti-tampering evidence storage module, and establish an index binding with the original video and sensor data to support post-event evidence collection. Evidence and records of the original video and sensor data can be quickly retrieved when an accident occurs for operation safety protection and responsibility tracing.

[0073] Example 2

[0074] A method for remote control of tower cranes based on multimodal fusion and dynamic twins includes the following steps:

[0075] S1. A virtual environment model is generated through multimodal sensor modeling, and a personalized multimodal interaction parameter set and communication baseline are constructed to provide stable and low-latency assurance for subsequent tower crane task planning and remote control. The modeling process is carried out before tower crane operation. The multimodal sensors include fixed lidar, vision sensors, millimeter-wave radar, and inertial units and short-range ranging sensors at the hook end. The multimodal sensors model the construction site environment and finally obtain a three-dimensional environment model containing high-precision point cloud maps and semantic annotations, which serves as the basic environmental baseline for the digital twin.

[0076] S1 includes the following steps:

[0077] S11. Collect raw data through multimodal sensors and perform time synchronization and spatial calibration on the raw data. After preliminary denoising and spatiotemporal alignment, a multimodal dataset is obtained. The multimodal dataset contains a unified spatiotemporal sequence of laser points, image features, radar targets, IMU attitude and near-range ranging data.

[0078] Specifically, (1) the data collection object and coordinate / time agreement:

[0079] The sensors used include a fixed LiDAR, a vision sensor (camera), a millimeter-wave radar, a hook-mounted IMU, and a near-field ranging sensor (ToF / ultrasonic). The outputs of each sensor are: point cloud P, image / feature. The radar target is R, its attitude is Q, and its range is D; the coordinate system used is the world system. and various sensor systems The time is agreed to be unified with the PTP master clock timeline. ;

[0080] (2) Time synchronization and timing resampling:

[0081] For each type of sensor Original measurement Perform time calibration and interpolation to compensate for its fixed / measurable delay. and on the public timeline Upsampling is:

[0082]

[0083] Finally, the aligned multimodal time series is obtained. This is to prepare for subsequent spatial calibration and integration;

[0084] in, Align to public timestamp Measurement; Indicates linear / spline interpolation or trigger-based nearest neighbor preservation;

[0085] Transmission-processing delay estimate obtained through time synchronization (PTP / Unified Timestamp) and cable / encoding delay calibration.

[0086] (3) Spatial calibration: Intrinsic / extrinsic parameters and coordinate unification:

[0087] Any sensor coordinate system The point (or target center) in the middle. Transform to World System :

[0088]

[0089] No. Sensor to the external parameters of the world system;

[0090] The extrinsic parameters are solved by minimizing the multi-source corresponding point error:

[0091]

[0092] Ultimately, all sensor data are mapped to a unified coordinate system. This lays the foundation for the integration of point cloud maps and semantic operation scenarios.

[0093] (4) Preliminary denoising and spatiotemporal alignment for library creation:

[0094] LiDAR: Voxel Mesh Downsampling (Voxel Side Length) ), statistical outlier removal (mean μ, standard deviation threshold κ);

[0095] Camera: Exposure / denoising and feature extraction (e.g., ORB / SIFT), preservation Key points and descriptors.

[0096] Radar: Constant False Alarm Rate (CFAR) Extraction Objective: To filter out false alarms using multipath thresholding.

[0097] IMU: Zero-bias estimation and gravity-constrained initial alignment; close-range ranging Perform sliding median filtering to combat spikes.

[0098] Combined with (2) and (3), a unified spatiotemporal sequence is formed, resulting in a multimodal dataset:

[0099] .

[0100] S12. Multimodal datasets are fused from multiple sources to generate high-precision point cloud maps and semantic operation scenarios, and synchronized to a digital twin to form a virtual environment model corresponding to the actual construction site. Semantic operation scenarios refer to adding semantic labels to the point cloud map, such as obstacles, restricted areas, pedestrian passages, height-restricted areas, tower cranes, storage areas, etc. These semantic labels provide support for subsequent path planning, safety envelope generation, and risk prediction.

[0101] S13. In the virtual environment model, complete the calibration of the sensitivity, threshold and mapping relationship of gestures and speech, and combine the operator's EEG characteristics to form a personalized multimodal interaction parameter set. The personalized multimodal interaction parameter set includes individualized parameters and permission policies, which are used for subsequent task input and control superposition.

[0102] Specifically, (1) the calibration stage: feature normalization and individual sensitivity:

[0103] Let Γ be the raw quantity of gesture intensity (obtained by scalarization from hand key point displacement, grip strength, or arm acceleration), Λ be the raw quantity of speech intent (obtained from keyword-triggered recognition confidence or acoustic energy ratio), and Θ be the raw quantity of EEG attention / stability (obtained by indexing from power spectrum / time-frequency features, such as a monotonic mapping of the θ / β ratio). During calibration, the individual thresholds (lower and upper limits) of each channel are recorded and denoted as: gesture thresholds. Speech threshold EEG threshold Based on this, the normalization index is obtained:

[0104]

[0105] Where α, β, γ∈[0,1], representing normalized gesture intensity, speech confidence, and EEG attention / stability, respectively.

[0106] (2) Multimodal intent fusion and output mapping:

[0107] Define individual weights (The relative weight of gestures / voice on the operator). Let ε be the amplification factor of EEG on intention enhancement, and let ε be the environmental risk index (derived from the semantic scene of S12, such as the constraint distance comprehensive score of obstacles / restrictions / height restrictions in pose and trajectory). Construct a concise fusion score J and control amplitude M (saturation mapping):

[0108]

[0109] in: χ is the risk suppression coefficient; χ is the output shape coefficient (which determines the response speed / saturation level).

[0110] Trigger threshold τ: When J < τ, no control superposition is generated (anti-false triggering); otherwise, the output M is superimposed on the reference trajectory (corresponding to the "superposition control instruction set" of S23 / S24).

[0111] (3) Personalized "permission policy" linkage:

[0112] Define the operational fatigue / load index δ (derived from physiological signals or operation duration), taking values ​​from 0 to 1;

[0113] Define permission score A:

[0114]

[0115] Where: ρ, σ, and υ are the policy weights;

[0116] Since A is a continuous score, its value usually ranges from 0 to 1. However, this continuous value alone is insufficient to directly determine the operation permission level. Therefore, a personalized threshold ξ is introduced to map the continuous value of A to a discrete permission level:

[0117] When A ≥ ξ, the system determines that the operator is in good condition and allows high-privilege operations (such as higher M limit, faster response, near-obstacle operation, etc.).

[0118] When A < ξ, the system automatically downgrades to a low-privilege mode (such as limiting the upper limit of M, limiting the speed, requiring secondary confirmation, or pausing).

[0119] The above parameters Interactive calibration is performed in the "virtual environment model" and persisted as a personalized multimodal interaction parameter set; during task execution, ε, δ, and γ can be updated over time within the digital twin, thereby forming online adaptation.

[0120] S14. Based on the virtual environment model and personalized multimodal interaction parameter set, 5G, Wi-Fi, data transmission radio and low-speed links are aggregated. Combining packet replication, forward error correction and QoS prediction, mapping is completed according to data flow level to obtain a communication baseline that supports differentiated data flows. Specifically, local fast loop control stays at the edge and does not rely on the public network; remote supervision commands go through the hardened link; video goes through the adaptive bitrate; telemetry and evidence storage go through the low-speed reliable link, providing stable, low-latency and availability communication guarantees for subsequent task planning and remote control execution.

[0121] Specifically, let the success probability of the k-th link be... (Probability of successful arrival of a single packet per unit time, 0–1), one-way delay is ;

[0122] Let the forward error correction redundancy factor be θ (>0 indicates more redundancy), and the number of concurrent replications be η (select η links for concurrent transmission);

[0123] The improvement in single-link success rate brought about by FEC can be simplified to:

[0124]

[0125] (The more redundancy, the higher the equivalent success rate; this is for engineering approximation and facilitates online calculation.)

[0126] After concurrent replication, the success rate of the combination is:

[0127]

[0128] To balance low latency and high reliability, a set is selected for "supervision command / remote control" type data. Make the objective function:

[0129]

[0130] Achieve maximum (intuitively equivalent to prioritizing "reliable / fast" links).

[0131] Mapping rule suggestions:

[0132] Local fast loop control: without going through the public network, it directly closes the loop at the edge (ensuring ω is minimized), corresponding to the original text "local fast loop control stays at the edge and does not depend on the public network".

[0133] Supervision instructions: Use the set of hardened links selected by the above formula. (small ω, large) ), concurrent replication (η≥2);

[0134] Video: Press Adaptive bitrate allocation based on instantaneous bandwidth;

[0135] Telemetry / Evidence Preservation: Retrieval For the largest but lowest bandwidth-required link (allowing for a larger ω), ensure "reachability".

[0136] S2, Based on virtual environment model, personalized multimodal interaction parameter set and communication baseline, generate, execute and return packets;

[0137] S2 includes the following steps:

[0138] S21. Based on the virtual environment model, combined with the personalized multimodal interaction parameter set, the instruction structuring is completed to obtain the structured task description package;

[0139] Specifically, the process involves parsing and converting the operator's natural interactive input (voice, gestures, EEG intentions, etc.) into a structured task description package. This description package is encapsulated using a unified data structure, facilitating subsequent path planning, risk assessment, and task scheduling by the digital twin.

[0140] A structured task description package should include at least the following fields:

[0141] Target pose sequence: Represents a series of key spatial positions and attitudes that the tower crane hook or suspended object needs to reach, represented by three-dimensional coordinates and orientation angles. Each point can be accompanied by a timestamp or sequence number.

[0142] Target object attributes: Used to describe the basic characteristics of the object being lifted, including size, weight, material type, and whether it has special constraints (such as fragile, long, or requiring horizontal holding).

[0143] Priority: Indicates the relative importance of the task in the current job plan, expressed as an integer or level to support multi-task scheduling and conflict resolution.

[0144] Deadline requirement: This indicates the expected time window for the task to be completed, including the start time, end time, or duration requirement, and is used as a time constraint in the scheduling strategy.

[0145] Work Area / Restricted Area Reference: By indexing semantic regions in the virtual environment model, the target work area that the hoisted object should enter is clearly specified, as well as restricted areas, height-restricted areas, etc., that are prohibited from being passed through or approached.

[0146] Allowed control mode set: List the control modes applicable to this task, such as "fully automatic planning and execution", "semi-automatic execution + operator fine-tuning", and "purely manual auxiliary control", so that the edge control node can switch in different scenarios.

[0147] Security policy reference: Points to the set of security envelope parameters within the digital twin, including minimum safe distance, buffer coefficient, and rate limit parameters, which are used for invocation during the path planning and execution phases.

[0148] Furthermore, the structured task description package is also bound to a personalized multimodal interaction parameter set. Specifically, this includes:

[0149] Operator ID: A unique identifier that identifies the operator from whom the task originated;

[0150] Permission level: Calculated by the permission policy in the personalized parameter set, used to restrict whether high-risk actions (such as near-obstacle operation, high-speed hoisting) are allowed in the task description package.

[0151] Interactive parameter references: These are associated with the operator's individualized sensitivity, thresholds, and mapping relationships, and are used to parse the operator's real-time fine-tuning instructions during task execution.

[0152] S22. Align the real-time working condition data continuously collected and updated by the multimodal sensors in step S11 to the digital twin during the operation with the virtual environment model to form the current working condition situation including dynamic obstacles and safety constraint sets.

[0153] Specifically, real-time operating data includes lidar point clouds, RGB-D images, inertial measurement units, anemometers, torque sensors, hook encoders, GPS / RTK positioning data, etc.; first, the real-time operating data is spatiotemporally aligned, and then it is updated into the virtual environment model of the digital twin.

[0154] Spatiotemporal alignment methods:

[0155] First, timestamp unification: A unified high-precision clock source (such as PTP Precision Time Protocol or GNSS time synchronization) is used to timestamp the output of each sensor; in the virtual environment model, resampling and interpolation are performed with a fixed step size (such as 50 ms) to ensure that different data sources have consistent observation status at the same time.

[0156] Next, coordinate registration: the coordinate systems of the LiDAR, camera, and tower crane are rigidly registered through offline calibration (external parameter matrix); real-time data is first converted to the tower crane coordinate system, and then uniformly mapped to the global coordinate system of the construction site (the world system defined by GPS / RTK); the digital twin maintains a strict one-to-one correspondence with the world system to ensure that the position of objects in the virtual environment is consistent with that on site.

[0157] Finally, the dynamic update cycle is as follows: the update frequency for the status of obstacles (such as vehicles and people) is 10–20 Hz; the update frequency for meteorological data such as wind speed and direction is 1–2 Hz; the update frequency for the position and stress status of the hook is 50–100 Hz; data of different frequencies are asynchronously sampled, buffered and fused in a unified manner, and then refreshed to the virtual environment at a cycle of 20 Hz.

[0158] The current operating status includes the following main fields:

[0159] Dynamic obstacle set O(t): records obstacle ID, type (personnel, vehicle, component), position vector, velocity vector, and predicted motion trajectory;

[0160] Safety constraint set S(t): includes obstacle safety radius, restricted area boundary, tower arm safety interaction zone during multi-tower operation, and height restriction surface;

[0161] Wind field estimation W(t): Wind speed magnitude, wind direction vector, and eddy intensity obtained by combining wind speed sensors at the tower top / hook with the environmental model;

[0162] Equipment status E(t): includes trolley position, hook height, load weight, torque margin, structural strain, etc.

[0163] The above fields are stored in the digital twin database in JSON / XML structured data format, with version numbers and timestamps to ensure traceability.

[0164] Data fusion and consistency check:

[0165] For dynamic obstacles, multi-sensor fusion (laser point cloud + video detection + RFID / Bluetooth tag) is used, and their state is estimated by Kalman filtering or particle filtering;

[0166] Short-time moving averages and outlier removal are introduced into wind field estimation to ensure that it is not affected by instantaneous spikes.

[0167] The equipment status is cross-verified using redundant sensors; for example, the hook height is compared between the encoder and the RTK dual-channel.

[0168] Output to the virtual environment model:

[0169] Digital twins are dynamically rendered in a virtual environment:

[0170] Project O(t) onto the point cloud map in real time to display the movement of people and vehicles;

[0171] Render S(t) as a semi-transparent volume region (red for no entry, yellow for warning);

[0172] Visualize W(t) as a vector arrow or field texture;

[0173] Display E(t) in the status panel (load weight, remaining torque margin).

[0174] This virtual environment model thus forms a digital mirror that corresponds one-to-one with the actual working conditions, providing complete situational input for subsequent path planning, risk prediction and control execution.

[0175] S23. After receiving the structured task description package and the current working situation, the digital twin performs constrained path planning and obstacle avoidance calculation based on the point cloud map and semantic operation scenario, and performs dynamic constraint correction in combination with the load swing model and wind load influence, thereby generating a reference trajectory package for the tower crane edge control node to call and execute. The reference trajectory package includes a time parameterized reference trajectory, safety envelope, control mode and swing suppression strategy reference trajectory package.

[0176] S23 includes the following steps:

[0177] S231. The tower crane edge control node receives the reference trajectory packet and tracks the time-parameterized reference trajectory in the local high-speed closed-loop control to obtain trajectory tracking status information, while performing swing suppression and emergency stop protection.

[0178] Specifically, the reference trajectory package is generated by the following optimization model:

[0179]

[0180] in, The spatial trajectory of the tower crane hook; For load swing angle, Its first and second derivatives; The acceleration command is given to the vehicle. For trajectory jump; To optimize the weights, corresponding to trajectory smoothness, swing angle constraint, and control energy consumption, respectively.

[0181] The load swing under wind load satisfies a simplified dynamic model:

[0182]

[0183] Where b>0 is the equivalent damping, c>0 is the equivalent stiffness, d>0 is the trolley acceleration coupling coefficient, e>0 is the wind load coupling coefficient, and v is the projection of the crosswind velocity onto the oscillating plane.

[0184] The system can adaptively adjust when operating conditions change (e.g., increased wind speed or the appearance of new obstacles). and safety radius This enables dynamic correction of the reference trajectory packet. In the high-speed closed-loop control of the edge control node, a cooperative control law is used to jointly suppress trajectory and oscillation:

[0185]

[0186] in, For trajectory error, Let n be the velocity error, and n be the unit vector of the principal direction of the oscillation. For position, velocity, swing angle, and swing speed feedback gain. When obstacle distance is detected. Upon that time, the emergency stop protection is immediately triggered, the acceleration is reduced to zero, and braking occurs. Corresponding safety radius.

[0187] S232. During execution, the tower crane edge control node analyzes the operator's gesture input based on trajectory tracking status information and combined with a personalized multimodal interaction parameter set, and generates corresponding fine-tuning amounts. Then, the fine-tuning amounts are superimposed within the safety envelope constraint range corresponding to the reference trajectory package to form a control instruction set after superimposed gesture fine-tuning, thereby improving operational flexibility while ensuring operational safety.

[0188] Specifically, the operator's gesture input is mapped to a fine-tuning amount via a personalized parameter set. and through the envelope projection operator The superposition rule is as follows, limited to the safe envelope of the reference trajectory packet:

[0189]

[0190] in, To overlay the finely adjusted command trajectory For reference trajectory, This indicates that the fine-tuning amount is projected with an envelope constraint to ensure that the command does not go out of bounds. This design ensures improved operational flexibility while maintaining operational safety.

[0191] S233. At the data transmission level, based on the communication baseline, the control command set, path command, real-time video stream, and telemetry data are mapped to the corresponding link channels respectively. This completes multi-link aggregation, packet replication, and error correction processing, forming a set of real-time data streams after multi-link transmission. This achieves low-latency and high-reliability transmission. The path command originates from the reference trajectory packet; the real-time video stream comes from real-time acquisition devices such as vision sensors and panoramic / zoom cameras at the construction site; and the telemetry data comes from the tower crane's operation monitoring and status monitoring sensors, which are collected in real-time by the edge control nodes.

[0192] S234. The tower crane control method renders a three-dimensional visualization scene based on a real-time data stream set in the operator interface, and provides safety prompts related to the operating status. Finally, it generates an execution and feedback packet, which includes trajectory tracking error, equipment operating status, telemetry data, video stream, and alarm prompts.

[0193] The tower crane control method includes tower crane edge control nodes, communication baselines, multimodal interaction modules, and operator interfaces.

[0194] S24. The tower crane control method performs high-speed closed-loop tracking of the reference trajectory package through the tower crane edge control node and executes swing suppression and emergency stop protection. It combines personalized multimodal interactive parameter set to parse operator gestures and generate superimposed control instruction set. Multi-link transmission is completed through the communication baseline. The operator interface renders a three-dimensional visualization scene and generates execution and feedback packets to the digital twin. The execution and feedback packets are fed back to the digital twin for continuous updating of the operation status and supporting necessary online replanning.

[0195] Specifically, in high-speed closed-loop control, the edge control node uses a cooperative control law to jointly adjust the position and swing angle based on trajectory error and oscillation state:

[0196]

[0197] in:

[0198] : Acceleration control quantity for position-oscillation coordination; For trajectory error; For trajectory velocity error; : Load swing angle and its angular velocity; n: Unit vector of the principal swing direction (determined by trajectory direction or wind direction); Feedback gain parameter.

[0199] The generated command trajectory and control instruction set are transmitted through the communication baseline. Using mechanisms such as multi-link aggregation, packet replication and error correction, control instructions, path instructions, video streams and telemetry data are mapped to the corresponding link channels to ensure low latency and high reliability.

[0200] On the user interface, the system renders a 3D visualization scene based on a real-time data stream and generates execution and feedback packets, which include: trajectory tracking error. Swing angle state The system transmits equipment operating status (speed, acceleration, load weight, etc.), telemetry data, video streams, and alarm prompts such as emergency stop or safety envelope triggering. This feedback packet is sent back to the digital twin in real time to continuously update the operational status and support necessary online reconfiguration.

[0201] S3. Based on the execution and feedback packets and the data collected and preprocessed by the status monitoring sensors, a multi-source real-time dataset is formed. Combined with the physical model and federated learning model, dynamic risk prediction is performed, a graded intervention strategy is generated and controlled disposal is executed, and the data and instructions are protected against tampering and stored.

[0202] S3 includes the following steps:

[0203] S31. Real-time status data related to the tower crane is collected through condition monitoring sensors, and time alignment and preprocessing are performed to form a multi-source real-time dataset (including tower crane structural stress, fatigue, and environmental parameters) for dynamic risk prediction. The real-time status data includes tower crane structural stress, fatigue, and environmental parameters. The condition monitoring sensors include structural monitoring sensors for measuring tower crane structural stress and fatigue, and environmental monitoring sensors for measuring environmental parameters. Structural monitoring sensors include stress sensors, strain gauges, accelerometers, vibration sensors, and fatigue monitoring modules. Environmental monitoring sensors include anemometers, temperature and humidity sensors, and environmental weather stations.

[0204] S32. Based on multi-source real-time datasets, dynamic risk prediction is performed by combining physical models and federated learning models. When structural anomalies or load risks are predicted, virtual pre-simulation is performed in the digital twin to derive corresponding graded intervention strategies. The graded intervention strategies include controlling the speed limit, hovering, or stopping the tower crane.

[0205] Specifically,

[0206] (1) Dual-domain fusion of physical domain risk (Physics) and data domain risk (Federated):

[0207] (a) Physical Domain Risk Indicators (Simplified):

[0208] Normalized features (truncated to [0,1]) are constructed from the multi-source real-time data of S31:

[0209] : Structural stress utilization rate (e.g., "Measured stress / Allowable stress" clipped to [0,1]);

[0210] : Normalized value of vibration / acceleration intensity (e.g., "rms / limit");

[0211] : Normalized wind speed value (e.g., "wind speed / wind limit").

[0212] A concise physical domain risk score B is given (linearly interpretable, easy to calculate online):

[0213]

[0214] B∈[0,1] (the sum can be truncated during implementation); To allow for calibrated weights, reflecting the relative contributions of stress, vibration, and wind to risk.

[0215] (b) Data domain risks (federated learning):

[0216] Each edge node j (different tower cranes / different construction sites) is trained / inferred locally using private data to output the local risk probability. The center does not aggregate the raw data, but only aggregates a weighted average of probabilities or gradients:

[0217]

[0218] F∈[0,1]; Node weights (which can be set based on sample quality, model confidence, or device health).

[0219] (c) The total risk H is obtained from the fusion of two domains:

[0220]

[0221] H∈[0,1]; A reliable allocation that reflects "physical mechanism vs. data experience" can be adaptively fine-tuned according to operating conditions (e.g., increasing when the wind is strong). Increase when data offset )

[0222] (2) Virtual pre-playing and hierarchical intervention mapping in digital twins

[0223] Scrolling previews of short future time windows (e.g., 10–30 s) are performed in the digital twin:

[0224] Using the reference trajectory and current load / wind field estimates as conditions, calculate the physical domain risk prediction. (Based on load / wind load model and speed limit assumptions); data domain risk prediction is obtained from short-term forecasts using the federated model (without uploading raw data). The predicted risks are derived as follows:

[0225]

[0226] Will The intervention level N is automatically generated by comparing it with the grading threshold.

[0227]

[0228] The policy threshold (e.g., 0.4 / 0.6 / 0.8, which can be calibrated according to site safety regulations).

[0229] N is directly mapped to control layer actions (speed limit / hover / stop) and written back to the execution policy and UI alerts.

[0230] S33. Control the tower crane according to the graded intervention strategy (speed limit, hovering or stopping); at the same time, generate hashes periodically for multi-source sensor data and control commands and store them in the anti-tampering evidence storage module, and establish an index binding with the original video and sensor data to support post-event evidence collection. Evidence and records of the original video and sensor data can be quickly retrieved when an accident occurs for operation safety protection and responsibility tracing.

[0231] Furthermore, speed limit / hover / stop commands are issued according to N=1 / 2 / 3 respectively, and the "trigger time, threshold, H / B / F value" is recorded in the return packet;

[0232] Furthermore, the "multi-source sensor data + control commands + intervention level N" are concatenated into a byte stream and hashed (such as SHA-256) at fixed intervals. The hash value, along with the video frame timestamp and the original file index, is written into an append-only ledger (WORM medium / blockchain sub-chain) to achieve immutability and traceability.

[0233] The technical features of this invention not described can be implemented by or using existing technology, and will not be repeated here. Of course, the above description is not a limitation of this invention, and this invention is not limited to the examples above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention should also be within the protection scope of this invention.

Claims

1. A method for remote control of tower cranes based on multimodal fusion and dynamic twins, characterized in that, Includes the following steps; S1. A virtual environment model is generated through multimodal sensor modeling, and a personalized multimodal interaction parameter set and communication baseline are constructed. Specifically: raw data is collected through multimodal sensors, and the raw data is time-synchronized and spatially calibrated. After preliminary denoising and spatiotemporal alignment, a multimodal dataset is obtained. The multimodal dataset is fused from multiple sources to generate a high-precision point cloud map and semantic operation scenario, and synchronized to a digital twin to form a virtual environment model corresponding to the actual construction site. The sensitivity, threshold, and mapping relationship of gestures and voice are calibrated in the virtual environment model, and a personalized multimodal interaction parameter set is formed by combining the operator's EEG characteristics. Based on the virtual environment model and the personalized multimodal interaction parameter set, 5G, Wi-Fi, data transmission radio, and low-speed links are aggregated. Combined with packet replication, forward error correction, and QoS prediction, mapping is completed according to data flow level to obtain a communication baseline that supports differentiated data flows. Local fast-loop control remains at the edge and does not rely on the public network. S2. Generate, execute, and send back packets based on the virtual environment model, personalized multimodal interaction parameter set, and communication baseline; Based on personalized multimodal interaction parameter sets, the following intent fusion and permission policy linkage are executed: (1) Based on the gesture threshold, speech threshold, and EEG threshold, the raw quantities of gesture intensity, speech intention, and EEG attention / stability are normalized to obtain the normalized gesture intensity. Voice confidence With EEG attention / stability ; Specifically, let's set The original quantity of gesture intensity is obtained by scalarization of the displacement of key points of the hand, grip strength, or arm acceleration; The raw quantity of speech intent is obtained from the recognition confidence or acoustic energy ratio triggered by keywords; The raw quantities for EEG attention / stability are obtained by indexing power spectrum / time-frequency characteristics; during calibration, the lower and upper limits of individual thresholds for each channel are recorded, and the gesture thresholds are denoted as follows: and The speech thresholds are denoted as follows: and The brainwave thresholds are denoted as follows: and Based on this, the normalized index is obtained: in, , representing normalized gesture intensity, speech confidence, and EEG attention / stability, respectively; (2) Based on the above A fusion score was constructed based on individual weights and combined with the amplification factor of EEG on intention enhancement and the environmental risk index. With control amplitude; and with trigger threshold Perform gating; when If no control superposition occurs, the output M is superimposed on the reference trajectory to form a superimposed control instruction set; Specifically, define individual weights. , representing the relative weight of gestures and voice on the operator, respectively; This represents the amplification factor of EEG on intention enhancement; The environmental risk index is derived from the constraint distances formed by obstacles, restricted areas, and height-restricted areas in the semantic scene of S12 on the pose and trajectory; a concise fusion score J and control amplitude M are constructed, where the control amplitude M is a saturation mapping: in: This is the risk inhibition coefficient; The output shape coefficient is used to determine the response speed and saturation level; trigger threshold. :when Control overlay is not generated at times to prevent accidental touches; otherwise, the output M is overlaid onto the reference trajectory to form an overlaid control instruction set. (3) Generate permission score A and introduce personalized threshold. Map continuous values ​​of A to discrete permission levels; when High-privilege operations are allowed when When this happens, the system automatically downgrades to a low-privilege mode. Specifically, an operational fatigue / load index δ is defined, which is derived from physiological signals or operation duration, and its value range is [value range missing]. ; Define permission score A: in: For strategy weights; Since A is a continuous score, its value range is usually between 0 and 1. However, the continuous value of A alone is insufficient to directly determine the operation permission level. Therefore, a personalized threshold ξ is introduced to map the continuous value of A to a discrete permission level: when When the system determines that the operator is in good condition, it allows high-privilege operations, including increasing the M limit, speeding up response, and performing near-obstacle operations. when When this happens, the system automatically downgrades to a low-privilege mode, which includes limiting the upper limit of M and performing rate limiting, secondary confirmation, or pausing. The above parameters Interactive calibration is performed in the "virtual environment model" and persisted as a personalized multimodal interaction parameter set; during task execution, ε, δ, and γ can be updated over time within the digital twin, thereby forming online adaptation; S3. Based on the execution and feedback packets and the multi-source real-time dataset collected and preprocessed by the status monitoring sensors, dynamic risk prediction is performed by combining the physical model and the federated learning model. A tiered intervention strategy is generated and controlled actions are executed. Simultaneously, the data and instructions are protected against tampering and stored as evidence. Specifically, real-time status data related to the tower crane is collected by the status monitoring sensors and time-aligned and preprocessed to form a multi-source real-time dataset for dynamic risk prediction. Based on the multi-source real-time dataset, dynamic risk prediction is performed by combining the physical model and the federated learning model. When structural anomalies or load risks are predicted, a virtual rehearsal is performed in the digital twin to derive the corresponding tiered intervention strategy. The tower crane is then controlled according to the tiered intervention strategy. Simultaneously, hashes are periodically generated from the multi-source sensor data and control instructions and stored in the tamper-proof evidence storage module. An index binding is established with the original video and sensor data to support post-event evidence retrieval.

2. The tower crane remote control method based on multimodal fusion and dynamic twin as described in claim 1, characterized in that, S2 includes the following steps: S21. Based on the virtual environment model, combined with the personalized multimodal interaction parameter set, the instruction structuring is completed to obtain the structured task description package; S22. Align the real-time working condition data continuously collected and updated by the multimodal sensors in step S11 during the operation to the virtual environment model to form the current working condition situation including dynamic obstacles and safety constraint sets. S23. After receiving the structured task description package and the current working situation, the digital twin performs constrained path planning and obstacle avoidance calculation based on the point cloud map and semantic operation scenario, and performs dynamic constraint correction in combination with the load swing model and wind load influence, thereby generating a reference trajectory package. S24. The tower crane control method performs high-speed closed-loop tracking of the reference trajectory package through the tower crane edge control node and executes swing suppression and emergency stop protection. It combines personalized multimodal interactive parameter set to parse operator gestures and generate superimposed control instruction set. Multi-link transmission is completed through the communication baseline. The operator interface renders a three-dimensional visualization scene and generates execution and feedback packets to the digital twin.

3. The tower crane remote control method based on multimodal fusion and dynamic twin according to claim 2, characterized in that, S23 includes the following steps: S231. The tower crane edge control node receives the reference trajectory packet and tracks the time-parameterized reference trajectory in the local high-speed closed-loop control to obtain trajectory tracking status information, while performing swing suppression and emergency stop protection. S232. During execution, the tower crane edge control node analyzes the operator's gesture input based on trajectory tracking status information and the personalized multimodal interaction parameter set, and generates corresponding fine-tuning amounts. The fine-tuning amount is then superimposed within the safety envelope constraint range corresponding to the reference trajectory packet to form a set of control instructions after superimposed gesture fine-tuning. S233. Based on the communication baseline, the control instruction set, path instruction, real-time video stream, and telemetry data are mapped to the corresponding link channels respectively to complete multi-link aggregation, packet replication, and error correction processing, forming a set of real-time data streams after multi-link transmission. S234. The tower crane control method renders a three-dimensional visualization scene based on the real-time data stream set in the operator interface, provides safety prompts related to the operating status, and finally generates execution and feedback packets.

Images