Digital twin virtual-reality interactive control decision fusion method for complex equipment

Abstract

The present invention proposes a digital twin virtual-reality interactive control decision fusion method for complex equipment, which integrates the control decisions of the physical end and the virtual end. It uses state alignment, evolution prediction, and instruction fusion methods to reduce the impact of inherent delay and delay fluctuation on decision accuracy and timeliness. Step 1: The equipment operates without abnormal state. Step 2: Determine whether abnormal state mapping point A1 appears. Step 3: Calculate the time Δt required from algorithm generation to execution. Step 4: Calculate the abnormal state A2 after Δt, and the virtual end makes a decision on A2. Step 5: If a decision is generated, the virtual and real decisions are integrated and executed. If no decision is generated, re-evaluate. Step 6: Determine whether the device state has returned to normal. The present invention is suitable for scenarios where virtual and real decisions are difficult to integrate due to delay issues during the interaction between the physical and virtual ends of complex equipment.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control of digital twins of complex equipment, and relates to a method for fusion of virtual-reality interactive control decisions of digital twins of complex equipment. Specifically, it proposes a method for fusion of virtual-reality interactive control decisions of physical and virtual ends to address the inherent delays and delay fluctuations caused by sensor acquisition and response, signal processing and calculation, network communication and delay in the virtual-reality interaction process of complex equipment. Background Art

[0002] Modern complex equipment often features a high degree of integration and intricate engineering structures. These devices incorporate numerous sensors, actuators, and control systems, and must operate under complex operating conditions while meeting the requirements for efficient, reliable, and safe operation. Intelligent operation of complex equipment can optimize its operation and control, improve operational and production efficiency, enhance safety, predict failures, and enable remote monitoring and management. To ensure personnel safety and improve equipment efficiency, research into intelligent control technologies for complex equipment is necessary.

[0003] Digital twin technology integrates the physical and digital worlds by building a two-way connection between the virtual and the real, allowing virtual models to simulate the behavior of physical entities. The intelligent operation of complex equipment requires the collaboration of both the physical and virtual ends. The physical end provides real-world operational status, while the virtual end uses algorithms and accumulated experience to provide precise decisions.

[0004] However, in actual virtual-reality interaction, due to inherent delays in sensor acquisition and response, signal processing and calculation, network communication, and latency, absolute synchronization of virtual and real-world decision-making cannot be achieved. The accumulation of these delays can reduce the timeliness of virtual decisions, or even render them unusable. Therefore, for certain virtual-reality interaction applications, it is crucial to consider the impact of latency. Summary of the Invention

[0005] In response to the problem of virtual-reality interaction decision delays that may be caused by the accumulation of multiple delays such as sensor information acquisition and response delays, information processing algorithm decision-making time, and network delay and communication delay, the present invention proposes a response plan to interactively integrate the control decisions of the physical end and the control decisions of the virtual end. Through state alignment, evolution prediction and instruction fusion methods, the impact of inherent delays and delay fluctuations on decision accuracy is reduced to ensure the timeliness and reliability of the decision-making of the virtual end.

[0006] In order to solve the above-mentioned delay problem of interaction between the physical end and the virtual end of the complex device, the present invention specifically includes the following steps:

[0007] Step 1: The equipment is running normally.

[0008] Step 2: Determine whether abnormal state mapping point A1 occurs;

[0009] Step 3: Calculate the time Δt required from algorithm generation to execution;

[0010] Step 4: Calculate the abnormal state A2 after Δt, and the virtual end makes a decision on A2;

[0011] Step 5: If a decision is generated, the virtual and real decisions are integrated and executed; if no decision is generated, re-evaluate;

[0012] Step 6: Determine whether the device status has returned to normal.

[0013] Specifically:

[0014] Step 1: The work starts, the physical control logic starts to execute, and the equipment operation status is periodically scanned. It ends when the target control state is reached, and the equipment operates normally;

[0015] Step 2: When the target control state is not reached, determine whether the control state is abnormal. If the state is normal, run the physical control logic. If it is abnormal, generate the abnormal working state mapping point A1;

[0016] Step 3: The abnormal working state triggers the virtual control logic, and the virtual end generates a virtual state mapping point A1. The time required for the virtual end algorithm decision to be generated and executed is Δt;

[0017] Step 4: The virtual end makes a control decision on the abnormal state mapping point A2 generated by the mapping point A1 after Δt;

[0018] Step 5: After Δt, if a virtual control decision is generated, it is fused with the abnormal working state A2 and executed. If no decision is generated, the new mapping point of the equipment operating state is re-evaluated based on the mapping point A2.

[0019] Step 6: Determine whether the control status has returned to normal. If it has returned to normal, perform a periodic scan of the equipment operation status. If it has not returned to normal, generate an abnormal state mapping point for virtual end decision control.

[0020] In this virtual-reality interactive control system, the control time scales of the physical and virtual ends differ. The physical end is primarily composed of hardware, primarily PLCs (Programmable Logic Controllers), with response times typically in the millisecond range. The physical end's mission is to monitor sensor data, execute control commands, and respond to external events to control and operate the physical system in real time.

[0021] The virtual side primarily consists of algorithms, models, and software, and its timescale can be more flexible, typically operating at sub-second levels or longer. Its mission is to process and analyze data from the physical side, generate control instructions, and perform optimization and decision-making.

[0022] The key to this invention lies in the use of a time-controlled framework for virtual-real interaction, where the virtual end evaluates and calculates the algorithm execution time Δt. Given the time delay Δt between the virtual decision reaching the physical end and the physical end's state changing from A1 to A2, the virtual end directly makes a decision based on the physical state A2 after Δt. This decision is then integrated based on the real-time state, ensuring the timeliness of the virtual end's decision.

[0023] Compared with the prior art, the present invention has the following technical effects:

[0024] Compared with the existing technology of directly applying virtual algorithm control decisions to the operation of physical equipment, the present invention takes into account the various inherent delays in virtual-reality interactive control that may cause the virtual end's decision lag or even unusability, and takes into account the offset workload of a series of inherent delays of the virtual end algorithm in the decision design of the virtual end algorithm. At the same time, the advantages of the physical end and the virtual end are combined to consider, and the physical end's fast response short-cycle control decision and the virtual end's precise correction long-cycle control decision are integrated and applied to the virtual-reality interactive control of complex equipment, thereby further improving the correction accuracy of the equipment's short-cycle decision and the overall stability of the long-cycle decision, preventing actual work from deviating from the expected equipment operation trajectory, realizing intelligent coordination of digital control and physical control, improving the intelligent control level of complex equipment under sudden disturbance conditions, and improving equipment work efficiency and safety.

[0025] Beneficial effects: Taking into account the influence of inherent delay and making decision designs on the virtual end, the new fusion decision makes a work plan for the most recent state of the device being disturbed, and the virtual end accumulates experience through algorithms to correct the overall control program of the physical end due to changes in the control benchmark caused by sudden disturbance events, thereby improving the overall work planning stability of the physical end of complex equipment and the level of accurate correction of abnormal device states on the virtual end.

[0026] The technical solution of the present invention combines the advantages of the physical and virtual ends by analyzing the time period of virtual-real interaction delay, and integrates the control decisions of the physical and virtual ends. In the presence of inherent delay, the precise control decisions of the virtual end are sent to the physical end, giving full play to the intelligent and precise correction advantages of the virtual end to control the operation of the equipment. The overall control program is corrected for long-term control targets, and the overall control targets of the physical and virtual ends are achieved. It not only makes short-term control decisions for sudden disturbances in the control of complex equipment, but also ensures the achievement of long-term control targets for long-term work layouts, realizes intelligent coordination between digital control and physical control, improves the intelligent control level of complex equipment under sudden disturbance conditions, and improves equipment work efficiency and safety.

[0027] Beneficial effect 1: The virtual terminal will provide accurate correction amount.

[0028] Beneficial effect 2: Correct the long-term control target when an emergency occurs, and correct the control benchmark changes caused by the sudden disturbance event in the overall control program. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the present invention;

[0030] Figure 2 It is the time point when the complex device of the embodiment is running in the state of virtual and real mutual control;

[0031] Figure 3 Schematic diagram of trajectory deviation of the device operation at time points T0-T4 in the embodiment.

[0032] Figure 4 Schematic diagram of the position of the cutting head of the roadheader running in the tunnel at time points T0-T4 in the embodiment. DETAILED DESCRIPTION

[0033] The specific technical solutions of the present invention are described with reference to the embodiments.

[0034] This embodiment adopts the digital twin virtual-reality interactive cutting control decision fusion in the remote intelligent control of the tunnel boring machine.

[0035] In the remote intelligent tunneling operation of underground coal mine roadheaders, intelligent control usually adopts the physical short-range control mode in the tunnel or the virtual remote control mode on the ground.

[0036] The physical end of the tunnel's short-range control is based on the coordinated control of multiple devices using multiple industrial controller systems. The overall work is arranged by setting up PLC programs, which places high demands on real-time decision-making and emergency stress handling capabilities. Accurate decision-making and corrections on the virtual end are necessary to deal with emergencies.

[0037] Remote control from the ground virtual end uses intelligent devices such as computer algorithms to make judgments and decisions based on sudden accidents. It is highly dependent on the perception of the operating status of all factors within the excavation working face, including personnel, equipment, tunnels, and the environment. However, the present invention analyzes that there is an inherent time delay in remote decision-making that cannot be avoided. Therefore, control will be carried out by combining the respective advantages of the physical and virtual ends.

[0038] For abnormal operation of the tunnel boring machine Figure 4 The specific analysis is as follows:

[0039] During the excavation process, the tunnel boring machine moves to the target position through the crawler walking mechanism, the front shovel transfer mechanism and the backup support mechanism of the tunnel boring machine adjust the support height, the cutting head cylinder extends, the hydraulic rotation and lifting part controls the position and height of the cutting arm, the cutting head rotates to start cutting the rock, and at the same time the water system sprays water for cooling and dust removal, so that it falls on the shovel and is transferred. The first conveyor transports it to the tail of the machine and is taken away by the transport vehicle.

[0040] like Figure 4 As shown, the solid arc-shaped trajectory represents the theoretical planned trajectory of the roadheader's cutting head entering the roadway, and the circle between T0 and T5 is a simplified diagram of the position where the cutting head contacts the roadway. Prior to T0, the roadheader's cutting arm operated normally, with no equipment anomalies. Testing of equipment operation and reaching the target control state was normal. At time T1, after T0, an emergency such as overbreak / underbreak or tunneling deviation is detected, requiring a virtual-side decision. At this point, the sensor information contains detailed information about the emergency. The sensor information arrives at the virtual side at time T2, and the virtual-side algorithm calculates the time Δt from the occurrence of the emergency to the receipt of the decision and the start of deviation correction. At time T3, the virtual side makes a decision based on the excess deflection within Δt, i.e., the physical device state A2 for time period T4. The decision is transmitted to the physical side and executed at time T4. During the time period T0-T4, before receiving the decision recommendation at time T4, the physical-side deviation pattern remains unchanged due to the short deviation period. Although the physical side can automatically repair and reduce the deviation, the decision lacks intelligence and accuracy, requiring the virtual side to leverage its precise decision-making advantages. After the virtual decision is executed at time T5, the TBM's cutting operation reaches the expected trajectory and a new round of work begins. The equipment then performs a periodic scan of the operating status to monitor and control the operating status of the physical-side equipment.

[0041] In response to abnormal operation of complex equipment, the virtual end makes corrections to the physical end control decision, taking into account the inherent delay of virtual end decision making, such as Figure 1 The process shown includes the following steps:

[0042] Step 1: The equipment is running normally.

[0043] Step 2: Determine whether abnormal state mapping point A1 occurs;

[0044] Step 3: Calculate the time Δt required from algorithm generation to execution;

[0045] Step 4: Calculate the abnormal state A2 after Δt, and the virtual end makes a decision on A2;

[0046] Step 5: If a decision is generated, the virtual and real decisions are integrated and executed; if no decision is generated, re-evaluate;

[0047] Step 6: Determine whether the device status has returned to normal.

[0048] Figure 2 It is the time point when complex equipment operates under the state of virtual and real mutual control. The equipment starts running at T0. At T1, it encounters emergencies such as damage and working offset, which require the virtual end to make a decision. The sensor information reaches the virtual end at T2. The virtual end completes the decision at T3 and transmits the decision to the physical end at T4 for fusion and execution. The decision is completed at T5.

[0049] Figure 3 This diagram shows the offset positions of a complex device during operation at time points T0-T5. The solid line represents the device's expected trajectory; the small black dots represent the device's position at each time point. The longer the distance from the solid line, the greater the offset; and the dashed line represents the degree of operational offset.

[0050] As can be seen from the figure, the system starts running at time T0, and an emergency occurs on the device at time T1. Before receiving the decision suggestion at time T4, the deviation pattern remains unchanged. Although the physical end can automatically repair and reduce the deviation, the decision lacks intelligence and accuracy. It is necessary to give full play to the precise decision-making advantages of the virtual end. After the virtual decision is executed at time T5, the device operation reaches the expected trajectory.

[0051] The working status of complex equipment varies at each time point. Its spatial position, cylinder extension length, mechanical component angle, and other specific data also vary at different time points. The equipment information at each timestamp is: t{x, y, z, α, β, γ...};

[0052] The virtual end monitors and predicts the operation of the physical end in real time. The physical and virtual ends of the complex equipment start working at time point T0. At time point T1, an abnormal state A1 {x1, y1, z1, α1, β1, γ1…} appears. The time period T2 reaches the virtual end for algorithm decision-making. At T3, the virtual end generates a decision. At time point T4, the decision arrives at the physical end. The virtual end directly corrects the state A2 {x2, y2, z2, α2, β2, γ2…} of the physical end where the decision arrives at time point T4, takes into account the excess deflection of the physical end, performs virtual and real decision judgment and fusion, and executes the new decision.

Claims

1. A digital twin virtual-reality interactive control decision fusion method for complex equipment, characterized by: Interactively integrate the control decisions of the physical side and the virtual side, and reduce the impact of inherent latency and latency fluctuations on decision accuracy and timeliness through state alignment, evolution prediction, and command fusion methods; The following steps are involved: Step 1: The equipment is running normally. Step 2: Determine whether abnormal state mapping point A1 occurs; Step 3: Calculate the time Δt required from algorithm generation to execution; Step 4: Calculate the abnormal state A2 after Δt, and the virtual end makes a decision on A2; Step 5: If a decision is generated, the virtual and real decisions are integrated and executed; if no decision is generated, re-evaluate; Step 6: Determine whether the device status has returned to normal; The specific steps are: Step 1: The work starts, the physical control logic starts to execute, and the equipment operation status is periodically scanned. It ends when the target control state is reached, and the equipment operates normally; Step 2: When the target control state is not reached, determine whether the control state is abnormal. If the state is normal, run the physical control logic. If it is abnormal, generate the abnormal working state mapping point A1; Step 3: The abnormal working state triggers the virtual control logic, and the virtual end generates a virtual state mapping point A1. The time required for the virtual end algorithm decision to be generated and executed is Δt; Step 4: The virtual end makes a control decision on the abnormal state mapping point A2 generated by the mapping point A1 after Δt; Step 5: After Δt, if a virtual control decision is generated, it is fused with the abnormal working state A2 and executed. If no decision is generated, the new mapping point of the equipment operating state is re-evaluated based on the mapping point A2; Step 6: Determine whether the control status has returned to normal. If it has returned to normal, perform a periodic scan of the equipment operation status. If it is not normal, generate an abnormal state mapping point for virtual end decision control.

2. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to claim 1.

3. A processor, characterized in that: The processor is configured to run a program, wherein the method according to claim 1 is executed when the program is run.

Images