Method and system for collaborative manipulation of multiple mobile devices

Abstract

The present application relates to the field of nuclear power plant field operation control technology, in particular to a kind of multi-action device collaborative operation control method and system. Including: multi-modal control terminal cluster, control safety control mechanism design module, collaborative control hub and high-reliability communication link module. The present application has the advantages that: it can realize the safe and efficient collaborative operation of multi-action device in the special environment of strong electromagnetic, high radiation and high noise in nuclear power plant, effectively solve the problems of multi-terminal control permission ambiguity, frequent instruction conflict and operation non-traceability, improve the operation safety, collaborative efficiency and compliance, effectively improve the overall operation efficiency and performance of the system.

Description

Technical Field

[0001] This invention relates to the field of nuclear power plant on-site operation control technology, specifically to a method and system for coordinated operation of multiple action devices. Background Technology

[0002] The accuracy and safety of multi-device collaborative operation are directly related to the operational safety of the unit. Issues such as unauthorized operation or conflicting commands can lead to serious consequences such as equipment collisions, radiation leaks, and unplanned shutdowns, even threatening nuclear safety. Therefore, the safe management of multi-device collaborative operation has become one of the core requirements for intelligent operation of nuclear power plants. However, the operating environment of nuclear power plants is characterized by high radiation, complex spaces, and strict regulations, resulting in significant limitations in existing operation methods.

[0003] Existing technologies mostly employ independent terminal distributed control systems, lacking a unified collaborative management and control architecture. This can easily lead to action conflicts, low efficiency, and limited safety, making it difficult to balance the safety and efficiency of collaborative operation of multiple action devices, posing a severe challenge to the intelligent operation of nuclear power plants. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for coordinating operations of multiple action devices. By constructing a multimodal control terminal cluster, a collaborative control hub, and a highly reliable communication link, it enables safe and efficient collaborative operation of multiple action devices in the special environment of nuclear power plants characterized by strong electromagnetic fields, high radiation, and high noise. It effectively solves the problems of ambiguous control permissions, frequent command conflicts, and untraceable operations among multiple terminals, improving operational safety, collaborative efficiency, and compliance, and significantly enhancing the overall operating efficiency and performance of the system.

[0005] The technical solution of the present invention is as follows: a multi-action device cooperative operation control system, comprising:

[0006] A multimodal control terminal cluster, comprising a master control terminal and slave control terminals;

[0007] The module for designing a safety control mechanism for manipulation includes a three-level permission division, instruction conflict resolution logic, and full-process operation record traceability function.

[0008] The collaborative control center includes a control effect simulation unit, a master-slave collaborative control module, and a control command parsing module. The control effect simulation unit pre-simulates the execution effect of control commands based on the three-level permission division to generate a control strategy; it then optimizes global operations based on the control strategy. The master-slave collaborative control module dynamically allocates temporary control permissions from the master control terminal to the slave control terminal based on the three-level permission division rules, clearly defining the corresponding work area range and time limit. The control command parsing module converts the interactive commands of the multimodal control terminals into control signals based on command conflict resolution logic.

[0009] A highly reliable communication link module is used to establish a communication link connecting the multimodal control terminal cluster and the collaborative control center; and to send the control signals to the corresponding multi-action devices based on the communication link.

[0010] The three-level permission division includes:

[0011] Level 1 access, which corresponds to core security functions, including emergency braking, radiation shielding start-stop, and power system redundancy switching;

[0012] Level 2 access control, which corresponds to core security functions, including precise operation of the robotic arm, material transfer and positioning, and real-time collection of detection data;

[0013] The three levels of permissions correspond to auxiliary functions, including status indicator light control, historical data query, and non-critical parameter adjustment.

[0014] In the event of a system failure, the first-level permission function will activate a resource exclusive mechanism, the second-level permission function will operate in a downgraded manner as needed, and the third-level permission function can be suspended.

[0015] The multi-action device collaborative operation control system includes mutually compatible operation logic among the multiple action devices. The mutually compatible operation logic includes a unified instruction set, standardized action mapping rules, and a logical priority protocol for collaborative operation.

[0016] The multi-device collaborative operation control system also includes dynamic allocation of operation permissions based on task division, real-time status sharing across terminals, and a time-series collaborative mechanism for multi-person operation.

[0017] The manipulation effect simulation unit simulates the execution effect of manipulation commands based on the three-level permission division to generate a control strategy, including:

[0018] The manipulation effect simulation unit establishes a real-time data interaction channel with the digital twin model of the nuclear power plant to simulate manipulation effect scenarios.

[0019] The execution effect of the pre-simulated manipulation commands is classified and labeled based on the three-level permission division rule, and the potential risks of overlapping execution paths, overlapping operation ranges, and insufficient safety distance of multiple action devices are simulated.

[0020] Based on the simulation results, a control strategy is generated that includes path adjustment, timing optimization, and action avoidance.

[0021] The manipulation effect simulation unit virtually executes the control strategy based on a twin model and outputs the pre-simulation results; the multi-action device collaborative operation manipulation system automatically iterates and optimizes the control strategy based on the pre-simulation results, and the pre-simulation results support visualization and instruction optimization.

[0022] The simulated manipulation scenarios include obstacle avoidance in narrow spaces for mobile devices, path coordination among multiple mobile devices, and interference checks on the actions of the operation modules.

[0023] Different terminals in the multimodal control terminal cluster control different functional parts of the same mobile device, including:

[0024] Control permissions are divided based on the structural modules of the mobile device;

[0025] Based on the control permissions, the collaborative control center allocates the timing synchronization, permission interlocking, and real-time status feedback of the actions of each of the mobile devices.

[0026] The manipulation command parsing logic includes:

[0027] In the event of conflicting commands issued by different terminals to the same mobile device, emergency commands shall be executed first, and the conflict results shall be fed back to the relevant terminals in real time.

[0028] In non-emergency situations, the determination is made according to the terminal level and the order of instruction generation time.

[0029] The manipulation instruction parsing logic employs multimodal fusion logic, which includes semantic verification and format conversion of the interaction instructions.

[0030] The terminals in the multimodal manipulation terminal cluster are configured to seamlessly switch between multiple interaction modes.

[0031] The multiple interaction methods include at least two of the following: keyboard, mouse, gamepad, gesture, voice, touch, and force feedback, and the multiple interaction methods include intelligent switching.

[0032] The diverse interaction methods include combined input, and when multiple interaction methods are triggered simultaneously, the logic of scene adaptation takes priority in responding.

[0033] The master control terminal dynamically assigns temporary control permissions to the slave control terminal, and the control permission assignment rules include: the master control terminal can assign temporary control permissions of the mobile device to the slave control terminal;

[0034] The scope of the temporary control includes the movement of a single action device and the start and stop of the operation module.

[0035] The multimodal control terminal cluster also includes an emergency control interface, which includes a physical interface and a wireless access module.

[0036] The physical interface and wireless access module are configured for access by portable emergency devices, which temporarily take over key control rights when the master and slave control terminals fail. The validity period and scope of the access rights are dynamically limited by the system.

[0037] Establish a communication link connecting the multimodal control terminal cluster and the collaborative control center, the communication link including a wired part and a wireless part;

[0038] The wired portion includes environmentally resistant cables, and the wireless portion includes anti-interference transmission technology.

[0039] The main control terminal is configured with a multi-screen interactive interface, force feedback control components, noise-reducing voice interaction module, and is equipped with a safety lock to prevent accidental touch.

[0040] The control terminal is configured with a lightweight control interface.

[0041] The multimodal control terminal cluster includes control mode switching, and the switching methods include: manual mode, semi-automatic mode and program mode.

[0042] The operation record traceability function includes tamper-proof storage technology;

[0043] The operation record tracing function records the following information: terminal identifier, operator information, instruction content, execution result, and time information.

[0044] The multi-action device collaborative operation control system also includes a VR cockpit module;

[0045] The VR cockpit module is configured as an immersive virtual environment to recreate the operating scenario of the mobile device, and is configured to control the mobile device from a first-person perspective, compatible with the system's permission division and conflict resolution rules.

[0046] A multi-mobility device collaborative operation control method, implemented based on a multi-mobility device collaborative operation control system, includes:

[0047] Construct a multimodal control terminal cluster; the multimodal control terminal cluster includes a master control terminal and slave control terminals;

[0048] Design an operation security control mechanism, which includes a three-level permission division and instruction conflict resolution logic;

[0049] A collaborative control center is constructed; the collaborative control center includes a control effect simulation unit, a master-slave collaborative control module, and a control command parsing module.

[0050] The manipulation effect simulation unit pre-simulates the execution effect of manipulation commands based on the three-level permission division to generate a control strategy; and optimizes the global operation based on the control strategy.

[0051] Based on the three-level permission division rules, the master-slave collaborative control module dynamically allocates temporary control permissions from the master control terminal to the slave control terminal, clearly defining the work area and time limit corresponding to the permissions.

[0052] The control command parsing module converts the interactive commands of the multimodal control terminal into control signals based on the command conflict resolution logic.

[0053] Establish a communication link connecting the multimodal control terminal cluster and the collaborative control center; based on the communication link, send the control signal to the corresponding multi-action devices.

[0054] The beneficial effects of this invention are: it enables safe and efficient collaborative operation of multiple action devices in the special environment of strong electromagnetic fields, high radiation, and high noise in nuclear power plants, effectively solves the problems of ambiguous control permissions of multiple terminals, frequent command conflicts, and untraceable operations, improves operational safety, collaborative efficiency and compliance, and effectively improves the overall operating efficiency and performance of the system. Attached Figure Description

[0055] Figure 1 This is a structural block diagram of a multi-action device collaborative operation control system provided in one embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of a multimodal manipulation terminal cluster provided in one embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of the collaborative control center provided in one embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the linkage between the collaborative manipulation center and multiple action devices provided in one embodiment of the present invention;

[0059] Figure 5 This is a flowchart of a multi-action device collaborative operation control method provided in one embodiment of the present invention;

[0060] In the diagram: 10 Multimodal control terminal cluster; 20 Control safety management and control mechanism design module; 30 Collaborative control hub; 40 High-reliability communication link module; 50 Multiple action devices; 11 Master control terminal; 12 Slave control terminal; 13 Emergency control interface; 31 Control effect simulation unit; 32 Master-slave collaborative control module; 33 Control command parsing module. Detailed Implementation

[0061] like Figure 1 As shown, a multi-device collaborative operation control system includes: a multi-modal control terminal cluster 10, a control safety management and control mechanism design module 20, a collaborative control center 30, and a high-reliability communication link module 40; wherein the multi-modal control terminal cluster 10 includes a master control terminal 11 and slave control terminals 12; the control safety management and control mechanism design module 20 includes a three-level permission division, instruction conflict resolution logic, and full-process operation record traceability function; the collaborative control center 30 includes a control effect simulation unit 31, a master-slave collaborative control module 32, and a control instruction parsing module 33; the control effect simulation unit 31 is based on... The three-level permission division is used to pre-simulate the execution effect of the control commands to generate a control strategy; the global operation is optimized based on the control strategy; the master-slave collaborative control module 32 dynamically allocates temporary control permissions from the master control terminal 11 to the slave control terminal 12 based on the three-level permission division rules, and clearly defines the work area and time limit corresponding to the permissions; the control command parsing module 33 converts the interactive commands of the multimodal control terminals into control signals based on the command conflict resolution logic; the high-reliability communication link module 40 is used to establish a communication link connecting the multimodal control terminal cluster 10 and the collaborative control center 30; based on the communication link, the control signals are sent to the corresponding multi-action devices 50.

[0062] In one optional embodiment, the three-level permission division may include:

[0063] Level 1 access corresponds to core safety assurance functions, such as emergency braking, radiation shielding activation / deactivation, and power system redundancy switching. This type of access is the core access to ensure the safety of the nuclear power plant's operating environment, the mobile device itself, and surrounding personnel and equipment. It is the highest level of access in the system, and its functions determine whether the mobile device can stop losses in time under sudden risks and whether special hazards such as nuclear radiation can be effectively controlled. For example, emergency braking can forcibly stop the device when it is about to collide or enter a high-radiation area; radiation shielding activation / deactivation can quickly activate protective devices to block radiation leakage; and power system redundancy switching can seamlessly switch to a backup power source to maintain the basic safe operation of the device in the event of a main power failure. In the event of a failure in the multi-mobile device collaborative operation control system, the Level 1 access function will activate a resource exclusive mechanism.

[0064] Level 2 access corresponds to core safety assurance functions, such as precise robotic arm operation, material transfer and positioning, and real-time data collection. This type of access affects the quality and efficiency of operations of multiple moving devices in nuclear power plants and is a necessary access for completing core operation and maintenance tasks. It must be prioritized and protected on the premise that Level 1 safety access functions are operating normally. Among them, precise robotic arm operation enables delicate operations such as reactor component disassembly and assembly and equipment maintenance; material transfer and positioning ensures that spare parts, consumables and other materials are accurately delivered to designated workstations; and real-time data collection can synchronously transmit key data such as equipment operating parameters and environmental radiation values ​​to support operational decisions.

[0065] Level 3 access permissions correspond to auxiliary functions, such as status indicator light control, historical data query, and non-critical parameter adjustment. These permissions do not directly affect the operational safety and core operational processes of the mobile device, and are the lowest level of access in the system. Among them, status indicator light control can switch the indicator light color of the device or console according to the operational status, serving as an indication of the operational status; historical data query can retrieve information such as past operation records and fault logs of the device, making it easier for operators to review the operational process; non-critical parameter adjustment can fine-tune non-core operating parameters of the device (such as the brightness of the console display screen, the low power mode threshold of the device, etc.).

[0066] In one optional embodiment, the multiple action devices 50 in the multi-action device collaborative operation control system adopt mutually compatible control logic. The mutually compatible control logic includes a unified instruction set, standardized action mapping rules, and a logical priority protocol for collaborative operation. The unified instruction set can ensure that different devices have a consistent semantic understanding of the same control command, avoiding command ambiguity. For example, the "forward" command will cause the wheeled robot to drive its wheels to rotate and the robotic arm with a base to move after locking its joints, taking into account both device characteristics and action uniformity. The collaborative priority protocol determines the order of actions according to the importance of the task and the safety attributes of the device when multiple devices have conflicting operations, ensuring orderly collaboration.

[0067] For example, in a nuclear power plant spare parts transfer scenario, the operator issues a unified command to "go to Spare Parts Warehouse Area B". The inspection robot moves forward by rotating its drive wheels, the six-axis robotic arm moves its base after locking its joints, and the automatic transfer platform adjusts its track speed to move. All three accurately understand and execute the command. When the inspection robot and the transfer platform intersect in a narrow passage, the transfer platform automatically pauses to avoid the robot according to the priority agreement, and continues to move after the robot passes, achieving conflict-free collaboration.

[0068] In an optional embodiment, the multi-device collaborative operation control system may further include a dynamic allocation of operation permissions based on task division, real-time status sharing across terminals, and a timing coordination mechanism for multi-person operations. The dynamic permission allocation will split the control responsibilities according to the task, and the master control terminal 11 will accurately allocate exclusive permissions to the slave control terminals 12 to avoid permission overlap or missing. The real-time status sharing across terminals will synchronize the location, operation progress, and permission occupancy of each device, allowing all operators to have a real-time grasp of the overall situation. The timing coordination mechanism will clarify the action sequence of multi-person operations, especially key actions can be set to require confirmation from two people before execution to prevent timing chaos and conflicts.

[0069] In an optional embodiment, the manipulation effect simulation unit 31 pre-simulates the execution effect of manipulation commands based on a three-level permission division to generate a control strategy. This includes: establishing a real-time data interaction channel between the manipulation effect simulation unit 31 and the nuclear power plant digital twin model to simulate manipulation effect scenarios; classifying and labeling the pre-simulated execution effect of manipulation commands based on the three-level permission division rules, and simulating potential risks such as overlapping execution paths, overlapping operating ranges, and insufficient safety distances of multiple action devices 50; generating a control strategy that includes path adjustment, timing optimization, and action avoidance based on the simulation results; wherein, the simulated manipulation effect scenarios include obstacle avoidance in narrow spaces by action devices, path coordination of multiple action devices 50, and action interference checks of operation modules; the manipulation effect simulation unit 31 virtually executes the control strategy based on the twin model and outputs the pre-simulation results; the multi-action device collaborative operation manipulation system automatically iterates and optimizes the control strategy according to the pre-simulation results, and the pre-simulation results support visualization and command optimization.

[0070] Therefore, it can be seen that by relying on the digital twin model to accurately simulate the execution path, operating range and safety distance of multiple action devices 50, and clarifying the pre-rehearsal status of core safety functions according to the three-level authority, the control strategy is automatically iterated and optimized, so as to identify potential risks in advance and avoid safety accidents caused by instruction omissions in actual operation.

[0071] In an optional embodiment, different terminals in the multimodal control terminal cluster 10 can be configured to control different functional parts of the same set of mobile devices, including: dividing control permissions based on the structural modules of the mobile device, and allocating the timing synchronization, permission interlocking and real-time status feedback of the actions of each mobile device through the collaborative control center 30 based on the control permissions.

[0072] In one optional embodiment, the manipulation command parsing logic includes: when different terminals issue conflicting commands to the same mobile device, the emergency command is executed first, and the conflict result is fed back to the relevant terminal in real time; in non-emergency situations, the command is determined according to the terminal level and the order of command generation time; wherein, the manipulation command parsing logic adopts multimodal fusion logic, which includes semantic verification and format conversion of interactive commands.

[0073] When different terminals send commands to the same mobile device, emergency commands are executed unconditionally; non-emergency commands are executed first according to terminal level, with commands from higher-level terminals being executed first. If the levels are the same, the commands are executed in the order of their generation time, ensuring that commands are executed in an orderly manner without chaotic competition in non-emergency scenarios.

[0074] Multimodal fusion logic can perform semantic verification on interactive commands, such as recognizing nuclear power plant terminology, excluding unauthorized commands, and filtering dangerous operation commands. Then, it performs format conversion, uniformly converting the verified commands into standardized control signals that can be recognized by mobile devices, compatible with mainstream industrial protocols of nuclear power plants, and ensuring that commands from different interaction methods and different terminals can be accurately understood and executed by the same mobile device.

[0075] In one optional embodiment, all terminals in the multimodal control terminal cluster 10 are configured to seamlessly switch between multiple interaction methods; wherein, the multiple interaction methods include at least two of keyboard, mouse, gamepad, gesture, voice, touch, and force feedback, and the multiple interaction methods are configured to switch intelligently; the multiple interaction methods can be input in combination, and when multiple interaction methods are triggered simultaneously, the scene adaptation logic takes priority in response, and the switching process does not interrupt the continuity of mobile device operation.

[0076] In an optional embodiment, the master control terminal 11 dynamically assigns temporary control permissions to the slave control terminal 12. The control permission assignment rules include: the master control terminal 11 can assign temporary control rights of the mobile device to the slave control terminal 12; wherein, the scope of the temporary control rights includes the movement of a single mobile device and the start and stop of the operation module. Permission changes require identity authentication. The operator of the master control terminal 11 needs to complete the system's preset authentication process (such as "password + fingerprint" two-factor authentication, operation permission code verification). Only after the verification is passed can the permission change be initiated. During the change process, the system will automatically record the master control terminal 11 operator information, permission assignment object, permission scope and effective duration.

[0077] As the core control node of the system, the main control terminal 11 has the dominant right to allocate permissions. It can delegate temporary permissions to specific slave control terminals 12 according to the division of work tasks (such as regional special control or single device function control). The slave control terminal 12 is only the recipient of permissions and cannot independently acquire or overstep the level of permissions, thus ensuring the controllability of permission allocation.

[0078] Temporary control permissions can be strictly limited to two basic and necessary functions: "single mobile device movement" and "operation module start / stop". This avoids security risks caused by excessive delegation of permissions. The "single mobile device movement" permission allows the operator terminal 12 to control the basic movement of a designated mobile device, including path adjustment, speed control, start / stop operations, etc., but cannot control across devices. The "operation module start / stop" permission allows the operator terminal 12 to start or stop the dedicated operation module of a designated mobile device, but cannot adjust the core parameters of the module, ensuring that the operator terminal 12 can only complete specific operations and does not interfere with the core function configuration of the device.

[0079] Therefore, the design of on-demand authorization, scope limitation, and security verification can not only meet the needs of flexible division of labor and security management in multi-terminal collaborative operations, but also prevent the abuse or loss of control of permissions.

[0080] In an optional embodiment, the multimodal control terminal cluster 10 further includes an emergency control interface 13, which includes a physical interface and a wireless access module. The physical interface can adopt military-grade standards and has characteristics of radiation resistance, electromagnetic interference resistance, and environmental corrosion resistance to ensure the reliability of wired access in extreme scenarios. The wireless access module can integrate anti-interference wireless technologies such as Beidou short message service and 4G private network to ensure that the physical interface cannot be approached and that portable devices can be connected to the system wirelessly. The physical interface and the wireless access module can also be configured for portable emergency device access, which can temporarily take over key control rights when the master and slave control terminals fail, and the validity period and scope of the authority are dynamically limited by the system.

[0081] In one optional embodiment, a communication link is established connecting the multimodal control terminal cluster 10 and the collaborative control center 30. The communication link includes a wired part and a wireless part. The wired part includes environmentally resistant cables, and the wireless part includes anti-interference transmission technology to ensure the stability of the transmission of control commands in the strong electromagnetic environment of the nuclear power plant, and to ensure that the transmission of critical commands is not affected by network congestion.

[0082] In one optional embodiment, the main control terminal 11 is configured as a multi-screen interactive interface, a force feedback control component, a noise-reducing voice interaction module, and is equipped with an anti-accidental touch safety lock. As the core control node of the system, the main control terminal 11's multi-screen interactive interface can be configured to display the overall status of multiple devices on the main screen and detailed parameters of individual devices on the secondary screen, enabling parallel control of both the overall and detailed aspects. The force feedback control component (such as a six-axis joystick) simulates the operating resistance of the moving device, helping the operator accurately judge the operating force and avoid blind operation. The noise-reducing voice interaction module integrates an array microphone and a nuclear power plant terminology database, filtering out high-noise interference and supporting reliable voice command issuance. Simultaneously, an anti-accidental touch safety lock is provided, requiring dual verification for unlocking, locking core operations such as emergency braking and permission allocation to avoid the risk of accidental touch.

[0083] In an optional embodiment, the slave terminal 12 is configured as a lightweight control interface that supports localized command generation and execution, responds only to specific mobile device control permissions assigned by the master terminal 11, and can synchronize control status with the master terminal 11 in real time.

[0084] In one optional embodiment, the multimodal control terminal cluster 10 includes control mode switching, which includes manual mode, semi-automatic mode, and program mode. In manual mode, the operator needs to directly control each step of the motion device, such as finely adjusting the joint angle of the robotic arm and manually planning the inspection path. In semi-automatic mode, the motion device can autonomously complete basic actions such as obstacle avoidance and basic movement. The operator only needs to specify the work target. This mode also includes intelligent assistance functions. After specifying the target point, the system will automatically plan the optimal path of the motion device and dynamically adjust the action parameters to ensure control accuracy. In program mode, the operation can be automatically executed according to a pre-programmed process, such as daily inspections of fixed routes and standardized material transfer processes. The operator does not need to operate in real time, but only needs to monitor the operating status of the device and deal with abnormal situations.

[0085] In one optional embodiment, the operation record tracing function includes tamper-proof storage technology; wherein, the recorded content of the operation record tracing function includes the terminal identifier, operator information, instruction content, execution result and time information.

[0086] In the specific implementation process, anti-tampering storage technologies (such as consortium blockchain architecture and encrypted distributed database) can be used to encrypt and store all operation data, preventing data from being maliciously modified or deleted and ensuring the authenticity of the records. The recorded content can cover "operation terminal identifier, operator information, instruction content, execution result, and time information", realizing full-process traceability of "who on which terminal, when, what instruction was issued, and what the final execution effect was", which is convenient for subsequent operation and maintenance audits, fault diagnosis, and security incident review.

[0087] In one optional embodiment, the multi-mobile device collaborative operation control system further includes a VR cockpit module. The VR cockpit module is configured as an immersive virtual environment to recreate the operational scenario of the mobile devices, and is configured for first-person perspective control of the mobile devices, compatible with the system's permission allocation and conflict resolution rules. By recreating the real operational scenario of the mobile devices through an immersive virtual environment, the operator can intuitively perceive the surrounding environment of the device from a first-person perspective; it also supports first-person perspective control of the mobile devices, improving the accuracy of operation in special scenarios such as confined spaces and high radiation; at the same time, it is strictly compatible with the system's permission allocation and conflict resolution rules, ensuring integration into the system's collaborative logic and preventing operational confusion.

[0088] The multi-action device collaborative operation system of the present invention can achieve safe and efficient collaborative operation of multiple action devices 50 in the special environment of strong electromagnetic, high radiation and high noise in nuclear power plants by constructing a multimodal operation terminal cluster 10, an operation safety control mechanism design module 20, a collaborative control center 30 and a high-reliability communication link module 40. It effectively solves the problems of ambiguous control authority of multiple terminals, frequent command conflicts and untraceable operation, improves operation safety, collaborative efficiency and compliance, and effectively improves the overall operating efficiency and performance of the system.

[0089] like Figure 2 As shown, the multimodal control terminal cluster 10 includes a master control terminal 11, slave control terminals 12, and an emergency control interface 13. The master control terminal 11 includes "global control, permission allocation, and multi-interaction adaptive switching," undertaking the core control tasks of the system and being responsible for dynamically allocating temporary permissions to slave control terminals 12 and monitoring the global status. The slave control terminal 12 includes "regionalized special control, localized instruction generation, and synchronization with the master terminal status," responding only to specific job permissions allocated by the master control terminal 11 and focusing on local job execution and data feedback. The emergency control interface 13 includes "master and slave terminal failure takeover and first-level permission function execution," serving as redundancy protection in extreme scenarios, automatically activating and taking over core security functions when the master and slave terminals fail.

[0090] As can be seen from the structural diagram of the multimodal control terminal cluster provided by the present invention, the hierarchical relationship of the permissions of each terminal is clearly defined (master control terminal 11 > slave control terminal 12 > emergency control interface 13). Through the functional differentiation and collaborative linkage of the three-level terminals, the uniformity of global control is ensured, and the accuracy of local operations is achieved. At the same time, the emergency interface strengthens the safety bottom line, effectively solving the pain points of the traditional control terminal's single mode and insufficient collaboration, and providing a stable and reliable control entry point for multi-device collaborative operation.

[0091] like Figure 3As shown, the collaborative control hub 30 integrates a control effect simulation unit 31, a master-slave collaborative control module 32, and a control command parsing module 33. The control command parsing module 33 includes "multimodal command conversion, semantic verification, and protocol compatibility," responsible for receiving heterogeneous commands from the multimodal control terminal cluster 10, verifying and converting them to generate standardized control signals. The master-slave collaborative control module 32 includes "dynamic permission allocation, conflict prediction, and permission revokement," implementing permission control of master and slave terminals based on a permission matrix, predicting and avoiding command conflicts. The control effect simulation unit 31 includes "digital twin pre-playing, path conflict early warning, action interference detection, and safety redundancy assessment," accessing the nuclear power plant's digital twin model to perform virtual pre-playing and risk assessment of command execution effects. Commands are first passed to the control command parsing module 33 for standardization processing, then synchronized to the master-slave collaborative control module 32 to complete permission verification and conflict determination. Finally, combined with the pre-playing optimization results of the control effect simulation unit 31, a safe and efficient final command is generated and output to the communication link, fully demonstrating the integrated scheduling capabilities of the collaborative control hub.

[0092] Therefore, it can be seen that the closed-loop collaboration of the three modules effectively solves the core problems of heterogeneous instructions from multiple terminals, chaotic permissions, and uncontrollable operational risks, providing core logical support for the safe and efficient operation of the system.

[0093] like Figure 4 As shown, the multi-dimensional commands input by the multimodal control terminal cluster 10 are converted by the control command parsing module 33 of the collaborative control center 30, verified by the master-slave collaborative control module 32, and pre-simulated and optimized by the control effect simulation unit 31. Then, they are sent to multiple action devices through the high-reliability communication link module 40. After the action devices execute the commands, they feed back data such as location, operation status, and equipment parameters to the collaborative control center 30 and the multimodal control terminal cluster 10 through the same link, forming a complete closed loop of "command issuance-execution-status feedback".

[0094] It can be seen that by using a full-link design with a collaborative hub as the core, communication links as the link, and security mechanisms as the guarantee, the pain points of conflicting actions of multiple devices, unstable command transmission, and uncontrollable operational risks are effectively solved, ensuring the safety, continuity, and efficiency of collaborative operation of 50 multiple action devices in nuclear power plants.

[0095] like Figure 5 As shown, the present invention provides a multi-action device collaborative operation control method, which includes the following steps: S11~S17.

[0096] S11: Construct a multimodal control terminal cluster; the multimodal control terminal cluster includes a master control terminal 11 and a slave control terminal 12.

[0097] S12: Design a control mechanism for manipulation security, which includes a three-level permission division and instruction conflict resolution logic.

[0098] S13: Construct a collaborative control center; the collaborative control center includes a control effect simulation unit 31, a master-slave collaborative control module 32, and a control command parsing module 33.

[0099] S14: The manipulation effect simulation unit 31 simulates the execution effect of manipulation instructions based on the three-level permission division to generate a control strategy; and optimizes the global operation based on the control strategy.

[0100] S15: The master-slave collaborative control module 32 dynamically allocates temporary control permissions from the master control terminal 11 to the slave control terminal 12 based on the three-level permission division rules, and clearly defines the scope and time limit of the corresponding work area of ​​the permission.

[0101] S16: The control instruction parsing module 33 converts the interactive instructions of the multimodal control terminal into control signals based on the instruction conflict resolution logic.

[0102] S17: Establish a communication link connecting the multimodal control terminal cluster and the collaborative control center; based on the communication link, send control signals to the corresponding multi-action devices 50.

[0103] The multi-action device collaborative operation method of the present invention enables safe and efficient collaborative operation of multiple action devices 50 in the special environment of strong electromagnetic, high radiation and high noise in nuclear power plants by constructing a multimodal operation terminal cluster, a collaborative control center and a highly reliable communication link. It effectively solves the problems of ambiguous control authority of multiple terminals, frequent command conflicts and untraceable operation, improves operation safety, collaborative efficiency and compliance, and effectively improves the overall operating efficiency and performance of the system.

[0104] In step S11, please refer to Figure 1 In step S11, a multimodal control terminal cluster is constructed; the multimodal control terminal cluster includes a master control terminal 11 and a slave control terminal 12.

[0105] In one optional embodiment, during the construction of a multimodal control terminal cluster, the hardware customization, function integration and collaborative adaptation can be completed by focusing on the hierarchical collaborative logic of "global control - regional execution" and combining the special scenarios of nuclear power plants with high radiation, strong electromagnetic fields and personnel wearing heavy protective equipment.

[0106] The main control terminal 11 can adopt an integrated radiation-resistant hardened design and is equipped with multimodal interactive components and a dual-screen information layered display interface. As the core of global control, the main control terminal 11 can be responsible for task breakdown, temporary control permission allocation, status monitoring of slave control terminals 12, and emergency command issuance. It can obtain the operation data of all slave control terminals 12 and the operating status of multiple action devices 50 in real time, support dynamic adjustment of permissions, modification of work area boundaries and emergency handling of abnormal situations, and has the ability to link with the digital twin model of the nuclear power plant, and can view the simulation and pre-show results of the global operation scenario.

[0107] The slave control terminal 12 is designed with lightweight and rugged features. As a regional special operation node, the slave control terminal 12 only responds to specific operation tasks assigned by the master control terminal 11. Its operation range is strictly limited to the operation area assigned by the master control terminal 11. Operation commands that exceed the boundary will automatically become invalid.

[0108] In an optional embodiment, the multimodal control terminal cluster also includes an emergency control interface 13. The emergency control interface 13 can be equipped with a USB-C military-grade physical interface and a Beidou short message and 4G private network wireless module, supporting the access of portable emergency devices weighing ≤3kg and with a battery life of ≥8 hours. It is automatically activated when the master and slave control terminals 12 fail, and only takes over first-level permission functions (such as emergency braking, radiation shielding start and stop), with a takeover time of ≤30 minutes. After the timeout, non-safe functions are automatically locked.

[0109] In an optional embodiment, in step S12, all three types of control terminals have built-in unified encrypted communication modules and anti-electromagnetic interference components, and access the collaborative control center through a standardized protocol; the main control terminal 11 can monitor the status of the slave control terminal 12 and the emergency control interface 13 in real time through 10Hz heartbeat detection. The data transmission of the three is encrypted throughout the process to ensure that the command and status information are secure and not leaked, forming a terminal cluster with complementary functions and collaborative linkage.

[0110] In step S12, please refer to Figure 1 In step S12, a control security management mechanism is designed, which includes a three-level permission division and instruction conflict resolution logic.

[0111] In one optional embodiment, the three-level permission division includes:

[0112] Level 1 access control includes emergency braking, radiation shielding start / stop, and power system redundancy switching; Level 2 access control includes precise operation of the robotic arm, material transfer positioning, and real-time acquisition of detection data; Level 3 access control includes status indicator light control, historical data query, and adjustment of non-critical parameters.

[0113] In an optional embodiment, the instruction conflict resolution logic includes: when multiple terminal instructions conflict in a multimodal control terminal cluster, identifying the permission level corresponding to the instruction; first-level permission instructions are executed first; for non-first-level permission instructions, the instruction from the master control terminal 11 takes precedence over the instruction from the slave control terminal 12; and emergency operation instructions take precedence over regular operation instructions.

[0114] In an optional embodiment, the manipulation safety control mechanism also includes a full-process operation record traceability function.

[0115] In one optional embodiment, the full-process operation record traceability function includes: real-time recording of the execution status of the control permissions of each terminal in the multimodal control terminal; recording content using distributed anti-tampering storage technology; the recorded content includes the permission allocation record of each terminal in the multimodal control terminal cluster, the generation time of each instruction, the execution process, the execution result, and the associated mobile device status data.

[0116] In step S13, please refer to Figure 1 In step S13, a collaborative control center is constructed; the collaborative control center includes a control effect simulation unit 31, a master-slave collaborative control module 32, and a control command parsing module 33.

[0117] In one optional embodiment, the three main modules can first interact at millisecond levels via a high-speed internal data bus using a radiation-resistant industrial server (equipped with an FPGA chip for electromagnetic interference resistance ≥80dB, a 316L stainless steel casing for radiation resistance ≥500kGy, dual-redundant power supply + RAID5 storage) deployed in a dedicated server room of a nuclear power plant. Specifically, the manipulation effect simulation unit 31 connects to a full-size digital twin model of the nuclear power plant with an accuracy of ±5cm, and can use the Unity3D engine to pre-simulate path conflicts, action interference, and safety redundancy. The pre-simulation report is synchronized to the dual screens of the main control terminal 11. The master-slave collaborative control module 32 constructs a permission matrix based on "terminal ID-device ID-function ID-timeliness," with built-in conflict prediction logic to support real-time permission revocation. The manipulation instruction parsing module 33 receives multimodal instructions, performs semantic verification according to a nuclear power plant terminology library, converts them into standardized signals compatible with protocols such as OPCUA / ModbusTCP, and marks them by permission as P0 / P1 / P2 priority, prioritizing the issuance of P0 / P1 level instructions, while a P2 level temporary cache queue (capacity ≥100 entries) is used.

[0118] In step S14, please refer to Figure 1 In step S14, the manipulation effect simulation unit 31 pre-simulates the execution effect of manipulation instructions based on the three-level permission division to generate a control strategy; and optimizes the global operation based on the control strategy.

[0119] In one optional embodiment, the simulation unit 31 can first establish a two-way real-time data interaction channel with the nuclear power plant digital twin model. Through an interface compatible with the nuclear power plant's industrial communication protocol, the equipment status, environmental parameters, and spatial layout information in the model are dynamically synchronized. The data synchronization frequency is not less than 10Hz to ensure the consistency between the pre-simulation scenario and the actual on-site state. Subsequently, the received pre-simulation manipulation commands are classified and marked according to a three-level permission division rule: commands corresponding to the first-level permission are marked as safety priority, and core simulation computing power (CPU usage ≥ 30%) is allocated first during the pre-simulation, and the compatibility of the commands with the nuclear power plant's safety red lines (such as radiation exclusion zone boundaries and equipment protection thresholds) is forcibly verified; commands corresponding to the second-level permission are marked as efficiency priority, and the temporary permission range from the manipulation terminal 12 is associated during the pre-simulation to simulate the execution path and resource usage of the commands within the defined boundaries; commands corresponding to the third-level permission are marked as auxiliary coordination, and simulation resources are allocated according to the resource sharing principle during the pre-simulation to simulate resource coordination and conflict with the first- and second-level permission commands.

[0120] Risks can be simulated for different types of instructions. Safety-priority instructions focus on verifying radiation exceeding limits and collision risks; efficiency-priority instructions detect path overlap and operation intersection; and auxiliary and collaborative instructions simulate resource contention. Then, a hierarchical strategy is generated, binding independent resources and emergency paths to first-level instructions, optimizing paths and timing for second-level instructions, and setting dynamic degradation rules for third-level instructions. Finally, the global operation is optimized from the dimensions of time, space, and resources, and the solution is synchronized to the master-slave collaborative control module 32 and the control instruction parsing module 33 of the collaborative control center to guide actual operations.

[0121] The manipulation effect simulation unit 31 can virtually execute the control strategy based on the twin model and output the pre-simulation results; the multi-action device collaborative operation control system automatically iterates and optimizes the control strategy according to the pre-simulation results, and the pre-simulation results support visualization and instruction optimization.

[0122] In step S15, please refer to Figure 1 In step S15, the master-slave collaborative control module 32 dynamically allocates temporary control permissions from the master control terminal 11 to the slave control terminal 12 based on the three-level permission division rules, and clearly defines the scope and time limit of the corresponding work area for each permission.

[0123] In an optional embodiment, the master-slave collaborative control module 32 can parse the job task based on the three-level permission division rules, clearly assigning only level two or three permissions to the slave control terminal 12, excluding the possibility of level one permission allocation; then, combining the complexity and risk level of the job task, it matches slave control terminals 12 with normal hardware status and compliant historical operations from the slave control terminal 12 library; subsequently, it calls the three-dimensional spatial coordinate system of the nuclear power plant digital twin model to convert the work area range into closed spatial parameters including coordinate boundaries and height range. If the operation command of the slave control terminal 12 exceeds these parameters, a permission verification check is triggered. The system includes: setting initial permission validity based on the preset completion time of the task, with a built-in early warning threshold; real-time tracking of task progress via a digital twin model; automatically extending the remaining permission validity if the actual progress lags behind the preset progress by more than 20% (the threshold can be adjusted as needed); shortening the validity if the progress is more than 30% ahead of schedule; terminating temporary permissions of the slave terminal 12 at any time via permission intervention commands; and immediately invalidating any unexecuted operation commands issued by the slave terminal 12 based on the intervention command, with the executed operations being taken over and monitored by the master terminal 11 to ensure secure operation transitions without security vulnerabilities.

[0124] In step S16, please refer to Figure 1 In step S16, the control instruction parsing module 33 converts the interactive instructions of the multimodal control terminal into control signals based on the instruction conflict resolution logic.

[0125] In an optional embodiment, the control instruction parsing module 33 first performs semantic recognition on the interactive instructions (such as voice, touch, and gesture) input by the multimodal control terminal, calls the built-in nuclear power plant professional terminology library and fuzzy instruction correction mechanism, and automatically completes the omitted parameters. For example, "transfer platform moves" is completed as "transfer platform moves along the preset C1 path to the maintenance station at a speed of 0.2 m / s". The module accurately extracts the operation object, action parameters, and execution priority in the instruction, while eliminating invalid instructions that do not conform to the nuclear power plant operation specifications. Then, based on the instruction conflict resolution logic, the module performs potential conflict pre-detection on the concurrent instructions from multiple control terminals. If a conflict is found, it is immediately marked and fed back to the relevant terminal. After the conflict is resolved, the module enters the conversion stage. Finally, based on the extracted operation object, action parameters, and execution priority information, the interactive instructions are standardized and converted to generate a structured control signal containing a unique terminal identifier, corresponding permission level, timestamp accurate to milliseconds, operation object identifier, and action parameter set. The module automatically matches the hardware interface protocol of the multiple action devices 50 through the built-in protocol adaptation layer to ensure that the control signal can be directly parsed and executed by the target action device.

[0126] In step S17, please refer to Figure 1 In step S17, a communication link is established connecting the multimodal control terminal cluster and the collaborative control center; based on the communication link, control signals are sent to the corresponding multi-action devices 50.

[0127] In one optional embodiment, when establishing a communication link connecting the multimodal control terminal cluster and the collaborative control center, a heterogeneous architecture of "wired backbone + wireless redundancy" can be adopted. The wired part uses bismuth-doped radiation-resistant optical fiber as the backbone transmission medium (radiation-induced attenuation ≤0.1dB / kGy, armor layer thickness ≥2mm), and the terminal access section uses shielded twisted pair cable (anti-interference bandwidth 100MHz). The link bandwidth is designed to be ≥1Gbps, and the control command transmission delay is controlled to be ≤20ms to ensure stable transmission of core commands between the master / slave control terminal 12 and the collaborative control center. The wireless part is built based on a 5G private network (3.5GHz band) and frequency hopping technology (frequency hopping rate ≥100 times / second), with a total bandwidth ≥200Mbps. At the same time, more than 4 wireless APs are deployed to achieve blind-spot-free coverage of the nuclear power plant operation area, and an interference monitoring module is configured. When the signal-to-noise ratio is <15dB, it automatically switches to the 2.6GHz backup frequency band with a switching time ≤100ms, serving as an emergency channel in case of wired link failure.

[0128] During the process of issuing control signals based on the communication link, the transmission priority is first marked according to the corresponding permission level of the control signal (level 1 permission command is marked as P0, level 2 as P1, and level 3 as P2). During the network transmission stage, priority is given to ensuring the bandwidth of P0 and P1 level commands (occupying ≥60% of the link bandwidth) to avoid network congestion causing delays in core commands. During the transmission process, the terminal's unique identifier, permission level, timestamp, and other verification information are carried synchronously. When the control signal reaches the target multi-action device 50, the action device first verifies the legality of the information. After confirming that it is correct, it parses and executes the signal. The execution result is transmitted back to the collaborative control center and the corresponding control terminal through the original link, forming a closed-loop transmission process of "issuance-execution-feedback".

[0129] In an optional embodiment, the multi-action device cooperative operation method of the present invention can be based on, for example... Figure 1 The multi-action device collaborative operation control system in the relevant embodiments is executed.

[0130] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A multi-action device collaborative operation control system, characterized in that, include: A multimodal control terminal cluster, comprising a master control terminal and slave control terminals; The module for designing a safety control mechanism for manipulation includes a three-level permission division, instruction conflict resolution logic, and full-process operation record traceability function. The collaborative control center includes a control effect simulation unit, a master-slave collaborative control module, and a control command parsing module. The master-slave collaborative control module establishes a real-time data transmission channel with the multimodal control terminal cluster through the collaborative control center. The control effect simulation unit pre-simulates the execution effect of control commands based on the three-level permission division to generate a control strategy. Optimize global operations based on the aforementioned control strategy; The master-slave collaborative control module, based on the three-level permission division rules, dynamically allocates temporary control permissions from the master control terminal to the slave control terminal, clearly defining the work area range and time limit corresponding to the permissions; the control instruction parsing module, based on instruction conflict resolution logic, converts the interactive instructions of the multimodal control terminal into control signals. A highly reliable communication link module is used to establish a communication link connecting the multimodal control terminal cluster and the collaborative control center; and to send the control signals to the corresponding multi-action devices based on the communication link. The manipulation effect simulation unit simulates the execution effect of manipulation commands based on the three-level permission division to generate a control strategy, including: The manipulation effect simulation unit establishes a real-time data interaction channel with the digital twin model of the nuclear power plant to simulate manipulation effect scenarios. The execution effect of the pre-simulated manipulation commands is classified and labeled based on the three-level permission division rule, and the potential risks of overlapping execution paths, overlapping operation ranges, and insufficient safety distance of multiple action devices are simulated. Based on the simulation results, a control strategy is generated that includes path adjustment, timing optimization, and action avoidance. The manipulation effect simulation unit virtually executes the control strategy based on a twin model and outputs the pre-simulation results; the multi-action device collaborative operation manipulation system automatically iterates and optimizes the control strategy based on the pre-simulation results, and the pre-simulation results support visualization and instruction optimization. Different terminals in the multimodal control terminal cluster control different functional parts of the same mobile device, including: Control permissions are divided based on the structural modules of the mobile device; Based on the control permissions, the collaborative control center allocates the timing synchronization, permission interlocking, and real-time status feedback of the actions of each of the mobile devices.

2. The multi-action device cooperative operation control system as described in claim 1, characterized in that: The three-level permission division includes: Level 1 access, which corresponds to core security functions, including emergency braking, radiation shielding start-stop, and power system redundancy switching; Level 2 access control, which corresponds to core security functions, including precise operation of the robotic arm, material transfer and positioning, and real-time collection of detection data; The three levels of permissions correspond to auxiliary functions, including status indicator light control, historical data query, and non-critical parameter adjustment.

3. The multi-action device cooperative operation control system as described in claim 1, characterized in that: The multi-action device collaborative operation control system also includes a dynamic allocation of operation permissions based on task division, real-time status sharing across terminals, and a time-series collaborative mechanism for multi-person operation. The master control terminal dynamically assigns temporary control permissions to the slave control terminal. The rules for assigning control permissions include: the master control terminal can assign temporary control rights of the mobile device to the slave control terminal; wherein, the scope of the temporary control rights includes the movement of a single mobile device and the start and stop of the operation module, and permission changes require identity authentication.

4. The multi-action device cooperative operation control system as described in claim 1, characterized in that: When conflicting commands are issued to the same mobile device by different terminals, the operation command parsing module prioritizes the execution of emergency commands and feeds back the conflict results to the relevant terminals in real time. In non-emergency situations, the determination is made according to the terminal level and the order of instruction generation time. The manipulation instruction parsing module employs multimodal fusion logic, which includes semantic verification and format conversion of the interaction instructions.

5. The multi-action device cooperative operation control system as described in claim 1, characterized in that: The multimodal control terminal cluster also includes an emergency control interface, which includes a physical interface and a wireless access module. The physical interface and wireless access module are configured for access by portable emergency devices, which temporarily take over key control rights when the master and slave control terminals fail. The validity period and scope of the access rights are dynamically limited by the system.

6. The multi-action device cooperative operation control system as described in claim 1, characterized in that: Establish a communication link connecting the multimodal control terminal cluster and the collaborative control center, the communication link including a wired part and a wireless part; The wired portion includes environmentally resistant cables, and the wireless portion includes anti-interference transmission technology.

7. The multi-action device cooperative operation control system as described in claim 1, characterized in that: The operation record traceability function includes tamper-proof storage technology; The operation record tracing function records the following information: terminal identifier, operator information, instruction content, execution result, and time information.

8. The multi-action device cooperative operation control system as described in claim 1, characterized in that: The multi-action device collaborative operation control system also includes a VR cockpit module; The VR cockpit module is configured as an immersive virtual environment to recreate the operating scenario of the mobile device, and is configured to control the mobile device from a first-person perspective, compatible with the system's permission division and conflict resolution rules.

9. A method for coordinating operation of multiple action devices, implemented based on the multi-action device coordinating operation system of claim 1, characterized in that: Construct a multimodal control terminal cluster; the multimodal control terminal cluster includes a master control terminal and slave control terminals; Design an operation security control mechanism, which includes a three-level permission division and instruction conflict resolution logic; A collaborative control center is constructed; the collaborative control center includes a control effect simulation unit, a master-slave collaborative control module, and a control command parsing module; the master-slave collaborative control module establishes a real-time data transmission channel with the multimodal control terminal cluster through the collaborative control center; The manipulation effect simulation unit pre-simulates the execution effect of manipulation commands based on the three-level permission division to generate a control strategy; Optimize global operations based on the aforementioned control strategy; Based on the three-level permission division rules, the master-slave collaborative control module dynamically allocates temporary control permissions from the master control terminal to the slave control terminal, clearly defining the work area and time limit corresponding to the permissions. The control command parsing module converts the interactive commands of the multimodal control terminal into control signals based on the command conflict resolution logic. Establish a communication link connecting the multimodal control terminal cluster and the collaborative control center; based on the communication link, send the control signal to the corresponding multi-action devices.

Images