Virtual robot control state management method, device, medium and program product

Abstract

The application provides a virtual mechanical arm control state management method, device, medium and program product, which is applied to a virtual surgery simulator system. The system comprises an operation platform, an intermediate control layer and a virtual simulator. The intermediate control layer comprises a system coordinator and a mechanical arm controller. The method comprises the following steps: the system coordinator receives and analyzes an initialization instruction sent by the operation platform, extracts virtual mechanical arm configuration information, instantiates a single-arm controller and issues configuration data; the mechanical arm controller initializes a control state; the system coordinator enters a motion preparation state after completing handshaking with the operation platform and sends a preparation signal to each single-arm controller; the single-arm controller completes motion preparation according to an activation mark; the mechanical arm controller feeds back a running state in real time; and the system coordinator coordinates the control state accordingly, thereby achieving dynamic management. The application adopts an intermediate control layer instantiation control and centralized scheduling mechanism, thereby achieving efficient cooperation and real-time response of multiple mechanical arms in a virtual surgery scene.

Description

Technical Field

[0001] This application relates to the field of mechanical control technology, and in particular to a method, device, medium and program product for virtual robotic arm control state management. Background Technology

[0002] Virtual surgical simulators, as a novel medical training aid, have demonstrated significant application value in areas such as surgical skills instruction, preoperative planning, and assessment. These systems are typically based on physical modeling, graphics rendering, and real-time interactive feedback mechanisms to provide users with a simulated surgical instrument operation experience, enhancing the immersion and precision of training.

[0003] Existing systems have significant limitations in their control architecture design. Most surgical robot simulators still employ direct motion mapping or a single-layer state machine architecture to construct the intermediate control layer: that is, the pose information output by the surgeon's operating platform is directly mapped to the virtual instrument's end effector, and the system calculates the virtual joint angles frame-by-frame using traditional inverse kinematics algorithms (such as the Jacobi pseudo-inverse method), and prevents over-limit operations through static limit strategies. While this architecture is simple to implement, it suffers from the following problems in multi-virtual robotic arm collaborative control scenarios:

[0004] The fact that all control processes share the same state machine makes it difficult to accurately respond to asynchronous instructions from different modules, resulting in response delays and state conflicts.

[0005] Complex operations such as clutch switching, perspective adjustment, and multi-arm coordination are difficult to coordinate dynamically, and fixed priority strategies are difficult to adapt to the real-time scheduling needs in complex scenarios.

[0006] Because state management is highly coupled with platform logic, the system can usually only be adapted to virtual devices with specific structures, making it difficult to be used across different simulator platforms, which reduces reusability and compatibility. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this application provides a virtual robotic arm control state management method, device, medium, and program product, which at least solves the problem of the lack of a unified state management and dynamic scheduling mechanism for the collaborative control process of multiple robotic arms in the existing technologies.

[0008] To achieve the above objectives and other advantages, some embodiments of this application provide the following aspects:

[0009] In a first aspect, some embodiments of this application provide a virtual robotic arm control state management method, applied to a virtual surgical simulator system. The virtual surgical simulator system includes an operating platform, an intermediate control layer, and a virtual simulator. The intermediate control layer includes a system coordinator and a robotic arm controller. The method includes:

[0010] The system coordinator receives the surgical scenario initialization command sent by the operation platform and parses the surgical scenario initialization command to extract the virtual robotic arm configuration information corresponding to the target surgical scenario;

[0011] The system coordinator instantiates a corresponding single-arm controller instance based on the virtual robotic arm configuration information and sends initial configuration data to the robotic arm controller.

[0012] The robotic arm controller initializes the control state of each single-arm controller instance according to the initial configuration data;

[0013] After the system coordinator and the operating platform complete the handshake information confirmation, the system enters the motion preparation state and sends a motion preparation signal to all the single-arm controller instances.

[0014] After receiving the ready-to-move signal, each of the single-arm controller instances completes the motion preparation for the corresponding virtual arm based on the arm activation marker;

[0015] The robotic arm controller provides real-time feedback to the system coordinator on the operating status information of each single-arm controller instance;

[0016] The system coordinator coordinates the running priority and execution sequence among the single-arm controller instances based on the operation instructions of the operating platform and the running status information, so as to realize dynamic management of the control status of the virtual robotic arm.

[0017] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising:

[0018] One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the virtual robotic arm control state management method as described above.

[0019] Thirdly, some embodiments of this application also provide a computer-readable storage medium having a computer program and / or instructions stored thereon, which, when executed by a processor, implement the virtual robotic arm control state management method as described above.

[0020] Fourthly, some embodiments of this application also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the virtual robotic arm control state management method as described above.

[0021] Compared with related technologies, the solution provided in this application constructs a hierarchical control architecture for virtual surgical simulator systems. The system coordinator in the intermediate control layer receives and parses the surgical scene initialization commands sent by the operating platform, automatically extracting the virtual robotic arm configuration information corresponding to the target scene. This enables the automatic generation and initial parameter configuration of single-arm controller instances, avoiding the inefficient process of manually setting instrument poses and parameters in traditional systems, significantly improving deployment efficiency and initialization accuracy. Based on this, single-arm controller instances complete differentiated motion preparation based on their respective activation flags. The system coordinator obtains the real-time operating status information of each single-arm controller and dynamically coordinates the control priority and execution sequence among multiple arms in conjunction with the operating platform's commands, achieving efficient collaboration and real-time response of multiple robotic arms in the virtual surgical scene. This method employs an intermediate control layer instantiation control and centralized scheduling mechanism, possessing good modularity and scalability, and can adapt to the configuration requirements of different types of instruments and a large number of robotic arms. Therefore, this solution enables rapid deployment and free migration of the virtual robotic arm control system across different platforms, greatly enhancing the system's practicality and promotional value in teaching and training, remote teaching, and cross-device integration. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other implementation methods can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall hardware connection structure of the virtual surgical simulator system in the embodiments of this application;

[0024] Figure 2 This is a flowchart illustrating a virtual robotic arm control state management method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the connection relationship between the instrument and the robotic arm in an embodiment of this application;

[0026] Figure 4 This is a system state operation flowchart of the virtual robotic arm control state management method provided in the embodiments of this application;

[0027] Figure 5 This is a schematic diagram of the single-arm controller instantiation process provided in the embodiments of this application;

[0028] Figure 6 This is a schematic diagram of the interaction process between the system coordinator and the single-arm controller provided in the embodiments of this application;

[0029] Figure 7 This is a rendering of the virtual simulator provided in the embodiments of this application;

[0030] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] First Embodiment

[0033] The first embodiment of this application relates to a virtual robotic arm control state management method, applied to a virtual surgical simulator system. The virtual surgical simulator system includes: an operating platform, an intermediate control layer, and a virtual simulator. The intermediate control layer includes a system coordinator and a robotic arm controller. The hardware deployment structure of the virtual surgical simulator system is as follows: Figure 1 As shown, the system includes: a main control computing platform for control and graphics processing, a standard Ethernet cable, and an operator platform (integrating an industrial control computer) for user operation. The virtual simulator and intermediate control layer can be deployed within the main control computing platform to perform task scheduling, graphics rendering, and status management. The operator platform, as the core interactive control terminal, is used to acquire the doctor's hand inputs, clutch and emergency stop operations, and image feedback. The main control computing platform can run an operating system that supports graphics rendering and control processing, such as Windows, Linux, or other compatible operating systems. The intermediate control layer, also known as the slave intermediate layer, is a subordinate execution control layer relative to the operator platform. Located between the operator platform and the virtual robotic arm simulator, it is responsible for parsing, distributing, and translating the doctor's operational intentions into control behaviors on the virtual robotic arm.

[0034] The operating platform includes human-computer interaction devices such as a right-hand controller, a left-hand controller, a clutch button, an emergency stop button, and a display. The operating platform collects user operation commands and can output various types of control information, such as surgical scenario initialization commands, instrument control commands, endoscope control commands, clutch or emergency stop control commands. At the same time, it collects the posture matrix, clamping status, and input events of the doctor's operating terminal and transmits them to the intermediate control layer through a network interface.

[0035] The intermediate control layer employs a dual-modal state management mechanism to manage the coordination and task scheduling of the entire control process. This is accomplished collaboratively by the system coordinator and the robotic arm controller. The system coordinator, as the core of the global mode management, is responsible for maintaining the state machine of the entire virtual surgical simulator system. Its states include, but are not limited to: initialization state, master-slave initialization state, motion preparation state, master-slave motion state, clutch state, emergency stop state, and system reset state. Based on the scene commands and control signals input from the operating platform, the system coordinator drives the state machine through orderly transitions and broadcasts its state updates to all relevant modules. Its control granularity is at the global control level, focusing on coordinating the logical relationships and behavioral timing between multiple virtual robotic arms.

[0036] As a local state management unit, the single-arm controller independently maintains the control state of each virtual robotic arm instance, including: standby state, active state, ready-to-move state, executing control state, clutch-pause state, and abnormal lockout state. Each single-arm controller instance receives high-level control commands from the system coordinator (such as ready-to-move, clutch commands, and control transfer commands) and determines whether to trigger a state transition based on its own state machine. Figure 3 As shown, the virtual instrument end effector is installed on the corresponding virtual robotic arm execution end via an interface component, used to accurately reproduce the motion behavior and spatial constraints of the real instrument during simulation. This connection structure design facilitates the replacement and initial binding of various surgical instruments, and supports the system coordinator in configuring and mapping instrument types, posture matrices, etc. during the initialization phase, so as to realize the logical association and fine control between the instrument and the robotic arm.

[0037] This dual-modal architecture decouples macroscopic coordination from microscopic control. The system coordinator centrally schedules system-level state transitions, ensuring the consistency and stability of the overall simulation environment; the single-arm controllers provide fine-grained response capabilities for each virtual robotic arm, ensuring that the operator's control intentions are accurately mapped to the target actuator. In this architecture, the system coordinator and each single-arm controller maintain synchronization through an event-driven communication mechanism. All critical state changes are communicated and recorded through state notification events. For example, when the operator releases the clutch button, the system coordinator proactively publishes a "clutch disengagement" event to all active single-arm controllers, and each controller instance decides whether to resume motion or remain on standby based on its current state.

[0038] Virtual simulator: As the system's graphics rendering and display module, it renders various virtual robotic arms, instruments, and scene animations based on the data stream from the intermediate control layer. The virtual simulator does not participate in control logic decisions; it only acts as a visual terminal responding to status updates from the intermediate control layer, achieving a WYSIWYG surgical simulation effect for the user. Simultaneously, the generated virtual scene images are transmitted in real-time to the operating platform's observation window via a physical connection interface or network transmission channel to ensure a high degree of consistency between the operating platform's display and the scene animations in the virtual simulator, providing immediate visual feedback on the operational actions.

[0039] The system physically connects the main control computing platform and the operation platform via standard Ethernet. It employs a predefined communication protocol to unify data structures and interface specifications, ensuring accurate parsing and processing of control commands and status feedback messages between the two parties. This communication protocol provides structured definitions for key fields (such as surgical scenario initialization commands, robotic arm control parameters, clutch and emergency stop events, etc.) and supports message type identification, parameter field validation, and status feedback flags. Regarding network transmission, by setting high-priority service types (such as QoS) and optimizing the buffer queue processing mechanism, the round-trip transmission delay of control commands is kept within 2ms to meet the stringent requirements of surgical simulation for low latency and high synchronization accuracy.

[0040] The three core modules communicate with each other via a standardized network interface and are deployed in a distributed architecture. The intermediate control layer acts as a bridge module, receiving data input from the operating platform and outputting drive commands to the virtual simulator, thereby constructing a virtual robotic arm simulation system with high responsiveness, high synchronization accuracy, and high control reliability.

[0041] Reference Figure 2 As shown, the virtual robotic arm control state management method applied to a virtual surgical simulator system specifically includes the following steps:

[0042] Step S1: The system coordinator receives the surgical scene initialization command sent by the operation platform and parses the surgical scene initialization command to extract the virtual robotic arm configuration information corresponding to the target surgical scene.

[0043] Specifically, in step S1, the system coordinator enters the initialization process during system startup. During this process, the system coordinator first receives a surgical scenario initialization command from the virtual simulator. This command encapsulates the key configuration information required to build the virtual surgical environment. This configuration information includes the following:

[0044] Spatial position and attitude information of the virtual robot base A reference frame used to determine the entire surgical procedure environment;

[0045] The number of virtual robotic arms, n, is used to initialize the corresponding number of single-arm controller instances;

[0046] Coordinates of each robotic arm's fixed points (such as the card-punching points) in the initial state. and the initial attitude matrix of its end effector Used to accurately restore the instrument to its initial state;

[0047] The identification ID of the instrument held by each robotic arm is used to load the corresponding virtual instrument model in the virtual simulator. At least one single-arm controller instance must carry an endoscopic instrument to support the field of vision guidance function.

[0048] The initial spatial pose of the endoscopic instrument To ensure accurate initial alignment of the vision system;

[0049] The list of robotic arms that have been activated for operation in the current surgical scenario has been identified. This typically corresponds to the left-hand and right-hand channels selected by the operator. Based on this activation information, the system coordinator establishes a control mapping relationship for the corresponding single-arm controller instance, and only initializes the motion state and issues tasks to these activated channels during the motion preparation phase, thereby avoiding invalid resource allocation to inactive arms.

[0050] Based on the above parameters, the system coordinator completes the logical construction of the virtual surgical scenario, laying the foundation for the subsequent state initialization and control scheduling of each virtual robotic arm control unit. See also... Figure 4 The state transition logic shown signifies that the initialization process also marks the formal transition of the entire system from the "initialization" state to the "master-slave initialization" state.

[0051] Step S2: The system coordinator instantiates the corresponding single-arm controller instance based on the virtual robotic arm configuration information and sends the initial configuration data to the robotic arm controller.

[0052] Specifically, in step S2, after parsing the scene initialization instructions, the system coordinator, based on the extracted virtual robotic arm configuration information, sequentially creates a corresponding single-arm controller instance for each virtual robotic arm that needs to participate in the operation. Each single-arm controller instance independently manages the motion control logic and state management information of one virtual robotic arm. For example... Figure 5 As shown, the system coordinator will dynamically generate n single-arm controller instances based on the number of robotic arms, n. For each single-arm controller instance, the system coordinator will construct a set of initial configuration data and send data packets through the control channel of the intermediate control layer. These data packets include the following key fields:

[0053] Initialization instruction flag This is used to trigger the initialization logic of the single-arm controller;

[0054] The target robotic arm ID is used to uniquely identify the virtual robotic arm corresponding to the current control instance.

[0055] Initial stamp point spatial coordinates Define the arm reference point as a reference for posture transition;

[0056] Initial pose of the instrument's end effector This is used to set the initial rendering and control state of the instrument;

[0057] The binding device type identifier Type indicates the category of the device currently held by the virtual robotic arm (such as scissors, pliers, endoscope, etc.).

[0058] Step S3: The robotic arm controller initializes the control state of each single-arm controller instance based on the initial configuration data.

[0059] Specifically, for step S3, after the system coordinator completes the parsing of the configuration information of each virtual robotic arm and sends the initialization command identifier... Upon reaching the robotic arm controller, the controller will trigger the initialization process for each target single-arm controller instance. First, the single-arm controller parses the received initial configuration data, binding it to its own instance to ensure consistent control mapping. Based on the initial configuration data, the single-arm controller will perform the following operations:

[0060] ID matching: Confirm that the robotic arm ID in the data packet matches the current instance identifier to ensure the uniqueness and accuracy of the instruction delivery;

[0061] Attitude matching: The initial checkpoint position and the initial pose matrix of the end effector are written as the initial spatial state of the instance into the local control cache to provide a benchmark for subsequent motion planning and attitude tracking;

[0062] Instrument matching: Load the corresponding virtual instrument model parameters according to the instrument type identifier, and configure control constraint logic, such as clamping method, number of movable joints and visual occlusion relationship, to ensure that the virtual robotic arm has the motion capability and control precision that conforms to the instrument attributes when performing operations, and avoid mismatch that may cause functional abnormalities or display errors.

[0063] Step S4: After the system coordinator and the operating platform complete the handshake information confirmation, the system enters the motion preparation state and sends a motion preparation signal to all single-arm controller instances.

[0064] Specifically, regarding step S4, after parsing all virtual robotic arm configuration information and initializing the single-arm controller instance, the system coordinator enters the master-side handshake phase. The system coordinator initiates a handshake request to the operating platform and receives the returned handshake signal data packet. This data packet contains the following key elements:

[0065] The initial pose matrix of the left and right hand controllers of the operator in the local coordinate system , , used to calibrate the current state of the operating handle;

[0066] Current clamping angle This reflects the initial state of the clamping device;

[0067] Emergency stop signal This is used to indicate whether the emergency stop protection state is in effect.

[0068] Clutch signal This is used to determine whether to enter the manual switching control channel state;

[0069] robotic arm switching signal This is used to indicate the current master-slave control channel allocation;

[0070] Handshake signal confirmation mark This indicates whether the operation platform is currently ready.

[0071] The system coordinator parses and verifies the validity of each received data packet, confirming that the current operation channel status, safety control signals, master-slave configuration, and scenario settings are consistent. If the verification passes, the system coordinator switches the state from "master-slave initialization" to "motion preparation" and sends a motion preparation signal to all active single-arm controller instances. This signal can be represented in a standardized format and includes information such as a cycle number and synchronization flag, indicating that each single-arm controller should complete local state preparation and feedback alignment before entering motion control.

[0072] Step S5: After receiving the motion preparation signal, each single-arm controller instance completes the motion preparation of the corresponding virtual arm based on the arm activation mark.

[0073] Specifically, for step S5, the first step is to determine whether the device itself belongs to a currently activated control channel, that is, to check whether its control number is included in the list of activated robotic arms. If the hit is successful, the single-arm controller enters the motion preparation phase and completes the following initialization actions:

[0074] Attitude reference alignment: based on the initial pose of the end effector received during the initialization phase. Adjust the local controller reference coordinate system to ensure spatial consistency of subsequent attitude control commands;

[0075] Checkpoint positioning judgment: Verify the current end state with the initially set checkpoint coordinates to ensure that the connection status between the device and the base is "ready".

[0076] Instrument model loading confirmation: Based on the instrument type identifier, confirm that the graphics layer rendering interface has correctly loaded the motion model of the instrument held by the arm to ensure that the rendering layer receives a model and instructions consistent with the physical control data;

[0077] Controller state transition: Switch the internal state machine of the current single-arm controller to the standby state, waiting to receive formal execution control commands.

[0078] If the single-arm controller is not present In the list, no motion preparation operation is performed; only the listening state is maintained, waiting for possible subsequent control channel switching or reactivation signals.

[0079] This motion preparation process ensures that each activated virtual robotic arm instance completes the necessary posture readiness and channel registration before entering the real-time control phase, effectively guaranteeing the consistency of the system state and the real-time responsiveness of subsequent control operations.

[0080] Step S6: The robotic arm controller provides real-time feedback to the system coordinator on the operating status information of each single-arm controller instance.

[0081] Specifically, regarding step S6, after completing the motion preparation phase, the motion preparation signal issued by the system coordinator to each single-arm controller instance has taken effect, and each single-arm controller instance then enters the master-slave motion control state. During this process, the robotic arm controller implements a periodic or event-driven state polling mechanism for all single-arm controller instances it manages, collecting the operational status information of each instance and transmitting this status data back to the system coordinator in real time via a standardized communication interface. The feedback operational status information may include: the current controller instance's status label (e.g., initialization complete, motion preparation complete, motion execution in progress, paused, emergency stop, etc.); and the virtual robotic arm's end effector pose matrix. The system coordinator continuously monitors the current clamping angle, instrument type identification, and binding status with the image layer. Through this feedback, the system coordinator can dynamically perceive the operational progress and status changes of each virtual robotic arm control unit, enabling global control process monitoring, task coordination, and fault tolerance.

[0082] Step S7: Based on the operation instructions and running status information of the operation platform, the system coordinator coordinates the running priority and execution sequence among the single-arm controller instances to achieve dynamic management of the control status of the virtual robotic arm.

[0083] Specifically, regarding step S7, after receiving the operation instructions from the operating platform (such as motion information of the master control handle, clamping actions, clutch or emergency stop trigger signals, etc.), the system coordinator first collects and analyzes the real-time operating status of each single-arm controller instance in the current system, including whether it is in an active state, whether it has entered the motion preparation mode, whether it has received an emergency stop signal, and whether it is currently executing an action. Then, based on the correspondence between the input instructions from the operating platform and the current operating status of each instance, the system coordinator dynamically determines the execution priority and operation sequence of each single-arm controller instance. For example, if the system detects a new motion input on the doctor's right-hand control channel and the corresponding right arm is in an active state, the system coordinator will assign a higher execution priority to that right arm and ensure that it receives priority resource allocation in control scheduling.

[0084] Simultaneously, the system coordinator dynamically adjusts the control channel mapping relationship based on control conditions such as clutch signals or switching commands, supporting state migration from control mode to autonomous mode and master-slave switching mode. Furthermore, when scheduling commands, the system coordinator also considers factors such as spatial conflict avoidance between robotic arms and collaborative operation of the devices, thereby achieving system-level collaborative control while maintaining high responsiveness. Finally, the coordination results are output to each single-arm controller instance in the form of a control signal stream, ensuring that they execute uniformly according to the coordination results, thus achieving dynamic and unified management of the virtual robotic arm's control state.

[0085] Compared with related technologies, the solution provided in this application constructs a hierarchical control architecture for virtual surgical simulator systems. The system coordinator in the intermediate control layer receives and parses the surgical scene initialization commands sent by the operating platform, automatically extracting the virtual robotic arm configuration information corresponding to the target scene. This enables the automatic generation and initial parameter configuration of single-arm controller instances, avoiding the inefficient process of manually setting instrument poses and parameters in traditional systems, significantly improving deployment efficiency and initialization accuracy. Based on this, single-arm controller instances complete differentiated motion preparation based on their respective activation flags. The system coordinator obtains the real-time operating status information of each single-arm controller and dynamically coordinates the control priority and execution sequence among multiple arms in conjunction with the operating platform's commands, achieving efficient collaboration and real-time response of multiple robotic arms in the virtual surgical scene. This method employs an intermediate control layer instantiation control and centralized scheduling mechanism, possessing good modularity and scalability, and can adapt to the configuration requirements of different types of instruments and a large number of robotic arms. Therefore, this solution enables rapid deployment and free migration of the virtual robotic arm control system across different platforms, greatly enhancing the system's practicality and promotional value in teaching and training, remote teaching, and cross-device integration.

[0086] Second Embodiment

[0087] The second embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The second embodiment provides a specific implementation method for dynamically adjusting the control state based on real-time input state of an operating platform, as described below. Figure 6 As shown, the method further includes:

[0088] The pose matrices of the right-hand control terminal and the left-hand control terminal output by the operating platform are both non-zero matrices, and the system coordinator enters the clutch state after confirming that the activated single-arm controller instance has completed motion preparation.

[0089] In the clutch state, the system coordinator executes the following control flow based on different control commands detected by the operating platform:

[0090] In response to the operating platform detecting the instrument operation command, the system coordinator sends the instrument motion command and the pose data packet corresponding to the master control terminal to the target single-arm controller example, and switches the control state to master-slave motion state.

[0091] In response to the operation platform detecting the endoscope control command, the system coordinator sends the endoscope displacement command and the pose data packet corresponding to the master control terminal to the single-arm controller instance with the instrument type of endoscope, and switches the control state to master-slave motion state.

[0092] In response to the operating platform detecting the clutch control command, the system coordinator sends the clutch control instruction to the target single-arm controller instance and switches the control state to master-slave motion state.

[0093] In response to the operating platform detecting a switching control command, the system coordinator switches the currently active single-arm controller instance and updates the control mapping relationship between the corresponding control channel and the target single-arm controller instance, but does not change the current control state of the system coordinator.

[0094] Specifically, when both the right-hand control terminal pose matrix and the left-hand control terminal pose matrix output by the operating platform are non-zero matrices, it indicates that the operator is ready to complete the operation. Simultaneously, the system coordinator confirms that all active single-arm controller instances have completed the corresponding motion preparation process. Under these conditions, the system coordinator switches the state machine to the clutch state. This clutch state serves as a pre-entry transition state before the system enters the master-slave motion control phase, establishing a safety buffer before formal motion control. During this process, the system does not immediately trigger the robotic arm's movement behavior but waits for the operator to explicitly issue control commands, such as instrument movement, endoscope control, clutch release, or control channel switching. In the clutch state, the system coordinator continuously monitors the control event input from the operating platform and executes the following control flow based on different types of control commands:

[0095] Responding to Instrument Operation Commands: When the system coordinator receives instrument control commands from the operating platform, such as movements of the operating end or clamping (gripper opening and closing), the system coordinator receives such signals, indicating that the operator has begun to execute the specific operation task. Since each operator is bound to a specific virtual robotic arm (managed by a single-arm controller instance), the system coordinator will send an instrument movement command identifier to that target single-arm controller instance. Simultaneously, it transmits the pose data packet of the operator's current control terminal, i.e., the motion matrix corresponding to the right and left hand control terminals. and This pose data packet encapsulates information such as the spatial position, orientation, and clamping angle of the operating terminal, used to drive the target virtual robotic arm to complete precise synchronized spatial movements in the simulation environment. After sending this control command and pose data, the system coordinator will switch the entire system state from "disengagement state" to "master-slave motion state," marking the official entry of the system into the real-time motion control phase, where each virtual robotic arm will begin to respond with high precision and synchronization based on the movements of the operating terminal.

[0096] Responding to Endoscope Control Commands: When the system coordinator receives an endoscope control command from the operating platform (e.g., the operator uses the control terminal to adjust the viewing angle, pan the field of view, or perform rotation), the system coordinator first identifies the target instance of the instrument type (endoscope) among the currently active single-arm controller instances based on preset instrument identification information. This identification process is based on the instrument type marker assigned during the system initialization phase, ensuring that control commands are accurately routed to the virtual robotic arm channel undertaking the vision task. The system coordinator then sends an endoscope displacement command identifier to the aforementioned target single-arm controller. It also synchronously transmits the current pose data packet of the operating terminal, that is, the pose matrix corresponding to the right and left control terminals. and Upon receiving the instruction and data packet, the target single-arm controller will drive the corresponding virtual endoscope to perform a pose update operation, thereby causing real-time changes in the viewing angle within the virtual scene. Simultaneously, the system coordinator will switch the overall control state from "disengaged state" to "master-slave motion state," signifying that the system has officially entered the operational phase where the operator dynamically controls the virtual lens.

[0097] Responding to Clutch Control Commands: When the system coordinator detects a clutch control command from the operating platform (i.e., the operator triggers the clutch button, indicating a desire to temporarily release control of the current robotic arm), the system coordinator first identifies the target single-arm controller instance corresponding to the currently active channel. Subsequently, the system coordinator sends a clutch control command identifier to the target single-arm controller, instructing it to pause its current motion control task. This clutch control command triggers the target single-arm controller's state machine to transition to the clutch pause state, meaning that although the system is generally in the motion control phase, this robotic arm channel will pause responding to new pose inputs to avoid unexpected actions during the operation transition. Simultaneously, the system coordinator still switches the global control state from "clutch state" to "master-slave motion state" to maintain the normal responsiveness of other active channels (such as robotic arms bound to other control handles).

[0098] Responding to Switching Control Commands: When the system coordinator receives a channel switching control command from the operating platform (e.g., to switch control from the virtual robotic arm currently controlled by the left-hand controller to another robotic arm bound to the right-hand controller), the system coordinator identifies the currently active operating channel and its corresponding target single-arm controller instance, and updates the control mapping table based on the switching command. This mapping table maintains the binding relationship between the operating channel and the single-arm controller instance. After the update, the new target robotic arm instance will take over the control commands input from the original channel, achieving an immediate transfer of control. During this control mapping update process, the system coordinator does not change its current global control state; that is, if the current system is in a "disengaged state" or "master-slave motion state," it remains unchanged. By decoupling the control mapping relationship from the system state management logic, the system can flexibly switch the robotic arm object corresponding to the user control channel without affecting the overall operating rhythm. This design ensures that the channel switching process does not trigger a jump or reset of the control state machine, thereby effectively avoiding problems such as response delay and motion abnormalities caused by state jitter or control interruption.

[0099] It is easy to see that, in this embodiment, by introducing a multi-type control command response mechanism under clutch and engagement states, the system's flexibility in responding to operator intentions and its coordination ability in control behavior are significantly enhanced. Compared to traditional control logic that relies on a single command flow, this embodiment supports the system coordinator in the clutch and engagement states to perform differentiated processing based on different control command types detected in real time by the operating platform, and then issue dedicated control commands and pose data packets to the corresponding target single-arm controller instances, and dynamically manage the control states and channel mapping relationships of each single-arm controller. This mechanism not only achieves a smooth transition from the clutch and engagement state to the master-slave motion state, ensuring the consistency of action timing and logical states during multi-arm control, but also allows for dynamic switching of control channels or temporary release of control without interrupting the global control flow, thereby effectively improving the system's operational continuity, safety, and user experience in multi-arm task collaboration, master-slave switching, and simulation of complex operation scenarios.

[0100] It should be noted that the second embodiment of this application may also be an improvement based on the first embodiment.

[0101] Third Embodiment

[0102] The third embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The third embodiment provides a specific implementation method for supporting automatic transition from a master-slave motion state to a global clutch control state, namely, the method further includes:

[0103] When the system coordinator is in master-slave motion state or other non-clutch control state, if it detects that the operator has left the operating position, or receives a global clutch control command from the operating platform, it immediately switches the control state to global clutch state and executes the following control flow:

[0104] Broadcast the global clutch control signal to all active single-arm controller instances, mark the task scheduling flag of each single-arm controller instance as suspended, and clear the unprocessed motion command cache.

[0105] Upon receiving a clutch control command, the single-arm controller instance actively stops the current action output of the end effector and updates its own state to standby.

[0106] Specifically, when the system coordinator is in a "master-slave motion state" or other non-clutch control state (such as an execution control state), if it detects that the operator has temporarily left the operating position (e.g., the operating platform detects handle release or loss of operating trajectory), or receives a global clutch control command from the operating platform (e.g., a manually triggered system-level clutch button), it immediately switches the control state to a "global clutch state." During this process, based on the global clutch control command received from the operating platform, the system coordinator immediately broadcasts the global clutch control signal to all active single-arm controller instances. Set the task scheduling flag of each single-arm controller to "suspended" to prevent it from continuing to read instructions from the local motion cache.

[0107] The system coordinator synchronously issues a clear command, requiring all single-arm controller instances to clear their unprocessed motion command caches to prevent operator errors or repetitive actions upon return. Furthermore, upon receiving a clutch control command, each single-arm controller instance will proactively halt its currently executing end effector action and update its state machine to "standby" to ensure the virtual robotic arm remains stationary and in a safe response mode after entering the global clutch state. This process achieves system freezing without requiring further user confirmation, effectively ensuring the safety of the operating platform.

[0108] It is easy to see that, in this embodiment, by introducing an automatic triggering mechanism for a global clutch control state, the response speed and system safety of the virtual robotic arm control system in emergency situations are significantly improved. When the operator temporarily leaves the control position or actively triggers a global clutch command, the system coordinator can immediately switch to the global clutch state, uniformly broadcast the global clutch control signal to all active single-arm controller instances, set their task scheduling flag to suspended, forcibly terminate the current control flow, and clear the motion command cache. This mechanism not only prevents system anomalies caused by erroneous input or unprocessed residual actions, but also ensures that each single-arm controller synchronously enters a "standby" state, keeping all virtual robotic arms stationary and avoiding potentially risky actions after the user leaves the operating platform. This enhances the fault tolerance and control robustness of the simulation system under abnormal operating conditions.

[0109] It should be noted that the third embodiment of this application may also be an improvement based on any one or more of the first to second embodiments.

[0110] Fourth embodiment

[0111] The fourth embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The fourth embodiment provides a specific implementation method supporting emergency stop response and control state freeze, namely, the method further includes:

[0112] When the system coordinator receives an emergency stop command from the operating platform, it immediately switches the control state to the global clutch state and executes the following control flow:

[0113] The system coordinator clears all mechanical control commands from the command queue, interrupts the motion command transmission channel with each single-arm controller, and marks the transmission channel as frozen to block the subsequent transmission of command data.

[0114] A global emergency stop flag is broadcast to all single-arm controller instances. Upon receiving the global emergency stop flag, each single-arm controller instance immediately suspends the task scheduling logic of its internal state machine. The global emergency stop flag is used to lock the control channel and output buffer of each single-arm controller instance.

[0115] The system coordinator stops uploading any received single-arm motion state data to the virtual simulator and freezes the display state of the virtual robotic arm at the current keyframe.

[0116] Specifically, when the system coordinator receives an emergency stop signal from the operating platform... (If the emergency stop button is triggered, or an emergency stop control event is generated when the system automatically detects an abnormal state), the system control state will be immediately switched to "global clutch state," and an emergency freeze mechanism will be activated to ensure that the virtual robotic arm system enters a safe shutdown mode. The system coordinator first clears all unprocessed instrument control commands in the current command queue to prevent existing commands from being delayed in execution after an emergency stop. At the same time, it actively interrupts the motion command transmission channels with all single-arm controllers and marks the transmission channels as "frozen," fundamentally cutting off the path for subsequent control data transmission.

[0117] The system coordinator broadcasts a global emergency stop flag to all active single-arm controller instances. Upon receiving the emergency stop flag, each single-arm controller instance immediately terminates its current task scheduling process, forcibly pauses its internal state machine, and locks its local control channel and output buffer to ensure that its output actions are completely frozen, thus avoiding unexpected behavior or delayed execution.

[0118] To ensure the operator's visual perception of the system status, the system coordinator synchronously stops uploading any new robotic arm status data to the virtual simulator and freezes the image frame status of the virtual robotic arm at the key frame when the emergency stop is triggered, presenting a frozen screen that intuitively shows that the current system is in an emergency stop protection state, and simultaneously obtains clear system response feedback on the operating platform.

[0119] It is easy to see that in this embodiment, when the operator triggers an emergency stop command or the system detects a critical anomaly, the system coordinator can immediately interrupt the transmission paths of various control signals and lock the output and task scheduling logic of all active single-arm controller instances, ensuring that the virtual robotic arm stops moving and freezes its output, thereby preventing erroneous commands from continuing to execute or causing uncontrollable behavior. Simultaneously, by switching the control state to a "global clutch state" and fixing the virtual display screen to a keyframe state, the operator can promptly perceive that the system has entered a safe mode, effectively reducing the risks caused by misoperation, delayed feedback, or visual misdirection. Therefore, by introducing an emergency stop response mechanism and a control state freezing strategy, rapid response and safe handling of sudden risk scenarios are achieved in the virtual surgical simulation system.

[0120] It should be noted that the fourth embodiment of this application may also be an improvement based on any one or more of the first to third embodiments.

[0121] Fifth Embodiment

[0122] The fifth embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The fifth embodiment provides a specific implementation method for supporting real-time linkage between the motion state of the virtual robotic arm and the virtual scene, namely, when the system coordinator is in a disengaged state or a master-slave motion state, it further includes the following processing steps:

[0123] The system coordinator receives real-time motion data from each single-arm controller instance. The real-time motion data includes the end-effector pose, execution path, joint angles, and operation status identifier of the virtual robotic arm under the current control state.

[0124] The system coordinator encapsulates the received real-time motion data and generates standardized transmission data frames according to the interface protocol of the virtual simulator.

[0125] The system coordinator synchronously transmits standardized data frames to the virtual simulator to drive the real-time rendering and status update of the corresponding virtual robotic arm in the virtual scene.

[0126] Specifically, the system coordinator continuously receives real-time motion data from each single-arm controller instance. This real-time motion data covers the key execution parameters of the corresponding virtual robotic arm in the current control state, such as the three-dimensional pose (position and attitude) of the end effector, path trajectory, joint angle information, and the identifier of the current control state (e.g., clamping state, execution state). The system coordinator uniformly encapsulates this raw data, packaging it into standardized data frames according to the interface communication protocol defined by the virtual simulator (e.g., field length, data structure, frame header identifier, etc.), ensuring that the data format is consistent and resolvable in cross-module communication. The system coordinator transmits this data frame to the virtual simulator via a high-speed bus (e.g., Ethernet or CAN-FD). After receiving the data frame, the simulator drives the virtual robotic arm to perform real-time graphics rendering and posture updates in the three-dimensional simulation scene based on the motion parameters and state information described within it. During this process, the motion trajectory, posture changes, and operational status of the virtual robotic arm can be synchronously presented in the user interface, such as... Figure 7 As shown, this ensures a high degree of consistency between the operator's physical actions on the operating platform and their responses in the virtual interface, enhancing the system's intuitiveness and immersive experience.

[0127] It is easy to see that, in this embodiment, by introducing a real-time linkage mechanism between the virtual robotic arm's motion state and the virtual scene, a high degree of coupling and synchronization between the system's physical control logic and image presentation is achieved. This mechanism significantly improves the real-time feedback and screen response accuracy of the virtual control system, enabling operators to obtain a more intuitive, accurate, and immersive interactive experience during operation training or simulation exercises, thereby enhancing training effectiveness and reducing the risk of misoperation.

[0128] It should be noted that the fifth embodiment of this application may also be an improvement based on any one or more of the first to fourth embodiments.

[0129] Sixth Embodiment

[0130] The sixth embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The sixth embodiment provides a specific implementation method supporting system control state reconstruction and master-slave control channel restart, namely, the method further includes:

[0131] When the system coordinator receives a reset signal from the operating platform, it executes the following control flow:

[0132] The system coordinator clears the current global state machine's running state, event queue, and state transition history, and simultaneously releases the cached configuration information, control parameters, and intermediate processing data in each single-arm controller instance;

[0133] After the cleanup operation is completed, the system coordinator enters the system reset initialization process, re-receives and parses the surgical scenario initialization command sent by the operation platform, re-extracts the virtual robotic arm configuration information and instantiates a new single-arm controller instance object, and establishes a new master-slave control channel and task scheduling mapping to complete the global state reconstruction and control process restart of the system.

[0134] Specifically, when the system coordinator receives a system reset signal from the operating platform (e.g., after a manually triggered system reset button or a platform self-check-triggered restart command) a global reset mechanism will be immediately initiated. This mechanism consists of two phases: a state clearing phase and a system reconstruction phase.

[0135] During the state clearing phase, the system coordinator first clears the current control state identifier, event processing queue, and historical state transition paths from the global state machine to completely remove contextual traces of previous control flows and prevent residual states from interfering with subsequent initialization logic. Simultaneously, the system coordinator issues a clear control command to all active single-arm controller instances, requiring each instance to release locally cached configuration information (such as bound device types, numbers, and channel mapping relationships), control parameters (such as maximum speed, acceleration limits, and motion boundaries), and intermediate state data (such as the current task's path cache, clamping status, and execution progress). This ensures that each instance returns to a unified initial standby state, ready to enter the reconfiguration process.

[0136] During the system reconfiguration phase, the system coordinator receives and parses the surgical scenario initialization command sent by the operating platform. This command includes the number of robotic arms required by the current virtual environment, their type configuration, master-slave control arm allocation relationship, fixed point position parameters, and activation channel IDs. Based on these parameters, the system coordinator re-instantiates single-arm controller objects, dynamically creates new controller instances, and rebuilds the master-slave control mapping relationship. Simultaneously, it initializes the status control flags and task allocation queues required for system scheduling, restoring the logical integrity and consistency of the control system.

[0137] It is easy to see that, in this embodiment of the application, by introducing a reset mechanism, the control disorder caused by sudden events, abnormal states or logical errors during system operation can be effectively addressed, ensuring that the system has the ability to recover quickly and a stable operating environment in complex operation tasks (such as virtual surgical simulation or remote collaborative control), and significantly improving the fault tolerance, maintainability and overall reliability of the system.

[0138] It should be noted that the sixth embodiment of this application may also be an improvement based on any one or more of the first to fifth embodiments.

[0139] Seventh Embodiment

[0140] The seventh embodiment of this application relates to a method for managing the control state of a virtual robotic arm. The seventh embodiment provides a specific implementation method for supporting handshake confirmation between the operating platform and the system coordinator, namely, the method further includes:

[0141] Before entering the motion preparation state, the system coordinator and the operating platform execute a handshake confirmation process from the master control terminal. The handshake confirmation process includes:

[0142] The system coordinator receives the handshake information data packet from the operating platform and verifies the integrity of the key control parameters contained therein. The key control parameters include: the pose matrix of the left and right hand control terminals on the operating platform, the operating clamping angle, the emergency stop flag, the clutch flag, the control channel switching flag, and the handshake flag.

[0143] After verification, the system coordinator returns a response confirmation message to the operation platform as a sign that the handshake between the two parties is complete;

[0144] After the handshake is completed, the system coordinator switches the current control state to the motion preparation state and sends a motion preparation signal to all single-arm controller instances to initiate the motion preparation process of each single-arm controller instance.

[0145] Specifically, before the system coordinator and the operating platform enter the motion preparation state, they perform a handshake confirmation process to ensure that the control parameters sent by the operating platform are valid, complete, and safe. The system coordinator receives a handshake information data packet from the operating platform, which carries several key parameters highly relevant to the current state of the operating platform, including the pose matrix of the left and right hand control terminals on the operating platform. , (Used to describe the position and posture information of the operator's hand device in three-dimensional space), operating clamping angle (Indicates whether the device is currently under clamping control), Emergency Stop Sign With clutch indicator (Used to indicate whether there is an interruption or pause requirement), Control Channel Switching Flag (Indicators of whether a channel switching operation has occurred) and handshake indicators (Used to mark whether the current data packet is an acknowledgment request during the system initialization phase).

[0146] After completing the above parameter verification, if the system coordinator confirms that all fields are valid, without missing data, and consistent in status, it immediately returns a response confirmation message to the operation platform. (This is followed by a seemingly unrelated sentence about handshake information.) After both parties confirm that there are no errors, the system coordinator determines that the handshake is successful and switches the current system control state to "motion preparation state". Subsequently, the system coordinator synchronously sends a motion preparation signal to all activated single-arm controller instances, triggering the motion preparation process of each single-arm controller, including attitude synchronization, end effector preheating, and buffer initialization, to ensure that the system has a stable starting state in the subsequent control task execution.

[0147] It is not difficult to see that in this embodiment of the application, by introducing a handshake confirmation mechanism, the integrity and consistency of key control parameters are first checked and confirmed before the system coordinator and the operation platform enter the motion preparation state. This prevents misoperation or action deviation caused by missing parameters, abnormal signals or unsynchronized states, and significantly improves the safety and stability of the system before entering motion control.

[0148] It should be noted that the seventh embodiment of this application may also be an improvement based on any one or more of the first to sixth embodiments.

[0149] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0150] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0151] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a virtual robotic arm control state management method as provided in any one or more of the above embodiments. Figure 8An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0152] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0153] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0154] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0155] In this embodiment, a computer-readable medium stores a computer program / instruction, which, when executed by a processor, implements a virtual robotic arm control state management method provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not be assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0156] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0157] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0160] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0162] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0163] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A virtual robot control state management method, characterized by, The method is applied to a virtual surgery simulator system, the virtual surgery simulator system comprises an operation platform, an intermediate control layer and a virtual simulator, the intermediate control layer comprises a system coordinator and a mechanical arm controller, and the method comprises the following steps: The system coordinator receives a surgery scene initialization instruction sent by the operation platform, and parses the surgery scene initialization instruction to extract virtual mechanical arm configuration information corresponding to a target surgery scene; The system coordinator instantiates a corresponding single-arm controller instance according to the virtual mechanical arm configuration information, and sends initial configuration data to the mechanical arm controller; The mechanical arm controller initializes the control state of each single-arm controller instance according to the initial configuration data; After the system coordinator and the operation platform complete handshake information confirmation, the system coordinator enters a motion preparation state, and sends a preparation motion signal to all single-arm controller instances; Each single-arm controller instance completes motion preparation of a corresponding virtual arm according to an arm activation mark after receiving the preparation motion signal; The mechanical arm controller feeds back running state information of each single-arm controller instance to the system coordinator in real time; The system coordinator coordinates the running priority and execution time sequence between single-arm controller instances based on operation instructions of the operation platform and the running state information, so as to realize dynamic management of the control state of a virtual mechanical arm.

2. The virtual robot control state management method of claim 1, wherein, The right-hand control end pose matrix and the left-hand control end pose matrix output by the operation platform are both non-zero matrices, and the system coordinator enters a clutch state after confirming that the motion preparation of the activated single-arm controller instance is completed; In the clutch state, the system coordinator executes the following control process based on different control commands detected by the operation platform: In response to detection of an instrument operation command by the operation platform, the system coordinator sends an instrument motion instruction and a pose data packet corresponding to a master control end to a target single-arm controller instance, and switches the control state to a master-slave motion state; In response to detection of an endoscope control command by the operation platform, the system coordinator sends an endoscope displacement instruction and a pose data packet corresponding to a master control end to a single-arm controller instance of an instrument type of an endoscope, and switches the control state to a master-slave motion state; In response to detection of a clutch control command by the operation platform, the system coordinator sends a clutch control instruction to a target single-arm controller instance, and switches the control state to a master-slave motion state; In response to detection of a switching control command by the operation platform, the system coordinator switches the currently activated single-arm controller instance, and updates the control mapping relationship between a corresponding control channel and a target single-arm controller instance, but does not change the current control state of the system coordinator.

3. The virtual robot control state management method of claim 1, wherein, When the system coordinator is in a master-slave motion state or other non-clutch control state, if it is detected that an operator leaves an operation position or a global clutch control command is received from the operation platform, the control state is immediately switched to a global clutch state, and the following control process is executed: Broadcast a global clutch control signal to all activated single-arm controller instances, mark the task scheduling flag of each single-arm controller instance as suspended, and clear the motion instruction cache that has not been processed; After receiving the clutch control instruction, the single-arm controller instance actively stops the current motion output of the end effector and updates its own state to standby.

4. The virtual robot control state management method of claim 1, wherein, When the system coordinator receives an emergency stop operation instruction from the operation platform, it immediately switches the control state to the global clutch state and performs the following control process: The system coordinator clears the instruction queue of various instrument control instructions, interrupts the motion instruction transmission channel between each single-arm controller, and marks the transmission channel as frozen to block the subsequent issuance of instruction data; Broadcast a global emergency stop flag to all single-arm controller instances. Upon receiving the global emergency stop flag, each single-arm controller instance immediately suspends the task scheduling logic of the internal state machine. The global emergency stop flag is used to lock the control channel and output cache of each single-arm controller instance. The system coordinator stops uploading any single-arm motion state data received to the virtual simulator and freezes the display state of the virtual robot arm at the current key frame.

5. The virtual robot control state management method of claim 1, wherein, During the clutch state or master-slave motion state, the system coordinator also includes the following processing steps: The system coordinator receives real-time motion data from each single-arm controller instance, which includes the end pose, execution path, joint angle, and operation state identification of the virtual robot arm under the current control state. The system coordinator encapsulates the received real-time motion data and generates standardized transmission data frames according to the interface protocol of the virtual simulator; The system coordinator synchronously transmits the standardized transmission data frames to the virtual simulator to drive the real-time rendering and state update of the corresponding virtual robot arm in the virtual scene.

6. The virtual robot control state management method of claim 1, wherein, When the system coordinator receives a reset signal from the operation platform, it performs the following control process: The system coordinator clears the running state, event queue, and state transition history of the current global state machine, and simultaneously releases the cached configuration information, control parameters, and intermediate processing data in each single-arm controller instance; After completing the clearing operation, the system coordinator enters the system reset initialization process, re-receives and parses the surgical scene initialization instructions sent by the operation platform, re-extracts the virtual robot arm configuration information and instantiates new single-arm controller instance objects, establishes new master-slave control channels and task scheduling mappings, to complete the global state reconstruction and control process restart of the system.

7. The virtual robot control state management method of claim 1, wherein, Before entering the motion preparation state, the system coordinator and the operation platform perform a handshaking confirmation process for the master control end, which includes: The system coordinator receives a handshaking information data packet from the operation platform and verifies the integrity of the key control parameters contained therein, including the pose matrix of the left and right hand control ends on the operation platform, the operation clamping angle, the emergency stop flag, the clutch flag, the control channel switching flag, and the handshaking flag. The system coordinator returns response confirmation information to the operation platform after verification is completed, as a sign of completion of handshaking between the two parties; After the handshaking is completed, the system coordinator switches the current control state to a motion preparation state, and sends a motion preparation signal to all single-arm controller instances to start the motion preparation process of each single-arm controller instance.

8. An electronic device, comprising: The electronic device comprises: One or more processors; and a memory having stored computer program instructions that, when executed, cause the processors to perform the virtual robot control state management method of any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program and / or instructions, characterized in that, The computer program and / or instructions, when executed by a processor, implement the virtual robot control state management method of any one of claims 1-7.

10. A computer program product comprising computer programs and / or instructions, characterized in that, The computer program and / or instructions, when executed by a processor, implement the virtual robot control state management method of any one of claims 1-7.

Images