An intraoperative semantic-driven surgical robot risk closed-loop control method and system

Abstract

The application discloses an intraoperative semantic driving surgical robot risk closed-loop control method and system, and belongs to the technical field of medical robot control. The method acquires an intraoperative endoscope image and an actuator state, extracts semantic features and determines a surgical stage, and calculates a dynamic risk level for each actuator arm. Three-dimensional distances are calculated by fusing depth information, point cloud reconstruction and actuator pose calculation, key anatomical structures are converted to a robot coordinate system through coordinate transformation to form a three-dimensional space risk area. According to anatomical structure risk, instrument action, surgical stage and recognition confidence, a safety margin is dynamically determined and a safety domain is constructed. A safety constraint is established by controlling a barrier function to pre-position safety correction of the original motion instruction. The corrected instruction is sent to the bottom controller through a real-time channel to form a closed loop. The application converts intraoperative semantics into three-dimensional space dynamic safety constraints to realize pre-position preventive safety control.

Description

Technical Field

[0001] This application relates to the field of medical robot control technology, specifically to a method and system for intraoperative semantic-driven closed-loop risk control of surgical robots. Background Technology

[0002] Most existing surgical robots employ a master-slave teleoperation control structure. The operator sends position, velocity, or force-related commands to the slave actuators via the master control mechanism, and the slave controller executes motion control based on preset parameters. Existing safety mechanisms typically include fixed speed limits, collision responses based on contact force thresholds, preset geometric boundary restrictions, and emergency stops for exceeding these limits. However, in actual surgery, the risks faced by the actuators are not constant but are closely related to the type of instrument in the current surgical field, the operational behavior, the type of anatomical structure, the stage of the procedure, and the spatial distance between the end effector and the target anatomical structure. The same motion speed carries different risks in the tissue mobilization and transection stages; different actuators perform different tasks at the same time, and their control constraint strengths should also differ.

[0003] Existing technologies typically only use intraoperative image analysis results for prompts, displays, or visualization overlays, failing to directly drive actuator parameter updates as semantic results as functional inputs to the control layer; or they merely use static mapping tables to set fixed upper limits according to instrument categories, failing to achieve multi-arm, multi-stage, continuously changing closed-loop control. More critically, existing technologies fail to effectively transform key anatomical structures such as blood vessels and nerves identified in endoscopic images into three-dimensional spatial safety constraints in the robot coordinate system: two-dimensional image recognition results lack fusion with depth perception and robot pose, making it difficult to accurately represent the three-dimensional position of danger zones in the robot's motion space; safety margins often use fixed values, failing to dynamically adjust based on the risk level of anatomical structures, instrument movements, surgical stage, and recognition confidence; safety interventions are mostly reactive responses through speed limits and emergency stops after contact or boundary breaches occur, lacking a mechanism for proactive safety correction at the command level through control barrier functions; and when recognition confidence decreases or input is missing due to obstruction or smoke, there is a lack of systematic conservative strategies related to safety boundary expansion, speed limits, and soft stops.

[0004] The existing technology has the following specific defects: (1) The control layer lacks intraoperative semantic perception capabilities. The existing controller inputs usually only include position, speed and force feedback, and cannot perceive the semantic information of instrument type, operation behavior and anatomical structure. (2) There is a lack of a mechanism to integrate the results of two-dimensional semantic recognition with depth and pose to form a three-dimensional spatial risk area in the robot coordinate system, resulting in imprecise spatial safety constraints; (3) The safety margin and safety boundary adopt fixed parameters and are not dynamically adjusted according to the anatomical structure risk, instrument operation, process stage and identification confidence. When there is perceived uncertainty, the safety boundary cannot be adaptively expanded. (4) Safety constraints are mostly reactive and lack a mechanism for pre-emptive correction at the motion command level through control barrier functions; (5) In multi-arm collaboration scenarios, existing technologies struggle to achieve independent calculation of risks and differentiated parameter adjustments for each arm; (6) The lack of a clear real-time control channel and parameter update mechanism between semantic feature extraction, risk level calculation and the underlying controller makes it easy to be limited to information processing rather than control invention. Summary of the Invention

[0005] To overcome the aforementioned shortcomings, the purpose of this application is to provide a surgical robot risk closed-loop control method and system driven by intraoperative semantics. By introducing intraoperative image semantic features, surgical procedure stage information, and spatial distance correction information into the underlying control layer, dynamic risk levels are generated for each actuator arm. The identified key anatomical structures are converted into three-dimensional spatial risk regions in the robot coordinate system through depth and pose fusion. The safety margin is dynamically determined and a safety domain is constructed based on the degree of anatomical structure risk, instrument movements, procedure stage, and recognition confidence. Safety constraints are established through control barrier functions to pre-correct the operator's original motion commands. The speed scaling factor, acceleration limit parameter, damping coefficient, workspace boundary, and emergency stop triggering condition of the actuator are adjusted in real time based on the risk level, thereby improving the adaptability of the control system to complex operation scenarios, its pre-emptive prevention capability, and the safety of multi-arm collaboration.

[0006] To achieve the above objectives, this application adopts the following technical solution: This application provides a method for intraoperative semantic-driven closed-loop risk control of surgical robots, comprising the following steps: Acquire real-time intraoperative image data and actuator status data captured by the surgical robot endoscope; Semantic features of retrieval instruments, operational behaviors, and anatomical structures for each actuator arm; Determine the current stage of the surgical procedure based on continuous image sequences; Based on the semantic features, process stages, and master-slave operation context, a dynamic risk level is calculated for each execution arm. The dynamic risk level is corrected by introducing a three-dimensional spatial distance between the end of the actuator arm and the target anatomical structure. The three-dimensional spatial distance is obtained by fusing endoscope depth information, point cloud reconstruction and actuator pose. The identified key anatomical structures are transformed into the robot coordinate system by the coordinate transformation of the depth information and actuator pose, forming a three-dimensional spatial risk area. The safety margin is dynamically determined based on the modified dynamic risk level, and a safety domain that changes with risk is constructed. The safety margin is dynamically adjusted according to the risk level of the anatomical structure, the risk of instrument movement, the risk of process stage, and the confidence level of semantic recognition. Based on the safety domain, safety constraints are established through a control barrier function. The control barrier function changes its inequality constraint boundary according to the dynamic risk level to realize the dynamic contraction or expansion of the safety domain. The original motion command input by the operator is then modified through the safety constraints so that the modified motion command satisfies the safety domain constraints. The safety-corrected motion commands are sent to the surgical robot's underlying controller through a real-time control channel. The underlying controller then updates the motion control parameters of the corresponding actuators in real time and obtains motion feedback to form a closed-loop control.

[0007] Furthermore, in the process of extracting semantic features for each actuator arm, the instrument type includes at least one of gripping, separating, energy, cutting, and stitching. The operation includes at least one of approaching, clamping, pulling, freeing, cutting, and suturing; The anatomical structures include at least one of the following: blood vessels, nerves, bile ducts, bronchi, parenchymal tissue, and serous membranes. The semantic features form an arm-level semantic feature vector, which is used to characterize the operational semantic state of the corresponding execution arm.

[0008] Furthermore, in the process of determining the current surgical procedure stage based on a continuous image sequence, the procedure stage includes at least one or more of the following: tissue mobilization stage, anatomical exposure stage, cutting stage, and suturing stage. The process stage determination uses a timing model to continuously determine stage switching and smooths the stage boundaries to avoid sudden changes in control parameters during stage switching.

[0009] Furthermore, the process of forming the three-dimensional spatial risk area includes: Key anatomical structures in intraoperative images are identified and their two-dimensional regions are obtained through semantic segmentation. By combining the endoscopic depth information, the two-dimensional region is reconstructed into a point cloud to obtain the three-dimensional spatial point cloud of the key anatomical structure. Based on the coordinate transformation relationship between the camera coordinate system and the robot coordinate system determined by the actuator pose, the three-dimensional point cloud is transformed into the robot coordinate system; The three-dimensional spatial point clouds after transformation of multiple consecutive frames are fused to form the three-dimensional spatial risk area of ​​the key anatomical structure in the robot coordinate system; The three-dimensional spatial risk region carries the corresponding anatomical structure type and risk level attributes, and the distance from the end of the execution arm to the boundary of the three-dimensional spatial risk region is used as the three-dimensional spatial distance.

[0010] Furthermore, in the process of dynamically determining the safety margin, the safety margin is positively correlated with the risk level of the anatomical structure, the risk of instrument movement, and the risk of the process stage, and negatively correlated with the confidence level of semantic recognition. The safety margin corresponds to the amount of shrinkage of the workspace boundary. When the safety margin increases, the workspace boundary shrinks accordingly and the safety domain shrinks. The higher the dynamic risk level, the tighter the inequality constraint boundary of the control barrier function, and the greater the degree of safety domain contraction.

[0011] Furthermore, it also includes adaptive safety boundary adjustment for uncertainty: When the confidence level of semantic recognition is lower than the first confidence threshold, the safety margin is increased to expand the safety boundary; When the semantic recognition confidence level is lower than the second confidence threshold, speed scaling limits are imposed on motion commands while increasing the safety margin. When key anatomical structures are not identified for several consecutive frames, a soft stop is achieved by gradually reducing the speed scaling factor, causing the actuator to gradually decelerate to zero speed. When key inputs are missing or sensors fail, the system enters a conservative degradation mode, maintaining the risk level of the previous cycle, using higher damping, tightening the workspace boundaries, and prohibiting automatic release of the emergency stop state. When surgical field occlusion or surgical smoke leads to a decrease in semantic recognition confidence or the absence of key inputs, the corresponding safety boundary adjustment or conservative downgrade is triggered.

[0012] Furthermore, the process of establishing safety constraints and making safety corrections to the original motion commands by controlling the barrier function includes: The control barrier function uses the relationship between the distance from the end of the execution arm to the boundary of the three-dimensional risk area and the safety margin as an inequality constraint. When the original motion command input by the operator causes the end of the actuator arm to tend to intrude into the safety margin range, the control barrier function corrects the original motion command through its inequality constraints, restricts the motion component of the actuator arm toward the risk area in three-dimensional space, and keeps the corrected motion command within the safety domain. When the original motion command satisfies the inequality constraint, the original motion command is not modified. The safety modification preemptively restricts the movement of the actuator before a contact event occurs.

[0013] Furthermore, the dynamic risk level is calculated independently for each of the multiple actuators, a three-dimensional spatial risk region is formed, the safety margin is determined and the safety constraint is corrected, and differentiated motion commands for each actuator are output. Different actuators obtain different risk levels and control adjustment signals based on their semantic features, process stages and spatial distances. High-risk actuators are subject to stronger speed limits, enhanced damping and shrinking workspace boundaries, while low-risk actuators maintain more relaxed control parameters.

[0014] Furthermore, the initial risk value of the dynamic risk level is calculated as follows: R0_j(t) = w_s·Sscore_j(t) + w_p·Pscore(t) + w_i·Iscore_j(t), where R0_j(t) represents the initial risk value of the j-th execution arm at time t, Sscore_j(t) represents the arm-level semantic risk score, Pscore(t) represents the process stage risk weight, Iscore_j(t) represents the interactive correction term obtained from the master-slave operation context, the relative position relationship between the two arms, or the multi-arm collaborative load, and w_s, w_p, and w_i are configurable weights; The final risk value is obtained by applying a historical smoothing mechanism to the initial risk value: R_j(t) = λ·R0_j(t) + (1-λ)·R_j(t-1), where λ is the smoothing coefficient and 0 < λ ≤ 1; The real-time control channel is at least one of the following: shared memory interface, industrial Ethernet, fieldbus, and ROS2 DDS channel.

[0015] This application also provides an intraoperative semantic-driven surgical robot risk closed-loop control system for implementing any of the intraoperative semantic-driven surgical robot risk closed-loop control methods described in this application, comprising: The image acquisition module is used to acquire real-time intraoperative image data and actuator status data; The semantic feature extraction module is used to extract semantic features of instrument type, operation behavior and anatomical structure for each actuator arm; The process stage determination module is used to determine the current process stage of the surgery based on a continuous image sequence; The multi-arm risk calculation module is used to calculate the dynamic risk level of each execution arm based on the semantic features, process stages, and master-slave operation context. The distance fusion module is used to fuse endoscope depth information, point cloud reconstruction and actuator pose, calculate the three-dimensional spatial distance between the end of the actuator arm and the target anatomical structure, and transform the identified key anatomical structures into the robot coordinate system to form a three-dimensional spatial risk area. The control parameter generation module is used to dynamically determine the safety margin and construct the safety domain based on the dynamic risk level, establish safety constraints through the control barrier function, and make safety corrections to the operator's original motion commands, thereby generating control adjustment signals. The actuator control module is used to send safety-corrected motion commands and control adjustment signals to the underlying controller and update the actuator motion control parameters in real time. The feedback and monitoring module is used to obtain motion feedback from the actuator to form closed-loop control. Beneficial effects

[0016] Compared with the prior art, the present invention has the following beneficial effects: By fusing depth information, point cloud reconstruction, and actuator pose with key anatomical structures identified in endoscopic images and transforming their coordinates, the resulting structures are converted into three-dimensional spatial risk areas in the robot coordinate system. This gives the two-dimensional semantic recognition results three-dimensional spatial safety constraints, moving beyond simply providing display prompts.

[0017] Safety margins are dynamically determined based on the degree of risk of anatomical structures, instrument movement risks, process stage risks, and identification confidence, rather than using a fixed parameter table. This allows the safety domain to contract or expand in real time according to the intraoperative scenario, ensuring safety while avoiding unnecessary operational restrictions.

[0018] By establishing safety constraints through the control barrier function and pre-correcting the operator's original motion commands, safety intervention is moved from post-event speed limits and emergency stops to the command level. No intervention is given when the original command is safe, and the motion component toward the risk area is automatically restricted when the original command tends to be dangerous, thus balancing operational freedom and safety.

[0019] The system directly links the recognition confidence level and uncertainties such as occlusion, smoke, and target loss with the expansion of safety margin, speed scaling, soft stop, and conservative degradation. When uncertainty is perceived, the system automatically tends to be conservative to avoid the loss of safety constraints due to recognition errors or failures.

[0020] It supports independent risk calculation for multiple execution arms, construction of independent security domains, and adjustment of differentiated parameters, thereby enhancing security in multi-arm collaborative scenarios.

[0021] By ensuring a smooth timing transition, transient instability in control during stage switching is reduced, thus guaranteeing the continuity of actuator movement and consistency of operator feel. Attached Figure Description

[0022] The accompanying drawings are provided to illustrate the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure. The shapes and sizes of the components in the drawings do not reflect actual proportions and are only intended to illustrate the content of this application.

[0023] Figure 1 This is the overall system architecture diagram for this application.

[0024] Figure 2 This is a flowchart of the method used in this application.

[0025] Figure 3 This application presents a multi-arm independent risk closed-loop diagram.

[0026] Figure 4 This is a mapping diagram of risk level and control parameters for this application.

[0027] Figure 5 This is a distance correction and control barrier constraint diagram for this application.

[0028] Figure 6 This is the timing continuous stage switching control chart for this application. Detailed Implementation

[0029] The above-described solution will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of this application. The implementation conditions used in the embodiments may be further adjusted according to the conditions of specific manufacturers, and the implementation conditions not specified are generally those in routine experiments.

[0030] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In this document, "electrical connection" includes the situation where constituent elements are connected together by an element having some electrical function. There is no particular limitation on the "electrically functioning element," as long as it enables the transmission and reception of electrical signals between the connected constituent elements. An "electrically functioning element" can be, for example, an electrode or wiring, a switching element such as a transistor, or other functional elements such as a resistor, inductor, or capacitor. "Up," "down," "left," and "right" are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0031] In this application, the terms "upper," "lower," "inner," "middle," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0032] This invention constructs a semantic-aware control closed loop with intraoperative images as input and dynamic updates of actuator parameters as output. This closed loop uses the results of image semantic feature extraction, surgical procedure stage determination, and spatial distance information between the actuator arm and the target anatomical structure as joint inputs. The multi-arm risk calculation module outputs the dynamic risk level of each actuator arm, and the control parameter generation module maps these risk levels into control adjustment signals, which are then sent to the underlying controller via a real-time control channel to update the control parameters of each actuator. Specifically, the identified key anatomical structures are fused with depth information, point cloud reconstruction, and actuator pose, and then transformed using coordinates to form a three-dimensional spatial risk region in the robot coordinate system. The safety margin is dynamically determined based on the degree of anatomical structure risk, instrument movements, procedure stage, and recognition confidence. The control barrier function data changes the inequality constraint boundary to dynamically shrink or expand the safety domain and performs pre-operator safety corrections to the operator's original motion commands. The dynamic risk level and safety correction results are not for display or prompting, but serve as functional input variables of the control layer, directly driving changes in control parameters such as speed scaling, acceleration limiting, damping enhancement, workspace boundary contraction, and emergency stop triggering, thereby forming closed-loop control. Example

[0033] like Figure 2 As shown, the method of the present invention includes the following steps: Step S1: Input Acquisition and Synchronization First, the intraoperative image sequence acquired by the surgical robot's endoscope is obtained in real time. Simultaneously, the actuator pose, end effector velocity, master input, force feedback, and real-time control cycle number are acquired. Then, the image frames, control states, and pose information are uniformly time-aligned to form a joint state record with the control cycle as the granularity. The control cycle of the surgical robot is typically in the millisecond range; information synchronization ensures that all inputs correspond to the same moment, avoiding delays in risk level calculations and safety constraint updates due to information asynchrony.

[0034] Step S2: Semantic Feature Extraction For each control cycle t and each execution arm j, three semantic features—instrument type, operational behavior, and anatomical structure—are extracted from the local surgical field to form an arm-level semantic feature vector S_j(t). The instrument type may include categories such as grasping, separating, energy, cutting, and suturing; the operational behavior may include categories such as approaching, clamping, pulling, freeing, cutting, and suturing; and the anatomical structure may include categories such as blood vessels, nerves, bile ducts, bronchi, parenchymal tissue, and serous surfaces.

[0035] The semantic feature extraction module includes an instrument detection submodule, a behavior recognition submodule, and an anatomical structure recognition submodule. Instrument detection can use an object detection network to output instrument category and bounding box; behavior recognition can determine the type of operation behavior based on local temporal frames; anatomical structure recognition can output structure category and region based on semantic segmentation or classification networks. Each submodule simultaneously outputs recognition confidence, which is used for subsequent dynamic adjustment of safety margin and uncertainty adaptive strategies.

[0036] Step S3: Process Stage Determination Based on a continuous image sequence or an image sequence and operational context, the current surgical procedure stage is determined to form a stage variable P(t). The procedure stage includes at least one or more of the following: tissue mobilization stage, anatomical exposure stage, cutting stage, and suturing stage. The stage determination module uses a temporal model to continuously determine stage transitions and smooths stage boundaries to avoid abrupt changes in control parameters during stage transitions.

[0037] Step S4: Multi-arm Independent Risk Calculation Calculate the dynamic risk level for each actuator arm. The initial risk value R0_j(t) of the j-th actuator arm at time t can be calculated as follows: R0_j(t) = w_s·Sscore_j(t) + w_p·Pscore(t) + w_i·Iscore_j(t) Here, Sscore_j(t) represents the arm-level semantic risk score, obtained based on the extracted semantic features. Different semantic features correspond to different risks: energy cutting devices have higher risk scores, while gripping devices have lower risk scores; cutting operations have higher risk scores, while traction operations have lower risk scores; key anatomical structures such as blood vessels and nerves have higher risk scores, while solid tissues have lower risk scores. Pscore(t) represents the risk weight of each process stage, with higher risk weights in the cutting stage and lower risk weights in the freeing stage. Iscore_j(t) represents the interactive correction term obtained from the master-slave operation context, the relative positional relationship between the two arms, or the multi-arm collaborative load; the risk increases when the two arms are closer together. w_s, w_p, and w_i are configurable weights.

[0038] To avoid sudden changes in the risk level, a historical smoothing mechanism can be adopted: R_j(t) = λ·R0_j(t) + (1-λ)·R_j(t-1), where 0<λ ≤ 1. The risk level can be divided into discrete levels, or a continuous risk quantity can be used.

[0039] Step S5: Formation of three-dimensional spatial risk region and distance correction To enhance the spatial sensitivity of risk calculation and convert the two-dimensional semantic recognition result into the three-dimensional spatial safety constraint of the robot, the three-dimensional spatial distance d_j(t) between the end of the execution arm and the target anatomical structure is introduced to correct the risk level and form a three-dimensional spatial risk region, as Figure 3 shown in the figure.

[0040] First, a semantic segmentation network is used to identify key anatomical structures (such as blood vessels, nerves, etc.) in the intraoperative image, and obtain their regions in the two-dimensional image. Combined with the endoscope depth information (which can be obtained by binocular stereo matching, monocular depth estimation or structured light), point cloud reconstruction is performed on the two-dimensional region to obtain the three-dimensional spatial point cloud of the key anatomical structure.

[0041] Since the point cloud data is in the camera coordinate system, while the robot motion control is performed in the robot base coordinate system, the three-dimensional spatial point cloud is converted to the robot coordinate system based on the coordinate transformation relationship between the camera coordinate system and the robot coordinate system determined by the current pose of the execution mechanism. The converted point clouds of consecutive multiple frames are fused to eliminate region incompleteness caused by single-frame observation noise and occlusion, so as to form the three-dimensional spatial risk region Z_k of the key anatomical structure in the robot coordinate system. The three-dimensional spatial risk region carries the corresponding anatomical structure type and risk degree attributes.

[0042] The shortest distance from the end of the execution arm to the boundary of the three-dimensional spatial risk region is the three-dimensional spatial distance d_j(t). When d_j(t) is less than the first distance threshold d1, a first correction amount is added to the risk value; when d_j(t) is less than the second distance threshold d2 and d2<d1, a second correction amount is added to the risk value, where the second correction amount is greater than the first correction amount. Preferred example: when d_j(t)<5mm, the risk level is raised by one level; when d_j(t)<3mm, the risk level is raised by two levels or the highest risk control is directly triggered.

[0043] Step S6: Dynamic safety margin and safety domain construction Based on the corrected dynamic risk level, the safety margin m(t) is dynamically determined and a risk-varying safety domain is constructed, as Figure 4 shown in the figure.

[0044] The safety margin is dynamically adjusted based on the following factors: (1) the risk level of the anatomical structure, with a large safety margin for key blood vessels, nerves and other structures with high risk; (2) the risk of the instrument's actions, with a large safety margin for energy cutting actions with high risk, and a small safety margin for grasping actions with low risk; (3) the risk of the process stage, with a large safety margin for the cutting stage and a small safety margin for the freeing stage; (4) the confidence level of semantic recognition, with a larger safety margin for lower confidence levels in order to cope with positioning uncertainty.

[0045] Safety margin corresponds to the shrinkage of the workspace boundary: as the safety margin increases, the workspace boundary shrinks away from the danger zone, and the safe area for robot movement shrinks accordingly. In a preferred embodiment, the dynamic risk level can be quantified into multiple levels, each level corresponding to a set of safety margins and control parameters; alternatively, a continuous safety margin can be directly used.

[0046] A safety domain is defined as the set of positions in the robot's motion space where the distance from the end effector of the actuator to the boundary of each three-dimensional spatial risk region is not less than the corresponding safety margin. The safety domain dynamically shrinks or expands as the risk level changes: when the risk increases, the safety margin increases and the safety domain shrinks; when the risk decreases, the safety margin decreases and the safety domain expands.

[0047] Step S7: Security instruction correction based on control barrier function Based on the safety domain, safety constraints are established through control barrier functions (CBFs) to safely modify the original motion commands input by the operator, such as... Figure 4 As shown.

[0048] The control barrier function uses the relationship between the distance from the end of the execution arm to the boundary of the three-dimensional risk region and the safety margin as an inequality constraint. The control barrier function changes its inequality constraint boundary according to the dynamic risk level: when the risk level increases, the safety margin increases, the inequality constraint boundary tightens, and the safety region shrinks; when the risk level decreases, the constraint boundary relaxes, and the safety region expands.

[0049] When the operator inputs the initial motion command through the master terminal, causing the end effector to tend towards intruding into the safety margin range (i.e., moving towards the three-dimensional risk area and approaching the safety margin), the control barrier function modifies the initial motion command through its inequality constraints, limiting the motion component of the end effector towards the risk area and keeping the modified motion command within the safety domain. When the initial motion command satisfies the inequality constraints, no modification is made to the initial motion command, and the operator's operation is unaffected.

[0050] Therefore, safety intervention preemptively restricts the movement of the actuator before a contact event occurs, rather than responding only after contact or boundary breach has occurred. In addition to control barrier functions, equivalent safety control constraints such as artificial potential fields, safety domain projection methods, or model-predicted safety constraints can also be used to achieve the same function.

[0051] Based on the safety constraint correction, the control parameter generation module also outputs control adjustment signals such as speed scaling factor, acceleration limit parameter, damping enhancement coefficient, workspace constraint parameter and emergency stop trigger flag according to the risk level, which work together with the safety constraint correction to control the actuator.

[0052] Step S8: Uncertainty Adaptive Safety Boundary Adjustment and Degradation Strategy This step directly links semantic recognition confidence and surgical field quality with safety boundary adjustments, such as... Figure 6 As shown.

[0053] When the semantic recognition confidence level is lower than the first confidence level threshold, it indicates that there is some uncertainty in the location of dangerous areas. At this time, the safety margin is increased to expand the safety boundary and keep the robot at a greater distance from the risk area.

[0054] When the semantic recognition confidence level is lower than the second confidence level threshold (lower than the first threshold), while increasing the safety margin, the motion command is limited by the speed scaling factor to reduce the motion speed of the execution arm.

[0055] When a critical anatomical structure is not identified for several consecutive frames (i.e., the target is lost), the system achieves a soft stop by gradually reducing the speed scaling factor: the speed scaling factor is gradually reduced over several control cycles, causing the actuator to gradually decelerate to zero speed and maintain its position, rather than stopping abruptly, thus avoiding traction damage to the tissue caused by the inertia of a sudden stop.

[0056] When critical inputs are missing (such as image transmission interruption) or sensors fail, the system does not directly relax control, but enters a conservative degradation mode: maintaining the risk level of the previous cycle, using higher damping, tightening the workspace boundaries, and prohibiting automatic release of the emergency stop state, which can only be restored after operator confirmation.

[0057] When surgical field obstruction or surgical smoke occurs during surgery, it can lead to a decrease in semantic recognition confidence or the loss of key inputs. Based on this, the system will automatically trigger the corresponding safety boundary adjustment or conservative degradation strategy mentioned above.

[0058] Step S9: Differentiated Safety Control for Multiple Actuators For surgical robots with multiple actuators, such as Figure 5As shown, each actuator independently executes steps S2 to S8: each arm extracts semantic features based on its own local surgical field, calculates the risk level, forms a three-dimensional spatial risk area, determines the safety margin, and corrects the safety command through the control barrier function, outputting differentiated motion commands and control adjustment signals for each actuator.

[0059] Different actuators have different risk levels and safety margins due to differences in instrument type, operation behavior and adjacent anatomical structures: high-risk actuators are subject to stronger speed limits, enhanced damping and shrinking workspace boundaries, while low-risk actuators maintain more relaxed control parameters, thereby ensuring the safety of high-risk operations while maintaining the efficiency of low-risk operations.

[0060] Step S10: Actuator control closed loop The safety-corrected motion commands and control adjustment signals are sent to the robot's underlying controller via a real-time control channel. This real-time control channel can be a shared memory interface, industrial Ethernet, fieldbus, ROS2 DDS channel, or an equivalent real-time communication mechanism, ensuring that command transmission and parameter updates are completed within the control cycle.

[0061] In each control cycle, the underlying controller receives correction commands and control adjustment signals from each actuator, updating the corresponding actuator's speed limit, acceleration constraints, damping parameters, and workspace boundary constraints. After execution, the actuator returns posture, velocity, acceleration, and out-of-bounds status feedback, which serves as one of the inputs for the next control cycle, forming a closed-loop control.

[0062] Example Scenario 1: Differentiated Control in Multi-Arm Cooperative Isolation-Disconnection Scenarios In this embodiment, as Figure 5 As shown, a three-arm surgical robot platform is used: arm 1 is responsible for tissue traction, arm 2 is responsible for energy cutting, and arm 3 is responsible for maintaining the field of vision.

[0063] The system extracts semantic features from the local surgical field of each actuator arm. For actuator arm 2, the instrument type is identified as an energy device, the operation is cutting, the anatomical structure is a blood vessel, and the process stage is determined to be the cutting stage. The distance fusion module reconstructs the blood vessels identified in the image into a three-dimensional spatial risk region in the robot coordinate system through depth information and pose coordinate transformation. Due to the high risk of blood vessels, high risk of energy cutting action, and high risk of the cutting stage, a large dynamic safety margin is adopted, and the safety domain shrinks.

[0064] When the operator manipulates the actuator arm 2 to move towards the blood vessel, the control barrier function constrains and restricts its motion component toward the blood vessel, while simultaneously outputting strong velocity limiting and damping enhancement parameters. In a specific parameter configuration of this embodiment, the velocity scaling factor of actuator arm 2 can be 0.50, and the damping coefficient can be 1.5.

[0065] Meanwhile, actuator arm 1 detects the pulling action but is far from the high-risk structure, resulting in a small safety margin. The original command satisfies the CBF constraint and requires no correction; the speed scaling factor can be set to 0.80, allowing the operator to pull normally. Actuator arm 3 maintains the field of view, carries lower risk, and maintains normal control parameters. This achieves differentiated safety control for each actuator arm in a multi-arm collaborative scenario.

[0066] Example Scenario 2: High-risk anatomical proximity control based on 3D spatial risk area and CBF In this embodiment, the doctor is using an actuation arm to detach tissue, and the end of the actuation arm is approaching a key nerve.

[0067] First, the semantic segmentation network identifies the nerve in the image and obtains a two-dimensional region. Point cloud reconstruction is then performed using endoscopic depth information to obtain a three-dimensional point cloud of the nerve. Based on the actuator pose, the point cloud is transformed to the robot's base coordinate system, and after multi-frame fusion, a three-dimensional spatial risk region Z_k of the nerve is formed.

[0068] The distance fusion module calculates the three-dimensional distance d_j(t) from the end of the execution arm to the Z_k boundary in real time. Initially, d_j(t) is large (greater than 5mm), the distance correction is not effective, the safety margin is small, the original instruction satisfies the CBF constraint, and the operator operates normally.

[0069] As the distal end approaches the nerve, when d_j(t) < 5 mm, the risk level increases by one level, the safety margin increases, the CBF inequality constraint boundary tightens, and the safety domain shrinks. If the operator continues to issue commands toward the nerve, the CBF begins to limit the normal motion component toward the nerve, the distal end approach speed automatically decreases, and the speed scaling factor can be reduced to 0.80, the damping coefficient increases to 1.2, and the workspace boundary shrinks by 2 mm.

[0070] When d_j(t) < 3mm, the risk level increases by two levels or directly enters the highest risk control, further increasing the safety margin. CBF constraint limits the motion component towards the nerve to near zero, and the distal end is pre-constrained at the safety domain boundary without touching the nerve. Simultaneously, the velocity scaling factor can be reduced to 0.20, the damping coefficient increased to 2.0, the workspace boundary shrinks by 8mm, and an emergency stop flag is triggered. The entire process completes pre-constraint before contact occurs, rather than responding after contact.

[0071] Example Scenario 3: Uncertainty Adaptive Safety Boundary and Soft Stop In this embodiment, the smoke generated by the ultrasonic scalpel during the operation leads to a decrease in the quality of the endoscopic image, and the semantic segmentation network's confidence in recognizing the vascular region decreases.

[0072] When the confidence level falls below the first confidence threshold, the system automatically increases the safety margin of the blood vessel region, expanding the safety boundary to provide a larger buffer distance for the actuator arm. When the smoke intensifies and the confidence level falls further below the second confidence threshold, the system continues to expand the safety margin while applying a speed scaling limit to reduce the end effector's movement speed.

[0073] When smoke completely obscures the field of vision and blood vessels are not detected for several consecutive frames, the system triggers a soft stop: the speed scaling factor gradually decreases over several control cycles, the actuator arm gradually decelerates to zero speed along the feasible CBF direction and maintains its position, avoiding sudden stops that could cause the instrument to bounce and damage tissue. Once the smoke dissipates, recognition is restored, and confidence recovers, the system automatically resumes normal control.

[0074] If the depth sensor suddenly malfunctions and causes a loss of critical input, the system enters a conservative degradation mode: maintaining the current risk level, increasing the damping coefficient, tightening the workspace boundaries, and prohibiting automatic release of the emergency stop state. The system can only be restored after manual confirmation by the operator.

[0075] Example Scenario 4: Continuous parameter transition during stage switching In this embodiment, the surgery switches from the tissue mobilization phase to the transection phase. During the mobilization phase, the risk weight of the process stage is low, the safety margin is small, and the control parameters are lenient; after entering the transection phase, the risk weight of the process stage increases, the safety margin increases, and the CBF constraint is strengthened.

[0076] Instead of abruptly switching control parameters, the system employs a historical smoothing mechanism for risk values, allowing safety margins and control parameters to gradually transition over several consecutive control cycles. This results in smooth changes in the actuator arm movement, eliminating the operator's perception of sudden parameter changes. This smoothing mechanism also prevents control transient instability caused by jitter in the stage-based decision model at boundary conditions.

[0077] In one specific parameter configuration of this embodiment, the preferred mapping relationship between risk level and control parameters is shown in the table below. The values ​​in the table are only preferred embodiments and are not intended to limit the scope of protection:

[0078] The above mapping table is a preferred embodiment. In practical applications, it can be configured individually according to different instrument categories, different process stages, or different actuators.

[0079] like Figure 1 As shown, the system of the present invention includes the following modules: The image acquisition module is used to acquire intraoperative image sequences in real time and output them to the semantic processing link, while acquiring status data such as the pose, velocity, and force feedback of the actuator.

[0080] The semantic feature extraction module is used to extract semantic information such as instrument type, operation behavior and anatomical structure and recognition confidence, forming an arm-level semantic feature vector.

[0081] The process stage determination module is used to determine the current process stage based on image timing or image-context joint information, and to smooth the stage boundaries.

[0082] The multi-arm risk calculation module is used to independently calculate the dynamic risk level of each execution arm and perform smoothing and correction.

[0083] The distance fusion module is used to fuse endoscope depth information, point cloud reconstruction and actuator pose, calculate the three-dimensional spatial distance between the end effector and the target anatomical structure, and transform the identified key anatomical structures into a three-dimensional spatial risk area in the robot coordinate system.

[0084] The control parameter generation module is used to dynamically determine the safety margin and construct the safety domain based on the dynamic risk level. It establishes safety constraints through the control barrier function to make safety corrections to the original motion commands and generates control adjustment signals such as speed, acceleration, damping, boundary and emergency stop.

[0085] The actuator control module is used to send correction commands and control adjustment signals to the underlying controller through a real-time control channel, and update the actuator motion control parameters in real time.

[0086] The feedback and monitoring module is used to acquire motion feedback, out-of-bounds status, and control execution status of the actuator, forming a closed loop, and using the actuator status feedback as the input for the next cycle.

[0087] In summary, the intraoperative semantic-driven risk closed-loop control method and system for surgical robots of the present invention transforms the intraoperative semantic recognition results into a three-dimensional spatial risk region in the robot coordinate system through depth and pose fusion. It dynamically determines the safety margin based on multiple factors and constructs a safety domain. It performs pre-emptive safety correction on the original motion commands through a control barrier function and correlates the recognition uncertainty with the adaptive adjustment of the safety boundary and soft-stop degradation level. This forms a complete technical chain of "semantic perception—three-dimensional risk region—dynamic safety domain—command correction—closed-loop feedback," effectively solving problems in existing technologies such as the disconnect between semantic information and the control layer, fixed safety constraints and post-event responses, and the lack of conservative strategies when perception is uncertain. It has high practical application value.

[0088] The above embodiments are only for illustrating the technical concept and features of this application, and are intended to enable those skilled in the art to understand the content of this application and implement it accordingly. They should not be used to limit the scope of protection of this application. All equivalent changes or modifications made in accordance with the spirit and essence of this application should be included within the scope of protection of this application.

Claims

1. A method for intraoperative semantic-driven closed-loop risk control of a surgical robot, characterized in that, Includes the following steps: Acquire real-time intraoperative image data and actuator status data captured by the surgical robot endoscope; Semantic features of retrieval instruments, operational behaviors, and anatomical structures for each actuator arm; Determine the current stage of the surgical procedure based on continuous image sequences; Based on the semantic features, process stages, and master-slave operation context, a dynamic risk level is calculated for each execution arm. The dynamic risk level is corrected by introducing a three-dimensional spatial distance between the end of the actuator arm and the target anatomical structure. The three-dimensional spatial distance is obtained by fusing endoscope depth information, point cloud reconstruction and actuator pose. The identified key anatomical structures are transformed into the robot coordinate system by the coordinate transformation of the depth information and actuator pose, forming a three-dimensional spatial risk area. The safety margin is dynamically determined based on the modified dynamic risk level, and a safety domain that changes with risk is constructed. The safety margin is dynamically adjusted according to the risk level of the anatomical structure, the risk of instrument movement, the risk of process stage, and the confidence level of semantic recognition. Based on the safety domain, safety constraints are established through a control barrier function. The control barrier function changes its inequality constraint boundary according to the dynamic risk level to realize the dynamic contraction or expansion of the safety domain. The original motion command input by the operator is then modified through the safety constraints so that the modified motion command satisfies the safety domain constraints. The safety-corrected motion commands are sent to the surgical robot's underlying controller through a real-time control channel. The underlying controller then updates the motion control parameters of the corresponding actuators in real time and obtains motion feedback to form a closed-loop control.

2. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: In the process of extracting semantic features for each actuator arm, the instrument type includes at least one of gripping, separating, energy, cutting, and stitching. The operation includes at least one of approaching, clamping, pulling, freeing, cutting, and suturing; The anatomical structures include at least one of the following: blood vessels, nerves, bile ducts, bronchi, parenchymal tissue, and serous membranes. The semantic features form an arm-level semantic feature vector, which is used to characterize the operational semantic state of the corresponding execution arm.

3. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: In the process of determining the current surgical procedure stage based on a continuous image sequence, the procedure stage includes at least one or more of the following: tissue mobilization stage, anatomical exposure stage, cutting stage, and suturing stage. The process stage determination uses a timing model to continuously determine stage switching and smooths the stage boundaries to avoid sudden changes in control parameters during stage switching.

4. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: The process of forming a three-dimensional spatial risk zone includes: Key anatomical structures in intraoperative images are identified and their two-dimensional regions are obtained through semantic segmentation. By combining the endoscopic depth information, the two-dimensional region is reconstructed into a point cloud to obtain the three-dimensional spatial point cloud of the key anatomical structure. Based on the coordinate transformation relationship between the camera coordinate system and the robot coordinate system determined by the actuator pose, the three-dimensional point cloud is transformed into the robot coordinate system; The three-dimensional spatial point clouds after transformation of multiple consecutive frames are fused to form the three-dimensional spatial risk area of ​​the key anatomical structure in the robot coordinate system; The three-dimensional spatial risk region carries the corresponding anatomical structure type and risk level attributes, and the distance from the end of the execution arm to the boundary of the three-dimensional spatial risk region is used as the three-dimensional spatial distance.

5. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: In the process of dynamically determining the safety margin, the safety margin is positively correlated with the risk level of the anatomical structure, the risk of instrument movement, and the risk of the process stage, and negatively correlated with the confidence level of semantic recognition. The safety margin corresponds to the amount of shrinkage of the workspace boundary. When the safety margin increases, the workspace boundary shrinks accordingly and the safety domain shrinks. The higher the dynamic risk level, the tighter the inequality constraint boundary of the control barrier function, and the greater the degree of safety domain contraction.

6. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 5, characterized in that: It also includes adaptive safety boundary adjustment for uncertainty: When the confidence level of semantic recognition is lower than the first confidence threshold, the safety margin is increased to expand the safety boundary; When the semantic recognition confidence level is lower than the second confidence threshold, speed scaling limits are imposed on motion commands while increasing the safety margin. When key anatomical structures are not identified for several consecutive frames, a soft stop is achieved by gradually reducing the speed scaling factor, causing the actuator to gradually decelerate to zero speed. When key inputs are missing or sensors fail, the system enters a conservative degradation mode, maintaining the risk level of the previous cycle, using higher damping, tightening the workspace boundaries, and prohibiting automatic release of the emergency stop state. When surgical field occlusion or surgical smoke leads to a decrease in semantic recognition confidence or the absence of key inputs, the corresponding safety boundary adjustment or conservative downgrade is triggered.

7. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: The process of establishing safety constraints and making safety corrections to the original motion commands by controlling the barrier function includes: The control barrier function uses the relationship between the distance from the end of the execution arm to the boundary of the three-dimensional risk area and the safety margin as an inequality constraint. When the original motion command input by the operator causes the end of the actuator arm to tend to intrude into the safety margin range, the control barrier function corrects the original motion command through its inequality constraints, restricts the motion component of the actuator arm toward the risk area in three-dimensional space, and keeps the corrected motion command within the safety domain. When the original motion command satisfies the inequality constraint, the original motion command is not modified. The safety modification preemptively restricts the movement of the actuator before a contact event occurs.

8. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: The system independently calculates the dynamic risk level for each of the multiple actuators, forms a three-dimensional spatial risk region, determines the safety margin, and corrects the safety constraints, outputting differentiated motion commands for each actuator. Different actuators obtain different risk levels and control adjustment signals based on their semantic features, process stages and spatial distances. High-risk actuators are subject to stronger speed limits, enhanced damping and shrinking workspace boundaries, while low-risk actuators maintain more relaxed control parameters.

9. The intraoperative semantic-driven surgical robot risk closed-loop control method according to claim 1, characterized in that: The initial risk value of the dynamic risk level is calculated as follows: R0_j(t) = w_s·Sscore_j(t) + w_p·Pscore(t) + w_i·Iscore_j(t), where R0_j(t) represents the initial risk value of the j-th execution arm at time t, Sscore_j(t) represents the arm-level semantic risk score, Pscore(t) represents the process stage risk weight, Iscore_j(t) represents the interactive correction term obtained from the master-slave operation context, the relative position relationship between the two arms, or the multi-arm collaborative load, and w_s, w_p, and w_i are configurable weights; The final risk value is obtained by applying a historical smoothing mechanism to the initial risk value: R_j(t) = λ·R0_j(t) + (1-λ)·R_j(t-1), where λ is the smoothing coefficient and 0 < λ ≤ 1; The real-time control channel is at least one of the following: shared memory interface, industrial Ethernet, fieldbus, and ROS2 DDS channel.

10. A surgical robot risk closed-loop control system driven by intraoperative semantics, used to implement the surgical robot risk closed-loop control method driven by intraoperative semantics as described in any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire real-time intraoperative image data and actuator status data; The semantic feature extraction module is used to extract semantic features of instrument type, operation behavior and anatomical structure for each actuator arm; The process stage determination module is used to determine the current process stage of the surgery based on a continuous image sequence; The multi-arm risk calculation module is used to calculate the dynamic risk level of each execution arm based on the semantic features, process stages, and master-slave operation context. The distance fusion module is used to fuse endoscope depth information, point cloud reconstruction and actuator pose, calculate the three-dimensional spatial distance between the end of the actuator arm and the target anatomical structure, and transform the identified key anatomical structures into the robot coordinate system to form a three-dimensional spatial risk area. The control parameter generation module is used to dynamically determine the safety margin and construct the safety domain based on the dynamic risk level, establish safety constraints through the control barrier function, and make safety corrections to the operator's original motion commands, thereby generating control adjustment signals. The actuator control module is used to send safety-corrected motion commands and control adjustment signals to the underlying controller and update the actuator motion control parameters in real time. The feedback and monitoring module is used to obtain motion feedback from the actuator to form closed-loop control.

Images