Dynamic decision-making method for surgical robot

Abstract

The invention provides a dynamic decision-making method for a surgical robot, and the method comprises the steps: obtaining surgical video data and a corresponding surgical gesture tag sequence in the operation process of the surgical robot, and generating a structured text description based on the gesture tag sequence; respectively extracting a visual feature sequence and a text feature sequence for the operation video data and the structured text description, carrying out cross-modal semantic alignment on the visual feature sequence and the text feature sequence, carrying out action segmentation on obtained fusion features, and dynamically integrating multi-scale spatial-temporal features through an adaptive weight learning mechanism to obtain a predicted operation action sequence; and inputting the result into a preset causal rule device, performing causal time sequence inspection, correcting a prediction result when a causal rule is violated, and giving causal explanation of the detected operation state transition point and the abnormal event. According to the method, fuzziness of semantic understanding and adaptivity of multi-modal fusion can be cooperatively solved, a causal test mechanism is introduced, and a reliable cognitive basis is provided for autonomous decision making of the surgical robot.

Description

Technical Field

[0001] This invention belongs to the field of surgical robot technology, and specifically relates to a dynamic decision-making method for surgical robots based on multimodal semantic alignment perception and causal temporal inference. It is mainly applied to the autonomous cognition and operational decision-making of surgical robots and surgical quality assessment systems. Background Technology

[0002] Advanced autonomous operation capabilities of surgical robots require precise perception of the surgical scene as a fundamental support. Clinical practice shows that the decision-making quality of surgical robots is closely related to the accuracy of intraoperative gesture recognition; accurate state perception can effectively reduce operational risks and improve overall surgical quality. Traditionally, the assessment of the surgical state relies on surgeons' manual interpretation of real-time video footage. This method is not only inefficient but also inherently subjective and lacks standardization.

[0003] In recent years, artificial intelligence technology has been gradually applied to the field of surgical state perception. The main technical approaches include single visual perception, multimodal fusion, and temporal process modeling. However, existing technologies still face the following significant technical bottlenecks when dealing with the unstructured features of real surgical scenarios:

[0004] First, single visual modalities lack deep semantic understanding capabilities. Existing methods primarily map continuous video frames to discrete action labels, making it difficult to distinguish between operations with similar visual patterns but drastically different clinical intentions. For example, the "positioning of the needle tip" (preparatory action) and "pushing the needle through the tissue" (execution action) are visually highly similar, easily confusing pure visual models. Furthermore, in the event of sudden incidents such as instrument drops or bleeding, pure visual models often fail to understand the clinical semantics, resulting in the inability to trigger corrective decisions such as "retrieving the needle" or "stopping the bleeding." Second, existing multimodal fusion methods suffer from poor domain adaptability and rigid fusion mechanisms. While introducing textual information can aid visual understanding, there is a significant semantic gap between the surgical domain's specific terminology (such as "tissue penetration depth") and general visual-language models. Simultaneously, existing fusion methods often employ fixed-weight feature concatenation, failing to dynamically adjust the focus based on the surgical stage (e.g., focusing on needle tip details during suturing and the overall field of view during hemostasis), leading to insufficient effectiveness of feature fusion.

[0005] Furthermore, existing temporal modeling techniques generally lack causal logic verification, making them ill-suited for non-sequential tasks. Most current mainstream robot skill learning and state recognition methods are based on the Markov process assumption, which holds that the current state depends only on a finite number of preceding states. This assumption has two major drawbacks: first, it cannot distinguish between "statistical correlation" and "true causality," making the model prone to learning spurious associations in the data (such as misinterpreting environmental noise as an operational signal); second, it cannot capture long-range causal dependencies. Real surgery often involves dynamic adjustments to procedures due to patient anatomical differences or sudden anomalies (such as "needle drop"). The impact of these abnormal events often has long-range lag effects, and traditional models cannot establish a causal chain of "abnormal event - preceding operation - corrective action" like human experts, resulting in poor decision robustness.

[0006] Finally, existing architectures typically treat semantic understanding, temporal modeling, and logical reasoning as independent modules. This decoupled design neglects the collaborative relationships between tasks: semantic description should guide temporal logic, while temporal context should feed back into semantic understanding. The lack of a unified framework for perception and causal inference makes it difficult for robots to achieve human-like cognition and decision-making in complex and dynamic environments. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems existing in the prior art.

[0008] To address this, the present invention provides a dynamic decision-making method for surgical robots based on multimodal semantic alignment perception and Granger causal temporal inference. This method can collaboratively solve the ambiguity of semantic understanding and the adaptability of multimodal fusion, and introduces a rigorous causal verification mechanism to overcome the logical blind spots of traditional probability models, thus providing a reliable cognitive basis for the autonomous decision-making of surgical robots.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention provides a dynamic decision-making method for surgical robots, comprising:

[0011] Step S100: Obtain surgical video data and corresponding surgical gesture label sequences during the operation of the surgical robot; preprocess the surgical video data to obtain keyframe sequences; and generate structured text descriptions for the surgical gesture label sequences through semantic enhancement and prompting engineering.

[0012] Step S200: Visual features are extracted frame by frame from the keyframe sequence using a pre-trained surgical action sequence prediction model. The structured text description is encoded to obtain a text feature sequence. The obtained visual feature sequence and text feature sequence are cross-modal semantically aligned to construct a fusion sequence containing surgical action count labels and temporal labels. The visual features and text features in the fusion sequence are fused to obtain a semantically enhanced visual feature sequence. Action segmentation is performed on the semantically enhanced visual feature sequence. Multi-scale spatiotemporal features of surgical operation details and surgical operation process are dynamically integrated through an adaptive weight learning mechanism to obtain a preliminary predicted surgical action sequence.

[0013] Step S300: Input the preliminarily predicted surgical action sequence into the preset causal rule generator and perform causal time sequence test. When the test result violates the causal rule, the preliminarily predicted surgical action sequence is corrected to obtain the corrected surgical action sequence, and the causal explanation of the detected surgical state transition points and abnormal events is given.

[0014] The preset causal rule generator is constructed according to the following steps:

[0015] Step S310: Decompose the surgical action sequence into standard operation and sudden abnormal event, encode them as standard operation state and random event state respectively, and together form stage variables. Encode the executable operation after the occurrence of the standard operation or sudden abnormal event into executable operation state and form action variables. The stage variables and action variables form a structured data representation.

[0016] Step S320: Use a VAR model to model the temporal relationship between historical stage variables and current action variables to obtain a VAR-based stage-action model, so as to capture the continuous effect of historical states on the current operation of the surgical robot.

[0017] Step S330: Perform Granger causality tests on the variables in the VAR-based stage-action model to confirm the causal dependencies between variables, and construct a causal relationship network in the surgical robot operation process to form a causal rule generator.

[0018] In some embodiments, the surgical video data is preprocessed, including image resizing and normalization and keyframe sampling.

[0019] In some embodiments, the surgical gesture label sequence is used to generate a structured text description through semantic enhancement and cueing engineering, including:

[0020] A dynamic template library containing 14 basic text templates was established. Each template is designed for different language expression methods. The {} are placeholders used to fill in specific surgical action descriptions.

[0021] A surgical terminology mapping dictionary is established to map surgical gesture labels to surgical terms. The mapping relationship is based on the clinical semantics of surgical operations, and surgical gesture labels are converted into professional descriptions that include instrument, action, and direction information.

[0022] The dynamic template library and the surgical terminology mapping dictionary are used for each keyframe sequence. and its corresponding gesture label sequence Four types of complementary text descriptions are dynamically generated, including: sequence counting information. Description of each action Complete sequence description and location information .

[0023] In some embodiments, the surgical action sequence prediction model includes a multimodal feature extraction network, a cross-modal semantic alignment and fusion network, and a temporal action recognition network connected in sequence;

[0024] The multimodal feature extraction network employs a visual-language dual encoder adapted to the surgical scenario. The keyframe sequence is input into the visual encoder to obtain a visual feature sequence, and the structured text description is input into the language encoder to obtain a text feature sequence.

[0025] The cross-modal semantic alignment and fusion network takes the fusion sequence as input, enabling the extracted visual feature sequence and text feature sequence to enhance and understand each other at multiple semantic levels, and then passes the resulting semantically enhanced visual feature sequence to the temporal action recognition network.

[0026] The temporal action recognition network uses MS-TCN++ as the basic temporal modeling network and introduces a dynamic weighted fusion module. The dynamic weighted fusion module uses an attention mechanism to learn the feature importance weights to obtain a fused feature sequence. After the fused feature sequence is passed through the subsequent layers of MS-TCN++, an initial action category prediction score is generated frame by frame. The output layer of MS-TCN++ normalizes the initial action category prediction score to obtain the probability of each frame belonging to each action category.

[0027] The surgical action sequence prediction model is trained in two stages. In the first stage, the multimodal feature extraction network and the cross-modal semantic alignment and fusion network are jointly trained, while the temporal action recognition network is not involved in the training. In the second stage, the parameters of the multimodal feature extraction network and the cross-modal semantic alignment and fusion network trained in the first stage are frozen, and only the temporal action recognition network is trained.

[0028] In some embodiments, the fusion sequence is constructed according to the following formula. :

[0029]

[0030] in, Indicates feature concatenation operation; , and For three predefined learnable tags, Used to capture and fuse counts and global semantic information at the entire fusion sequence level. Used to inject the timing order and timing position information of video frames on the timeline. As a separator, it is used to distinguish between text guidance information and visual feature sequences; For text feature sequences; It is a sequence of visual features;

[0031] The cross-modal semantic alignment and fusion network processes the fused sequence and splits it into the following two parts:

[0032] Sequence counting feature sequences Corresponding to The output of the location contains information about the keyframe sequence. Global information on action types and quantities;

[0033] Semantically enhanced visual feature sequences : Output corresponding to the position of the original visual feature sequence;

[0034] The dynamic weighted fusion module includes a global average pooling layer, a 1×1 convolutional layer, and a Sigmoid activation function connected in sequence.

[0035] In some embodiments, a multi-level contrastive learning strategy is employed during the first stage of training, using a multi-level contrastive learning loss function. Defined as:

[0036]

[0037] in, For global semantic alignment loss, For action-level alignment loss, For sequence-level alignment loss, The weights corresponding to each alignment loss;

[0038] During the first stage of training, a classification loss function is used to optimize the temporal action recognition network.

[0039] In some embodiments, in step S310, the robot is placed in The state variables at time t are described as follows:

[0040]

[0041]

[0042] In the formula, for The stage variable at time, by Standard operating status and A random event state composition; for The action variable at time t, by Executable operation status Composition; among which, in time:

[0043] For the first The first standard operating state is used to represent the first... Are the standard operations performed? ; express The time robot is executing the first A standard operating procedure; express The time robot did not execute the first A standard operating procedure;

[0044] For the first The nth random event state is used to represent the nth random event state. Whether a random event occurs. ; express The moment occurred A random event; express The time has not yet occurred. A random event;

[0045] For the first The executable operation state is used to represent the first executable operation state. Whether an executable operation is triggered. ; express The moment triggered the first An executable operation, then in The time-lapse robot will perform this executable operation; express The time did not trigger the first One executable operation.

[0046] In some embodiments, in step S320, the VAR-based phase-action model is expressed as follows:

[0047]

[0048] in, It is a sequence of historical stage variables consisting of p past stage variables; It is a linear function; The matrix represents the parameters to be estimated. for The random perturbation term at time step;

[0049] Will The state of each executable operation at any given time Represented as:

[0050]

[0051] in, express The first moment There are 1 executable operation state; p is the total lag order; q is the lag order; Indicates past moment The The standard operating state for the first Executable operation status The impact; It is in the past moment No. A standard operating state. ; Indicates past moment The state of the j-th random event is paired with the th Executable operation status The impact; It is in the past moment No. A random event state, ; yes Time of the first Random noise term of each executable operation state.

[0052] In some embodiments, in step S330

[0053] Granger causality tests were performed on the variables in the VAR-based stage-action model to confirm the causal relationships between the variables, including:

[0054] We set the time dimension n, the total lag order p, and the significance level α, and utilize the historical stage variable series. and action variables Constructing time-series data;

[0055] A constrained model without lagged terms of causal variables and an unconstrained model with lagged terms of causal variables are constructed respectively. The expression of the constrained model is as follows:

[0056]

[0057] in, These are constant terms in the constraint model; In the constraint model Its own lag q-th order term The coefficients during regression;

[0058] The expression for the unconstrained model is as follows:

[0059]

[0060] in, This refers to the constant term in the unconstrained model. In representing an unconstrained model Its own lag q-th order term The coefficients during regression;

[0061] The F-statistic is calculated to assess whether the improvement in predictive power of the unconstrained model compared to the constrained model is statistically significant. The F-statistic is calculated based on the sum of squared residuals.

[0062]

[0063] in, and , respectively, are the sum of squared residuals for the constrained and unconstrained models; Q is the total number of lagged terms of the constrained causal variable. ;

[0064] If the calculated F-statistic is greater than the critical value of the F-distribution corresponding to the significance level α, This indicates the sequence of variables in the historical stage. It constitutes Action variables at time The causal driving factors; otherwise, it indicates a historical stage variable sequence. Not constituted Action variables at time The causal driving factors.

[0065] In some embodiments, step S330 involves constructing a causal relationship network during robot operation based on the results of the Granger causality test, forming a causal rule generator, including:

[0066] Extract significant causal pairs from the results of Granger causality tests;

[0067] All extracted causal pairs are represented as edges of a directed graph, and a causal graph is constructed with each variable as a node;

[0068] Check the causal graph for cyclic or redundant causal relationships to ensure logical consistency;

[0069] The paths and relationships in a logically consistent causal graph are transformed into rules to form the causal rule generator.

[0070] The dynamic decision-making method for surgical robots based on multimodal semantic alignment perception and Granger causal temporal inference provided by this invention has the following characteristics and beneficial effects:

[0071] 1. Deep fusion of multimodal semantics, breaking through the bottleneck of visual perception.

[0072] This invention utilizes intelligent text prompting engineering and the Bridge-Prompt cross-modal semantic bridging mechanism to transform abstract numerical labels into semantically rich natural language descriptions, upgrading the model from pure visual pattern matching to deep semantic understanding. This mechanism can distinguish visually similar surgical actions with different clinical intentions (such as "positioning the needle tip" versus "pushing the needle"), effectively solving the semantic gaps inherent in traditional methods and simulating the cognitive process of surgeons integrating real-time observation with procedural knowledge.

[0073] 2. Introduce Granger causality testing to construct reliable decision-making logic.

[0074] This invention innovatively introduces the Granger causality test mechanism, solving the problem that traditional time series models cannot distinguish between "statistical correlation" and "true causality." By constructing a VAR model and comparing the prediction residuals of constrained and unconstrained models, this method can mathematically verify whether a preceding operation (such as "holding a needle") is a necessary condition for the current action (such as "stitching"). This enables the robot to eliminate spurious correlation features caused by environmental noise, establish a decision network based on true causal logic, and significantly improve the robustness and interpretability of decision-making.

[0075] 3. Improved dynamic weighting and temporal modeling to adapt to the unstructured characteristics of surgery.

[0076] The Wfusion dynamic weighted fusion module proposed in this invention replaces the traditional fixed-weight fusion method with adaptive weight learning, enabling dynamic balancing of local detailed features and global process features. Combined with Granger causal inference, the system can not only identify standard processes but also effectively handle non-sequential sudden anomalies (such as "dropped needle" triggering "needle retrieval"). By accurately tracing the root cause of the problem through causal chains and generating error correction strategies, it overcomes the limitation of the Markov assumption that relies only on local states.

[0077] 4. Multi-level alignment and phased training ensure feature accuracy.

[0078] This invention designs a multi-level contrastive learning optimization strategy, including global semantic alignment, action-level alignment, and sequence-level alignment, to ensure accurate alignment of visual features and text features at different semantic levels. Combined with a two-stage training strategy (first multimodal alignment, then temporal / causal optimization), gradient conflicts are effectively avoided, improving the model's convergence speed and final performance.

[0079] 5. Broad clinical applicability and generalization ability

[0080] This invention's method is independent of specific surgical types. By introducing general textual knowledge to enhance visual understanding and combining it with a data-driven causal rule discovery mechanism, it exhibits excellent cross-task generalization capabilities. The constructed causal relationship network provides the robot with clear decision-making basis in unstructured environments, making it suitable for various complex types of surgical state recognition and quality assessment tasks. Attached Figure Description

[0081] Figure 1 This is an overall flowchart of the surgical decision-making method based on multimodal perception and causal temporal inference provided in the embodiments of the present invention;

[0082] Figure 2 This is a flowchart of the multimodal semantic alignment mechanism provided in the embodiments of the present invention;

[0083] Figure 3 This is a schematic diagram of the improved MS-TCN++ network architecture and Wfusion module provided in the embodiments of the present invention;

[0084] Figure 4 It is a causal relationship diagram constructed by applying the method of the embodiments of the present invention to the surgical procedure;

[0085] Figure 5 In the figures (a) and (b), respectively, the confusion matrix analysis diagrams of the method of the present invention at scales of 5,000 samples and 10,000 samples are presented. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application.

[0087] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0088] To ensure accuracy, the main terms and symbols used in this invention are explained as follows:

[0089] 1. Terminology Explanation:

[0090] Surgical status recognition: refers to analyzing the types of actions and operational quality during the surgical process, that is, identifying the types of surgical actions that the surgeon is performing (such as "positioning the needle tip", "pushing the needle through the tissue", etc.).

[0091] Multimodal semantic alignment refers to establishing deep semantic relationships between information from different sources (visual images and structured text descriptions in this embodiment) by fusing information from different sources, thereby improving the ability to understand the clinical intent and contextual meaning of actions in surgical scenarios and solving the problem that simple visual pattern matching is difficult to distinguish between visually similar but different actions.

[0092] Temporal modeling refers to modeling the evolution and dependencies of surgical actions over time. Since surgical actions are a continuous sequence, understanding how the current action depends on previous actions (temporal dependency) is crucial for accurately identifying the overall surgical procedure status.

[0093] Granger causality test: A statistical hypothesis testing method used to determine whether a time series (specifically, the state of a surgical procedure in this application) helps predict another time series (specifically, the robot's next action in this application). If incorporating historical information about variable X significantly reduces the prediction error of variable Y, then variable X is said to be a Granger cause of variable Y.

[0094] 2. Symbol definition:

[0095] Surgical video sequence ,in This represents the total number of frames in the video.

[0096] : No. The frame image has a size of 224×224×3;

[0097] Surgical gesture label sequence, ,in ;

[0098] Visual feature sequence, dimension 1 ;

[0099] Text feature sequence, dimension 1 , Batch size;

[0100] : A semantically enhanced visual feature sequence with dimension . ;

[0101] : Sequence counting feature sequence, dimension is ;

[0102] Dynamic weight parameters, satisfying ;

[0103] CLIP model image encoder;

[0104] : Text encoder of CLIP model;

[0105] Action variable, representing time. The executable operation status;

[0106] Phase Variable: Includes standard operating status. and random event states ;

[0107] Sum Squared Residuals.

[0108] See Figure 1 The first aspect of this invention provides a dynamic decision-making method for surgical robots based on multimodal semantic alignment perception and causal temporal inference, comprising the following steps:

[0109] Step S100: Obtain surgical video data and corresponding surgical gesture label sequences during the operation of the surgical robot. The surgical video data is preprocessed, and the surgical gesture label sequences are generated into structured text descriptions through semantic enhancement and prompting engineering.

[0110] Step S200: Extract visual features frame by frame from the preprocessed surgical video data, encode the above structured text description to obtain high-dimensional text features; perform cross-modal semantic alignment on the obtained visual feature sequence and text feature sequence, and construct a fusion sequence containing surgical action count labels and temporal labels accordingly; fuse the visual features and text features in the fusion sequence to obtain a semantically enhanced visual feature sequence; perform action segmentation on the sequence; and dynamically integrate the multi-scale spatiotemporal features of surgical operation details and surgical operation process through an adaptive weight learning mechanism to obtain a preliminary predicted surgical action sequence.

[0111] Step S300: Input the preliminarily predicted surgical action sequence into the preset causal rule generator and perform causal time sequence test. When the test result violates the causal rule, the preliminarily predicted surgical action sequence is corrected to obtain the corrected surgical action sequence, and the causal explanation of the detected surgical state transition points and abnormal events is given.

[0112] The pre-defined causal rule generator is constructed according to the following steps:

[0113] Step S310: Decompose the surgical action sequence into standard operation and sudden abnormal event, encode them as standard operation state and random event state respectively, and together they constitute stage variables. Encode the executable operation after the occurrence of standard operation or sudden abnormal event as executable operation state and constitute action variable. The stage variable and action variable form a structured data representation.

[0114] Step S320: Use a vector autoregression (VAR) model to model the temporal relationship between historical stage variables and current action variables to obtain a VAR-based stage-action model, so as to capture the continuous effect of historical states on the current operation of the surgical robot.

[0115] Step S330: Perform Granger causality test on the variables in the VAR-based stage-action model to confirm the real causal dependency between variables, and construct the causal relationship network in the operation process of the surgical robot to form a causal rule generator.

[0116] In some embodiments, step S100 includes:

[0117] Step S110: Acquisition of raw data

[0118] Real-time surgical video data and corresponding surgical gesture label sequences are acquired during the operation of the surgical robot. In this embodiment of the invention, 15 surgical operations of the surgical robot are assigned the following surgical gesture labels G0, G1, ..., G14, with the definitions of each label as follows:

[0119] G0: Idle / Silent (SIL)

[0120] G1: Reaching for needle with right hand

[0121] G2: Positioning needle

[0122] G3: Pushing needle through tissue

[0123] G4: Transferring needle from left to right

[0124] G5: Moving to center with needle in grip

[0125] G6: Pulling the suture with the left hand

[0126] G7: Pulling the suture with the right hand

[0127] G8: Orienting needle

[0128] G9: Using the right hand to help tighten the suture.

[0129] G10: Loosening more sutures

[0130] G11: Dropping suture at the end and moving to the endpoints.

[0131] G12: Grasping needle

[0132] G13: Making a C-loop

[0133] G14: Inserting needle

[0134] Step S120: Surgical video data preprocessing

[0135] The raw surgical video data collected in step S110 is standardized to prepare for model input, specifically including:

[0136] Image resizing and normalization: First, resize each frame of the original surgical video. The image is scaled uniformly to the standard size required for the model input, such as 224x224 pixels, usually using methods such as bilinear interpolation. Then, the image pixel values ​​are normalized, for example, the original pixel value range [0,255] is normalized to the range [0,1] or [-1,1].

[0137] Keyframe sampling: To avoid processing redundant frames and improve efficiency, an equal-interval sampling strategy is adopted from the original surgical video sequence. Extracting keyframe sequences For example, every few frames, one frame is extracted to form a subsequence of fixed length; simultaneously, gesture tag sequences that precisely correspond to these sampled keyframes are extracted. This step ensures the temporal representativeness and manageability of the input data.

[0138] Step S130: Generation of structured text description

[0139] See Structured Text Description Generation (Intelligent Text Hint Project) Figure 2 This step employs a dynamic template selection and technical terminology mapping strategy, generating four types of structured text descriptions through 14 dynamic template libraries and surgical terminology mapping, transforming abstract numerical labels into semantically rich natural language descriptions. Specifically, it includes:

[0140] Step S131: Construct the dynamic template library T templates

[0141] A dynamic template library containing 14 basic text templates is established, each targeting a different language expression style. {} are placeholders used to fill in specific surgical action descriptions. The template design aims to cover different linguistic perspectives of action description (such as subject, action, object, state, etc.). The template content designed in this embodiment is as follows:

[0142] T templates= ["the person is {}", "the person is performing the action of {}", "the character is {}", "he or she is {}", "the action {} is being played", "it is the action of {}", "the human is {}", "the person is working on {}", "the scene is {}", "the person is focusing on {}", "the person is completing the action of {}", "the step is {}", "the action is {}", "theaction step is {}"]

[0143] The Chinese description of the above template is: T templates = ["This person is currently {}", "This person is performing {} action", "This character is currently {}", "He / She is currently {}", "Action {} is in progress", "This is {}'s action", "This person is currently {}", "This person is processing {}", "The scene is {}", "This person is focusing on {}", "This person is completing {} action", "The step is {}", "The action is {}", "The action step is {}"]

[0144] Step S132: Establish a surgical terminology mapping dictionary M, mapping surgical gesture labels to surgical terms. The mapping relationship is based on the clinical semantics of surgical procedures, converting abstract surgical gesture labels into professional descriptions containing information such as instruments, movements, and directions.

[0145] M = {0: "SIL", 1: "reaching for needle with right hand", 2: "positioning needle", 3: "pushing needle through tissue", 4: "transferringneedle from left to right", 5: "moving to center with needle in grip", 6: "pulling suture with left hand", 7: "pulling suture with right hand", 8: "orienting needle", 9: "using right hand to help tighten suture", 10: "loosening more suture", 11: "dropping suture at end and moving to endpoints", 12: "grasping needle", 13: "making C loop", 14: "inserting needle",15: "pulling suture with both hands"}

[0146] The Chinese description of the dictionary M above is:

[0147] M = {0: "Stationary state", 1: "Right-hand needle pick", 2: "Needle tip positioning", 3: "Push the needle through the tissue", 4: "Transfer the needle from left to right", 5: "Move the needle to the center", 6: "Left-hand suture pull", 7: "Right-hand suture pull", 8: "Direction needle", 9: "Right-hand assist in tightening the suture", 10: "Loosen more suture", 11: "Drop the suture at the end and move it to the endpoint", 12: "Grasp the needle", 13: "Make a C-loop", 14: "Insert the needle", 15: "Pull the suture with both hands"}

[0148] For special labels such as "SIL" (silent state), semantic transformation is performed in the surgical scenario: "SIL" at the beginning of the sequence is transformed into "waiting and preparing", "SIL" at the end of the sequence is transformed into "finishing and waiting", and "SIL" in the middle of the sequence is transformed into "pausing and adjusting".

[0149] Step S133: Utilize the aforementioned dynamic template library A dictionary M mapping surgical terminology to each sampled keyframe sequence. and its gesture label sequence Four types of complementary text descriptions are dynamically generated, including: sequence counting information. Description of each action Complete sequence description and location information (Location information mainly describes the position of the surgical instrument tip or key surgical site, such as the spatial relationship between the needle tip and the tissue.) See this step for details. Figure 2 The dynamic parameter injection module in the program.

[0150] In some embodiments, step S200 uses a pre-trained surgical action sequence prediction model to perform multimodal feature extraction, cross-modal semantic alignment and fusion, and temporal action recognition on the preprocessed surgical video data and structured text description to obtain a preliminary predicted surgical action sequence. See also Figure 2 Step S200 specifically includes:

[0151] Step S210: Construction of Surgical Action Sequence Prediction Model

[0152] The constructed surgical action sequence prediction model consists of a multimodal feature extraction network, a cross-modal semantic alignment and fusion network, and a temporal action recognition network connected in sequence. The structure of each network is described below:

[0153] Step S211: Construction of Multimodal Feature Extraction Network

[0154] The multimodal feature extraction network employs a vision-language dual encoder adapted to surgical scenarios, specifically the dual encoders of the CLIP (Contrastive Language-Image Pre-training) model: an image encoder and a language encoder. and text encoder To extract high-dimensional features. Among them:

[0155] Image encoder It adopts the ViT-B / 16 architecture, which contains 12 image encoder layers for processing the pre-processed keyframe sequences. Extracting classification tags frame by frame (Representing the input sequence) corresponding to visual features, and forming a visual feature sequence. Its dimensions are Where 16 represents the number of keyframes and 512 represents the visual feature dimension; specifically, the image encoder E imgFirst, each frame of the image is divided into 196 16×16 pixel patches, with a classification label added, for a total of 197 labels. Then, after processing through 12 image encoder layers, the features corresponding to the classification labels are extracted as image representations, resulting in a visual feature sequence with dimensions (B, 16, 512). :

[0156] (1)

[0157] Text Encoder Employing a Transformer architecture, it contains 12 text encoder layers to extract high-dimensional text features from four types of structured text descriptions (sequence counts, action descriptions, complete sequences, and location information) corresponding to the keyframe sequences of the input image encoder. A separate text feature sequence is constructed for each type of structured text description. Its dimension is (B, 512); specifically, it will be compared with the input image encoder. keyframe sequence The four corresponding types of structured text descriptions are input into the text encoder E respectively. txt Text feature encoding is performed; text encoder E txt Each type of structured text description is segmented into words, with the maximum sequence length set to 77. Any shortfall is filled with padding markers. After processing through 12 text encoder layers, the features corresponding to the classification markers are extracted as text representations. The dimension of each text feature sequence is (B, 512).

[0158] Step S212: Construction of Cross-Modal Semantic Alignment and Fusion Network

[0159] The constructed cross-modal semantic alignment and fusion network is one of the core innovations of this invention, aiming to deeply bridge visual and textual modalities to achieve accurate semantic alignment. Its key is to enable the extracted visual feature sequences... and text feature sequences They enhance and improve understanding at multiple semantic levels.

[0160] In this embodiment, the cross-modal semantic alignment and fusion network adopts a Bridge-Prompt model architecture, specifically a 6-layer Transformer encoder with a hidden layer dimension of 512. The input to this cross-modal semantic alignment and fusion network is a sequence of visual features. and text feature sequences Constructed fusion sequence The cross-modal semantic alignment and fusion network splits the fused sequence into sequence counting feature sequences. and semantically enhanced visual feature sequences Two parts, and semantically enhanced visual feature sequences. It is then passed to the timing action recognition network.

[0161] Furthermore, fusion sequence The construction process includes:

[0162] Step S2121: Define three learnable tags for cross-modal semantic bridging. These tags are learnable parameters that learn semantic representations for surgical scenarios through a training process.

[0163] Used to capture and fuse count and global semantic information at the entire fusion sequence level; It is a learnable feature container, which is similar to a text encoder. Extracted sequence counting information The text features exhibit semantic alignment relationships, which will be crucial during subsequent training. The output vector, after aggregating the extracted visual feature sequences, is subjected to a loss function by the text encoder. Extracted text feature sequences Under supervision, they learn to represent the quantity of actions;

[0164] Used to inject the timing order and timing position information of video frames on the timeline;

[0165] : Used as a separator to distinguish between textual guidance information and visual feature sequences.

[0166] Step S2122: For each keyframe sequence (Representing a video segment) Extracted visual feature sequence and its corresponding four types of text feature sequences A fusion sequence is constructed by concatenating learnable tags, text feature sequences, and visual feature sequences according to a predetermined logical order. The typical order is:

[0167] (2)

[0168] in, This indicates a feature concatenation operation; it should be noted that, due to the text encoder... There are four types of text feature sequences extracted, which will ultimately be used for each keyframe sequence. Construct four such fusion sequences.

[0169] Once the fused sequence is constructed, it is input into the cross-modal semantic alignment and fusion network. The Transformer encoder's self-attention mechanism allows all elements in the fused sequence (learnable tags, text feature sequences, and visual feature sequences) to freely attend to and interact with each other, achieving cross-modal semantic bridging. Through this deep interaction, the cross-modal semantic alignment and fusion network can learn:

[0170] How visual feature sequences correspond to the semantics of text descriptions;

[0171] How text descriptions guide the understanding of visual content (especially distinguishing visually similar but intentionally different actions).

[0172] Global sequence information ( ) and timing information ( How to integrate the understanding of local features.

[0173] Cross-modal semantic alignment and fusion network processes fusion sequence Then, the fused sequence was split into two parts:

[0174] Sequence counting feature sequences : Usually corresponds to The output of the position contains information about the entire keyframe sequence. Global information on action types and quantities;

[0175] Semantically enhanced visual feature sequences The output corresponds to the position of the original visual feature sequence (i.e., those vector segments in the sequence output by the 6-layer Transformer encoder that correspond to the "visual feature part" in the input sequence). At this point, these visual features have been semantically enhanced and refined through interaction with text, global, and temporal information. The dimension is Where 512 is the feature dimension after projection and semantic enhancement, and K is... The total number of video frames.

[0176] Step S213: Construction of the temporal action recognition network

[0177] The temporal action recognition network constructed in this embodiment of the invention is used to process the semantically enhanced visual feature sequences obtained by cross-modal semantic alignment and fusion networks. Finally, the predicted action category for each frame is output.

[0178] Furthermore, this embodiment uses MS-TCN++ (Multi-Stage Temporal Convolutional Network for Action Segmentation) as the basic temporal modeling network, and introduces a dynamic weighted fusion module on top of it to construct a temporal action recognition network. See [link to documentation]. Figure 3 .in:

[0179] The core of MS-TCN++ is its multi-stage structure and one-dimensional dilated temporal convolutions. Each stage contains multiple dilated convolutional layers with increasing dilation rates to capture temporal dependencies at different time scales (short-range and long-range). The multi-stage structure is cascaded through residual connections to progressively refine the predictions.

[0180] The introduced dynamic weighted fusion module Wfusion is a key innovation in temporal modeling in this embodiment of the invention. This module is integrated within the MS-TCN++ network (typically at each level or key layers). Traditional methods fuse feature maps obtained from convolutions with different dilation rates using fixed weights (such as simple concatenation or averaging). The core innovation of the Wfusion module in this embodiment lies in using an attention mechanism to learn feature importance weights, achieving adaptive weight learning. Weight calculation is implemented through a global average pooling layer, a 1×1 convolutional layer, and a sigmoid activation function. The dynamic weighted fusion formula is as follows:

[0181]

[0182] In the formula, For MS-TCN++ network, the first The output feature sequence of each branch Wfusion is a dynamic weighted fusion module. The assigned normalized weights satisfy... This ensures the reasonableness of weight allocation; by normalizing the weights of the learned feature sequences of each branch. Output feature sequence of the corresponding branch Multiply the weighted feature sequences, then sum the weighted feature sequences to obtain the fused feature sequence. ; The number of branches that the dynamically weighted fusion module Wfusion runs in parallel.

[0183] In one specific embodiment of this application, the dynamic weighted fusion module Wfusion supports three parallel branches (local details, intermediate range, and global context, corresponding to dilation rates of 1, 8, and 16, respectively). The dynamic weighted fusion module Wfusion employs an attention mechanism to learn the weights of the three branches. This enables more refined multi-scale feature fusion. This allows the temporal action recognition network to dynamically decide, based on the specific content and temporal context of the current surgical segment, whether to rely more on local features that capture subtle changes in movement or on global features that understand the overall process (for example, assigning higher weight to local features in delicate needle-threading actions and higher weight to global features when judging transitions in surgical stages).

[0184] The fused feature sequences are then processed through subsequent layers of the MS-TCN++ network (containing more convolutions and activation functions such as ReLU and Dropout). Finally, a 1x1 convolutional layer (equivalent to a fully connected layer) maps the features to the number of action categories, generating an initial action category prediction score for each frame. Subsequently, the raw prediction scores output by the MS-TCN++ network are Softmax normalized to obtain the probability of each frame belonging to each action category. Temporal smoothing techniques (such as median filtering or Conditional Random Fields, CRF) are typically used to eliminate unreasonable transient jitter in the prediction results, making the output action transition points smoother and more realistic.

[0185] Step S220: Training the surgical action sequence prediction model

[0186] The surgical action sequence prediction model constructed in step S210 is trained in two stages. In the first stage, the multimodal feature extraction network and the cross-modal semantic alignment and fusion network are jointly trained, while the temporal action recognition network is not trained. In the second stage, the parameters of the multimodal feature extraction network and the cross-modal semantic alignment and fusion network trained in the first stage are frozen, and only the temporal action recognition network is trained. Specifically, this includes:

[0187] Step S221, First Phase Training

[0188] Using surgical datasets and through a multi-level contrastive loss function Backpropagation is performed to jointly train the multimodal feature extraction network and the cross-modal semantic alignment and fusion network. Among these,

[0189] The CLIP model's dual encoder was fine-tuned using a surgical dataset to adapt it to the specialized visual patterns and terminology of the surgical field. The surgical dataset includes subsets of data and corresponding textual description data.

[0190] The image data subset is obtained by first collecting raw video stream data in a simulated environment (such as a desktop model) and in a real clinical surgical scenario using a surgical robot system (such as the da Vinci system) to perform target tasks (such as suturing, knotting, threading needles, etc.) according to a set video frame rate (such as 30 frames per second) and resolution (such as 640×480 pixels). The video stream data collected in the real environment usually has challenges such as low image contrast and frequent instrument occlusion, which can be used to improve the robustness of the model in complex clinical environments. The image data subset is obtained by processing various surgical gestures defined by surgical experts corresponding to the image data subset according to step S130.

[0191] The CLIP model's dual encoders were trained for a total of 50 epochs with a learning rate of 5e-06 to adapt the dual encoders to the professional knowledge of the surgical field and to retain the dual encoder weights of the CLIP model with the highest recognition accuracy in the epochs.

[0192] The first training phase employs a multi-level contrastive learning strategy, using a multi-level contrastive learning loss function. Training a cross-modal semantic alignment and fusion network. Defined as:

[0193] (4)

[0194] in, The global semantic alignment loss measures the entire keyframe sequence. Global visual representation (e.g.) or Pooling characteristics and complete sequence description semantic consistency Commonly used KL divergence or contrast loss;

[0195] To achieve action-level alignment loss, cross-modal semantic alignment and fusion networks need to be aligned at a finer granular level. Ensure the visual features of each frame in the video (i.e. The features corresponding to each time step can be correlated with the action descriptions at that time (i.e., ... To ensure correct matching of the corresponding parts, frame-word contrast loss is commonly used.

[0196] The sequence-level alignment loss ensures cross-modal semantic alignment and enables the fusion network to understand the overall structure of the action sequence. The requirement is that the cross-modal semantic alignment and fusion network can correctly match sequence counting information. Sequence counting features that contain sequence counting information Commonly used loss methods include classification loss or regression loss.

[0197] These three parts of the loss are weighted accordingly. Combined into total loss Used for backpropagation and optimization. In this embodiment, the weight parameters... It can be adjusted according to the actual effect, and the value range is: .

[0198] Step S222, Second Phase Training

[0199] The network parameters of the multimodal feature extraction network and the cross-modal semantic alignment and fusion network trained in step S221 are fixed. Only the temporal action recognition network is trained. The temporal action recognition network is optimized using standard classification loss (such as cross-entropy loss) to focus on learning the temporal evolution pattern of surgical actions. In this step, because the improved MS-TCN++ temporal modeling network uses causal convolution, it ensures that the prediction of the current frame depends only on the information of the current frame and previous frames, meeting the needs of real-time state recognition during surgery.

[0200] Step S230, Surgical Action Sequence Prediction

[0201] The keyframe sequence extracted in step S120 The image encoder E is input into the trained multimodal feature extraction network. img After forward propagation calculation, the output is a visual feature sequence containing temporal information. Its dimensions are (B, 16, 512), where B is the batch size, 16 is the number of keyframes, and 512 is the feature dimension. The four types of structured text descriptions (sequence count, action description, complete sequence, and location information) generated in step S130 are input into the text encoder E in the trained multimodal feature extraction network. txt Extract the hidden state corresponding to the classification label in each text sequence as the text feature sequence. Its dimensions are (B, 512); the extracted visual feature sequence and text feature sequences It is passed as a multimodal input to the trained cross-modal semantic alignment and fusion network module;

[0202] The cross-modal semantic alignment and fusion network module first utilizes and A fusion sequence is constructed, and then semantically enhanced visual features are obtained based on the fusion sequence. It is then passed to the trained sequential action recognition network;

[0203] The trained sequential action recognition network uses semantically enhanced visual features. The output yields a preliminary predicted sequence of surgical actions.

[0204] It should be noted that, as Figure 1 As shown in the "Granger Causal Time Series Inference and Verification" stage, in order to overcome the limitation that pure probability models cannot distinguish between "statistical correlation" and "true causality," step S300 of this embodiment uses the Granger causality test to logically verify the prediction results output in step S200, and constructs decision rules based on causal mechanisms. The causal rule generator used in the Granger causality test is constructed according to the following steps:

[0205] Step S310, Status-Event Coding

[0206] The surgical action sequence predicted in step S200 is decomposed into standard operation and sudden abnormal event, which are respectively encoded as standard operation state and random event state and together constitute stage variables. The executable operation after the occurrence of standard operation or sudden abnormal event is encoded as executable operation state and constitutes action variable. The stage variables and action variables form a structured data representation.

[0207] Furthermore, step S310 aims to transform continuous action sequences into structured temporal data using a "phase-action" encoding method, providing a foundation for subsequent temporal relationship modeling and causal verification. The phase dimension is defined by the continuously executed historical standard operating states and sudden abnormal events; the action dimension represents the current response action the robot can perform after a standard operating procedure or a random event. If the historical phase state is a standard operating state (no unexpected events), the current action is the corresponding action in the regular standard procedure; if the historical phase state is a random event state (an anomaly occurs), the current action is the response action to that anomaly. This decomposition method aligns with the cognitive patterns of human experts and effectively distinguishes between the decision logic of regular procedures and anomaly handling, providing an interpretable dynamic response framework for the surgical robot system.

[0208] Specifically, suppose there exists a continuous sequence of actions. Standard operating status (e.g., "stitching") A random event state (e.g., "needle drop", "bleeding") and Executable operation status (Such as "needle picking" and "electrocautery"), their definitions are as follows:

[0209] At any moment :

[0210] For the first The first standard operating state is used to represent the first... Are the standard operations performed? ; express The time robot is executing the first A standard operating procedure; express The time robot did not execute the first A standard operating procedure;

[0211] For the first The nth random event state is used to represent the nth random event state. Whether a random event occurs. ; express The moment occurred A random event, express The time has not yet occurred. A random event; if step S200 detects that the probability of a random event exceeds a set threshold, then its state is set to 1;

[0212] For the first The executable operation state is used to represent the first executable operation state. Whether an executable operation is triggered. ; express The moment triggered the first An executable operation, then in The time-lapse robot will perform this executable operation; express The time did not trigger the first One executable operation;

[0213] The standard operation that the robot is performing before a random event occurs is defined as the preceding operation of the random event, and the subsequent operation of the preceding operation when the random event occurs is defined as the response operation to the random event.

[0214] Therefore, the robot in The state variables at time t are described as follows:

[0215] (5)

[0216] (6)

[0217] in, for The stage variable at time, by Standard operating status and A random event state composition; for The action variable at time t, by Executable operation status composition.

[0218] Step S320, VAR modeling

[0219] The VAR model is used to model the temporal relationship between historical stage variables and current action variables, resulting in a VAR-based stage-action model to capture the persistent effect of historical states on the current operation of the surgical robot.

[0220] Furthermore, step S320 aims to characterize the inter-stage variables in the complex task of surgery using a VAR-based stage-action model. Dynamic interaction relationships and their impact on action variables The potential impact of this approach lies in its core advantage: by introducing lag terms, it can simultaneously model the interactions between multiple standard operating states and random events, adapting to dynamic changes in the operational process. Furthermore, by capturing the persistent effects of historical states on current operations through lag terms, it provides a structured foundation for subsequent Granger causal analysis identifying the relationship between stage variables and action variables. Specifically, this includes:

[0221] By leveraging VAR models to uncover temporal patterns between historical information and the robot's current actions, a dynamic decision function is learned from continuous action sequences based on these patterns. This dynamic decision function precisely describes the mapping relationship between historical information and the robot's current actions, representing the conditional probability distribution of the robot taking a predetermined action within the historical context of the surgical task. Its general expression is:

[0222] (7)

[0223] Where p is the total lag order, representing the length of the historical time window affecting the current decision; a state transition process is defined through the above conditional probability distribution, which is a sequence of historical stage variables consisting of the past p stage variables. Mapped to Action variables at time points .

[0224] Construct a VAR-based phase-action model, expressed as follows:

[0225] (8)

[0226] in, It is a linear function; The parameter matrix is ​​used to present the pattern of how historical stages affect current actions in numerical form, providing a quantitative basis for subsequent causal analysis. for The random perturbation term at each time step. The stage-action model constructed in this embodiment represents action decisions as a linear combination of variables from the past p stages, providing interpretable parameter estimates for subsequent analysis.

[0227] Furthermore, The state of each executable operation at any given time Represented as:

[0228] (9)

[0229] in, express The first moment There are several executable operation states; q is the hysteresis order; Indicates past moment The The standard operating state for the first Executable operation status The impact; It is in the past moment No. A standard operating state. ; Indicates past moment The state of the j-th random event is paired with the th Executable operation status The impact; It is in the past moment No. A random event state, ; yes Time of the first The random noise term for each executable operation state is a random variable that follows a normal distribution.

[0230] Understandably, this representation simultaneously depicts the impact of historical standard operating states and anomalous event states on the current executable operating state, providing a clear and interpretable framework for modeling robot operational decisions. Specifically, some executable operations may be triggered by anomalous events (e.g., "bleeding" in a surgical scenario leading to "electrocoagulation"), or they may originate from routine operational logic (e.g., "suturing" in a surgical scenario naturally progresses to "knotting"). In the subsequent step S330, Granger causality tests will be introduced and integrated into the time series modeling framework to identify and eliminate spurious correlations, thereby constructing decision logic with clear causal interpretability and good generalization ability.

[0231] Step S330: Granger causality test and rule generation

[0232] Granger causality tests were performed on the variables in the VAR-based stage-action model to confirm the true causal dependencies between the variables. Based on this, a causal relationship network in the operation process of the surgical robot was constructed to form a causal rule generator.

[0233] Further, step S330 aims to establish a predictive model between variables using the aforementioned VAR-based stage-action model, and to determine the statistical significance of the causal relationship through hypothesis testing. Utilizing the idea of ​​Granger causality testing, the predictive ability of models including and not including historical data of potential causal variables for the outcome variable is compared to determine whether preceding actions and random events are Granger causes of the response operation. More specifically, using the VAR-based stage-action model constructed in step S320, Granger causality testing is employed to identify the temporal causal relationship between historical stage variables and the robot's action variables. In non-Markovian surgical operation scenarios, if historical standard operation state sequences are included... and historical random event state sequence Information can significantly improve the current state of executable operations. The prediction accuracy is considered to be and yes Granger causality, otherwise it is considered and no The Granger causal source is determined. This testing process is performed independently for each potential causal pair (stage-action). The specific implementation steps are as follows:

[0234] Step S331: Construct time series data:

[0235] Set the time dimension n ( The total lag order p and significance level α are used to analyze the historical variable series. and action variables Construct time-series data.

[0236] Step S332: Construct two sets of regression models:

[0237] The two sets of regression models constructed include a constrained model without lagged terms of causal variables and an unconstrained model with lagged terms of causal variables; among them,

[0238] Constrained model (without lagged causal variables): This model uses only historical data of the action variable (outcome variable) itself to predict the current value of the action variable, ignoring the influence of historical information (potential causal variables). The constrained model can be represented as:

[0239] (10)

[0240] in, q represents the constant term in the constraint model; q is the lag order. In the constraint model Its own lag q-th order term The coefficients during regression, i.e. yes The autoregressive coefficient at lag order q.

[0241] Unconstrained Model (with Lagged Terms of Causal Variables): This model incorporates lagged terms (potential causal variables) of all historical stage variables to explicitly capture the causal impact of past standard operating conditions and random event states on the current action variable (outcome variable). In other words, while retaining the historical information of the outcome variable itself, it introduces the historical information of potential causal variables to assist in predicting the current action variable. The expression for the unconstrained model is as follows:

[0242] (11)

[0243] in, This refers to the constant term in the unconstrained model. In representing an unconstrained model Its own lag q-th order term The coefficients during regression; Indicates past moment The The standard operating state for the first Executable operation status The impact; It is in the past moment No. A standard operating state. ; Indicates past moment The j-th random event state corresponds to the k-th executable operation state. The impact; It is in the past moment No. A random event state, ; yes Time of the first The random noise term for each executable operation state is a random variable that follows a normal distribution.

[0244] Step S333: Calculate the F-statistic to assess whether the improvement in predictive ability of the unconstrained model compared to the constrained model is statistically significant. The F-statistic is based on the sum of squared residuals and is calculated using the following formula:

[0245] (12)

[0246] in, and These are the sums of squared residuals for the constrained and unconstrained models, respectively; the sum of squared residuals for the constrained model is calculated. This yields the result when the lagged terms of causal variables are not considered. The prediction error level is used as a benchmark for subsequent comparisons; the sum of squared residuals of the unconstrained model is calculated. We can obtain the result after adding lagged terms of causal variables for... The prediction error level; Q is the total number of lagged terms of the constrained causal variables (when only a single standard operating state or a single random event state is considered), i.e. .

[0247] Step S334: Perform a causal significance test:

[0248] Determine the critical value of the F-distribution for a given significance level α by consulting the F-distribution table. ;

[0249] If the F-statistic calculated in step S333 is greater than This indicates the sequence of variables in the historical stage. (Including standard operating states and random event states) constitutes Action variables at time The causal driving factors, namely, there is a significant causal relationship between "preceding operation - random event - response operation";

[0250] If the F-statistic calculated in step S333 is less than or equal to This indicates the sequence of variables in the historical stage. (Including standard operating conditions and random event conditions) does not constitute Action variables at time There is no significant causal relationship between the causal driving factors, namely "preceding operation - random event - response operation".

[0251] Step S335: Extract significant causal pairs from the results of the Granger causality test. That is, if variable X is a Granger cause of variable Y, then record it as X→Y.

[0252] Step S336: Represent all extracted causal relationship pairs as edges of a directed graph, and construct a causal graph with each variable as a node;

[0253] Step S337: Check if there are cyclic or redundant causal relationships in the cause-effect graph. If cyclic or redundant causal relationships exist, perform the following processing to ensure logical consistency:

[0254] For cases involving loops:

[0255] When using graph theory algorithms (such as depth-first search) to identify directed cycles, if node A is both the cause and effect of node B and there is no time delay, the following methods can be used: a) introduce a mediating variable to break the cycle; or b) redefine the temporal relationship between variables.

[0256] For situations where redundancy exists:

[0257] First, redundancy type determination is performed: when both ①A→B→C and B is the only intermediary from A to C and ②A→C are satisfied, then A→B→C and A→C are considered equivalent paths; when both A→{B1,B2}→C and A→C are satisfied, then path A→{B1,B2}→C is considered to be contained in path A→C.

[0258] Subsequently, the d-separation criterion is used to test conditional independence and determine whether the path needs to be deleted. If conditional independence exists, it is determined to be a redundant path, and the direct path A→C is deleted; if conditional independence does not exist, it is determined to be an independent direct causal relationship, and the path is retained and not considered redundant.

[0259] In one specific embodiment of this application, for a surgical scenario, a directed acyclic graph is used to represent the causal relationship graph of the surgical process, such as... Figure 4 As shown.

[0260] Step S338: Finally, the paths and relationships in the causal graph are transformed into rule form, that is, if A→B and B→C, then the rule "If A occurs, then C may occur" is constructed, forming a causal rule generator that is easy to understand and apply.

[0261] Understandably, based on the results of Granger causality tests, a dynamic correction mechanism for robot operation processes can be constructed. When the robot detects an abnormal event, it automatically triggers the corresponding corrective action. For example, in a surgical scenario, by combining the causal relationship between the abnormal event of "needle drop" and its preceding operation state "G6 operation," the corrective strategy of "needle pickup + re-execution of G6" can be automatically inferred. Unlike traditional predefined operation processes, this mechanism generates adaptive response strategies through data-driven causal inference, thereby enabling process reconfiguration in non-sequential or sudden task scenarios.

[0262] In some embodiments, in step S300, when the logic of the initially predicted surgical action sequence significantly violates Granger causality rules, for example, "needle picking" is predicted even though "needle drop" did not occur, or "bleeding" occurs but "hemostasis" is not predicted, the predicted surgical action sequence will be automatically corrected using a causal network, or the confidence of the prediction will be reduced and an early warning will be issued; finally, a high-confidence surgical action category and the causal explanation of the detected surgical state transition points and abnormal events will be output, for example: "A needle drop event was detected, triggering the needle picking error correction procedure".

[0263] Experimental verification and effect analysis:

[0264] To further verify the effectiveness of the causal temporal inference framework proposed in this invention in dynamically modeling surgical temporal logic, this verification example uses synthetic data to simulate abnormal clinical scenarios and quantitatively evaluates the model's ability to distinguish between "statistical correlation" and "true causality".

[0265] 1. Dataset Construction and Variable Definition

[0266] Considering the scarcity of labeled data for anomalous events (especially long-range causal chains) in real-world surgical environments, this validation example constructs a synthetic dataset containing explicit causal logic. The design closely revolves around the causal relationship between "abnormal events and response actions," and includes three core binary time-series variables:

[0267] : Original gesture variable, representing standard operating procedure, such as G6 "left hand pulls the stitch";

[0268] : Abnormal event variables, representing sudden disturbances, such as "dropped needle";

[0269] : Restore action variables to represent error correction behavior, such as "pick up the needle + execute G6 again".

[0270] The above variable takes a value of 1 to indicate that the event / action has occurred, and a value of 0 to indicate that it has not occurred.

[0271] 2. Sample generation strategy

[0272] To test the discrimination boundary of the model, positive and negative samples were constructed:

[0273] Positive Samples: Constructed based on clinical surgical logic, with pre-defined causal driving relationships (e.g., ), that is, the recovery action It is the original gesture of history and abnormal events Co-driven. All pre-specified causal relationships were reviewed by multiple senior surgeons (consistency Kappa value 0.92), and simulated clinical noise (such as visual occlusion) was introduced in 20% of the sample to verify the robustness of the model.

[0274] Negative Samples: Samples generated randomly without causal association. Sequence composition. In this type of sample, there may be statistically random correlations between variables, but there is no real clinical logical dependence, which is used to exclude the interference of temporal spurious correlations.

[0275] This validation example constructs two sample sets of different sizes: one set contains 5,000 data points (2,500 positive and 2,500 negative examples), and the other set contains 10,000 data points (5,000 positive and 5,000 negative examples) to evaluate the generalization stability of the method under different data sizes.

[0276] 3. Model parameter optimization

[0277] The choice of the total lag order p is crucial when constructing constrained and unconstrained models.

[0278] Clinical prerequisite settings: Considering that intraoperative abnormal events typically span approximately two operation cycles from occurrence to triggering a response (e.g., after a "needle drop" occurs, it is necessary to identify the event before performing a "needle retrieval" operation), the initial settings are as follows: .

[0279] To further optimize the selection of the total lag order, the Akaike Information Criterion (AIC) was used for verification: by calculating the AIC index under different orders, the results showed that when... Minimum residual of the time model ( ), better than ( )and ( Therefore, this embodiment ultimately selects the optimal total lag order. It can accurately capture dynamic processes while avoiding overfitting.

[0280] 4. Experimental Results and Analysis

[0281] The experimental parameters are set as follows: probability of generating normal gestures Probability of abnormal event generation lagged term weight coefficient The regression coefficients were estimated using ordinary least squares (OLS), and the total number of lag terms was... .

[0282] Table 1: Performance Indicators of Causality Tests under Different Sample Sizes

[0283]

[0284] Figure 5 For confusion matrix analysis of different sample sizes, Figure 5 In Figure (a), the results of the method of this embodiment of the invention are shown on a scale of 5000 samples (2500 positive samples and 2500 negative samples); Figure 5 (b) shows the results of the method of this embodiment on a scale of 10,000 samples (5,000 positive samples and 5,000 negative samples).

[0285] It is worth noting that the recall rate of the embodiments of the present invention is consistently higher than the precision rate, indicating that it tends to reduce the false negative rate of true causal relationships, i.e., reducing false negatives. In safety-critical surgical scenarios, this "better to falsely accuse than to miss" bias aligns with the principle of prioritizing clinical safety, helping to ensure that all critical anomalies are effectively identified and trigger response mechanisms. The confusion matrix further shows that the false negative rate of the embodiments of the present invention is consistently below 5%, demonstrating its good application potential in surgical anomaly modeling and autonomous response.

[0286] Compared with existing methods, the surgical state recognition method proposed in this invention, based on multimodal semantic alignment and temporal modeling, addresses the issues of semantic gaps and dynamic adaptation in surgical gesture recognition. It continuously optimizes the model's recognition results for semantic understanding and temporal modeling of surgical actions in the intelligent text prompting module, the Bridge-Prompt cross-modal fusion module, and the improved MS-TCN++ temporal modeling module. The Granger criterion partially solves the problem of "unclear logic": in unstructured surgical environments, it can distinguish between statistical correlation and true causality. For example, it can mathematically verify that "dropping the needle" is a necessary precondition for "picking up the needle," thereby eliminating misjudgments caused by environmental noise (such as visual occlusion) and ensuring that the robot's decisions conform to clinical logic. By combining multimodal semantic alignment (step S200) with Granger causal inference (step S300), a closed loop of "perception + cognition" is achieved.

[0287] Alternative implementation plan:

[0288] The technical solution of this invention has good scalability and adaptability. Those skilled in the art can make the following modifications according to actual needs: other pre-trained multimodal models can be used to replace CLIP, such as ALIGN, BLIP, and other vision-language pre-trained models. The ALIGN model uses larger-scale image-text pairs for pre-training and has stronger cross-modal understanding capabilities; the BLIP model combines a unified framework for understanding and generation tasks and performs better in the medical field; the Florence model is a large-scale vision-language model developed by Microsoft and has better fine-grained recognition capabilities.

[0289] The Bridge-Prompt fusion encoder can employ other attention mechanisms, such as cross attention, sparse attention, and other variations of attention. Cross attention calculates the attention of visual features to text features and the attention of text features to visual features, respectively; sparse attention reduces computational complexity and is suitable for long sequence processing; multi-scale attention calculates attention weights at different scales, capturing multi-level semantic information.

[0290] Temporal networks can be replaced with other architectures, such as Transformer-based temporal models, LSTM, GRU, and other recurrent neural networks. Transformer-based temporal models use a pure Transformer architecture for temporal modeling, exhibiting stronger long-range dependency modeling capabilities; LSTM and GRU networks use recurrent neural networks for temporal modeling, offering higher computational efficiency; 3D CNN networks perform convolution operations directly in the spatiotemporal dimension, making them suitable for short-term temporal modeling. Furthermore, hierarchical loss functions can be designed to assign different weights to different semantic levels: L frame For frame-level loss, L segment For fragment-level loss, L sequence For sequence-level loss, the weight parameters are dynamically adjusted according to the importance of the task.

[0291] To address the highly nonlinear characteristics of surgical dynamics systems, the VAR model in step S300 can be replaced with transfer entropy or a Bayesian network. By calculating the direction and intensity of information flow, nonlinear causal dependencies can be captured, further improving decision robustness in complex scenarios.

[0292] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A dynamic decision-making method for a surgical robot, characterized in that, include: Step S100: Obtain surgical video data and corresponding surgical gesture label sequences during the operation of the surgical robot; preprocess the surgical video data to obtain keyframe sequences; and generate structured text descriptions for the surgical gesture label sequences through semantic enhancement and prompting engineering. Step S200: Visual features are extracted frame by frame from the keyframe sequence using a pre-trained surgical action sequence prediction model. The structured text description is encoded to obtain a text feature sequence. The obtained visual feature sequence and text feature sequence are cross-modal semantically aligned to construct a fusion sequence containing surgical action count labels and temporal labels. The visual features and text features in the fusion sequence are fused to obtain a semantically enhanced visual feature sequence. Action segmentation is performed on the semantically enhanced visual feature sequence. Multi-scale spatiotemporal features of surgical operation details and surgical operation process are dynamically integrated through an adaptive weight learning mechanism to obtain a preliminary predicted surgical action sequence. Step S300: Input the preliminarily predicted surgical action sequence into the preset causal rule generator and perform causal time sequence test. When the test result violates the causal rule, the preliminarily predicted surgical action sequence is corrected to obtain the corrected surgical action sequence, and the causal explanation of the detected surgical state transition points and abnormal events is given. The preset causal rule generator is constructed according to the following steps: Step S310: Decompose the surgical action sequence into standard operation and sudden abnormal event, encode them as standard operation state and random event state respectively, and together form stage variables. Encode the executable operation after the occurrence of the standard operation or sudden abnormal event into executable operation state and form action variables. The stage variables and action variables form a structured data representation. Step S320: Use a VAR model to model the temporal relationship between historical stage variables and current action variables to obtain a VAR-based stage-action model, so as to capture the continuous effect of historical states on the current operation of the surgical robot. Step S330: Perform Granger causality tests on the variables in the VAR-based stage-action model to confirm the causal dependencies between variables, and construct a causal relationship network in the surgical robot operation process to form a causal rule generator.

2. The dynamic decision-making method for surgical robots according to claim 1, characterized in that, The surgical video data is preprocessed, including image resizing and normalization, and keyframe sampling.

3. The dynamic decision-making method for surgical robots according to claim 1, characterized in that, The surgical gesture label sequence is used to generate a structured text description through semantic enhancement and cueing engineering, including: A dynamic template library containing 14 basic text templates was established. Each template is designed for different language expression methods. The {} are placeholders used to fill in specific surgical action descriptions. A surgical terminology mapping dictionary is established to map surgical gesture labels to surgical terms. The mapping relationship is based on the clinical semantics of surgical operations, and surgical gesture labels are converted into professional descriptions that include instrument, action, and direction information. The dynamic template library and the surgical terminology mapping dictionary are used for each keyframe sequence. and its corresponding gesture label sequence Four types of complementary text descriptions are dynamically generated, including: sequence counting information. Description of each action Complete sequence description and location information .

4. The dynamic decision-making method for surgical robots according to claim 1, characterized in that, The surgical action sequence prediction model includes a multimodal feature extraction network, a cross-modal semantic alignment and fusion network, and a temporal action recognition network connected in sequence. The multimodal feature extraction network employs a visual-language dual encoder adapted to the surgical scenario. The keyframe sequence is input into the visual encoder to obtain a visual feature sequence, and the structured text description is input into the language encoder to obtain a text feature sequence. The cross-modal semantic alignment and fusion network takes the fusion sequence as input, enabling the extracted visual feature sequence and text feature sequence to enhance and understand each other at multiple semantic levels, and then passes the resulting semantically enhanced visual feature sequence to the temporal action recognition network. The temporal action recognition network uses MS-TCN++ as the basic temporal modeling network and introduces a dynamic weighted fusion module. The dynamic weighted fusion module uses an attention mechanism to learn the feature importance weights to obtain a fused feature sequence. After the fused feature sequence is passed through the subsequent layers of MS-TCN++, an initial action category prediction score is generated frame by frame. The output layer of MS-TCN++ normalizes the initial action category prediction score to obtain the probability of each frame belonging to each action category. The surgical action sequence prediction model is trained in two stages. In the first stage, the multimodal feature extraction network and the cross-modal semantic alignment and fusion network are jointly trained, while the temporal action recognition network is not involved in the training. In the second stage, the parameters of the multimodal feature extraction network and the cross-modal semantic alignment and fusion network trained in the first stage are frozen, and only the temporal action recognition network is trained.

5. The dynamic decision-making method for surgical robots according to claim 4, characterized in that, The fusion sequence is constructed according to the following formula. : in, Indicates feature concatenation operation; , and For three predefined learnable tags, Used to capture and fuse counts and global semantic information at the entire fusion sequence level. Used to inject the timing order and timing position information of video frames on the timeline. As a separator, it is used to distinguish between text guidance information and visual feature sequences; For text feature sequences; It is a sequence of visual features; The cross-modal semantic alignment and fusion network processes the fused sequence and splits it into the following two parts: Sequence counting feature sequences Corresponding to The output of the location contains information about the keyframe sequence. Global information on action types and quantities; Semantically enhanced visual feature sequences : Output corresponding to the position of the original visual feature sequence; The dynamic weighted fusion module includes a global average pooling layer, a 1×1 convolutional layer, and a Sigmoid activation function connected in sequence.

6. The dynamic decision-making method for surgical robots according to claim 4, characterized in that, The first phase of training employs a multi-level contrastive learning strategy, using a multi-level contrastive learning loss function. Defined as: in, For global semantic alignment loss, For action-level alignment loss, For sequence-level alignment loss, The weights corresponding to each alignment loss; During the first stage of training, a classification loss function is used to optimize the temporal action recognition network.

7. The dynamic decision-making method for surgical robots according to claim 1, characterized in that, In step S310, the robot is placed in The state variables at time t are described as follows: In the formula, for The stage variable at time, by Standard operating status and A random event state composition; for The action variable at time t, by Executable operation status Composition; among which, in time: For the first The first standard operating state is used to represent the first... Are the standard operations performed? ; express The time robot is executing the first A standard operating procedure; express The time robot did not execute the first A standard operating procedure; For the first The nth random event state is used to represent the nth random event state. Whether a random event occurs. ; express The moment occurred A random event; express The time has not yet occurred. A random event; For the first The executable operation state is used to represent the first executable operation state. Whether an executable operation is triggered. ; express The moment triggered the first An executable operation, then in The time-lapse robot will perform this executable operation; express The time did not trigger the first One executable operation.

8. The dynamic decision-making method for surgical robots according to claim 7, characterized in that, In step S320, the VAR-based phase-action model is expressed as follows: in, It is a sequence of historical stage variables consisting of p past stage variables; It is a linear function; The matrix represents the parameters to be estimated. for The random perturbation term at time step; Will The state of each executable operation at any given time Represented as: in, express The first moment There are 1 executable operation state; p is the total lag order; q is the lag order; Indicates past moment The The standard operating state for the first Executable operation status The impact; It is in the past moment No. A standard operating state. ; Indicates past moment The state of the j-th random event is paired with the th Executable operation status The impact; It is in the past moment No. A random event state, ; yes Time of the first Random noise term of each executable operation state.

9. The dynamic decision-making method for surgical robots according to claim 8, characterized in that, In step S330, Granger causality tests were performed on the variables in the VAR-based stage-action model to confirm the causal relationships between the variables, including: We set the time dimension n, the total lag order p, and the significance level α, and utilize the historical stage variable series. and action variables Constructing time-series data; A constrained model without lagged terms of causal variables and an unconstrained model with lagged terms of causal variables are constructed respectively. The expression of the constrained model is as follows: in, These are constant terms in the constraint model; In the constraint model Its own lag q-th order term The coefficients during regression; The expression for the unconstrained model is as follows: in, This refers to the constant term in the unconstrained model. In representing an unconstrained model Its own lag q-th order term The coefficients during regression; The F-statistic is calculated to assess whether the improvement in predictive power of the unconstrained model compared to the constrained model is statistically significant. The F-statistic is calculated based on the sum of squared residuals. in, and , respectively, are the sum of squared residuals for the constrained and unconstrained models; Q is the total number of lagged terms of the constrained causal variable. ; If the calculated F-statistic is greater than the critical value of the F-distribution corresponding to the significance level α, This indicates the sequence of variables in the historical stage. It constitutes Action variables at time The causal driving factors; otherwise, it indicates a historical stage variable sequence. Not constituted Action variables at time The causal driving factors.

10. The dynamic decision-making method for surgical robots according to claim 1, characterized in that, In step S330, a causal relationship network in the robot operation process is constructed based on the results of the Granger causality test, forming a causal rule generator, including: Extract significant causal pairs from the results of Granger causality tests; All extracted causal pairs are represented as edges of a directed graph, and a causal graph is constructed with each variable as a node; Check the causal graph for cyclic or redundant causal relationships to ensure logical consistency; The paths and relationships in a logically consistent causal graph are transformed into rules to form the causal rule generator.

Images