Surgical robot control method, apparatus and product based on world motion model

Abstract

Embodiments of the present application disclose a surgical robot control method and device based on a world action model, and a product; the method comprises: acquiring observation data of a current surgical scene, a surgical robot state and a task instruction; inputting the observation data, the surgical robot state and the task instruction into a world action model to generate a candidate visual-action pair, the candidate visual-action pair comprising a future surgical visual state and a candidate action sequence; performing safety evaluation on the candidate action sequence according to the future surgical visual state; in response to the candidate action sequence meeting a surgical safety requirement, determining the candidate action sequence as a target action sequence; and rolling out the target action sequence and re-acquiring the observation data of the surgical scene and the surgical robot state to generate a next target action sequence, so that the surgical robot can continuously update the action sequence according to the change of the surgical scene.

Description

Technical Field

[0001] This application relates to the field of surgical robot control technology, and in particular to a surgical robot control method, device and product based on a world motion model. Background Technology

[0002] With the development of minimally invasive surgery and robot-assisted surgery technologies, surgical robots can assist in performing operations such as clamping, suturing, cutting, traction, and tissue dissection through master-slave teleoperation. In order to reduce the burden of repetitive operations on doctors and improve the stability of standardized surgical procedures, surgical robots are gradually developing towards having a certain degree of autonomous control capabilities.

[0003] In the field of robot control, neural network models can be used to generate robot actions based on current scene observations, robot state, and task instructions. However, surgical scenarios present challenges such as non-rigid soft tissue deformation, surgical instrument obstruction, bleeding, smoke, liquid reflection, suture movement, and changes in endoscopic perspective. Therefore, the actions generated by surgical robots must also meet safety requirements such as anatomical structure protection, tissue deformation control, instrument collision avoidance, and robot range of motion.

[0004] Existing control methods often generate actions directly based on current observations, lacking sufficient correlation between future scenario prediction and action generation. Consequently, they struggle to adequately account for potential changes in the surgical scenario after action execution. Furthermore, when the actual surgical scenario changes, actions generated but not yet executed may no longer be suitable for the current surgical situation, impacting the safety and adaptability of surgical robot control.

[0005] Therefore, how to enable the motion control of surgical robots to combine predicted changes in the surgical scenario and adapt to the dynamic changes in the actual surgical scenario is a technical problem that needs to be solved. Summary of the Invention

[0006] One objective of this application is to provide a surgical robot control method, device, and product based on a world motion model, which at least solves the problem in the prior art that surgical robots have difficulty performing pre-execution safety assessments based on the future surgical visual state corresponding to candidate actions and continuously updating the action sequence according to changes in the surgical scenario.

[0007] To achieve the above objectives, some embodiments of this application provide the following aspects: In a first aspect, some embodiments of this application provide a surgical robot control method based on a world motion model, the method comprising: Acquire observation data of the current surgical scene, the status of the surgical robot, and task instructions; The observation data, the state of the surgical robot, and the task instructions are input into the world action model to generate candidate vision-action pairs, which include future surgical vision states and candidate action sequences. The candidate action sequence is safety evaluated based on the future surgical visual state; In response to the candidate action sequence meeting the surgical safety requirements, the candidate action sequence is determined as the target action sequence; The target action sequence is executed continuously, and the observation data of the surgical scene and the state of the surgical robot are reacquired to generate the next target action sequence.

[0008] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0009] Thirdly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0010] Compared with related technologies, the solution provided in this application generates candidate visual-action pairs, including the future surgical visual state and candidate action sequences, through a world action model. A pre-execution safety evaluation is performed on the candidate action sequences based on the future surgical visual state, and target action sequences that meet surgical safety requirements are executed in a rolling manner, establishing a correspondence between candidate actions and their predicted surgical scenarios. Simultaneously, through re-prediction evaluation after action correction, selection of the best candidate action, surgical-specific safety evaluation, and feedback on the deviation between the actual and predicted scenarios, subsequent actions can be updated when the surgical scenario changes, which helps improve the adaptability of action selection to the current surgical scenario. Attached Figure Description

[0011] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0012] Figure 1 An exemplary flowchart of a surgical robot control method based on a world motion model is provided for some embodiments of this application; Figure 2 An exemplary flowchart of another surgical robot control method based on a world motion model provided in some embodiments of this application; Figure 3An exemplary flowchart of another surgical robot control method based on a world motion model provided in some embodiments of this application; Figure 4 An exemplary structural diagram of the electronic device provided for some embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Figure 1 This application provides a flowchart of a surgical robot control method based on a world motion model. The method can be executed by a control device for the surgical robot, which can be integrated into the surgical robot or function as a separate electronic device communicating with the surgical robot. The surgical robot state refers to the overall state of the surgical robot, including one or more of the following: instrument end-effector pose, robot joint state, and gripper opening / closing state.

[0015] The world model is a computational model used to characterize the state of the environment and the laws governing changes in that state. Based on the current state of the environment and the actions acting upon it, the world model can predict the future state of the environment after an action is performed. In a surgical robot control scenario, the environmental state can be jointly characterized by the visual information of the surgical scene and the state of the surgical robot. Changes in the environmental state can include the movement of surgical instruments and the changes in the scene caused by the surgical instruments acting on the tissue.

[0016] The world-action model in this application combines the state prediction capability of a world model with the motion generation process of a surgical robot. Based on observation data of the current surgical scene, the state of the surgical robot, and task instructions, the world-action model generates candidate motion sequences and predicts the future surgical visual state corresponding to these candidate motion sequences. The future motion sequences represent the motion sequences that the model needs to predict; during the model inference phase, the future motion sequences generated by the model serve as candidate motion sequences.

[0017] The World Action Model (WAM) can be pre-trained using surgical operation data. This surgical operation data can include observation data of the surgical scene, the corresponding surgical robot state, task instructions, robot action sequences, and the surgical scene data formed after the execution of these actions. Through this surgical operation data, the WAM can learn the relationship between robot actions and subsequent scene changes in different surgical scenarios. During the model application phase, the WAM can generate the corresponding future surgical visual state before the actual execution of candidate actions, thus providing a predictive basis for the selection of candidate actions.

[0018] The surgical robot control method may include the following steps.

[0019] S101. Obtain observation data of the current surgical scene, surgical robot status, and task instructions.

[0020] Specifically, such as Figure 2 and Figure 3 The observation data is used to characterize the surgical scene in which the surgical robot is currently located, and may include current surgical images acquired by the endoscope or image sequences composed of continuous surgical images. Through this observation data, visual information about surgical instruments, target tissues, and related anatomical structures in the current surgical scene can be obtained.

[0021] In one embodiment, raw images can be acquired from the endoscopic imaging device of the surgical robot, and processed such as resizing, image normalization, or feature extraction to obtain observation data suitable for input into the world motion model. The observation data can be in the form of surgical images or in the form of visual features extracted from the surgical images.

[0022] Surgical robot state is used to characterize the current operating state of the surgical robot, and can include one or more of the following: the pose of surgical instruments, the joint state of the surgical robot, and the operating state of the instrument end effector. The pose of surgical instruments can be represented by their position and orientation in the surgical robot coordinate system, and the operating state of the instrument end effector can indicate whether the instrument end effector is currently in an open, closed, or other operating state.

[0023] Task instructions characterize the surgical procedure to be performed, such as clamping, traction, suturing, cutting, or tissue dissection. These instructions can be in natural language or predefined structured instruction formats. For natural language instructions, task features can be obtained through text encoding; for structured instructions, task features can be formed based on information such as task type, target object, and task stage.

[0024] Furthermore, the observation data and the surgical robot state can be correlated according to the data acquisition time, so that the observation data and the surgical robot state input into the world motion model can represent the surgical scene and robot operation state under the same control moment or the same control cycle. Subsequently, the observation data, the surgical robot state, and the task instructions are used together as input to the world motion model.

[0025] S102. Input the observation data, the state of the surgical robot and the task instructions into the world action model to generate candidate vision-action pairs, which include future surgical vision states and candidate action sequences.

[0026] Specifically, the world action model performs feature processing on the observed data, the surgical robot's state, and the task instructions, and forms the current surgical control context based on the processed visual information, robot state information, and task information. This surgical control context characterizes the current surgical scenario, the current state of the surgical robot, and the surgical task to be completed.

[0027] The world motion model generates candidate motion sequences based on the surgical control context. These candidate motion sequences can include multiple robot actions arranged in execution order, each of which can be used to control the position, orientation, or operational state of a surgical instrument. Robot actions can be represented in the form of target pose of the instrument's end effector, pose change, motion speed, robot joint control parameters, or end effector manipulation parameters.

[0028] While generating candidate action sequences, the world action model predicts the future visual state of the surgical scene after executing the candidate action sequences, based on the current surgical scene and the candidate action sequences. The future visual state may include one or more predicted surgical images at future moments, or it may include visual features that can characterize changes in the future surgical scene.

[0029] In one embodiment, the world action model can use the current surgical scene as the initial state and predict subsequent surgical scenes based on the action changes represented by the candidate action sequence. In this way, the visual results that the candidate action sequence may produce can be obtained without actually executing the candidate action sequence.

[0030] The world action model combines candidate action sequences and their corresponding future surgical visual states into candidate vision-action pairs. The future surgical visual state in the candidate vision-action pair represents the surgical scene predicted after the execution of the corresponding candidate action sequence, thus establishing a correspondence between robot actions and the possible changes in the surgical scene that the robot actions may produce.

[0031] S103. Perform a safety evaluation on the candidate action sequence based on the future surgical visual state.

[0032] Specifically, before the candidate action sequence is actually executed, the future surgical visual state in the candidate vision-action pair is input into the safety evaluation process. The safety evaluation process is used to determine whether the surgical scene predicted after the candidate action sequence is executed meets the surgical safety requirements corresponding to the current surgical task.

[0033] In one embodiment, surgical safety constraints corresponding to the current surgical task can be preset, and state information related to the surgical safety constraints can be extracted from future surgical visual states. By comparing the extracted state information with the surgical safety constraints, a safety evaluation result for the candidate action sequence can be obtained.

[0034] When the future surgical visual state includes predictions of multiple future moments, the predicted surgical scenario at each future moment can be evaluated separately, and the overall safety evaluation result of the candidate action sequence can be determined by combining the evaluation results of each future moment. Therefore, it is possible to evaluate the changes in the surgical scenario that may occur during the continuous execution of the candidate action sequence, rather than being limited to evaluating the surgical scenario at the end of the candidate action sequence execution.

[0035] Safety evaluation results can be expressed as safety level, safety score, or a judgment on whether surgical safety requirements are met. The safety evaluation results are correlated with corresponding candidate action sequences to determine whether the candidate action sequences can be used to control the surgical robot.

[0036] S104. In response to the candidate action sequence meeting the surgical safety requirements, the candidate action sequence is determined as the target action sequence.

[0037] Specifically, when the safety evaluation results indicate that the candidate action sequence meets the surgical safety requirements, the candidate action sequence is determined to be suitable for the current surgical scenario, and the candidate action sequence is determined as the target action sequence.

[0038] The control device can convert the target motion sequence into control commands that the surgical robot can recognize, based on the surgical robot's control interface. For example, when the target motion sequence is represented by the instrument end-effector pose or pose change, the target motion sequence can be converted into robot joint control quantities based on the surgical robot's kinematic relationships; when the target motion sequence is already represented by robot joint control quantities, robot control commands can be generated according to the corresponding execution sequence.

[0039] The target action sequence can include multiple actions controlling the continuous movement of surgical instruments, or it can include the movement of the surgical instruments and end-effector actions. The control device can send the corresponding control commands to the actuators of the surgical robot according to the order of the actions in the target action sequence.

[0040] S105. Execute the target action sequence in a rolling manner, and reacquire the observation data of the surgical scene and the state of the surgical robot to generate the next target action sequence.

[0041] Specifically, rolling execution refers to the repeated generation and execution of actions based on constantly updated surgical scenario information during the control of the surgical robot, rather than generating a fixed sequence of actions covering the entire surgical procedure at the initial moment and executing it all at once.

[0042] In one embodiment, the control device can first control the surgical robot to perform a portion of the target action sequence. After or during the execution of the corresponding action, observation data of the surgical scene is acquired again through the endoscopic imaging device, and the updated state of the surgical robot is obtained from the control system of the surgical robot.

[0043] The reacquired observation data is used to characterize the actual surgical scenario after the corresponding actions are performed, and the reacquired surgical robot state is used to characterize the actual operating state of the surgical robot after the corresponding actions are performed. Since soft tissue may deform under the action of surgical instruments, and the perspective of surgical instruments, sutures, or endoscopes may also change during the execution of actions, the reacquired data can be used to form the current surgical control context for the next control cycle.

[0044] Furthermore, the reacquired observation data and the surgical robot's state, along with the current task instructions, are input into the world action model. The processes of candidate vision-action pair generation, safety assessment, and target action sequence determination are then re-executed to generate the next target action sequence. As the target action sequence is executed, the current task instructions can remain unchanged or be updated to the next stage's task instructions, depending on the completion status of the surgical task.

[0045] The above process can be repeated according to the control cycle of the surgical robot, so that the target action sequence generated in each control cycle is based on the current actual surgical scenario and the current state of the surgical robot, until the surgical operation corresponding to the task instruction is completed, or the execution of the surgical robot's actions is stopped according to the control conditions.

[0046] In this embodiment, the world action model learns the relationship between the surgical scene, robot actions, and subsequent scene changes to generate candidate vision-action pairs, including the future surgical visual state and candidate action sequences. This establishes a correspondence between the candidate action sequences and the surgical scene changes they may cause. By performing a safety assessment based on the future surgical visual state before action execution, the action selection process can incorporate predicted surgical scene changes. By continuously executing the target action sequence and re-acquiring observation data and the surgical robot state, subsequent actions can adapt to the dynamic changes of the actual surgical scene, thereby improving the safety and adaptability of robot control in dynamic surgical scenarios.

[0047] In one embodiment, after performing a safety evaluation of the candidate action sequence based on the future surgical visual state, the method further includes: In response to the candidate action sequence failing to meet surgical safety requirements but meeting preset correction conditions, the candidate action sequence is corrected based on the safety evaluation results.

[0048] When evaluating the candidate action sequence for safety, a continuous safety score and a hard safety constraint judgment result can be obtained. The hard safety constraints may include anatomical no-entry zone constraints, instrument collision constraints, non-target tissue deformation constraints, and robot kinematic constraints. The continuous safety score is used to characterize the degree of deviation or safety margin between the candidate action sequence and the corresponding safety requirements.

[0049] The preset correction conditions are used to determine whether candidate action sequences that do not meet surgical safety requirements have feasible correction space. In one embodiment, when the candidate action sequence does not trigger hard safety constraints and the safety evaluation results indicate that the corresponding safety deviation can be eliminated by adjusting the action, it can be determined that the candidate action sequence meets the preset correction conditions.

[0050] For example, if the continuous safety score corresponding to the candidate action sequence does not meet the safety score requirements, but the corresponding prediction result has not entered the anatomical restricted area, has not caused instrument collision, has not caused unacceptable non-target tissue deformation, and has not exceeded the allowable range of robot kinematics, the position, posture, speed or acceleration of the corresponding action can be adjusted according to the safety evaluation result.

[0051] Based on the safety evaluation results, the timing or segment of the action that causes safety risks in the candidate action sequence, the corresponding risk type, and the degree of deviation from surgical safety requirements can be determined. For candidate action sequences comprising multiple actions, the action causing the safety risk, as well as some actions adjacent to it, can be adjusted; alternatively, the entire candidate action sequence can be considered as the object to be corrected. The following will refer to the candidate action sequence or action block within it that requires correction as the action block to be corrected.

[0052] In one embodiment, the correction result, which is relatively close to the action block to be corrected, can be solved within the safe and feasible set. The objective function for action correction is: Among them, among them, The action block to be corrected. For the corrected action block, Adjust the weight matrix for the action. A safe and feasible set that simultaneously meets the requirements for anatomical restricted area distance, instrument collision gap, tissue deformation, joint limitation, speed, and acceleration.

[0053] Action correction weight matrix This is used to represent the weights of actions at different action dimensions or at different times during the correction process. (Through the safe and feasible set) Minimizing the weighted difference between the modified action block and the action block to be modified can reduce the change to the original candidate action while meeting the surgical safety requirements, so that the modified action maintains the task execution intention of the original candidate action sequence.

[0054] In practice, corresponding constraints can be set for the safety feasibility set based on the risks reflected in the safety assessment results. For example, when the safety assessment results indicate that the distance between the instrument position and the anatomical restricted area is close to the safety boundary, the distance can be increased by adjusting the position or direction of the corresponding action; when the gap between instruments is close to the safety boundary, the movement trajectory of the relevant instruments can be adjusted; when the deformation of non-target tissue is close to the upper limit of the allowable range, the displacement of the corresponding action can be reduced or the direction of action of the instrument can be changed; when the position, speed, or acceleration of the robot joint is close to the allowable boundary, the corresponding robot action parameters can be adjusted.

[0055] By solving the action correction objective function, the safe and feasible set can be obtained. The modified action block inside When the object to be corrected is part of a candidate action sequence, the corresponding action block in the candidate action sequence can be replaced with the corrected action block to obtain the corrected action sequence.

[0056] The corrected action sequence is input as action condition into the world action model for re-prediction, and the safety of the corrected action sequence is re-evaluated based on the re-prediction results.

[0057] Specifically, action correction alters one or more actions in the candidate action sequence, and the surgical scenario that may result from the execution of the corresponding actions will also change accordingly. Therefore, the future surgical visual state corresponding to the candidate action sequence before correction is no longer used as the basis for safety evaluation of the corrected action sequence. Instead, the world action model is used to re-predict the future surgical visual state that the corrected action sequence may produce.

[0058] In one embodiment, the modified action block Or it may include the modified action block. The action sequence is used as known action conditions and input into the same world action model. The world action model keeps the action feature sequence corresponding to the modified action unchanged and performs progressive denoising on the future visual feature sequence to obtain the future surgical visual state corresponding to the modified action sequence.

[0059] During the re-prediction process, the world action model predicts the surgical scene to be formed after executing the corrected action sequence, based on the observation data of the current surgical scene, the current state of the surgical robot, the task instructions, and the corrected action sequence as known action conditions. Since the corrected action sequence is used as the input of known action conditions, the future surgical visual state obtained through re-prediction corresponds to the corrected action sequence.

[0060] After obtaining the future surgical visual state corresponding to the corrected action sequence, the same safety evaluation method as before the correction is used to re-evaluate the safety of the corrected action sequence based on the re-predicted future surgical visual state. Therefore, it is possible to re-determine whether the corrected action sequence meets the surgical safety requirements, rather than simply determining whether the corrected action sequence can be executed based on whether the action parameters are within the safe and feasible set.

[0061] In response to the modified action sequence meeting surgical safety requirements, the modified action sequence is determined as the target action sequence.

[0062] Specifically, when the corrected action sequence meets the surgical safety requirements based on the re-predicted future surgical visual state, it indicates that the predicted surgical scenario corresponding to the corrected action sequence has passed the safety evaluation. At this point, the corrected action sequence can be determined as the target action sequence, and control instructions for the surgical robot can be generated based on the target action sequence.

[0063] When determining the target action sequence, the correspondence between the corrected action sequence and the re-predicted future surgical visual state can be preserved so that the corresponding future surgical visual state can be used to characterize the predicted execution result of the target action sequence during subsequent rolling execution.

[0064] In response to the candidate action sequence failing to meet the surgical safety requirements and the preset correction conditions, or the corrected action sequence still failing to meet the surgical safety requirements, a new candidate visual-action pair is generated. Specifically, when a candidate action sequence triggers a hard safety constraint, there is no feasible solution in the safe and feasible set, or the candidate action sequence cannot be adjusted to enter the safe and feasible set based on the current range of motion of the surgical robot, it can be determined that the candidate action sequence does not meet the preset correction conditions. In this case, the candidate action sequence is not corrected, nor is it sent to the surgical robot for execution.

[0065] The control device can re-invoke the world-action model, resample candidate action sequences based on observation data of the current surgical scene, the state of the surgical robot, and task instructions, and predict the future surgical visual state corresponding to the resampled candidate action sequences, thereby forming new candidate visual-action pairs. When using a generative world-action model, candidate visual-action pairs different from the previous generation results can be generated by changing the initial noise of the visual generation branch and the action generation branch.

[0066] Even if the corrected action sequence is within the safe and feasible set corresponding to the action parameters, the surgical scenario predicted based on the corrected action sequence may still fail to meet surgical safety requirements. Therefore, if it is determined from the re-predicted future surgical visual state that the corrected action sequence still does not meet surgical safety requirements, the corrected action sequence will not be identified as the target action sequence.

[0067] The control equipment re-inputs the observation data of the current surgical scene, the state of the surgical robot, and the task instructions into the world motion model, resamples candidate motion sequences, and generates corresponding future surgical visual states. The regenerated candidate visual-motion pairs re-enter the safety evaluation process to redetermine the target motion sequences that can meet the surgical safety requirements.

[0068] In this embodiment, for candidate action sequences that do not meet surgical safety requirements but have feasible correction space, a small correction is made within the safe and feasible set, which can balance action safety and the original task execution intention. The corrected action sequence is re-input into the same world action model as action conditions, and a safety evaluation is performed again based on the future surgical visual state predicted by the re-prediction. This can verify the possible changes in the surgical scene caused by action correction and reduce the situation where correction is completed only based on action parameters without verifying the corresponding visual consequences. For action sequences that do not meet the correction conditions or still do not meet the surgical safety requirements after correction, candidate visual-action pairs are regenerated, which can prevent the corresponding action sequence from entering the execution process.

[0069] In one embodiment, the world action model includes a scene understanding expert, a surgical state prediction expert, and a motion control expert; The scene understanding expert is used to determine the semantic and anatomical information of the surgical scene based on the observation data and the task instructions; The surgical status prediction expert is used to predict the future surgical visual status; The motion control expert is used to generate the candidate motion sequence; The scene understanding expert, the surgical state prediction expert, and the motion control expert interact with each other through semantic-visual-motion joint attention; The world action model combines the future surgical visual state output by the surgical state prediction expert with the candidate action sequence output by the action control expert to form the candidate visual-action pair.

[0070] Specifically, the scene understanding expert processes observation data and task instructions to obtain scene semantics and anatomical structure information related to the current surgical task. The scene understanding expert may include a pre-trained visual language model and a semantic Transformer layer. The pre-trained visual language model is used to jointly encode endoscopic images and task instructions, and the semantic Transformer layer is used to perform semantic association processing on the jointly encoded features to obtain a semantic feature sequence. .

[0071] In practice, the endoscopic images can first be encoded into visual label sequences using an image encoder, and the task instructions in natural language or structured form can be encoded into task label sequences using a text encoder. Subsequently, the visual label sequences and task label sequences can be input into a pre-trained visual-language model, establishing a connection between the surgical instruments, tissues, and anatomical structures in the images and the objects and intentions in the task instructions.

[0072] The semantic Transformer layer forms a sequence of semantic features based on the associated visual labels and task labels. The semantic feature sequence It can characterize the semantic relationships between the current surgical task, target tissue, non-target tissue, surgical instruments, and anatomical structures.

[0073] In one embodiment, a scene understanding expert can base their work on semantic feature sequences. Output surgical scene semantics and anatomical structure information. The surgical scene semantics can characterize the task type, operation object, and task stage of the current surgical operation; the anatomical structure information can be represented by tissues, surgical instruments, needles, or masks, key points, positional features, or corresponding confidence levels of anatomical restricted areas.

[0074] The surgical state prediction expert is used to process visual information of the current surgical scene and predict changes in the surgical scene as the robot moves. The surgical state prediction expert may include a video encoder, a video generation Transformer, and a video decoder.

[0075] A video encoder can encode the current surgical image or consecutive surgical images from previous moments to obtain a sequence of visual features. The visual feature sequence This is used to characterize surgical instruments, tissues, needles, sutures, and their spatial relationships and motion states in the current surgical scene. The video generation Transformer is used to predict the visual features of the future surgical scene in the visual feature space, and the video decoder is used to decode the predicted visual features into the future surgical visual state.

[0076] The future surgical visual state can be represented by future surgical images, future surgical image sequences, or future visual feature sequences. The future surgical visual state is used to characterize the possible changes in instrument positions, tissue states, and surgical scene after the execution of candidate actions.

[0077] The motion control expert is used to generate candidate motion sequences based on the surgical robot's state, task information, and visual and semantic information obtained through feature interaction. The motion control expert may include a motion encoder, a motion Transformer, and a motion decoder.

[0078] The motion encoder is used to encode the state of the surgical robot and the motion information during the motion generation process into a sequence of motion features. The surgical robot state can include the end effector pose, joint states, and gripper opening / closing states. The motion Transformer is used to determine the motion feature sequence. The semantic and visual information obtained through joint attention are used to generate subsequent action features, and the action decoder is used to convert the subsequent action features into candidate action sequences.

[0079] The candidate action sequence may include one or more of the following: position and orientation of the surgical instrument end effector, position change, orientation change, robot joint control parameters, and gripper opening / closing parameters. Multiple actions in the candidate action sequence are arranged in a predetermined execution order to control the surgical robot to complete the operations corresponding to the task instructions.

[0080] Semantic-visual-motor joint attention facilitates feature interaction among the scene understanding expert, the surgical state prediction expert, and the motion control expert. Semantic feature sequence. Visual feature sequences and action feature sequence The features were derived from different experts. To enable each expert to utilize information from other modalities when processing their respective modal information, semantic-visual-action joint attention computation was performed on the feature sequences of the three experts.

[0081] For experts According to the characteristic sequences of the relevant experts Generate query ,key Sum : ; in, Indicates a scene understanding expert, Indicates an expert in predicting surgical outcomes. Indicates a motion control expert; Experts The corresponding feature sequence; , and Experts Generate the parameter matrix used for querying, keys, and values.

[0082] Scene understanding experts based on semantic feature sequences Generate query Surgical status prediction experts based on visual feature sequences Generate query Motion control experts base their decisions on motion characteristic sequences. Generate query .

[0083] The common key is obtained by concatenating the keys generated by the three experts. The values ​​generated by the three experts are concatenated to obtain a common value. : The semicolon indicates concatenation along the direction of the feature sequence. Common key and common value It also includes scene semantic information, visual state information, and robot action information.

[0084] For any expert Using the query generated by the expert himself With common key Perform matching, and based on the matching results, extract common values. Select features relevant to the current processing task to obtain expert... Corresponding update features : in, For common key transpose, To query the feature dimensions corresponding to the keys, Used to scale the results of a query matching the key. The updated features are obtained by expert m after semantic-visual-action joint attention calculation.

[0085] For scene understanding experts, they use queries From common key and common value The process involves acquiring visual change and action information relevant to the current scene understanding to obtain updated semantic features. Therefore, scene understanding experts can combine the state and motion information of the surgical robot when identifying task objects and anatomical structures.

[0086] For surgical status prediction experts, the use of queries From common key and common value Obtain task semantics and action information related to future scene prediction to obtain updated visual features. Therefore, surgical status prediction experts can combine the operational intent represented by the task instructions with the motion characteristics generated by the motion control experts to predict the future changes in the surgical scenario that the corresponding actions may cause.

[0087] For motion control experts, the use of queries From common key and common value The process involves acquiring semantic and visual state information of the surgical scene related to action generation to obtain updated action features. Therefore, motion control experts can combine target tissues, anatomical structures, and predicted visual changes to generate candidate motion sequences.

[0088] Updated semantic features Visual features and action characteristics The data is then returned to the corresponding expert, whose feedforward network continues processing. Through the joint attention computation, the three experts can exchange features relevant to the current surgical task without having to use the final output of one expert as a one-way input to another.

[0089] Scene understanding experts can analyze the updated semantic features The system processes the data to determine the corresponding location of the operation object and operation intention in the current surgical scene, and identifies surgical instruments, target tissues, non-target tissues and anatomical structures related to the current surgical task.

[0090] For example, when the task instruction is to clamp the needle, the scene understanding expert can determine the needle, the needle holder, and the target clamping position; when the task instruction is to pull the tissue, the scene understanding expert can determine the scene semantics corresponding to the target tissue, the pulling position, and the pulling direction; when the task instruction is to cut along the marked line, the scene understanding expert can determine the marked line, the cutting instrument, and the adjacent tissue that needs to be protected.

[0091] The semantic and anatomical information of the surgical scene output by the scene understanding expert can be returned to the joint attention process to constrain the surgical state prediction expert and the action control expert, so that the future surgical visual state and candidate action sequence correspond to the current task object and anatomical structure.

[0092] Surgical status prediction experts can predict surgical status based on updated visual features. The system generates a Transformer from video to predict future visual feature sequences, and then uses a video decoder to obtain the future surgical visual state. This prediction process combines visual information from the current surgical scene, task semantics and anatomical structure information provided by scene understanding experts, and motion features provided by motion control experts. Therefore, the obtained future surgical visual state is used to characterize the surgical scene predicted after the execution of the corresponding candidate motion sequences.

[0093] Motion control experts based on updated motion characteristics The system generates motion features for multiple consecutive actions using an Action Transformer, and obtains candidate action sequences using an Action Decoder. This action generation process combines the current state of the surgical robot, the operational intent represented by the task instructions, the anatomical structure of the current surgical scene, and visual change information provided by a surgical state prediction expert.

[0094] In the joint prediction process, the scene understanding results can constrain the prediction of future surgical visual state and the generation of candidate actions. The future visual changes obtained by the surgical state prediction expert can also affect the generation of candidate actions through joint attention, so that the candidate action sequence generated by the action control expert is correlated with the predicted changes in the surgical scene.

[0095] The world-action model can obtain the future surgical visual state and candidate action sequence in the same joint prediction process, based on the same observation data, surgical robot state, and task commands. The joint conditional distribution learned by the world-action model can be represented as: Where θ represents the model parameters of the world action model. This indicates the current observation of the surgical scene. This indicates the current state of the surgical robot. Indicates task instructions. Indicates the sequence of visual states in the future surgery. Indicates a sequence of future actions. This indicates the number of predicted moments included in the future surgical visual state sequence. This indicates the number of actions contained in the future action sequence.

[0096] The world-action model associates the future surgical visual state output by the surgical state prediction expert with the candidate action sequence output by the motion control expert, forming candidate vision-action pairs. The future surgical visual state in the candidate vision-action pair represents the surgical scene predicted after the corresponding candidate action sequence is executed, and the candidate action sequence represents the robot action that produces the corresponding scene change.

[0097] Therefore, in the subsequent safety evaluation process, the candidate action sequence can be evaluated based on the future surgical visual state in the candidate visual-action pair, without having to re-establish the correlation between the independent visual prediction results and action generation results.

[0098] In this embodiment, scene understanding experts, surgical state prediction experts, and motion control experts process scene semantics, visual state, and robot actions respectively. Feature interaction is achieved among the three experts through semantic-visual-motion joint attention. This allows task semantics and anatomical structure information to participate in future visual state prediction and candidate action generation, while also enabling predicted visual changes to participate in candidate action generation. By forming candidate visual-action pairs with the future surgical visual state output by the surgical state prediction expert and the candidate action sequence output by the motion control expert, a corresponding data foundation can be provided for evaluating the corresponding actions based on the predicted surgical scene.

[0099] In one embodiment, the motion control expert generates the candidate motion sequence according to a first prediction frequency, and the surgical state prediction expert generates multiple future surgical visual states according to a second prediction frequency, wherein the first prediction frequency is higher than the second prediction frequency. Based on the ratio between the first prediction frequency and the second prediction frequency, the candidate action sequence is divided into multiple action groups, and each action group is associated with a future surgical visual state, wherein the future surgical visual state represents the predicted surgical scenario after the corresponding action group is executed.

[0100] Specifically, the motion control of surgical robots requires high temporal resolution to enable surgical instruments to perform position adjustments, posture adjustments, and end-effector manipulations in a continuous sequence. Compared to robot motion, the generation of future surgical visual states requires the prediction and decoding of surgical images or visual features; the prediction frequency can be lower than the motion prediction frequency. Therefore, the world motion model employs a combination of high-frequency motion prediction and low-frequency visual prediction.

[0101] Let the first prediction frequency, i.e. the action prediction frequency, be... The second prediction frequency, i.e., the visual state prediction frequency, is ,and > The ratio between the first prediction frequency and the second prediction frequency is expressed as... In one embodiment, the first prediction frequency and the second prediction frequency are set to make a ratio It is a positive integer. Therefore, within the same prediction time range, the motion control expert generates a continuous... Each action, the surgical status prediction expert generates a result corresponding to the described... The visual state of the future surgery corresponding to each action.

[0102] Motion control experts generate a prediction process containing A candidate action sequence for each action, wherein the candidate action sequence can be represented as... Where t represents the current time, Indicates the sequence from the first action after the current moment to the last action. A candidate action sequence consisting of several actions.

[0103] Surgical state prediction experts generate multiple future surgical visual states in a single prediction process. The number of future surgical visual states is based on the number of actions contained in the candidate action sequence. and frequency ratio Sure.

[0104] The candidate action sequence can be divided into multiple action groups based on the ratio between the first prediction frequency and the second prediction frequency. Specifically, according to each... The method of forming an action group from consecutive actions, for candidate action sequences Perform sequential partitioning. When the number of remaining actions in the candidate action sequence is insufficient... When the time is up, the last remaining action is taken as the last action group.

[0105] In one embodiment, starting from the first action in the candidate action sequence, proceeding according to each consecutive... The actions are divided into groups, with each action forming a separate group. When the number of remaining actions is insufficient... At this point, the remaining actions are divided into the last action group. The number of action groups is equal to the number of future surgical visual states. And it can be determined according to the following formula: in, · Indicates rounding up. Indicates the number of action groups. Indicates the first The total number of movements included at the end of each set of movements. This indicates the total number of actions included before the start of the first action group, and its value is 0.

[0106] Due to the previous The action set has accumulated to include The first action, therefore, the first The action group starts from the next... Starting with one action, and reaching a cumulative total of [number of actions]. The time ends. For each action group before the last action group, each action group includes... A series of actions; when Cannot be When divisible, the last action group includes the remaining actions in the candidate action sequence.

[0107] The above grouping process only segments consecutive actions in the candidate action sequence, without changing the original execution order of each action. The resulting multiple action groups correspond to the future surgical visual state at different prediction times according to their order in the candidate action sequence.

[0108] In one embodiment, the first The future surgical visual state represents the cumulative pre-operative visual state from the current moment. The surgical scenario is predicted after each action. The future surgical visual state and the first The correspondence between the action groups is as follows: in, This indicates that the previous action in the candidate action sequence has been executed. The future surgical visual state is predicted after each action. Indicates the first A set of actions.

[0109] For the first action group, the future surgical visual state This indicates that the execution of the first action will begin from the current moment until the next action. The surgical scene predicted after each action. For the second action group, the future surgical visual state. This indicates that the second set of actions will continue and the total number of actions completed will be accumulated. The surgical scene is predicted after each action. Subsequent action groups correspond sequentially to the future surgical visual state in the same way.

[0110] It should be noted that a future surgical visual state represents the predicted surgical scenario after the corresponding action group is completed, and also represents the surgical scenario accumulated from the current moment until the end of the action group. Therefore, when evaluating the corresponding action group based on the future surgical visual state, the cumulative impact of previous action groups on the surgical scenario can be considered.

[0111] Based on the time correspondence, the surgical scene predicted after the execution of the corresponding action group can be determined according to each future surgical visual state. Alternatively, the future surgical visual state used to evaluate the corresponding action can be determined according to the action position in the candidate action sequence.

[0112] In this embodiment, the motion control expert generates candidate motion sequences using a higher prediction frequency than the surgical state prediction expert, which can maintain the temporal resolution of the surgical robot's motion control. By dividing the motion into groups according to the ratio between the motion prediction frequency and the visual state prediction frequency, and making each motion group correspond to each future surgical visual state, the number of future surgical visual state predictions can be reduced while establishing the temporal relationship between robot actions and predicted surgical scenarios, providing a corresponding data foundation for safety evaluation according to the motion execution stage.

[0113] In one embodiment, the world action model generates multiple candidate visual-action pairs; When there are multiple candidate action sequences that meet the surgical safety requirements, the smoothness of each candidate action sequence is determined based on the amount of change between adjacent actions in each candidate action sequence and the degree of change of the amount of change between adjacent actions. Based on the task completion rate, safety evaluation results, and action smoothness corresponding to each candidate action sequence, multiple candidate action sequences that meet the surgical safety requirements are sorted, and the target action sequence is determined based on the sorting results.

[0114] Specifically, during the inference phase, the observation data of the current surgical scene, the state of the surgical robot, and the task instructions are input into the world action model. The world action model generates future surgical visual states and candidate action sequences based on the input, and forms candidate visual-action pairs by combining the corresponding future surgical visual states and candidate action sequences.

[0115] In one embodiment, the world action model includes a visual generation branch and an action generation branch. The initial noise of the visual generation branch and the action generation branch can be sampled once or multiple times, allowing the world action model to generate multiple candidate visual-action pairs based on different initial noise. Multiple candidate visual-action pairs can be represented as: in, Indicates the number of candidate visual-action pairs. Indicates the first A sequence of future surgical visual states from candidate vision-action pairs. Indicates the first Candidate action sequences from candidate vision-action pairs. Sequence of visual states for each future surgery. All are candidate action sequences in the same candidate visual-action pair Correspondingly.

[0116] By changing the initial noise of the vision generation branch and the action generation branch, multiple candidate vision-action pairs with different action changes and different predicted vision results can be generated under the same current surgical scene, surgical robot state and task instructions, thereby providing multiple candidate results for the selection of target action sequences.

[0117] For each candidate vision-action pair, a safety evaluation is performed on the corresponding candidate action sequence based on the future surgical visual state, resulting in a safety evaluation result. The safety evaluation result may include a judgment on whether surgical safety requirements are met and a continuous safety score characterizing the degree of safety.

[0118] Candidate action sequences that do not meet surgical safety requirements can be corrected or regenerated according to the corresponding safety control procedures. Candidate action sequences that meet surgical safety requirements are retained as candidate action sequences to be sorted. When only one candidate action sequence meets the surgical safety requirements, it can be determined as the target action sequence; when multiple candidate action sequences meet the surgical safety requirements, the task completion rate and action smoothness of each candidate action sequence are further determined.

[0119] For any candidate action sequence that meets the surgical safety requirements, the amount of change between adjacent actions and the degree of change of the amount of change between adjacent actions are determined according to the order of each action in the candidate action sequence.

[0120] Suppose a candidate action sequence includes The action, the first action in the candidate action sequence The action vectors are The smoothness of the candidate action sequence can be determined according to the following action smoothing loss: in, The first in the candidate action sequence One action vector, The number of actions contained in the candidate action sequence. The weighting coefficient for the second term. The action smoothing loss is used for candidate action sequences.

[0121] Action vectors This can include one or more of the following: the position and orientation of the surgical instrument's end effector, robot joint control variables, or gripper opening and closing variables. For motion vectors that include multiple control variables, motion smoothing loss can be calculated under a unified motion representation.

[0122] The first term in the motion smoothing loss is used to determine the amount of change between adjacent motions. When the changes in position, posture, joint control, or gripper opening / closing between adjacent motions are large, the first term increases accordingly; when the changes between adjacent motions are small, the first term decreases accordingly.

[0123] The second term in the motion smoothing loss is used to determine the degree of change in the amount of change between adjacent actions. Among them, This represents the difference between the changes in the rate of change of two consecutive actions. When there is a large abrupt change in the rate of change of the action, the second term increases accordingly; when the change between consecutive actions remains stable, the second term decreases accordingly.

[0124] motion smoothing loss The smaller the value, the smoother the changes between adjacent actions and the magnitude of those changes are in the candidate action sequence, resulting in higher motion smoothness of the candidate action sequence. (Motion smoothing loss) The larger the value, the more significant the action changes or abrupt changes in the rate of action change in the candidate action sequence, and the lower the smoothness of the corresponding candidate action sequence.

[0125] For each candidate action sequence that meets surgical safety requirements, the task completion degree can be determined based on the task instructions and the corresponding future surgical visual state. The task completion degree is used to characterize the degree of closeness between the predicted surgical scene and the task objective after the candidate action sequence is executed.

[0126] For example, for the task of holding a needle, the degree of task completion can be determined based on the relationship between the surgical instrument and the target holding position in the future surgical vision state; for the task of pulling tissue, the degree of task completion can be determined based on the predicted position or predicted deformation direction of the target tissue; for the task of cutting along a marked line, the degree of task completion can be determined based on the correspondence between the predicted motion result of the surgical instrument and the marked line.

[0127] The safety evaluation results are used to characterize the degree to which candidate action sequences meet surgical safety requirements. For multiple candidate action sequences that all meet surgical safety requirements, the safety margin of each candidate action sequence can be further distinguished based on its continuous safety score. The higher the continuous safety score, the greater the safety margin of the candidate action sequence.

[0128] The action smoothness is used to characterize the degree of continuity between adjacent actions in a candidate action sequence. Action smoothing loss can be used... As a criterion for evaluating motion smoothness, the smaller the motion smoothness loss, the higher the motion smoothness of the corresponding candidate motion sequence.

[0129] In one embodiment, according to a pre-set sorting rule, the task completion degree, safety evaluation result, and action smoothness corresponding to each candidate action sequence are comprehensively compared, so that the candidate action sequence with higher task completion degree, better safety evaluation result, and higher action smoothness has a higher sorting position.

[0130] When discrepancies exist between different evaluation results, the candidate action sequences can be prioritized based on the control requirements of the current task, while ensuring that they meet surgical safety requirements. For example, for operations close to the anatomical safety boundary, safety evaluation results can be given priority; for operations that require maintaining the continuity of instrument movement, such as continuous suturing or tissue traction, smoothness of movement and task completion can be further considered among the candidate action sequences that meet surgical safety requirements.

[0131] Based on the sorting results, the candidate action sequence whose sorting position meets the target selection criteria is determined as the target action sequence. In one embodiment, the candidate action sequence with the highest sorting position can be determined as the target action sequence, and the relationship between the target action sequence and its corresponding future surgical visual state is preserved for use in subsequent rolling execution processes.

[0132] In this embodiment, multiple candidate visual-action pairs are generated through a world action model, enabling multiple action selections under the same surgical scenario and task instructions. Among the candidate action sequences that meet surgical safety requirements, the smoothness of the action is determined based on the amount of change between adjacent actions and the degree of change of the amount of change between adjacent actions, which can reduce abrupt changes in the amplitude or rate of change of the action. By combining the task completion degree, safety evaluation results, and action smoothness, multiple candidate action sequences are sorted, and a target action sequence that balances task execution effect and action continuity can be selected while meeting surgical safety requirements.

[0133] In one embodiment, the safety evaluation of the candidate action sequence based on the future surgical visual state includes: The tissue deformation, the contact state between surgical instruments and tissues, and the distance between surgical instruments and anatomical restricted areas are determined based on the future surgical visual state. The gaps between surgical instruments and the robot's kinematic risks are determined based on the candidate action sequence and the state of the surgical robot. The safety evaluation result is determined based on the tissue deformation, the contact state, the distance, the gap, and the robot kinematic risk, and the candidate action sequence is determined based on the safety evaluation result to determine whether it meets the surgical safety requirements.

[0134] Specifically, scene understanding processing can be performed on each future surgical visual state in the candidate vision-action pair to identify surgical instruments, needles, target tissues, non-target tissues and anatomical restricted areas in each future surgical visual state, and output the mask or key points of the corresponding objects and their confidence scores.

[0135] The mask is used to characterize the image region of the corresponding object in the future surgical visual state, and the key points are used to characterize the instrument tip, needle tip, tissue boundary, or other feature location of the corresponding object. By mapping the masks or key points in multiple future surgical visual states, the predicted changes of surgical instruments and tissues during the execution of candidate action sequences can be determined.

[0136] Before determining tissue deformation based on the future surgical visual state, compensation can be made for the overall image changes caused by endoscopic motion, ensuring that the tissue position in different future surgical visual states is under a comparable reference coordinate system. Lens motion compensation can reduce the impact of changes in the endoscopic viewing angle on the judgment of tissue displacement, avoiding the direct misinterpretation of image displacement caused by endoscopic motion as tissue deformation.

[0137] After completing camera motion compensation, tissue deformation can be determined based on the displacement of key tissue points or the displacement of tissue mask boundaries. Specifically, the positions of key points or mask boundaries of the same tissue in different future surgical visual states can be compared, and the predicted displacement and degree of tissue deformation can be determined based on their positional changes.

[0138] For the target tissue, it can be determined whether the deformation of the corresponding tissue is consistent with the expected operation direction based on the current task instructions. For example, in a tissue traction task, the deformation of the target tissue along the traction direction corresponding to the task instructions can be considered a result of task execution. For non-target tissues, it can be determined whether the candidate action sequence might cause unexpected traction or compression based on its predicted deformation.

[0139] The contact state between surgical instruments and tissues can be determined based on the minimum distance between the boundaries of surgical instruments and tissues, the relative direction of movement between surgical instruments and tissues, and the cooperative displacement after contact.

[0140] When the distance between the boundary of the surgical instrument and the boundary of the tissue gradually decreases and the surgical instrument moves toward the tissue, it can be determined that the surgical instrument is in a state of approaching the tissue; when the distance between the two reaches the contact range and the tissue undergoes coordinated displacement with the surgical instrument in the subsequent future surgical vision state, it can be determined that the surgical instrument and the tissue are in a state of contact; when the surgical instrument moves away from the tissue or the distance between the two increases, it can be determined that the surgical instrument and the tissue are in a state of separation.

[0141] Furthermore, by combining the semantics of the surgical scenario and the task instructions output by the expert with the contextual understanding, it can be determined whether the corresponding contact is the target contact required by the task. For example, contact between surgical instruments and the target tissue specified in the task instructions can be considered as expected contact; contact between surgical instruments and non-target tissue, or contact positions deviating from the operating positions specified in the task instructions, can be considered as unexpected contact requiring safety evaluation.

[0142] The distance between the surgical instrument and the anatomical restricted area can be determined based on the mask of the anatomical restricted area. Specifically, a distance transformation can be performed on the anatomical restricted area mask to obtain the distance between each position and the anatomical restricted area in the future surgical visual state, and the minimum distance between the needle tip or the end of the surgical instrument and the anatomical restricted area can be determined based on the position of the needle tip or the end of the surgical instrument.

[0143] When the future surgical visual state includes predictions from multiple time points, the distance between the needle tip or surgical instrument tip and the anatomical no-entry zone can be determined at each prediction time point to assess the change in this distance during the execution of the candidate action sequence. A smaller distance indicates that the needle tip or surgical instrument tip is closer to the anatomical no-entry zone, and consequently, a higher risk associated with the no-entry zone.

[0144] The gaps between surgical instruments can be determined based on the candidate action sequence, the state of the surgical robot, and the geometric information of the surgical instruments. Specifically, based on the current state of the surgical robot and the candidate action sequence, the predicted joint state after executing each candidate action is obtained, and the predicted pose of the surgical instruments at each action moment is calculated using the robot's forward kinematics.

[0145] The geometric envelopes of the surgical instruments are established based on their predicted poses, and the minimum distance between the geometric envelopes of different surgical instruments is calculated. This minimum distance is then used as the predicted gap between the surgical instruments. In one embodiment, the minimum gap between the surgical instruments and the endoscope can also be determined based on their geometric envelopes.

[0146] When the predicted gap between surgical instruments decreases, it indicates an increased risk of instrument collision caused by the candidate action sequence; when the predicted gap is lower than the minimum gap required to prevent instrument collision, it can be determined that the candidate action sequence does not meet the corresponding instrument collision prevention requirements.

[0147] Robot kinematic risk can be determined based on the margins of predicted joint positions, joint velocities, and joint accelerations relative to allowable ranges when executing candidate action sequences. Specifically, the predicted joint states at each action moment can be obtained based on the current surgical robot state and the candidate action sequence, and the distances between the predicted joint positions, joint velocities, and joint accelerations and their corresponding allowable ranges can be compared.

[0148] When the predicted joint position, joint velocity, or joint acceleration is close to the boundary of the corresponding allowable range, the robot's kinematic risk increases; when the predicted joint position, joint velocity, or joint acceleration exceeds the corresponding allowable range, it can be determined that the candidate action sequence violates the robot's kinematic hard safety constraints.

[0149] After obtaining the tissue deformation, the contact state between the surgical instruments and the tissue, the distance between the surgical instruments and the anatomical no-entry zone, the gap between the surgical instruments, and the robot's kinematic risk, the no-entry zone risk, collision risk, tissue deformation risk, and kinematic risk can be determined separately. These are then normalized and weighted fusions to obtain the [missing information - likely a specific result or parameter]. Safety risk value of candidate results for the group: in, Let i be the security risk value of the i-th candidate result. Due to the risk of restricted areas, To mitigate collision risk, To mitigate organizational deformation risks, For robot kinematic risks, These are the weights for the corresponding risks.

[0150] Risk of restricted areas The risk of collision increases as the distance between the needle tip or surgical instrument tip and the anatomical no-entry zone decreases. The risk of tissue deformation increases as the gap between surgical instruments decreases. It increases with the increase of the predicted deformation of the non-target organization. Robot kinematic risk It increases as the joint position, joint velocity, or joint acceleration approaches the boundary of the allowable range.

[0151] The contact state between surgical instruments and tissue can be used to determine the corresponding tissue deformation risk or collision risk. When the surgical instrument makes expected contact with the target tissue corresponding to the task instruction, the corresponding action can be evaluated in conjunction with the degree of deformation of the target tissue; when the surgical instrument makes unexpected contact with non-target tissue, the corresponding tissue deformation risk can be increased or the contact state can be determined as a basis for judging a violation of safety requirements.

[0152] According to the Safety risk value of candidate results The safety score can then be calculated further: in, For the first Security score of candidate results. Security risk value. The larger the value, the higher the safety score. The smaller the value; the lower the safety risk value The smaller the value, the higher the safety score. The larger the value, the higher the safety score. The larger the value, the safer the corresponding candidate action.

[0153] In one embodiment, a safety risk value and safety score can be determined for each action group in the candidate action sequence, and the overall safety evaluation result of the candidate action sequence can be determined by combining the safety evaluation results of each action group. When any action group violates the anatomical no-entry zone, instrument collision, non-target tissue deformation, or robot kinematic hard safety constraints, it can be determined that the corresponding candidate action sequence does not meet the surgical safety requirements.

[0154] When none of the action groups violate the hard safety constraints, the safety scores of each action group can be used to determine whether the candidate action sequence meets the surgical safety requirements. When the safety scores of each action group are within the preset safety score range, the candidate action sequence is determined to meet the surgical safety requirements; when the safety score of at least one action group is not within the preset safety score range, the candidate action sequence is determined to not meet the surgical safety requirements.

[0155] The safety evaluation results may include the restricted area risk, collision risk, tissue deformation risk, robot kinematic risk, safety risk value, and safety score corresponding to each action group, as well as the judgment result on whether the candidate action sequence meets the surgical safety requirements. The safety evaluation results can be used to determine whether the candidate action sequence can be identified as the target action sequence, and can also be used to determine the risk type and action position corresponding to the candidate action sequence that does not meet the surgical safety requirements.

[0156] In this embodiment, by determining tissue deformation, the contact state between surgical instruments and tissue, and the distance between surgical instruments and anatomical restricted areas based on the future surgical visual state, the predicted changes in tissue and anatomical structures after the execution of candidate actions can be evaluated. By determining the gap between surgical instruments and the robot's kinematic risks based on the candidate action sequence and the state of the surgical robot, the executability and collision risks of candidate actions in the robot's motion space can be evaluated. By fusing various risks and converting them into safety scores, a safety evaluation result can be generated for candidate action selection and action correction, thereby determining whether the candidate action sequence meets surgical safety requirements before actual execution.

[0157] In one embodiment, the scrolling execution of the target action sequence includes: In each round, a preset number of actions in the target action sequence are executed, and the observation data of the surgical scene and the state of the surgical robot are reacquired. The preset number is less than the number of actions contained in the target action sequence. Based on the reacquired observation data and the future surgical visual state corresponding to the performed actions, the deviation between the actual surgical scenario and the predicted surgical scenario is determined; In response to the deviation exceeding a preset range, or the scene understanding confidence determined based on the reacquired observation data being lower than a preset threshold, the execution of actions that have not yet been executed in the target action sequence is stopped and discarded, and candidate visual-action pairs are regenerated to determine the next target action sequence; In response to multiple consecutive failures to generate a target action sequence that meets the surgical safety requirements, a pause command is output.

[0158] Specifically, rolling execution means that only a portion of the target action sequence is executed each time. After obtaining the actual surgical scene and surgical robot state after executing the portion of the action, subsequent actions are regenerated, rather than executing all the actions in the target action sequence at once.

[0159] Suppose that the target action sequence generated by the world action model each time includes Each round executes the first action in the target action sequence. One action, of which: ; The preset number of actions to be executed in each round. The number of actions contained in the target action sequence. Because... Less than Therefore, after each round of execution, there are still actions in the target action sequence that have not yet been executed.

[0160] The control device can send the preceding actions to the surgical robot sequentially according to the order of the actions in the target action sequence. The control commands correspond to each action. These control commands can be used to control the position and orientation of the surgical instrument's end effector, the robot's joint states, or the opening and closing states of the grippers. Before execution... After one action, the control device pauses sending control commands for the remaining actions in the target action sequence and reacquires the observation data of the current surgical scene and the state of the surgical robot.

[0161] The reacquired observation data may include actual surgical images or sequences of actual surgical images acquired by the endoscope. The reacquired surgical robot state may include the instrument end-effector pose, robot joint state, and gripper opening / closing state after performing the aforementioned partial actions. The observation data and surgical robot state are used to characterize the actual surgical scenario and robot operating state after performing this round of actions.

[0162] In one embodiment, the preset number of actions executed in each round This can be matched with the cumulative number of movements corresponding to a future surgical visual state. Therefore, before execution... After a certain action, the execution of the preceding action can be determined from the future surgical visual state corresponding to the target action sequence. The surgical scenario is predicted after each action.

[0163] In another embodiment, a future surgical visual state corresponding to the executed action can be selected from a sequence of future surgical visual states based on the position of the executed action in the target action sequence. The selected future surgical visual state is used to characterize the surgical scene predicted after the execution of this round of actions.

[0164] Based on the reacquired observation data, the actual surgical scene formed after this round of actions can be determined; based on the future surgical visual state corresponding to the already performed actions, the originally predicted surgical scene formed after this round of actions can be determined. By comparing the two, the deviation between the actual surgical scene and the predicted surgical scene can be determined.

[0165] In practice, the same visual encoding method can be used to process the reacquired actual surgical image and the corresponding future surgical visual state to obtain the visual features of the actual surgical scene and the visual features of the predicted surgical scene, and the deviation is determined based on the difference between the two sets of visual features.

[0166] The deviation can reflect changes in surgical instruments, target tissues, non-target tissues, needles, sutures, or anatomical structures in the actual surgical scenario relative to the predicted surgical scenario. For example, sudden tissue movement, instrument shifting, changes in needle posture, or changes in the location of anatomical forbidden zones can all increase the deviation between the actual and predicted surgical scenarios.

[0167] In one embodiment, masks or key points of surgical instruments, tissues and anatomical structures can be obtained from the reacquired observation data and the corresponding future surgical visual state, respectively, and the deviation between the actual surgical scene and the predicted surgical scene can be determined based on the positional differences of the corresponding masks or key points.

[0168] When there is a difference in endoscopic perspective between the newly acquired observation data and the corresponding future surgical visual state, the image changes caused by lens movement can be compensated first, and then the actual surgical scene and the predicted surgical scene can be compared to reduce the impact of endoscopic movement on the bias judgment.

[0169] In addition to the discrepancy between the actual and predicted surgical scenarios, scene understanding confidence can also be determined based on reacquired observation data. Specifically, scene understanding experts process the reacquired observation data to identify surgical instruments, needles, target tissues, non-target tissues, and anatomical restricted areas, and output corresponding masks or key points and their confidence levels. The scene understanding confidence of the current surgical scenario can be determined based on the confidence level of the identification results.

[0170] When reacquired observation data contains bleeding, smoke, instrument obstruction, tissue obstruction, or liquid reflection, the scene understanding expert's confidence in recognizing surgical instruments, tissues, or anatomical structures may decrease. A scene understanding confidence level below a preset confidence threshold indicates that the current observation data is insufficient to support the continuation of the original target action sequence.

[0171] During the execution of the target action sequence, actual surgical images and the state of the surgical robot can be continuously acquired, not just before execution is completed. After each action, a check is performed. If the preset number of actions for this round has not been completed, but the deviation between the actual surgical scenario and the predicted surgical scenario has been detected to exceed the preset range, or the scenario understanding confidence is lower than the preset confidence threshold, the execution of this round of actions can be stopped immediately.

[0172] In response to a deviation between the actual surgical scenario and the predicted surgical scenario exceeding a preset range, or a scenario understanding confidence level falling below a preset confidence threshold, the control device stops sending subsequent control commands in the original target action sequence and discards actions in the original target action sequence that have not yet been executed. The unexecuted actions include actions not yet executed in this round and actions in the original target action sequence that follow the actions in this round.

[0173] Discarding actions that have not yet been executed means that these actions will no longer be used as the basis for subsequent control of the surgical robot. Since these actions are generated based on the surgical scenario before the change, they will not be executed if the actual surgical scenario deviates from the predicted result or if the current scenario cannot be reliably identified.

[0174] After halting execution and discarding any unexecuted actions, candidate vision-action pairs are generated again using the reacquired observation data, the reacquired surgical robot state, and the current task instructions, based on the world action model. The regenerated candidate action sequences are then subjected to a safety evaluation, and those sequences that meet surgical safety requirements are designated as the next target action sequences.

[0175] The next target action sequence is generated based on the changed actual surgical scenario and the state of the surgical robot, thereby replacing the actions that have not yet been executed in the original target action sequence. After the next target action sequence is determined, it can be executed in the same rolling execution mode, while continuing to acquire the actual surgical scenario and the state of the surgical robot.

[0176] In one specific embodiment, the world action model generates a target action sequence comprising 16 consecutive actions each time. The surgical robot executes only the first four of these 16 actions per round, and then reacquires the observation data of the surgical scene and the state of the surgical robot. The reacquired data is used to generate the next target action sequence. If tissue movement, instrument displacement, occlusion, or bleeding occurs during execution, the actions that have not yet been executed are immediately discarded, and candidate vision-action pairs are regenerated based on the changed surgical scene.

[0177] After regenerating candidate visual-action pairs, situations may arise where none of the regenerated candidate action sequences meet surgical safety requirements, the candidate action sequences do not meet preset correction conditions, or the corrected action sequences still fail to meet surgical safety requirements. The control device can record the number of consecutive times a target action sequence that fails to meet surgical safety requirements is generated.

[0178] Once a target action sequence that meets surgical safety requirements is regenerated, the rolling execution can resume. If a predetermined number of consecutive attempts fail to generate a target action sequence that meets surgical safety requirements, it indicates that a safe robotic action has not yet been obtained for the current surgical scenario, and the control device outputs a pause command.

[0179] In response to the pause command, the surgical robot stops executing the target action sequence and remains in a paused state to avoid continuing to generate and execute robot actions when a sequence of actions that meets the surgical safety requirements cannot be obtained continuously.

[0180] In this embodiment, only a portion of the target action sequence is executed in each round, and the next target action sequence is generated based on the reacquired observation data and the state of the surgical robot. This allows subsequent robot actions to adapt to changes in the actual surgical scenario. By comparing the actual surgical scenario with the predicted surgical scenario and combining the scenario understanding confidence to determine whether to continue executing actions that have not yet been executed, the original action sequence can be stopped in a timely manner in cases of tissue movement, instrument displacement, obstruction, or bleeding. When a target action sequence that meets the surgical safety requirements cannot be generated continuously, a pause command is output to prevent the surgical robot from continuing to operate when there are no actionable actions.

[0181] In one embodiment, the world action model is trained in the following manner: The first image coordinates of the surgical instruments are determined from the future surgical visual state output by the surgical state prediction expert, and the motion sequence output by the motion control expert is subjected to forward kinematics calculation and camera projection to obtain the second image coordinates corresponding to the first image coordinates. The visual-action consistency loss is determined based on the difference between the first image coordinates and the second image coordinates, and the world action model is trained by combining the visual prediction loss, action prediction loss, action smoothing loss and safety constraint loss.

[0182] Specifically, the world motion model can be trained using a phased training approach.

[0183] In the first training phase, surgical state prediction experts are trained using surgical instructional videos and simulated surgical videos. Training data may include surgical videos and corresponding surgical task descriptions. Through this training phase, the surgical state prediction experts learn the evolutionary patterns of surgical scenarios, such as the movement of surgical instruments, changes in needle posture, suture movement, tissue deformation, and occlusion changes.

[0184] In the second training phase, robot teaching data—aligned with surgical videos, robot movements, and task instructions—is used to jointly train scene understanding experts, surgical state prediction experts, and motion control experts. This robot teaching data can originate from different surgical robot platforms.

[0185] For robot movements originating from different surgical robot platforms, coordinate system unification and normalization can be performed on the robot movements to map the robot movements from different robot platforms to a unified movement representation. Specifically, the end-effector pose can be unified to the surgical robot base coordinate system or the endoscope coordinate system, and the joint angles, end-effector pose, and gripper opening / closing amount can be normalized according to the respective surgical robot's range of motion.

[0186] In the third training phase, the world motion model is fine-tuned using the teaching trajectory of the target surgical robot to adapt the candidate motion sequences generated by the motion control expert to the motion space of the target surgical robot. During the fine-tuning process, vision-motion consistency constraints, motion smoothness constraints, and surgical safety constraints can be added.

[0187] The world-action model is used to jointly generate a sequence of future surgical visual states and a sequence of future actions based on observation data of the current surgical scene, the state of the surgical robot, and task instructions. The joint prediction relationship can be represented by the following conditional distribution: Where θ represents the model parameters of the world action model. This indicates the current surgical observation. This indicates the current state of the surgical robot. Indicates task instructions. Indicates the sequence of visual states in the future surgery. Indicates a sequence of future actions. This indicates the number of visual states included in the future surgical visual state sequence. This indicates the number of actions contained in the future action sequence.

[0188] Surgical state prediction experts can include a visual generation branch, and motion control experts can include a motion generation branch. When training the visual and motion generation branches, real-world future visual features from the training samples are used. and real action sequences Interpolate with the corresponding Gaussian noise respectively.

[0189] Assume realistic future visual features Gaussian noise with the same dimension is , and the actual action sequence Gaussian noise with the same dimension is The scalar time step parameter corresponding to the visual generation branch is The scalar time step parameter corresponding to the action generation branch is The noisy visual features and noisy action sequences are as follows: in, and The values ​​of are all in the range of 0 to 1. Indicates at time step Noisy visual features constructed Indicates at time step The constructed noisy action sequence.

[0190] when or When it equals 1, the corresponding input is Gaussian noise; when it equals 0, the corresponding input is real training data.

[0191] The visual generation branch is based on noisy visual features. Predicting the visual velocity field The action generation branch is based on the noisy action sequence. Predicting motion velocity field Visual prediction loss And action prediction loss They are respectively: in, This represents the target velocity field of the visually generated branch. This represents the target velocity field of the action generation branch. The visual generation branch and the action generation branch can each use corresponding scalar time step parameters and are based on visual prediction loss. And action prediction loss Joint optimization was carried out.

[0192] To ensure that the position of surgical instruments in the future surgical visual state corresponds to the position of instruments calculated from the candidate action sequence, a visual-action consistency constraint can be applied to the future surgical visual state and the candidate action sequence.

[0193] Specifically, from the first The first image coordinates of the instrument tip or needle tip are obtained from the predicted images. The execution step is determined based on the motion sequence output by the motion control expert. Predicted joint status after a movement Through the robot's forward kinematics function and camera projection function The predicted joint state is converted into second image coordinates. : in, This represents the cumulative number of actions corresponding to the j-th future surgical visual state. Indicates execution of the first Predicted joint status after each movement Let be the positive kinematics function of the robot. This is the coordinate transformation matrix from the robot's base coordinate system to the endoscope camera's coordinate system. This is the camera projection function.

[0194] Therefore, the visual-action consistency loss can be determined based on the first image coordinates and the second image coordinates: in, The number of visual states required for future surgery. It is an L1 loss function or a smoothed L1 loss function. This is the visual-motor consistency loss. Visual-motor consistency loss is used to limit the difference between the instrument position in the future surgical visual state and the instrument position calculated based on the candidate action sequence.

[0195] In one embodiment, the motion smoothing loss used during the training of the world action model is: in, For the first action in the action sequence One action vector, The number of actions contained in the action sequence. This is the weighting coefficient for the second term. The first term is used to limit the change between adjacent actions, and the second term is used to limit abrupt changes in the rate of change of actions.

[0196] During training, safety constraint losses can also be calculated based on the predicted distance between the needle tip or surgical instrument and the anatomical no-entry zone, the predicted minimum distance between surgical instruments, the predicted deformation of non-target tissues, and robot kinematic constraints. in, For predicting time index, This is the predicted distance between the needle tip or surgical instrument and the anatomical no-entry zone. The safety threshold for the distance is... The predicted minimum distance between surgical instruments. The minimum clearance required to prevent surgical instruments from colliding. For predictive variables of non-target organizations, This represents the maximum allowable deformation.

[0197] ReLU(·) is used to impose a penalty on portions exceeding a corresponding safety threshold. Penalties for exceeding joint limits, joint velocity, and joint acceleration limits. To constrain losses for safety.

[0198] In one embodiment, the total loss of the world action model is: in, The total loss during model training. For visual prediction loss, For predicting loss in action, For visual-motor consistency loss, For motion smoothing loss, To constrain losses for safety reasons; These are the weighting coefficients corresponding to each loss term.

[0199] By using visual prediction loss and action prediction loss, the ability to generate future surgical visual states and candidate actions can be trained respectively; by using visual-action consistency loss, the position of surgical instruments in the predicted image can be constrained to correspond to the position of instruments calculated from candidate actions; by using action smoothing loss, the changes between adjacent actions and abrupt changes in the rate of action change can be restricted; by using safety constraint loss, constraints can be imposed on anatomical no-entry zones, instrument collisions, deformation of non-target tissues, and robot kinematic boundary violations during model training.

[0200] In this embodiment, the world action model learns the evolution of the surgical scene, the correspondence between vision, action, and command, and the action space of the target surgical robot through a phased training method. By jointly optimizing the visual prediction loss, action prediction loss, vision-action consistency loss, action smoothing loss, and safety constraint loss, the world action model can generate future surgical visual states and candidate action sequences with corresponding relationships, and make the generated results adaptable to the surgical scene and the control requirements of the surgical robot.

[0201] Furthermore, in one specific embodiment, the surgical robot includes two robotic arms, each equipped with a needle holder, and a stereoscopic endoscope, and the surgical robot is used to perform incision suturing operations.

[0202] A stereoscopic endoscope is used to acquire endoscopic images of the current surgical scene. The surgical robot's state can include the end-effector poses of the two robotic arms, the joint states of the robots, and the gripper opening and closing status of the needle holder. Task instructions are used to direct the surgical robot to perform suturing operations on the target incision.

[0203] In this embodiment, robot actions are represented by the pose increments of the dual-arm instrument end effectors and the gripper opening / closing amount. Each action may include the position increments, pose increments, and gripper opening / closing amount of the two needle holders at the corresponding control moment. By representing robot actions using pose increments, their target positions and target poses can be continuously updated based on the current positions and poses of the two needle holders.

[0204] The world motion model comprises scene understanding experts, surgical state prediction experts, and motion control experts. The scene understanding expert extracts information about the needle, suture, needle holder, target tissue, and anatomical restricted areas based on endoscopic images and suturing task instructions. The surgical state prediction expert predicts the future surgical visual state formed after the needle holder performs corresponding actions, and the motion control expert generates candidate action sequences for the two robotic arms.

[0205] Scene understanding experts, surgical state prediction experts, and motion control experts interact through semantic-visual-motion joint attention. Information about needles, sutures, target tissues, and anatomical restricted areas obtained by scene understanding experts is used to predict future surgical visual states and generate candidate actions, while information about future visual changes obtained by surgical state prediction experts is used to generate candidate actions.

[0206] During training, surgical state prediction experts were first trained using surgical videos and corresponding surgical task descriptions, enabling them to learn needle holder movements, needle posture changes, suture movements, tissue deformation, and occlusion changes. Subsequently, three experts were jointly trained using robot teaching data aligned with surgical videos, robot movements, and task instructions. Finally, the world motion model was fine-tuned using the suture teaching trajectory of the target surgical robot.

[0207] When robot teaching data comes from different surgical robot platforms, coordinate system unification and normalization are performed on the robot movements of different platforms to map the robot movements of different platforms to a unified motion representation. After joint training is completed, the unified motion representation is then adapted to the end-effector pose increments and gripper opening and closing amounts of the target surgical robot.

[0208] In this embodiment, the motion control expert generates a candidate action sequence of 16 consecutive actions at a prediction frequency of 30Hz, and the surgical state prediction expert generates the future surgical visual state at a prediction frequency of 5Hz. The 16 consecutive actions are divided into three action groups: the first action group includes actions 1 to 6, the second action group includes actions 7 to 12, and the third action group includes actions 13 to 16.

[0209] The surgical status prediction expert generates the future surgical visual state after the completion of the 6th, 12th, and 16th actions, respectively. The future surgical visual state after the completion of the 6th action corresponds to the first action group, the future surgical visual state after the completion of the 12th action corresponds to the second action group, and the future surgical visual state after the completion of the 16th action corresponds to the third action group.

[0210] By altering the initial noise of the visual generation branch and the action generation branch, the world action model can generate multiple candidate visual-action pairs. Each candidate visual-action pair includes a candidate action sequence and a future surgical visual state that has a temporal correspondence with the candidate action sequence.

[0211] For each candidate vision-action pair, the deformation of the target and non-target tissues, the contact state between the needle holder and the tissue, the distance between the needle body or needle holder and the anatomical restricted area, the gap between the two needle holders, and the robot kinematic risk are determined based on the future surgical vision state and candidate action sequence, and the safety evaluation results are determined accordingly.

[0212] Candidate action sequences that violate anatomical no-entry zones, instrument collisions, non-target tissue deformation, or robot kinematic hard safety constraints will not be identified as target action sequences. For multiple candidate action sequences that meet surgical safety requirements, they will be ranked according to task completion, safety evaluation results, and action smoothness, and the target action sequence will be determined based on the ranking results.

[0213] For candidate action sequences that do not meet surgical safety requirements but have feasible correction space, they are corrected within the safe and feasible set to minimize the difference between the corrected action sequence and the original candidate action sequence. The corrected action sequence is then input into the same world action model as known action conditions to regenerate the future surgical visual state corresponding to the corrected action sequence.

[0214] Based on the regenerated future surgical visual state, the corrected action sequence is re-evaluated for safety. If the corrected action sequence meets the surgical safety requirements, it is determined as the target action sequence; if the corrected action sequence still does not meet the surgical safety requirements, candidate action sequences are resampled.

[0215] In one embodiment, a preset number of times can be set to allow candidate action generation and action correction. If an action sequence that meets the surgical safety requirements is not obtained within the preset number of times, a pause command or a manual takeover request is output.

[0216] After determining the target action sequence, the surgical robot performs only the first 4 of the 16 actions in each round, then reacquires the endoscopic images and the state of the surgical robot, and generates the next target action sequence based on the reacquired data.

[0217] During action execution, the feature deviation between the actual surgical image and the predicted surgical visual state, as well as the scene understanding confidence, are continuously monitored. When tissue movement, needle holder displacement, occlusion, or bleeding is detected, causing the deviation between the actual surgical scene and the predicted surgical scene to exceed a preset range, or causing the scene understanding confidence to fall below a preset confidence threshold, execution is stopped and unexecuted actions in the target action sequence are discarded.

[0218] After discarding actions that have not yet been performed, the newly acquired endoscopic images, surgical robot status, and current suturing task instructions are input into the world action model to regenerate candidate vision-action pairs and perform a safety evaluation to determine the next target action sequence for the adapted surgical scenario. If a target action sequence that meets surgical safety requirements is not obtained multiple times consecutively, the surgical robot pauses action execution and can output a manual takeover request.

[0219] In this embodiment, by using a surgical robot equipped with dual needle holders and a stereoscopic endoscope to perform incision suturing, scene understanding of the world action model, prediction of future visual states, generation of dual-arm movements, safety assessment, movement correction, and rolling execution can be applied to the same surgical task. By executing only a portion of the target action sequence in each round and continuously updating subsequent actions based on the reacquired surgical scene, the movements of the two needle holders can adapt to changes in the needle body, suture, and tissue state. When a sequence of actions that meets the surgical safety requirements cannot be obtained continuously, the execution of actions can be paused or manual intervention can be requested, thus avoiding the continued execution of robot actions that are unsuitable for the current surgical scenario.

[0220] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0221] In one embodiment, this application also provides a surgical robot system, the surgical robot system including a surgical robot and a control device communicatively connected to the surgical robot, the control device being used to execute the surgical robot control method based on the world motion model described in any of the above embodiments.

[0222] The surgical robot may include at least one robotic arm, surgical instruments disposed at the end of the robotic arm, an endoscopic imaging device for acquiring images of the surgical scene, and an actuator for driving the movement of the robotic arm and the surgical instruments. The surgical instruments may include needle holders, clamps, pullers, cutters, or tissue dissectors. The actuator can control the position, orientation, and end-effector state of the surgical instruments according to control commands output by the control device.

[0223] The surgical robot may further include a state acquisition device. The state acquisition device is used to acquire one or more of the following: joint state, end-effector pose, and gripper opening / closing state of the surgical robot, and sends the acquired data as the surgical robot's state to the control device. The endoscopic imaging device is used to acquire images or image sequences of the current surgical scene, and sends the images or image sequences as observation data of the current surgical scene to the control device.

[0224] The control device acquires observation data of the current surgical scene, the state of the surgical robot, and task instructions, and inputs the observation data, the state of the surgical robot, and the task instructions into a world action model to generate candidate vision-action pairs. Each candidate vision-action pair includes a candidate action sequence and the corresponding future surgical visual state.

[0225] The control device performs a safety evaluation on the corresponding candidate action sequences based on the future surgical visual state. When the candidate action sequence meets the surgical safety requirements, the control device determines the candidate action sequence as the target action sequence and generates control commands based on the target action sequence to control the robotic arm and surgical instruments to perform corresponding actions.

[0226] When a candidate action sequence does not meet surgical safety requirements but meets preset correction conditions, the control device can correct the candidate action sequence based on the safety evaluation results, and use the world action model to re-predict and evaluate the safety of the corrected action sequence. When the candidate action sequence does not meet the preset correction conditions, or the corrected action sequence still does not meet surgical safety requirements, the control device regenerates candidate visual-action pairs.

[0227] During the execution of the target action sequence, the control device controls the surgical robot to execute a preset number of actions in the target action sequence in each round, and re-acquires the observation data of the surgical scene and the state of the surgical robot through the endoscopic imaging device and the state acquisition device to generate the next target action sequence. When the deviation between the actual surgical scene and the predicted surgical scene exceeds a preset range, or when the scene understanding confidence is lower than a preset threshold, the control device stops sending control commands corresponding to the actions that have not yet been executed, and generates new candidate vision-action pairs based on the re-acquired data.

[0228] In one specific embodiment, the surgical robot includes two robotic arms, each equipped with a needle holder, and the endoscopic imaging device is a stereoscopic endoscope. The control device generates candidate action sequences and future surgical visual states corresponding to the two needle holders based on the surgical images acquired by the stereoscopic endoscope, the states of the two robotic arms, and suturing task instructions. It then uses safety evaluation and rolling execution control to control the two needle holders to complete the incision suturing operation.

[0229] The control device can be integrated into the surgical robot or it can be a standalone device that communicates with the surgical robot. When the control device is integrated into the surgical robot, the surgical robot can be configured to execute the surgical robot control method based on the world motion model described in any of the above embodiments.

[0230] The surgical robot system can evaluate the possible changes in the surgical scene caused by the action based on the corresponding future surgical visual state before the surgical instruments actually perform the candidate action, and continuously update the subsequent actions based on the reacquired surgical scene and the state of the surgical robot during the action execution.

[0231] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0232] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 4 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0233] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0234] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0235] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0236] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0237] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0238] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0239] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0240] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0241] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0242] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0243] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0244] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0245] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0246] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A surgical robot control method based on a world motion model, characterized in that, The method includes: Acquire observation data of the current surgical scene, the status of the surgical robot, and task instructions; The observation data, the state of the surgical robot, and the task instructions are input into the world action model to generate candidate vision-action pairs, which include future surgical vision states and candidate action sequences. The candidate action sequence is safety evaluated based on the future surgical visual state; In response to the candidate action sequence meeting the surgical safety requirements, the candidate action sequence is determined as the target action sequence; The target action sequence is executed continuously, and the observation data of the surgical scene and the state of the surgical robot are reacquired to generate the next target action sequence.

2. The surgical robot control method based on a world motion model as described in claim 1, characterized in that, After performing a safety evaluation of the candidate action sequence based on the future surgical visual state, the method further includes: In response to the candidate action sequence failing to meet the surgical safety requirements but meeting the preset correction conditions, the candidate action sequence is corrected based on the safety evaluation results; The corrected action sequence is input as action condition into the world action model for re-prediction, and the safety assessment of the corrected action sequence is re-evaluated based on the re-prediction results. In response to the modified action sequence meeting the surgical safety requirements, the modified action sequence is determined as the target action sequence; In response to the candidate action sequence failing to meet the surgical safety requirements and the preset correction conditions, or the corrected action sequence still failing to meet the surgical safety requirements, a new candidate visual-action pair is generated.

3. The surgical robot control method based on a world motion model as described in claim 1, characterized in that, The world action model includes scene understanding experts, surgical status prediction experts, and motion control experts; The scene understanding expert is used to determine the semantic and anatomical information of the surgical scene based on the observation data and the task instructions; The surgical status prediction expert is used to predict the future surgical visual status; The motion control expert is used to generate the candidate motion sequence; The scene understanding expert, the surgical state prediction expert, and the motion control expert interact with each other through semantic-visual-motion joint attention; The world action model combines the future surgical visual state output by the surgical state prediction expert with the candidate action sequence output by the action control expert to form the candidate visual-action pair.

4. The surgical robot control method based on a world motion model as described in claim 3, characterized in that, The motion control expert generates the candidate motion sequence according to a first prediction frequency, and the surgical state prediction expert generates multiple future surgical visual states according to a second prediction frequency, wherein the first prediction frequency is higher than the second prediction frequency. Based on the ratio between the first prediction frequency and the second prediction frequency, the candidate action sequence is divided into multiple action groups, and each action group is associated with a future surgical visual state, wherein the future surgical visual state represents the predicted surgical scenario after the corresponding action group is executed.

5. The surgical robot control method based on a world motion model as described in claim 1, characterized in that, The world action model generates multiple candidate visual-action pairs; When there are multiple candidate action sequences that meet the surgical safety requirements, the smoothness of each candidate action sequence is determined based on the amount of change between adjacent actions in each candidate action sequence and the degree of change of the amount of change between adjacent actions. Based on the task completion rate, safety evaluation results, and action smoothness corresponding to each candidate action sequence, multiple candidate action sequences that meet the surgical safety requirements are sorted, and the target action sequence is determined based on the sorting results.

6. The surgical robot control method based on a world motion model as described in claim 1, characterized in that, The safety evaluation of the candidate action sequence based on the future surgical visual state includes: The tissue deformation, the contact state between surgical instruments and tissues, and the distance between surgical instruments and anatomical restricted areas are determined based on the future surgical visual state. The gaps between surgical instruments and the robot's kinematic risks are determined based on the candidate action sequence and the state of the surgical robot. The safety evaluation result is determined based on the tissue deformation, the contact state, the distance, the gap, and the robot kinematic risk, and the candidate action sequence is determined based on the safety evaluation result to determine whether it meets the surgical safety requirements.

7. The surgical robot control method based on a world motion model as described in claim 1, characterized in that, The scrolling execution of the target action sequence includes: In each round, a preset number of actions in the target action sequence are executed, and the observation data of the surgical scene and the state of the surgical robot are reacquired. The preset number is less than the number of actions contained in the target action sequence. Based on the reacquired observation data and the future surgical visual state corresponding to the performed actions, the deviation between the actual surgical scenario and the predicted surgical scenario is determined; In response to the deviation exceeding a preset range, or the scene understanding confidence determined based on the reacquired observation data being lower than a preset threshold, the execution of actions that have not yet been executed in the target action sequence is stopped and discarded, and candidate visual-action pairs are regenerated to determine the next target action sequence; In response to multiple consecutive failures to generate a target action sequence that meets the surgical safety requirements, a pause command is output.

8. The surgical robot control method based on a world motion model as described in claim 3, characterized in that, The world action model was trained in the following way: The first image coordinates of the surgical instruments are determined from the future surgical visual state output by the surgical state prediction expert, and the motion sequence output by the motion control expert is subjected to forward kinematics calculation and camera projection to obtain the second image coordinates corresponding to the first image coordinates. The visual-action consistency loss is determined based on the difference between the first image coordinates and the second image coordinates, and the world action model is trained by combining the visual prediction loss, action prediction loss, action smoothing loss and safety constraint loss.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

Images