A surgical robot strategy model training data generation method, device, medium and program product

Abstract

The application provides a surgical robot strategy model training data generation method, device, medium and program product. The method comprises the following steps: acquiring a candidate surgical video sequence generated by a pre-trained surgical world model, wherein the surgical world model is used to generate a candidate surgical video sequence containing continuous surgical scene images according to a surgical initial scene and a to-be-executed surgical action process; determining a robot action sequence corresponding to a visual state change in the candidate surgical video sequence according to the candidate surgical video sequence; and constructing a training data set of a surgical robot strategy model based at least on the candidate surgical video sequence and the corresponding robot action sequence. The application can expand the source of surgical robot strategy model training data, and improve the scene coverage and scalability of the training data.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and surgical robot technology, and in particular to a method, device, medium and program product for generating training data for a surgical robot strategy model. Background Technology

[0002] With the development of surgical robot autonomous operation, assisted decision-making, and intelligent control technologies, strategy models such as visual language action models, imitation learning models, and robot control strategies are increasingly being applied to surgical operations such as instrument gripping, needle delivery, puncture, traction, suturing, and knot tying. These strategy models typically require training using training data containing surgical visual observations and their corresponding robot movements. These robot movements can include robot joint states, instrument end-effector positions and postures, and gripper opening and closing states.

[0003] Currently, the collection of real surgical robot teaching data relies on specialized robotic equipment, professional operators, experimental task setup, and compliant collection procedures. This results in high data collection costs, long cycles, and difficulty in fully covering different instrument postures, occlusion conditions, tissue interaction methods, and surgical operation stages. While existing clinical teaching videos, publicly available surgical videos, and surgical robot endoscopic recordings contain surgical visual information such as instrument movement, tissue deformation, needle posture changes, and suture movement, they typically do not record motion information corresponding to the corresponding surgical procedures, such as robot joint states, instrument end-effector poses, and gripper states. Therefore, it is difficult to directly generate visual observation and robot action correspondence data that can be used for strategy model training.

[0004] Therefore, how to obtain training data that includes surgical visual observations and their corresponding robot actions to expand the sources of training data for surgical robot strategy models has become an urgent problem to be solved by existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a method, device, medium, and program product for generating training data for surgical robot strategy models, which at least solves the problems of difficulty in obtaining training data for existing surgical robot strategy models and the lack of corresponding robot motion information in surgical videos, making them difficult to use for strategy model training.

[0006] To achieve the above objectives and other advantages, some embodiments of this application provide the following aspects: In a first aspect, some embodiments of this application provide a method for generating training data for a surgical robot strategy model, including: Obtain a candidate surgical video sequence generated by a pre-trained surgical world model, wherein the surgical world model is used to generate a candidate surgical video sequence containing continuous surgical scene images based on the initial surgical scene and the surgical action process to be performed; Based on the candidate surgical video sequence, determine the robot action sequence corresponding to the visual state changes in the candidate surgical video sequence; A training dataset for the surgical robot strategy model is constructed based at least on the candidate surgical video sequences and their corresponding robot action sequences.

[0007] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the surgical robot strategy model training data generation method as described above.

[0008] Thirdly, some embodiments of this application also provide a computer-readable storage medium having a computer program and / or instructions stored thereon, which, when executed by a processor, implement the surgical robot strategy model training data generation method as described above.

[0009] Fourthly, some embodiments of this application also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the surgical robot strategy model training data generation method as described above.

[0010] Compared with existing technologies, the solution provided in this application utilizes a pre-trained surgical world model to generate a candidate surgical video sequence containing continuous surgical scene images based on the initial surgical scene and the surgical actions to be performed. Furthermore, it determines the corresponding robot action sequence based on visual state changes within the candidate surgical video sequence, forming a data generation chain from candidate surgical video generation and robot action determination to the construction of a strategy model training dataset. This allows the continuous surgical scene videos generated by the surgical world model to be transformed into training data containing both visual and robot action information, expanding the sources of training data for the surgical robot strategy model. It also enables the targeted generation of corresponding training samples based on different initial surgical scenes and surgical action processes, thereby reducing the dependence of training data on large-scale real-world teaching acquisition, improving the scene coverage and scalability of the training data, and providing a more sufficient data foundation for training the surgical robot strategy model. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other implementation methods can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is one of the flowcharts illustrating a method for generating training data for a surgical robot strategy model provided in this application embodiment; Figure 2 This is a second flowchart illustrating a method for generating training data for a surgical robot strategy model, as provided in an embodiment of this application. Figure 3 This is a schematic diagram illustrating a process of generating robot motion sequences from candidate surgical video sequences and performing verification and screening, as provided in an embodiment of this application. Figure 4 A schematic diagram illustrating the closed-loop data supplementation process for training a surgical robot strategy model based on weak scenario assessment, provided in an embodiment of this application. Figure 5 A schematic diagram illustrating the process of generating, filtering, and constructing training datasets for a surgical robot strategy model in a needle delivery task, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Some embodiments of this application relate to a method for generating training data for a surgical robot strategy model. This method can be applied to scenarios involving the construction of training data, data expansion, and model iteration for surgical robot strategy models. It can also be deployed as an offline training data generation service on a local training server, an internal computing center of a medical institution, a cloud computing platform, or a computing cluster composed of multiple computing nodes. The computing device executing this method can communicate with the target surgical robot platform and the data storage device. The target surgical robot platform can be a surgical robot system configured with one or more surgical instrument arms, an image acquisition device, and a motion control system, used to acquire surgical videos and robot kinematic data during real-world teaching processes. The data storage device can be used to store surgical videos, real-world teaching data, model parameters, candidate surgical video sequences, robot motion sequences, and the generated training dataset.

[0015] In one implementation, the training of the surgical world model, the generation of candidate surgical video sequences, the training of the inverse dynamics model, the generation of robot action sequences, the verification and screening, and the construction of the training dataset can be centrally deployed on the same training server or computing cluster. In another implementation, the above processes can be distributed across different computing nodes, with each node performing model training, data generation, verification and screening, and training dataset management. The training dataset constructed using this method can be used to train or update the surgical robot strategy model. The trained surgical robot strategy model can be deployed in the control computing device corresponding to the target surgical robot platform to generate robot actions based on surgical scene information, task information, and robot status. This method can be applied to training data generation scenarios corresponding to surgical operations such as needle clamping, needle delivery, tissue puncture, suture traction, suturing, or knot tying. The embodiments of this application can be used for offline data generation and training before the deployment of the surgical robot strategy model or during model iteration. For ease of description, the embodiments of this application use an endoscope as an example for illustrative purposes, but this application does not limit the specific type of image acquisition device; refer to... Figure 1 , Figure 2 As shown, this application may include the following steps: Step S1: Obtain candidate surgical video sequences generated by a pre-trained surgical world model, wherein the surgical world model is used to generate candidate surgical video sequences containing continuous surgical scene images based on the initial surgical scene and the surgical action process to be performed.

[0016] In step S1, the surgical world model is used to infer the changes in visual state after the initial surgical scene based on the given initial surgical scene and the surgical actions to be performed, thereby generating a candidate surgical video sequence containing images of continuous surgical scenes. The candidate surgical video sequence has not yet been associated with robot action information corresponding to its changes in visual state, and therefore is not directly used as training data for the surgical robot strategy model.

[0017] In some embodiments, the initial surgical scene can be represented by an initial surgical image, and the surgical procedure to be performed can be represented by task prompts describing the corresponding surgical procedure. Accordingly, the initial surgical image and the task prompts describing the surgical procedure to be generated can be acquired, and the initial surgical image and task prompts can be input into a pre-trained surgical world model, which then generates candidate surgical video sequences.

[0018] The initial surgical image is used to characterize the state of the surgical scene at the start of the surgical procedure to be generated. The initial surgical image can be an image frame output from an endoscope, laparoscope, surgical microscope, external surgical camera, or surgical simulation platform, or it can be an image frame selected from a pre-acquired surgical video or surgical scene dataset. The initial surgical image can include surgical instruments, needles, sutures, target tissues, and the initial spatial relationships between various surgical objects within the current field of view.

[0019] Task prompts describe the expected surgical procedure and can be constructed based on preset data to generate a target. Task prompts may include at least one of the following: surgical procedure, instruments involved in the procedure, spatial relationships between instruments or between instruments and target tissue, the anatomical structure being manipulated, and the interaction between instruments and tissue.

[0020] The surgical actions can include needle clamping, needle delivery, tissue puncture, suture pulling or knotting; the spatial relationships of instruments can include the relative positions and orientations of different surgical instruments, the clamping relationship between instruments and needles, and the position and orientation of the needle tip relative to the target tissue; the instrument-tissue interaction process can include sequential state changes such as the instrument approaching the target object, contacting the target object, clamping the target object, releasing the target object, moving the target object, or passing through the target tissue. Using the above information, the surgical object, action content, spatial relationships, and interaction state changes involved in the surgical actions to be performed can be defined, enabling the surgical world model to generate continuous surgical scene images according to the corresponding surgical action process.

[0021] For example, in a needle delivery data generation scenario, the initial surgical image can reflect the initial state where the first needle holder clamps the needle body, and the second needle holder is located on the other side of the needle body and facing it. The task prompt information can describe that the second needle holder approaches and clamps the needle body; after the second needle holder forms a clamp, the first needle holder releases the needle body, and the second needle holder drives the needle body to move. The pre-trained surgical world model can generate continuous image frames based on the initial surgical image and task prompt information, sequentially reflecting the second needle holder approaching the needle body, clamping the needle body, both needle holders clamping the needle body together, the first needle holder releasing the needle body, and the needle body moving with the second needle holder.

[0022] For example, in a tissue puncture data generation scenario, the initial surgical image may include a needle holder holding the needle and the target tissue to be punctured, with the needle tip pointing towards the target tissue. The task prompt information can describe that the needle holder moves the needle towards the target tissue, and the needle tip sequentially approaches, contacts, and crosses the boundary of the target tissue. Based on this, a pre-trained surgical world model can generate candidate surgical video sequences that reflect changes in the position and posture of the needle, changes in the spatial relationship between the needle and the target tissue, and corresponding changes in the state of the target tissue.

[0023] Specifically, the pre-trained surgical world model can extract the initial visual state represented by the initial surgical image and semantically encode the task prompts to form image generation conditions and text generation conditions. Under the joint constraints of the image generation conditions and text generation conditions, the surgical world model generates future video latent variables corresponding to the surgical action process in the video latent variable space, and then decodes the future video latent variables into consecutive image frames arranged in chronological order to obtain a candidate surgical video sequence. Among them, the consecutive image frames can reflect at least one of the following changes in position, posture, spatial relationship, or interaction state of surgical instruments, manipulated objects, and target tissues during the surgical action to be performed.

[0024] In some embodiments, for the same initial surgical image and task prompts, different initial noise levels can be used to perform video generation, resulting in multiple candidate surgical video sequences. Furthermore, by adjusting the action stage, initial relative position of instruments, target tissue region, or video generation length in the task prompts, candidate surgical video sequences corresponding to different data generation conditions can be generated. Thus, corresponding candidate surgical video sequences can be generated for different initial surgical scenarios or surgical actions to be performed, thereby expanding the surgical scenarios and types of surgical actions covered by subsequent training data.

[0025] It should be noted that the task prompts in this step are used in the reasoning process of the surgical world model, while the training prompts used when training the surgical world model correspond to real surgical action video clips, enabling the model to learn the correspondence between the semantics of surgical actions and changes in real future surgical scenarios. Both task prompts and training prompts can be described using natural language, but they are used in the reasoning and training phases of the surgical world model, respectively.

[0026] In a preferred embodiment, the pre-trained surgical world model is trained as follows: Step A1: Obtain the surgical video, divide the surgical video into multiple surgical action video segments, and determine the training prompts that describe the corresponding surgical action process for each surgical action video segment.

[0027] In step A1, surgical videos for training the surgical world model can be acquired. These videos may include compliantly acquired clinical surgical videos, surgical teaching videos, animal experimental surgical videos, simulated surgical videos, or surgical robot experimental videos. The acquired surgical videos can undergo de-identification, frame rate standardization, resolution standardization, image quality screening, and temporal continuity checks to reduce the impact of differences in video sources, imaging parameters, and image quality on model training.

[0028] After preprocessing, the surgical video can be segmented into multiple surgical action video segments based on the surgical actions occurring within it. Each surgical action video segment corresponds to a temporally continuous surgical action process, such as needle clamping, needle delivery, tissue puncture, suture traction, or knotting. The segmented surgical action video segments can retain the continuous state changes of surgical instruments, needles, sutures, and target tissues during the execution of the corresponding surgical action.

[0029] The start and end positions of surgical action video clips can be determined based on pre-marked start and end times, surgical action stage labels, or changes in the interactive state between surgical subjects. For example, for the action of transferring the needle from the first needle holder to the second needle holder, the start time of the corresponding surgical action video clip can be determined by the moment when the second needle holder begins to approach the needle body, and the end time of the corresponding surgical action video clip can be determined by the moment when the first needle holder releases the needle body and the needle body moves with the second needle holder. For the action of the needle body puncturing the target tissue, the start time of the corresponding surgical action video clip can be determined by the moment when the needle tip begins to move towards the target tissue, and the end time of the corresponding surgical action video clip can be determined by the moment when the needle tip crosses the boundary of the target tissue or completes the preset puncture stage.

[0030] For each surgical action video segment, the surgical action and its execution attributes, the instruments and objects involved in the surgical action, the operated anatomical structures, the initial spatial relationships between the surgical objects, and the sequential changes in interactive states during the action execution can be determined based on the image content and the changes in the states of each surgical object over time. The instrument-tissue interaction process can be divided into multiple action stages arranged in chronological order, and the interactive objects and interactive states in each action stage can be recorded separately. Furthermore, according to the order of occurrence of the surgical actions, the values ​​of each semantic element are organized into fine-grained natural language descriptions to obtain corresponding training prompts. The surgical objects, spatial relationships, action stages, and their order described in the training prompts should be consistent with the actual action process presented in the corresponding surgical action video segment, and the time range covered by the training prompts should correspond to the time range covered by the subsequently determined sequence of real future surgical scenarios.

[0031] For example, for a surgical video clip of a needle being passed from the first needle holder to the second needle holder, the corresponding training prompts can be described as follows: The first needle holder holds the needle body and passes it to the second needle holder located on its other side; after the second needle holder approaches and holds the needle body, the first needle holder releases the needle body, and the needle body moves with the second needle holder.

[0032] For example, for a video clip of a surgical procedure involving needle puncturing the target tissue, the corresponding training prompts can be described as follows: the needle holder clamps the needle body, the needle tip faces the target tissue area and moves towards the target tissue; the needle tip approaches, contacts and crosses the boundary of the target tissue in sequence, and the target tissue undergoes corresponding morphological changes during the contact and penetration of the needle tip.

[0033] Step A2: Construct a second training sample based on each surgical action video clip. Each second training sample includes the first frame of the corresponding surgical action video clip as the initial training image, the continuous images after the first frame of the real future surgical scene sequence, and the corresponding training prompts.

[0034] In step A2, for each surgical action video segment obtained in step A1, the first frame of the surgical action video segment can be selected as the initial training image, and the consecutive images arranged in chronological order after the first frame can be selected as the real future surgical scene sequence. At the same time, the training prompts corresponding to the surgical action video segment are associated with the initial training image and the real future surgical scene sequence to construct the corresponding second training sample.

[0035] Among them, the initial training image is used to represent the current surgical state at the start of the corresponding surgical action video segment, the real future surgical scene sequence is used to represent the continuous changes of surgical instruments, operating objects and target tissues as the corresponding surgical action progresses from the current surgical state, and the training prompts are used to describe the surgical action process corresponding to the real future surgical scene sequence.

[0036] For example, for a video clip of a needle delivery surgery, the initial training image can reflect the initial state where the first needle holder holds the needle and the second needle holder is located on the other side of the needle; the sequence of real future surgical scenes can sequentially reflect the process of the second needle holder approaching the needle, the second needle holder holding the needle, the two needle holders holding the needle together, the first needle holder releasing the needle, and the needle moving with the second needle holder; the training prompts can describe the corresponding needle delivery direction, the needle holders and needles involved in the needle delivery, the spatial relationship between the needle holders, and the needle handover process.

[0037] For video clips of tissue puncture surgery, the initial training image can reflect the initial state of the needle holder holding the needle body with the needle tip pointing towards the target tissue; the real future surgical scene sequence can sequentially reflect the process of the needle tip approaching the target tissue, contacting the target tissue, crossing the boundary of the target tissue, and the target tissue undergoing corresponding morphological changes in the contact area; the training prompts can describe the direction of movement of the needle body, the target tissue to be punctured, and the interaction process between the needle body and the target tissue.

[0038] Therefore, the initial training image, training cues, and sequences of real future surgical scenarios together constitute a second training sample, establishing a temporal correspondence between the current surgical state, surgical action semantics, and future surgical state. By performing the above processing on surgical action video clips corresponding to different surgical actions, instrument spatial relationships, target tissue regions, and instrument-tissue interaction processes, a second training sample set for training the surgical world model can be obtained, wherein the initial training image, training cues, and sequences of real future surgical scenarios in each second training sample are mutually aligned.

[0039] Step A3: Configure a low-rank adaptation module in the pre-trained basic video world model, use the initial training image and training cues as generation conditions, construct the flow matching training target based on the video latent variables corresponding to the real future surgical scene sequence, and train the parameters of the low-rank adaptation module based on the flow matching loss to obtain the pre-trained surgical world model.

[0040] In step A3, a pre-trained basic video world model can be obtained, and a low-rank adaptation module can be configured in the Transformer structure of the basic video world model. The surgical domain adaptation training can be performed using the second training samples constructed in step A2.

[0041] The low-rank adaptation module can be a LoRA module, including a first low-rank parameter matrix and a second low-rank parameter matrix used to form the weight increment of the target linear transformation layer. During training, the original weight parameters of the pre-trained base video world model can be kept unchanged or at least partially fixed, and the two low-rank parameter matrices can be updated according to the flow matching loss. This allows the model to further learn the evolution of surgical scenes from the endoscopic perspective, such as instrument movement, needle and suture state changes, tissue deformation, occlusion changes, and instrument-tissue interaction, while retaining its existing video generation capabilities.

[0042] For each second training sample, the initial training image and training cues can be used together as generation conditions. The initial training image represents the current surgical state at the start of the surgical action video segment, while the training cues describe the surgical actions occurring within the corresponding time frame, the surgical objects involved, and the spatial relationships and interactions between them. The video latent variables obtained from the sequence of real future surgical scenes after latent variable transformation are used to construct the stream matching training objective.

[0043] During training, the pre-trained basic video world model can learn the evolution path of latent variables in the latent variable space of real future videos under the constraints of generation conditions, and the flow matching loss is determined based on the difference between the model's prediction results and the flow matching training objective. By minimizing the flow matching loss, the parameters of the low-rank adaptation module are iteratively updated to obtain a pre-trained surgical world model adapted to the surgical scenario.

[0044] In the solution provided in this embodiment, through steps A1 to A3, the first frame, subsequent continuous images, and corresponding training prompts of the surgical action video clip are organized into mutually aligned second training samples, thereby establishing the correspondence between the initial surgical state, the semantics of surgical actions, and the future evolution of the surgical scene. Furthermore, by configuring a low-rank adaptation module in the basic video world model and performing flow matching training, the model learns the evolutionary laws of the surgical scene, such as the movement of surgical instruments, the interaction between instruments and tissues, and tissue changes. Thus, while largely retaining the original video generation capabilities of the basic video world model, surgical scene adaptation can be completed without retraining all model parameters, reducing the number of model parameters that need to be updated and the consumption of training resources. It also improves the matching degree between the generated candidate surgical video sequences and the initial surgical state and task prompt information, enhancing the temporal continuity of the surgical action process and the rationality of scene evolution in the candidate surgical video sequences.

[0045] In a preferred embodiment, the surgical world model includes a spatiotemporal encoder, a Transformer latent dynamics module, and a video decoder. The low-rank adaptation module is configured within the attention layer and feedforward network of the Transformer latent dynamics module. Training the parameters of the low-rank adaptation module based on flow matching loss may specifically include the following steps: Step A301: Convert the sequence of real future surgical scenes into real future video latent variables using a spatiotemporal encoder.

[0046] In step A301, the sequence of real future surgical scenarios from the second training sample can be input into the spatiotemporal encoder. The sequence of real future surgical scenarios can be denoted as: in, Including the first frame of the corresponding surgical action video clip, and the first to the next frames arranged in chronological order. Frame image, Represents the first in a sequence of real future surgical scenarios Frame image, This indicates the number of image frames contained in the sequence of real future surgical scenes.

[0047] The spatiotemporal encoder can extract spatial appearance information and temporal change information from a sequence of real future surgical scenes, and encode continuous images in pixel space into a video latent variable space to obtain the real future video latent variables. The encoding process can be represented as: in, Indicates a space-time encoder. The latent variables of the real future video corresponding to the sequence of real future surgical scenarios.

[0048] Real Future Video Latent Variables Used to characterize sequences of real future surgical scenarios in the video latent variable space. The continuous scene evolution process includes changes in the position and posture of surgical instruments, movement of needles and sutures, changes in the interaction state between instruments or between instruments and tissues, changes in the morphology of target tissues, and changes in the occlusion relationship between different surgical objects.

[0049] For example, in a sequence of real-future surgical scenarios corresponding to needle delivery, the real-future video latent variables can characterize the continuous changes in the needle body as the receiving-side needle holder approaches and clamps the needle, the original clamping-side needle holder releases the needle, and the needle moves with the receiving-side needle holder. In a sequence of real-future surgical scenarios corresponding to tissue puncture, the real-future video latent variables can characterize the continuous process as the needle tip approaches the target tissue, contacts the target tissue, crosses the target tissue boundary, and the target tissue undergoes corresponding morphological changes.

[0050] By converting real future surgical scene sequences into real future video latent variables, continuous images in high-dimensional pixel space can be compressed and represented in video latent variable space. This allows the subsequent Transformer latent dynamics module to learn the continuous evolution of future surgical scenes in video latent variable space without having to perform flow matching training directly in high-dimensional pixel space.

[0051] Step A302: Based on the real future video latent variables, sampling noise, and training time steps, determine the intermediate latent variables and the stream matching target speed corresponding to the training time step.

[0052] In step A302, latent variables of the true future video can be sampled from the standard normal distribution. Matching sampling noise The training time steps are obtained by sampling from the Logit normal distribution. ,in, .

[0053] Based on real future video latent variables Sampling noise and training time steps The training time step can be determined according to the following formula. Corresponding intermediate latent variables : Among them, intermediate latent variables It is based on training time steps Latent variables of real future video and sampling noise The latent variables are obtained by linear interpolation. (When training time step...) When close to 0, intermediate latent variables More realistic future video latent variables When training time steps When close to 1, intermediate latent variables Closer to sampling noise .

[0054] Furthermore, the training time step can be determined according to the following calculation formula. Corresponding flow matching target speed : Stream matching target speed Used to characterize latent variables from real future videos and sampling noise The interpolation path constitutes the direction of target change relative to the training time step. Due to intermediate latent variables... The target speed of the corresponding interpolation path can be determined by the sampling noise, based on the linear interpolation relationship described above. Latent variables in real future videos The difference between them is determined.

[0055] During training in different training rounds or for different second training samples, different sampling noise and training time steps can be sampled to obtain intermediate latent variables with different noise levels, enabling the Transformer latent dynamics module to learn the conditional velocity corresponding to different positions in the latent variable evolution path.

[0056] Step A303: Using the Transformer latent dynamics module, predict the conditional velocity corresponding to the intermediate latent variable based on the intermediate latent variable, the training time step, and the generation conditions consisting of the initial training image and training cues.

[0057] In step A303, the initial training image can be... and training tips Composition generation conditions ,Right now: Among them, the initial training image This is the first frame of the corresponding surgical action video clip; training hints It is used to describe surgical actions, surgical instruments and objects, spatial relationships and instrument-tissue interaction processes that occur within the time range covered by a sequence of real future surgical scenarios, starting from the time corresponding to the initial training image.

[0058] The intermediate latent variables obtained in step A302 can be used as... Training time steps and generation conditions Input the Transformer latent dynamics module to predict intermediate latent variables. The corresponding conditional velocity, which can be expressed as: in, This represents the conditional velocity field function implemented by the Transformer latent dynamics module. This represents the low-rank adaptation parameters configured in the Transformer latent dynamics module.

[0059] Specifically, the Transformer latent dynamics module can be based on intermediate latent variables. Extract the latent variable features of the video at the current training time step and combine them with the initial training images. The current surgical status and training prompts are represented. The semantic representation of surgical actions is used to predict the direction of change of intermediate latent variables under the corresponding generation conditions.

[0060] In some implementations, the Transformer latent dynamics module may include an attention layer and a feedforward network connected in sequence. Intermediate latent variables can be transformed into multiple latent variable feature units, and the initial training image and training cues can be transformed into corresponding image-conditional features and text-conditional features, respectively. The attention layer is used to establish spatiotemporal relationships between different latent variable feature units and to introduce image-conditional features and text-conditional features into the feature processing of intermediate latent variables; the feedforward network is used to perform feature transformations on each latent variable feature unit after attention processing.

[0061] In one implementation, the attention layer may include a linear transformation layer for generating query features, key features, and value features. A low-rank adaptation module may be configured in at least one of these linear transformation layers. For a linear transformation layer configured with a low-rank adaptation module, the input features can be subjected to a basic linear transformation using the original weight parameters of the linear transformation layer, and then an incremental linear transformation can be performed using the low-rank adaptation module. The results of the basic linear transformation and the incremental linear transformation are then superimposed to obtain the corresponding query features, key features, or value features.

[0062] For example, the output of a linear transformation layer used to generate query features can be represented as: in, For the latent variable features of the input attention layer, These are the original weight parameters used to generate query features in the pre-trained base video world model. The weight increments are formed for the low-rank adaptation module. These weight increments can be determined by the product of two low-rank parameter matrices. The linear transformation layers used to generate key and value features can also employ corresponding incremental transformation methods.

[0063] When performing spatiotemporal correlation processing within latent variables, query features, key features, and value features can all be generated from the latent variable features corresponding to intermediate latent variables. This allows the attention layer to aggregate the corresponding latent variable features based on the correlation between latent variable features at different time and spatial locations. Consequently, correlations can be established between different time and spatial locations in continuous images, enabling the Transformer latent dynamics module to combine information from multiple spatiotemporal locations to process continuous evolutionary features such as instrument movement, changes in needle and suture states, tissue deformation, and changes in occlusion relationships.

[0064] When introducing generation conditions into the latent variable processing, query features can be generated from the latent variable features corresponding to intermediate latent variables, and key and value features can be generated from the image condition features corresponding to the initial training image and the text condition features corresponding to the training prompts. The attention layer can fuse the corresponding image condition information and text condition information into the intermediate latent variables based on the correlation between the latent variable features and each condition feature. Thus, the instrument position, instrument posture, needle state, tissue state, and occlusion relationship reflected in the initial training image, as well as the surgical actions, instrument spatial relationships, and instrument-tissue interaction processes described in the training prompts, can all participate in the prediction of conditional velocity.

[0065] The low-rank adaptation module configured in the attention layer can incrementally adjust at least one of the aforementioned query features, key features, and value features based on the second training sample. For example, when the training prompt describes the first needle holder passing the needle to the second needle holder, the low-rank adaptation module can strengthen the association between the latent variable features and the spatial relationship between the two needle holders and the needle body and their changes over time during the attention processing; when the training prompt describes the needle body puncturing the target tissue, the low-rank adaptation module can further associate the latent variable features with the needle tip position, needle tip movement direction, target tissue area, and tissue response. The above association adjustment refers to changing the attention association results between the corresponding features through training, and does not mean that the low-rank adaptation module performs surgical object recognition or action determination alone.

[0066] After processing by the attention layer, the resulting attention output features can be input into the feedforward network. The feedforward network may include at least two linear transformation layers and nonlinear processing units located between the linear transformation layers, used to perform feature dimension transformation and nonlinear feature recombination on each latent variable feature unit. A low-rank adaptation module may be configured in at least one linear transformation layer in the feedforward network and incrementally adjust the original feature transformation results of that linear transformation layer.

[0067] For example, feedforward networks for input features The processing can be expressed as: in, The latent variable features output by the attention layer, The latent variable features output by the feedforward network, and These are the original weight parameters of the corresponding linear transformation layer in the feedforward network. and To correspond to the weight increments formed by the low-rank adaptation module, This indicates nonlinear processing. In practical applications, a low-rank adaptation module can be configured in only one of the linear transformation layers, or it can be configured in multiple linear transformation layers respectively.

[0068] The low-rank adaptation module configured in the feedforward network is mainly used to adjust the feature combination method within each latent variable feature unit. The attention layer first collects spatiotemporal information and generation condition information related to the corresponding latent variable features. The feedforward network then performs a nonlinear transformation on the collected information, enabling the latent variable features to further characterize the local morphology and dynamic changes in the endoscopic surgical scene. For example, it can enhance the expression of features related to instrument edges and postures, needle bending direction, suture local morphology, tissue deformation areas, and foreground and background occlusion relationships, while suppressing features with low correlation to the corresponding surgical action conditions.

[0069] The attention layer and the feedforward network can process latent variable features sequentially through residual connections. The low-rank adaptation module in the attention layer focuses on adjusting the relationships between different latent variable features and between latent variable features and generation conditions; the low-rank adaptation module in the feedforward network focuses on adjusting the transformation of latent variable features after information aggregation. Together, these two modules enable the Transformer latent dynamics module to learn the conditional relationships and latent variable evolution characteristics specific to the surgical scene, while largely preserving the original parameters and video generation capabilities of the pre-trained basic video world model, thereby predicting conditional velocities that match the current surgical state and training cues.

[0070] Step A304: Determine the flow matching loss based on the predicted conditional velocity and the flow matching target velocity, and iteratively update the parameters of the low-rank adaptation module by minimizing the flow matching loss.

[0071] In step A304, the flow matching loss can be determined based on the difference between the conditional velocity predicted by the Transformer latent dynamics module and the flow matching target velocity determined in step A302.

[0072] In some implementations, the flow matching loss can be determined according to the following formula: in, For flow matching loss; Conditional velocities predicted for the Transformer potential dynamics module; Match the target velocity to the flow; Represents latent variables for the real future video Sampling noise Generation conditions and training time steps Seeking expectations; This represents the square norm 2.

[0073] Specifically, conditional velocity can be calculated. Matching the target velocity with the flow The mean squared error between the two training samples is calculated, and the flow matching loss for the corresponding second training sample or training batch is determined based on the mean squared error. Subsequently, backpropagation can be performed based on the flow matching loss to determine the gradient of the low-rank adaptation parameters with respect to the flow matching loss, and the low-rank adaptation parameters configured in the attention layer and feedforward network of the Transformer latent dynamics module are updated according to the gradient.

[0074] During training, the original model parameters of the pre-trained base video world model can be kept unchanged or at least partially fixed, with the parameters in the low-rank adaptation module being updated iteratively. By repeatedly executing steps A301 to A304 using multiple second training samples and minimizing the flow matching loss, the conditional velocity predicted by the Transformer latent dynamics module gradually approximates the corresponding flow matching target velocity, thereby learning the conditional velocity field that continuously evolves with the progress of the surgical action in the real future video latent variables under the joint constraints of the initial training image and training cues.

[0075] After training, a pre-trained surgical world model adapted to the surgical scenario is obtained. Given an initial surgical image and task prompts, the pre-trained surgical world model can generate corresponding future video latent variables through the Transformer latent dynamics module, and decode the generated future video latent variables into consecutive image frames arranged in chronological order through a video decoder, thereby forming a candidate surgical video sequence.

[0076] In the solution provided in this embodiment, through steps A301 to A304, the sequence of real future surgical scenes is encoded into the video latent variable space. Intermediate latent variables and the target velocity for flow matching are constructed based on sampling noise and training time steps. This allows the Transformer latent dynamics module to learn the conditional velocity field of the future surgical scene under the constraints of the initial training image and training cues. The low-rank adaptation module is trained by the difference between the predicted velocity and the target velocity for flow matching, enabling the model to learn the continuous evolutionary laws of surgical instrument movement, instrument-tissue interaction, and tissue changes. Therefore, the training overhead of directly processing high-dimensional video data is reduced. While retaining the original generation capabilities of the basic video world model, the matching degree between the generated video and the initial surgical state and surgical action semantics is improved, and the temporal continuity and rationality of changes in the surgical scene are enhanced.

[0077] Step S2: Based on the candidate surgical video sequence, determine the robot action sequence corresponding to the visual state changes in the candidate surgical video sequence.

[0078] In step S2, based on multiple frames of surgical scene images arranged chronologically in the candidate surgical video sequence, the visual state changes of surgical instruments, manipulated objects, or target tissue between adjacent image frames can be determined, and a robot action sequence corresponding to the visual state changes in time can be obtained. Visual state changes may include at least one of the following: changes in the position, posture, spatial relationship, and interaction state of the surgical instruments or manipulated objects.

[0079] The robot action sequence may include multiple robot actions arranged in chronological order. Each robot action may correspond to a change in visual state between a set of adjacent image frames in a candidate surgical video sequence, or to a change in visual state within a preset video window. The robot actions may be represented in joint space or Cartesian space, and may include at least one of joint angles, joint displacements, instrument end-effector positions, instrument end-effector postures, and gripper opening / closing amounts.

[0080] Determining robot motion sequences based on candidate surgical video sequences can be achieved through various implementation methods.

[0081] In one implementation, candidate surgical video sequences are input into an inverse dynamics model to obtain robot motion sequences corresponding to changes in visual state in the candidate surgical video sequences. The inverse dynamics model is obtained by training a basic inverse dynamics model based on real teaching data of the target surgical robot platform, and is used to establish a mapping relationship between changes in visual state in the surgical scene and the motion space of the target surgical robot platform.

[0082] In a preferred embodiment, training the basic inverse dynamics model based on real teaching data from the target surgical robot platform specifically includes: Step B1: Synchronously resample the video sequence and robot kinematic data sequence in the real teaching data to obtain a time-aligned video frame sequence and a resampled robot kinematic data sequence.

[0083] In step B1, real teaching data of the target surgical robot platform can be acquired. Real teaching data includes video sequences and robot kinematic data sequences acquired during the execution of teaching actions on the target surgical robot platform. The video sequences can be continuous images acquired by an endoscope, used to record the visual state changes of surgical instruments, needles, sutures, and target tissue as the teaching actions occur. The robot kinematic data sequences can include at least one of the joint states of the left and right instruments, the end-effector pose, and the gripper opening and closing states, used to record the robot's motion state during the execution of the corresponding teaching actions.

[0084] Since video sequences and robot kinematic data sequences can be obtained from different data acquisition channels, their sampling frequencies, sampling times, and data start and end times may differ. Therefore, the effective time range covered by both can be determined based on the timestamps carried by the video frames and robot kinematic data, and the video sequences and robot kinematic data sequences can be resampled to a unified sampling frequency.

[0085] For each synchronous sampling moment at a uniform sampling frequency, the corresponding video frames can be determined from the video sequence, and the corresponding robot kinematics data can be determined from the robot kinematics data sequence. When the synchronous sampling moment does not completely coincide with the original sampling moment, the robot kinematics data for the corresponding synchronous sampling moment can be determined based on temporally adjacent data. This results in a time-aligned video frame sequence and a resampled robot kinematics data sequence, ensuring that the video frames and robot kinematics data corresponding to the same synchronous sampling moment reflect the visual and motion states of the target surgical robot platform at that moment, respectively. The uniform sampling frequency can be determined based on the training frame rate of the basic inverse dynamics model, thus matching the timescale of the real teaching data with the motion prediction timescale of the inverse dynamics model.

[0086] Step B2: Process the resampled robot kinematic data sequence according to the action label format corresponding to the target surgical robot platform and the action normalization parameters corresponding to the real teaching data to obtain the action sequence.

[0087] In a preferred embodiment, step B2 specifically includes: Step B201: Based on the motion space of the target surgical robot platform, determine the motion label format and the motion dimension of the target platform. The motion label format can be either joint space representation or Cartesian space representation.

[0088] In step B201, the motion label format used to characterize the motion of the target surgical robot platform can be determined based on the mechanical configuration, motion control interface, and motion output format of the basic inverse dynamics model of the target surgical robot platform, and the motion dimension of the target platform can be determined based on the motion label format.

[0089] Action labels can be in joint space or Cartesian space. When using joint space representation, the action of each instrument can include the joint angles or displacements of each joint, as well as the gripper opening / closing amount, so that the corresponding action directly reflects the joint motion state of the target surgical robot platform. When using Cartesian space representation, the action of each instrument can include the instrument end effector position, end effector orientation, and gripper opening / closing amount, so that the corresponding action reflects the motion state of the instrument end effector in a preset reference coordinate system. The end effector orientation can be represented using quaternions or six-dimensional rotation.

[0090] The target platform motion dimension is determined based on the number of instruments included in the target surgical robot platform and the number of motion components contained in each instrument under the selected motion label format. In a dual-arm surgical robot platform, the target platform motion dimension includes the motion dimension of the left-side instrument and the motion dimension of the right-side instrument.

[0091] For example, in one implementation, the end-effector pose of the dual-arm instruments in the endoscopic coordinate system is used as the motion label. The motion of each instrument includes three-dimensional end-effector position, six-dimensional rotation representation, and gripper opening / closing amount. Therefore, each instrument corresponds to ten motion dimensions, and the dual-arm motion includes a total of twenty motion dimensions. The above motion dimensions are for illustrative purposes only and can be determined based on the structural configuration and motion control interface of the target surgical robot platform.

[0092] Step B202: Extract the kinematic data of the left and right devices corresponding to each synchronous sampling time from the resampled robot kinematic data sequence, and represent the kinematic data of the left and right devices as left device actions and right device actions respectively according to the action label format.

[0093] In step B202, the kinematic data of the left and right devices corresponding to each synchronous sampling moment can be extracted from the resampled robot kinematic data sequence according to the action label format determined in step B201.

[0094] When the motion label is represented in joint space, the joint angles or joint displacements of the left and right instruments can be extracted from the robot kinematic data at the corresponding synchronous sampling time. These are then combined with the gripper opening and closing amounts of the respective instruments to obtain the motions of the left and right instruments. The order of the motion components can be consistent with the joint numbering order of the target surgical robot platform or the motion input order specified by the control interface.

[0095] When the action labels are represented in Cartesian space, the end-effector positions, end-effector postures, and gripper opening / closing amounts of the left and right devices can be extracted from the robot kinematic data at the corresponding synchronous sampling moments, thus obtaining the actions of the left and right devices. The end-effector positions and end-effector postures can be uniformly represented in a preset reference coordinate system.

[0096] In one implementation, the preset reference coordinate system can be the endoscope coordinate system. For the instrument end-effector pose represented by the robot base coordinate system or other reference coordinate system in the original robot kinematics data, the corresponding instrument end-effector pose can be transformed to the endoscope coordinate system based on the calibration relationship of the target surgical robot platform, so that the instrument movement reflected by the action label and the instrument movement observed in the video frame adopt the corresponding spatial reference.

[0097] Therefore, at each synchronous sampling moment, the left and right instrument movements corresponding to the video frame at the same moment can be obtained.

[0098] Step B203: Combine the left and right instrument movements corresponding to the same synchronous sampling time into a two-arm movement.

[0099] In step B203, the left and right instrument movements corresponding to the same synchronous sampling time can be combined according to the pre-set sequence of motion components to obtain the double arm movements corresponding to the synchronous sampling time.

[0100] No. The arm movements corresponding to each synchronous sampling moment can be represented as: in, For the first The arm movement vectors corresponding to each synchronous sampling moment and These are movements using the left-side apparatus and movements using the right-side apparatus, respectively. For the target platform's action dimension.

[0101] When combining two-arm movements, the order of the left and right apparatus movements in the two-arm movement vectors can be fixed, as can the order of the movement components within each apparatus on each side. For example, the left and right apparatus movements can be arranged sequentially; for apparatus movements represented using Cartesian space, the apparatus end position, apparatus end posture, and gripper opening / closing amount can be arranged sequentially within each apparatus movement.

[0102] The order of the same action components can be maintained throughout the processes of real teaching data processing, inverse dynamics model training, candidate surgical video action prediction, and action executability verification, in order to avoid misalignment of action label meanings due to inconsistent action dimension order.

[0103] Step B204: Determine the motion normalization parameters based on the statistical range of each motion dimension in the real teaching data, and normalize each double-arm motion based on the motion normalization parameters.

[0104] In step B204, the actual teaching data of the target surgical robot platform can be statistically analyzed to determine the statistical range of each motion dimension during the actual teaching process, and the corresponding motion normalization parameters can be determined based on the statistical range of each motion dimension. Specifically, the bi-arm movements corresponding to each synchronous sampling moment in the actual teaching data can be summarized according to the motion component arrangement order determined in step B203, and the value range of each motion dimension can be statistically analyzed. Corresponding motion normalization parameters are determined for different motion dimensions to ensure that motion components with different physical meanings or numerical scales, such as joint angles, joint displacements, instrument end-effector positions, instrument end-effector postures, and gripper opening / closing, can be converted to a unified numerical range used by the basic inverse dynamics model.

[0105] Subsequently, based on the motion normalization parameters corresponding to each motion dimension, the arm movements corresponding to each synchronous sampling moment can be processed dimension by dimension to obtain the normalized arm movements. The normalization process only changes the numerical scale of each motion component, without changing the motion dimensions, the meaning of the motion components, or the order of arrangement of the arm movements.

[0106] For motion dimensions with the same physical meaning but different statistical ranges in the left and right apparatuses, motion normalization parameters can be determined separately based on the actual statistical range of the corresponding motion dimension in real teaching data. Similarly, for different types of motion components such as apparatus end-effector position, apparatus end-effector posture, and gripper opening / closing amount, corresponding motion normalization parameters can also be determined separately.

[0107] The motion normalization parameters determined based on real teaching data from the target surgical robot platform can remain consistent during inverse dynamics model training and subsequent candidate surgical video motion prediction. When performing motion executability verification, the normalized motion output by the inverse dynamics model can be denormalized back to the actual motion space of the target surgical robot platform using the corresponding motion normalization parameters.

[0108] Step B205: Arrange the normalized arm movements according to the order of each synchronous sampling moment to obtain the movement sequence.

[0109] In step B205, the normalized dual-arm movements obtained in step B204 can be arranged according to the order of the synchronous sampling times to obtain a movement sequence. Each dual-arm movement in the movement sequence has a movement label format, target platform movement dimension, and movement component arrangement order that matches the target surgical robot platform, and a time correspondence is established between the synchronous sampling times and the video frame sequence obtained in step B1.

[0110] When constructing subsequent first training samples, the continuous actions corresponding to the visual state changes of each adjacent image frame between the starting and ending image frames can be extracted from the action sequence based on the synchronization relationship between the video frames and the arm movements. For example, the... The arm movements corresponding to the first synchronous sampling moment can be used as the first... Action labels corresponding to visual state changes between video frames and the next video frame, and subsequent arm movements corresponding to visual state changes between adjacent video frames in chronological order.

[0111] Thus, the resampled robot kinematics data is converted into action sequences that are represented by unified action labels, have fixed action dimensions, are normalized, and correspond to the video frame sequence in time, providing action supervision data for the first training samples for extracting continuous action sequences and constructing inverse dynamics models.

[0112] In the solution provided in this embodiment, through steps B201 to B205, a unified action label format and action dimension are determined based on the action space of the target surgical robot platform. The kinematic data of the left and right instruments at each synchronous sampling moment are converted and combined into bi-arm movements. Then, normalization and temporal arrangement are performed based on the statistical range of each action dimension in the real teaching data. This allows robot kinematic data with different physical meanings and dimensions to be converted into action sequences with clear action meanings, fixed dimensions, unified numerical scales, and temporal correspondence with the video frame sequence. This avoids label bias caused by inconsistent action component order or numerical scales, providing standardized supervisory data adapted to the target surgical robot platform for the inverse dynamics model, improving the stability of model training and the consistency of action prediction results.

[0113] Step B3: Select a start image frame and an end image frame with a preset number of frames from the video frame sequence, and extract the continuous action sequences corresponding to the visual state changes between each adjacent image frame from the start image frame to the end image frame from the action sequence to construct the first training sample.

[0114] In step B3, a preset number of frames can be determined based on the action prediction window length of the basic inverse dynamics model, and a start and end image frame with a preset number of frames intervals can be selected from the time-aligned video frame sequence. Specifically, the first frame in the video frame sequence can be selected from the second frame in the second frame sequence. The image frame corresponding to each synchronous sampling time is used as the starting image frame. and the first The image frame corresponding to each synchronous sampling moment is used as the end image frame. .in, The number of frame intervals included in the motion prediction window, that is, the preset number of frames between the start image frame and the end image frame.

[0115] From the start image frame Until the end of the image frame This includes, in sequence, the changes in visual state between the starting image frame and the next image frame, as well as the changes in visual state between subsequent groups of adjacent image frames. These changes in visual state can reflect changes in the position or posture of the left and right instruments, changes in the opening and closing state of the grippers, and changes in the state of the needle, suture, or target tissue as the instrument is operated.

[0116] Based on the temporal correspondence between the video frame sequence and the action sequence, continuous action sequences corresponding to the visual state changes between each adjacent image frame can be extracted from the action sequence obtained in step B2. This continuous sequence of actions includes actions arranged in chronological order. There are two arm movements, with the first arm movement corresponding to an image frame. The visual state changes between subsequent image frames correspond to the visual state changes between each group of adjacent image frames.

[0117] The first training sample is constructed based on the start image frame, the end image frame, and the corresponding continuous action sequence: in, The starting time The corresponding first training sample, The starting image frame, To end an image frame, For the action prediction window containing A supervisory sequence consisting of consecutive actions.

[0118] For example, in a needle delivery demonstration video, the image frame when the left instrument is holding the needle and the right instrument is approaching the needle can be selected as the starting image frame, and the image frame when the right instrument has already held the needle or completed receiving the needle can be selected as the ending image frame. The continuous action sequence between the two frames can sequentially reflect the actions of the right instrument approaching the needle, adjusting the end posture, closing the gripper to hold the needle, and the left instrument releasing the needle.

[0119] The start time can be changed along the time direction of the video frame sequence. For different start times, start and end image frames are selected, and continuous action sequences within the corresponding time intervals are extracted to construct multiple first training samples from the same real teaching data. For each first training sample, a supervised correspondence is established between the start and end image frames and the continuous action sequences within the corresponding time intervals.

[0120] Step B4: Input the starting image frame and the ending image frame from the first training sample into the basic inverse dynamics model, and train the basic inverse dynamics model using the continuous action sequence from the first training sample as supervision data to obtain the inverse dynamics model.

[0121] In step B4, the starting image frame from the first training sample can be... and end image frame Input the basic inverse dynamics model and the continuous action sequence from the first training sample. As supervised data, the basic inverse dynamics model is trained.

[0122] The basic inverse dynamics model can include a visual encoder, a diffusion Transformer, and an action decoder. The visual encoder extracts visual features from the starting and ending image frames, respectively; the diffusion Transformer models the generation process of a continuous action sequence under the constraints of the visual features; and the action decoder generates continuous actions within the action prediction window based on the output of the diffusion Transformer.

[0123] Specifically, the initial image frame can be processed separately using a visual encoder. and end image frame Feature extraction is performed to obtain the visual features corresponding to the initial image frame. Visual features corresponding to the end image frame Visual features can characterize the visual state of surgical instruments, needles, sutures, and target tissues at the start and end of the action prediction window, respectively.

[0124] The basic inverse dynamics model can employ a flow-matched action head. Let the true continuous action sequence in the first training sample be... Action noise is The exercise training time is... ,in, Based on the actual continuous action sequence, action noise, and action training time steps, determine the corresponding intermediate action sequence and the target velocity for action flow matching: in, Steps for movement training The corresponding intermediate action sequence, The target velocity to match the action flow.

[0125] intermediate action sequence Exercise training time steps Visual features corresponding to the starting image frame and the visual features corresponding to the end image frame. Input the diffusion Transformer, and use the action velocity prediction network to predict the action velocity corresponding to the intermediate action sequence.

[0126] The inverse dynamic flow matching loss is determined based on the difference between the predicted motion velocity and the target velocity of the motion flow matching. in, For inverse dynamic flow matching loss, For parameters Action speed prediction network, Indicates the expectation. This represents the square norm 2.

[0127] Backpropagation can be performed based on the inverse dynamics flow matching loss to update the model parameters of the basic inverse dynamics model. This allows the motion velocity prediction network to gradually approximate the corresponding motion flow matching target velocity, thereby learning the correspondence between the initial visual state, the final visual state, and the continuous motion sequence between them. The motion decoder can generate the motion prediction window based on the output of the diffusion Transformer. A series of actions.

[0128] In some implementations, the parameters of the basic inverse dynamics model, which are pre-trained using publicly available general robot video motion data, can be loaded first. Since the robot mechanism and motion space corresponding to the publicly available data may differ from the target surgical robot platform, the visual encoder and temporal modeling parameters in the basic inverse dynamics model can be retained, and the input and output dimensions of the motion encoder and motion decoder can be adjusted according to the motion dimensions of the target platform.

[0129] Subsequently, the first training sample was constructed using the general instrument motion data of the target surgical robot platform and the real teaching data corresponding to tasks such as needle clamping and needle delivery. The basic inverse dynamics model was then trained to adapt it to the instrument appearance, mechanical configuration, action label form, action dimension, and the correspondence between visual state changes under the endoscope view and robot actions of the target surgical robot platform.

[0130] The action label format, action dimension, and action normalization parameters used during model training can be consistent with the settings used when processing real teaching data in step B2. By repeatedly performing visual feature extraction, intermediate action sequence construction, action speed prediction, inverse dynamics flow matching loss determination, and model parameter updates using multiple first training samples until the preset training termination condition is met, an inverse dynamics model adapted to the target surgical robot platform is obtained.

[0131] In the solution provided in this embodiment, steps B1 to B4 are used to temporally align the video sequence and robot kinematic sequence in the real teaching data, converting the robot kinematic data into a normalized motion sequence that conforms to the motion space of the target surgical robot platform. A first training sample is constructed based on the starting image frame, the ending image frame, and the continuous motion sequence within their corresponding time intervals. This first training sample is then used to train the basic inverse dynamics model. This enables the establishment of an accurate correspondence between changes in the visual state of the surgical scene and the continuous motions of the target surgical robot, allowing the trained inverse dynamics model to adapt to the motion form and scale of the target surgical robot platform, and improving the accuracy and stability of its prediction of continuous robot motions based on candidate surgical video sequences.

[0132] In a preferred embodiment, the process of generating robot motion sequences from candidate surgical video sequences and performing verification and screening can be referred to... Figure 3 As shown. The steps of inputting candidate surgical video sequences into the inverse dynamics model to obtain robot action sequences corresponding to changes in visual state within the candidate surgical video sequences include: Step S2A101: Resample the candidate surgical video sequence according to the sampling frequency used when training the inverse dynamics model.

[0133] In step S2A101, candidate surgical video sequences generated by a pre-trained surgical world model can be obtained, and the candidate surgical video sequences can be resampled according to the sampling frequency used when training the inverse dynamics model. The original sampling frequency of the candidate surgical video sequence may be different from the sampling frequency of the video frame sequence used when training the inverse dynamics model. If the two are directly input into the inverse dynamics model using different time scales, the actual time lengths covered by adjacent image frames will be different, and the same visual displacement may correspond to robot actions of different amplitudes, which can easily cause a mismatch between the action labels output by the model and the visual state changes in the video. Therefore, the sampling frequency used when training the inverse dynamics model can be used as the target sampling frequency, and the candidate surgical video sequence can be resampled according to the temporal order of each image frame in the candidate surgical video sequence to obtain the resampled candidate surgical video sequence.

[0134] The time intervals between adjacent image frames after resampling are the same as or match the time intervals between adjacent image frames when constructing the first training sample of the inverse dynamics model. Therefore, the video data received by the inverse dynamics model during the inference phase has the same time scale as the video data used during the training phase, enabling the robot action output by the model to correspond to the visual state changes between a set of adjacent image frames in the resampling candidate surgical video sequence.

[0135] For example, when the inverse dynamics model constructs the first training sample using real teaching videos of the target surgical robot platform at a preset training frame rate, it can resample the candidate needle delivery videos at the same training frame rate, so that visual changes in the needle delivery videos, such as the instrument approaching, clamping the needle, releasing the needle, and driving the needle to move, are represented at time intervals consistent with the model training process.

[0136] Step S2A102: Divide the resampled candidate surgical video sequence into multiple video windows according to the action prediction window length used when training the inverse dynamics model.

[0137] In step S2A102, the resampled candidate surgical video sequence can be segmented according to the action prediction window length used when training the inverse dynamics model, resulting in multiple video windows arranged in chronological order. The action prediction window length can be the same as the preset number of frames between the starting and ending image frames when constructing the first training sample. Let the action prediction window length be... The interval between the start and end image frames of each video window is... Frames, containing a total of Frame images, including those between the two The visual state changes between adjacent image frames in a group. Accordingly, each video window is used to predict the changes in visual state between the above-mentioned groups using an inverse dynamics model. The changes in visual state correspond to the following groups A robot's actions.

[0138] In one implementation, video windows can be sequentially divided according to the chronological order of the candidate surgical video sequence, with the ending image frame of the previous video window serving as the starting image frame of the next video window. Thus, the last robot action corresponding to the previous video window and the first robot action corresponding to the next video window are temporally continuous, and each visual state change between adjacent image frames in the candidate surgical video sequence can be incorporated into its respective video window.

[0139] Step S2A103: Input the start and end image frames of each video window into the inverse dynamics model to obtain the robot action subsequence corresponding to each video window. The robot actions in each robot action subsequence correspond to the visual state changes between adjacent image frames in the corresponding video window.

[0140] In step S2A103, the start and end image frames of each video window can be determined respectively, and the start and end image frames of each video window are input into the trained inverse dynamics model, which predicts the robot action subsequence corresponding to the video window.

[0141] For the starting time is The action prediction window length is The video window can be denoted as the starting image frame of the video window. The end image frame is denoted as By inputting the starting and ending image frames into the inverse dynamics model, we obtain: in, and These are the start and end image frames for the corresponding video window. For the inverse dynamics model obtained through training, For the robot action subsequence corresponding to the respective video window, including the actions within that video window. A robot's actions.

[0142] The inverse dynamics model can extract the visual features of the starting and ending image frames respectively, and generate robot action subsequences that continuously change from the starting visual state to the ending visual state based on the differences in the visual states of surgical instruments, needles, sutures and target tissues reflected by the two.

[0143] for The first subsequence of robot actions The robot action corresponds to the [number]th action in the candidate surgical video sequence. Frame to the The visual state changes between frames. Therefore, although each video window takes the start and end image frames as model inputs, the model output is not just a single action corresponding to the overall state difference between the two frames, but a series of robot actions corresponding to the visual state changes between each group of adjacent image frames within the video window.

[0144] For example, in the needle delivery video window, the robot action sub-sequence can correspond to the continuous visual state changes in chronological order, such as the right instrument approaching the needle, adjusting the posture of the instrument end, closing the gripper to hold the needle, the left instrument releasing the gripper, and the right instrument moving the needle.

[0145] The robot motion subsequence adopts the motion label format, target platform motion dimension, and motion normalization parameters determined when training the inverse dynamics model. For example, when training uses the end-effector pose of a dual-arm instrument in the endoscope coordinate system, each robot motion output by the inverse dynamics model can include the end-effector position, end-effector posture, and gripper opening / closing amount of the left and right instruments, respectively, while maintaining the same motion component arrangement order and normalization scale as the real teaching data.

[0146] Step S2A104: Integrate the robot action subsequences according to the time order of each video window in the candidate surgical video sequence to obtain the robot action sequence.

[0147] In step S2A104, the temporal order of each video window can be determined based on its start time in the candidate surgical video sequence. The robot action sub-sequences corresponding to each video window are then arranged sequentially according to this temporal order to obtain the robot action sequence corresponding to the candidate surgical video sequence. Specifically, for multiple video windows sequentially divided according to the temporal direction of the candidate surgical video sequence, the robot action sub-sequence corresponding to the first video window can be used as the starting part of the robot action sequence. Then, the robot action sub-sequences corresponding to subsequent video windows are sequentially appended to the previous robot action sub-sequence. Since the robot actions in each robot action sub-sequence are arranged according to the temporal order of adjacent image frames within the corresponding video window, the integrated robot action sequence maintains the same temporal order as the continuous visual state changes in the candidate surgical video sequence.

[0148] When adjacent video windows use the ending image frame of the previous video window as the starting image frame of the next video window, the robot action sub-sequence corresponding to the previous video window covers the visual state changes from the starting image frame to its ending image frame. The robot action sub-sequence corresponding to the next video window starts from that ending image frame and continues to cover the subsequent visual state changes. Therefore, the robot action sub-sequences can be sequentially connected without repeating the visual state changes corresponding to the same set of adjacent image frames.

[0149] For example, when the candidate needle delivery video is divided into the first video window, the second video window, and the third video window in sequence, the robot action sub-sequences corresponding to the three video windows can be integrated in sequence so that the final robot action sequence continuously reflects the process of the instrument approaching the needle, completing the needle clamping, handing over the needle, and driving the needle away from the handover position.

[0150] The integrated robot motion sequence is a sequence of motion labels derived from candidate surgical video sequences using an inverse dynamics model. This robot motion sequence can then be associated with candidate surgical video sequences and denormalized to the motion space of the target surgical robot platform for further verification of motion executability and consistency between visual and motion information.

[0151] In the solution provided in this embodiment, through steps S2A101 to S2A104, the candidate surgical video sequence is resampled and windowed according to the sampling frequency and action prediction window length used during the inverse dynamics model training. Then, robot action subsequences corresponding to visual state changes in adjacent image frames within each video window are predicted based on the starting and ending image frames, and finally integrated in chronological order. This maintains consistency in the time scale and action prediction range between the model training and inference phases, establishes a segment-by-segment correspondence between continuous visual state changes and robot actions in the candidate surgical video sequence, and forms a temporally continuous and sequentially consistent robot action sequence. This improves the accuracy and overall coherence of the correspondence between the action prediction results and the candidate surgical video content.

[0152] In another implementation, the robot motion sequence can be determined based on the candidate surgical video sequence by synchronous recording on a surgical simulation platform.

[0153] Specifically, the pre-trained surgical world model can work collaboratively with the surgical robot simulation platform. The surgical robot simulation platform pre-constructs mechanical models, kinematic models, and control interfaces corresponding to the target surgical robot platform. While the surgical world model generates candidate surgical video sequences based on initial surgical images and task prompts, the surgical robot simulation platform can simultaneously run the surgical action simulation process corresponding to the task prompts. The simulation platform records in real-time the robot kinematic data generated during the execution of this surgical action simulation process, including robot joint states, instrument end-effector poses, and gripper opening and closing states. This kinematic data is then processed according to action label formats and action normalization parameters that match the target surgical robot platform, forming robot action sequences corresponding to the visual state changes between adjacent image frames in the candidate surgical video sequence.

[0154] In this embodiment, the robot motion sequence is derived from the forward kinematics solution of the simulation platform, naturally maintaining temporal alignment and semantic consistency with the candidate surgical video sequence, without the need for inverse dynamics models to infer the visual-to-motion relationship. It should be understood that the above two embodiments can be used individually or in combination, and this application does not limit the specific method for determining the robot motion sequence.

[0155] Step S3: Construct a training dataset for the surgical robot strategy model based at least on the candidate surgical video sequences and their corresponding robot action sequences.

[0156] In step S3, the surgical scene images in the candidate surgical video sequence can be associated with the corresponding robot actions based on the temporal correspondence between the candidate surgical video sequence and the robot action sequence, thereby constructing training samples for training the surgical robot strategy model, and forming a training dataset based on multiple training samples.

[0157] The temporal correspondence between candidate surgical video sequences and robot motion sequences can be established based on the method used to determine the robot motion sequences. For example, when the robot motion sequences are generated by an inverse dynamics model, the correspondence between image frames and robot motions can be established based on the image frames covered by each video window and the motion sub-sequences output by the inverse dynamics model for the corresponding video window. When the robot motion sequences are synchronously recorded by a surgical robot simulation platform, each image frame can be associated with the robot kinematic data or control commands at the corresponding time based on the simulation time corresponding to each image frame in the candidate surgical video sequence.

[0158] Each training sample may include video data and robot motion data. The video data may include one or more consecutive image frames from a candidate surgical video sequence, and the robot motion data may include one or more robot actions corresponding to changes in visual state reflected in the one or more consecutive image frames. Depending on the input format of the surgical robot strategy model, the training sample may also include at least one of task text and robot state. The task text may be task prompts used when generating the candidate surgical video sequence, or determined based on the surgical actions presented in the corresponding candidate surgical video sequence; the robot state may include at least one of the following at a given moment: joint state, instrument end-effector pose, and gripper opening / closing state.

[0159] When constructing the training dataset, candidate surgical video sequences and their corresponding robot motion sequences can be converted into the data format required by the surgical robot policy model. For example, the video sampling frequency, sample time range, robot motion representation method, motion dimension arrangement order, and motion value range can be standardized, while maintaining the temporal correspondence between video data and robot motion data.

[0160] In some embodiments, the training dataset may consist only of training samples constructed based on candidate surgical video sequences and their corresponding robot motion sequences. In other embodiments, the training samples constructed based on candidate surgical video sequences and their corresponding robot motion sequences may be used together with real teaching data from the target surgical robot platform to construct the training dataset.

[0161] For real-world teaching data, video data, robot status, robot actions, and task text can be extracted from synchronously acquired video sequences, robot kinematic data sequences, and task instructions, respectively. Specifically, video data can be surgical videos or video clips from the corresponding real-world teaching process; robot status can include at least one of the following at a given moment: joint status, instrument end-effector pose, and gripper opening / closing state; robot actions can be determined based on the robot kinematic data between adjacent sampling moments, according to the action label format corresponding to the target surgical robot platform; and task text can be determined based on the corresponding task instructions or surgical actions in the real-world teaching data.

[0162] When unifying the format of data based on candidate surgical video sequences with real teaching data, the input and output requirements of the surgical robot strategy model can be followed to ensure that both types of data use the same video sampling frequency, sample time range, robot state representation method, and robot motion representation method. Robot motions in both types of data can use the same motion label format, target platform motion dimension, motion component arrangement order, and motion normalization parameters, giving robot motions from different data sources consistent physical meaning and numerical scale.

[0163] Specifically, when using joint space representation, the robot motions in both types of data can be represented as the joint angles or joint displacements of the left and right instruments, as well as the gripper opening and closing amounts. When using Cartesian space representation, the robot motions in both types of data can be represented as the instrument end-effector positions, instrument end-effector postures, and gripper opening and closing amounts. The corresponding robot states and robot motions can be uniformly represented in the reference coordinate system used by the target surgical robot platform.

[0164] Each training sample may include corresponding video data, task text, robot state, and robot motion. The video data represents the surgical scene and instrument operation process; the task text describes the corresponding surgical task or action requirements; the robot state represents the motion state of the target surgical robot platform at a given time; and the robot motion serves as the supervision data that the surgical robot strategy model needs to predict.

[0165] Data based on candidate surgical video sequences and real teaching data can be divided into multiple training samples according to image frames or time windows, while maintaining the temporal correspondence between video data, robot state, and robot actions in each training sample. For surgical robot strategy models using action prediction windows, the video data within the corresponding time window, the robot state at the start of the window, and the continuous robot actions within the window can be combined to form a single training sample.

[0166] The training dataset for the surgical robot strategy model is obtained by combining the formatted generated training samples with the real teaching training samples. The training dataset can include real training samples obtained from actual teaching actions performed by the target surgical robot platform, or generated training samples constructed from candidate surgical video sequences and their corresponding robot action sequences generated based on the surgical world model. By standardizing the format of the generated training samples and the real teaching training samples and jointly constructing the training dataset, the sources of training data and the scope of surgical scenarios covered can be expanded, providing a more sufficient data foundation for training the surgical robot strategy model.

[0167] In the solution provided in this application embodiment, through steps S1 to S3, a pre-trained surgical world model is used to generate a candidate surgical video sequence containing continuous surgical scene images based on the initial surgical scene and the surgical actions to be performed. Then, the corresponding robot action sequence is determined based on the visual state changes in the candidate surgical video sequence, forming a data generation link from candidate surgical video generation and robot action determination to the construction of a strategy model training dataset. Therefore, the continuous surgical scene videos generated by the surgical world model can be transformed into training data containing visual and robot action information, expanding the sources of training data for the surgical robot strategy model. Furthermore, corresponding training samples can be generated in a targeted manner according to different initial surgical scenes and surgical action processes, thereby reducing the dependence of training data on large-scale real-world teaching acquisition, improving the scene coverage and scalability of the training data, and providing a more sufficient data foundation for the training of the surgical robot strategy model.

[0168] In some embodiments, the following steps are included before constructing the training dataset: The robot's action sequence is verified based on the action executability conditions corresponding to the target surgical robot platform; and / or, Based on the consistency condition between visual information and motion information determined by the real teaching data of the target surgical robot platform, the candidate surgical video sequences and their corresponding robot motion sequences are verified. Candidate surgical video sequences and their corresponding robot motion sequences are selected based on the verification results to construct a training dataset.

[0169] The executability conditions for the target surgical robot platform are used to determine whether the robot's motion sequence conforms to the mechanical motion capabilities of the target surgical robot platform and the requirements of the surgical operating space. The executability conditions can be pre-determined based on at least one of the following: the mechanical configuration of the target surgical robot platform, kinematic relationships, controller safety configuration, device motion parameters, and preoperative calibration results. The executability conditions may include at least one of the following: inverse kinematics solvability constraints, joint position constraints, joint velocity constraints, joint acceleration constraints, deviation constraints of the instrument axis relative to the remote motion center of the cannula, instrument insertion depth constraints, gripper opening and closing range constraints, left and right instrument spacing constraints, and instrument-image acquisition device spacing constraints.

[0170] For example, if the end-effector pose of any instrument in a robot motion sequence does not have an inverse kinematic solution that satisfies the mechanical configuration of the target surgical robot platform, or if the joint position, joint velocity, or joint acceleration exceeds the allowable range of the target surgical robot platform, or if the instrument motion violates constraints such as remote center of motion, insertion depth, or instrument spacing, then the robot motion sequence can be determined to not meet the motion executability conditions. Conversely, if the robot motion sequence satisfies all the adopted constraints, then the robot motion sequence can be determined to meet the motion executability conditions.

[0171] The consistency condition between visual and motion information determined by the real teaching data of the target surgical robot platform is used to determine whether the instrument movement represented by the robot motion sequence corresponds to the instrument movement presented in the candidate surgical video sequence. Specifically, the real teaching data of the target surgical robot platform can include synchronously acquired real teaching videos and robot kinematic data. Based on the robot kinematic data in the real teaching data and the kinematic relationship of the target surgical robot platform, the instrument end effector trajectory during the real teaching process can be determined, and the instrument end effector trajectory can be mapped to the image coordinate system of the real teaching video according to the camera projection relationship to obtain the instrument end effector projection trajectory; the instrument end effector visual trajectory can also be determined from the real teaching video. By statistically analyzing the trajectory deviation between the instrument end effector projection trajectory and the instrument end effector visual trajectory in multiple sets of real teaching data, the allowable deviation range when the target surgical robot platform normally performs the corresponding surgical action can be determined, and this allowable deviation range is used as the consistency condition between visual and motion information.

[0172] It should be noted that the action executability condition and the visual information-action information consistency condition target different data attributes. The action executability condition determines whether the robot action sequence can be executed by the target surgical robot platform; the visual information-action information consistency condition determines whether the instrument movements corresponding to the robot action sequence match the instrument movements presented in the candidate surgical video sequence. Depending on the actual data generation method and verification requirements, only the action executability verification, only the visual information-action information consistency verification, or both types of verification can be performed simultaneously. Specifically, the verification and screening process may include the following steps.

[0173] Step C1: Based on the robot motion sequence and the kinematic relationship of the target surgical robot platform, determine the joint motion trajectory and instrument end effector motion trajectory for verification.

[0174] In step C1, the robot motion sequence determined in step S2 can be obtained, and each robot motion is converted to the actual motion space corresponding to the target surgical robot platform according to the source and motion representation of the robot motion sequence. The actual motion space refers to the motion representation space that can be interpreted according to the mechanical configuration, kinematic relationship, or control interface of the target surgical robot platform.

[0175] When the robot motion sequence is a normalized sequence, the motion normalization parameters used when determining the robot motion sequence can be used to inversely normalize each motion dimension. When the robot motion sequence is synchronously recorded by a surgical robot simulation platform and has adopted the motion representation format corresponding to the target surgical robot platform, the corresponding robot motion can be directly adopted. When the coordinate system, motion dimension arrangement order, or motion representation format used in the robot motion sequence is inconsistent with the target surgical robot platform, coordinate transformation, motion dimension mapping, or motion format conversion can be performed on the robot motion sequence. The converted robot motion can include at least one of the following: joint angles of the left and right instruments, joint displacement, instrument end-effector position, instrument end-effector posture, and gripper opening / closing amount.

[0176] When the robot motion sequence is represented in Cartesian space, the corresponding joint states can be determined based on the instrument end-effector position and pose at each sampling time, according to the mechanical configuration and inverse kinematics of the target surgical robot platform. When the instrument end-effector pose is represented using quaternions or six-dimensional rotation, it can be converted into the pose representation used for kinematic calculations by the target surgical robot platform. If no inverse kinematics solution satisfying the mechanical configuration of the target surgical robot platform exists for any instrument end-effector pose, the corresponding inverse kinematics solution failure result can be recorded for subsequent determination of whether the robot motion sequence meets the motion executability conditions.

[0177] When a robot motion sequence is represented in joint space, the joint angles or joint displacements at corresponding sampling moments can be extracted from each robot motion to obtain the corresponding joint states. Arranging the joint states in the order of the sampling moments yields the joint motion trajectory.

[0178] Based on the forward kinematics of the target surgical robot platform, the positions and postures of the left and right instrument ends corresponding to each joint state in the joint motion trajectory can be determined. These instrument end poses are then arranged in chronological order to obtain the instrument end motion trajectory. The gripper opening / closing range in the robot motion sequence can be time-correlated with the joint motion trajectory and the instrument end motion trajectory, which is used for subsequent gripper opening / closing range verification.

[0179] Step C2: In the case of performing action executability verification, based on the robot action sequence, joint motion trajectory and instrument end effector motion trajectory, perform robot body motion constraint verification and surgical operation space constraint verification on the robot action sequence corresponding to the target surgical robot platform, and determine whether the robot action sequence meets the action executability conditions based on the verification results.

[0180] In step C2, the executability conditions for the target surgical robot platform may include at least one of robot body motion constraints and surgical operation space constraints, used to determine whether the robot motion sequence meets the mechanical motion capabilities of the target surgical robot platform and the spatial requirements during the surgical operation. Robot body motion constraints may include at least one of inverse kinematics solvability constraints, joint position constraints, joint velocity constraints, and joint acceleration constraints.

[0181] When the robot motion sequence is represented in Cartesian space, the inverse kinematics solution obtained in step C1 can be used to determine whether the end effector pose at each sampling moment has an inverse kinematics solution. For the joint motion trajectory obtained in step C1, the joint position can be compared with the allowable range of motion of the corresponding joint on the target surgical robot platform, and the joint motion velocity and acceleration can be determined based on the joint state at adjacent sampling moments. It can then be determined whether the joint motion velocity and acceleration exceed the corresponding velocity and acceleration limits.

[0182] Surgical operation space constraints may include at least one of the following: deviation constraints of the instrument axis relative to the remote motion center of the cannula, instrument insertion depth constraints, gripper opening and closing range constraints, left and right instrument spacing constraints, and instrument-image acquisition device spacing constraints.

[0183] Based on the instrument's end effector trajectory and the structural relationship of the target surgical robot platform, the deviation of the instrument axis relative to the cannula's remote motion center and the instrument insertion depth at each sampling moment can be determined, and the deviation and insertion depth can be compared with the corresponding allowable ranges. Furthermore, the gripper opening / closing amount at each sampling moment can be compared with the allowable gripper range of the target surgical robot platform, and based on the spatial relationship between the left and right instruments and between the instrument and the image acquisition device, it can be determined whether the corresponding spacing meets safety requirements.

[0184] The allowable range corresponding to the above constraints can be determined based on the safety configuration of the target surgical robot platform controller, the device motion parameters, and the preoperative calibration results. When the pose of any instrument end effector does not have an inverse kinematic solution, or when the robot motion sequence violates any of the adopted robot body motion constraints or surgical operation space constraints, it can be determined that the robot motion sequence does not meet the motion executability conditions; when the robot motion sequence meets all the adopted constraints, it can be determined that the robot motion sequence meets the motion executability conditions.

[0185] Step C3: While performing the consistency check between visual information and motion information, determine the projection trajectory of the instrument end effector in the image coordinate system of the candidate surgical video sequence based on the instrument end effector motion trajectory and the camera projection relationship, and determine the instrument end effector visual trajectory from the candidate surgical video sequence.

[0186] In step C3, based on the instrument end motion trajectory obtained in step C1 and the predetermined camera projection relationship, the spatial positions of the left and right instrument ends corresponding to each sampling time can be mapped to the image coordinate system of the candidate surgical video sequence to obtain the instrument end projection position corresponding to each sampling time.

[0187] When the trajectory of the instrument's end effector is represented using the robot's base coordinate system or another robot coordinate system, the spatial position of the instrument's end effector can be transformed to the camera coordinate system based on the spatial relationship between the target surgical robot platform and the image acquisition device. Then, the spatial position of the instrument's end effector in the camera coordinate system is mapped to the image coordinate system according to the camera projection relationship. By arranging the corresponding projected positions of the instrument's end effector in the order of each sampling moment, the projected trajectory of the instrument's end effector can be obtained.

[0188] The visual positions of the left and right instrument ends in the image coordinate system can also be determined from the corresponding image frames of the candidate surgical video sequence, and the visual positions of the same instrument end in different image frames can be associated according to the time sequence of the image frames to obtain the visual trajectory of the instrument end.

[0189] The instrument end-effector projection trajectory is used to characterize the motion process of the instrument end-effector image determined based on the robot motion sequence, the kinematic relationship of the target surgical robot platform, and the camera projection relationship; the instrument end-effector visual trajectory is used to characterize the actual motion process of the instrument end-effector image presented in the candidate surgical video sequence. Both types of trajectories correspond to the corresponding image frames or sampling times in the candidate surgical video sequence.

[0190] Step C4: Based on the trajectory deviation between the instrument end-projection trajectory and the instrument end-visual trajectory, and the allowable deviation range determined based on the real teaching data of the target surgical robot platform, determine whether the candidate surgical video sequence and the robot motion sequence meet the consistency condition of visual information and motion information.

[0191] In step C4, the projection trajectory and visual trajectory of the instrument end can be time-aligned according to the corresponding image frame or sampling time, and the trajectory deviation between the two can be determined based on the projection position and visual position of the instrument end corresponding to the same image frame or the same sampling time.

[0192] For a dual-arm surgical robot platform, the deviation between the projection trajectory of the left instrument end and the visual trajectory of the left instrument end can be determined, as well as the deviation between the projection trajectory of the right instrument end and the visual trajectory of the right instrument end. Based on the corresponding deviations of the left and right instruments, the trajectory deviation between the candidate surgical video sequence and the robot motion sequence can be determined.

[0193] The trajectory deviation corresponding to the candidate surgical video sequence is compared with the allowable deviation range. When the trajectory deviation does not exceed the allowable deviation range, it can be determined that the instrument movement represented by the robot motion sequence matches the instrument movement presented in the candidate surgical video sequence, and the candidate surgical video sequence and the robot motion sequence satisfy the consistency condition of visual information and motion information; when the trajectory deviation exceeds the allowable deviation range, it can be determined that the two do not satisfy the consistency condition of visual information and motion information.

[0194] Step C5: Based on the verification results of the actual verification, filter candidate surgical video sequences and their corresponding robot action sequences to construct a training dataset.

[0195] In step C5, candidate surgical video sequences and their corresponding robot motion sequences can be used as data units to be screened. Based on at least one of the motion executability verification results obtained in step C2 and the visual information and motion information consistency verification results obtained in step C4, it is determined whether to retain the corresponding data.

[0196] When only performing action executability verification, candidate surgical video sequences and their corresponding robot action sequences that satisfy the action executability condition can be retained. When only performing visual information and action information consistency verification, candidate surgical video sequences and their corresponding robot action sequences that satisfy the visual information and action information consistency condition can be retained. When performing both types of verification simultaneously, candidate surgical video sequences and their corresponding robot action sequences that simultaneously satisfy both the action executability condition and the visual information and action information consistency condition can be retained.

[0197] During the screening process, the complete candidate surgical video sequence and its corresponding robot motion sequence can be used as the screening unit, or the candidate surgical video sequence and robot motion sequence can be divided into corresponding video segments and motion sub-sequences according to a preset time window, and each video segment and its corresponding motion sub-sequence can be used as the screening unit.

[0198] For the data retained after screening, based on the correspondence between the candidate surgical video sequence and the robot action sequence in terms of image frames, sampling time, or time window, the two are used as video action pairing data for training datasets to build the surgical robot strategy model in step S3.

[0199] In the solution provided in this embodiment, through steps C1 to C5, by performing action executability verification, robot action data that exceeds the mechanical movement capabilities of the target surgical robot platform or does not meet the surgical operation space requirements can be filtered out; by performing visual information and action information consistency verification, data whose instrument movements determined based on the robot action sequence do not match the instrument movements presented in the candidate surgical video sequence can be filtered out; by performing both types of verification simultaneously, the filtered and retained data can possess both platform executability and visual action consistency. Therefore, appropriate verification combinations can be selected according to the generation method and quality requirements of the training data, reducing the amount of data that exceeds the movement capabilities of the target surgical robot platform, violates surgical operation constraints, or whose instrument movements in the video do not match the action labels entering the training dataset, thus improving the platform adaptability of the training data, the accuracy of video action correspondence, and the overall reliability.

[0200] In a preferred embodiment, the process of performing closed-loop supplementation of the training dataset based on the evaluation results of the surgical robot strategy model can be referred to... Figure 4 As shown. The method also includes: Step D1: Train the surgical robot policy model using the training dataset.

[0201] In step D1, the surgical robot strategy model can be trained using the training dataset constructed in step S3. The surgical robot strategy model can be a visual-language-action strategy model, an imitation learning model, or a robot control strategy model, used to predict the subsequent robot action sequence to be executed by the target surgical robot platform based on the current endoscopic image, task text, and robot state. The robot state can include at least one of joint states, instrument end-effector pose, and gripper opening / closing states; the robot action sequence uses action labeling and action normalization parameters that match the target surgical robot platform.

[0202] In one implementation, a surgical robot strategy model can be trained using video motion matching data, enabling the model to learn the mapping relationship between different surgical scenarios, task requirements, and robot states and corresponding robot actions. The model can then be fine-tuned using real teaching data from the target surgical robot platform to match the robot actions output by the model with the motion space of the target surgical robot platform.

[0203] For needle delivery tasks, the surgical robot strategy model can predict continuous robot actions such as instrument approaching the needle, adjusting the end-effector posture, clamping the needle, releasing the needle, and driving the needle to move, based on the spatial relationship between the needle and the instruments on the left and right sides in the current endoscopic image, the opening and closing state of the grippers, and the needle delivery task text.

[0204] Step D2: Evaluate the surgical robot strategy model under multiple preset scenario groups to obtain the performance indicators corresponding to each preset scenario group.

[0205] In step D2, evaluation samples for assessing the surgical robot strategy model can be constructed, and these samples can be divided into multiple preset scenario groups according to preset scenario semantic elements. The preset scenario groups can correspond to different action categories, needle postures, instrument relative distances, occlusion levels, target tissue regions, or instrument-tissue interaction processes. Action categories can include needle clamping, needle delivery, tissue puncture, suture traction, or knotting; needle postures can be used to distinguish different angles or orientations of the needle in the image; instrument relative distances can be used to distinguish the distance between left and right instruments or between the instrument and the object being operated on; occlusion levels can be used to distinguish scenarios where the needle, instrument tip, or target tissue is not occluded, partially occluded, or has a high degree of occlusion; instrument-tissue interaction processes can be used to distinguish different interaction states such as approach, contact, clamping, puncture, traction, or release.

[0206] Each evaluation sample can be divided into one or more preset scene groups based on its corresponding scene label. For example, a needle delivery evaluation sample may simultaneously belong to scene groups such as "needle delivery action", "needle body posture deviation", "distance between left and right instruments is large" and "needle body is partially obscured".

[0207] Under each preset scenario group, performance metrics for the surgical robot strategy model can be calculated separately. Performance metrics can include task success rate and trajectory error between predicted and actual motion trajectories. Depending on the motion labeling format used by the target surgical robot platform, trajectory error can include joint motion error, instrument end-effector position error, instrument end-effector posture error, and gripper motion error. Performance metrics can also include motion smoothness and safety constraint fulfillment. Thus, performance metrics corresponding to each preset scenario group can be obtained to reflect the specific performance of the surgical robot strategy model under different motion types, instrument states, and surgical scenario conditions, rather than evaluating model performance solely based on the overall metrics of all evaluation samples.

[0208] Step D3: Group the preset scenarios whose performance indicators do not meet the preset indicator conditions and identify them as weak scenarios.

[0209] In step D3, the performance indicators corresponding to each preset scenario group can be compared with the preset indicator conditions, and the preset scenario group whose performance indicators do not meet the preset indicator conditions can be identified as a weak scenario.

[0210] The preset index conditions can be determined based on the training objectives of the surgical robot strategy model, the control accuracy requirements of the target surgical robot platform, and the operational requirements of the corresponding surgical task. For example, when the task success rate corresponding to a certain preset scenario group is lower than the preset success rate requirement, the preset scenario group can be identified as a weak scenario; when any trajectory error among the joint motion error, instrument end position error, instrument end posture error, and gripper motion error corresponding to the preset scenario group is higher than the corresponding preset error requirement, the preset scenario group can also be identified as a weak scenario.

[0211] For example, when the needle delivery strategy model has a low success rate in scenarios where the needle posture is skewed, the relative distance between the left and right instruments is large, or the needle is partially occluded, or when the prediction errors for the instrument end position, posture, and gripper movements are high, the corresponding scenario groups can be identified as weak scenarios. Weak scenarios indicate that the surgical robot strategy model's action prediction or task execution capabilities under the corresponding scenario conditions have not yet met the preset requirements, and also suggest that the current training dataset may have insufficient coverage of this type of scenario or an insufficient number of effective training samples.

[0212] Step D4: Determine the scene semantic elements corresponding to the weak scenes, and generate supplementary task prompt information based on the scene semantic elements.

[0213] In step D4, corresponding failure samples in the weak scene can be obtained, and scene semantic elements used to characterize the weak scene can be determined based on the task text, initial image, and scene label of the failure samples. Scene semantic elements may include action type, operating instruments, instrument spatial relationship, needle or target tissue state, occlusion status, and expected interaction result. Among them, action type describes the surgical action that needs to be supplemented; operating instruments describe the left-side instruments, right-side instruments, or other surgical instruments involved in the corresponding action; instrument spatial relationship describes the relative position between instruments and between instruments and needles or target tissue; needle or target tissue state describes the posture, position, or current interaction state of the operated object; occlusion status describes the visibility of the needle, instrument tip, or target tissue; and expected interaction result describes the target state that should be formed after the corresponding action is completed.

[0214] Supplementary task prompts can be obtained by organizing the semantic elements of each scene into a natural language description according to the occurrence process of surgical actions in the corresponding weak scenarios. The action type, operating instruments, spatial relationships, state of the operating object, occlusion situation, and expected interaction result described in the supplementary task prompts should jointly define the data scene that needs to be supplemented.

[0215] For example, when the weak scenario is "needle handover operation with needle body tilted, left and right instruments far apart and needle body partially obscured", the supplementary task prompt information can describe that the receiving instrument moves towards the partially obscured and tilted needle body in the initial state of being far away from the needle body, adjusts the end posture of the instrument and clamps the needle body, and releases the needle body from the original clamping instrument, thus forming the expected result of needle body handover.

[0216] Step D5: Based on the supplementary task prompts, obtain supplementary candidate surgical video sequences for weak scenarios generated by the surgical world model.

[0217] In step D5, an initial surgical image matching the weak scene can be selected based on the scene semantic elements corresponding to the weak scene. This initial surgical image and supplementary task prompts are then input into a pre-trained surgical world model to obtain a supplementary candidate surgical video sequence for the weak scene generated by the pre-trained surgical world model. The initial surgical image can be selected from real surgical videos, surgical teaching videos, or other surgical scene videos. In one example, the initial surgical image can be an endoscopic image.

[0218] For example, in a weak needle delivery scenario where the needle is partially occluded and the instruments on the left and right sides are far apart, an initial surgical image can be selected that presents the corresponding needle posture, occlusion state, and initial relative position of the instruments. This image, along with supplementary task prompts describing the corresponding needle delivery process, is then input into a pre-trained surgical world model. Based on the initial surgical state represented by the initial surgical scene image and the surgical action process described by the supplementary task prompts, the pre-trained surgical world model generates one or more supplementary candidate surgical video sequences that reflect the continuous evolution of the corresponding weak scenario.

[0219] Step D6: Generate and verify robot motion sequences for the supplementary candidate surgical video sequences to obtain supplementary video motion pairing data, and update the training dataset using the supplementary video motion pairing data.

[0220] In step D6, the supplementary candidate surgical video sequence generated in step D5 can be input into the inverse dynamics model, and the supplementary robot action sequence corresponding to the visual state changes in the supplementary candidate surgical video sequence can be obtained according to the robot action sequence generation method in the aforementioned steps.

[0221] Using the verification and screening methods described above, the executability of the supplementary robot action sequences corresponding to the target surgical robot platform is verified, and the consistency of visual and action information between the supplementary candidate surgical video sequences and the supplementary robot action sequences is verified. For data that simultaneously meets the conditions of action executability and consistency of visual and action information, the corresponding supplementary candidate surgical video sequences and their supplementary robot action sequences are determined as supplementary video action pairing data.

[0222] Following the data format adopted in the aforementioned steps, the supplementary video-action pairing data is converted to the same video, task text, robot state, and robot action format as the current training dataset, and then added to the training dataset to update it. Specifically, the corresponding supplementary task prompts can be used as task text, while maintaining the temporal correspondence between video frames, robot states, and robot actions.

[0223] The updated training dataset includes video-action pairing training samples for weak scenarios, which can be used to retrain or continue training the surgical robot policy model. After training, the surgical robot policy model can be re-evaluated under the same preset scenario groups, and the sample size, task success rate, and corresponding trajectory error of each scenario group can be updated.

[0224] In some implementations, steps D1 to D6 can be repeated to continuously feed the evaluation results of the surgical robot strategy model back to the training data generation process. Data supplementation can end when the performance metrics of each preset scenario group meet the preset requirements; the iterative loop can also end when data supplementation for a preset number of consecutive iterations fails to effectively improve model performance, reaches the maximum number of iterations, or fails to obtain new and effective supplementary video action pairing data.

[0225] In the solution provided in this embodiment, steps D1 to D6 feed back the scenario-specific performance evaluation results of the surgical robot strategy model to the training data generation process. Weak scenarios are identified based on performance indicators, and supplementary task prompts are generated using corresponding scenario semantic elements. This drives the pre-trained surgical world model to generate supplementary candidate surgical video sequences. Supplementary video action pairing data is obtained through action sequence generation, verification, and filtering, and the training dataset is updated accordingly. Therefore, effective training samples can be supplemented for specific scenarios where the strategy model's performance is insufficient, improving the coverage of training data across different surgical actions, instrument states, and scenario conditions. This reduces the randomness of data supplementation and gradually improves the strategy model's action prediction ability and task execution performance in weak scenarios.

[0226] Based on the above embodiments, the following example illustrates the process of generating, filtering, and supplementing training data for the surgical robot strategy model, using the needle delivery task where the left-side instrument holds the needle and hands it to the right-side instrument. The process of generating, filtering, and constructing the training dataset for the needle delivery task can be referred to... Figure 5 As shown. The methods for model training, action sequence generation, verification and filtering, and training dataset updating that are not specifically described in this embodiment can be referred to the relevant descriptions in the preceding embodiments, and will not be repeated here.

[0227] In this embodiment, a surgical video containing the needle delivery process can be acquired and segmented into video clips depicting surgical actions such as needle clamping, tissue puncture, suture pulling, and knot tying. For the needle delivery video clips, the spatial relationship between the left and right instruments, the posture and position of the needle, the interaction between each instrument and the needle, and the needle handover process can be determined based on the actual surgical actions presented in the video. These semantic elements are then organized into natural language descriptions to obtain corresponding training prompts.

[0228] Surgical videos, bi-arm kinematic data, and gripper opening and closing distances during the actual needle delivery teaching process can be simultaneously acquired on the target surgical robot platform. The acquired data is then synchronously resampled and subjected to coordinate transformation to ensure temporal alignment between the video frame sequence and the robot's kinematic sequence, and to represent the instrument's end-effector pose using a unified coordinate system. The resulting realistic teaching data can be used to train inverse dynamics models, determine motion normalization parameters, and serve as training samples for constructing surgical robot strategy models.

[0229] When training the surgical world model, a low-rank adaptation module can be configured in the attention layer and feedforward network of the pre-trained base video world model. Second training samples are constructed using the initial surgical image corresponding to the needle-handling action video clip, training cues, and sequences of real future surgical scenes. A flow-matching training objective is constructed based on the video latent variables corresponding to the sequences of real future surgical scenes, and the parameters of the low-rank adaptation module are updated based on the flow-matching loss to obtain the pre-trained surgical world model.

[0230] When generating candidate needle delivery video sequences, initial surgical images containing the needle and instruments on both sides can be selected from the needle delivery action video clips. These initial surgical images, along with task prompts describing the needle delivery process, are then input into a pre-trained surgical world model. The pre-trained surgical world model can generate multiple candidate needle delivery video sequences based on different initial noise levels, resulting in variations in needle posture, relative instrument positions, and the needle delivery process. Because the surgical world model is trained on real surgical videos, the generated candidate needle delivery video sequences maintain the corresponding visual features and video style of the surgical scene. Compared to video data generated by traditional simulators, this approach reduces the visual domain difference between the generated data and the real surgical video to a certain extent.

[0231] When training the inverse dynamics model, the parameters of the basic inverse dynamics model, pre-trained based on general robot video motion data, can be loaded. Then, using general instrument motion teaching data and needle delivery teaching data from the target surgical robot platform, the basic inverse dynamics model is trained to obtain an inverse dynamics model adapted to the target surgical robot platform. The inverse dynamics model is used to generate robot motion sequences corresponding to the changes in the initial and final visual states in the surgical video. By training the basic inverse dynamics model using real teaching data from the target surgical robot platform, a mapping relationship between changes in the visual state of the surgical scene and the motion space of the target surgical robot can be established even with a relatively limited scale of real teaching data. This allows for the use of candidate surgical video sequences to supplement corresponding robot motion labels, reducing the reliance on large-scale real teaching data acquisition during the training data construction process.

[0232] In this embodiment, Cartesian space representation can be used as the action label form for the target surgical robot platform. The actions of each instrument can include the three-dimensional instrument end-effector position, the six-dimensional instrument end-effector rotation, and the gripper opening and closing amount. The actions of the left and right instruments are combined to form the actions of both arms. Each action dimension is normalized according to the statistical range corresponding to the real teaching data of the target surgical robot platform, and the same action representation method, action dimension arrangement order, and action normalization parameters are used in the training and inference phases of the inverse dynamics model.

[0233] When constructing training samples for the inverse dynamics model, a start and end image frame with a preset frame interval can be selected from a time-aligned video frame sequence. Then, continuous dual-arm motion sequences corresponding to visual state changes between several groups of adjacent image frames within that time interval are extracted from the corresponding motion sequences. The start and end image frames are input into the basic inverse dynamics model, and the continuous dual-arm motion sequences are used as supervised data to train the basic inverse dynamics model. This allows the trained inverse dynamics model to generate robot motion sequences corresponding to the continuous visual state changes between the start and end visual states. The preset frame number can be determined based on the video sampling frequency, the rate of change of the surgical movements, and the motion prediction window length of the inverse dynamics model.

[0234] For candidate needle delivery video sequences generated by the surgical world model, resampling can be performed according to the sampling frequency used during inverse dynamics model training, and the sequences can be divided into multiple video windows according to the action prediction window length of the inverse dynamics model. Each video window includes a start image frame, an end image frame, and several image frames in between. The frame interval between the start and end image frames corresponds to the preset number of frames used during inverse dynamics model training. The start and end image frames of each video window are input into the inverse dynamics model to obtain corresponding action subsequences. Each arm movement in the action subsequence corresponds to the visual state changes between several groups of adjacent image frames within the corresponding video window. The corresponding action subsequences are integrated according to the temporal order of each video window to obtain the robot action sequence corresponding to the candidate needle delivery video sequence.

[0235] When verifying and filtering robot motion sequences, the robot motion sequences can be inversely normalized to the actual motion space of the target surgical robot platform based on the aforementioned motion normalization parameters. Then, inverse kinematics can be solved based on the end-effector poses of each instrument to obtain the corresponding joint motion trajectories. For robot motion sequences that do not have an inverse kinematics solution, or whose joint positions, joint velocities, and joint accelerations exceed the corresponding limits of the target surgical robot platform, it can be determined that they do not satisfy the robot body motion constraints.

[0236] Furthermore, the surgical operation space constraint verification of the robot's motion sequence can be performed based on the movement trajectory of the instrument's end effector. When the deviation of the instrument axis relative to the telecentric center of motion, the instrument insertion depth, or the opening and closing amount of the gripper exceeds the corresponding allowable range, or when the distance between the left and right instruments or between the instrument and the image acquisition device does not meet the safety constraints, it can be determined that the corresponding robot motion sequence does not meet the conditions for motion executability.

[0237] For robot motion sequences that meet the conditions for action executability, the end effector projection trajectory can be determined based on the relationship between the end effector's motion trajectory and the camera projection, and the end effector visual trajectory can be determined from candidate needle delivery video sequences. The end effector projection trajectory and the end effector visual trajectory are time-aligned, and the trajectory deviation between them is determined. When the trajectory deviation does not exceed the allowable deviation range determined based on real teaching data, it can be determined that the candidate needle delivery video sequence and the robot motion sequence meet the consistency condition of visual information and motion information, and the two are identified as video-motion pairing data.

[0238] Through the above verification and screening, candidate needle delivery video sequences and their robot action labels can be examined from aspects such as robot motion constraints, surgical operation space constraints, and the correspondence between video content and action information. Data with unexecutable actions, motion states exceeding the target platform limits, or mismatches between instrument visual motion and robot actions can be screened out, thereby improving the action executability, platform adaptability, and video action consistency of video action pairing data.

[0239] In this embodiment, the selected video-action pairing data and the actual teaching data of the target surgical robot platform can be converted into the same video, task text, robot state, and robot action format to construct a training dataset for the surgical robot strategy model. The video-action pairing data has both the visual features of the surgical scene learned from real surgical videos and robot action labels adapted to the target surgical robot platform, which can serve as a supplement to the actual teaching data to expand the training samples corresponding to the needle delivery operation.

[0240] The training dataset can be used in stages according to the data source. For example, video motion matching data can be used to initially train the surgical robot strategy model, and real needle delivery teaching data can be used to fine-tune the surgical robot strategy model after initial training so that it can adapt to the actual motion distribution and control requirements of the target surgical robot platform.

[0241] Based on semantic elements of the scene, such as needle posture, relative distance between left and right instruments, and degree of needle occlusion, the needle delivery evaluation samples can be divided into multiple preset scenario groups. Under each preset scenario group, the task success rate, instrument end position error, instrument end rotation error, and gripper action error of the surgical robot strategy model can be evaluated.

[0242] When the success rate of the surgical robot strategy model in scenarios where the needle posture is skewed, the distance between instruments on the left and right sides is large, or the needle is partially occluded is lower than the preset success rate requirement, or when the corresponding position error, rotation error, or gripper motion error exceeds the preset error requirement, the corresponding scenario group can be identified as a weak scenario. New supplementary task prompts are generated based on the semantic elements of the weak scenario, such as the needle posture, instrument spatial relationship, occlusion state, and needle handover process.

[0243] By inputting initial surgical images matching the weak scenario and supplementary task prompts into a pre-trained surgical world model, supplementary candidate needle delivery video sequences for the weak scenario can be generated. A robot action sequence corresponding to the supplementary candidate needle delivery video sequence is generated using an inverse dynamics model. This sequence is then validated and filtered according to the aforementioned action executability conditions and the consistency condition between visual and action information, resulting in supplementary video-action pairing data. Adding this supplementary video-action pairing data to the training dataset updates the training dataset, allowing for further training and evaluation of the surgical robot strategy model.

[0244] Therefore, based on the actual performance of the surgical robot strategy model in different scenario groups, training samples corresponding to scenarios with insufficient performance, such as needle posture deviation, instrument distance, needle occlusion, and others, can be supplemented in a targeted manner. This avoids relying solely on human experience to expand training data without specific targeting, and improves the coverage of training data for weak motion scenarios and complex surgical states. The above closed-loop supplementation process can improve the action prediction ability and task execution performance of the surgical robot strategy model under different scenario conditions, enhance the generalization ability of the strategy model, and improve the efficiency of training data utilization during the autonomous operation of the surgical robot.

[0245] The above training data supplementation process can be repeated until the performance indicators corresponding to each preset scenario group meet the preset indicator conditions, or until the performance of the surgical robot strategy model is effectively improved after a preset number of consecutive training data supplementations.

[0246] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0247] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0248] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a method for generating training data for a surgical robot strategy model as provided in any one or more of the above embodiments. Figure 6 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses 1105 and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses 1105 can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0249] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus 1105 or other means. Figure 6 Taking the connection between China and Israel via bus 1105 as an example.

[0250] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0251] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0252] In this embodiment, a computer-readable medium stores a computer program / instruction, which, when executed by a processor, implements a method for generating training data for a surgical robot strategy model provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0253] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0254] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0255] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0256] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0257] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0258] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0259] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0260] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0261] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Terms such as "first," "second," etc., are used only to distinguish descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0262] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A method for generating training data for a surgical robot strategy model, characterized in that, include: Obtain a candidate surgical video sequence generated by a pre-trained surgical world model, wherein the surgical world model is used to generate a candidate surgical video sequence containing continuous surgical scene images based on the initial surgical scene and the surgical action process to be performed; Based on the candidate surgical video sequence, determine the robot action sequence corresponding to the visual state changes in the candidate surgical video sequence; A training dataset for the surgical robot strategy model is constructed based at least on the candidate surgical video sequences and their corresponding robot action sequences.

2. The method for generating training data for a surgical robot strategy model according to claim 1, characterized in that, The step of determining the robot action sequence corresponding to the visual state changes in the candidate surgical video sequence includes: The candidate surgical video sequence is input into the inverse dynamics model to obtain the robot action sequence, wherein the inverse dynamics model is obtained by training the basic inverse dynamics model based on the real teaching data of the target surgical robot platform.

3. The method for generating training data for a surgical robot strategy model according to claim 2, characterized in that, The training of the basic inverse dynamics model based on real teaching data from the target surgical robot platform includes: The video sequence and robot kinematic data sequence in the real teaching data are synchronously resampled to obtain a time-aligned video frame sequence and a resampled robot kinematic data sequence; According to the action label format corresponding to the target surgical robot platform and the action normalization parameters corresponding to the real teaching data, the resampled robot kinematic data sequence is processed to obtain the action sequence; Select a start image frame and an end image frame with a preset number of frames interval from the video frame sequence, and extract continuous action sequences from the action sequence that correspond to the visual state changes between each adjacent image frame from the start image frame to the end image frame, and construct the first training sample; The starting and ending image frames from the first training sample are input into the basic inverse dynamics model, and the basic inverse dynamics model is trained using the continuous action sequence from the first training sample as supervision data to obtain the inverse dynamics model.

4. The method for generating training data for a surgical robot strategy model according to claim 3, characterized in that, The step of processing the resampled robot kinematic data sequence according to the action label format corresponding to the target surgical robot platform and the action normalization parameters corresponding to the real teaching data to obtain the action sequence includes: Based on the motion space of the target surgical robot platform, the motion label format and the motion dimension of the target platform are determined, wherein the motion label format is a joint space representation or a Cartesian space representation; From the resampled robot kinematic data sequence, extract the kinematic data of the left and right devices corresponding to each synchronous sampling time, and represent the kinematic data of the left and right devices as left device actions and right device actions respectively according to the action label format; The left-side instrument movement and the right-side instrument movement corresponding to the same synchronous sampling time are combined into a two-arm movement; The motion normalization parameters are determined based on the statistical range of each motion dimension in the real teaching data, and each of the two arm movements is normalized based on the motion normalization parameters. The normalized arm movements are arranged according to the order of the synchronous sampling times to obtain the movement sequence.

5. The method for generating training data for a surgical robot strategy model according to claim 2, characterized in that, The step of inputting the candidate surgical video sequence into the inverse dynamics model to obtain the robot action sequence includes: The candidate surgical video sequences are resampled according to the sampling frequency used when training the inverse dynamics model; Based on the action prediction window length used when training the inverse dynamics model, the resampled candidate surgical video sequence is divided into multiple video windows; The starting and ending image frames of each video window are input into the inverse dynamics model to obtain the robot action subsequence corresponding to each video window. The robot actions in each robot action subsequence correspond to the visual state changes between adjacent image frames within the corresponding video window. The robot action subsequences are integrated according to the time order of each video window in the candidate surgical video sequence to obtain the robot action sequence.

6. The method for generating training data for a surgical robot strategy model according to any one of claims 1-5, characterized in that, Before constructing the training dataset, the following is also included: The robot action sequence is verified based on the action executability conditions corresponding to the target surgical robot platform; and / or, Based on the consistency condition of visual information and motion information determined by the real teaching data of the target surgical robot platform, the candidate surgical video sequence and its corresponding robot motion sequence are verified. The candidate surgical video sequences and their corresponding robot motion sequences are selected based on the verification results to construct the training dataset.

7. The method for generating training data for a surgical robot strategy model according to claim 1, characterized in that, The pre-trained surgical world model is trained in the following manner: Acquire a surgical video, divide the surgical video into multiple surgical action video segments, and determine training prompts that describe the surgical action process corresponding to each of the surgical action video segments; A second training sample is constructed based on each of the surgical action video clips. Each second training sample includes the first frame of the corresponding surgical action video clip as the initial image for training, the continuous images after the first frame as a sequence of real future surgical scenes, and the corresponding training prompts. A low-rank adaptation module is configured in the pre-trained basic video world model. The initial training image and the training cues are used as generation conditions. A flow matching training objective is constructed based on the video latent variables corresponding to the real future surgical scene sequence. The parameters of the low-rank adaptation module are trained based on the flow matching loss to obtain the pre-trained surgical world model.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions, which, when executed, cause the processors to perform the surgical robot strategy model training data generation method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the surgical robot strategy model training data generation method as described in any one of claims 1-7.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the surgical robot strategy model training data generation method as described in any one of claims 1-7.

Images