Surgical robot auxiliary system, readable storage medium and surgical robot

Abstract

The invention provides a surgical robot auxiliary system, a readable storage medium and a surgical robot, and the system comprises a motion state obtaining unit which is configured to obtain the current motion state information of a main control arm; the interaction state acquisition unit is configured to acquire a current operation scene image; and the interaction state prediction unit is configured to predict non-delay instrument and tissue interaction state information according to the current motion state information and the current operation scene image. According to the method, the interaction state of the tail end of the instrument and the tissue can be predicted according to the current motion state information of the main control arm, so that the influence of time delay on an operator is eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and in particular to a surgical robot-assisted system, a readable storage medium and a surgical robot. Background Art

[0002] The current teleoperated surgical robot includes the operator, the master end, the slave end, the communication module and the image trolley. The operator issues a command by manipulating the master control arm, and the command is transmitted to the slave end through the communication module. After receiving the command, the control system of the slave end will control the slave end robot to reproduce the operation of the master end, and the slave end robot will transmit the surgical screen to the master end through the communication module. With the continuous development of technology, the application scope and functions of surgical robots are also expanding. At present, surgical robots can not only be applied to a variety of surgical fields (such as: cardiac surgery, neurosurgery, urology, gynecology, etc.), but also can perform remote surgical operations. However, with the increase in the control distance, delay has become an unavoidable prominent problem. Delay will cause the slave end robot to fail to respond to the doctor's instructions in time, affecting the accuracy of the operation. If the delay is too large, the doctor may make a wrong judgment on the interaction between the end of the instrument and the tissue, affecting the safety of the operation.

[0003] Current solutions mainly include: using high-speed network connections to increase data transmission speed; or reducing data transmission volume through software; or improving the hardware performance of surgical robots to reduce processing delays and display delays. However, these solutions can only reduce delays to a limited extent. Short-distance delays are still tens of milliseconds, and long-distance delays can still reach hundreds of milliseconds, which still affects doctors' operations.

[0004] It should be noted that the information disclosed in the background technology section of the invention is only intended to deepen the understanding of the general background technology of the invention, and should not be regarded as an admission or suggestion in any form that the information constitutes prior art already known to those skilled in the art. Summary of the invention

[0005] The object of the present invention is to provide a surgical robot-assisted system, a readable storage medium and a surgical robot, which can predict the interaction state between the instrument end and the tissue according to the current motion state information of the main control arm, thereby eliminating the impact of delay on the operator.

[0006] To achieve the above-mentioned purpose, the present invention provides a surgical robot-assisted system, which includes: a motion state acquisition unit, configured to acquire the current motion state information of the main control arm; an interaction state acquisition unit, configured to acquire the current operation scene image; and an interaction state prediction unit, configured to predict the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image.

[0007] Optionally, the interaction state prediction unit is configured to predict the interaction state information between the non-delay instrument and the tissue using a pre-trained first diffusion model based on the current motion state information and the current operation scene image.

[0008] Optionally, the interaction state prediction unit is further configured to extract instrument end key features and target tissue key features from the current operation scene image, and input the extracted instrument end key features and target tissue key features into the first diffusion model as additional information.

[0009] Optionally, the surgical robot-assisted system provided by the present invention further includes an auxiliary display unit configured to generate a zero-delay operation scene image based on the zero-delay instrument and tissue interaction state information.

[0010] Optionally, the delay-free instrument and tissue interaction state information includes first delay-free instrument end posture information and first delay-free target tissue shape information; the auxiliary display unit is configured to: perform field of view conversion on the first delay-free instrument end posture information and the first delay-free target tissue shape information according to the motion state information of the main control arm in the time period from the moment corresponding to the current operation scene image to the current moment, so as to obtain second delay-free instrument end posture information and second delay-free target tissue shape information under the endoscopic field of view corresponding to the current motion state information of the main control arm; and generate a delay-free operation scene image according to the second delay-free instrument end posture information and the second delay-free target tissue shape information.

[0011] Optionally, the auxiliary display unit is configured to: obtain a first posture information of the endoscope at a moment corresponding to the current operation scene image and a second posture information of the endoscope at the current moment according to the motion state information of the main control arm in the time period from the moment corresponding to the current operation scene image to the current moment; obtain relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image to the current moment according to the first posture information and the second posture information; and perform field of view conversion on the first delay-free instrument end posture information and the first delay-free target tissue shape information according to the relative transformation matrix information, so as to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the field of view of the endoscope corresponding to the current motion state information of the main control arm.

[0012] Optionally, the auxiliary display unit is configured to generate the delay-free operation scene image based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, and a pre-acquired preset instrument end wireframe model and a preset target tissue wireframe model.

[0013] Optionally, the auxiliary display unit is configured to: perform image segmentation on the current operation scene image to obtain an instrument end segmentation image and a target tissue segmentation image; and generate the delay-free operation scene image based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, the instrument end segmentation image and the target tissue segmentation image.

[0014] Optionally, the auxiliary display unit is configured to generate the delay-free operation scene image using a pre-trained second diffusion model based on the current operation scene image, the second delay-free instrument end posture information and the second delay-free target tissue shape information.

[0015] Optionally, the surgical robot-assisted system provided by the present invention further includes an image processing unit configured to superimpose the zero-delay operation scene image with the current operation scene image.

[0016] To achieve the above-mentioned purpose, the present invention provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining the current motion state information of the main control arm; obtaining the current operation scene image; and predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image.

[0017] Optionally, when the computer program is executed by the processor, the following steps are further implemented: generating a zero-delay operation scene image based on the zero-delay instrument and tissue interaction state information.

[0018] Optionally, when the computer program is executed by a processor, the following step is further implemented: superimposing the zero-delay operation scene image with the current operation scene image.

[0019] In order to achieve the above-mentioned objectives, the present invention also provides a surgical robot, which includes the surgical robot-assisted system described in any one of the above items or the readable storage medium described in any one of the above items.

[0020] Compared with the prior art, the surgical robot-assisted system, readable storage medium and surgical robot provided by the present invention have the following beneficial effects: the surgical robot-assisted system provided by the present invention can obtain the current motion state information of the main control arm through the motion state acquisition unit; can obtain the current operation scene image through the interaction state acquisition unit; can predict the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image through the interaction state prediction unit, thereby eliminating the impact of the delay on the operator, allowing the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the operation, and at the same time, it can also avoid the discontinuous motion of the slave robot caused by the delay and the waiting time of the operator, thereby improving the efficiency of the surgical operation.

[0021] Since the readable storage medium and surgical robot provided by the present invention belong to the same inventive concept as the surgical robot-assisted system provided by the present invention, the readable storage medium and surgical robot provided by the present invention at least have all the beneficial effects of the surgical robot-assisted system provided by the present invention. For details, please refer to the relevant description of the beneficial effects of the surgical robot-assisted system provided by the present invention in the above text. Therefore, the beneficial effects of the readable storage medium and surgical robot provided by the present invention will not be described one by one here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the application scenarios of surgical robots.

[0023] Figure 2 Schematic diagram of the partial structure of the main control arm.

[0024] Figure 3 Schematic diagram of the control principle of the surgical robot.

[0025] Figure 4 It is a structural schematic diagram of the visual capture device.

[0026] Figure 5 Schematic diagram of the interaction between device and tissue.

[0027] Figure 6 This is a schematic diagram of local surgery delay.

[0028] Figure 7 This is a schematic diagram of remote surgery delay.

[0029] Figure 8 This is a structural block diagram of a surgical robot-assisted system provided in one embodiment of the present invention.

[0030] Fig. 9 This is a time-delay diagram of the interaction status between the device and the tissue.

[0031] Fig.10 A schematic diagram of generating a delay-free operation scene image based on a wireframe model provided in one embodiment of the present invention.

[0032] Fig.11 A schematic diagram of generating a non-delay operation scene image based on a segmented image provided by an embodiment of the present invention.

[0033] Fig.12 A schematic diagram of generating a delay-free operation scene image based on a diffusion model provided in one embodiment of the present invention.

[0034] Fig.13 A schematic diagram of an image superposition effect provided by one embodiment of the present invention.

[0035] Fig.14 A flowchart of steps that can be implemented by a readable storage medium provided in one embodiment of the present invention.

[0036] The reference numerals are as follows: doctor cart-100; main doctor cart-100A; remote doctor cart-100B; main control arm-110; doctor handle-111; main end display-120; main controller-130; patient cart-200; local patient cart-200A; robotic arm-210; surgical instrument-220; instrument end-221; visual capture device-230; slave controller-240; image cart-300; local image cart-300A ; Target tissue-400; Server-500; Motion state acquisition unit-610; Interaction state acquisition unit-620; Interaction state prediction unit-630; Auxiliary display unit-640; Image processing unit-650; Current operation scene image-11; Actual operation scene image-12; Non-delay operation scene image-13; Preset instrument end wireframe model-21; Preset target tissue wireframe model-22; Instrument end segmentation image-31; Target tissue segmentation image-32. DETAILED DESCRIPTION

[0037] The following is a further detailed description of the surgical robot-assisted system, readable storage medium and surgical robot proposed in the present invention in combination with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the condition that the effect that the present invention can produce and the purpose that can be achieved are the same or similar, should still fall within the scope of the technical content disclosed in the present invention.

[0038] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. The singular forms "a", "an" and "the" include plural objects, the term "or" is generally used in a sense including "and / or", the term "several" is generally used in a sense including "at least one", and the term "at least two" is generally used in a sense including "two or more". In addition, the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or suggesting relative importance or implicitly indicating the number of the indicated technical features.

[0039] In addition, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0040] For ease of understanding, the research background of the surgical robot-assisted system, readable storage medium, and surgical robot provided by the present invention is briefly described. Figure 1 , which is a schematic diagram of the application scenarios of surgical robots. Figure 1 As shown, the surgical robot includes a doctor's trolley 100 (master end) and a patient's trolley 200 (slave end), and a master control arm 110 is mounted on the doctor's trolley 100. The patient's trolley 200 is equipped with at least one mechanical arm 210, and surgical instruments 220 and visual capture devices 230 (such as endoscopes) can be mounted on the mechanical arms 210 respectively. The operator (doctor) performs remote operation through the master control arm 110 on the doctor's trolley 100 to achieve minimally invasive surgery on the patient on the operating table. Among them, the master control arm 110 and the mechanical arm 210, as well as the surgical instruments 220 and visual capture devices 230 (such as endoscopes) mounted on the mechanical arm 210 form a master-slave control relationship. Specifically, the mechanical arm 210, the surgical instruments 220 and the visual capture devices 230 (such as endoscopes) move in accordance with the movement of the master control arm 110 during the operation, that is, they move in accordance with the operation of the operator's hand.

[0041] Please continue to refer to Figure 2 , which is a schematic diagram of a partial structure of the master control arm 110. Figure 2 As shown, the main control arm 110 receives the operator's hand motion state information through the doctor's handle 111 at the end as the motion control input of the surgical robot. Figure 1 ,like Figure 1As shown, the doctor's trolley 100 is also provided with a main-end display 120 (doctor's observation window), which can display the operation scene image collected by the visual capture device 230 (such as an endoscope). The operator controls the mechanical arm 210, the surgical instrument 220 and the visual capture device 230 (such as an endoscope) to move through the main control arm 110 according to the operation scene image displayed by the main-end display 120. The surgical instrument 220 and the visual capture device 230 (such as an endoscope) can enter the lesion position (the location of the target tissue 400) through the wound or natural opening on the patient's body.

[0042] Please continue to refer to Figure 1 ,like Figure 1 As shown, the surgical robot further includes an image trolley 300. The operation scene image acquired by the visual capture device 230 (eg, an endoscope) can be transmitted to the image trolley 300 for display.

[0043] Please continue to refer to Figure 3 , which is a schematic diagram of the control principle of the surgical robot. Figure 3 As shown, the doctor's cart 100 includes a main controller 130 and a main display 120. The main controller 130 is used to sense the operator's hand movements and identify the operator's operation intentions. The main display 120 (doctor's observation window) is used to present the surgical screen (operation scene image). The patient cart 200 includes a slave controller 240 and a visual capture device 230 (e.g., an endoscope, such as Figure 4 As shown, which is a structural diagram of the visual capture device 230), the slave controller 240 is used to receive instructions from the main controller 130, and after processing, controls the slave end robot 210 to reproduce the master end operation. The visual capture device 230 (such as an endoscope) collects the current surgical screen (operation scene image) and outputs the current surgical screen (operation scene image) to the master end display 120 on the doctor's trolley 100 through the image trolley 300 for display.

[0044] Please continue to refer to Figure 5 , which is a schematic diagram of the interaction between the device and the tissue. Figure 5 As shown, the instrument-tissue interaction state information collected by the visual capture device 230 (eg, an endoscope) includes the position information of the instrument tip 221 and the shape information of the target tissue 400 .

[0045] Please continue to refer to Figure 6 , which is a schematic diagram of local surgery delay. Figure 6 As shown, when the surgical robot performs local surgery, the delay includes: t 1. The local doctor cart 100A outputs a control signal to the local patient cart 200A, which receives the signal; delay t2. The local patient cart 200A performs the action until the local visual capture device 230 (such as an endoscope) captures the current surgical screen (operation scene image); delay t 3. The local image cart 300A outputs the current surgical screen (operation scene image) to the main end display 120 (doctor observation window) on the local doctor cart 100A for display.

[0046] Please continue to refer to Figure 7 , which is a schematic diagram of remote surgery delay. Figure 7 As shown, when the surgical robot performs remote surgery, the delay includes: t 1. The remote doctor trolley 100B outputs a control signal to the server 500, which receives the signal; delay t 2. The server 500 outputs a control signal to the local patient cart 200A to receive the signal; delay t 3. The local patient cart 200A performs the action until the local visual capture device 230 (such as an endoscope) captures the current surgical screen (operation scene image); delay t 4. The local image trolley 300A outputs the current surgical screen (operation scene image) to the server 500 to receive the signal (a decoding process may be involved during this period); delay t 5. The server 500 outputs the signal to the main display 120 (doctor observation window) on the remote doctor cart 100B for imaging display.

[0047] It can be seen that the delay in the operation performed by the surgical robot includes the following two categories: delay A, which is the delay from the doctor's trolley 100's instruction to the patient's trolley 200's response, which is t 1. For remote surgery t 1+ t 2; Delay B, for the patient cart 200 to respond to the doctor cart 100 to produce the image, for local surgery t 2+ t 3. For remote surgery t 3+ t 4+ t 5.

[0048] Based on this, the core idea of ​​the present invention is to provide a surgical robot-assisted system, a readable storage medium and a surgical robot, which can predict the interaction state between the instrument end 221 and the tissue according to the current motion state information of the main control arm 110, thereby eliminating the impact of delay on the operator.

[0049] It should be noted that the surgical robot-assisted system and readable storage medium provided by the present invention can be set on the main end of the surgical robot (doctor's trolley 100). It should also be noted that, as can be understood by those skilled in the art, for the slave end, the "end" referred to herein refers to the end close to the lesion; for the main end, the "end" referred to herein refers to the end close to the operator.

[0050] To realize the above idea, the present invention provides a surgical robot-assisted system, please refer to Figure 8 , which is a structural block diagram of a surgical robot-assisted system provided by one embodiment of the present invention, such as Figure 8 As shown, the surgical robot-assisted system provided by the present invention includes: a motion state acquisition unit 610, configured to acquire current motion state information of the main control arm 110; an interaction state acquisition unit 620, configured to acquire the current operation scene image 11 (such as Fig. 9 As shown); and an interaction state prediction unit 630, configured to predict the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11.

[0051] Therefore, the present invention can accurately predict the interaction state information of the non-delayed instrument (instrument end 221) and the tissue (target tissue 400) based on the current motion state information of the main control arm 110 and the current operation scene image 11 (including the current instrument and tissue interaction state information), thereby eliminating the impact of the delay on the operator, allowing the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the operation, and at the same time avoiding the discontinuous movement of the slave robot caused by the delay and the waiting time of the operator, thereby improving the efficiency of the surgical operation.

[0052] It should be noted that, as can be understood by those skilled in the art, the "current operation scene image 11" referred to herein refers to the operation scene image currently received by the doctor's trolley 100 (which is the same as the operation scene image currently received by the master control arm 110). t k-A-B Motion status at all times Pt k-A-B Correspondingly, compared with the current motion state of the master control arm 110 ( t k Motion status at all times Pt k ) There is a certain delay. Please continue to refer to Fig. 9 , which is a time-delay diagram of the interaction between the device and the tissue. Fig. 9 As shown, due to the existence of delay A and delay B, when the doctor handle 111 of the master control arm 110 corresponds to t k Actual motion status at the moment Pt k When the patient cart 200 corresponds to the doctor's handle 111 (main control arm 110), t k-A Motion status at all times Pt k-A The surgical screen (operation scene image) presented by the doctor's observation window (main terminal display 120) corresponds to the doctor's handle 111 (main control arm 110) in t k-A-B Motion status at all times Pt k-A-B The non-delayed instrument-tissue interaction state predicted by the present invention refers to the image displayed in the doctor's observation window (the main end display 120) corresponding to the doctor's handle 111 (the main control arm 110) in t k-A-B Motion status at all times Pt k-A-B When the doctor handles 111, t k Motion status at all times Pt k The corresponding device and tissue interaction state. It should be noted that, as can be understood by those skilled in the art, t k At the moment (i.e., the current moment), the surgical screen (i.e., the current operation scene image 11) presented in the doctor's observation window (the main terminal display 120) includes the surgical scene image 110 and the surgical scene image 111 (the main control arm 110) at the moment (i.e., the current moment). t k-A-B Motion status at all times Pt k-A-B The corresponding instrument and tissue interaction state information, the actual operation scene image 12 collected by the visual capture device 230 (such as an endoscope) includes the doctor's handle 111 (main control arm 110) in t k-A Motion status at all times Pt k-A Corresponding device and tissue interaction status information.

[0053] In some exemplary embodiments, the motion state acquisition unit 610 is configured to acquire the current motion state information of the master control arm 110 according to the current position information of each joint of the master control arm 110. Specifically, the current position information of each joint of the master control arm 110 can be acquired by encoders installed on each joint of the master control arm 110.

[0054] Furthermore, the current motion state information of the master control arm 110 includes the current position and posture of the end of the master control arm 110 (the current position and posture of the doctor's handle 111 ).

[0055] Furthermore, the current motion state information of the master control arm 110 also includes the current speed and current acceleration of the end (doctor's handle 111) of the master control arm 110. Therefore, by obtaining the current speed and current acceleration of the end (doctor's handle 111) of the master control arm 110, the accuracy of the predicted non-delay instrument and tissue interaction state information can be further guaranteed.

[0056] In some exemplary embodiments, the interaction state prediction unit 630 is configured to: preprocess the current motion state information and the current operation scene image 11, and predict the delay-free instrument and tissue interaction state information based on the preprocessed current motion state information and the current operation scene image 11.

[0057] Therefore, by preprocessing the current motion state information and the current operation scene image 11, noise can be effectively removed and image quality can be improved, thereby effectively ensuring the accuracy of the predicted non-delay instrument and tissue interaction state information.

[0058] Furthermore, since the data such as the posture, velocity and acceleration of the end of the master control arm 110 may be affected by factors such as sensor noise, inconsistent acquisition frequency or slight shaking of the operator's hand, the current motion state information may be subjected to preprocessing steps such as denoising, data normalization and time series interpolation.

[0059] Specifically, filtering technology (such as Kalman filtering or sliding average filtering) can be used to smooth the data such as the posture, velocity and acceleration in the current motion information of the master control arm 110 to reduce noise interference. By normalizing the data such as the posture, velocity and acceleration in the current motion information of the master control arm 110, these data can be normalized to a unified dimension (for example, the position coordinates are normalized to [-1,1]) to match the input requirements of the first diffusion model below, thereby ensuring the accuracy of the delay-free device and tissue interaction state information predicted by the first diffusion model. Furthermore, if the motion state data collection of the master control arm 110 has a discontinuous timestamp or insufficient sampling rate, the missing data points can be supplemented by interpolation methods such as linear interpolation or spline interpolation to ensure the continuity and integrity of the data.

[0060] Furthermore, since the current operation scene image 11 is a real-time surgical image captured by the visual capture device 230 (such as an endoscope), it may be affected by changes in illumination, reflections on the tissue surface, differences in image resolution, or compression distortion during transmission, etc. In order to improve the image quality and adapt to the input requirements of the first diffusion model described below, pre-processing steps such as denoising and enhancement, and image standardization may be performed on the current operation scene image 11.

[0061] Specifically, an image denoising algorithm (such as Gaussian blur or median filtering) can be used to remove noise in the current operation scene image 11, and contrast enhancement or histogram equalization can be used to improve the visibility of the image to highlight the features of the instrument end 221 and the target tissue 400. The pixel value of each pixel of the denoised and enhanced current operation scene image 11 is then normalized to the range of [0, 1] or [-1, 1], and the resolution of the normalized current operation scene image 11 is adjusted to match the input size requirement of the first diffusion model described below (for example, the resolution of the normalized current operation scene image 11 is adjusted to 256×256 pixels).

[0062] In some exemplary embodiments, the interaction state prediction unit 630 is configured to predict the non-delayed instrument and tissue interaction state information using a pre-trained first diffusion model based on the current motion state information and the current operation scene image 11. Since the diffusion model can theoretically approximate any complex data distribution, and the training process of the diffusion model is relatively stable and is not prone to problems such as mode collapse, and at the same time, by adjusting the parameters in the forward diffusion and reverse generation processes, fine control of the generated content can be achieved. Therefore, the present invention can predict more accurate non-delayed instrument and tissue interaction state information corresponding to the current motion state of the main control arm 110 (doctor's handle 111) by using a pre-trained first diffusion model.

[0063] Specifically, the first diffusion model aims to obtain a given diffusion model with the master control arm 110 (doctor handle 111) in t k-A-B Motion status at all times Pt k-A-B The corresponding operation scene image and the attached main control arm 110 (doctor handle 111) are shown in t k The motion state information at the moment is generated by the main control arm 110 (doctor handle 111) at t k The interaction status information between the device and the tissue corresponding to the motion status information at any moment.

[0064] Furthermore, before training the first diffusion model, it is necessary to first collect sample data required for training (including motion state data of the main control arm 110 and corresponding operation scene images, etc.), and then process the sample data, including but not limited to denoising, filtering, interpolation and other operations to improve the accuracy and stability of the sample data, and then train the pre-created controllable diffusion model based on the processed sample data and preset model parameters and initial state to obtain a trained first diffusion model.

[0065] Furthermore, the training of the first diffusion model depends on high-quality sample data, which needs to reflect the true correspondence between the motion state (motion information) of the master control arm 110 and the interactive state information between the instrument and the tissue. Specifically, the sample data can be obtained in the following ways: 1) Collection of motion state data, through encoders or other motion sensors (such as inertial measurement units IMU) installed on each joint of the master control arm, real-time recording of the motion state information of the end of the master control arm 110 (such as the doctor's handle 111), including posture, velocity and acceleration. Data collection can be carried out in simulated surgery or real surgery scenes, covering a variety of operation modes (such as translation, rotation, clamping, etc.) to ensure the diversity of samples; 2) Collection of operation scene images, using a visual capture device 230 (such as an endoscope) to synchronously record the operation scene images corresponding to the motion state information of the master control arm 110. The operation scene image includes the interaction state information between the instrument end 221 and the target tissue 400 (including the position and posture of the instrument end 221 and the deformation of the target tissue 400), and the acquisition frequency of the operation scene image is consistent with the acquisition frequency of the motion state data of the master control arm 110; 3) Data pairing and annotation, aligning the motion state data of the master control arm 110 at each moment with the corresponding operation scene image by time stamp to form a paired sample. To improve the training effect, the position and posture of the instrument end 221 and the shape of the target tissue 400 in the image can be extracted as a supervisory signal through manual annotation or automatic algorithms (such as image segmentation technology). The sample data set needs to include local surgery and remote surgery scenes to cover the interaction states under different delay conditions.

[0066] Furthermore, the first diffusion model adopts a generative training framework based on noise addition and denoising to predict the non-delayed instrument and tissue interaction state information from the current motion state information and the current operation scene image 11. The specific training process is as follows.

[0067] 1) Model initialization: Build an initial controllable diffusion model, usually based on the U-Net architecture, combined with a conditional input module to receive motion state information and operation scene images. Model parameters (such as weights) are initialized to random values ​​or pre-trained values ​​(such as those transferred from general image generation tasks).

[0068] 2) Forward diffusion process: Gaussian noise is gradually added to the target data in the paired samples (i.e., the state information of the interaction between the device and the tissue without delay) to generate a series of noisy versions of the data. This process simulates the degradation of data from the real distribution to the noise distribution. Usually, a fixed number of steps (such as 1000 steps) is set, and the noise intensity of each step is controlled by a predefined variance schedule.

[0069] 3) Reverse generation process: The training model gradually reconstructs the delay-free interaction state information from the noisy data through reverse denoising. The input conditions include the current motion state information and the current operation scene image11. The model learns to predict the denoising result of each step under given conditions. The loss function uses the mean square error (MSE) to calculate the difference between the denoising result predicted by the model and the actual delay-free interaction state.

[0070] 4) Optimization process: Use gradient descent method (such as Adam optimizer) to iteratively update model parameters. Learning rate scheduling (such as cosine annealing) can be introduced during training to balance convergence speed and stability. Use batch samples (for example, batch size is 32) for each iteration, and randomly shuffle the data set to avoid overfitting.

[0071] Furthermore, in order to ensure that the first diffusion model can accurately predict the delay-free instrument and tissue interaction state information based on the current motion state information of the master control arm 110 and the current operation scene image 11, the training end condition of the first diffusion model needs to comprehensively measure the model performance and stability, specifically including the following standards.

[0072] 1) Loss convergence: When the average loss (such as MSE) on the training set changes less than a preset threshold (such as 0.001) within several consecutive iterations (such as 10 epochs), the model is considered to have converged.

[0073] 2) Validation set performance: Reserve a portion of sample data as a validation set and regularly evaluate the prediction accuracy of the model on the validation set (such as pixel-level error or pose error of interaction state information). Stop training when the validation set loss no longer decreases significantly or there are signs of overfitting (such as an increase in validation set loss).

[0074] 3) Generation quality assessment: The similarity between the non-delayed interactive state information generated by the model and the real data is evaluated through manual inspection or automatic indicators (such as structural similarity SSIM and peak signal-to-noise ratio PSNR). When the generated results meet the accuracy requirements of surgical applications (such as posture error less than 1 mm and tissue shape error less than 5%), the training ends.

[0075] 4) Maximum iteration limit: To avoid infinite training, you can set a maximum number of iterations (such as 500 epochs) or training duration (such as 72 hours). If the upper limit is reached and the above conditions are not fully met, the model version with the best performance on the validation set is selected as the final result.

[0076] In some exemplary embodiments, the interaction state prediction unit 630 is further configured to extract instrument end key features and target tissue key features of the current operation scene image 11, and input the extracted instrument end key features and target tissue key features as additional information into the first diffusion model.

[0077] Therefore, by inputting the extracted key features of the instrument end and the target tissue as additional information into the first diffusion model, the accuracy of the non-delay instrument-tissue interaction state information predicted by the first diffusion model can be further improved.

[0078] Specifically, the key features of the instrument end and the key features of the target tissue can be extracted from the current operation scene image 11 using an edge detection algorithm (such as the Canny operator) or a pre-trained convolutional neural network.

[0079] Please continue to refer to Figure 8 ,like Figure 8 As shown, in some exemplary embodiments, the surgical robot-assisted system provided by the present invention further includes an auxiliary display unit 640, which is configured to generate a non-delay operation scene image 13 (such as Fig.10 Thus, by generating a non-delay operation scene image 13 according to the non-delay instrument and tissue interaction state information, the current motion state of the master control arm 110 (in t k Motion status at all times Pt k ) The corresponding delay-free instrument and tissue interaction status is more intuitively displayed to the operator.

[0080] In some exemplary embodiments, the non-delay instrument and tissue interaction state information includes first non-delay instrument end position information and first non-delay target tissue shape information. The auxiliary display unit 640 is configured to: perform field of view conversion on the first non-delay instrument end position information and the first non-delay target tissue shape information according to the motion state information of the main control arm 110 in the time period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain the second non-delay instrument end position information and the second non-delay target tissue shape information under the endoscope field of view corresponding to the current motion state information of the main control arm 110; and generate a non-delay operation scene image 13 according to the second non-delay instrument end position information and the second non-delay target tissue shape information.

[0081] Since the first non-delay instrument end posture information and the first non-delay target tissue shape information are based on the main control arm 110 (doctor handle 111) t k Motion status at all times Pt k and the main control arm 110 (doctor handle 111) t k-A-B Motion status at all times Pt k-A-B The corresponding device and tissue interaction state information (current operation scene image 11) is predicted. t k-A-B Time to t k During this period of time, the motion state of the main control arm 110 (doctor's handle 111) may cause the posture of the endoscope to change, thereby causing the field of view of the endoscope to change. Therefore, by t k-A-B time) to the current time (i.e. t k The motion state information within the time period of (i.e., the current motion state information of the master control arm 110) can be obtained by converting the first non-delay instrument end posture information and the first non-delay target tissue shape information into a field of view. t k Motion status at all times Pt k ) The second delay-free instrument end posture information and the second delay-free target tissue shape information under the corresponding endoscopic field of view can further eliminate the impact of the delay on the operator, so as to further improve the accuracy and safety of the operation.

[0082] Specifically, the master control arm 110 (doctor's handle 111) can be used to t k-A-B Time to t k The motion state information during this period of time is transformed through geometric transformation (such as rigid body transformation) t k-A-B Endoscopic view at all times (with the doctor's handle 111 in t k-A-B Motion status at all times Pt k-A-B The first non-delay instrument end position information and the first non-delay target tissue shape information under the corresponding endoscope field of view are converted to t k Endoscopic view at all times (with the doctor's handle 111 in tk Motion status at all times Pt k The second delay-free instrument end posture information and the second delay-free target tissue shape information under the corresponding endoscopic field of view.

[0083] In some exemplary embodiments, the auxiliary display unit 640 is configured to: obtain a first posture information of the endoscope at a moment corresponding to the current operation scene image 11 and a second posture information at the current moment based on the motion state information of the main control arm 110 in the time period from the moment corresponding to the current operation scene image 11 to the current moment; obtain relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image 11 to the current moment based on the first posture information and the second posture information; and perform field of view conversion on the first delay-free instrument end posture information and the first delay-free target tissue shape information based on the relative transformation matrix information to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the field of view of the endoscope corresponding to the current motion state information of the main control arm 110.

[0084] Specifically, the master control arm 110 may be configured to ... t k-A-B time) to the current time (i.e. t k The motion state information of the main control arm 110 at the moment (i.e., the position change of the endoscope during this period of time) is calculated. The motion state of the main control arm 110 is usually collected by the joint encoder, including the three-dimensional position coordinates (x, y, z) and posture (e.g., rotation represented by Euler angles or quaternions) of the end of the main control arm 110 (the doctor's handle 111). Through kinematic forward solution calculation, it can be obtained that the endoscope is at the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) and the current moment (i.e. t k The positions at the time are recorded as and , each pose can be represented by a 4×4 homogeneous transformation matrix, as shown below:

[0085] in, T Express posture, R is a 3×3 rotation matrix, t is a 3×1 translation vector.

[0086] Further, the time at which the endoscope starts from the moment corresponding to the current operation scene image 11 (i.e. t k-A-B time) to the current time (i.e. t k The relative transformation matrix of T rel :

[0087] The relative transformation matrix T rel Describes the endoscope from the moment corresponding to the current operation scene image 11 (ie t k-A-B time) to the current time (i.e. t k The rigid body motion of the object (at time instant) includes rotational and translational components.

[0088] In some exemplary embodiments, the auxiliary display unit 640 is configured to display the second non-delay instrument end posture information, the second non-delay target tissue shape information, the preset instrument end wireframe model 21 (such as Fig.10 as shown) and the preset target organization wireframe model 22 (as shown Fig.10 As shown in FIG. 1 ), the non-delay operation scene image 13 is generated. Thus, by generating the non-delay operation scene image 13 according to the second non-delay instrument end posture information, the second non-delay target tissue shape information, the preset instrument end wireframe model 21, and the preset target tissue wireframe model 22, the operator can more intuitively view the current motion state of the master control arm 110 (in FIG. 1 ). t k Motion status at all times Pt k ) corresponds to the delay-free device-tissue interaction state.

[0089] Please continue to refer to Fig.10 , which is a schematic diagram of generating a non-delayed operation scene image based on a wireframe model provided by an embodiment of the present invention. Fig.10 As shown, based on the second non-delay instrument end posture information, the second non-delay target tissue shape information, the preset instrument end wireframe model 21 and the preset target tissue wireframe model 22, the non-delay operation scene image 13 can be generated, and the surface contours of the instrument end 221 and the target tissue 400 can be presented in the form of lines, and only the edges and vertices of the instrument end 221 and the target tissue 400 are displayed, without displaying the color, texture and other information of their surface.

[0090] In some other exemplary embodiments, the auxiliary display unit 640 is configured to: perform image segmentation on the current operation scene image 11 to obtain an instrument end segmentation image 31 (eg, Fig.11 ) and the target tissue segmentation image 32 (as shown Fig.11 As shown); and according to the second non-delay instrument end posture information, the second non-delay target tissue shape information, the instrument end segmentation image 31 and the target tissue segmentation image 32, the non-delay operation scene image 13 is generated. Thus, by performing image segmentation on the current operation scene image 11, the structural features (such as color, texture, brightness, etc.) of the instrument end 221 and the target tissue 400 can be separated, so that the non-delay operation scene image 13 generated based on the second non-delay instrument end posture information, the second non-delay target tissue shape information, the instrument end segmentation image 31 and the target tissue segmentation image 32 can be guaranteed to have the real structural features (such as color, texture, brightness, etc.) of the instrument end 221 and the target tissue 400, thereby ensuring that the generated non-delay operation scene image 13 is more realistic.

[0091] Please refer to Fig.11 , which is a schematic diagram of generating a non-delay operation scene image based on a segmented image provided by an embodiment of the present invention. Fig.11 As shown, the delay-free operation scene image 13 generated based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, the instrument end segmentation image 31 and the target tissue segmentation image 32 can not only display the edges and vertices of the instrument end 221 and the target tissue 400, but also display the color, texture and other information of its surface.

[0092] Specifically, the auxiliary display unit 640 is further configured to: pre-process the current operation scene image 11 before performing image segmentation on the current operation scene image 11. Thus, by pre-processing the current operation scene image 11 (including but not limited to converting the current operation scene image 11 into a grayscale image or a binary image, denoising, etc.), and then performing image segmentation on the pre-processed current operation scene image 11, the accuracy of the image segmentation result can be effectively guaranteed.

[0093] Furthermore, the auxiliary display unit 640 is configured to perform image segmentation on the current operation scene image 11 (the preprocessed current operation scene image 11) through the following process: performing edge detection on the current operation scene image 11 (the preprocessed current operation scene image 11) using an edge detection algorithm to obtain an edge image; and extracting the contours of the instrument end 221 and the target tissue 400 from the edge image using a contour extraction algorithm to obtain an instrument end segmentation image 31 and a target tissue segmentation image 32.

[0094] Therefore, by using an edge detection algorithm to perform edge detection on the current operation scene image 11 (the current operation scene image 11 after preprocessing), the boundaries in the image can be detected, thereby facilitating the subsequent identification of the contours of the instrument end 221 and the target tissue 400; by using a contour extraction algorithm, different contours can be identified and extracted from the image after edge detection (edge ​​image), and by analyzing the size, shape and position of the contours, it is possible to distinguish which are the instrument end 221 and which are soft tissues (target tissue 400).

[0095] It should be noted that, as can be understood by those skilled in the art, in some other embodiments, the auxiliary display unit 640 is configured to: use a pre-trained image segmentation model to perform semantic segmentation on the current operation scene image 11 (the pre-processed current operation scene image 11) to obtain the instrument end segmentation image 31 and the target tissue segmentation image 32. Therefore, by using a pre-trained image segmentation model (deep learning model such as U-Net, Mask R-CNN, etc.) to perform semantic segmentation on the current operation scene image 11 (the pre-processed current operation scene image 11), the accuracy of the segmented instrument end segmentation image 31 and the target tissue segmentation image 32 can be effectively guaranteed.

[0096] In some further exemplary embodiments, the auxiliary display unit 640 is configured to generate the non-delay operation scene image 13 using a pre-trained second diffusion model based on the current operation scene image 11, the second non-delay instrument end posture information and the second non-delay target tissue shape information. Fig.12 As shown, it is a schematic diagram of generating a delay-free operation scene image based on a diffusion model provided by an embodiment of the present invention), which can generate more realistic and delicate images. Therefore, compared with the traditional generation model, the diffusion model shows higher clarity and fewer artifacts in image generation, and the output image quality is higher.

[0097] Please continue to refer to Figure 8 ,like Figure 8As shown, in some exemplary embodiments, the surgical robot-assisted system provided by the present invention further includes an image processing unit 650 configured to superimpose the non-delayed operation scene image 13 with the current operation scene image 11. t k The motion state P at the current moment t k The corresponding non-delay operation scene image 13 is superimposed on the current operation scene image 11, and the non-delay operation scene image 13 and the current operation scene image 11 can be superimposed into a picture to be displayed on the same interface (such as Fig.13 As shown, it is a schematic diagram of the image superposition effect provided by one embodiment of the present invention), which makes it easier for the operator to observe.

[0098] The present invention also provides a readable storage medium, wherein the readable storage medium stores a computer program. Fig.14 , which is a flow chart of the steps that can be implemented by a readable storage medium provided by an embodiment of the present invention. Fig.14 As shown, when the computer program is executed by the processor, the following steps can be implemented: step S100, obtaining the current motion state information of the main control arm 110; step S200, obtaining the current operation scene image 11; step S300, predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11.

[0099] Therefore, the present invention can accurately predict the interaction state information of the instrument and tissue without delay based on the current motion state information of the master control arm 110 and the current operation scene image 11, thereby eliminating the impact of the delay on the operator, allowing the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the operation, and at the same time avoiding the discontinuous movement of the slave robot caused by the delay and the waiting time of the operator, thereby improving the efficiency of the surgical operation.

[0100] In some exemplary embodiments, predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11 includes: predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11 using a pre-trained first diffusion model.

[0101] In some exemplary embodiments, when the computer program is executed by a processor, the following steps may be implemented: generating a zero-delay operation scene image 13 based on the zero-delay instrument and tissue interaction state information.

[0102] In some exemplary embodiments, when the computer program is executed by the processor, the following steps are also implemented: extracting the key features of the instrument end and the key features of the target tissue from the current operation scene image 11, and inputting the extracted key features of the instrument end and the key features of the target tissue into the first diffusion model as additional information.

[0103] In some exemplary embodiments, the zero-delay instrument and tissue interaction state information includes first zero-delay instrument end posture information and first zero-delay target tissue shape information.

[0104] The method of generating a delay-free operation scene image 13 based on the delay-free instrument and tissue interaction state information includes: performing a field of view conversion on the first delay-free instrument end posture information and the first delay-free target tissue shape information based on the motion state information of the master control arm 110 in the time period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110; and generating the delay-free operation scene image 13 based on the second delay-free instrument end posture information and the second delay-free target tissue shape information.

[0105] In some exemplary embodiments, the first non-delay instrument end posture information and the first non-delay target tissue shape information are converted according to the motion state information of the master control arm 110 in the time period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain the second non-delay instrument end posture information and the second non-delay target tissue shape information in the endoscope field of view corresponding to the current motion state information of the master control arm 110, including: obtaining the endoscope according to the motion state information of the master control arm 110 in the time period from the moment corresponding to the current operation scene image 11 to the current moment, The endoscope is configured to obtain the first posture information of the endoscope at the moment corresponding to the current operation scene image 11 and the second posture information of the endoscope at the current moment; according to the first posture information and the second posture information, the relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image 11 to the current moment is obtained; and according to the relative transformation matrix information, the first delay-free instrument end posture information and the first delay-free target tissue shape information are converted into a field of view to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the field of view of the endoscope corresponding to the current motion state information of the main control arm 110.

[0106] In some exemplary embodiments, the method of generating a delay-free operation scene image 13 based on the second delay-free instrument end posture information and the second delay-free target tissue shape information includes: generating the delay-free operation scene image 13 based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, and a pre-acquired preset instrument end wireframe model 21 and a preset target tissue wireframe model 22.

[0107] In other exemplary embodiments, the generating of the delay-free operation scene image 13 according to the second delay-free instrument end posture information and the second delay-free target tissue shape information comprises: performing image segmentation on the current operation scene image 11 to obtain an instrument end segmentation image 31 and a target tissue segmentation image 32; and generating the delay-free operation scene image 13 according to the second delay-free instrument end posture information, the second delay-free target tissue shape information, the instrument end segmentation image 31 and the target tissue segmentation image 32.

[0108] In some other exemplary embodiments, the method of generating a delay-free operation scene image 13 based on the second delay-free instrument end posture information and the second delay-free target tissue shape information includes: generating the delay-free operation scene image 13 by using a pre-trained second diffusion model based on the current operation scene image 11, the second delay-free instrument end posture information and the second delay-free target tissue shape information.

[0109] In some exemplary embodiments, when the computer program is executed by a processor, the following steps may be implemented: superimposing the non-delay operation scene image 13 with the current operation scene image 11 .

[0110] Further, the readable storage medium provided by the present invention can adopt any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0111] Further, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, etc., or any suitable combination of the above.

[0112] To achieve the above-mentioned object, the present invention further provides a surgical robot, the surgical robot comprising the surgical robot-assisted system described above or the readable storage medium described above. Since the surgical robot provided by the present invention comprises the surgical robot-assisted system described above or the readable storage medium described above, the surgical robot provided by the present invention at least comprises all the beneficial effects of the surgical robot-assisted system described above or the readable storage medium described above. For details, reference may be made to the above description of the beneficial effects of the surgical robot-assisted system or the readable storage medium provided by the present invention. Therefore, the beneficial effects of the surgical robot provided by the present invention will not be described one by one here.

[0113] To sum up, compared with the prior art, the surgical robot-assisted system, readable storage medium and surgical robot provided by the present invention have the following beneficial effects: the present invention can accurately predict the delay-free instrument and tissue interaction state information based on the current motion state information of the master control arm 110 and the current operation scene image 11, thereby eliminating the impact of the delay on the operator, allowing the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the operation, and at the same time avoiding the discontinuous movement of the slave robot caused by the delay and the waiting time of the operator, thereby improving the efficiency of the surgical operation.

[0114] It should be noted that the computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0115] It should be noted that the devices and methods disclosed in the embodiments of this article can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of this article. In this regard, each box in the flowchart or block diagram can represent a module, a part of a program or a code, and the module, a part of a program segment or a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented by a dedicated hardware-based system for performing a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0116] It should also be noted that the above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of the present invention. Any changes and modifications made by a person skilled in the art in the field of the present invention based on the above disclosure are within the scope of protection of the present invention. Obviously, a person skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A surgical robot-assisted system, characterized in that: include: A motion state acquisition unit configured to acquire current motion state information of the master control arm; An interaction state acquisition unit, configured to acquire a current operation scene image; as well as The interaction state prediction unit is configured to predict the interaction state information between the non-delayed instrument and the tissue according to the current motion state information and the current operation scene image.

2. The surgical robot-assisted system according to claim 1, characterized in that: The interaction state prediction unit is configured to predict the interaction state information between the non-delay instrument and the tissue using a pre-trained first diffusion model according to the current motion state information and the current operation scene image.

3. The surgical robot-assisted system according to claim 2, characterized in that: The interaction state prediction unit is also configured to extract instrument end key features and target tissue key features from the current operation scene image, and input the extracted instrument end key features and target tissue key features into the first diffusion model as additional information.

4. The surgical robot-assisted system according to claim 1, characterized in that: It also includes an auxiliary display unit configured to generate a zero-delay operation scene image based on the zero-delay instrument and tissue interaction state information.

5. The surgical robot-assisted system according to claim 4, characterized in that: The non-delay instrument and tissue interaction state information includes first non-delay instrument end position information and first non-delay target tissue shape information; The auxiliary display unit is configured as follows: Performing a field of view conversion on the first non-delay instrument end position information and the first non-delay target tissue shape information according to the motion state information of the master control arm in the time period from the moment corresponding to the current operation scene image to the current moment, so as to obtain the second non-delay instrument end position information and the second non-delay target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm; as well as A zero-delay operation scene image is generated according to the second zero-delay instrument end posture information and the second zero-delay target tissue shape information.

6. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured as follows: Acquire the first posture information of the endoscope at the moment corresponding to the current operation scene image and the second posture information thereof at the current moment according to the motion state information of the master control arm in the time period from the moment corresponding to the current operation scene image to the current moment; Acquire, according to the first posture information and the second posture information, relative transformation matrix information of the endoscope from a moment corresponding to the current operation scene image to a current moment; as well as The first delay-free instrument end posture information and the first delay-free target tissue shape information are subjected to a field of view conversion according to the relative transformation matrix information to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the endoscopic field of view corresponding to the current motion state information of the main control arm.

7. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured to generate the delay-free operation scene image based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, and the pre-acquired preset instrument end wireframe model and preset target tissue wireframe model.

8. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured as follows: Performing image segmentation on the current operation scene image to obtain an instrument end segmentation image and a target tissue segmentation image; as well as The zero-delay operation scene image is generated according to the second zero-delay instrument end posture information, the second zero-delay target tissue shape information, the instrument end segmentation image and the target tissue segmentation image.

9. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured to generate the delay-free operation scene image using a pre-trained second diffusion model based on the current operation scene image, the second delay-free instrument end posture information and the second delay-free target tissue shape information.

10. The surgical robot-assisted system according to claim 4, characterized in that: It also includes an image processing unit configured to superimpose the zero-delay operation scene image with the current operation scene image.

11. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are implemented: Obtain the current motion state information of the main control arm; Acquire the current operation scene image; and The non-delayed instrument and tissue interaction state information is predicted based on the current motion state information and the current operation scene image.

12. The readable storage medium according to claim 11, characterized in that: When the computer program is executed by a processor, the following steps are also implemented: A zero-delay operation scene image is generated based on the zero-delay instrument and tissue interaction state information.

13. The readable storage medium according to claim 12, characterized in that: When the computer program is executed by a processor, the following steps are also implemented: The non-delayed operation scene image is superimposed on the current operation scene image.

14. A surgical robot, characterized in that: A surgical robot-assisted system comprising any one of claims 1 to 10 or a readable storage medium according to any one of claims 11 to 13.

Images