Surgical robot-assisted system, readable storage medium, and surgical robot

Abstract

The present invention provides a surgical robot assistance system, a readable storage medium, and a surgical robot. The system includes a motion state acquisition unit configured to acquire the current motion state information of a master control arm; an interaction state acquisition unit configured to acquire an image of the current operation scenario; and an interaction state prediction unit configured to predict the interaction state information between a delay-free instrument and tissue based on the current motion state information and the image of the current operation scenario. The present invention can predict the interaction state between the end of the instrument and tissue according to the current motion state information of the master control arm, thereby eliminating the influence brought by the delay to the operator.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and particularly to a surgical robot assistance system, a readable storage medium, and a surgical robot. Background Art

[0002] Current teleoperated surgical robots include an operator, a master end, a slave end, a communication module, and an image trolley. The operator issues commands by manipulating the master control arm, and these commands will be transmitted to the slave end through the communication module. After receiving the commands, the control system of the slave end will control the slave-end robot to reproduce the operations of the master end. At the same time, the slave-end robot will transmit the surgical images to the master end through the communication module. With the continuous development of technology, the application scope and functions of surgical robots are also constantly expanding. Currently, surgical robots can not only be applied to a variety of surgical fields (such as cardiac surgery, neurosurgery, urology, gynecology, etc.), but can also perform operations at a long distance. However, with the increase in the operating distance, latency has become a prominent problem that cannot be avoided. Latency will cause the slave-end robot to fail to respond to the doctor's commands in a timely manner, affecting the surgical precision. If the latency is too large, the doctor may make incorrect judgments about the interaction between the end of the instrument and the tissue, affecting the safety of the surgery.

[0003] Current solutions mainly include: using a high-speed network connection to improve the data transmission speed; or reducing the amount of data transmission through software; or improving the hardware performance of the surgical robot to reduce the processing latency and display latency. However, the latency that these solutions can reduce is limited. The short-distance latency is still dozens of milliseconds, and the long-distance latency can still reach hundreds of milliseconds. Therefore, it will still affect the doctor's operation.

[0004] It should be noted that the information disclosed in the background art of this invention is only intended to deepen the understanding of the general background technology of the present invention, and should not be regarded as an admission or any form of suggestion that this information constitutes the prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a surgical robot assistance system, a readable storage medium, and a surgical robot, which can predict the interaction state between the end of the instrument and the tissue according to the current motion state information of the master control arm, so as to eliminate the influence brought by latency to the operator.

[0006] To achieve the above purpose, the present invention provides a surgical robot assistance system, which includes: a motion state acquisition unit configured to acquire the current motion state information of the master control arm; an interaction state acquisition unit configured to acquire the current operation scene image; and an interaction state prediction unit configured to predict the interaction state information between the instrument without latency and the tissue according to the current motion state information and the current operation scene image.

[0007] Optionally, the interaction state prediction unit is configured to predict the interaction state information between the delay-free instrument and the tissue by using a pre-trained first diffusion model according to the current motion state information and the current operation scene image.

[0008] Optionally, the interaction state prediction unit is further configured to extract the key features of the instrument end and the key features of the target tissue from the current operation scene image, and input the extracted key features of the instrument end and the key features of the target tissue as additional information into the first diffusion model.

[0009] Optionally, the surgical robot assistance system provided by the present invention further includes an auxiliary display unit, which is configured to generate a delay-free operation scene image according to the interaction state information between the delay-free instrument and the tissue.

[0010] Optionally, the interaction state information between the delay-free instrument and the tissue includes the first delay-free instrument end pose information and the first delay-free target tissue shape information; the auxiliary display unit is configured to: perform a field-of-view conversion on the first delay-free instrument end pose information and the first delay-free target tissue shape information according to the motion state information of the master control arm during the period from the moment corresponding to the current operation scene image to the current moment, so as to obtain the second delay-free instrument end pose information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm; and generate a delay-free operation scene image according to the second delay-free instrument end pose information and the second delay-free target tissue shape information.

[0011] Optionally, the auxiliary display unit is configured to: obtain the first pose information of the endoscope at the moment corresponding to the current operation scene image and its second pose information at the current moment according to the motion state information of the master control arm during the period from the moment corresponding to the current operation scene image to the current moment; obtain the relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image to the current moment according to the first pose information and the second pose information; and perform a field-of-view conversion on the first delay-free instrument end pose information and the first delay-free target tissue shape information according to the relative transformation matrix information, so as to obtain the second delay-free instrument end pose information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm.

[0012] Optionally, the auxiliary display unit is configured to generate the delay-free operation scenario image according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, and a pre-acquired preset instrument end wireframe model and a preset target tissue wireframe model.

[0013] Optionally, the auxiliary display unit is configured to: perform image segmentation on the current operation scenario image to obtain an instrument end segmentation image and a target tissue segmentation image; and generate the delay-free operation scenario image according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the instrument end segmentation image, and the target tissue segmentation image.

[0014] Optionally, the auxiliary display unit is configured to generate the delay-free operation scenario image according to the current operation scenario image, the second delay-free instrument end pose information, and the second delay-free target tissue shape information by using a pre-trained second diffusion model.

[0015] Optionally, the surgical robot assistance system provided by the present invention further includes an image processing unit configured to superimpose the delay-free operation scenario image on the current operation scenario image.

[0016] To achieve the above object, the present invention provides a readable storage medium storing a computer program, which when executed by a processor, implements the following steps: obtaining the current motion state information of the main control arm; obtaining the current operation scenario image; and predicting the delay-free instrument-tissue interaction state information according to the current motion state information and the current operation scenario image.

[0017] Optionally, when the computer program is executed by the processor, the following steps are further implemented: generating a delay-free operation scenario image according to the delay-free instrument-tissue interaction state information.

[0018] Optionally, when the computer program is executed by the processor, the following steps are further implemented: superimposing the delay-free operation scenario image on the current operation scenario image.

[0019] To achieve the above object, the present invention further provides a surgical robot, which includes the surgical robot assistance system described in any one of the above or the readable storage medium described in any one of the above.

[0020] Compared with the prior art, the surgical robot assistance system, readable storage medium, and surgical robot provided by the present invention have the following beneficial effects: The surgical robot assistance system provided by the present invention can obtain the current motion state information of the master control arm through the motion state acquisition unit; can obtain the current operation scene image through the interaction state acquisition unit; and can predict the interaction state information between the non-delay instrument and the tissue according to the current motion state information and the current operation scene image, thereby eliminating the influence brought by the delay to the operator, enabling the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, further improving the accuracy and safety of the surgery, and at the same time avoiding the incoherence of the slave robot's motion and the waiting time of the operator caused by the delay, thereby improving the surgical operation efficiency.

[0021] Since the readable storage medium and surgical robot provided by the present invention belong to the same inventive concept as the surgical robot assistance system provided by the present invention, the readable storage medium and surgical robot provided by the present invention have at least all the beneficial effects of the surgical robot assistance system provided by the present invention. Specifically, reference can be made to the relevant descriptions of the beneficial effects of the surgical robot assistance system provided by the present invention in the above text. Therefore, the beneficial effects of the readable storage medium and surgical robot provided by the present invention will not be elaborated one by one here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the application scenario of the surgical robot.

[0023] Figure 2 It is a schematic diagram of the partial structure of the master control arm.

[0024] Figure 3 It is a schematic diagram of the control principle of the surgical robot.

[0025] Figure 4 It is a schematic diagram of the structure of the visual capture device.

[0026] Figure 5 It is a schematic diagram of the interaction state between the instrument and the tissue.

[0027] Figure 6 It is a schematic diagram of local surgical delay.

[0028] Figure 7 It is a schematic diagram of remote surgical delay.

[0029] Figure 8 It is a block diagram of the structure of the surgical robot assistance system provided by an embodiment of the present invention.

[0030] Figure 9 It is a schematic diagram of the delay of the interaction state between the instrument and the tissue.

[0031] Figure 10 Schematic diagram of generating a non-delay operation scenario image based on a wireframe model provided by an embodiment of the present invention.

[0032] Figure 11 Schematic diagram of generating a non-delay operation scenario image based on a segmented image provided by an embodiment of the present invention.

[0033] Figure 12 Schematic diagram of generating a non-delay operation scenario image based on a diffusion model provided by an embodiment of the present invention.

[0034] Figure 13 Schematic diagram of the image overlay effect provided by an embodiment of the present invention.

[0035] Figure 14 Flowchart of the steps that can be achieved by a readable storage medium provided by an embodiment of the present invention.

[0036] Among them, the reference numerals are explained as follows: doctor's trolley - 100; main body doctor's trolley - 100A; remote doctor's trolley - 100B; main control arm - 110; doctor's handle - 111; main end display - 120; main controller - 130; patient's trolley - 200; local patient's trolley - 200A; robotic arm - 210; surgical instrument - 220; instrument end - 221; visual capture device - 230; slave controller - 240; image trolley - 300; local image trolley - 300A; target tissue - 400; server - 500; motion state acquisition unit - 610; interaction state acquisition unit - 620; interaction state prediction unit - 630; auxiliary display unit - 640; image processing unit - 650; current operation scenario image - 11; actual operation scenario image - 12; non-delay operation scenario image - 13; preset instrument end wireframe model - 21; preset target tissue wireframe model - 22; instrument end segmented image - 31; target tissue segmented image - 32. Detailed implementation manners

[0037] The surgical robot-assisted system, readable storage medium, and surgical robot proposed by the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and are all drawn using non-precise scales, only for the convenience and clarity of assisting in explaining the purpose of the present invention. In order to make the purpose, features, and advantages of the present invention more obvious and understandable, please refer to the accompanying drawings. It should be noted that the structures, scales, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those skilled in this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship, or adjustment of the size, in the case of being the same or similar to the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0038] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article, or device comprising the element. The singular forms "a", "an", and "the" include plural objects, the term "or" is generally used in the sense of including "and / or", the term "several" is generally used in the sense of including "at least one", the term "at least two" is generally used in the sense of including "two or more", and in addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features.

[0039] In addition, in the description of this specification, the descriptions with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0040] For ease of understanding, first, a brief description of the research background of the surgical robot-assisted system, readable storage medium, and surgical robot provided by the present invention will be given. Please refer to Figure 1 , which is a schematic diagram of the application scenario of the surgical robot. As Figure 1 shown, the surgical robot includes a doctor's cart 100 (master end) and a patient's cart 200 (slave end). The doctor's cart 100 is equipped with a main control arm 110. The patient's cart 200 is equipped with at least one robotic arm 210. A surgical instrument 220 and a visual capture device 230 (such as an endoscope) can be respectively mounted on the robotic arm 210. The operator (doctor) performs teleoperation through the main control arm 110 on the doctor's cart 100 to perform minimally invasive surgical treatment on the patient on the operating table. Among them, the main control arm 110 and the robotic arm 210, as well as the surgical instrument 220 and the visual capture device 230 (such as an endoscope) mounted on the robotic arm 210, form a master-slave control relationship. Specifically, the robotic arm 210, the surgical instrument 220, and the visual capture device 230 (such as an endoscope) move following the movement of the main control arm 110 during the operation, that is, following the operation of the operator's hand.

[0041] Please continue to refer to Figure 2 , which is a partial structural schematic diagram of the main control arm 110. As Figure 2 shown, the main control arm 110 receives the hand movement state information of the operator through the doctor's handle 111 at the end as the motion control input of the surgical robot. Please continue to refer to Figure 1 , as Figure 1As shown, a main-end display 120 (doctor's observation window) is also provided on the doctor's trolley 100. The main-end display 120 can display the operation scene images collected by the visual capture device 230 (such as an endoscope). The operator controls the robotic arm 210, the surgical instrument 220, and the visual capture device 230 (such as an endoscope) to move through the main control arm 110 according to the operation scene images displayed on the main-end display 120. The surgical instrument 220 and the visual capture device 230 (such as an endoscope) can respectively enter the lesion location (the location where the target tissue 400 is located) through the wounds or natural openings on the patient's body.

[0042] Please continue to refer to Figure 1 , as Figure 1 shown, the surgical robot further includes an image trolley 300. The operation scene images collected by the visual capture device 230 (such as an endoscope) can be transmitted to the image trolley 300 for display.

[0043] Please continue to refer to Figure 3 , which is a schematic diagram of the control principle of the surgical robot. As Figure 3 shown, the doctor's trolley 100 includes a main controller 130 and a main-end display 120. The main controller 130 is used to sense the movements of the operator's hand and recognize the operator's operation intention. The main-end display 120 (doctor's observation window) is used to present the surgical images (operation scene images). The patient trolley 200 includes a slave controller 240 and a visual capture device 230 (such as an endoscope, as Figure 4 shown, which is a schematic diagram of the structure of the visual capture device 230). The slave controller 240 is used to receive the instructions from the main controller 130, and after processing, control the robotic arm 210 at the slave end to reproduce the main-end operation. The visual capture device 230 (such as an endoscope) collects the current surgical images (operation scene images), and outputs the current surgical images (operation scene images) to the main-end display 120 on the doctor's trolley 100 through the image trolley 300 for display.

[0044] Please continue to refer to Figure 5 , which is a schematic diagram of the interaction state between the instrument and the tissue. As Figure 5 shown, the interaction state information between the instrument and the tissue collected by the visual capture device 230 (such as an endoscope) includes the pose information of the instrument end 221 and the shape information of the target tissue 400.

[0045] Please continue to refer to Figure 6 , which is a schematic diagram of the local surgical delay. As Figure 6 shown, when the surgical robot performs local surgery, the delay includes: delay t 1 , the local doctor's trolley 100A outputs a control signal to the local patient trolley 200A to receive the signal; delay t2 , the local patient trolley 200A performs an action until the local vision capture device 230 (such as an endoscope) captures the current surgical image (operation scene image); with a time delay t 3 , the local image trolley 300A outputs the current surgical image (operation scene image) to the main monitor 120 (doctor's observation window) on the local doctor trolley 100A for display.

[0046] Please continue to refer to Figure 7 , which is a schematic diagram of the time delay in remote surgery. As Figure 7 shown, when the surgical robot performs remote surgery, the time delay includes: time delay t 1 , the remote doctor trolley 100B outputs a control signal to the server 500 to receive the signal; with a time delay t 2 , the server 500 outputs a control signal to the local patient trolley 200A to receive the signal; with a time delay t 3 , the local patient trolley 200A performs an action until the local vision capture device 230 (such as an endoscope) captures the current surgical image (operation scene image); with a time delay t 4 , the local image trolley 300A outputs the current surgical image (operation scene image) to the server 500 to receive the signal (a decoding process may be involved during this period); with a time delay t 5 , the server 500 outputs a signal to the main monitor 120 (doctor's observation window) on the remote doctor trolley 100B for imaging display.

[0047] It can be seen that the time delay during the operation of the surgical robot includes the following two categories: time delay A, which is the time delay from the instruction of the doctor trolley 100 to the response of the patient trolley 200. For local surgery, it is t 1 , and for remote surgery, it is t 1 + t 2 ; time delay B, which is the time delay from the response of the patient trolley 200 to the image output on the doctor trolley 100. For local surgery, it is t 2 + t 3 , and for remote surgery, it is t 3 + t 4 + t 5 .

[0048] Based on this, the core idea of the present invention is to provide a surgical robot assistance system, a readable storage medium, and a surgical robot, which can predict the interaction state between the end of the instrument 221 and the tissue according to the current motion state information of the master control arm 110, so as to eliminate the influence brought by the delay to the operator.

[0049] It should be noted that the surgical robot assistance system and the readable storage medium provided by the present invention can be set on the master end (doctor's cart 100) of the surgical robot. It should also be noted that, as can be understood by those skilled in the art, for the slave end, the "end" referred to in this article means the end close to the lesion; for the master end, the "end" referred to in this article means the end close to the operator.

[0050] To achieve the above idea, the present invention provides a surgical robot assistance system. Please refer to Figure 8 , which is the structural block diagram of the surgical robot assistance system provided by an embodiment of the present invention. As Figure 8 shown, the surgical robot assistance system provided by the present invention includes: a motion state acquisition unit 610 configured to acquire the current motion state information of the master control arm 110; an interaction state acquisition unit 620 configured to acquire the current operation scene image 11 (as Figure 9 shown); and an interaction state prediction unit 630 configured to predict the interaction state information between the instrument without delay and the tissue according to the current motion state information and the current operation scene image 11.

[0051] Thus, the present invention can accurately predict the interaction state information between the instrument without delay (the end of the instrument 221) and the tissue (the target tissue 400) by according to the current motion state information of the master control arm 110 and the current operation scene image 11 (including the current interaction state information between the instrument and the tissue), so as to eliminate the influence brought by the delay to the operator, enabling the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the surgery. At the same time, it can also avoid the discontinuous movement of the slave robot and the waiting time of the operator caused by the delay, thus improving the surgical operation efficiency.

[0052] It should be noted that, as can be understood by those skilled in the art, the "current operation scene image 11" referred to in this article means the operation scene image currently received by the doctor's cart 100 (corresponding to the motion state of the master control arm 110 at t k-A-B moment), compared with the current motion state of the master control arm 110 ( Pt k-A-B moment's motion state t k moment's motion state Pt kThere is a certain delay. Please continue to refer to Figure 9 , which is a schematic diagram of the delay in the interaction state between the instrument and the tissue. As Figure 9 shown, due to the existence of delay A and delay B, when the doctor's handle 111 of the main control arm 110 corresponds to t k the actual motion state at the moment Pt k , the patient trolley 200 corresponds to the motion state of the doctor's handle 111 (main control arm 110) at t k-A the moment Pt k-A , the surgical image (operation scene image) presented on the doctor's observation window (main end display 120) corresponds to the motion state of the doctor's handle 111 (main control arm 110) at t k-A-B the moment Pt k-A-B , the predicted interaction state between the instrument without delay and the tissue in the present invention means that the image displayed on the doctor's observation window (main end display 120) corresponds to the motion state of the doctor's handle 111 (main control arm 110) at t k-A-B the moment Pt k-A-B , and the predicted interaction state between the instrument and the tissue corresponding to the motion state of the doctor's handle 111 at t k the moment Pt k . It should be noted that, as can be understood by those skilled in the art, at t k the moment (i.e., the current moment), the surgical image (i.e., the current operation scene image 11) presented on the doctor's observation window (main end display 120) contains the interaction state information between the instrument and the tissue corresponding to the motion state of the doctor's handle 111 (main control arm 110) at t k-A-B the moment Pt k-A-B , and the actual operation scene image 12 collected by the visual capture device 230 (such as an endoscope) contains the interaction state information between the instrument and the tissue corresponding to the motion state of the doctor's handle 111 (main control arm 110) at t k-A the moment Pt k-A .

[0053] In some exemplary embodiments, the motion state acquisition unit 610 is configured to obtain the current motion state information of the master control arm 110 according to the current position information of each joint of the master control arm 110. Specifically, the current position information of each joint of the master control arm 110 can be collected by encoders installed on each joint of the master control arm 110.

[0054] Furthermore, the current motion state information of the master control arm 110 includes the current pose of the end of the master control arm 110 (the current pose of the doctor's handle 111).

[0055] Moreover, the current motion state information of the master control arm 110 further includes the current speed and current acceleration of the end of the master control arm 110 (the doctor's handle 111). Thus, by obtaining the current speed and current acceleration of the end of the master control arm 110 (the doctor's handle 111), the accuracy of the predicted interaction state information between the delay-free instrument and the tissue can be further ensured.

[0056] In some exemplary embodiments, the interaction state prediction unit 630 is configured to: preprocess the current motion state information and the current operation scene image 11, and predict the interaction state information between the delay-free instrument and the tissue based on the preprocessed current motion state information and the current operation scene image 11.

[0057] Thus, by preprocessing the current motion state information and the current operation scene image 11, noise can be effectively removed and the image quality can be improved, thereby effectively ensuring the accuracy of the predicted interaction state information between the delay-free instrument and the tissue.

[0058] Furthermore, since data such as the pose, speed, and acceleration of the end of the master control arm 110 may be affected by factors such as sensor noise, inconsistent acquisition frequencies, or slight tremors of the operator's hand, preprocessing steps such as denoising, data normalization, and time series interpolation can be performed on the current motion state information.

[0059] Specifically, filtering techniques (such as Kalman filtering or moving average filtering) can be used to smooth data such as pose, velocity, and acceleration in the current motion information of the main control arm 110 to reduce noise interference. By normalizing the data such as pose, velocity, and acceleration in the current motion information of the main control arm 110, these data can be normalized to a unified dimension (for example, normalizing the position coordinates to [-1, 1]) to match the input requirements of the first diffusion model in the following text, thereby ensuring the accuracy of the interaction state information between the delay-free instrument and the tissue predicted by the first diffusion model. Further, when there are situations where the time stamps of the motion state data acquisition of the main control arm 110 are discontinuous or the sampling rate is insufficient, interpolation methods such as linear interpolation or spline interpolation can be used to supplement the missing data points to ensure the continuity and integrity of the data.

[0060] Further, since the current operation scene image 11 is a real-time surgical image captured by a visual capture device 230 (such as an endoscope), it may be affected by factors such as light changes, tissue surface reflection, image resolution differences, or compression distortion during transmission. To improve the image quality and adapt to the input requirements of the first diffusion model in the following text, preprocessing steps such as denoising and enhancement, and image standardization can be taken for the current operation scene image 11.

[0061] Specifically, an image denoising algorithm (such as Gaussian blur or median filtering) can be used to remove the noise in the current operation scene image 11, and at the same time, the visibility of the image can be improved through contrast enhancement or histogram equalization to highlight the features of the instrument tip 221 and the target tissue 400. Then, the pixel values of each pixel point of the denoised and enhanced current operation scene image 11 are normalized to the range of [0, 1] or [-1, 1], and the resolution of the normalized current operation scene image 11 is adjusted to match the input size requirements of the first diffusion model in the following text (for example, adjusting the resolution of the normalized current operation scene image 11 to 256×256 pixels).

[0062] In some exemplary embodiments, the interaction state prediction unit 630 is configured to predict the interaction state information between the delay-free instrument and the tissue according to the current motion state information and the current operation scene image 11 by using a pre-trained first diffusion model. Since the diffusion model can theoretically approximate any complex data distribution, and the training process of the diffusion model is relatively stable and not prone to problems such as mode collapse. At the same time, by adjusting the parameters in the forward diffusion and reverse generation processes, fine control of the generated content can be achieved. Therefore, in the present invention, by using a pre-trained first diffusion model, more accurate interaction state information between the delay-free instrument and the tissue corresponding to the current motion state of the main control arm 110 (doctor's handle 111) can be predicted.

[0063] Specifically, the purpose of the first diffusion model is to generate the interaction state information between the instrument and the tissue corresponding to the motion state information of the master control arm 110 (doctor's handle 111) from the given operation scenario image corresponding to the motion state of the master control arm 110 (doctor's handle 111) at t k-A-B a moment Pt k-A-B and the additional motion state information of the master control arm 110 (doctor's handle 111) at t k a moment. t k Specifically, the purpose of the first diffusion model is to generate the interaction state information between the instrument and the tissue corresponding to the motion state information of the master control arm 110 (doctor's handle 111) from the given operation scenario image corresponding to the motion state of the master control arm 110 (doctor's handle 111) at

[0064] Furthermore, before training the first diffusion model, it is necessary to first collect the sample data required for training (including the motion state data of the master control arm 110 and the corresponding operation scenario images, etc.), and then process the sample data, including but not limited to denoising, filtering, interpolation and other operations, to improve the accuracy and stability of the sample data. Then, according to the processed sample data, preset model parameters and initial states, the pre-created controllable diffusion model is trained to obtain the trained first diffusion model.

[0065] Furthermore, the training of the first diffusion model relies on high-quality sample data, which need to reflect the true correspondence between the motion state (motion information) of the master control arm 110 and the interaction state information between the instrument and the tissue. Specifically, the sample data can be obtained in the following ways: 1) Acquisition of motion state data: By installing encoders or other motion sensors (such as inertial measurement units IMUs) on each joint of the master control arm, the motion state information of the end of the master control arm 110 (such as the doctor's handle 111) is recorded in real time, including pose, speed, and acceleration. Data acquisition can be carried out in simulated surgery or real surgery scenarios, covering various operation modes (such as translation, rotation, clamping, etc.) to ensure the diversity of samples; 2) Acquisition of operation scene images: Using a visual capture device 230 (such as an endoscope) to synchronously record the operation scene images corresponding to the motion state information of the master control arm 110. The operation scene images include the interaction state information between the instrument end 221 and the target tissue 400 (including the pose of the instrument end 221 and the deformation of the target tissue 400), and the acquisition frequency of the operation scene images is consistent with the acquisition frequency of the motion state data of the master control arm 110; 3) Data pairing and annotation: Align the motion state data of the master control arm 110 at each moment with the corresponding operation scene images by timestamp to form paired samples. To improve the training effect, the pose of the instrument end 221 and the shape of the target tissue 400 in the image can be extracted through manual annotation or automatic algorithms (such as image segmentation technology) as supervision signals. The sample data set needs to include local surgery and remote surgery scenarios to cover the interaction states under different latency conditions.

[0066] Furthermore, the first diffusion model adopts a generative training framework based on noise addition and denoising to predict the interaction state information of the instrument and tissue without latency from the current motion state information and the current operation scene image 11. The specific training process is as follows.

[0067] 1) Model initialization: Construct an initial controllable diffusion model, usually based on the U-Net architecture, combined with a conditional input module to receive motion state information and operation scene images. The model parameters (such as weights) are initialized to random values or pre-trained values (such as migrated from general image generation tasks).

[0068] 2) Forward diffusion process: Gradually add Gaussian noise to the target data (i.e., the interaction state information of the instrument and tissue without latency) in the paired samples to generate a series of noisy versions of the data. This process simulates the degradation of data from the real distribution to the noise distribution, usually setting a fixed number of steps (such as 1000 steps), and the noise intensity at each step is controlled by a predefined variance schedule.

[0069] 3) Reverse generation process: The training model gradually reconstructs the delay-free interaction state information from the noisy data through reverse denoising. The input conditions include the current motion state information and the current operation scene image 11. The model learns to predict the denoising result at each step under the given conditions. The loss function uses the mean squared error (MSE) to calculate the difference between the denoising result predicted by the model and the true delay-free interaction state.

[0070] 4) Optimization process: Use the gradient descent method (such as the Adam optimizer) to iteratively update the model parameters. During the training process, a learning rate scheduler (such as cosine annealing) can be introduced to balance the convergence speed and stability. Each iteration uses a batch of samples (for example, the batch size is 32), and the dataset is randomly shuffled to avoid overfitting.

[0071] Furthermore, in order to ensure that the first diffusion model can accurately predict the delay-free instrument-tissue interaction state information based on the current motion state information of the main control arm 110 and the current operation scene image 11, the training end condition of the first diffusion model needs to comprehensively measure the model performance and stability, specifically including the following criteria.

[0072] 1) Loss convergence: When the average loss (such as MSE) on the training set changes less than a preset threshold (such as 0.001) within a continuous number of iterations (such as 10 epochs), the model is considered to have converged.

[0073] 2) Validation set performance: Reserve a part of the sample data as the validation set and regularly evaluate the prediction accuracy of the model on the validation set (such as the pixel-level error or pose error of the interaction state information). When the validation set loss no longer decreases significantly or signs of overfitting appear (such as the validation set loss increases), stop the training.

[0074] 3) Generation quality assessment: Evaluate the similarity between the delay-free interaction state information generated by the model and the real data through manual inspection or automatic metrics (such as structural similarity SSIM, peak signal-to-noise ratio PSNR). When the generated result meets the accuracy requirements of surgical applications (such as the pose error is less than 1 mm and the tissue shape error is less than 5%), the training ends.

[0075] 4) Maximum iteration limit: To avoid infinite training, a maximum number of iterations (such as 500 epochs) or training duration (such as 72 hours) can be set. If the upper limit is reached and the above conditions are not fully met, select the model version with the best performance on the validation set as the final result.

[0076] In some exemplary embodiments, the interaction state prediction unit 630 is further configured to extract the key features of the instrument end and the key features of the target tissue from the current operation scene image 11, and input the extracted key features of the instrument end and the key features of the target tissue as additional information into the first diffusion model.

[0077] Thus, by inputting the extracted key features of the instrument end and the key features of the target tissue as additional information into the first diffusion model, the accuracy of the interaction state information between the delay-free instrument and the tissue predicted by the first diffusion model can be further improved.

[0078] Specifically, the key features of the instrument end and the key features of the target tissue can be extracted from the current operation scene image 11 by an edge detection algorithm (such as the Canny operator) or a pre-trained convolutional neural network.

[0079] Please continue to refer to Figure 8 , such as Figure 8 shown, in some exemplary embodiments, the surgical robot-assisted system provided by the present invention further includes an auxiliary display unit 640, configured to generate a delay-free operation scene image 13 according to the interaction state information between the delay-free instrument and the tissue (such as Figure 10 shown). Thus, by generating a delay-free operation scene image 13 according to the interaction state information between the delay-free instrument and the tissue, the delay-free interaction state between the instrument and the tissue corresponding to the current motion state of the master control arm 110 (at t k the motion state at time Pt k ) can be more intuitively displayed to the operator in the form of an image.

[0080] In some exemplary embodiments, the interaction state information between the delay-free instrument and the tissue includes the first delay-free instrument end pose information and the first delay-free target tissue shape information. The auxiliary display unit 640 is configured to: perform a field of view conversion on the first delay-free instrument end pose information and the first delay-free target tissue shape information according to the motion state information of the master control arm 110 during the period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain the second delay-free instrument end pose information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110; and generate a delay-free operation scene image 13 according to the second delay-free instrument end pose information and the second delay-free target tissue shape information.

[0081] Since the first delay-free instrument end pose information and the first delay-free target tissue shape information are based on the motion state of the master control arm 110 (doctor's handle 111) at t k moment Pt k and the interaction state information between the instrument and the tissue corresponding to the motion state of the master control arm 110 (doctor's handle 111) at t k-A-B moment Pt k-A-B are predicted, and during the period from t k-A-B moment to t k moment, the motion state of the master control arm 110 (doctor's handle 111) may cause the pose of the endoscope to change, resulting in a change in the field of view of the endoscope. Therefore, by performing a field-of-view conversion on the first delay-free instrument end pose information and the first delay-free target tissue shape information according to the motion state information of the master control arm 110 from the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) to the current moment (i.e., t k moment), the second delay-free instrument end pose information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110 (i.e., t k the motion state at Pt k ) can be obtained, thereby further eliminating the influence of the delay on the operator and further improving the accuracy and safety of the surgery.

[0082] Specifically, based on the motion state information of the master control arm 110 (doctor's handle 111) from t k-A-B moment to t k moment, through geometric transformation (such as rigid body transformation), the t k-A-B endoscope field of view at t k-A-B moment (the endoscope field of view corresponding to the motion state of the doctor's handle 111 at Pt k-A-B ) of the first delay-free instrument end pose information and the first delay-free target tissue shape information is converted to the t k endoscope field of view at tk Motion state at a moment Pt k The second non-delayed instrument end pose information and the second non-delayed target tissue shape information under the corresponding endoscopic view).

[0083] In some exemplary embodiments, the auxiliary display unit 640 is configured to: obtain the first pose information of the endoscope at the moment corresponding to the current operation scene image 11 and its second pose information at the current moment according to the motion state information of the main control arm 110 during the period from the moment corresponding to the current operation scene image 11 to the current moment; obtain the relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image 11 to the current moment according to the first pose information and the second pose information; and perform a view transformation on the first non-delayed instrument end pose information and the first non-delayed target tissue shape information according to the relative transformation matrix information to obtain the second non-delayed instrument end pose information and the second non-delayed target tissue shape information under the endoscopic view corresponding to the current motion state information of the main control arm 110.

[0084] Specifically, it can be based on the motion state information of the main control arm 110 at the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) to the current moment (i.e., t k moment) to calculate the pose change of the endoscope during this period. The motion state of the main control arm 110 is usually collected by joint encoders, including the three-dimensional position coordinates (x, y, z) and the attitude (such as rotation represented by Euler angles or quaternions) of the end of the main control arm 110 (doctor's handle 111). Through forward kinematic calculation, the poses of the endoscope at the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) and the current moment (i.e., t k moment) can be obtained, which are respectively denoted as and , and each pose can be represented by a 4×4 homogeneous transformation matrix, as shown specifically below:

[0085]

[0086] Among them, T represents the pose, R is a 3×3 rotation matrix, t is a 3×1 translation vector.

[0087] Further, the relative transformation matrix of the endoscope from the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) to the current moment (i.e., t k moment) can be calculated according to the following formula T rel :

[0088]

[0089] This relative transformation matrix T rel describes the rigid body motion of the endoscope from the moment corresponding to the current operation scene image 11 (i.e., t k-A-B moment) to the current moment (i.e., t k moment), including rotation and translation components.

[0090] In some exemplary embodiments, the auxiliary display unit 640 is configured to generate the delay-free operation scene image 13 according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the preset instrument end wireframe model 21 (as Figure 10 shown) and the preset target tissue wireframe model 22 (as Figure 10 shown). Thus, by generating the delay-free operation scene image 13 according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the preset instrument end wireframe model 21 and the preset target tissue wireframe model 22, it is convenient for the operator to more intuitively view the delay-free instrument-tissue interaction state corresponding to the current motion state of the main control arm 110 (at the t k moment's motion state Pt k ).

[0091] Please continue to refer to Figure 10 , which is a schematic diagram of generating a delay-free operation scene image based on a wireframe model provided by an embodiment of the present invention. As Figure 10 shown, the delay-free operation scene image 13 generated according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the preset instrument end wireframe model 21 and the preset target tissue wireframe model 22 can present the surface contours of the instrument end 221 and the target tissue 400 in the form of lines, only showing the edges and vertices of the instrument end 221 and the target tissue 400, without showing information such as the surface color and texture.

[0092] In some other exemplary embodiments, the auxiliary display unit 640 is configured to: perform image segmentation on the current operation scene image 11 to obtain an instrument end segmentation image 31 (as shown in Figure 11 ), and a target tissue segmentation image 32 (as shown in Figure 11 ); and generate the delay-free operation scene image 13 according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the instrument end segmentation image 31, and the target tissue segmentation image 32. Thus, by performing image segmentation on the current operation scene image 11, the structural features (such as color, texture, brightness, etc.) of the instrument end 221 and the target tissue 400 can be separated, so that the delay-free operation scene image 13 generated based on the second delay-free instrument end pose information, the second delay-free target tissue shape information, the instrument end segmentation image 31, and the target tissue segmentation image 32 can have the structural features (such as color, texture, brightness, etc.) of the real instrument end 221 and the target tissue 400, and further, the generated delay-free operation scene image 13 can be made more real.

[0093] Please refer to Figure 11 , which is a schematic diagram of generating a delay-free operation scene image based on a segmentation image provided by an embodiment of the present invention. As shown in Figure 11 , the delay-free operation scene image 13 generated based on the second delay-free instrument end pose information, the second delay-free target tissue shape information, the instrument end segmentation image 31, and the target tissue segmentation image 32 can not only display the edges and vertices of the instrument end 221 and the target tissue 400, but also display information such as the color and texture of their surfaces.

[0094] Specifically, the auxiliary display unit 640 is further configured to: preprocess the current operation scene image 11 before performing image segmentation on the current operation scene image 11. Thus, by first preprocessing the current operation scene image 11 (including but not limited to converting the current operation scene image 11 into a grayscale image or a binary image, denoising, etc.), and then performing image segmentation on the preprocessed current operation scene image 11, the accuracy of the image segmentation result can be effectively guaranteed.

[0095] Further, the auxiliary display unit 640 is configured to perform image segmentation on the current operation scenario image 11 (the preprocessed current operation scenario image 11) through the following process: performing edge detection on the current operation scenario image 11 (the preprocessed current operation scenario image 11) by using an edge detection algorithm to obtain an edge image; and performing extraction of the contours of the instrument end 221 and the target tissue 400 on the edge image by using a contour extraction algorithm to obtain an instrument end segmentation image 31 and a target tissue segmentation image 32.

[0096] Thus, by performing edge detection on the current operation scenario image 11 (the preprocessed current operation scenario image 11) by using an edge detection algorithm, the boundaries in the image can be detected, facilitating subsequent identification of the contours of the instrument end 221 and the target tissue 400; by using a contour extraction algorithm, different contours can be identified and extracted from the image after edge detection (the edge image), and by analyzing the size, shape, and position of the contours, it can be distinguished which are the instrument ends 221 and which are the soft tissues (the target tissue 400).

[0097] It should be noted that, as can be understood by those skilled in the art, in some other embodiments, the auxiliary display unit 640 is configured to: perform semantic segmentation on the current operation scenario image 11 (the preprocessed current operation scenario image 11) by using a pre-trained image segmentation model to obtain an instrument end segmentation image 31 and a target tissue segmentation image 32. Thus, by performing semantic segmentation on the current operation scenario image 11 (the preprocessed current operation scenario image 11) by using a pre-trained image segmentation model (deep learning models such as U-Net, Mask R-CNN, etc.), the accuracy of the segmented instrument end segmentation image 31 and target tissue segmentation image 32 can be effectively guaranteed.

[0098] In still some exemplary embodiments, the auxiliary display unit 640 is configured to generate the delay-free operation scenario image 13 by using a pre-trained second diffusion model according to the current operation scenario image 11, the second delay-free instrument end pose information, and the second delay-free target tissue shape information. Since the diffusion model can generate more real and delicate images through a fine noise addition and removal process (as Figure 12 shown, which is a schematic diagram of generating a delay-free operation scenario image based on the diffusion model provided by an embodiment of the present invention), compared with traditional generation models, the diffusion model shows higher clarity and fewer artifacts in image generation, and the quality of the output image is higher.

[0099] Please continue to refer to Figure 8 as Figure 8As shown, in some exemplary embodiments, the surgical robot-assisted system provided by the present invention further includes an image processing unit 650 configured to superimpose the non-delayed operation scene image 13 with the current operation scene image 11. t k The motion state P at the current moment t k The corresponding non-delay operation scene image 13 is superimposed on the current operation scene image 11, and the non-delay operation scene image 13 and the current operation scene image 11 can be superimposed into a picture to be displayed on the same interface (such as Figure 13 As shown, it is a schematic diagram of the image superposition effect provided by one embodiment of the present invention), which makes it easier for the operator to observe.

[0100] The present invention also provides a readable storage medium, wherein the readable storage medium stores a computer program. Figure 14 , which is a flow chart of the steps that can be implemented by a readable storage medium provided by an embodiment of the present invention. Figure 14 As shown, when the computer program is executed by the processor, the following steps can be implemented: step S100, obtaining the current motion state information of the main control arm 110; step S200, obtaining the current operation scene image 11; step S300, predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11.

[0101] Therefore, the present invention can accurately predict the interaction state information of the instrument and tissue without delay based on the current motion state information of the master control arm 110 and the current operation scene image 11, thereby eliminating the impact of the delay on the operator, allowing the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, thereby improving the accuracy and safety of the operation, and at the same time avoiding the discontinuous movement of the slave robot caused by the delay and the waiting time of the operator, thereby improving the efficiency of the surgical operation.

[0102] In some exemplary embodiments, predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11 includes: predicting the delay-free instrument and tissue interaction state information based on the current motion state information and the current operation scene image 11 using a pre-trained first diffusion model.

[0103] In some exemplary embodiments, when the computer program is executed by a processor, the following steps may be implemented: generating a zero-delay operation scene image 13 based on the zero-delay instrument and tissue interaction state information.

[0104] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are further implemented: extracting key features of the instrument end and key features of the target tissue from the current operation scene image 11, and inputting the extracted key features of the instrument end and the key features of the target tissue as additional information into the first diffusion model.

[0105] In some exemplary embodiments, the non-delay instrument-tissue interaction state information includes first non-delay instrument end pose information and first non-delay target tissue shape information.

[0106] Generating the non-delay operation scene image 13 according to the non-delay instrument-tissue interaction state information includes: performing a field of view transformation on the first non-delay instrument end pose information and the first non-delay target tissue shape information according to the motion state information of the master control arm 110 during the period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain second non-delay instrument end pose information and second non-delay target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110; and generating the non-delay operation scene image 13 according to the second non-delay instrument end pose information and the second non-delay target tissue shape information.

[0107] In some exemplary embodiments, performing a field of view transformation on the first non-delay instrument end pose information and the first non-delay target tissue shape information according to the motion state information of the master control arm 110 during the period from the moment corresponding to the current operation scene image 11 to the current moment, so as to obtain second non-delay instrument end pose information and second non-delay target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110 includes: obtaining first pose information of the endoscope at the moment corresponding to the current operation scene image 11 and its second pose information at the current moment according to the motion state information of the master control arm 110 during the period from the moment corresponding to the current operation scene image 11 to the current moment; obtaining relative transformation matrix information of the endoscope from the moment corresponding to the current operation scene image 11 to the current moment according to the first pose information and the second pose information; and performing a field of view transformation on the first non-delay instrument end pose information and the first non-delay target tissue shape information according to the relative transformation matrix information, so as to obtain second non-delay instrument end pose information and second non-delay target tissue shape information under the endoscope field of view corresponding to the current motion state information of the master control arm 110.

[0108] In some exemplary embodiments, generating the delay-free operation scenario image 13 according to the second delay-free instrument end pose information and the second delay-free target tissue shape information includes: generating the delay-free operation scenario image 13 according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, and a pre-acquired preset instrument end wireframe model 21 and a preset target tissue wireframe model 22.

[0109] In some other exemplary embodiments, generating the delay-free operation scenario image 13 according to the second delay-free instrument end pose information and the second delay-free target tissue shape information includes: performing image segmentation on the current operation scenario image 11 to obtain an instrument end segmentation image 31 and a target tissue segmentation image 32; and generating the delay-free operation scenario image 13 according to the second delay-free instrument end pose information, the second delay-free target tissue shape information, the instrument end segmentation image 31, and the target tissue segmentation image 32.

[0110] In some further exemplary embodiments, generating the delay-free operation scenario image 13 according to the second delay-free instrument end pose information and the second delay-free target tissue shape information includes: generating the delay-free operation scenario image 13 according to the current operation scenario image 11, the second delay-free instrument end pose information, and the second delay-free target tissue shape information by using a pre-trained second diffusion model.

[0111] In some exemplary embodiments, when the computer program is executed by a processor, the following steps may be implemented: superimposing the delay-free operation scenario image 13 and the current operation scenario image 11.

[0112] Furthermore, the readable storage medium provided by the present invention may adopt any combination of one or more computer-readable media. The readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive listing) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0113] Furthermore, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber, etc., or any suitable combination of the above.

[0114] To achieve the above object, the present invention also provides a surgical robot, which includes the surgical robot assistance system or the readable storage medium described above. Since the surgical robot provided by the present invention includes the surgical robot assistance system or the readable storage medium described above, the surgical robot provided by the present invention at least includes all the beneficial effects of the surgical robot assistance system or the readable storage medium described above. For specific details, reference may be made to the relevant descriptions of the beneficial effects of the surgical robot assistance system or the readable storage medium provided by the present invention in the above text. Therefore, the beneficial effects of the surgical robot provided by the present invention will not be elaborated one by one here.

[0115] In summary, compared with the prior art, the surgical robot assistance system, readable storage medium, and surgical robot provided by the present invention have the following beneficial effects: By according to the current motion state information of the master control arm 110 and the current operation scene image 11, the present invention can accurately predict the interaction state information between the delay-free instrument and the tissue, thereby eliminating the influence brought by the delay to the operator, enabling the operator to control the slave robot to perform high-precision operations such as fine cutting and suturing, and further improving the accuracy and safety of the surgery. At the same time, it can also avoid the discontinuous movement of the slave robot caused by the delay and the waiting time of the operator, thereby improving the surgical operation efficiency.

[0116] It should be noted that computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0117] It should be noted that the devices and methods disclosed in the embodiments herein can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments herein. In this regard, each block in the flowchart or block diagram can represent a module, program, or part of the code, and the module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions. Additionally, the functional modules in each embodiment herein can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0118] It should also be noted that the above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure fall within the protection scope of the present invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A surgical robot-assisted system, characterized in that: include: A motion state acquisition unit configured to acquire current motion state information of the master control arm; An interaction state acquisition unit, configured to acquire a current operation scene image; as well as An interaction state prediction unit, configured to predict the interaction state information between the non-delayed instrument and the tissue according to the current motion state information and the current operation scene image; The non-delay instrument and tissue interaction state information includes first non-delay instrument end position information and first non-delay target tissue shape information; The surgical robot-assisted system also includes an auxiliary display unit, which is configured to: perform a field of view conversion on the first delay-free instrument end posture information and the first delay-free target tissue shape information according to the motion state information of the main control arm in the time period from the moment corresponding to the current operation scene image to the current moment, so as to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the endoscopic field of view corresponding to the current motion state information of the main control arm.

2. The surgical robot-assisted system according to claim 1, characterized in that: The interaction state prediction unit is configured to predict the interaction state information between the non-delay instrument and the tissue using a pre-trained first diffusion model according to the current motion state information and the current operation scene image.

3. The surgical robot-assisted system according to claim 2, characterized in that: The interaction state prediction unit is also configured to extract instrument end key features and target tissue key features from the current operation scene image, and input the extracted instrument end key features and target tissue key features into the first diffusion model as additional information.

4. The surgical robot-assisted system according to claim 1, characterized in that: The auxiliary display unit is also configured to generate a zero-delay operation scene image based on the zero-delay instrument and tissue interaction state information.

5. The surgical robot-assisted system according to claim 4, characterized in that: The auxiliary display unit is also configured to generate a zero-delay operation scene image based on the second zero-delay instrument end posture information and the second zero-delay target tissue shape information.

6. The surgical robot-assisted system according to claim 1, characterized in that: The auxiliary display unit is configured as follows: Acquire the first posture information of the endoscope at the moment corresponding to the current operation scene image and the second posture information thereof at the current moment according to the motion state information of the master control arm in the time period from the moment corresponding to the current operation scene image to the current moment; Acquire, according to the first posture information and the second posture information, relative transformation matrix information of the endoscope from a moment corresponding to the current operation scene image to a current moment; as well as The first delay-free instrument end posture information and the first delay-free target tissue shape information are subjected to a field of view conversion according to the relative transformation matrix information to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the endoscopic field of view corresponding to the current motion state information of the main control arm.

7. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured to generate the delay-free operation scene image based on the second delay-free instrument end posture information, the second delay-free target tissue shape information, and the pre-acquired preset instrument end wireframe model and preset target tissue wireframe model.

8. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured as follows: Performing image segmentation on the current operation scene image to obtain an instrument end segmentation image and a target tissue segmentation image; as well as The zero-delay operation scene image is generated according to the second zero-delay instrument end posture information, the second zero-delay target tissue shape information, the instrument end segmentation image and the target tissue segmentation image.

9. The surgical robot-assisted system according to claim 5, characterized in that: The auxiliary display unit is configured to generate the zero-delay operation scene image using a pre-trained second diffusion model based on the current operation scene image, the second zero-delay instrument end posture information and the second zero-delay target tissue shape information.

10. The surgical robot-assisted system according to claim 4, characterized in that: It also includes an image processing unit configured to superimpose the zero-delay operation scene image with the current operation scene image.

11. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are implemented: Obtain the current motion state information of the main control arm; Acquire the current operation scene image; and Predicting the interaction state information between the non-delayed instrument and the tissue according to the current motion state information and the current operation scene image; The non-delay instrument and tissue interaction state information includes first non-delay instrument end position information and first non-delay target tissue shape information; When the computer program is executed by the processor, the following steps are also implemented: the first delay-free instrument end posture information and the first delay-free target tissue shape information are converted into a field of view based on the motion state information of the main control arm in the time period from the moment corresponding to the current operation scene image to the current moment, so as to obtain the second delay-free instrument end posture information and the second delay-free target tissue shape information under the endoscope field of view corresponding to the current motion state information of the main control arm.

12. The readable storage medium according to claim 11, characterized in that: When the computer program is executed by a processor, the following steps are also implemented: A zero-delay operation scene image is generated based on the zero-delay instrument and tissue interaction state information.

13. The readable storage medium according to claim 12, characterized in that: When the computer program is executed by a processor, the following steps are also implemented: The non-delayed operation scene image is superimposed on the current operation scene image.

14. A surgical robot, characterized in that: A surgical robot-assisted system comprising any one of claims 1 to 10 or a readable storage medium according to any one of claims 11 to 13.

Images