Remote surgical robot and corresponding stereoscopic visual effect generation method and medium

Abstract

The present application discloses a remote surgical robot and a corresponding method and medium for generating a stereoscopic visual effect. The remote surgical robot includes a patient surgical platform, an image acquisition device provided with a first shooting unit and a second shooting unit, a remote doctor console and an image display device provided with a first display unit and a second display unit; the patient surgical platform and the remote doctor console are in communication connection; the method includes obtaining first device information of the patient surgical platform, second device information of the remote doctor console, and user interaction data; according to the first device information and the second device information, the patient surgical platform or the remote doctor console is selected, and based on the user interaction data, an overlay operation is performed in the left view image and the right view image respectively, to obtain a left view overlay image and a right view overlay image, and display them in the first display unit and the second display unit respectively. The present invention realizes a stereoscopic visual effect of displaying user interaction graphics in the surgical field of view.

Description

Technical Field

[0001] The present invention belongs to the field of medical devices, and in particular relates to a remote surgical robot and a corresponding method and medium for generating a stereoscopic visual effect. Background Art

[0002] In a remote surgical robot, the imaging system is mainly divided into the monitor on the master end (remote doctor console) and the endoscope on the slave end (patient surgical platform). The monitor on the master end and the endoscope on the slave end of the imaging system need to be highly coordinated to ensure that the doctor can observe the patient's surgical area captured by the endoscope on the slave end in real time and accurately on the monitor on the master end, and perform the surgical operation.

[0003] The master hand is the control center for doctors to perform surgical operations. The imaging system provides real-time visual feedback. The display has parameters of single-channel / dual-channel, two-dimensional (2D) / three-dimensional (3D), 2K resolution / 4K resolution, usually including single-channel 3D2K display and dual-channel 2D4K display. The slave hand is where the surgical robot performs actual operations. The imaging system is responsible for capturing real-time images of the surgical area and transmitting them to the master hand. The endoscope also has parameters of single-channel / dual-channel / triple-channel, two-dimensional (2D) / three-dimensional (3D), 2K resolution / 4K resolution, usually including dual-channel 2D4K endoscope and dual-channel 2D2K endoscope.

[0004] Currently, dual-channel 2D endoscopes and dual-channel 2D displays / single-channel 3D displays usually display 3D stereoscopic endoscope images (surgical images) + 2D flat user interaction graphics, such as UI icons (User Interface). User interaction graphics have visual effect deviations in display, which may affect doctors' UI interaction operations and observations during surgery, reduce the immersive experience of surgical operations, and thus affect surgical operations. Summary of the invention

[0005] In order to solve the problem that the existing remote surgical robot has visual effect deviation in the surgical field, which may affect the doctor's UI interactive operation and observation during the operation, reduce the immersive experience of the surgical operation and thus affect the surgical operation, the present invention proposes the following technical solutions:

[0006] A first aspect of the present invention provides a method for generating a stereoscopic visual effect applied to a remote surgical robot, comprising: a patient surgical platform and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit and a second shooting unit, the first shooting unit is used to shoot a left-view image, and the second shooting unit is used to shoot a right-view image; a remote doctor console and an image display device connected to each other, wherein the image display device is provided with a first display unit and a second display unit; wherein the patient surgical platform and the remote doctor console are communicatively connected; the method for generating a stereoscopic visual effect comprises: obtaining first device information of the patient surgical platform, second device information of the remote doctor console, and user interaction data; selecting at least one end of the patient surgical platform and the remote doctor console according to the first device information and the second device information, and performing an overlay operation in the left-view image and the right-view image respectively based on the user interaction data to obtain a left-view overlay image and a right-view overlay image; and displaying the left-view overlay image and the right-view overlay image in the first display unit and the second display unit respectively.

[0007] Preferably, the remote surgical robot further comprises: a first remote host and a second remote host, wherein the first remote host is connected to the patient surgical platform, and the second remote host is connected to the remote doctor console; wherein the patient surgical platform and the remote doctor console are communicatively connected via the first remote host and the second remote host, and the first remote host comprises a first remote port and a second remote port; according to the first device information and the second device information, at least one end of the patient surgical platform and the remote doctor console is selected, and based on the user interaction data, an overlay operation is performed in the left view image and the right view image respectively to obtain a left view overlay image and a right view overlay image, including: if the first device information and the second device information If the second device information satisfies the preset first condition, the patient surgical platform transmits the left-view image, the right-view image and the user interaction data to the remote doctor console through the first remote port; the remote doctor console performs an overlay operation on the left-view image and the right-view image based on the user interaction data to obtain a left-view overlay image and a right-view overlay image; if the first device information and the second device information meet the preset second condition, the patient surgical platform performs an overlay operation on the left-view image and the right-view image based on the user interaction data to obtain a left-view overlay image and a right-view overlay image, and transmits the left-view overlay image and the right-view overlay image to the remote doctor console through the second remote port.

[0008] Preferably, based on the user interaction data, an overlay operation is performed in the left-view image and the right-view image respectively to obtain a left-view overlay image and a right-view overlay image, including: generating a user interaction graphic based on the user interaction data, and extracting first assembly information in the first device information and second assembly information in the second device information; calculating, according to the first assembly information, the parallax of the user interaction graphic when it is superimposed on the left-view image and the right-view image respectively; correcting the parallax according to the second assembly information, and determining a first overlay position of the user interaction graphic in the left-view image and a second overlay position of the user interaction graphic on the right-view image based on the corrected parallax; superimposing the user interaction graphic on the first overlay position of the left-view image to obtain a left-view overlay image, and superimposing the user interaction graphic on the second overlay position of the right-view image to obtain a right-view overlay image.

[0009] Preferably, before obtaining the first device information of the patient surgical platform, the second device information of the remote doctor console, and the user interaction data, it also includes: in response to the interactive operation between the remote doctor console and the patient surgical platform, the patient surgical platform determines the interactive state corresponding to the interactive operation, and the user interaction data includes the interactive state; wherein the user interaction graphic includes a first state icon corresponding to the interactive state, and the interactive state includes: the connection state, pairing state and network state between the remote doctor console and the patient surgical platform.

[0010] Preferably, the patient surgical platform also includes a graphical user interface, which includes at least one trigger control; before obtaining the first device information of the patient surgical platform, the second device information of the remote doctor console, and the user interaction data, it also includes: in response to a touch operation on the trigger control, the patient surgical platform determines a touch state corresponding to the touch operation, and the user interaction data includes the touch state; wherein the user interaction graphic includes a second state icon corresponding to the touch state.

[0011] Preferably, the graphical user interface also includes a marked area, and the marked area includes at least a partial area of ​​the left-view image or the right-view image; in response to a sliding operation on the marked area, a sliding trajectory of the sliding operation on the left-view image and the right-view image is determined, and the user interaction data includes the sliding trajectory; wherein the user interaction graphic includes a trajectory graphic corresponding to the sliding trajectory.

[0012] Preferably, the remote surgical robot includes a first remote doctor console and a left-view image display device connected to each other, and a second remote doctor console and a right-view image display device connected to each other; the patient surgical platform includes at least a first mechanical arm and a second mechanical arm, wherein the first remote doctor console has control authority over the first mechanical arm, and the second remote doctor console has control authority over the second mechanical arm; the user interaction graphic includes a first user interaction graphic and a second user interaction graphic, the first user interaction graphic is displayed on the left-view image display device, and the second user interaction graphic is displayed on the right-view image display device; before acquiring the first device information of the patient surgical platform, the second device information of the remote doctor console, and the user interaction data, it also includes: in response to a first control operation of the first remote doctor console on the first mechanical arm, the patient surgical platform determines a first control state corresponding to the first control operation, the user interaction data includes the first control state, and the first user interaction image includes a third state icon corresponding to the first control state; or, in response to a second control operation of the second remote doctor console on the second mechanical arm, the patient surgical platform determines a second control state corresponding to the second control operation, the user interaction data includes the second control state, and the second user interaction image includes a fourth state icon corresponding to the second control state.

[0013] Preferably, the user interaction graphic includes: a first part that overlaps with the left-view image and the right-view image when superimposed, and a second part that does not overlap, wherein the first part includes a track graphic and the second part includes a status icon.

[0014] Preferably, the image acquisition device is connected to the first remote port and the second remote port, and the graphical user interface is connected to the second remote port, and is at least used to acquire and display at least one of the left view image and the right view image from the patient surgical platform; the stereoscopic visual effect generation method also includes: in response to a sliding operation in the left view image and the right view image, superimposing the trajectory graphic on the left view image and the right view image through the graphical user interface, generating a left-view superimposed image and a right-view superimposed image respectively, and outputting them to the second remote port; in response to the generation of user interaction data corresponding to the status icon, the patient surgical platform outputs the left view image, the right view image and the user interaction data corresponding to the status icon to the first remote port; or, the patient surgical platform superimposes the status icon on the left view image and the right view image, generating a left-view superimposed image and a right-view superimposed image respectively, and outputting them to the second remote port.

[0015] A second aspect of the present invention provides a remote surgical robot, comprising: a patient surgical platform and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit and a second shooting unit, the first shooting unit is used to shoot a left-view image, and the second shooting unit is used to shoot a right-view image; a remote doctor console and an image display device connected to each other, wherein the image display device is provided with a first display unit and a second display unit; wherein the patient surgical platform and the remote doctor console are communicatively connected; a processor is configured to: obtain first device information of the patient surgical platform, second device information of the remote doctor console, and user interaction data; select at least one end of the patient surgical platform and the remote doctor console according to the first device information and the second device information, and perform an overlay operation in the left-view image and the right-view image based on the user interaction data to obtain a left-view overlay image and a right-view overlay image; and display the left-view overlay image and the right-view overlay image in the first display unit and the second display unit, respectively.

[0016] A third aspect of the present invention provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned method for generating a stereoscopic visual effect applied to a remote surgical robot are implemented.

[0017] The beneficial effect of the present invention is that after the image acquisition device of the patient's surgical platform, such as a binocular endoscope, acquires the left and right view images (left view image and right view image, binocular endoscope image), one or both ends are selected according to the hardware design of the patient's surgical platform and the remote doctor console to superimpose the interactive graphics of the user interactive data on the left and right view images, so that the dual-channel two-dimensional display of the remote doctor console can simultaneously display the endoscope image (surgical image) with three-dimensional stereoscopic visual effect + the user interactive graphics with three-dimensional stereoscopic visual effect. At the same time, the remote doctor console adapted to different hardware designs can realize the stereoscopic visual effect of the user interactive graphics without changing the hardware design of the original remote doctor console. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of an embodiment of a patient surgery platform of the present invention;

[0019] Figure 2 It is a structural schematic diagram of an embodiment of a manipulation component in a surgical robot of the present invention;

[0020] Figure 3 is a structural schematic diagram of another embodiment of the patient surgery platform of the present invention;

[0021] Figure 4It is a structural schematic diagram of an embodiment of a remote doctor console of the present invention;

[0022] Figure 5 is a schematic diagram of an image interconnection system of an embodiment of a remote surgical robot of the present invention;

[0023] Figure 6 It is a flowchart of an embodiment of the image processing method of the present invention;

[0024] Figure 7 It is a flowchart of an embodiment of the image processing method of the present invention;

[0025] Figure 8 It is a data flow diagram of an embodiment of the image processing method of the present invention;

[0026] Fig. 9 It is a flowchart of an embodiment of the image processing method of the present invention;

[0027] Fig.10 It is a data flow diagram of an embodiment of the image processing method of the present invention;

[0028] Fig.11 It is a flowchart of an embodiment of the image processing method of the present invention;

[0029] Fig.12 It is a flowchart of an embodiment of the image processing method of the present invention;

[0030] Fig.13 It is a data flow diagram of an embodiment of the image processing method of the present invention;

[0031] Fig.14 It is a flowchart of an embodiment of the image processing method of the present invention;

[0032] Fig.15 It is a data flow diagram of an embodiment of a method for generating a stereoscopic visual effect of the present invention;

[0033] Fig.16 It is a flow chart of an embodiment of a method for generating a stereoscopic visual effect according to the present invention;

[0034] Fig.17 It is a simple schematic diagram of an embodiment of a method for generating a stereoscopic visual effect of the present invention;

[0035] Fig.18 It is a data flow diagram of an embodiment of a method for generating a stereoscopic visual effect of the present invention. DETAILED DESCRIPTION

[0036] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0037] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those understood by a person skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For example, the term "plurality" includes two or more.

[0038] 1. Remote surgical robots

[0039] Robotic remote surgery technology refers to the use of information network channels as a carrier for the transmission of medical information, and adopts a master-slave method to combine robotics, virtual reality, artificial intelligence, and computer technology to expand the use of remote diagnosis, remote guidance, and remote operation of medical surgery, so that remote doctors can control surgical robots through information network channels to perform surgery on remote patients. The remote surgical robot includes master and slave devices, where the master operating device is the remote doctor console and the slave operating device is the patient surgical platform. The doctor performs operations on the remote doctor console and sends control commands to the slave operating device to control the patient surgical platform to perform surgery on the patient.

[0040] The doctor and the remote doctor console are located in a first geographical location, and the patient and the patient surgery platform are located in a second geographical location; because the first geographical location and the second geographical location are usually far apart, such as being in different hospitals, buildings, regions, countries, etc., the remote doctor console and the patient surgery platform transmit surgical process data through an information network channel, such as master-slave control signaling, endoscopic video data, audio data, tactile feedback data, pairing data and other master-slave device interaction data.

[0041] Specifically, the remote doctor console is provided with a first remote host, and the patient surgery platform is provided with a second remote host. The remote doctor console and the patient surgery platform are connected to the information network channel through the first remote host and the second remote host respectively to realize the transmission of the surgery process data.

[0042] Remote surgical robots can specifically include: laparoscopic surgical robots (single-hole, multi-hole), natural cavity surgical robots, percutaneous puncture surgical robots, orthopedic surgical robots, pan-vascular surgical robots, dental surgical robots, neurosurgery surgical robots, etc., including the aforementioned remote doctor console and patient surgical platform, so that remote surgical technology can be realized.

[0043] These types of surgical robots usually include two or more relatively independent arms with joint components, and the movement of one arm can be used as input to control the associated movement of another arm. The control method involved in the present disclosure is applicable to a surgical robot with more than two arms, and the movement of one arm can cause the movement of another arm.

[0044] In some embodiments, the surgical robot includes a patient surgical platform and a remote doctor console that can manipulate the patient surgical platform, which may also be referred to as the patient surgical platform and the remote doctor console in the remote surgical robot. Different patient surgical platforms may be manipulated by remote doctor consoles of the same structure or by remote doctor consoles of different structures, that is, different patient surgical platforms and different remote doctor consoles may be interconnected to perform mapping control. Further, the specific types of laparoscopic surgical robots may be distinguished according to the different structures of the patient surgical platforms.

[0045] For example, a laparoscopic surgical robot corresponding to a patient surgical platform equipped with multiple medical instruments inserted into a patient's body through different surgical channels provided by a same puncture device connected to the patient can be called a single-port laparoscopic surgical robot. For another example, a laparoscopic surgical robot corresponding to a patient surgical platform equipped with multiple medical instruments inserted into a patient's body through surgical channels provided by multiple different puncture devices connected to the patient can be called a multi-port laparoscopic surgical robot. These medical instruments generally include imaging instruments and surgical instruments, the imaging instruments are used to provide a field of view, and the surgical instruments are used for surgical operations such as cutting and suturing.

[0046] The patient surgical platform is equipped with at least an imaging device, and the remote doctor console is equipped with at least a display device, which is used by the doctor to remotely observe the anatomical parts. Different types of patient surgical platforms may be equipped with imaging devices of different configurations, and different types of remote doctor consoles may be equipped with displays of different configurations; at the same time, as the hardware of the patient surgical platform is upgraded, it may also be equipped with imaging devices of different configurations, and as the hardware of the remote doctor console is upgraded, it may also be equipped with displays of different configurations. When the configuration of the imaging device is different from that of the display, the surgical image captured by the imaging device will be incompatible with the display of the display, causing display delays or even crashes of the display, affecting the surgical process.

[0047] Figure 1 The figure shows a single-port laparoscopic surgical robot. Figure 1 As shown, in a single-port laparoscopic surgical robot, a patient surgical platform 100 includes a driving arm, which includes a main arm 110 and a manipulator assembly 120 connected in sequence, both of which have joint assemblies, and the main arm 110 can adjust the position and / or posture of the manipulator assembly 120. Figure 2Referring to the figure, the manipulator assembly 120 includes a housing 130 connected to the end of the main arm 110, a manipulator 140 accommodated in the housing 130, and a medical device 150 detachably mounted on the manipulator 140. The manipulator 140 includes a plurality of manipulators 140, each of which is accommodated in the housing 130 and connected to the end of the main arm 110 in parallel through the housing 130. The movement of the main arm 110 can cause the joint movement of the plurality of manipulators 140 in position and / or posture. Generally, the manipulator 140 has at least one joint assembly to provide, for example, a feeding degree of freedom in an operating space to adjust the depth of insertion of the medical device 150 into the patient's body. The medical device 150 may include a plurality of, usually the same number as or less than the number of manipulators 140, and the medical device 150 also includes a plurality of joint assemblies. The joint assemblies of the medical device 150 can provide a plurality of degrees of freedom in the operating space under the drive of the manipulator 140 to realize, for example, all other degrees of freedom except the feeding degree of freedom, including the horizontal translation degree of freedom, the vertical translation degree of freedom, the rotation degree of freedom, the yaw degree of freedom and the pitch degree of freedom. The feeding degree of freedom, the horizontal translation degree of freedom and the vertical translation degree of freedom are the three positional degrees of freedom of the operating space, and the rotation degree of freedom, the yaw degree of freedom and the pitch degree of freedom are the three posture degrees of freedom of the operating space. Figure 2 The plurality of medical devices 150 shown may be Figure 1 The same puncture device 160 in is inserted into the patient's body.

[0048] Continue reading Figure 1 , and a natural orifice surgical robot 300 is also shown. The natural orifice surgical robot 300 shows a patient surgical platform 320, and the patient surgical platform 320 also includes an imaging vehicle 330. The patient surgical platform 320 is connected with a catheter instrument 340, and is connected with a sensor system 350, and a control system 360 for realizing control between the catheter instrument 340, the sensor system 350 and the imaging vehicle 330.

[0049] The patient surgical platform 320 can usually be moved to the side of the surgical bed to engage the catheter instrument 340, and under the control command, the catheter instrument 340 is controlled to move up and down in the vertical direction, or to translate in the horizontal direction, or to move in non-vertical and non-horizontal directions, so as to provide a better preoperative preparation angle for the operation of the catheter instrument 340. The control command can be a command triggered by the doctor operating the patient surgical platform 320, or a command triggered by the doctor directly clicking or pressing a button set on the patient surgical platform 320. Of course, in other embodiments, the control command can also be a command triggered by voice control or a force feedback mechanism.

[0050] like Figure 1As shown, further, the patient surgical platform 320 may include a base 321, a sliding seat body 322 that can be lifted and moved along the base 321, and two mechanical arms 323 fixedly connected to the sliding seat body 322. The mechanical arm 323 may include a plurality of arm segments connected at a joint, and the plurality of arm segments provide the mechanical arm 323 with a plurality of degrees of freedom, for example, seven degrees of freedom corresponding to the seven arm segments. A power unit (not shown in the figure) is installed at the end of the mechanical arm 323, and the power unit of the mechanical arm 323 is used to engage the catheter instrument 340, and under the driving action of the power unit, the end of the catheter instrument 340 is controlled to bend and turn accordingly. Among them, the two mechanical arms 323 may be structures that are completely the same or partially the same, one mechanical arm 323 is used to engage the inner catheter instrument 341, and the other mechanical arm 323 is used to engage the outer catheter instrument 342. During installation, the outer catheter device 342 may be installed first, and when the outer catheter device 342 is installed, the catheter of the inner catheter device 341 may be inserted into the catheter of the outer catheter device 342 .

[0051] The sensor system 350 has one or more subsystems for receiving information about the catheter device 340. The subsystems may include: a position sensor system; a shape sensor system for determining the position, orientation, speed, velocity, pose, and / or shape of the tip of the catheter device 340 and / or along one or more segments of the catheter that may constitute the catheter device 340; and / or a visualization system for capturing images from the tip of the catheter device 340.

[0052] The imaging vehicle 330 may be provided with a display system 331 and a flushing system (not shown in the figure), etc. The display system 331 is used to display images or representations of the surgical site and the catheter instrument 340 generated by the subsystem of the sensor system 350. Real-time images of the surgical site and the catheter instrument 340 captured by the visualization system may also be displayed. Images of the surgical site recorded before or during surgery may also be presented using image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, etc.

[0053] Wherein, the preoperative or intraoperative image data may be presented as a two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or rate-based) image and / or as an image from a model created based on the preoperative or intraoperative image data set, and a virtual navigation image may also be displayed. In the virtual navigation image, the actual position of the catheter device 340 is registered with the preoperative image to present a virtual image of the catheter device 340 within the surgical site to the operator from the outside.

[0054] The control system 360 includes at least one memory and at least one processor. It is understood that the control system 360 can be integrated into the patient surgical platform 320 or the imaging vehicle 330, or can be independently provided. The control system 360 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT, and wireless telemetry. The control system 360 can transmit one or more signals indicating movement of the catheter instrument 340 by the power unit to move the catheter instrument 340. The catheter instrument 340 can extend to a surgical location in the body via an opening of a natural cavity or a surgical incision of the patient.

[0055] Furthermore, the control system 360 may include a mechanical control system (not shown in the figure) and an image processing system (not shown in the figure), wherein the mechanical control system is used to control the movement of the catheter device 340, and thus, may be integrated into the patient surgical platform 320. The image processing system is used for virtual navigation path planning, and thus, may be integrated into the imaging vehicle 330. Of course, the various subsystems of the control system 360 are not limited to the specific situations listed above, and may also be reasonably set according to actual conditions.

[0056] The image processing system can image the surgical site using the above imaging technology based on the image of the surgical site recorded before or during the operation. The recorded image can also be converted into a two-dimensional or three-dimensional synthetic image of a part or the entire anatomical organ or segment by software used in combination with manual input. During the virtual navigation procedure, the sensor system 350 can be used to calculate the position of the catheter instrument 340 relative to the patient's anatomical structure, which can be used to generate an external tracking image and an internal virtual image of the patient's anatomical structure, so as to achieve the actual position of the catheter instrument 340 and the preoperative image registration, so that the virtual image of the catheter instrument 340 in the surgical site can be presented to the operator from the outside.

[0057] The structural composition of the inner catheter device 341 and the outer catheter device 342 is substantially the same, and each comprises a slender and flexible inner catheter 41 and an outer catheter 42, wherein the diameter of the outer catheter 42 is slightly larger than that of the inner catheter 41, so that the inner catheter 41 can pass through the outer catheter 42 and provide a certain support for the inner catheter 41, so that the inner catheter 41 can reach the target position in the patient's body, so as to facilitate operations such as tissue or cell sampling from the target position.

[0058] Figure 3 The figure shows a multi-port laparoscopic surgical robot. Figure 3As shown, in the multi-hole laparoscopic surgical robot, the patient surgical platform 200 includes a driving arm, and the driving arm includes a main arm 210, an adjustment arm 220 and a manipulator assembly 230 connected in sequence, and all three are equipped with joint assemblies. The adjustment arm 220 includes a plurality of them, and their proximal ends are all connected to the end of the main arm 210. The end of the main arm 210 includes a certain directional platform 215, that is, the proximal ends of each adjustment arm 220 are connected to the directional platform 215, and the main arm 210 can adjust the position and / or posture of the adjustment arm 220 and the manipulator assembly 230. The manipulator assembly 230 includes a plurality of them, and the number thereof is the same as the number of the adjustment arms 220. The proximal ends of the manipulator assembly 230 are connected to the end of the adjustment arm 220, and the adjustment arm 220 can adjust the position and / or posture of the manipulator assembly 230. The manipulator assembly 230 includes a manipulator 240 and a medical device 250 detachably mounted on the manipulator 240, both of which include a plurality of joint assemblies. The manipulator 240 includes a parallelogram mechanism. By using the parallelogram principle, the manipulator 240 can be limited to perform rotational motion around a remote motion center (i.e., Remote Center, RC). The joint assembly of the manipulator 240 can include multiple degrees of freedom in the operating space, including feed degrees of freedom, yaw degrees of freedom, and pitch degrees of freedom, and the yaw degrees of freedom and pitch degrees of freedom both perform rotational motion around RC. The joint assembly of the medical device 250 can provide multiple degrees of freedom in the operating space under the drive of the manipulator 240, including rotation degrees of freedom, yaw degrees of freedom, and pitch degrees of freedom. Among them, the feed degrees of freedom of the manipulator 240 adjust the depth of the medical device 250 inserted into the patient's body, the yaw degrees of freedom and the pitch degrees of freedom of the manipulator 240 can affect both the position and the posture of the medical device 250, and the rotation degrees of freedom, yaw degrees of freedom, and pitch degrees of freedom of the medical device 250 mainly affect the posture of the medical device 250, and the influence on the position of the medical device 250 can be ignored. Figure 3 In the embodiment, multiple medical devices 250 are inserted into the patient's body through different puncture devices 160 respectively.

[0059] In some embodiments, Figure 4 The remote doctor console 410, remote doctor console 420, and remote doctor console 430 shown can be connected to Figure 1 The patient surgical platform 100 shown or Figure 2 The patient surgical platforms 200 shown are relatively independent. They can be set up in close proximity, such as in the same room, within a few meters of each other; they can also be set up in long distances, such as in different cities, thousands of kilometers away.

[0060] Continue reading Figure 4, providing specific device structure examples of different types of remote doctor consoles, such as remote doctor console 410 may include operating unit 411 and display 412, remote doctor console 420 may include operating unit 421 and display 422, remote doctor console 430 may include operating unit 431 and display 432; a remote doctor console may include more than one operating unit, for example, two. Operating unit 411 may include multiple joint components, and operating unit 411 with multiple joint components may be regarded as an arm in a surgical robot; operating unit 431 may also be a handle including multiple buttons or torsion bars.

[0061] 2. Image interconnection system of remote surgical robot

[0062] The remote surgical robot comprises: at least one group of interconnected first terminals, a first remote host and an image acquisition device, wherein the first terminal is used to control the image acquisition device to acquire a first video stream of a surgical scene; at least one group of interconnected second terminals, a second remote host and an image display device, wherein the second terminal is used to control the image display device to display a second video stream of the surgical scene; wherein one end of the first terminal and the second terminal is a remote doctor's console, and the other end is a patient surgical platform.

[0063] In this embodiment, a master hand end or a master hand end of the remote surgical robot includes a group of interconnected first terminals, a first remote host and an image acquisition device, or a group of interconnected second terminals, a second remote host and an image display device.

[0064] When the first terminal is a remote doctor console, the image acquisition device includes a camera for capturing the overall scene of the doctor operating the remote doctor console, a camera for capturing the local scene of the doctor operating the remote doctor console operating part, etc. When the first terminal is a patient surgical platform, the image acquisition device includes an image instrument (endoscope), a camera for capturing the overall scene of the patient surgical platform performing a surgical operation on the patient (pan-tilt camera), a camera for capturing the local scene of the patient surgical platform contacting the patient (gun camera), etc. The surgical scene includes the aforementioned overall scenes and local scenes.

[0065] In this embodiment, when the second terminal is a remote doctor console, the image display device includes: Figure 4 The display 412, display 422, display 432 (imaging trolley screen), etc. are shown; when the second terminal is a patient surgery platform, the image display device includes the display 422, display 432 (imaging trolley screen), etc.

[0066] The processor can be set in the first terminal or the first remote host, and in the second terminal or the second remote host, and is configured to: in response to a successful communication pairing between one of the first remote hosts and one of the second remote hosts, obtain the first device information of one of the first remote hosts corresponding to the first terminal, and the second device information of one of the second remote hosts corresponding to the second terminal, and determine the device relationship between the first device information and the second device information; and adapt the first video stream captured by the image acquisition device to the second video stream displayed on the image display device according to the device relationship through the first remote host or the second remote host.

[0067] In this embodiment, after the communication pairing between the different first remote hosts and the second remote hosts is successful, the corresponding types of first terminals and second terminals can be interactively controlled through the communication network. Different types of first terminals are provided with image acquisition devices with different configurations, and different types of second terminals are provided with image display devices with different configurations; when the configurations of the image acquisition device and the image display device are different, adaptation processing is required to map the first video stream to the second video stream, so that there is no compatibility problem between the image acquisition device and the image display device.

[0068] In this embodiment, the first device information includes at least one of the following: first configuration information, first assembly information, first model information, and first type information of the image acquisition device / first terminal; the second device information includes at least one of the following: second configuration information, second assembly information, second model information, and second type information of the image display device / second terminal.

[0069] Specifically, the first model information / second model information includes at least one of the following: Jingfeng multi-port laparoscopic surgical robot MP1000 / MP2000, Jingfeng single-port laparoscopic surgical robot SP1000, Jingfeng natural orifice surgical robot CP1000, Intuitive Surgical multi-port surgical robot da Vinci X / da Vinci Xi, Intuitive Surgical single-port surgical robot da VinciSP, minimally invasive laparoscopic surgical robot Touma, etc.

[0070] Specifically, the first type of information / second type of information includes at least one of the following: laparoscopic surgical robot (single-hole, multi-hole), natural cavity surgical robot, percutaneous puncture surgical robot, orthopedic surgical robot, pan-vascular surgical robot, dental surgical robot, neurosurgery surgical robot and other surgical robot type information.

[0071] Among them, different models or types of remote doctor consoles or patient surgery platforms are fixedly provided with image display devices or image acquisition devices of different configurations based on the system design.

[0072] Specifically, the first configuration information includes at least one of the following: single-channel / dual-channel (number of image acquisition channels), two-dimensional (2D) / three-dimensional (3D) (image acquisition dimensions), 2K / 4K (image acquisition resolution); second configuration information: single-channel / dual-channel / triple-channel (number of image display channels), two-dimensional (2D) / three-dimensional (3D) (image display dimensions), 2K resolution / 4K resolution (image display resolution).

[0073] In this embodiment, the adaptation process includes at least one of the following: video stream transmission path adaptation and video stream format adaptation between the first remote host and the second remote host. Among them, the video stream transmission path adaptation includes the adaptation of the aforementioned image acquisition path and the image display path; the video stream format adaptation includes the adaptation of the aforementioned image acquisition dimension and image display dimension, the adaptation of the image acquisition resolution and the image display resolution, etc., and specifically includes at least one of the following: image offset processing, image synchronization processing, image scaling processing, image fusion processing, UI overlay processing, etc.

[0074] For example, Figure 5 As shown, it is assumed that the first terminal is a patient surgical platform, such as patient surgical platform A (connected to image acquisition device A), patient surgical platform B (connected to image acquisition device B), and patient surgical platform C (connected to image acquisition device C) of different configurations / types / models; the second terminal is a remote doctor console, such as remote doctor console A (connected to image display device A), remote doctor console B (connected to image display device B), and remote doctor console C (connected to image display device C) of different configurations / types / models. The processor can be specifically set in each of the aforementioned patient surgical platforms and each of the remote doctor consoles.

[0075] Different patient surgical platforms and different remote doctor consoles can be communicated and paired with each other through network channels. For example, the first remote host A, the first remote host B or the first remote host C can be communicated and paired with at least one of the second remote host A, the second remote host B and the second remote host C respectively.

[0076] Specifically, for example, the first remote host A and the second remote host B are successfully connected in communication: the model of the patient surgical platform A is "Jinfeng single-port laparoscopic surgical robot SP1000", the type is "multi-port laparoscopic surgical robot", and the configuration is "dual-channel two-dimensional 4K image acquisition device"; the model of the remote doctor console B is "Intuitive Surgical Multi-port Laparoscopic Surgical Robot da VinciXi", the type is "multi-port laparoscopic surgical robot", and the configuration is "single-channel three-dimensional 2K image display device". The equipment relationship between the two includes at least: dual-channel-single-channel relationship, two-dimensional-three-dimensional relationship, 4K-2K relationship; the following adaptation processing can be performed by the processor in the first remote host A / the second remote host B: image offset processing, image scaling processing, image fusion processing, and UI overlay processing, etc.

[0077] Furthermore, the first remote host and the second remote host include an image processing device (which may be the aforementioned imaging trolley) and a remote communication device, and the processor may be disposed in the image processing device.

[0078] The image interconnection system of the remote surgical robot was introduced above. The following is an explanation of the adaptation process of different types of remote doctor consoles and patient surgical platforms in image interconnection. Figure 6 , an embodiment of an image processing method applied to remote surgical robot image interconnection is provided, which is specifically as follows:

[0079] 601. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0080] In this embodiment, in response to one of the multiple first remote hosts and one of the multiple second remote hosts, a communication pairing request is initiated between each other by selecting a counterpart; authentication processing and communication connection are performed based on the communication pairing request, and after the authentication processing is completed and the communication connection is successful, it is determined that the communication pairing between the two is successful.

[0081] In this embodiment, after the communication pairing is successful, the first terminal connected to one of the first remote hosts can interactively control the second terminal connected to one of the second remote hosts. In the remote surgical robot, the image acquisition device and the image display device constitute an imaging system, which at least serves as the doctor's eyes when performing surgical operations to observe the patient's anatomical parts, surgical space, etc.

[0082] Specifically, the first device information is used to represent the configuration parameters of the image acquisition device, and the second device information is used to represent the configuration parameters of the image display device. According to the difference in the configuration parameters of the two, the device relationship between the two is determined, such as the aforementioned: dual-channel-single-channel relationship, two-dimensional-three-dimensional relationship, 4K-2K relationship, etc.

[0083] Among them, after the first remote host and the second remote host are successfully paired for communication, the first terminal sends its own device information to the first remote host, and the second terminal sends its own device information to the second remote host; so that the first remote host and / or the second remote host can subsequently perform adaptation processing between the first video stream and the second video stream to meet the compatible display of the second video stream on the image display device.

[0084] 602. Adapt the first video stream captured by the image capture device to the second video stream displayed on the image display device according to the device relationship through the first remote host or the second remote host.

[0085] In this embodiment, in addition to performing adaptation processing on the first video stream and the second video stream, according to the device relationship, a first remote host or a second remote host that performs adaptation processing may also be determined. For example, for device relationships including a dual-channel-single-channel relationship and a two-dimensional-three-dimensional relationship, image offset processing, image fusion processing, and UI overlay processing may be performed on the first remote host; for example, for device relationships including a dual-channel-dual-channel relationship and a two-dimensional-three-dimensional relationship, image offset processing, image synchronization processing, and UI overlay processing may be performed on the second remote host.

[0086] In one embodiment, the image acquisition device includes at least one shooting unit, and the image display device includes at least one display unit; the device relationship includes at least one of the following: the quantity relationship between the shooting units and the display units, the dimensional relationship and the resolution relationship between the first video stream and the second video stream; the adaptation processing includes at least one of the following: video stream transmission path adaptation and video stream format adaptation between the first remote host and the second remote host.

[0087] In this embodiment, the quantitative relationship between the shooting units and the display units represents: the relationship between the aforementioned image acquisition paths and the image display paths; the dimensional relationship between the first video stream and the second video stream represents: the relationship between the aforementioned image acquisition dimension and the image display dimension; the resolution relationship between the first video stream and the second video stream represents: the relationship between the aforementioned image acquisition resolution and the image display resolution.

[0088] In this embodiment, the video stream transmission path adaptation between the first remote host and the second remote host is executed in the first remote host. Based on different device relationships, corresponding transmission paths are set respectively. According to the device relationship of successful communication pairing, the corresponding transmission path is matched to transmit the first video stream / second video stream from the first remote host to the second remote host.

[0089] According to the different device relationships between the first terminal and the second terminal, several preferred embodiments of the image processing method applied to the remote surgical robot image interconnection are provided below, as shown below:

[0090] 1) Dual-channel 2D endoscope-single-channel 3D display

[0091] The aforementioned quantity relationship includes: a first number of shooting units and a second number of display units, and the aforementioned dimensional relationship includes a first video stream of a first dimension and a second video stream of a second dimension, wherein the first number is greater than the second number, and the first dimension is lower than the second dimension; the video stream transmission path adaptation includes: the video stream is transmitted between the first remote host and the second remote host via a first path, and the first path includes a second number of transmission links; please refer to the following Figure 7 The specific adaptation process based on the device relationship is as follows:

[0092] 701. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0093] 702. Collect a first number of first-dimensional surgical images simultaneously by using a first number of shooting units, wherein each video frame in the first video stream includes the first number of first-dimensional surgical images;

[0094] 703. Synthesize the first number of first-dimensional surgical images into a second number of second-dimensional surgical images through the first remote host, and transmit the second number of second-dimensional surgical images to the second remote host respectively by using a second number of transmission links in the first path, wherein each video frame in the second video stream includes the second number of second-dimensional surgical images;

[0095] 704. Display a second number of second-dimensional surgical images respectively using a second number of display units through the second remote host.

[0096] In this embodiment, the image acquisition device acquires a large number of low-dimensional images, and the image display device displays a small number of high-dimensional images; the representation adaptation process may involve image offset processing, image fusion processing, UI overlay processing, etc. If the first path includes more than 1 second number of transmission links, image synchronization processing may further be involved.

[0097] For example, Figure 8 As shown, the image acquisition device is a dual-path two-dimensional endoscope comprising two shooting units (shooting unit 811, shooting unit 812), and the dual-path two-dimensional endoscope is detachably connected to the patient surgical platform 810, and is used to acquire dual-path two-dimensional surgical images (two-dimensional surgical image 813, two-dimensional surgical image 814), and each video frame in the first video stream includes a dual-path two-dimensional surgical image.

[0098] The image display device is a single-channel three-dimensional display including a display unit 821; the single-channel three-dimensional display is arranged in the remote doctor console 820, and is used to display a single-channel three-dimensional surgical image (three-dimensional surgical image 822) synthesized from the two-channel two-dimensional surgical images, and each video frame in the second video stream includes a single-channel three-dimensional surgical image.

[0099] The first remote host 830 is provided with a plurality of remote ports, including at least one first remote port 831 , and the first path includes: a transmission link 832 formed based on the first remote port, for transmitting the single-channel three-dimensional surgical image.

[0100] Specifically, the patient surgical platform 810 simultaneously collects two two-dimensional surgical images 813 and two-dimensional surgical images 814 through the shooting unit 811 and the shooting unit 812 of the dual-path two-dimensional endoscope; through the first remote host 830, the two two-dimensional surgical images 813 and the two-dimensional surgical image 814 are synthesized into a three-dimensional surgical image 822, and the three-dimensional surgical image 822 is transmitted to the second remote host 840 through a transmission link 832 in a first path of a first remote port 831; the second remote host 840 displays the received three-dimensional surgical image 822 in a display unit 821.

[0101] 2) Dual-channel 2D endoscope-single-channel 3D display

[0102] The aforementioned quantitative relationship includes: a first number of shooting units and a first number of display units, and the dimensional relationship includes: a first video stream and a second video stream of a first dimension; the video stream transmission path adaptation includes: the first remote host and the second remote host transmit the video stream via a second path, and the second path includes a first number of transmission links; please refer to Fig. 9The specific adaptation process based on the device relationship is as follows:

[0103] 901. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0104] 902. Collect a first number of first-dimensional surgical images simultaneously by using a first number of shooting units, wherein each video frame in the first video stream includes the first number of first-dimensional surgical images;

[0105] 903. Transmitting a first number of first-dimensional surgical images to the second remote host respectively through the first remote host and using a first number of transmission links in the second path;

[0106] 904. Perform time synchronization processing on the first-dimensional surgical image of each of the video frames through the second remote host, and use a first number of display units to respectively display the time-synchronized first-dimensional surgical images, wherein each video frame in the second video stream includes a first number of time-synchronized first-dimensional surgical images.

[0107] In this embodiment, the surgical images captured by the image acquisition device and the surgical images displayed by the image display device are the same in number and dimension; at this time, the adaptation processing mainly involves image synchronization processing, so that the surgical images displayed by each display unit of the image display device are at the same moment, presenting a correct high-dimensional effect with multiple low-dimensional surgical images at the same moment.

[0108] For example, Fig.10 As shown, the image acquisition device is a dual-channel two-dimensional endoscope comprising two shooting units (shooting unit 1011, shooting unit 1012), and the dual-channel two-dimensional endoscope is detachably connected to the patient surgical platform 1010, and is used to acquire dual-channel two-dimensional surgical images (two-dimensional surgical image 1013, two-dimensional surgical image 1014), and each video frame in the first video stream includes a dual-channel two-dimensional surgical image.

[0109] The image display device is a dual-channel two-dimensional display comprising two display units (display unit 1021, display unit 1022); the dual-channel two-dimensional display is arranged in the remote doctor console 1020, and is used to display the time-synchronized dual-channel two-dimensional surgical images (two-dimensional surgical image 1041, two-dimensional surgical image 1042), and each video frame in the second video stream includes the time-synchronized dual-channel two-dimensional surgical image.

[0110] The first remote host 1030 is provided with a plurality of remote ports, and includes at least two second remote ports 1031 and a second remote port 1032, and the second path includes: two transmission links (transmission link 1033, transmission link 1034) formed based on the two second remote ports, for transmitting the dual-path two-dimensional surgical images.

[0111] Specifically, the patient surgery platform 1010 simultaneously collects two two-dimensional surgical images 1013 and two-dimensional surgical images 1014 through the shooting unit 1011 and the shooting unit 1012 of the dual-path two-dimensional endoscope; through the first remote host 1030, using the two transmission links 1033 and the transmission link 1034 formed by the second remote port 1031 and the second remote port 1032, each group of two two-dimensional surgical images 1013 and two-dimensional surgical images 1014 are respectively transmitted to the second remote host 1040; the second remote host 1040 synchronizes the received multiple groups of two two-dimensional surgical images 1013 and two-dimensional surgical images 1014 to obtain each group of two two-dimensional surgical images 1041 and two-dimensional surgical images 1042 with time matching, and synchronously displays them on the two display units 1021 and the display unit 1022.

[0112] Furthermore, since the two two-dimensional surgical images 1041 and 1042 need to form a three-dimensional display effect when displayed on the two display units 1021 and 1022, time synchronization needs to be performed to ensure that the displayed two-dimensional surgical images 1041 and 1042 are acquired at the same time.

[0113] 3) Dual-channel 2D endoscope-single-channel 3D display

[0114] The dual-channel two-dimensional surgical image includes a first channel two-dimensional surgical image and a second channel two-dimensional surgical image of each video frame; see below Fig.11 The specific adaptation process based on the device relationship is as follows:

[0115] 1101. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0116] 1102. Identify, through the second remote host, a first time stamp of the first two-dimensional surgical image and a second time stamp of the second two-dimensional surgical image of each video frame, and align the first time stamp and the second time stamp;

[0117] In this embodiment, the video streams captured by the two shooting units of the patient's surgical platform add timestamps to the first two-dimensional surgical image and the second two-dimensional surgical image in each video frame. The second remote host identifies the timestamps (first time stamp, second time stamp) from the first two-dimensional surgical image and the second two-dimensional surgical image, and performs time alignment based on the timestamps; if the timestamps are the same, it is considered that the first two-dimensional surgical image and the second two-dimensional surgical image are captured by the two shooting units at the same time.

[0118] In addition, when the remote surgical robot is detected to be busy, for example, the first remote host and the second remote host simultaneously perform tasks such as motion control, two-way video merging (merging the first two-dimensional surgical image and the second two-dimensional surgical image), video encoding, and data transmission, the transmitted first two-dimensional surgical image and the second two-dimensional surgical image may be out of order, packet loss, and other problems. A sliding window mechanism is used to sort the video frames in the window by timestamp, and the window size is dynamically adjusted according to the current network status to balance the delay and disorder problems. At the same time, a partial retransmission mechanism is used to handle the packet loss problem. Within the window, when the timestamps of two adjacent video frames that have been sorted are significantly greater than the acquisition frequency of the video frame, the patient's surgical platform is required to detect the timestamp and resend the lost video frame.

[0119] 1103. If the alignment result satisfies a preset error, then, according to the alignment result, time synchronization is performed on each of the first-channel two-dimensional surgical images and each of the second-channel two-dimensional surgical images;

[0120] In this embodiment, when the clock jitter occurs on the first remote host / patient surgical platform, when the jitter degree is higher than the threshold, the alignment result of the first time mark and the second time mark will be inaccurate; when the alignment result does not exceed the preset error, the time alignment is still performed according to the timestamps (first time mark and second time mark). If the alignment result exceeds the preset error, the first two-dimensional surgical image and the second two-dimensional surgical image may affect the three-dimensional visual effect formed because they display different surgical scenes. At this time, it is necessary to synchronize the time of the surgical scene screen actually displayed by both.

[0121] 1104. If the alignment result does not meet the preset error, extracting the first target motion trajectory and the first surgical background image in the first two-dimensional surgical image, and the second target motion trajectory and the second surgical background image in the second two-dimensional surgical image in each video frame;

[0122] In this embodiment, the objects of the first target motion trajectory and the second target motion trajectory are the same, such as target anatomical tissue, medical equipment, preset motion targets set on medical equipment / target anatomical tissue, etc. The first surgical background image represents the area of ​​the first two-dimensional surgical image except the first target motion trajectory, and the second surgical background image represents the area of ​​the second two-dimensional surgical image except the second target motion trajectory.

[0123] 1105. Match feature points in each of the first surgical background images and each of the second surgical background images; match each of the first two-dimensional surgical images and each of the second two-dimensional surgical images according to the feature points, the first target motion trajectory, and the second target motion trajectory;

[0124] In this embodiment, the basic matrix for feature point matching between the first surgical background image and the second surgical background image is calculated by the normalized eight-point algorithm; and the preset epipolar geometry constraint algorithm and the basic matrix are combined to achieve time-related trajectory point matching between the first target motion trajectory and the second target motion trajectory, indicating the matching relationship between the first two-dimensional surgical image and the second two-dimensional surgical image.

[0125] The time-correlated trajectory point matching set of the first target motion trajectory and the second target motion trajectory is calculated. The implementation steps are as follows: (1) one of the first two-dimensional surgical image and the second two-dimensional surgical image is used as the reference image and the other is used as the image to be synchronized. For each trajectory point q1(t i ), calculate each trajectory point q1(t i ) in the epipolar line I in the image to be synchronized q1(ti) :I q1(ti) =F 12 q1(t i ), where F12 is the aforementioned basic matrix; (2) Obtain the epipolar line I q1(ti) The intersection points with the motion trajectory of the second target in the image to be synchronized are used as a trajectory point pair matching set.

[0126] 1106. Fitting a first time line of each of the first two-dimensional surgical images and a second time line of each of the second two-dimensional surgical images according to the matching result;

[0127] In this embodiment, when fitting the first timeline and the second timeline, the track point pair matching set obtained above may contain some wrong track point matching pairs. In order to achieve a more accurate result, the track point pair matching set is fitted to obtain the time model parameters between the reference image and the image to be synchronized: frame rate ratio α (frame rate ratio between the two shooting units) and time offset ∆t (time difference between the two shooting units starting shooting). The above timeline model is the first timeline and the second timeline.

[0128] (1) Randomly select two points from the trajectory point pair matching set and calculate the time model parameters: frame rate ratio α and time offset ∆t;

[0129] (2) Substitute the trajectory point pair matching set into the straight line model to calculate the error: t`=α*t+∆t, where t and t′ are the frame numbers of the reference image and the image to be synchronized, respectively. If the error is less than or equal to the preset threshold d, it becomes a valid trajectory point and is recorded in the valid data set;

[0130] (3) Repeat steps (1)-(2) k times, compare the valid trajectory points of the valid data set, output the valid data set with the most valid trajectory points, and finally estimate the final time model parameters based on the output valid data set: frame rate ratio α and time offset ∆t, and determine the first timeline and the second timeline.

[0131] 1107. Based on the first timeline and the second timeline, time synchronize each of the first-channel two-dimensional surgical images and each of the second-channel two-dimensional surgical images.

[0132] Finally, based on the first timeline and the second timeline, the first two-dimensional surgical image and the second two-dimensional surgical image in the video frame at the same time point are synchronously output.

[0133] 4) Dual-channel 2D endoscope-single-channel 3D display

[0134] The aforementioned quantitative relationship includes: a first number of shooting units and a first number of display units, and the dimensional relationship includes: a first video stream of a first dimension and a second video stream of a first dimension; the video stream transmission path adaptation includes: the first remote host and the second remote host transmit the video stream via a third path, and the third path includes a transmission link, wherein the first number is greater than 1; please refer to Fig.12 The specific adaptation process based on the device relationship is as follows:

[0135] 1201. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0136] 1202. Collect a first number of first-dimensional surgical images simultaneously by using a first number of shooting units, wherein each video frame in the first video stream includes the first number of first-dimensional surgical images;

[0137] 1203. Splice the first number of first-dimensional surgical images into one first-dimensional surgical image through the first remote host, and transmit the first-dimensional surgical image to the second remote host using a transmission link in the third path;

[0138] 1204. Split one of the first-dimensional surgical images into a first number of first-dimensional surgical images through the second remote host, and use a first number of display units to respectively display the first number of first-dimensional surgical images, wherein each video frame in the second video stream includes the first number of first-dimensional surgical images after splitting.

[0139] In this embodiment, a first number of first-dimensional surgical images are spliced ​​into one first-dimensional surgical image, so as to maintain time synchronization when the first number of first-dimensional surgical images are displayed separately on a first number of display units.

[0140] For example, Fig.13 As shown, the image acquisition device is a dual-channel two-dimensional endoscope comprising two shooting units (shooting unit 1311, shooting unit 1312), and the dual-channel two-dimensional endoscope is detachably connected to the patient surgical platform 1310, and is used to acquire dual-channel two-dimensional surgical images (two-dimensional surgical image 1313, two-dimensional surgical image 1314), and each video frame in the first video stream includes a dual-channel two-dimensional surgical image.

[0141] The image display device is a dual-channel two-dimensional display comprising two display units (display unit 1321, display unit 1322); the dual-channel two-dimensional display is arranged in the remote doctor console 1320, and is used to display the split dual-channel two-dimensional surgical images (two-dimensional surgical image 1323, two-dimensional surgical image 1324), and each video frame in the second video stream includes the split dual-channel two-dimensional surgical image.

[0142] The first remote host 1330 is provided with a plurality of remote ports, including at least one third remote port 1331 , and the third path includes: a transmission link 1332 formed based on the third remote port 1331 , for transmitting a single-channel two-dimensional surgical image 1333 .

[0143] Specifically, the patient surgical platform 1310 simultaneously collects two two-dimensional surgical images 1313 and 1314 through the shooting unit 1311 and the shooting unit 1312 of the dual-channel two-dimensional endoscope; through the first remote host 1330, using the transmission link 1332 formed by the third remote port 1331, the single two-dimensional surgical image 1333 spliced ​​from the two two-dimensional surgical images 1313 and 1314 is transmitted to the second remote host 1340; the second remote host 1340 re-splits the single two-dimensional surgical image 1333 into two-dimensional surgical images 1323 and 1324, and displays them synchronously on the two display units 1321 and 1322.

[0144] The splicing of the two-dimensional surgical image 1313 and the two-dimensional surgical image 1314 ensures that the two are surgical images acquired at the same time during the transmission between the first remote host 1330 and the second remote host 1340.

[0145] In one embodiment, the resolution relationship includes: a shooting unit of a first acquisition resolution and a display unit of a first display resolution, wherein the first acquisition resolution is higher than the first display resolution; the video stream format adaptation includes a first interpolation strategy; In one embodiment, the resolution relationship includes: a shooting unit of a second acquisition resolution and a display unit of a second display resolution, wherein the second acquisition resolution is lower than the second display resolution; the video stream format adaptation includes a second interpolation strategy; Please refer to Fig.14 The specific adaptation process based on different resolution relationships is as follows:

[0146] 1401. In response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of a first terminal corresponding to one of the first remote hosts and second device information of a second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined;

[0147] 1402. Using the first interpolation strategy, reduce the first video stream captured by the shooting unit from the first capture resolution to the first display resolution to obtain a second video stream, and transmit the second video stream to the second remote host;

[0148] 1403. Displaying the second video stream of the first display resolution by the display unit through the second remote host; or

[0149] 1404. Transmitting the first video stream captured by the shooting unit to the second remote host through the first remote host;

[0150] 1405. Using the second interpolation strategy, the received first video stream is upgraded from the second acquisition resolution to the second display resolution through the second remote host, and the second video stream of the second display resolution is displayed using the display unit.

[0151] For example, the first acquisition resolution is 2K, and the first display resolution is 4K; the second acquisition resolution is 4K, and the second display resolution is 4K; the first interpolation strategy and the second interpolation strategy can be implemented by a bilinear interpolation algorithm.

[0152] Among them, when the first acquisition resolution is higher than the first display resolution, the first remote host executes a first interpolation strategy to reduce the resolution of the first video stream, and when the first acquisition resolution is lower than the second display resolution, the second remote host executes a second interpolation strategy to increase the resolution of the first video stream, the purpose is to reduce the amount of data when the first remote host and the second remote host transmit the first video stream or the second video stream.

[0153] 3. Generation of Stereoscopic Visual Effects for the Interactive Interface of a Remote Surgery Robot

[0154] like Fig.15 As shown, the remote surgical robot comprises: a patient surgical platform 1510 and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit 1511 and a second shooting unit 1512, wherein the first shooting unit 1511 is used to shoot a left-view image 1513, and the second shooting unit 1512 is used to shoot a right-view image 1514; a remote doctor console 1520 and an image display device connected to each other, wherein the image display device is provided with a first display unit 1521 and a second display unit 1522; wherein the patient surgical platform 1510 and the remote doctor console 1520 are in communication connection;

[0155] The processor is configured to: obtain first device information of the patient surgery platform 1510, second device information of the remote doctor console 1520, and user interaction data (corresponding to display effect such as image 1515); select at least one end of the patient surgery platform 1510 and the remote doctor console 1520 according to the first device information and the second device information, and perform overlay operations in the left view image 1513 and the right view image 1514 based on the user interaction data (corresponding to display effect such as image 1515) to obtain a left view overlay image 1523 and a right view overlay image 1524; display the left view overlay image 1523 and the right view overlay image 1524 in the first display unit 1521 and the second display unit 1522, respectively.

[0156] In this embodiment, the image acquisition device includes, for example, a binocular endoscope, a binocular gun camera, or a binocular pan / tilt camera, and the image display device includes a binocular display (such as display 412), a binocular display large screen, and a binocular display screen (such as display 432) of an imaging trolley provided on a remote doctor console. By observing the image display device at a set position of the remote doctor console, the doctor can observe the stereoscopic visualization effect formed on the display device when the user interaction data is in the left view image and the right view image.

[0157] The first device information in this embodiment may also include first configuration information, first assembly information, first model information, and first type information, etc., wherein the first configuration information includes whether an image processor is set in the patient's surgical platform, and the number of image processors set. The second device information in this embodiment may also include second configuration information, second assembly information, second model information, and second type information, etc., wherein the second configuration information includes whether an image processor is set in the remote doctor's console, and the number of image processors set. Here, the first model information and the first type information, as well as the second model information and the second type information have the same specific meanings as described above.

[0158] Further, based on whether an image processor is set in the patient's surgical platform and the number of image processors set, and whether an image processor is set in the remote doctor's console and the number of image processors set, the transmission mode of the user interaction data, the left view image and the right view image, and the superimposed position of the user interaction data on the left view image and the right view image are determined. The hardware design is adapted to different types of patient surgical platforms and remote doctor consoles.

[0159] The image interconnection system of the remote surgical robot was introduced above. The following is an explanation of the adaptation process of different types of remote doctor consoles and patient surgical platforms in image interconnection. Fig.16, an embodiment of an image processing method applied to remote surgical robot image interconnection is provided, which is specifically as follows:

[0160] 1601. Acquire first device information of the patient surgery platform, second device information of the remote doctor console, and user interaction data;

[0161] 1602. Select at least one of the patient surgery platform and the remote doctor console according to the first device information and the second device information, and perform a superposition operation in the left view image and the right view image based on the user interaction data to obtain a left view superposition image and a right view superposition image;

[0162] 1603. Display the left-view overlay image and the right-view overlay image in the first display unit and the second display unit, respectively.

[0163] In this embodiment, the remote surgical robot also includes: a first remote host and a second remote host, the first remote host is connected to the patient surgical platform, and the second remote host is connected to the remote doctor console; wherein the patient surgical platform and the remote doctor console are communicatively connected via the first remote host and the second remote host, and the first remote host includes a first remote port and a second remote port.

[0164] Among them, if the first device information and the second device information meet the preset first condition, the patient surgical platform transmits the left-view image, the right-view image and the user interaction data to the remote doctor console through the first remote port; the remote doctor console performs overlay operations in the left-view image and the right-view image based on the user interaction data to obtain a left-view overlay image and a right-view overlay image.

[0165] Exemplarily, the first condition includes: the patient's surgical platform is provided with at least one image processing device for performing the superimposition operation, or no image processing device is provided, and the remote doctor's console is provided with at least one image processing device. When the patient's surgical platform is provided with at least one image processing device for performing the superimposition operation, the at least one image processing device is used to process the offset processing of the left view image and the right view image to form a preset parallax, so that the left view image and the right view image are subsequently displayed on the first display unit and the second display unit to form a stereoscopic visual effect. When the patient's surgical platform is not provided with an image processing device, the left view image and the right view image are not subjected to the offset processing, and are the original images captured by the first shooting unit and the second shooting unit.

[0166] Among them, if the first device information and the second device information meet the preset second condition, the patient surgery platform performs an overlay operation in the left view image and the right view image respectively based on the user interaction data to obtain a left view overlay image and a right view overlay image, and transmits the left view overlay image and the right view overlay image to the remote doctor console through the second remote port.

[0167] Exemplarily, the second condition includes: the patient's surgical platform is provided with at least one image processing device for performing the superimposition operation, and the remote doctor's console is not provided with an image processing device. In this case, the image processing device provided on the patient's surgical platform for performing the superimposition operation independently performs the superimposition operation of the user interaction data in the left view image and the right view image, and the offset processing of the left view image and the right view image; the remote doctor's console is only responsible for displaying the left view superimposition image and the right view superimposition image.

[0168] In one embodiment, when user interaction data is superimposed on a left-view image and a right-view image, a user interaction graphic is generated based on the user interaction data, and first assembly information in the first device information and second assembly information in the second device information are extracted; based on the first assembly information, the parallax of the user interaction graphic when superimposed on the left-view image and the right-view image, respectively, is calculated; based on the second assembly information, the parallax is corrected, and based on the corrected parallax, a first superimposed position of the user interaction graphic in the left-view image and a second superimposed position of the user interaction graphic in the right-view image are determined; the user interaction graphic is superimposed on the first superimposed position of the left-view image to obtain a left-view superimposed image, and the user interaction graphic is superimposed on the second superimposed position of the right-view image to obtain a right-view superimposed image.

[0169] In this embodiment, the user interaction data is generated based on the user's interactive operation on the remote surgical robot. The user interaction graph includes: a first part that overlaps with the left view image and the right view image when superimposed, and a second part that does not overlap, wherein the first part includes a track graph and the second part includes a status icon. Among them, the status icon includes a static icon and a dynamic icon, the static icon is switched according to its own preset interaction logic based on the user operation, and the dynamic icon is automatically switched based on the state of at least one patient surgical platform; the track graph is a custom graph, which is generated based on the user operation track.

[0170] Among them, the dynamic icon is based on the remote doctor console and different patient surgical platforms / robotic arms with different control permissions, etc., to determine at least one icon and select one of the icons for superposition. The static icon is an icon that is fixed and superimposed based on the interaction logic.

[0171] In this embodiment, the first assembly information includes the assembly error of the first shooting unit and the second shooting unit, and the second assembly information includes the assembly error of the first display unit and the second display unit. When the user interaction icon is superimposed on the left view image and the right view image, the assembly conditions of the first shooting unit and the second shooting unit on each patient's surgical platform are adapted, and the parallax error generated by the two relative to the standard parallax when the surgical images are collected is determined, and the parallax when the user interaction graphics are superimposed on the left view image and the right view image is obtained. At the same time, the assembly conditions of the first display unit and the second display unit on each remote doctor's console are adapted, and the parallax error generated by the two relative to the aforementioned parallax when the surgical images are displayed is determined, and the corrected parallax is obtained. An accurate stereoscopic visual effect is presented.

[0172] Furthermore, a first standard overlay position is preset for the user interaction graphic in the left-view image, and a second standard overlay position is preset in the right-view image; after determining the corrected parallax, a first moving orientation of the user interaction graphic on the left-view image relative to the first standard overlay position and a second moving orientation of the user interaction graphic on the right-view image relative to the second standard overlay position are determined; the first standard overlay position is adjusted by the first moving orientation to obtain the first overlay position, and the second standard overlay position is adjusted by the second moving orientation to obtain the second overlay position.

[0173] For example, Fig.17 As shown, the first shooting unit and the second shooting unit respectively shoot the left view image 1711 and the right view image 1712, and generate the user interaction graphics 1713 and the user interaction graphics 1714 based on the user interaction data; after determining the final parallax based on the first assembly information and the second assembly information, determine the first superimposed position of the user interaction graphics on the left view image 1711 and the second superimposed position on the right view image 1712; in the left view image 1711, the user interaction graphics 1713 and the user interaction graphics 1714 are offset to the left based on the parallax to obtain the first superimposed position ( user interaction graphics 1721 and user interaction graphics 1722); in the right-view image 1712, the user interaction graphics 1713 and the user interaction graphics 1714 are offset to the right based on the parallax to obtain a second overlay position (user interaction graphics 1731 and user interaction graphics 1732); the user interaction graphics 1713 and the user interaction graphics 1714 are overlaid on the first overlay position of the left-view image 1711 to obtain the left-view overlay image 1720, and the user interaction graphics are overlaid on the second overlay position of the right-view image 1712 to obtain the right-view overlay image 1730.

[0174] In addition, if Fig.18As shown, the image acquisition device 1810 is connected to the first remote port 1821 and the second remote port 1822 of the first remote host 1820, and the graphical user interface 1830 is connected to the second remote port 1822, and is at least used to obtain and display at least one of the left view image 1813 and the right view image 1814 from the first display unit 1811 and the second display unit 1812 of the patient surgical platform;

[0175] In one embodiment, in response to a sliding operation in the left view image 1813 and the right view image 1814, the track graph is superimposed on the left view image 1813 and the right view image 1814 through the graphical user interface 1830 to generate a left view superimposed image 1831 and a right view superimposed image 1832, respectively, and output to the second remote port 1822;

[0176] In one embodiment, in response to the generation of user interaction data corresponding to the status icon, the patient surgical platform outputs the left-view image 1813, the right-view image 1814 and the user interaction data corresponding to the status icon to the first remote port; the patient surgical platform can also directly overlay the status icon on the left-view image 1813 and the right-view image 1814 to generate a left-view overlay image and a right-view overlay image, respectively, and output them to the second remote port 1822.

[0177] In this embodiment, when the left view image and the right view image are marked on the graphical user interface of the patient's surgical platform (such as a sliding operation), the left view image and the right view image are first transmitted to the graphical user interface for display, and then the superimposed marked trajectory graphics are directly completed and output to the second remote host through the second remote port of the first remote host, and directly displayed on the remote doctor's console.

[0178] For other interactive operations besides annotation, the user interaction data of the patient's surgical platform and the left view image and the right view image captured by the image acquisition device can be respectively transmitted to the first remote host, and the user interaction data can be superimposed on the left view image and the right view image at the first remote host / second remote host without the need to be transmitted to the graphical user interface.

[0179] In one embodiment of generating user interaction data by a patient surgical platform, in response to an interactive operation between the remote physician console and the patient surgical platform, the patient surgical platform determines an interaction state corresponding to the interactive operation, and the user interaction data includes the interaction state; wherein the user interaction graphic includes a first state icon corresponding to the interaction state, and the interaction state includes: a connection state, a pairing state and a network state between the remote physician console and the patient surgical platform.

[0180] In this embodiment, the interactive operations between the remote doctor console and the patient surgery platform include: connection operation, pairing operation and network communication operation, etc., and the corresponding interactive status is prompted for the connection process, pairing process, network communication process, and / or connection results, pairing results, network communication results, etc.; among which, the first status icon can be represented by corresponding text or an indicative icon.

[0181] Specifically, the first status icon is a dynamic icon, which switches to a preset icon corresponding to the connection process, pairing process, network communication process, and / or the connection result, pairing result, and network communication result based on the connection operation, pairing operation or network communication operation.

[0182] In one embodiment of generating user interaction data on a patient surgical platform, the patient surgical platform further includes a graphical user interface, which includes at least one trigger control; in response to a touch operation on the trigger control, the patient surgical platform determines a touch state corresponding to the touch operation, and the user interaction data includes the touch state; wherein the user interaction graphic includes a second state icon corresponding to the touch state.

[0183] In this embodiment, at least one level of trigger controls is preset according to the user's interaction logic on the graphical user interface. When the user performs a touch operation on the trigger control on the graphical user interface, such as a press operation, the patient surgical platform triggers the corresponding touch operation. According to the interaction logic, the user interaction image switches to a second state icon corresponding to the touch state, and the second state icon includes at least one of the following: a medical imaging icon, a microphone on / off icon, a volume adjustment icon, a menu icon, a camera icon, a brush icon, etc.

[0184] Specifically, the second state icon is a static icon, which is switched to the first state icon corresponding to the touch state under the interaction logic based on the user performing a touch operation on the trigger control on the graphical user interface.

[0185] In one embodiment of generating user interaction data on a patient surgical platform, the graphical user interface also includes a marking area, and the marking area includes at least a partial area in the left-view image or the right-view image; in response to a sliding operation on the marking area, a sliding trajectory of the sliding operation on the left-view image and the right-view image is determined, and the user interaction data includes the sliding trajectory; wherein the user interaction graphic includes a trajectory graphic corresponding to the sliding trajectory.

[0186] In this embodiment, the left view image and / or the right view image are displayed on the graphical user interface. At the patient's operating table, the doctor can make annotations on the left view image or the right view image to generate a sliding trajectory, and superimpose the trajectory graphics on the left view image and the right view image, and display them on the image display device of the patient's operating table; for example, annotate the target tissue position in the image,

[0187] In one embodiment of generating user interaction data on a patient surgical platform, the remote surgical robot includes a first remote doctor console and a left-view image display device connected to each other, and a second remote doctor console and a right-view image display device connected to each other; the patient surgical platform includes at least a first robotic arm and a second robotic arm, wherein the first remote doctor console has control authority over the first robotic arm, and the second remote doctor console has control authority over the second robotic arm; the user interaction graphics include a first user interaction graphics and a second user interaction graphics, the first user interaction graphics are displayed on the left-view image display device, and the second user interaction graphics are displayed on the right-view image display device.

[0188] In this embodiment, in response to the first remote doctor console activating the control authority over the first robotic arm and the second remote doctor console activating the control authority over the second robotic arm, the patient surgical platform generates user interaction data based on the control status of the first robotic arm and the control status of the second robotic arm; at this time, the user interaction data will not be directly synchronized to the image display devices of the two remote doctor consoles for synchronous display, but different user interaction data will be sent to the first remote doctor console and the second remote doctor console respectively according to the control authority to display the first user interaction graph and the second user interaction graph respectively.

[0189] In one embodiment of generating user interaction data on a patient surgical platform, in response to a first control operation of the first remote physician console on the first robotic arm, the patient surgical platform determines a first control state corresponding to the first control operation, and the user interaction data includes the first control state, wherein the first user interaction image includes a third state icon corresponding to the first control state; or, in response to a second control operation of the second remote physician console on the second robotic arm, the patient surgical platform determines a second control state corresponding to the second control operation, and the user interaction data includes the second control state, wherein the second user interaction image includes a fourth state icon corresponding to the second control state.

[0190] In addition, when a remote doctor console controls different robotic arms, status icons of the control status of different robotic arms can be displayed respectively. The control status can include functional status, such as sleeve hiding status, tactile feedback status, remote motion center dynamic control status, etc. It can also include instrument connection status, such as endoscope connection status, energy instrument connection status, suturing instrument connection status, closure device connection status, cutting instrument connection status, etc.

[0191] It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should also be within the protection scope of the present invention and will not be described in detail here.

[0192] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the mobile terminal can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned mobile terminal can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0193] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0194] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0195] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0196] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0197] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if the described condition or event is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce the described condition or event is detected" or "in response to detecting the described condition or event" depending on the context.

[0198] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0199] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0200] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0201] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0203] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0204] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0205] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for generating a stereoscopic visual effect for a remote surgical robot, characterized in that: The remote surgical robot comprises: A patient surgical platform and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit and a second shooting unit, the first shooting unit is used to shoot a left-view image, and the second shooting unit is used to shoot a right-view image; A remote doctor console and an image display device connected to each other, wherein the image display device is provided with a first display unit and a second display unit; Wherein, the patient surgery platform and the remote doctor console are communicatively connected; a first remote host and a second remote host, wherein the first remote host is connected to the patient surgery platform, and the second remote host is connected to the remote doctor console; Wherein, the patient surgery platform and the remote doctor console are communicatively connected via the first remote host and the second remote host, and the first remote host includes a first remote port and a second remote port; The stereoscopic visual effect generation method comprises: Acquire first device information of the patient surgery platform, second device information of the remote doctor console, and user interaction data; According to the first device information and the second device information, at least one end of the patient surgery platform and the remote doctor console is selected, and based on the user interaction data, an overlay operation is performed in the left view image and the right view image respectively to obtain a left view overlay image and a right view overlay image, wherein if the first device information and the second device information meet the preset first condition, the patient surgery platform transmits the left view image, the right view image and the user interaction data to the remote doctor console through the first remote port; the remote doctor console performs an overlay operation in the left view image and the right view image respectively based on the user interaction data to obtain a left view overlay image and a right view overlay image; if the first device information and the second device information meet the preset second condition, the patient surgery platform performs an overlay operation in the left view image and the right view image respectively based on the user interaction data to obtain a left view overlay image and a right view overlay image, and transmits the left view overlay image and the right view overlay image to the remote doctor console through the second remote port; The left-view overlay image and the right-view overlay image are displayed in the first display unit and the second display unit respectively.

2. The method for generating a stereoscopic visual effect according to claim 1, characterized in that: Based on the user interaction data, a superposition operation is performed in the left-view image and the right-view image respectively to obtain a left-view superposition image and a right-view superposition image, including: generating a user interaction graph based on the user interaction data, and extracting first assembly information in the first device information and second assembly information in the second device information; Calculating, according to the first assembly information, the parallax when the user interaction graphic is superimposed on the left-view image and the right-view image respectively; Correcting the parallax according to the second assembly information, and determining a first superimposed position of the user interaction graphic on the left-view image and a second superimposed position on the right-view image based on the corrected parallax; The user interaction graphic is superimposed on a first superimposed position of the left-view image to obtain a left-view superimposed image, and the user interaction graphic is superimposed on a second superimposed position of the right-view image to obtain a right-view superimposed image.

3. The method for generating a stereoscopic visual effect according to claim 2, wherein: Before acquiring the first device information of the patient surgery platform, the second device information of the remote doctor console, and the user interaction data, the method further includes: In response to an interactive operation between the remote physician console and the patient surgery platform, the patient surgery platform determines an interactive state corresponding to the interactive operation, and the user interaction data includes the interactive state; The user interaction graphic includes a first status icon corresponding to the interaction status, and the interaction status includes: a connection status, a pairing status and a network status between the remote doctor console and the patient surgery platform.

4. The method for generating a stereoscopic visual effect according to claim 2, wherein: The patient surgical platform further includes a graphical user interface, the graphical user interface including at least one trigger control; Before acquiring the first device information of the patient surgery platform, the second device information of the remote doctor console, and the user interaction data, the method further includes: In response to a touch operation on the trigger control, the patient surgery platform determines a touch state corresponding to the touch operation, and the user interaction data includes the touch state; The user interaction graphic includes a second state icon corresponding to the touch state.

5. The method for generating a stereoscopic visual effect according to claim 4, characterized in that: The graphical user interface further includes a marking area, wherein the marking area includes at least a portion of the left-view image or the right-view image; In response to a sliding operation on the marked area, determining a sliding track of the sliding operation on the left-view image and the right-view image, wherein the user interaction data includes the sliding track; Wherein, the user interaction graph includes a track graph corresponding to the sliding track.

6. The method for generating a stereoscopic visual effect according to claim 2, wherein: The remote surgical robot comprises a first remote doctor console and a left-view image display device connected to each other, and a second remote doctor console and a right-view image display device connected to each other; The patient surgical platform comprises at least a first robotic arm and a second robotic arm, wherein the first remote doctor console has control authority over the first robotic arm, and the second remote doctor console has control authority over the second robotic arm; The user interaction graph includes a first user interaction graph and a second user interaction graph, the first user interaction graph is displayed on the left-view image display device, and the second user interaction graph is displayed on the right-view image display device; Before acquiring the first device information of the patient surgery platform, the second device information of the remote doctor console, and the user interaction data, the method further includes: In response to a first control operation of the first remote doctor console on the first robotic arm, the patient surgery platform determines a first control state corresponding to the first control operation, the user interaction data includes the first control state, wherein the first user interaction image includes a third state icon corresponding to the first control state; or, In response to the second control operation of the second remote doctor console on the second robotic arm, the patient surgical platform determines a second control state corresponding to the second control operation, and the user interaction data includes the second control state, wherein the second user interaction image includes a fourth state icon corresponding to the second control state.

7. The method for generating a stereoscopic visual effect according to any one of claims 2 to 6, characterized in that: The user interaction graphic includes: a first portion that overlaps with the left-view image and the right-view image when superimposed, and a second portion that does not overlap, wherein the first portion includes a track graphic and the second portion includes a status icon.

8. The method for generating a stereoscopic visual effect according to claim 7, characterized in that: The image acquisition device is connected to the first remote port and the second remote port, and the graphical user interface is connected to the second remote port, and is at least used to acquire and display at least one of the left view image and the right view image from the patient surgical platform; The stereoscopic visual effect generation method further comprises: In response to a sliding operation in the left-view image and the right-view image, the trajectory graphic is superimposed on the left-view image and the right-view image through the graphical user interface to generate a left-view superimposed image and a right-view superimposed image, respectively, and output to the second remote port; In response to the generation of the user interaction data corresponding to the status icon, the patient surgery platform outputs the left view image, the right view image and the user interaction data corresponding to the status icon to the first remote port; or, In response to generation of user interaction data corresponding to the status icon, the patient surgical platform overlays the status icon on the left view image and the right view image to generate a left view overlay image and a right view overlay image, respectively, and outputs them to the second remote port.

9. A remote surgical robot, characterized in that: The remote surgical robot comprises: A patient surgical platform and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit and a second shooting unit, the first shooting unit is used to shoot a left-view image, and the second shooting unit is used to shoot a right-view image; A remote doctor console and an image display device connected to each other, wherein the image display device is provided with a first display unit and a second display unit; Wherein, the patient surgery platform and the remote doctor console are communicatively connected; a first remote host and a second remote host, wherein the first remote host is connected to the patient surgery platform, and the second remote host is connected to the remote doctor console; Wherein, the patient surgery platform and the remote doctor console are communicatively connected via the first remote host and the second remote host, and the first remote host includes a first remote port and a second remote port; The processor is configured as: Acquire first device information of the patient surgery platform, second device information of the remote doctor console, and user interaction data; According to the first device information and the second device information, at least one end of the patient surgery platform and the remote doctor console is selected, and based on the user interaction data, an overlay operation is performed in the left view image and the right view image respectively to obtain a left view overlay image and a right view overlay image, wherein if the first device information and the second device information meet the preset first condition, the patient surgery platform transmits the left view image, the right view image and the user interaction data to the remote doctor console through the first remote port; the remote doctor console performs an overlay operation in the left view image and the right view image respectively based on the user interaction data to obtain a left view overlay image and a right view overlay image; if the first device information and the second device information meet the preset second condition, the patient surgery platform performs an overlay operation in the left view image and the right view image respectively based on the user interaction data to obtain a left view overlay image and a right view overlay image, and transmits the left view overlay image and the right view overlay image to the remote doctor console through the second remote port; The left-view overlay image and the right-view overlay image are displayed in the first display unit and the second display unit respectively.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for generating a stereoscopic visual effect applied to a remote surgical robot as described in any one of claims 1 to 8.

Images