Remote surgical robot system

Abstract

The utility model discloses a remote surgical robot system. The system comprises at least one group of first terminals and image acquisition devices which are connected with each other; the second terminal and the image display device are connected with each other. One end is a remote doctor console, and the other end is a patient operation platform; the first terminal comprises: a determination unit, used for determining an equipment relationship with any second terminal in response to successful communication pairing with any second terminal; and the adaptation unit is used for selecting an adaptation strategy corresponding to the equipment relationship from multiple stored adaptation strategies, and carrying out adaptation processing on the first video stream acquired by the image acquisition device and the second video stream displayed by the image display device according to the adaptation strategy. According to the invention, image interconnection of the remote surgical robot is realized, so that different types of master hand ends and slave hand ends can be freely matched.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical devices, in particular to a teleoperation robot system. BACKGROUND

[0002] In a teleoperation robot, the image system is mainly composed of a display at the master end (remote doctor console) and an endoscope at the slave end (patient surgery platform). The master end display and the slave end endoscope of the image system need to be highly coordinated to ensure that the doctor can observe the patient's surgery area collected by the slave end endoscope in real time and accurately on the master end display, and perform surgery operation.

[0003] The master end is the control center for the doctor to perform surgery operation, and the image system provides real-time visual feedback function. The display has single / dual, two-dimensional (2D) / three-dimensional (3D), 2K resolution / 4K resolution parameters, usually including single 3D 2K display and dual 2D 4K display. The slave end is the position where the surgery robot performs actual operation, and the image system is responsible for capturing real-time images of the surgery area and transmitting them to the master end. The endoscope also has single / dual / triple, two-dimensional (2D) / three-dimensional (3D), 2K resolution / 4K resolution parameters, usually including dual 2D 4K endoscope and dual 2D 2K endoscope.

[0004] However, the image system configurations of the master end and the slave end may be different, resulting in compatibility problems. For example: the master end uses a dual 2D 4K display, while the slave end uses a dual 2D 2K endoscope, resulting in a resolution mismatch problem; the master end uses a single 3D 2K display, while the slave end uses a dual 2D 4K endoscope, resulting in a display mode inconsistency problem. Especially during a teleoperation surgery, different types of slave ends are usually controlled by the same master end, or the same slave end is controlled by multiple different types of master ends, which involves the interconnection of image systems with different configurations, and the compatibility problem will cause the image system to delay or even crash at the master end, while reducing the number of matching types of the master end and the slave end, and reducing the applicable surgery scenarios of the master end or the slave end. In summary, the existing various types of teleoperation robots cannot interconnect images to match different types of master ends and slave ends. SUMMARY

[0005] In order to solve the problem that the existing different types of teleoperation robots cannot interconnect images to match different types of master ends and slave ends, the present application proposes the following technical solutions:

[0006] The first aspect of the present application provides a remote surgery robot system, comprising: at least one set of a first terminal and an image acquisition device connected to each other, wherein the first terminal is used to control the image acquisition device to acquire a first video stream of a surgery scene; at least one set of a second terminal and an image display device connected to each other, wherein the second terminal is used to control the image display device to display a second video stream of the surgery scene; wherein one of the first terminal and the second terminal is a remote doctor console, and the other is a patient surgery platform, and the first terminal stores a plurality of adaptation strategies; the first terminal comprises: a determination unit, configured to determine a device relationship with any one second terminal in response to successful communication pairing with the any one second terminal; and an adaptation unit, configured to select an adaptation strategy corresponding to the device relationship from the plurality of stored adaptation strategies, and perform adaptation processing on the image acquisition device acquiring the first video stream and the image display device displaying the second video stream according to the adaptation strategy.

[0007] Preferably, the image acquisition device comprises at least one shooting unit, and the image display device comprises at least one display unit; the device relationship comprises at least one of the following: a quantity relationship of the shooting unit and the display unit, a relationship of shooting dimensions and display dimensions, and a relationship of acquisition resolution and display resolution; and the adaptation strategy comprises at least one of the following: video stream transmission path adaptation between the first terminal and the second terminal, and video stream format adaptation.

[0008] Preferably, the device relationship comprises: a first quantity of shooting units and a second quantity of display units, and a first dimension of shooting and a second dimension of display, wherein the first quantity is greater than the second quantity, and the first dimension is lower than the second dimension; the video stream transmission path in the adaptation strategy comprises: transmitting the video stream between the first terminal and the second terminal through a first path, and the first path comprises a second quantity of transmission links; and the adaptation unit is further configured to: simultaneously acquire a first quantity of first dimension surgery images through a first quantity of shooting units, wherein each video frame in the first video stream comprises the first quantity of first dimension surgery images; synthesize the first quantity of first dimension surgery images into a second quantity of second dimension surgery images, and transmit the second quantity of second dimension surgery images to the second terminal through the second quantity of transmission links in the first path respectively, wherein each video frame in the second video stream comprises the second quantity of second dimension surgery images; and display the second quantity of second dimension surgery images through the second terminal using a second quantity of display units respectively.

[0009] Preferably, the image acquisition device is a two-way two-dimensional endoscope comprising two shooting units, and the image display device is a single-way three-dimensional display comprising one display unit; the two-way two-dimensional endoscope is detachably connected to the patient surgery platform, and is configured to acquire two-way two-dimensional surgery images, each video frame in the first video stream comprising a two-way two-dimensional surgery image; the single-way three-dimensional display is arranged in the remote doctor console, and is configured to display a single-way three-dimensional surgery image synthesized from the two-way two-dimensional surgery images, each video frame in the second video stream comprising a single-way three-dimensional surgery image; the first terminal is provided with a first remote port, and the first path comprises a transmission link formed based on the first remote port, and is configured to transmit the single-way three-dimensional surgery image.

[0010] Preferably, the device relationship comprises a third number of shooting units and display units, and a third dimension of the shooting dimension and the display dimension; the video stream transmission path adaptation in the adaptation strategy comprises that the video stream is transmitted between the first terminal and the second terminal through a second path, and the second path comprises a third number of transmission links; the adaptation unit is further configured to simultaneously acquire a third number of third-dimensional surgery images through the third number of shooting units, wherein each video frame in the first video stream comprises the third number of third-dimensional surgery images; the third number of third-dimensional surgery images are respectively transmitted to the second terminal by using the third number of transmission links in the second path; and the third-dimensional surgery images of each video frame are subjected to time synchronization processing by the second terminal, and the time-synchronized third-dimensional surgery images are respectively displayed by using the third number of display units, wherein each video frame in the second video stream comprises the time-synchronized third number of third-dimensional surgery images.

[0011] Preferably, the image acquisition device is a two-way two-dimensional endoscope comprising two shooting units, and the image display device is a two-way two-dimensional display comprising two display units; the two-way two-dimensional endoscope is detachably connected to the patient surgery platform, and is configured to acquire two-way two-dimensional surgery images, each video frame in the first video stream comprising a two-way two-dimensional surgery image; the two-way two-dimensional display is arranged in the remote doctor console, and is configured to display time-synchronized two-way two-dimensional surgery images, each video frame in the second video stream comprising a time-synchronized two-way two-dimensional surgery image; the first terminal is provided with two second remote ports, and the second path comprises two transmission links formed based on the two second remote ports, and is configured to transmit the two-way two-dimensional surgery images.

[0012] Preferably, the device relationship comprises: a fourth number of the shooting units and the display units, and the shooting dimension and the display dimension are both fourth dimensions; the video stream transmission path adaptation in the adaptation strategy comprises: transmitting the video stream between the first terminal and the second terminal through a third path, the third path comprising a transmission link, wherein the fourth number is greater than 1; the adaptation unit is further configured to: simultaneously collect fourth number of fourth dimension surgical images through the fourth number of shooting units, wherein each video frame in the first video stream comprises fourth number of fourth dimension surgical images; splice the fourth number of fourth dimension surgical images into one fourth dimension surgical image, and transmit the one fourth dimension surgical image to the second terminal by using one transmission link in the third path; split the one fourth dimension surgical image into fourth number of fourth dimension surgical images by using the second terminal, and display the fourth number of fourth dimension surgical images by using the fourth number of display units respectively, wherein each video frame in the second video stream comprises the fourth number of fourth dimension surgical images after splitting; wherein the fourth number of fourth dimension surgical images are spliced into one fourth dimension surgical image for maintaining time synchronization when the fourth number of fourth dimension surgical images are displayed on the fourth number of display units respectively.

[0013] Preferably, the image acquisition device is a double two-dimensional endoscope comprising two shooting units, and the image display device is a double two-dimensional display comprising two display units; the double two-dimensional endoscope is detachably connected to the patient surgery platform, and is used for collecting double two-dimensional surgical images, wherein each video frame in the first video stream comprises double two-dimensional surgical images; the double two-dimensional display is arranged in the remote doctor console, and is used for displaying the double two-dimensional surgical images after splitting, wherein each video frame in the second video stream comprises the double two-dimensional surgical images after splitting; the first terminal is provided with a third remote port, and the third path comprises a transmission link formed based on the third remote port, and is used for transmitting single two-dimensional surgical images.

[0014] Preferably, the device relationship comprises: a first acquisition resolution and a first display resolution, wherein the first acquisition resolution is higher than the first display resolution; the video stream format adaptation in the adaptation strategy comprises a first interpolation strategy; the adaptation unit is further configured to: reduce the first video stream collected by the shooting unit from the first acquisition resolution to the first display resolution by using the first interpolation strategy to obtain a second video stream, and transmit the second video stream to the second terminal; display the second video stream with the first display resolution by using the display unit through the second terminal.

[0015] Preferably, the device relationship comprises a second acquisition resolution and a second display resolution, wherein the second acquisition resolution is lower than the second display resolution; the video stream format adaptation of the adaptation strategy comprises a second interpolation strategy; the adaptation unit is further configured to transmit the first video stream acquired by the shooting unit to the second terminal; and by using the second interpolation strategy, the second terminal is configured to upscale the received first video stream from the second acquisition resolution to the second display resolution, and display the second video stream of the second display resolution by using the display unit.

[0016] The present application has the beneficial effects that: after the image acquisition device (such as an endoscope) of the first terminal (such as a patient surgery platform) acquires a first video stream of a surgery scene, the first video stream is adapted according to the device relationship of the currently connected patient surgery platform and remote doctor console, so that the adjusted second video stream is adapted to the display on the image display device (such as a display) of the second terminal (such as a remote doctor console). The image interconnection of the remote surgery robot is realized, and it is ensured that different types of master and slave hands can be freely matched, which is suitable for more surgery scenes. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a structural schematic diagram of an embodiment of the patient surgery platform of the present application;

[0018] Figure 2 is a structural schematic diagram of an embodiment of the manipulation assembly in the surgery robot of the present application;

[0019] Figure 3 is a structural schematic diagram of another embodiment of the patient surgery platform of the present application;

[0020] Figure 4 is a structural schematic diagram of an embodiment of the remote doctor console of the present application;

[0021] Figure 5 is a schematic diagram of an image interconnection system of an embodiment of the remote surgery robot of the present application;

[0022] Figure 6 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0023] Figure 7 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0024] Figure 8 is a data flow conversion schematic diagram of an embodiment of the image processing method of the present application;

[0025] Figure 9 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0026] Figure 10 is a data flow conversion schematic diagram of an embodiment of the image processing method of the present application;

[0027] Figure 11 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0028] Figure 12 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0029] Figure 13 is a data flow conversion schematic diagram of an embodiment of the image processing method of the present application;

[0030] Figure 14 is a flow schematic diagram of an embodiment of the image processing method of the present application;

[0031] Figure 15 is a data flow conversion schematic diagram of an embodiment of the stereoscopic visual effect generation method of the present application;

[0032] Figure 16 is a flow schematic diagram of an embodiment of the stereoscopic visual effect generation method of the present application;

[0033] Figure 17 is a simple schematic diagram of an embodiment of the stereoscopic visual effect generation method of the present application;

[0034] Figure 18 is a data flow conversion schematic diagram of an embodiment of the stereoscopic visual effect generation method of the present application. DETAILED DESCRIPTION

[0035] In order to facilitate the understanding of the present application, the present application will be described in more detail below in conjunction with the drawings and specific embodiments. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described in the specification. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0036] It should be noted that, unless otherwise defined, all technical and scientific terms used in the specification have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. For example, the term "a plurality of" includes two or more.

[0037] I. Remote surgery robot

[0038] The robot remote surgery technology refers to taking an information network channel as a medical information transmission carrier, adopting a master-slave method, combining robot technology, virtual reality technology, artificial intelligence technology, computer technology, etc. to expand remote diagnosis, remote guidance, remote operation, etc. for medical surgery, so as to realize that a remote doctor controls a surgery robot through an information network channel to perform surgery operation on a patient at a remote end. The remote surgery robot includes master-slave devices, wherein the master operation device is a remote doctor console, and the slave operation device is a patient surgery platform. The doctor performs operation at the remote doctor console and sends a control command to the slave operation device to control the patient surgery platform to perform surgery on the patient.

[0039] The doctor and the remote doctor console are located at a first geographical position, and the patient and the patient surgery platform are located at a second geographical position. Because the first geographical position and the second geographical position are usually far away, such as being in different hospitals, buildings, regions, countries, etc., the remote doctor console and the patient surgery platform transmit surgery process data through an information network channel, such as master-slave control signaling, endoscope video data, audio data, haptic feedback data, and pairing data, etc. interactive data of master-slave devices.

[0040] Specifically, the remote doctor console is provided with a first remote host, and the patient surgery platform is provided with a second remote host. The remote doctor console and the patient surgery platform are connected to the information network channel through the first remote host and the second remote host respectively, to realize transmission of the surgery process data.

[0041] The remote surgery robot can specifically include: a laparoscopic surgery robot (single-hole, multi-hole), a natural orifice surgery robot, a percutaneous puncture surgery robot, an orthopedic surgery robot, a general vascular surgery robot, a dental surgery robot, a neurosurgery robot, etc. surgery robots including the foregoing remote doctor console and patient surgery platform, so that the remote surgery technology can be realized.

[0042] These types of surgery robots usually include two or more arms that are relatively independent and have joint assemblies. The movement of one arm can be used as input to control the associated movement of the other arm. The control method involved in the present disclosure is applicable to a surgery robot having two or more arms and the movement of one arm can cause the movement of the other arm.

[0043] In some embodiments, the surgical robot includes a patient surgery platform and a remote physician console that can manipulate the patient surgery platform, which can also be referred to as a patient surgery platform and a remote physician console in a remote surgery robot. Different patient surgery platforms can be manipulated by the same structure of remote physician console, or can be manipulated by different structures of remote physician console, that is, different patient surgery platforms and different remote physician consoles can be connected to each other, and mapping control can be performed. Further, the specific type of laparoscopic surgery robot can be distinguished according to the structure of the patient surgery platform.

[0044] For example, a laparoscopic surgery robot provided with multiple medical instruments inserted into the patient's body through different surgical channels provided by the same puncture device connected to the patient and corresponding to the patient surgery platform can be referred to as a single-hole laparoscopic surgery robot. For another example, a laparoscopic surgery robot provided with multiple medical instruments inserted into the patient's body through multiple surgical channels provided by multiple puncture devices connected to the patient and corresponding to the patient surgery platform can be referred to as a multi-hole laparoscopic surgery robot. These medical instruments usually include image instruments for providing a field of view and surgical instruments for surgical operations such as cutting and suturing.

[0045] The patient surgery platform at least has an image instrument, and the remote physician console at least has a display device for the physician to remotely observe the anatomical site. Different types of patient surgery platforms can be equipped with different configurations of image instruments, and different types of remote physician consoles can be equipped with different configurations of displays; at the same time, with the iteration and upgrade of the hardware of the patient surgery platform, different configurations of image instruments can also be equipped, and with the iteration and upgrade of the hardware of the remote physician console, different configurations of displays can also be equipped. When the configuration of the image instrument and the configuration of the display are different, the surgical image collected by the image instrument and the display of the display are not compatible, which causes the display of the display to be delayed or even to crash, affecting the progress of the surgery.

[0046] Figure 1 A single-hole laparoscopic surgery robot is illustrated. As shown in Figure 1 In the single-hole laparoscopic surgery robot, the patient surgery platform 100 includes a driving arm, which includes a main arm 110 and a manipulator assembly 120 connected in sequence, both of which have joint assemblies, and the main arm 110 can adjust the position and / or attitude of the manipulator assembly 120. In combination with Figure 2Referring to FIG. 1, a manipulator assembly 120 is shown. The manipulator assembly 120 includes a housing 130 connected to the distal end of a master arm 110, a manipulator 140 housed in the housing 130, and a medical instrument 150 detachably mounted on the manipulator 140. The manipulator 140 includes a plurality of manipulators 140, each of which is housed in the housing 130 and connected to the distal end of the master arm 110 in parallel through the housing 130. Movement of the master arm 110 can cause the plurality of manipulators 140 to move collectively in position and / or attitude. Typically, the manipulator 140 has at least one joint assembly to provide, for example, a feed degree of freedom in an operational space to adjust the depth of insertion of the medical instrument 150 into a patient. The medical instrument 150 can include a plurality of medical instruments 150, which can typically include the same number of medical instruments 150 as the number of manipulators 140 or less. The medical instrument 150 also includes a plurality of joint assemblies, which can provide a plurality of degrees of freedom in the operational space under the drive of the manipulator 140 to achieve, for example, all degrees of freedom other than the feed degree of freedom, including horizontal translation degrees of freedom, vertical translation degrees of freedom, roll degrees of freedom, yaw degrees of freedom, and pitch degrees of freedom. The feed degree of freedom, the horizontal translation degrees of freedom, and the vertical translation degrees of freedom are three positional degrees of freedom of the operational space, and the roll degrees of freedom, the yaw degrees of freedom, and the pitch degrees of freedom are three attitude degrees of freedom of the operational space. Figure 1 In some embodiments, a plurality of medical instruments 150 are inserted into a patient through the same puncture device 160.

[0047] Continuing to refer to FIG. 1, Figure 1 In addition, a natural orifice transluminal surgery robot 300 is shown. The natural orifice transluminal surgery robot 300 shows a patient surgery platform 320, which also includes an imaging cart 330. The patient surgery platform 320 is coupled with a catheter instrument 340, and is coupled with a sensor system 350, and a control system 360 for achieving control between the catheter instrument 340, the sensor system 350, and the imaging cart 330, etc.

[0048] The patient surgery platform 320 can typically be moved to the side of an operating bed for coupling with the catheter instrument 340, and under control instructions, the catheter instrument 340 is controlled to move up and down in a vertical direction, or to move horizontally, or to move in a non-vertical and non-horizontal direction, thereby providing a better preoperative preparation angle for the operation of the catheter instrument 340. The control instructions can be triggered by the doctor operating the patient surgery platform 320, or can be triggered by the doctor directly clicking or pressing the keys provided on the patient surgery platform 320. Of course, in other embodiments, the control instructions can also be voice control or triggered by a force feedback mechanism.

[0049] As Figure 3As shown, further, the patient procedure platform 320 can include a base 321, a sliding carriage 322 that can move up and down along the base 321, and two robotic arms 323 fixedly connected to the sliding carriage 322. The robotic arms 323 can include a plurality of arm segments coupled at joints, which provide the robotic arms 323 with a plurality of degrees of freedom, e.g., seven degrees of freedom corresponding to seven arm segments. The distal end of the robotic arms 323 is equipped with a powered portion (not shown in the figure) for engaging the catheter instrument 340 and controlling the distal end of the catheter instrument 340 to bend and turn accordingly under the driving action of the powered portion. The two robotic arms 323 can be of the same or partially the same structure, one robotic arm 323 for engaging the inner catheter instrument 341 and the other robotic arm 323 for engaging the outer catheter instrument 342. When equipped, the outer catheter instrument 342 can be installed first, and after the outer catheter instrument 342 is installed, the catheter of the inner catheter instrument 341 is inserted into the catheter of the outer catheter instrument 342.

[0050] The sensor system 350 has one or more subsystems for receiving information about the catheter instrument 340. The subsystems can include a position sensor system, a shape sensor system for determining the position, orientation, velocity, speed, pose, and / or shape of the distal end of the catheter instrument 340 and / or along one or more segments of the catheter that can make up the catheter instrument 340, and / or a visualization system for capturing images from the distal end of the catheter instrument 340.

[0051] The imaging cart 330 can be provided with a display system 331 and an irrigation system (not shown in the figure), etc. The display system 331 is used to display images or representations of the surgical site and the catheter instrument 340 generated by the subsystems of the sensor system 350. Real-time images of the surgical site and the catheter instrument 340 captured by the visualization system can also be displayed. Image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, etc. can also be used to present images of the preoperatively or intraoperatively recorded surgical site.

[0052] Among them, preoperative or intraoperative image data can be presented as two-dimensional, three-dimensional or four-dimensional (such as time-based or speed-based information) images and / or as images from models created according to preoperative or intraoperative image data sets, and virtual navigation images can also be displayed. In the virtual navigation image, the actual position of the catheter instrument 340 is registered with the preoperative image to present the virtual image of the catheter instrument 340 within the surgical site to the operator from the outside.

[0053] The control system 360 includes at least one memory and at least one processor. It can be appreciated that the control system 360 can be integrated in the patient surgery platform 320 or the image cart 330, or can be independently arranged. The control system 360 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT, and wireless telemetry, etc. The control system 360 can transmit one or more signals instructing the catheter instrument 340 to move, which are generated by the powered unit to move the catheter instrument 340. The catheter instrument 340 can extend to the surgical position in the body via the opening of the natural cavity of the patient or the surgical incision.

[0054] Further, the control system 360 can include a mechanical control system (not shown in the figure) for controlling the movement of the catheter instrument 340, and thus can be integrated in the patient surgery platform 320. The control system 360 can also include an image processing system (not shown in the figure) for virtual navigation path planning, and thus can be integrated in the image cart 330. Of course, the various subsystems of the control system 360 are not limited to the specific cases listed above, but can be reasonably arranged according to actual conditions.

[0055] The image processing system can use the above imaging techniques to image the surgical site based on the images of the surgical site recorded before or during surgery. Software that can be used in combination with manual input can also be used to convert the recorded images into two-dimensional or three-dimensional composite images of part or the entire anatomical organ or segment. During the virtual navigation program, the sensor system 350 can be used to calculate the position of the catheter instrument 340 relative to the patient's anatomical structure, which can be used to generate external tracking images and internal virtual images of the patient's anatomical structure, to achieve registration of the actual position of the catheter instrument 340 with the preoperative images, so that the operator can be presented with a virtual image of the catheter instrument 340 inside the surgical site from the outside.

[0056] The internal catheter instrument 341 and the external catheter instrument 342 have substantially the same structure, each having an elongated flexible internal catheter 41 and an external catheter 42, wherein the diameter of the external catheter 42 is slightly larger than that of the internal catheter 41, so that the internal catheter 41 can pass through the external catheter 42 and provide certain support for the internal catheter 41, so that the internal catheter 41 can reach the target position in the patient's body to facilitate tissue or cell sampling operations at the target position.

[0057] Figure 3 A multi-hole laparoscopic surgery robot is illustrated. As shown in FIG. 1, the multi-hole laparoscopic surgery robot includes a patient surgery platform 320, an image cart 330, a control system 360, an internal catheter instrument 341, and an external catheter instrument 342. Figure 3As shown, in the multi-hole laparoscopic surgery robot, the patient surgery platform 200 comprises a driving arm, which comprises a main arm 210, an adjusting arm 220 and a manipulator assembly 230 connected in sequence, all of which have joint assemblies. The adjusting arm 220 comprises a plurality of adjusting arms, the proximal ends of which are connected to the distal end of the main arm 210, and the distal end of the main arm 210 comprises a directional platform 215, i.e. the proximal ends of the adjusting arms 220 are connected to the directional platform 215, and the main arm 210 can adjust the position and / or attitude of the adjusting arm 220 and the manipulator assembly 230. The manipulator assembly 230 comprises a plurality of manipulator assemblies, the number of which is the same as the number of the adjusting arms 220, and the proximal ends of the manipulator assemblies 230 are connected to the distal ends of the adjusting arms 220, and the adjusting arms 220 can adjust the position and / or attitude of the manipulator assemblies 230. The manipulator assembly 230 comprises a manipulator 240 and a medical instrument 250 detachably mounted on the manipulator 240, both of which comprise a plurality of joint assemblies. The manipulator 240 comprises a parallelogram mechanism, and by using the parallelogram principle, it can be defined that the manipulator 240 can rotate around a remote center (i.e. Remote Center, RC). The joint assemblies of the manipulator 240 can include a plurality of degrees of freedom in the operation space, including, for example, a feed degree of freedom, a yaw degree of freedom and a pitch degree of freedom, both of which rotate around the RC. The joint assemblies of the medical instrument 250 can provide a plurality of degrees of freedom in the operation space under the drive of the manipulator 240, including, for example, a roll degree of freedom, a yaw degree of freedom and a pitch degree of freedom. Among them, the feed degree of freedom of the manipulator 240 adjusts the depth of the medical instrument 250 inserted into the patient's body, and the yaw degree of freedom and the pitch degree of freedom of the manipulator 240 can affect both the position and the attitude of the medical instrument 250, and the roll degree of freedom, the yaw degree of freedom and the pitch degree of freedom of the medical instrument 250 mainly affect the attitude of the medical instrument 250, and the influence on the position of the medical instrument 250 can be ignored. Figure 2 Among them, a plurality of medical instruments 250 are inserted into the patient's body through different puncture devices 500.

[0058] In some embodiments, as Figure 4 The remote physician consoles 410, 420 and 430 shown can be independently arranged with Figure 1 The patient surgery platform 100 shown or Figure 2 The patient surgery platform 200 shown. They can be arranged in close proximity, for example, arranged in the same room, within a few meters apart; they can also be arranged at a distance, for example, arranged in different cities, thousands of kilometers away.

[0059] Continuing to refer to Figure 4, provide specific device structure examples of different types of remote physician consoles, such as the remote physician console 410 can include an operation part 411 and a display 412, the remote physician console 420 can include an operation part 421 and a display 422, and the remote physician console 430 can include an operation part 431 and a display 432; one remote physician console can include more than one operation part, for example, including two. The operation part 411 can include a plurality of joint assemblies, and the operation part 411 with a plurality of joint assemblies can be regarded as one arm body in the surgical robot; the operation part 431 can also be a handle including a plurality of buttons or twist rods.

[0060] II. Image interconnection system of remote surgical robot

[0061] The remote surgical robot comprises: at least one set of interconnected first terminals and image acquisition devices, wherein the first terminals are used to control the image acquisition devices to collect first video streams of a surgical scene; at least one set of interconnected second terminals and image display devices, wherein the second terminals are used to control the image display devices to display second video streams of the surgical scene; wherein one of the first terminals and the second terminals is a remote physician console, and the other is a patient surgery platform, and the first terminal stores a plurality of adaptation strategies.

[0062] The first terminal comprises: a determination unit, configured to determine a device relationship with any one of the second terminals in response to successful communication pairing with the any one of the second terminals; and an adaptation unit, configured to select an adaptation strategy corresponding to the device relationship from the stored plurality of adaptation strategies, and perform adaptation processing on the image acquisition devices collecting the first video streams and the image display devices displaying the second video streams according to the adaptation strategy.

[0063] In this embodiment, one master hand end of the remote surgical robot or one master hand end comprises a set of interconnected first terminals and image acquisition devices, or a set of interconnected second terminals and image display devices.

[0064] In this embodiment, the first terminal can also be connected with a first remote host, and the second terminal can also be connected with a second remote host, the device relationship can be determined by the first terminal / second terminal, or can be determined by the first remote host / second remote host; in response to successful communication pairing between one of the first remote hosts and one of the second remote hosts, first device information of the first terminal corresponding to one of the first remote hosts and second device information of the second terminal corresponding to one of the second remote hosts are obtained, and a device relationship between the first device information and the second device information is determined. The following describes the first remote host / second remote host as the execution end.

[0065] When the first terminal is a remote doctor console, the image acquisition device includes a camera for shooting the overall scene of the remote doctor console operated by the doctor, a camera for shooting the local scene of the operation part of the remote doctor console operated by the doctor, etc.; when the first terminal is a patient surgery platform, the image acquisition device includes an image instrument (endoscope), a camera (pan-tilt camera) for shooting the overall scene of the patient surgery platform performing surgery operation on the patient, a camera (gun camera) for shooting the local scene of the contact position between the patient surgery platform and the patient, etc. The surgery scene includes the aforementioned overall scene and local scene.

[0066] In this embodiment, when the second terminal is a remote doctor console, the image display device includes a display 412 as shown in the figure, a large screen, an image trolley screen 432, etc.; when the second terminal is a patient surgery platform, the image display device includes a large screen, an image trolley screen 432, etc. Figure 4

[0067] The processor can be arranged in the first terminal or the first remote host, and arranged in the second terminal or the second remote host, and configured to: in response to the communication pairing between one of the first remote hosts and one of the second remote hosts being successful, acquire first device information of the first terminal corresponding to one of the first remote hosts and second device information of the second terminal corresponding to one of the second remote hosts, and determine the device relationship between the first device information and the second device information; and through the first remote host or the second remote host, perform adaptive processing on the second video stream displayed on the image display device based on the device relationship, the second video stream being the first video stream collected by the image acquisition device.

[0068] In this embodiment, after the communication pairing between different first remote hosts and second remote hosts is successful, the first terminal and the second terminal of the corresponding type can interact and control through the communication network. Different types of first terminals are provided with image acquisition devices of different configurations, and different types of second terminals are provided with image display devices of different configurations; when the configurations of the image acquisition device and the image display device are different, adaptive processing is needed to map the first video stream to the second video stream, so that there is no compatibility problem between the image acquisition device and the image display device.

[0069] In this embodiment, the first device information includes at least one of the following: first configuration information, first assembly information, first model information and first type information of the image acquisition device / first terminal, etc.; the second device information includes at least one of the following: second configuration information, second assembly information, second model information and second type information of the image display device / second terminal, etc.

[0070] ​Specifically, the first type information / second type information includes at least one of the following: JINGFENG multi-port endoscopic surgery robot MP1000 / MP2000, JINGFENG single-port endoscopic surgery robot SP1000, JINGFENG natural orifice surgery robot CP1000, da Vinci X / da Vinci Xi of da Vinci multi-port surgery robot, da Vinci SP of da Vinci single-port surgery robot, and TOMEY minimally invasive endoscopic surgery robot.

[0071] Specifically, the first type information / second type information includes at least one of the following: endoscopic surgery robot (single-port, multi-port), natural orifice surgery robot, percutaneous surgery robot, orthopedic surgery robot, general vascular surgery robot, dental surgery robot, neurosurgery robot, and other types of surgery robot information.

[0072] Among them, different models or different types of remote physician consoles or patient surgery platforms are fixedly provided with different configurations of image display devices or image acquisition devices based on system design.

[0073] Specifically, the first configuration information includes at least one of the following: single / dual (image acquisition channel number), two-dimensional (2D) / three-dimensional (3D) (image acquisition dimension), 2K / 4K (image acquisition resolution); second configuration information: single / dual / three (image display channel number), two-dimensional (2D) / three-dimensional (3D) (image display dimension), 2K resolution / 4K resolution (image display resolution).

[0074] In this embodiment, the adaptation processing includes at least one of the following: video stream transmission path adaptation between the first remote host and the second remote host, video stream format adaptation. Among them, the video stream transmission path adaptation includes the adaptation of the image acquisition channel number and the image display channel number mentioned above; the video stream format adaptation includes the adaptation of the image acquisition dimension and the image display dimension mentioned above, the adaptation of the image acquisition resolution and the image display resolution, etc., and specifically includes at least one of the following: image offset processing, image synchronization processing, image scaling processing, image fusion processing, UI superposition processing, etc.

[0075] For example, as shown in FIG. 1, the first remote host 100 and the second remote host 200 are connected through a network 300, and the first remote host 100 and the second remote host 200 are connected through a network 300. Figure 5As shown, it is assumed that the first terminal is a patient surgery platform, such as patient surgery platform A (connected to image acquisition device A), patient surgery platform B (connected to image acquisition device B), and patient surgery platform C (connected to image acquisition device C) of different configurations / types / models; and the second terminal is a remote physician console, such as remote physician console A (connected to image display device A), remote physician console B (connected to image display device B), and remote physician console C (connected to image display device C) of different configurations / types / models. The processor can be specifically arranged in each of the patient surgery platforms and each of the remote physician consoles, and includes a determination unit and an adaptation unit.

[0076] Different patient surgery platforms and different remote physician consoles can be paired for communication through a network channel, such as first remote host A, first remote host B, or first remote host C, each of which can be paired for communication with at least one of second remote host A, second remote host B, and second remote host C.

[0077] Specifically, for example, the first remote host A and the second remote host B are successfully connected in communication: the patient surgery platform A is of the model "Jingfeng single-hole laparoscopic surgery robot SP1000", the type is "multi-hole laparoscopic surgery robot", and the configuration is "dual-path two-dimensional 4K image acquisition device"; the remote physician console B is of the model "da Vinci Xi visual surgical multi-hole laparoscopic surgery robot", the type is "multi-hole laparoscopic surgery robot", and the configuration is "single-path three-dimensional 2K image display device". The equipment relationship of the two includes at least: dual-path-single-path relationship, two-dimensional-three-dimensional relationship, 4K-2K relationship; and the processor in the first remote host A / second remote host B can perform adaptation processing such as image offset processing, image scaling processing, image fusion processing, and UI superposition processing.

[0078] Further, the first remote host and the second remote host include an image processing device (which can be the aforementioned image cart) and a remote communication device, and the processor can be arranged in the image processing device.

[0079] The foregoing introduces the image interconnection system of the remote surgery robot, and the adaptation process of different types of remote physician consoles and patient surgery platforms in image interconnection is described below, please refer to Figure 6 , which provides an embodiment of an image processing method applied to the image interconnection of the remote surgery robot, as shown below:

[0080] 601、in response to successful pairing for communication of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the equipment relationship with the other terminal;

[0081] In the embodiment, the first terminal can also be connected with a first remote host, and the second terminal can also be connected with a second remote host. The device relationship can be determined by the first terminal / second terminal, or can be determined by the first remote host / second remote host. In response to the communication pairing success between one of the first remote hosts and one of the second remote hosts, the first device information corresponding to the first terminal of one of the first remote hosts and the second device information corresponding to the second terminal of one of the second remote hosts are acquired, and the device relationship between the first device information and the second device information is determined. The following is described with the first remote host / second remote host as the execution end.

[0082] In the embodiment, in response to the communication pairing success between one of the first remote hosts and one of the second remote hosts, the first device information corresponding to the first terminal of one of the first remote hosts and the second device information corresponding to the second terminal of one of the second remote hosts are acquired, and the device relationship between the first device information and the second device information is determined. The following is described with the first remote host / second remote host as the execution end.

[0083] In the embodiment, after the communication pairing success, the first terminal connected with the one of the first remote hosts can interactively control the second terminal connected with the one of the second remote hosts, in a remote surgery robot. An image acquisition device and an image display device form an image system, and at least serve as the eyes of a doctor performing a surgery operation to observe an anatomical part, a surgery space, and the like of a patient.

[0084] Specifically, the first device information is used to represent the configuration parameter of the image acquisition device, and the second device information is used to represent the configuration parameter of the image display device. According to the difference between the configuration parameters of the two, the device relationship between the two is determined, such as the aforementioned double path-single path relationship, two-dimensional-three-dimensional relationship, 4K-2K relationship, and the like.

[0085] In the embodiment, after the communication pairing success between the first remote host and the second remote host, the first terminal sends its device information to the first remote host, and the second terminal sends its device information to the second remote host. The first remote host and / or the second remote host can subsequently perform the adaptation processing between the first video stream and the second video stream, so that the compatible display of the second video stream on the image display device is met.

[0086] 602、From the stored multiple adaptation strategies, an adaptation strategy corresponding to the device relationship is selected, and the adaptation processing is performed on the first video stream collected by the image acquisition device and the second video stream displayed by the image display device according to the adaptation strategy.

[0087] In the embodiment, the selection and execution of the adaptation strategy can be processed by the first terminal / second terminal or the first remote host / second remote host; the first remote host or the second remote host performs adaptation processing on the second video stream displayed on the image display device according to the device relationship.

[0088] In the embodiment, according to the device relationship, in addition to the adaptation processing on the first video stream and the second video stream, the first remote host or the second remote host performing the adaptation processing can also be determined. For example, for the device relationship including the two-way-single-way relationship and the two-dimensional-three-dimensional relationship, the image offset processing, image fusion processing and UI superposition processing can be performed in the first remote host; for the device relationship including the two-way-two-way relationship and the two-dimensional-three-dimensional relationship, the image offset processing, image synchronization processing and UI superposition processing can be performed in the second remote host.

[0089] In an embodiment, the image acquisition device includes at least one shooting unit, and the image display device includes at least one display unit; the device relationship includes at least one of the following: the number relationship of the shooting unit and the display unit, the relationship of the shooting dimension and the display dimension, and the relationship of the acquisition resolution and the display resolution; the adaptation strategy includes at least one of the following: the video stream transmission path adaptation between the first terminal and the second terminal, and the video stream format adaptation.

[0090] In the embodiment, the number relationship of the shooting unit and the display unit represents the relationship between the image acquisition channel and the image display channel; the relationship of the shooting dimension and the display dimension represents the relationship between the dimension of the image acquisition device for finally presenting the acquired image and the dimension of the image display device for finally displaying the image; and the relationship of the acquisition resolution and the display resolution represents the relationship between the resolution of the image acquisition device for finally presenting the acquired image and the resolution of the image display device for finally displaying the image.

[0091] In the embodiment, the video stream transmission path adaptation between the first remote host and the second remote host is performed in the first remote host, and different transmission paths are respectively set according to different device relationships; according to the successfully paired device relationship, the corresponding transmission path is matched to transmit the first video stream / second video stream from the first remote host to the second remote host.

[0092] The following provides several preferred embodiments of the image processing method applied to the remote surgical robot image interconnection according to different device relationships of the first terminal and the second terminal, as shown below:

[0093] 1) two-way two-dimensional endoscope-single three-dimensional display

[0094] The aforementioned device relationship includes: a first number of shooting units and a second number of display units, and a shooting dimension of a first dimension and a display dimension of a second dimension, wherein the first number is greater than the second number, and the first dimension is lower than the second dimension; the video stream transmission path in the adaptation strategy includes: transmitting a video stream between the first terminal and the second terminal through a first path, and the first path includes a second number of transmission links;

[0095] Please refer to Figure 7 The specific adaptation process based on the device relationship is as follows:

[0096] 701, in response to the communication pairing success of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the device relationship with the other terminal;

[0097] 702, select an adaptation strategy corresponding to the device relationship from a plurality of stored adaptation strategies;

[0098] 703, simultaneously collect a first number of first dimension surgical images through a first number of shooting units, wherein each video frame in the first video stream includes a first number of first dimension surgical images;

[0099] 704, synthesize the first number of first dimension surgical images into a second number of second dimension surgical images, and transmit the second number of second dimension surgical images to the second terminal respectively by using the second number of transmission links in the first path, wherein each video frame in the second video stream includes a second number of second dimension surgical images;

[0100] 705, by using a second number of display units, the second terminal displays a second number of second dimension surgical images respectively.

[0101] In this embodiment, the image acquisition device acquires a larger number of low-dimensional images, and the image display device displays a smaller number of high-dimensional images; it is indicated that the adaptation process can involve image offset processing, image fusion processing, UI superposition processing, etc. If the first path includes more than one second number of transmission links, image synchronization processing can also be further involved.

[0102] For example, Figure 8As shown, the image acquisition device is a two-way two-dimensional endoscope comprising two shooting units (shooting unit 811, shooting unit 812), which is detachably connected to the patient surgery platform 810 for acquiring two-way two-dimensional surgery images (two-dimensional surgery image 813, two-dimensional surgery image 814), and each video frame in the first video stream comprises a two-way two-dimensional surgery image.

[0103] The image display device is a single-way three-dimensional display comprising a display unit 821; the single-way three-dimensional display is arranged in the remote doctor console 820 for displaying a single-way three-dimensional surgery image (three-dimensional surgery image 822) synthesized from the two-way two-dimensional surgery images, and each video frame in the second video stream comprises a single-way three-dimensional surgery image.

[0104] The first remote host 830 is provided with a plurality of remote ports, and at least comprises a first remote port 831; the first path comprises a transmission link 832 formed based on the first remote port, for transmitting the single-way three-dimensional surgery image.

[0105] Specifically, the patient surgery platform 810 simultaneously acquires two two-dimensional surgery images 813 and 814 through the shooting units 811 and 812 of the two-way two-dimensional endoscope; the first remote host 830 synthesizes the two two-dimensional surgery images 813 and 814 into a three-dimensional surgery image 822, and transmits the three-dimensional surgery image 822 to the second remote host 840 through a transmission link 832 in a first path of a first remote port 831; the second remote host 840 displays the received three-dimensional surgery image 822 in a display unit 821.

[0106] 2) Two-way two-dimensional endoscope-single-way three-dimensional display

[0107] The aforementioned quantity relationship includes: device relationship includes: a third number of shooting units and display units, and the shooting dimension and the display dimension are both third dimensions; the video stream transmission path adaptation in the adaptation strategy includes: transmitting the video stream between the first terminal and the second terminal through a second path, and the second path comprises a third number of transmission links; please refer to Figure 9 The specific adaptation process based on the device relationship is as follows:

[0108] 901, in response to the communication pairing success of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the device relationship with the other terminal;

[0109] 902. Select an adaptation strategy that corresponds to the device relationship from the various stored adaptation strategies;

[0110] 903. Simultaneously acquire a third number of third-dimensional surgical images through a third number of shooting units, wherein each video frame in the first video stream includes a third number of third-dimensional surgical images;

[0111] 904. Using the third number of transmission links in the second path, transmit the third number of third-dimensional surgical images to the second terminal respectively;

[0112] 905. Through the second terminal, the third-dimensional surgical image of each video frame is time-synchronized, and the time-synchronized third-dimensional surgical image is displayed using a third number of display units, wherein each video frame in the second video stream includes a third number of time-synchronized third-dimensional surgical images.

[0113] In this embodiment, the surgical images acquired by the image acquisition device and the surgical images displayed by the image display device are the same in number and dimension. At this time, the adaptation process mainly involves image synchronization processing, so that the surgical images displayed by each display unit of the image display device are at the same moment, presenting the correct high-dimensional effect with multiple low-dimensional surgical images at the same moment.

[0114] For example, such as Figure 10 As shown, the image acquisition device is a dual-channel two-dimensional endoscope containing two imaging units (imaging unit 1011 and imaging unit 1012). The dual-channel two-dimensional endoscope is detachably connected to the patient surgical platform 1010 and is used to acquire dual-channel two-dimensional surgical images (two-dimensional surgical image 1013 and two-dimensional surgical image 1014). Each video frame in the first video stream includes dual-channel two-dimensional surgical images.

[0115] The image display device is a dual-channel two-dimensional display containing two display units (display unit 1021 and display unit 1022); the dual-channel two-dimensional display is set in the remote doctor console 1020 and is used to display time-synchronized dual-channel two-dimensional surgical images (two-dimensional surgical image 1023 and two-dimensional surgical image 1024), and each video frame in the second video stream includes time-synchronized dual-channel two-dimensional surgical images.

[0116] The first remote host 1030 is provided with multiple remote ports, including at least two second remote ports 1031 and 1032. The second path includes two transmission links (transmission link 1033 and transmission link 1034) formed based on the two second remote ports, which are used to transmit the dual-channel two-dimensional surgical images.

[0117] Specifically, the patient surgery platform 1010 collects two two-dimensional surgery images 1013 and 1014 through the shooting units 1011 and 1012 of the dual two-dimensional endoscope; the two two-dimensional surgery images 1013 and 1014 are transmitted to the second remote host 1040 through the two transmission links 1033 and 1034 formed by the second remote ports 1031 and 1032 of the first remote host 1030; the second remote host 1040 synchronizes the received two two-dimensional surgery images 1013 and 1014 to obtain time-matched two two-dimensional surgery images 1041 and 1042, and displays the two two-dimensional surgery images 1041 and 1042 on the two display units 1021 and 1022.

[0118] Further, since the two two-dimensional surgery images 1041 and 1042 need to form a three-dimensional display effect when displayed on the two display units 1021 and 1022, time synchronization needs to be performed to ensure that the two two-dimensional surgery images 1041 and 1042 are acquired at the same time.

[0119] 3) dual two-dimensional endoscope-single three-dimensional display

[0120] The dual two-dimensional surgery images include a first two-dimensional surgery image and a second two-dimensional surgery image of each video frame; please refer to Figure 11 The specific adaptation process based on the device relationship is as follows:

[0121] 1101, in response to the communication pairing success of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the device relationship with the other terminal;

[0122] 1102, select an adaptation strategy corresponding to the device relationship from the stored multiple adaptation strategies; 1103, identify the first time mark of the first two-dimensional surgery image and the second time mark of the second two-dimensional surgery image of each video frame through the second remote host, and align the first time mark and the second time mark;

[0123] In this embodiment, the video stream collected by the two shooting units of the patient surgery platform has a time stamp in the first two-dimensional surgery image and the second two-dimensional surgery image of each video frame, and the second remote host identifies the time stamp (first time mark and second time mark) from the first two-dimensional surgery image and the second two-dimensional surgery image, and performs time alignment based on the time stamp; if the time stamps are the same, it is considered that the first two-dimensional surgery image and the second two-dimensional surgery image collected by the two shooting units are at the same time.

[0124] In addition, when it is detected that the remote surgery robot is busy, such as the first remote host and the second remote host simultaneously performing motion control, two-way video merging (merging of the first two-dimensional surgery image and the second two-dimensional surgery image), video encoding, data sending, and the like, the transmitted first two-dimensional surgery image and the second two-dimensional surgery image can have problems such as out-of-order and packet loss. A sliding window mechanism is used, the video frames in the window are sorted according to the time stamp, and the window size is dynamically adjusted according to the current network state to balance the delay and out-of-order problems. At the same time, a partial retransmission mechanism is used to handle the packet loss problem. When the time stamps of two adjacent video frames sorted are obviously greater than the acquisition frequency of the video frames, the patient surgery platform is required to detect the time stamp and resend the lost video frames.

[0125] 1104、If the alignment result meets the preset error, time synchronization is performed on each of the first two-dimensional surgery image and each of the second two-dimensional surgery image according to the alignment result;

[0126] In this embodiment, when the clock jitter occurs in the first remote host / patient surgery platform, and the jitter degree is higher than the threshold, the alignment result of the first time mark and the second time mark will be inaccurate. When the alignment result does not exceed the preset error, time alignment is still performed according to the time stamp (the first time mark and the second time mark). If the alignment result exceeds the preset error, the first two-dimensional surgery image and the second two-dimensional surgery image can affect the three-dimensional visual effect formed because they display pictures of different surgery scenes. At this time, time synchronization is needed based on the actual display of the surgery scene pictures of the two.

[0127] 1105、If the alignment result does not meet the preset error, the first target motion trajectory and the first surgery background image in each video frame of the first two-dimensional surgery image, and the second target motion trajectory and the second surgery background image in the second two-dimensional surgery image are extracted;

[0128] In this embodiment, the objects of the first target motion trajectory and the second target motion trajectory are the same, such as a target anatomical tissue, a medical instrument, a preset motion target arranged on the medical instrument / target anatomical tissue, and the like. The first surgery background image represents a region in the first two-dimensional surgery image except the first target motion trajectory, and the second surgery background image represents a region in the second two-dimensional surgery image except the second target motion trajectory.

[0129] 1106、Matching feature points in each of the first surgery background image and each of the second surgery background image; and matching each of the first two-dimensional surgery image and each of the second two-dimensional surgery image according to the feature points, the first target motion trajectory, and the second target motion trajectory;

[0130] In the embodiment, the basis matrix of the feature point matching between the first surgical background image and the second surgical background image is calculated by a normalized eight-point algorithm; and the time-related trajectory point matching of the first target motion trajectory and the second target motion trajectory is realized by combining a preset epipolar geometry constraint algorithm and the basis matrix, representing the matching relationship corresponding to the first two-dimensional surgical image and the second two-dimensional surgical image.

[0131] The time-related trajectory point matching set of the first target motion trajectory and the second target motion trajectory is calculated, and the implementation steps are as follows: (1) taking one of the first two-dimensional surgical image and the second two-dimensional surgical image as a reference image and the other as a to-be-synchronized image, for each trajectory point q1(t i ) in the first target motion trajectory of the reference image, the epipolar line I i of each trajectory point q1(t q1(ti) ) in the to-be-synchronized image is calculated according to the following formula: I q1(ti) =F 12 ·q1(t i ), wherein F12 is the aforementioned basis matrix; (2) obtaining the intersection of the epipolar line I q1(ti) and the second target motion trajectory in the to-be-synchronized image as a trajectory point pair matching set.

[0132] 1107、According to the matching result, the first time line of each of the first two-dimensional surgical images and the second time line of each of the second two-dimensional surgical images are fitted;

[0133] In the embodiment, when the fitting of the first time line and the second time line is performed, the trajectory point pair matching set obtained based on the trajectory point pair matching set may contain some incorrect trajectory point pairs. In order to achieve more accurate results, the trajectory point pair matching set is fitted to obtain the time model parameters between the reference image and the to-be-synchronized image: the frame rate ratio α (the frame rate ratio between the two shooting units) and the time offset ∆t (the time difference between the start of shooting of the two shooting units). The aforementioned time line model is the first time line and the second time line.

[0134] (1) two points are randomly selected from the trajectory point pair matching set, and the time model parameters: the frame rate ratio α and the time offset ∆t are calculated;

[0135] (2) the error is calculated by substituting the trajectory point pair matching set into the straight line model: t`=α*t+∆t, wherein, t and t' are the frame numbers of the reference image and the to-be-synchronized image, respectively. If the error is less than or equal to a preset threshold d, it becomes a valid trajectory point and is recorded in the valid data set;

[0136] (3) repeating steps (1)-(2) for k times, comparing the effective track points of the effective data sets, outputting the effective data set with the most effective track points, and finally estimating the final time model parameters: frame rate ratio a and time offset ∆t based on the output effective data set, to determine the first timeline and the second timeline.

[0137] 1108、based on the first timeline and the second timeline, time synchronizing each of the first two-dimensional surgical images and each of the second two-dimensional surgical images.

[0138] Finally, based on the first timeline and the second timeline, synchronously outputting the first two-dimensional surgical images and the second two-dimensional surgical images in the video frames at the same time point.

[0139] 4) dual two-dimensional endoscope-single three-dimensional display

[0140] The aforementioned device relationship includes: a fourth number of shooting units and display units, and the shooting dimension and the display dimension are both fourth dimensions; the video stream transmission path adaptation in the adaptation strategy includes: transmitting the video stream between the first terminal and the second terminal through a third path, and the third path includes a transmission link, wherein the fourth number is greater than 1; please refer to Figure 12 , the specific adaptation process based on the device relationship is as follows:

[0141] 1201, in response to the communication pairing success of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the device relationship with the other terminal;

[0142] 1202, selecting an adaptation strategy corresponding to the device relationship from a plurality of stored adaptation strategies; 1203, simultaneously collecting a fourth number of fourth-dimensional surgical images through a fourth number of shooting units, wherein each video frame in the first video stream includes a fourth number of fourth-dimensional surgical images;

[0143] 1204, splicing the fourth number of fourth-dimensional surgical images into one fourth-dimensional surgical image, and transmitting the one fourth-dimensional surgical image to the second terminal using a transmission link in the third path;

[0144] 1205, using the second terminal, splitting the one fourth-dimensional surgical image into a fourth number of fourth-dimensional surgical images, and displaying the fourth number of fourth-dimensional surgical images using a fourth number of display units, respectively, wherein each video frame in the second video stream includes the fourth number of fourth-dimensional surgical images after splitting.

[0145] In the embodiment, the fourth number of fourth-dimension surgery images are spliced into one fourth-dimension surgery image, which is used to keep time synchronization when the fourth number of fourth-dimension surgery images are displayed respectively on the fourth number of display units.

[0146] As shown in the example, Figure 13 The image acquisition device is a double two-dimensional endoscope including two shooting units (shooting unit 1311, shooting unit 1312), which is detachably connected to the patient surgery platform 1310, and used to acquire double two-dimensional surgery images (two-dimensional surgery image 1313, two-dimensional surgery image 1314). Each video frame in the first video stream includes double two-dimensional surgery images.

[0147] The image display device is a double two-dimensional display including two display units (display unit 1321, display unit 1322), which is arranged in the remote doctor console 1320, and used to display the split double two-dimensional surgery images (two-dimensional surgery image 1323, two-dimensional surgery image 1324). Each video frame in the second video stream includes split double two-dimensional surgery images.

[0148] The first remote host 1330 is provided with a plurality of remote ports, at least including a third remote port 1331. The third path includes a transmission link 1332 formed based on the third remote port 1331, which is used to transmit a single two-dimensional surgery image 1333.

[0149] Specifically, the patient surgery platform 1310 simultaneously acquires two two-dimensional surgery images 1313, 1314 through the shooting units 1311 and 1312 of the double two-dimensional endoscope. The first remote host 1330 transmits the single two-dimensional surgery image 1333 obtained by splicing the two two-dimensional surgery images 1313, 1314 to the second remote host 1340 through the transmission link 1332 formed by the third remote port 1331. The second remote host 1340 re-splits the single two-dimensional surgery image 1333 into two two-dimensional surgery images 1323, 1324, and synchronously displays them on the two display units 1321 and 1322.

[0150] The splicing of the two two-dimensional surgery images 1313, 1314 ensures that they are always surgery images acquired at the same time during the transmission process of the first remote host 1330 and the second remote host 1340.

[0151] In an embodiment, the resolution relationship comprises a shooting unit of a first acquisition resolution and a display unit of a first display resolution, wherein the first acquisition resolution is higher than the first display resolution; the video stream format adaptation comprises a first interpolation strategy; in an embodiment, the resolution relationship comprises a shooting unit of a second acquisition resolution and a display unit of a second display resolution, wherein the second acquisition resolution is lower than the second display resolution; the video stream format adaptation comprises a second interpolation strategy; see the following for specific adaptation processing flow based on different resolution relationships: Figure 14

[0152] 1401, in response to successful communication pairing of any one first terminal and any one second terminal, one of the any one first terminal and the any one second terminal determines the device relationship with the other terminal;

[0153] 1402, select an adaptation strategy corresponding to the device relationship from a plurality of stored adaptation strategies;

[0154] 1403, using the first interpolation strategy, reduce the first video stream collected by the shooting unit from the first acquisition resolution to the first display resolution to obtain a second video stream, and transmit the second video stream to the second terminal;

[0155] 1404, by the second terminal, using the display unit to display the second video stream of the first display resolution; or,

[0156] 1405, transmit the first video stream collected by the shooting unit to the second terminal;

[0157] 1406, by the second terminal, using the second interpolation strategy, the received first video stream is promoted from the second acquisition resolution to the second display resolution, and the second video stream of the second display resolution is displayed by using the display unit.

[0158] For example, the first acquisition resolution is 2K, the first display resolution is 4K; the second acquisition resolution is 4K, and the second display resolution is 4K; the first interpolation strategy and the second interpolation strategy can be realized by a bilinear interpolation algorithm.

[0159] Wherein, when the first acquisition resolution is higher than the first display resolution, the first interpolation strategy is executed on the first remote host to reduce the resolution of the first video stream, and when the second acquisition resolution is lower than the second display resolution, the second interpolation strategy is executed on the second remote host to increase the resolution of the first video stream, so as to reduce the data amount of the first remote host and the second remote host in transmitting the first video stream or the second video stream. ​

[0160] III. Stereoscopic visual effect generation of remote surgery robot interaction interface

[0161] As shown in Figure 15 The remote surgery robot comprises: a patient surgery platform 1510 and an image acquisition device connected to each other, wherein the image acquisition device is provided with a first shooting unit 1511 and a second shooting unit 1512, the first shooting unit 1511 is used for shooting a left view image 1513, and the second shooting unit 1512 is used for shooting a right view image 1514; a remote doctor console 1520 and an image display device connected to each other, wherein the image display device is provided with a first display unit 1521 and a second display unit 1522; wherein the patient surgery platform 1510 and the remote doctor console 1520 are in communication connection.

[0162] a processor configured to: acquire first device information of the patient surgery platform 1510, second device information of the remote doctor console 1520, and user interaction data (corresponding to the display effect such as the image 1515); according to the first device information and the second device information, select at least one end of the patient surgery platform 1510 and the remote doctor console 1520, and based on the user interaction data (corresponding to the display effect such as the image 1515), perform superposition operation in the left view image 1513 and the right view image 1514 respectively to obtain a left view superimposed image 1523 and a right view superimposed image 1524; display the left view superimposed image 1523 and the right view superimposed image 1524 in the first display unit 1521 and the second display unit 1522 respectively.

[0163] In this embodiment, the image acquisition device comprises a binocular endoscope, a binocular gun camera or a binocular holder camera, and the image display device comprises a binocular display (such as the display 412) provided on the remote doctor console, a binocular display large screen, an image trolley binocular display screen (such as the screen 432) and the like. The doctor can observe the stereoscopic visual effect formed on the display device when the user interaction data is in the left view image and the right view image by observing the image display device at the set position of the remote doctor console.

[0164] The first device information in the embodiment can also include first configuration information, first assembly information, first model information and first type information, etc., wherein the first configuration information includes whether an image processor is arranged in the patient surgery platform and the number of image processors arranged. The second device information in the embodiment can also include second configuration information, second assembly information, second model information and second type information, etc., wherein the second configuration information includes whether an image processor is arranged in the remote doctor console and the number of image processors arranged. The first model information and the first type information and the second model information and the second type information herein have the same specific meanings as the foregoing.

[0165] Further, based on whether an image processor is arranged in the patient surgery platform and the number of image processors arranged, and whether an image processor is arranged in the remote doctor console and the number of image processors arranged, the transmission mode of the user interaction data, the left view image and the right view image, and the superposition position of the user interaction data on the left view image and the right view image are determined. The hardware design of different types of patient surgery platforms and remote doctor consoles is adapted.

[0166] The image interconnection system of the remote surgery robot is introduced above, and the adaptation process of different types of remote doctor consoles and patient surgery platforms in image interconnection is described below, referring to Figure 16 , an embodiment of an image processing method applied to image interconnection of a remote surgery robot is provided, which is specifically shown as follows:

[0167] 1601, obtaining first device information of the patient surgery platform, second device information of the remote doctor console and user interaction data;

[0168] 1602, selecting at least one end of the patient surgery platform and the remote doctor console according to the first device information and the second device information, and performing superposition operation in the left view image and the right view image based on the user interaction data respectively to obtain a left view superposition image and a right view superposition image;

[0169] 1603, displaying the left view superposition image and the right view superposition image in the first display unit and the second display unit respectively.

[0170] In the embodiment, the remote surgery robot further includes a first remote host and a second remote host, the first remote host is connected with the patient surgery platform, and the second remote host is connected with the remote doctor console; wherein the patient surgery platform and the remote doctor console are communicatively connected through the first remote host and the second remote host, and the first remote host includes a first remote port and a second remote port.

[0171] If the first device information and the second device information satisfy a preset first condition, the patient surgery platform transmits the left view image, the right view image and the user interaction data to the remote doctor console through the first remote port; and the remote doctor console performs superimposition operation on the left view image and the right view image based on the user interaction data, to obtain a left view superimposed image and a right view superimposed image.

[0172] For example, the first condition includes that the patient surgery platform is provided with at least one image processing device for performing superimposition operation, or is not provided with an image processing device, and the remote doctor console is provided with at least one image processing device. When the patient surgery platform is provided with at least one image processing device for performing superimposition operation, the at least one image processing device is used to perform offset processing on the left view image and the right view image, to form a preset parallax, so that the left view image and the right view image are subsequently displayed on the first display unit and the second display unit, to form a stereoscopic visual effect. When the patient surgery platform is not provided with an image processing device, the left view image and the right view image are not subjected to offset processing, and are original images collected by the first shooting unit and the second shooting unit.

[0173] If the first device information and the second device information satisfy a preset second condition, the patient surgery platform performs superimposition operation on the left view image and the right view image based on the user interaction data, to obtain a left view superimposed image and a right view superimposed image, and transmits the left view superimposed image and the right view superimposed image to the remote doctor console through the second remote port.

[0174] For example, the second condition includes that the patient surgery platform is provided with at least one image processing device for performing superimposition operation, and the remote doctor console is not provided with an image processing device. At this time, the image processing device provided by the patient surgery platform for performing superimposition operation simultaneously completes superimposition operation of the user interaction data on the left view image and the right view image, and offset processing of the left view image and the right view image; and the remote doctor console is only responsible for displaying the left view superimposed image and the right view superimposed image.

[0175] In an embodiment, when superimposing the user interaction data onto the left view image and the right view image, a user interaction graphic is generated based on the user interaction data, and first assembly information in the first device information and second assembly information in the second device information are extracted; according to the first assembly information, a parallax of the user interaction graphic when superimposed on the left view image and the right view image respectively is calculated; the parallax is corrected according to the second assembly information, and a first superimposition position of the user interaction graphic on the left view image and a second superimposition position on the right view image are determined based on the corrected parallax; the user interaction graphic is superimposed on the first superimposition position of the left view image to obtain a left view superimposed image, and the user interaction graphic is superimposed on the second superimposition position of the right view image to obtain a right view superimposed image.

[0176] In the embodiment, the user interaction data is generated based on the user's interactive operation on the remote surgery robot. The user interaction graphic includes a first part that overlaps when superimposed on the left view image and the right view image, and a second part that does not overlap, wherein the first part includes a trajectory graphic, and the second part includes a status icon. The status icon includes a static icon and a dynamic icon, the static icon is switched according to its own preset interaction logic based on user operation, and the dynamic icon is automatically switched based on the state of at least one patient surgery platform; the trajectory graphic is a custom graphic generated based on user operation trajectory.

[0177] The dynamic icon is determined based on the remote doctor console and different patient surgery platforms / mechanical arms with different control permissions, at least one icon is determined, and one of the icons is superimposed. The static icon is an icon that is fixedly superimposed based on the interaction logic.

[0178] In the embodiment, the first assembly information includes assembly errors of the first shooting unit and the second shooting unit, and the second assembly information includes assembly errors of the first display unit and the second display unit. When the user interaction icon is superimposed on the left view image and the right view image, the assembly of the first shooting unit and the second shooting unit on each patient surgery platform is adapted, the parallax error generated by the two when collecting surgery images relative to the standard parallax is determined, and the parallax of the user interaction graphic when superimposed on the left view image and the right view image respectively is obtained. At the same time, the assembly of the first display unit and the second display unit on each remote doctor console is adapted, the parallax error generated by the two when displaying surgery images relative to the aforementioned parallax is determined, and the corrected parallax is obtained. Accurate stereoscopic visual effect is presented.

[0179] Further, the user interaction figure is preset with a first standard superimposition position in the left view image and a second standard superimposition position in the right view image; after determining the corrected parallax, a first moving direction of the user interaction figure relative to the first standard superimposition position in the left view image and a second moving direction of the user interaction figure relative to the second standard superimposition position in the right view image are determined; the first standard superimposition position is adjusted by the first moving direction to obtain a first superimposition position, and the second standard superimposition position is adjusted by the second moving direction to obtain a second superimposition position.

[0180] As shown in the figure, Figure 17 The first shooting unit and the second shooting unit respectively shoot the left view image 1711 and the right view image 1712, and generate the user interaction figure 1713 and the user interaction figure 1714 based on the user interaction data; after determining the final parallax based on the first assembly information and the second assembly information, the first superimposition position of the user interaction figure in the left view image 1711 and the second superimposition position of the user interaction figure in the right view image 1712 are determined; in the left view image 1711, the user interaction figure 1713 and the user interaction figure 1714 are offset to the left based on the parallax to obtain the first superimposition position (the user interaction figure 1721 and the user interaction figure 1722); in the right view image 1712, the user interaction figure 1713 and the user interaction figure 1714 are offset to the right based on the parallax to obtain the second superimposition position (the user interaction figure 1731 and the user interaction figure 1732); the user interaction figure 1713 and the user interaction figure 1714 are superimposed to the first superimposition position of the left view image 1711 to obtain the left view superimposed image 1720, and the user interaction figure is superimposed to the second superimposition position of the right view image 1712 to obtain the right view superimposed image 1730.

[0181] In addition, as shown in the figure, Figure 18 The image acquisition device 1810 and the first remote port 1821 and the second remote port 1822 of the first remote host 1820 are connected, the graphical user interface 1830 and the second remote port 1822 are connected, and at least one of the left view image 1813 and the right view image 1814 is acquired and displayed from the first display unit 1811 and the second display unit 1812 of the patient surgery platform;

[0182] In an embodiment, in response to a sliding operation in the left view image 1813 and the right view image 1814, the trajectory figure is superimposed on the left view image 1813 and the right view image 1814 through the graphical user interface 1830, left view superimposed image 1831 and right view superimposed image 1832 are generated respectively, and output to the second remote port 1822;

[0183] In an embodiment, in response to generation of the user interaction data corresponding to the status icon, the patient surgery platform outputs the left view image 1813, the right view image 1814 and the user interaction data corresponding to the status icon to the first remote port; the patient surgery platform can also superimpose the status icon directly on the left view image 1813 and the right view image 1814 to generate left view superimposed images and right view superimposed images respectively and output to the second remote port 1822.

[0184] In this embodiment, when the left view image and the right view image are labeled on the graphical user interface of the patient surgery platform (such as a sliding operation), the left view image and the right view image are first transmitted to the graphical user interface for display, and then the superimposed trajectory graphics are directly completed, output to the second remote port of the first remote host, and displayed directly on the remote physician console.

[0185] For other interactive operations other than labeling, the user interaction data of the patient surgery platform and the left view image and the right view image collected by the image collection device can be transmitted to the first remote host respectively, and the superposition of the user interaction data on the left view image and the right view image is completed in the first remote host / second remote host, at which time there is no need to transmit to the graphical user interface.

[0186] In an embodiment of the patient surgery platform generating user interaction data, in response to an interactive operation between the remote physician console and the patient surgery platform, the patient surgery platform determines an interaction state corresponding to the interactive operation, and the user interaction data includes the interaction state; wherein the user interaction graphics include a first status icon corresponding to the interaction state, and the interaction state includes a connection state, a pairing state and a network state between the remote physician console and the patient surgery platform.

[0187] In this embodiment, the interactive operation between the remote physician console and the patient surgery platform includes connection operations, pairing operations and network communication operations, etc., and for connection processes, pairing processes, network communication processes, and / or connection results, pairing results, network communication results, etc., the corresponding interaction states are prompted; wherein the first status icon can be represented by corresponding text or symbolic icon.

[0188] Specifically, the first status icon is a dynamic icon, which is switched to a preset icon corresponding to the connection process, the pairing process, the network communication process, and / or the connection result, the pairing result, the network communication result based on the connection operation, the pairing operation or the network communication operation.

[0189] In one embodiment of generating the user interaction data by the patient surgery platform, the patient surgery platform further comprises a graphical user interface, the graphical user interface comprises at least one trigger control; in response to a touch operation on the trigger control, the patient surgery platform determines a touch state corresponding to the touch operation, and the user interaction data comprises the touch state; wherein the user interaction image comprises a second state icon corresponding to the touch state.

[0190] In this embodiment, according to the interaction logic of the user on the graphical user interface, at least one level of trigger control is preset, and the user performs a touch operation such as a pressing operation on the trigger control on the graphical user interface, and the patient surgery platform triggers a corresponding touch operation. According to the interaction logic, the user interaction image is switched to a second state icon corresponding to the touch state, and the second state icon comprises at least one of the following: a medical image icon, an on / off microphone icon, a volume adjustment icon, a menu icon, a camera icon, a brush icon, etc.

[0191] Specifically, the second state icon is a static icon, and based on the user performing a touch operation on the trigger control on the graphical user interface, the first state icon corresponding to the touch state under the interaction logic is switched.

[0192] In one embodiment of generating the user interaction data by the patient surgery platform, the graphical user interface further comprises a labeling area, and the labeling area contains at least part of the left view image or the right view image; in response to a sliding operation on the labeling area, a sliding track of the sliding operation on the left view image and the right view image is determined, and the user interaction data comprises the sliding track; wherein the user interaction image comprises a track image corresponding to the sliding track.

[0193] In this embodiment, the left view image and / or the right view image are displayed on the graphical user interface, and the doctor can make annotations on the left view image or the right view image on the patient surgery platform to generate a sliding track, and the track image is superimposed on the left view image and the right view image, and displayed on the image display device of the patient surgery platform; for example, the target tissue position in the annotated image,

[0194] In one embodiment of generating user interaction data in a patient surgery platform, the remote surgery robot comprises a first remote physician console and a left view image display device connected to each other, and a second remote physician console and a right view image display device connected to each other; the patient surgery platform comprises at least a first mechanical arm and a second mechanical arm, wherein the first remote physician console has control authority over the first mechanical arm, and the second remote physician console has control authority over the second mechanical arm; the user interaction graphics comprise first user interaction graphics and second user interaction graphics, the first user interaction graphics being displayed on the left view image display device, and the second user interaction graphics being displayed on the right view image display device.

[0195] In this embodiment, in response to the first remote physician console activating the control authority over the first mechanical arm, and the second remote physician console activating the control authority over the second mechanical arm, the patient surgery platform generates user interaction data according to the control state of the first mechanical arm and the control state of the second mechanical arm; at this time, the user interaction data is not directly synchronized to the image display devices of the two remote physician consoles for synchronous display, but different user interaction data is respectively sent to the first remote physician console and the second remote physician console according to the control authority, so as to respectively display the first user interaction graphics and the second user interaction graphics.

[0196] In one embodiment of generating user interaction data in a patient surgery platform, in response to a first control operation of the first remote physician console on the first mechanical arm, the patient surgery platform determines a first control state corresponding to the first control operation, and the user interaction data comprises the first control state, wherein the first user interaction graphics comprise a third state icon corresponding to the first control state; or, in response to a second control operation of the second remote physician console on the second mechanical arm, the patient surgery platform determines a second control state corresponding to the second control operation, and the user interaction data comprises the second control state, wherein the second user interaction graphics comprise a fourth state icon corresponding to the second control state.

[0197] In addition, when one remote physician console controls different mechanical arms, state icons of control states of different mechanical arms can also be displayed respectively. The control state can include a function state, such as a sleeve state, a haptic feedback state, a remote center of motion dynamic control state, etc. It can also include an instrument connection state, such as an endoscope connection state, an energy instrument connection state, a suture instrument connection state, a closure instrument connection state, a cutting instrument connection state, etc.

[0198] It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should also be within the protection scope of the present application, and will not be described here.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional units or modules according to needs, that is, the internal structure of the mobile terminal is divided into different functional units or modules to complete all or part of the functions described above. Each functional module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific name of each functional module is only for the convenience of mutual differentiation, and does not serve to limit the protection scope of the present application. The specific working process of the module in the mobile terminal can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0200] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in each method embodiment.

[0201] The embodiment of the present application provides a computer program product, when the computer program product is run on a mobile terminal, so that the mobile terminal executes to realize the steps in each method embodiment.

[0202] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0203] It should also be understood that the term "and / or" as used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0204] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if a described condition or event is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting the described condition or event" or "in response to detecting the described condition or event" depending on the context.

[0205] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0206] In the present application, the reference to "one embodiment" or "some embodiments" or the like means that the particular feature, structure or characteristic described in connection with this embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments" and so on, in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specifically stated. The terms "comprising", "comprises", "including", "includes" and "having" and their variants are meant to be equivalent to "including but not limited to", unless otherwise specifically stated.

[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for description, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0208] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0209] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0210] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0211] If the integrated module / unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0212] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A tele-surgical robotic system, comprising: The remote surgery robot system comprises: At least one set of first terminal and image acquisition device connected to each other, wherein the first terminal is used to control the image acquisition device to acquire a first video stream of a surgery scene; At least one set of second terminal and image display device connected to each other, wherein the second terminal is used to control the image display device to display a second video stream of the surgery scene; Wherein one of the first terminal and the second terminal is a remote physician console, and the other is a patient surgery platform, and the first terminal stores a plurality of adaptation strategies; The first terminal comprises: A determination unit for determining a device relationship with any one of the second terminals in response to a successful communication pairing with the any one of the second terminals; An adaptation unit for selecting an adaptation strategy corresponding to the device relationship from the plurality of stored adaptation strategies, and performing adaptation processing on the image acquisition device to acquire the first video stream and the image display device to display the second video stream according to the adaptation strategy.

2. The tele-surgical robotic system of claim 1, wherein, The image acquisition device comprises at least one shooting unit, and the image display device comprises at least one display unit; The device relationship comprises at least one of the following: a quantity relationship between the shooting unit and the display unit, a relationship between a shooting dimension and a display dimension, and a relationship between an acquisition resolution and a display resolution; The adaptation strategy comprises at least one of the following: video stream transmission path adaptation between the first terminal and the second terminal, and video stream format adaptation.

3. The tele-surgical robotic system of claim 2, wherein, The device relationship comprises a first number of shooting units and a second number of display units, and a shooting dimension of a first dimension and a display dimension of a second dimension, wherein the first number is greater than the second number, and the first dimension is lower than the second dimension; The video stream transmission path in the adaptation strategy comprises: transmitting the video stream between the first terminal and the second terminal through a first path, and the first path comprises a second number of transmission links; The adaptation unit is further configured to: Acquire a first number of first dimension surgery images simultaneously through a first number of shooting units, wherein each video frame in the first video stream comprises a first number of first dimension surgery images; Synthesize the first number of first dimension surgery images into a second number of second dimension surgery images, and transmit the second number of second dimension surgery images to the second terminal respectively through a second number of transmission links in the first path, wherein each video frame in the second video stream comprises a second number of second dimension surgery images; Display the second number of second dimension surgery images respectively through a second number of display units in the second terminal.

4. The tele-surgical robotic system of claim 3, wherein, The image acquisition device is a double-path two-dimensional endoscope comprising two shooting units, and the image display device is a single-path three-dimensional display comprising one display unit; The double-path two-dimensional endoscope is detachably connected to the patient surgery platform and is used to acquire double-path two-dimensional surgery images, and each video frame in the first video stream comprises double-path two-dimensional surgery images; The double-path two-dimensional endoscope is detachably connected to the patient surgery platform and is used to acquire double-path two-dimensional surgery images, and each video frame in the first video stream comprises double-path two-dimensional surgery images; The single-path three-dimensional display is arranged in the remote physician console and is configured to display a single-path three-dimensional surgery image synthesized from the two-path two-dimensional surgery image, each video frame in the second video stream comprising a single-path three-dimensional surgery image; The first terminal is provided with a first remote port, and the first path comprises a transmission link formed based on the first remote port and configured to transmit the single-path three-dimensional surgery image.

5. The tele-surgical robotic system of claim 2, wherein, The device relationship comprises a third number of photographing units and display units, and the photographing dimension and the display dimension are both third dimensions; The video stream transmission path adaptation in the adaptation strategy comprises transmission of the video stream between the first terminal and the second terminal through a second path, the second path comprising a third number of transmission links; The adaptation unit is further configured to: collect a third number of third-dimensional surgery images through the third number of photographing units simultaneously, wherein each video frame in the first video stream comprises the third number of third-dimensional surgery images; transmit the third number of third-dimensional surgery images to the second terminal respectively by using the third number of transmission links in the second path; perform time synchronization processing on the third-dimensional surgery images of each video frame by using the second terminal, and display the time-synchronized third-dimensional surgery images respectively by using a third number of display units, wherein each video frame in the second video stream comprises the time-synchronized third number of third-dimensional surgery images.

6. The tele-surgical robotic system of claim 5, wherein, The image acquisition device is a two-path two-dimensional endoscope comprising two photographing units, and the image display device is a two-path two-dimensional display comprising two display units; The two-path two-dimensional endoscope is detachably connected to the patient surgery platform and is configured to acquire two-path two-dimensional surgery images, each video frame in the first video stream comprising two-path two-dimensional surgery images; The two-path two-dimensional display is arranged in the remote physician console and is configured to display time-synchronized two-path two-dimensional surgery images, each video frame in the second video stream comprising time-synchronized two-path two-dimensional surgery images; The first terminal is provided with two second remote ports, and the second path comprises two transmission links formed based on the two second remote ports and configured to transmit the two-path two-dimensional surgery images.

7. The tele-surgical robotic system of claim 2, wherein, The device relationship comprises a fourth number of photographing units and display units, and the photographing dimension and the display dimension are both fourth dimensions; The video stream transmission path adaptation in the adaptation strategy comprises transmission of the video stream between the first terminal and the second terminal through a third path, the third path comprising one transmission link, wherein the fourth number is greater than 1; The adaptation unit is further configured to: collect a fourth number of fourth-dimensional surgery images through the fourth number of photographing units simultaneously, wherein each video frame in the first video stream comprises the fourth number of fourth-dimensional surgery images; splice the fourth number of fourth-dimensional surgery images into one fourth-dimensional surgery image, and transmit the one fourth-dimensional surgery image to the second terminal by using one transmission link in the third path; The second terminal splits the one fourth-dimension surgery image into a fourth number of fourth-dimension surgery images, and displays the fourth number of fourth-dimension surgery images respectively by using a fourth number of display units, wherein each video frame in the second video stream comprises the fourth number of fourth-dimension surgery images after splitting; The fourth number of fourth-dimension surgery images are spliced into one fourth-dimension surgery image, so as to keep time synchronization when the fourth number of fourth-dimension surgery images are displayed respectively by using the fourth number of display units.

8. The tele-surgical robotic system of claim 7, wherein, The image acquisition device is a double two-dimensional endoscope comprising two shooting units, and the image display device is a double two-dimensional display comprising two display units; The double two-dimensional endoscope is detachably connected to the patient surgery platform, and is used to acquire double two-dimensional surgery images, wherein each video frame in the first video stream comprises the double two-dimensional surgery images; The double two-dimensional display is arranged in the remote doctor console, and is used to display the split double two-dimensional surgery images, wherein each video frame in the second video stream comprises the split double two-dimensional surgery images; The first terminal is provided with a third remote port, and the third path comprises a transmission link formed based on the third remote port, and is used to transmit the single two-dimensional surgery image.

9. The tele-surgical robotic system of claim 2, wherein, The device relationship comprises a first acquisition resolution and a first display resolution, wherein the first acquisition resolution is higher than the first display resolution; The video stream format adaptation of the adaptation strategy comprises a first interpolation strategy; The adaptation unit is further used to: By using the first interpolation strategy, the first video stream acquired by the shooting unit is reduced from the first acquisition resolution to the first display resolution to obtain a second video stream, and the second video stream is transmitted to the second terminal; By using the display unit, the second video stream of the first display resolution is displayed by the second terminal.

10. The tele-surgical robotic system of claim 9, wherein, The device relationship comprises a second acquisition resolution and a second display resolution, wherein the second acquisition resolution is lower than the second display resolution; The video stream format adaptation of the adaptation strategy comprises a second interpolation strategy; The adaptation unit is further used to: The first video stream acquired by the shooting unit is transmitted to the second terminal; By using the second interpolation strategy, the received first video stream is raised from the second acquisition resolution to the second display resolution by the second terminal, and the second video stream of the second display resolution is displayed by using the display unit.

Images