A rope-driven laparoscopic surgical instrument system and control method thereof

Abstract

The present application discloses a rope-driven laparoscopic surgical instrument system and a control method thereof. The perception module is used to obtain the force information of the hand and the end of the surgical instrument and the motion information of the hand; the data distribution module is used to send the force information and the motion information to the data preprocessing module; the image acquisition module is used to obtain the image taken by the laparoscope; the end positioning module is used to obtain the motion information of the end according to the image; the data preprocessing module is used to denoise and fit the force information and the motion information respectively to obtain the fitting information; the data analysis module is used to determine the correlation between the force information and the motion information according to the fitting information. The surgical instrument system of the present application combines the surgical instrument with the perception module, and can obtain the motion information and force information of the surgical instrument during the operation, providing a data basis for the intelligent control of the laparoscopic surgical instrument, facilitating the updating and iteration of the surgical instrument design, and can be widely used in the field of medical equipment technology.

Description

Technical Field

[0001] The present application relates to the technical field of medical equipment, and in particular to a rope-driven laparoscopic surgical instrument system and a control method thereof. Background Art

[0002] As human society enters the information age, the intelligentization of all walks of life has become a general trend, especially for laparoscopic surgical instruments that require precise operation. Intelligent control of laparoscopic surgery helps to update and iterate laparoscopic surgical instruments. However, it is difficult to collect operation data from current laparoscopic surgical instruments, resulting in a lack of data basis for the update and iteration of laparoscopic surgical instruments. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a rope-driven laparoscopic surgical instrument system and a control method thereof to obtain the operating data of the laparoscopic surgical instrument and provide a data basis for the update and iteration of the laparoscopic surgical instrument.

[0004] To achieve the above objectives, one aspect of an embodiment of the present application provides a rope-driven laparoscopic surgical instrument system, the system comprising:

[0005] Laparoscopes, surgical instruments, machine vision systems, and data analysis systems;

[0006] The surgical instrument is provided with a sensor system, which includes a perception module and a data distribution module; the machine vision system includes an image acquisition module and an end positioning module; the data analysis system includes a data preprocessing module and a data analysis module;

[0007] The sensing module is used to obtain first force information of the hand of the surgical instrument, second force information of the end of the surgical instrument, and first motion information of the hand of the surgical instrument;

[0008] The data distribution module is used to send the first force information, the second force information and the first motion information to the data preprocessing module;

[0009] The image acquisition module is used to acquire the image captured by the laparoscope; the image includes the end of the surgical instrument;

[0010] The end positioning module is used to obtain second motion information of the end of the surgical instrument according to the image;

[0011] The data preprocessing module is used to perform denoising and fitting on the first force information, the second force information, the first motion information and the second motion information respectively to obtain fitting information;

[0012] The data analysis module is used to determine the correlation between the first force information and the second force information according to the fitting information, and to determine the correlation between the first motion information and the second motion information according to the fitting information.

[0013] In some embodiments, the sensing module includes a force sensor and an encoder;

[0014] The force sensor is used to obtain first force information of the hand of the surgical instrument and second force information of the end of the surgical instrument;

[0015] The encoder is used to obtain the rotation angle of the hand transmission mechanism of the surgical instrument as the first motion information.

[0016] In some embodiments, the terminal positioning module includes an image preprocessing unit, a marker positioning unit, and a terminal turning angle calculation unit;

[0017] The image preprocessing unit is used to set a plurality of marking points on the image;

[0018] The marking point positioning unit is used to determine the coordinates of the center of each marking point in the coordinate system of the image;

[0019] The end rotation angle calculation unit is used to calculate the opening and closing angle and rotation angle of the end of the surgical instrument in the horizontal direction, as well as the rotation angle of the end of the surgical instrument in the vertical direction according to the central coordinates of each of the marking points.

[0020] In some embodiments, the marker point positioning unit is used for determining the coordinates of the center of each marker point in the coordinate system of the image by a marker point detection algorithm;

[0021] The calculation formula of the marker detection algorithm is:

[0022] (M1,M2,M3,M4,d v )=detection(I cell )

[0023] Wherein, detection() represents the marker detection algorithm, I cell represents the image, M1 and M2 are two marking points located at the end clamp of the surgical instrument, M3 is a marking point located at the end wrist joint of the surgical instrument, M4 is a marking point located at the end fixing rod of the surgical instrument, d v is the diameter length of the marking point M4 in the coordinate system of the image;

[0024] The end rotation angle calculation unit is used to calculate the rotation angle of the end of the surgical instrument in the vertical direction according to the first calculation formula;

[0025] The first calculation formula is:

[0026]

[0027] μ and α are:

[0028]

[0029]

[0030] Wherein, μ is the mapping ratio between the image coordinate system and the real coordinate system; d t is the diameter length of the marking point M4 in the real coordinate system, h is the vertical distance between the laparoscope and the horizontal plane, d3 is the distance between the marking point M3 and the vertical rotation axis in the real coordinate system, d4 is the horizontal distance between the marking point M4 and the vertical rotation axis in the real coordinate system, θ p is the rotation angle of the distal end of the surgical instrument in the vertical direction;

[0031] The end angle calculation unit is used to calculate the opening and closing angle of the end of the surgical instrument in the horizontal direction according to the second calculation formula;

[0032] The second calculation formula is:

[0033]

[0034] Among them, θ o is the horizontal rotation angle of the end of the surgical instrument, M yt is the coordinate of the horizontal axis in the real coordinate system, M 1t is the coordinate of the marker point M1 in the real coordinate system, M 2t is the coordinate of the marker point M2 in the real coordinate system;

[0035] The end rotation angle calculation unit is used to calculate the rotation angle of the end of the surgical instrument in the horizontal direction according to a third calculation formula;

[0036] The third calculation formula is:

[0037]

[0038] Among them, θ y is the horizontal rotation angle of the end of the surgical instrument, M mt is the coordinate of the midpoint of line segment M1M2 in the real coordinate system, is the normal vector of the vertical plane where the distal end of the surgical instrument is located.

[0039] In some embodiments, the data preprocessing module is used to use a high-order polynomial fitting algorithm or a least squares method to denoise the first force information, the second force information, the first motion information, and the second motion information, respectively, and fit them into smooth curves, and each of the smooth curves serves as the fitting information.

[0040] In some embodiments, the data analysis module includes an ideal data calculation unit, a data comparison unit, an expert system construction unit, an expert system evaluation unit, and a neural network learning unit;

[0041] The ideal data calculation unit is used to calculate the third motion information of the end of the surgical instrument under ideal conditions according to the first motion information of the hand of the surgical instrument; and calculate the third force information of the end of the surgical instrument under ideal conditions according to the first force information of the hand of the surgical instrument;

[0042] The data comparison unit is used to compare the second motion information and the third motion information of the end of the surgical instrument, compare the second force information and the third force information of the end of the surgical instrument, and evaluate the performance of the surgical instrument according to the comparison result;

[0043] The expert system building unit is used to obtain the force information and motion information corresponding to the hand of the surgical instrument, and build a laparoscopic surgery expert system based on an artificial neural network according to the force information and motion information;

[0044] The expert system evaluation unit is used to evaluate the effect of the operation according to the laparoscopic surgery expert system, the first force information and the first motion information;

[0045] The neural network learning unit is used to train a first fully connected neural network and a second fully connected neural network; use the first fully connected neural network to determine the association relationship between the first force information and the second force information according to the fitting information; use the second fully connected neural network to determine the association relationship between the first motion information and the second motion information according to the fitting information.

[0046] In some embodiments, it is used to calculate the third motion information according to the ideal motion calculation equation group;

[0047] The ideal motion calculation equations include:

[0048]

[0049]

[0050]

[0051] Among them, r1 is the radius of the two pulleys at the end of the hand of the surgical instrument, r2 is the radius of the pulley at the horizontal rotation axis of the hand of the surgical instrument, r3 is the radius of the two pulleys at the end clamp of the surgical instrument, r4 is the radius of the pulley at the vertical rotation axis of the hand of the surgical instrument, and r5 is the radius of the pulley at the end vertical rotation axis of the surgical instrument; when the hand of the surgical instrument is opened or closed by 2θ1 degrees, the two pulleys at the end of the hand of the surgical instrument rotate θ1 degrees in different directions respectively, pulling the pulley fixed on the pulley The rope on the surgical instrument stretches or contracts by a length d1, thereby pulling the end clamp of the surgical instrument to open or close by θ3 degrees; when the hand of the surgical instrument rotates along the horizontal axis by θ2 degrees, the rope moves as a whole by a length d2, thereby pulling the end clamp of the surgical instrument to rotate by θ4 degrees on the same side in the horizontal direction; when the hand of the surgical instrument rotates along the vertical axis by θ5 degrees, the rope on the pulley at the vertical axis of the hand of the surgical instrument stretches or contracts by a length d3, thereby pulling the end clamp of the surgical instrument to rotate by θ6 degrees on the same side in the vertical direction; 3i ,θ 4i ,θ 6i They are θ3, θ4, and θ6 under ideal conditions respectively;

[0052] The ideal data calculation unit is used to calculate the third force information according to the ideal hand force calculation formula;

[0053] The ideal hand force calculation formula is:

[0054]

[0055] Among them, F h is the unilateral force on the hand of the surgical instrument, F e is the unilateral force of the end clamp of the surgical instrument, F i is the friction force provided at the i-th pulley, and n is the total number of pulleys.

[0056] In some embodiments, the expert system building unit is used to build a laparoscopic surgery expert system based on an artificial neural network according to the opening or closing angle of the hand of the surgical instrument, the rotation angle of the hand of the surgical instrument along the horizontal axis, the rotation angle of the hand of the surgical instrument along the vertical axis, and the unilateral force of the hand of the surgical instrument.

[0057] In some embodiments, the data distribution module is further used to send the first force information, the second force information, the first motion information and the second motion information to a host computer.

[0058] To achieve the above object, another aspect of the embodiment of the present application provides a control method for a rope-driven laparoscopic surgical instrument system, which is applied to a rope-driven laparoscopic surgical instrument system as described above, and the method comprises:

[0059] Acquire first force information of a hand of a surgical instrument, second force information of a distal end of the surgical instrument, and first motion information of the hand of the surgical instrument;

[0060] Acquire an image captured by a laparoscope; the image includes the distal end of the surgical instrument;

[0061] Acquire second motion information of the distal end of the surgical instrument according to the image;

[0062] Denoising and fitting the first force information, the second force information, the first motion information, and the second motion information respectively to obtain fitting information;

[0063] The association relationship between the first force information and the second force information is determined according to the fitting information, and the association relationship between the first motion information and the second motion information is determined according to the fitting information.

[0064] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0065] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0066] The embodiments of the present application include at least the following beneficial effects:

[0067] The perception module of the present application is used to obtain the first force information of the hand of the surgical instrument, the second force information of the end of the surgical instrument, and the first motion information of the hand of the surgical instrument; the data distribution module is used to send the first force information, the second force information, and the first motion information to the data preprocessing module; the image acquisition module is used to obtain the image taken by the laparoscope; the image includes the end of the surgical instrument; the end positioning module is used to obtain the second motion information of the end of the surgical instrument according to the image; the data preprocessing module is used to denoise and fit the first force information, the second force information, the first motion information, and the second motion information respectively to obtain the fitting information; the data analysis module is used to determine the correlation between the first force information and the second force information according to the fitting information, and determine the correlation between the first motion information and the second motion information according to the fitting information. The rope-driven laparoscopic surgical instrument system of the present application combines the surgical instrument with the perception module, which can obtain the motion information and force information of the surgical instrument during the operation, provide a data basis for the intelligent control of the laparoscopic surgical instrument, and facilitate the update and iteration of the surgical instrument design. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 An exemplary structural diagram of a rope-driven laparoscopic surgical instrument system provided in an embodiment of the present application;

[0069] Figure 2 A module framework diagram of a rope-driven laparoscopic surgical instrument system provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of a flow chart of functions implemented by a terminal positioning module provided in an embodiment of the present application;

[0071] Figure 4 A flowchart of the functions implemented by a data analysis module provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of calculating terminal horizontal motion information of a data analysis module provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of calculating terminal vertical motion information of a data analysis module provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of calculating horizontal hand motion information of a data analysis module provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of hand vertical motion information calculation of a data analysis module provided in an embodiment of the present application;

[0076] Fig. 9A flow chart of an operation evaluation of a surgical instrument provided in an embodiment of the present application;

[0077] Fig.10 A schematic flow chart of a control method of a rope-driven laparoscopic surgical instrument system provided in an embodiment of the present application;

[0078] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0080] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0081] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0083] like Figure 1As shown, an embodiment of the present application provides a rope-driven laparoscopic surgical instrument system, which includes a rope-driven surgical instrument 1, which can be divided into an instrument hand 11 and an instrument end 12, a plurality of sensing modules 2 installed on the surgical instrument body 1, and a data distribution module 3 installed on the surgical instrument body 1. The surgical instrument system also includes a laparoscope 4.

[0084] A machine vision system and a data analysis system based on a laparoscope 4, wherein the machine vision system includes a module for acquiring laparoscopic surgery images (i.e., an image acquisition module) and an end positioning module, and the data analysis system includes a data preprocessing module and a data analysis module.

[0085] like Figure 2 As shown, the input end of the perception module is the force magnitude at the surgical instrument hand and the end clamp and the rotation angle of the hand transmission mechanism in three degrees of freedom, the output end of the perception module is connected to the input end of the data distribution module; the output end of the data distribution module is connected to the input end of the data preprocessing module; the output end of the data preprocessing module is connected to the input end of the data analysis module; the input end of the laparoscopic surgery image acquisition module is the real three-dimensional space scene within the laparoscope field of view, the output end of the laparoscopic surgery image acquisition module is connected to the input end of the end positioning module; the output end of the end positioning module is connected to the input end of the data preprocessing module.

[0086] Next, each functional module in the embodiment of the present application is explained in detail:

[0087] Acquisition module for laparoscopic surgery images (i.e., image acquisition module):

[0088] The input of the module for acquiring laparoscopic surgical images is the real three-dimensional space scene within the laparoscope field of view, and the output end is connected to the input end of the terminal positioning module.

[0089] Used to capture images of the abdominal cavity during surgery through a laparoscope, including the tips of surgical instruments.

[0090] End positioning module:

[0091] The input end of the terminal positioning module is connected to the laparoscopic surgery image acquisition module, and the output end is connected to the data preprocessing module.

[0092] It is used to obtain images acquired by the laparoscopic surgery image module and obtain the motion information of the end of the surgical instrument through the machine vision algorithm.

[0093] like Figure 3 As shown, the terminal positioning module includes an image preprocessing unit, a marker point positioning unit and a terminal turning angle calculation unit.

[0094] The image preprocessing unit is used to preprocess the original laparoscopic surgery image, remove noise and other useless information in the image, and facilitate the subsequent determination of the center coordinates of each marking point by the marking point positioning unit.

[0095] The marker point positioning unit is used to determine the coordinates of the center coordinates of each marker point in the image coordinate system using a marker point detection algorithm.

[0096] Assume that a laparoscopic surgery image unit is I cell , the coordinates of the centers of the four identical circular marking points at the end of the surgical instrument in the image coordinate system M i =(x i ,y i )(i∈{1,2,3,4}), then:

[0097] (M1, M2, M3, M4, d v )=detection(I cell )

[0098] Where detection() represents the marker point detection algorithm used. In the specific implementation process, traditional feature extraction algorithms such as Hough transform or feature extraction algorithms based on deep learning can be used according to the actual situation; M1 and M2 are the marker points located at the two end grippers, M3 is the marker point located at the end wrist joint, M4 is the marker point located at the end fixed rod, and d v is the diameter length of the M4 marker point in the image coordinate system.

[0099] The end angle calculation unit is used to calculate the opening and closing angle, rotation angle and vertical rotation angle of the end clamp of the surgical instrument using the center coordinates of each marking point obtained by the marking point positioning unit.

[0100] For ease of calculation, assume that the laparoscope is perpendicular to the horizontal plane where the end of the instrument is located, and the M4 marker is located in the center of the laparoscope. Assuming that the mapping ratio between the image coordinate system and the real coordinate system is μ, then:

[0101]

[0102]

[0103]

[0104] Where, d t is the diameter length of the M4 mark point in the real coordinate system, h is the vertical distance between the laparoscope and the horizontal plane, d3 is the distance between the M3 mark point and the vertical rotation axis in the real coordinate system, d4 is the horizontal distance between the M4 mark point and the vertical rotation axis in the real coordinate system, θ pIt is the rotation angle of the end of the instrument in the vertical direction.

[0105] Take the coordinates of the M4 marker point as the zero point, and assume that the coordinates of the M1 and M2 marker points in the real coordinate system are it =(x it ,y it ,z it )(i∈{1,2}), since the M1, M2 and M3 marking points are always coplanar, we have:

[0106]

[0107]

[0108] M yt =(x3,d4+cosθ p *d y ,sinθ p *d y )

[0109] Where M yt is the coordinate of the horizontal axis in the real coordinate system, d y It is the distance between the horizontal axis and the vertical axis in the real coordinate system.

[0110] Due to M it With M yt The following relationship is satisfied between two points:

[0111] |M yt M it |=d i (i∈{1,2})

[0112] Where, d i M i The distance between the marked point and the horizontal axis in the real coordinate system can be calculated according to the above formula. it The coordinates of the M1 and M2 markers in the real coordinate system are obtained. it After that, the opening and closing angle and horizontal rotation angle of the end of the instrument can be calculated:

[0113]

[0114]

[0115] In the formula, θ o is the opening and closing angle of the instrument end, θ y M is the horizontal rotation angle of the end of the instrument. mt is the coordinate of the midpoint of line segment M1M2 in the real coordinate system, is the normal vector of the vertical plane where the end of the instrument is located.

[0116] In the specific implementation process, in order to simplify the calculation and improve the detection accuracy, the laparoscope can also be placed in a vertical or horizontal position relative to the horizontal plane where the end of the instrument is located. p ,θ o and θ y Calculate separately.

[0117] Perception module:

[0118] The input of the perception module is the force magnitude at the surgical instrument hand and the end clamp and the rotation angle of the hand transmission mechanism in three degrees of freedom, and the output end is connected to the data distribution module.

[0119] Used to obtain the force and motion information of the hand and end of the surgical instrument, and convert it into analog or digital signals that can be processed by a computer.

[0120] Data distribution module:

[0121] The input end of the data distribution module is connected to the perception module, and the output end is connected to the data preprocessing module.

[0122] It is used to receive the force and motion information obtained by the perception module and transmit it to the host computer wirelessly.

[0123] Data preprocessing module:

[0124] The input end of the data preprocessing module is connected to the data distribution module and the terminal positioning module, and the output end is connected to the data analysis module.

[0125] It is used to pre-process the received force and motion information, remove noise from the data and fit it, and use multiple smooth curves to represent the motion data of the instrument hand, the force data of the instrument hand, the motion data of the instrument end and the force data of the instrument end, so as to facilitate the analysis of the subsequent data analysis module.

[0126] Data Analysis Module:

[0127] The input end of the data analysis module is connected to the data preprocessing module, and the output end is the data analysis result, including the learned neural network, expert system, surgical implementation effect evaluation result and instrument performance evaluation result.

[0128] It is used to analyze the data processed by the data preprocessing module, including using neural networks to learn the relationship between the movement and force data of the instrument hand and the instrument end, using data to build an expert system and evaluate the effectiveness of the surgery, and comparing theoretical calculation data with actual data to evaluate the performance of the instrument.

[0129] like Figure 4As shown, the data analysis module includes an ideal data calculation unit, a data comparison unit, an expert system building unit, an expert system evaluation unit and a neural network learning unit.

[0130] The ideal data calculation unit is used to obtain the motion information of the end of the instrument under ideal conditions through theoretical calculation based on the motion information of the instrument hand.

[0131] like Figures 5 to 8 As shown, in the specific calculation process, it is assumed that the radius of the two pulleys at the end of the handle is r1, the radius of the pulley at the horizontal axis of the hand is r2, the radius of the two pulleys at the clamping claw at the end of the instrument is r3, the radius of the pulley at the vertical axis of the hand is r4, and the radius of the pulley at the vertical axis of the instrument is r5. Then, when the doctor operates the handle to open or close 2θ1 degrees, the two pulleys at the end of the handle rotate θ1 degrees in different directions, pulling the rope fixed on the pulley to extend or contract the length d1, thereby pulling the end of the instrument The jaws open or close by θ3 degrees; when the doctor's operating handle rotates by θ2 degrees along the horizontal axis of the hand, the lower tangent points of the rope and the pulley at the horizontal axis rotate by θ2 degrees in different directions, so that the rope moves by a length d2 as a whole, thereby pulling the jaws at the end of the instrument to rotate by θ4 degrees on the same side in the horizontal direction; when the doctor's operating handle rotates by θ5 degrees along the vertical axis of the hand, the rope fixed on the pulley at the vertical axis of the hand is stretched or contracted by a length d3, thereby pulling the jaws at the end of the instrument to rotate by θ6 degrees on the same side in the vertical direction, and:

[0132] d1=θ1r1=θ3r3

[0133] d2=θ2r2=θ4r3

[0134] d3=θ5r4=θ6r5

[0135] Therefore, after obtaining the hand motion information θ1, θ2 and θ5, the motion information θ of the end of the instrument under ideal conditions can be obtained by the following formula: 3i ,θ 4i and θ 6i :

[0136]

[0137]

[0138]

[0139] The data comparison unit is used to compare the θ obtained by the ideal data calculation unit 3i ,θ 4i and θ 6i Compared with the actual end motion information θ3, θ4 and θ6 provided by the data preprocessing module, the force information Fh With F e By comparison, an objective and quantitative evaluation of the performance of the surgical instrument is carried out.

[0140] Since the drive of the surgical instrument from the hand to the end is completely realized by the rope, the force information of the hand and the end is as follows:

[0141]

[0142] In the formula, F h F is the force on the handle on one side. e F is the force on one side of the clamping jaw. i is the friction force provided at the i-th pulley, and n is the total number of pulleys. In actual use, in order to make the doctor's operation more accurate, F is required h ≈F e ,Right now As close to 0 as possible.

[0143] Expert system building unit for using hand motion and force information θ1, θ2, θ5, F h , built an artificial neural network-based laparoscopic surgery expert system, and studied the optimal hand movement and force range in laparoscopic surgery.

[0144] The expert system evaluation unit is used to evaluate the hand movement and force information θ1, θ2, θ5, F after the expert system is built. h , to evaluate the effect of the operation, and the operation evaluation flow chart of surgical instruments can be referred to Fig. 9 .

[0145] The neural network learning unit is used to train two neural networks so that they can respectively learn the relationship between the motion and force information of the hand and the end.

[0146] The expression of the trained neural network is as follows:

[0147] (θ3, θ4, θ6) = network1 (θ1, θ2, θ5)

[0148] (F e )=network2(F h )

[0149] Since the relationship between the two is relatively simple, the simplest fully connected neural network can be used for learning, and the specific parameters of the neural network can be adjusted appropriately according to the actual results. Using these two neural networks, the motion and force information of the instrument end can be directly obtained through the motion and force information of the instrument hand. In the actual operation, there is no need to install a force sensor at the end and visually perceive and locate the end, making the surgical instrument system more streamlined.

[0150] The embodiment of the present application also provides a control method for a rope-driven laparoscopic surgical instrument system, which relates to the field of medical equipment technology. The control method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the control method, etc., but is not limited to the above forms.

[0151] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0152] Reference Fig.10 The present application embodiment provides a control method for a rope-driven laparoscopic surgical instrument system, which is applied to a rope-driven laparoscopic surgical instrument system of the present application embodiment. The method may include but is not limited to S100 to S140, as follows:

[0153] S100: Acquire first force information of a hand of a surgical instrument, second force information of a distal end of the surgical instrument, and first motion information of the hand of the surgical instrument;

[0154] S110: Acquire an image captured by a laparoscope; the image includes the distal end of the surgical instrument;

[0155] S120: Acquire second motion information of the distal end of the surgical instrument according to the image;

[0156] S130: De-noising and fitting the first force information, the second force information, the first motion information, and the second motion information respectively to obtain fitting information;

[0157] S140: Determine an association relationship between the first force information and the second force information according to the fitting information, and determine an association relationship between the first motion information and the second motion information according to the fitting information.

[0158] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above control method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0159] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0160] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0161] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0162] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the control method of the embodiment of the present application;

[0163] Input / output interface 1103, used to implement information input and output;

[0164] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0165] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0166] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0167] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned control method when executed by a processor.

[0168] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0171] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0172] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0174] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0175] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0180] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A rope-driven laparoscopic surgical instrument system, characterized in that: The system includes: a laparoscope, surgical instruments, a machine vision system and a data analysis system; Wherein, the surgical instrument is provided with a sensor system, which includes a perception module and a data distribution module; the machine vision system includes an image acquisition module and an end positioning module; the data analysis system includes a data preprocessing module and a data analysis module; The sensing module is used to obtain first force information of the hand of the surgical instrument, second force information of the end of the surgical instrument, and first motion information of the hand of the surgical instrument; The data distribution module is used to send the first force information, the second force information and the first motion information to the data preprocessing module; The image acquisition module is used to acquire the image captured by the laparoscope; the image includes the end of the surgical instrument; The end positioning module is used to obtain second motion information of the end of the surgical instrument according to the image; The data preprocessing module is used to perform denoising and fitting on the first force information, the second force information, the first motion information and the second motion information respectively to obtain fitting information; The data analysis module is used to determine the correlation between the first force information and the second force information according to the fitting information, and to determine the correlation between the first motion information and the second motion information according to the fitting information.

2. A rope-driven laparoscopic surgical instrument system according to claim 1, characterized in that: The sensing module includes a force sensor and an encoder; The force sensor is used to obtain first force information of the hand of the surgical instrument and second force information of the end of the surgical instrument; The encoder is used to obtain the rotation angle of the hand transmission mechanism of the surgical instrument as the first motion information.

3. The rope-driven laparoscopic surgical instrument system according to claim 1, characterized in that: The terminal positioning module includes an image preprocessing unit, a marking point positioning unit and a terminal turning angle calculation unit; The image preprocessing unit is used to set a plurality of marking points on the image; The marking point positioning unit is used to determine the coordinates of the center of each marking point in the coordinate system of the image; The end rotation angle calculation unit is used to calculate the opening and closing angle and rotation angle of the end of the surgical instrument in the horizontal direction, as well as the rotation angle of the end of the surgical instrument in the vertical direction according to the central coordinates of each of the marking points.

4. A rope-driven laparoscopic surgical instrument system according to claim 3, characterized in that: The marker point positioning unit is used for determining the coordinates of the center of each marker point in the coordinate system of the image by a marker point detection algorithm; The calculation formula of the marker detection algorithm is: (M1,M2,M3,M4,d v )=detection(I cell ) Wherein, detection() represents the marker detection algorithm, I cell represents the image, M1 and M2 are two marking points located at the end clamp of the surgical instrument, M3 is a marking point located at the end wrist joint of the surgical instrument, M4 is a marking point located at the end fixing rod of the surgical instrument, d v is the diameter length of the marking point M4 in the coordinate system of the image; The end rotation angle calculation unit is used to calculate the rotation angle of the end of the surgical instrument in the vertical direction according to the first calculation formula; The first calculation formula is: μ and α are: Wherein, μ is the mapping ratio between the image coordinate system and the real coordinate system; d t is the diameter length of the marking point M4 in the real coordinate system, h is the vertical distance between the laparoscope and the horizontal plane, d3 is the distance between the marking point M3 and the vertical rotation axis in the real coordinate system, d4 is the horizontal distance between the marking point M4 and the vertical rotation axis in the real coordinate system, θ p is the rotation angle of the distal end of the surgical instrument in the vertical direction; The end angle calculation unit is used to calculate the opening and closing angle of the end of the surgical instrument in the horizontal direction according to the second calculation formula; The second calculation formula is: Among them, θ o is the horizontal rotation angle of the end of the surgical instrument, M yt is the coordinate of the horizontal axis in the real coordinate system, M 1t is the coordinate of the marker point M1 in the real coordinate system, M 2t is the coordinate of the marker point M2 in the real coordinate system; The end rotation angle calculation unit is used to calculate the rotation angle of the end of the surgical instrument in the horizontal direction according to a third calculation formula; The third calculation formula is: Among them, θ y is the horizontal rotation angle of the end of the surgical instrument, M mt is the coordinate of the midpoint of line segment M1M2 in the real coordinate system, is the normal vector of the vertical plane where the distal end of the surgical instrument is located.

5. The rope-driven laparoscopic surgical instrument system according to claim 1, characterized in that: The data preprocessing module is used to use a high-order polynomial fitting algorithm or a least squares method to denoise the first force information, the second force information, the first motion information and the second motion information respectively, and fit them into smooth curves respectively, and each of the smooth curves serves as the fitting information.

6. The rope-driven laparoscopic surgical instrument system according to claim 1, characterized in that: The data analysis module includes an ideal data calculation unit, a data comparison unit, an expert system construction unit, an expert system evaluation unit and a neural network learning unit; The ideal data calculation unit is used to calculate the third motion information of the end of the surgical instrument under ideal conditions according to the first motion information of the hand of the surgical instrument; and calculate the third force information of the end of the surgical instrument under ideal conditions according to the first force information of the hand of the surgical instrument; The data comparison unit is used to compare the second motion information and the third motion information of the end of the surgical instrument, compare the second force information and the third force information of the end of the surgical instrument, and evaluate the performance of the surgical instrument according to the comparison result; The expert system building unit is used to obtain the force information and motion information corresponding to the hand of the surgical instrument, and build a laparoscopic surgery expert system based on an artificial neural network according to the force information and motion information; The expert system evaluation unit is used to evaluate the effect of the operation according to the laparoscopic surgery expert system, the first force information and the first motion information; The neural network learning unit is used to train a first fully connected neural network and a second fully connected neural network; use the first fully connected neural network to determine the association relationship between the first force information and the second force information according to the fitting information; use the second fully connected neural network to determine the association relationship between the first motion information and the second motion information according to the fitting information.

7. The rope-driven laparoscopic surgical instrument system according to claim 6, characterized in that: The ideal data calculation unit is used to calculate the third motion information according to the ideal motion calculation equation group; The ideal motion calculation equations include: Among them, r1 is the radius of the two pulleys at the end of the hand of the surgical instrument, r2 is the radius of the pulley at the horizontal rotation axis of the hand of the surgical instrument, r3 is the radius of the two pulleys at the end clamp of the surgical instrument, r4 is the radius of the pulley at the vertical rotation axis of the hand of the surgical instrument, and r5 is the radius of the pulley at the end vertical rotation axis of the surgical instrument; when the hand of the surgical instrument is opened or closed by 2θ1 degrees, the two pulleys at the end of the hand of the surgical instrument rotate θ1 degrees in different directions respectively, pulling the pulley fixed on the pulley The rope on the surgical instrument stretches or contracts by a length d1, thereby pulling the end clamp of the surgical instrument to open or close by θ3 degrees; when the hand of the surgical instrument rotates along the horizontal axis by θ2 degrees, the rope moves as a whole by a length d2, thereby pulling the end clamp of the surgical instrument to rotate by θ4 degrees on the same side in the horizontal direction; when the hand of the surgical instrument rotates along the vertical axis by θ5 degrees, the rope on the pulley at the vertical axis of the hand of the surgical instrument stretches or contracts by a length d3, thereby pulling the end clamp of the surgical instrument to rotate by θ6 degrees on the same side in the vertical direction; 3i ,θ 4i ,θ 6i They are θ3, θ4, and θ6 under ideal conditions respectively; The ideal data calculation unit is used to calculate the third force information according to the ideal hand force calculation formula; The ideal hand force calculation formula is: Among them, F h is the unilateral force on the hand of the surgical instrument, F e is the unilateral force of the end clamp of the surgical instrument, F i is the friction force provided at the i-th pulley, and n is the total number of pulleys.

8. The rope-driven laparoscopic surgical instrument system according to claim 7, characterized in that: The expert system building unit is used to build a laparoscopic surgery expert system based on an artificial neural network according to the opening or closing angle of the hand of the surgical instrument, the rotation angle of the hand of the surgical instrument along the horizontal axis, the rotation angle of the hand of the surgical instrument along the vertical axis, and the unilateral force of the hand of the surgical instrument.

9. A rope-driven laparoscopic surgical instrument system according to any one of claims 1 to 8, characterized in that: The data distribution module is further used to send the first force information, the second force information, the first motion information and the second motion information to a host computer.

10. A control method for a rope-driven laparoscopic surgical instrument system, characterized in that: Applied to a rope-driven laparoscopic surgical instrument system as claimed in claim 1, the method comprising: Acquire first force information of a hand of a surgical instrument, second force information of a distal end of the surgical instrument, and first motion information of the hand of the surgical instrument; Acquire an image captured by a laparoscope; the image includes the distal end of the surgical instrument; Acquire second motion information of the distal end of the surgical instrument according to the image; Denoising and fitting the first force information, the second force information, the first motion information, and the second motion information respectively to obtain fitting information; The association relationship between the first force information and the second force information is determined according to the fitting information, and the association relationship between the first motion information and the second motion information is determined according to the fitting information.

Images