Method for evaluating surgery based on artificial intelligence and surgical robot system

Abstract

The invention relates to the field of medical instruments, and discloses a method for evaluating an operation based on artificial intelligence and an operation robot system. The operation comprises a plurality of actual operation stages, and the method comprises the following steps: obtaining an operation information set of the operation; and evaluating the operation based on the operation information set and a pre-trained artificial intelligence evaluation model.

Description

Technical Field

[0001] The present disclosure relates to the field of medical devices, and in particular to a method for evaluating surgery based on artificial intelligence and a surgical robot system. Background Art

[0002] Minimally invasive surgery causes less trauma to patients and has higher postoperative output, and has occupied an important position in surgical operations. Objective, fair, comprehensive, accurate, professional and effective surgical evaluation is an important part of surgical monitoring and management.

[0003] At present, objective surgical evaluation technology is still in its infancy. Due to the lack of unified standards for surgical evaluation, the development of surgical quality assessment lags behind. Summary of the invention

[0004] In some embodiments, the present disclosure provides a method for evaluating surgery based on artificial intelligence, wherein the surgery includes multiple actual surgical stages, and the method includes: obtaining an operation information set of the surgery; and evaluating the surgery based on the operation information set and a pre-trained artificial intelligence evaluation model.

[0005] In some embodiments, the present disclosure provides a surgical robot system, comprising: at least one master trolley; at least one slave trolley, the slave trolley comprising at least one robotic arm, the robotic arm being used to carry surgical instruments, the master trolley being communicatively connected with the slave trolley; and a control device, configured to execute a method as in any embodiment of the present disclosure.

[0006] In some embodiments, the present disclosure provides a computer-readable storage medium, the storage medium is used to store at least one instruction, and the at least one instruction is executed by a computer to configure the computer to perform the method in any embodiment of the present disclosure.

[0007] In some embodiments, the present disclosure provides a computer system, comprising: a memory for storing at least one instruction; and a processor, coupled to the memory and configured to execute the at least one instruction to perform the method in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. The drawings described below only illustrate some embodiments of the present disclosure. For ordinary technicians in this field, other embodiments can be obtained based on the contents of the embodiments of the present disclosure and these drawings without creative work.

[0009] Figure 1 A structural block diagram of a surgical robot system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic diagram showing the structure of a driven trolley of a surgical robot system according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing the structure of a main control vehicle according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram showing the structure of a main operator according to some embodiments of the present disclosure is shown;

[0013] Figure 5A A flowchart showing a method for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure;

[0014] Figure 5B A block diagram showing an artificial intelligence evaluation model according to some embodiments of the present disclosure;

[0015] Fig. 6A A flowchart showing a method for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure;

[0016] Figure 6B A structural block diagram of an artificial intelligence evaluation model according to some embodiments of the present disclosure is shown;

[0017] Figure 6C A structural block diagram of a first feature extractor according to some embodiments of the present disclosure is shown;

[0018] Fig.6D A structural block diagram of a second feature extractor according to some embodiments of the present disclosure is shown;

[0019] Fig. 6E A structural block diagram of a third feature extractor according to some embodiments of the present disclosure is shown;

[0020] Fig. 7A A flowchart showing a method for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure;

[0021] Figure 7B A structural block diagram of an artificial intelligence evaluation model according to some embodiments of the present disclosure is shown;

[0022] Figure 7C A structural block diagram of a first feature extractor according to some embodiments of the present disclosure is shown;

[0023] Fig.7D A structural block diagram of a second feature extractor according to some embodiments of the present disclosure is shown;

[0024] Fig. 7E A structural block diagram of a third feature extractor according to some embodiments of the present disclosure is shown;

[0025] Fig. 8A A flowchart showing a method for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure;

[0026] Figure 8B A structural block diagram of an artificial intelligence evaluation model according to some embodiments of the present disclosure is shown;

[0027] Figure 8C A structural block diagram of a first feature extractor according to some embodiments of the present disclosure is shown;

[0028] Fig.8D A structural block diagram of a second feature extractor according to some embodiments of the present disclosure is shown;

[0029] Fig. 8E A structural block diagram of a third feature extractor according to some embodiments of the present disclosure is shown;

[0030] Fig. 9 A structural block diagram of a computer system according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the technical problems solved by the present disclosure, the technical solutions adopted and the technical effects achieved more clearly, the technical solutions of the embodiments of the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only exemplary embodiments of the present disclosure, rather than all embodiments.

[0032] In the description of the present disclosure, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "coupled" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances. In the present disclosure, the end close to the operator (e.g., doctor) is defined as the proximal end, near part, or rear end, and the end close to the surgical patient is defined as the distal end, far end, or front end, or front part. Those skilled in the art will appreciate that the embodiments of the present disclosure may be used for medical instruments or surgical robots, and may also be used for other non-medical devices.

[0033] In the present disclosure, the term "position" refers to the positioning of an object or a part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using changes in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X axis, Y axis, and Z axis, respectively). In the present disclosure, the term "attitude" refers to the rotational setting of an object or a part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In the present disclosure, the term "pose" refers to the combination of the position and attitude of an object or a part of an object, such as can be described using six parameters of the six degrees of freedom mentioned above.

[0034] Some embodiments of the present disclosure provide a surgical robot system. Figure 1 FIG. 1 is a block diagram of a surgical robot system 100 according to some embodiments of the present disclosure. Figure 1 As shown, the robot system 100 may include at least one slave trolley 1, at least one master trolley 2 and a control device 3. The control device 3 may be connected to the master trolley 2 and the slave trolley 1 in communication, for example, by cable connection or wireless connection, to achieve communication between the master trolley 2 and the slave trolley 1. The master trolley 2 includes a master operator 20 for remote operation by an operator, and a display for displaying an image of an operation area. The slave trolley 1 includes at least one robotic arm 10 and at least one surgical instrument 30 disposed at the distal end of at least one robotic arm 10. The master-slave mapping between the master operator 20 in the master trolley 2 and the surgical instrument 30 in the slave trolley 1 is achieved by the control device 3, so as to achieve motion control of the surgical instrument 30 by the master operator 20. It should be understood that the control device 3 may be configured on a computer device and disposed inside the master trolley 2. Those skilled in the art may understand that the master trolley 2 and the slave trolley 1 may adopt other structures or forms, such as a base, a bracket or a building.

[0035] Figure 2 FIG. 1 is a schematic diagram showing the structure of the driven trolley 1 of the surgical robot system 100 according to some embodiments of the present disclosure. Figure 2 As shown, the driven trolley 1 of the surgical robot system 100 may include at least one robotic arm 10 and at least one surgical instrument 30 disposed at the distal end of the at least one robotic arm 10. For example, Figure 2 The driven carriage 1 of the robot system 100 shown includes a plurality of robot arms 10 ( Figure 24 robotic arms are shown) and surgical instruments 30 disposed at the distal end of each robotic arm 10. Those skilled in the art will appreciate that the robotic system 100 may include multiple slave trolleys 1, other numbers of robotic arms 10, and surgical instruments 30. It should be understood that the multiple robotic arms may be disposed on the same slave trolley or on different slave trolleys.

[0036] In some embodiments, in some embodiments, the surgical instrument 30 may include a surgical tool or an endoscope. The distal end of the surgical instrument 30 forms a surgical instrument end, and the surgical tool end may be provided with a surgical actuator, such as a separation forceps, a grasping forceps, a scissors, a bipolar grasping forceps, a single-stage curved scissors, etc., to perform a surgical operation. It should be understood that the end of the endoscope may include an image acquisition device or a lighting device, etc.

[0037] Figure 3 FIG. 2 shows a schematic diagram of the structure of the main control vehicle 2 according to some embodiments of the present disclosure. Figure 3 As shown, the master control vehicle 2 may include a main operator 20 and a display (eg, Figure 3 The stereo display 21, the main control external display 22, and the main control touch display 23 are shown in the figure. The image acquisition device can be used to acquire images of the operation area, and the acquired images are processed by the video processing module and displayed on the display of the main control trolley 2. The operator obtains the posture of the end of the surgical instrument relative to the reference coordinate system (such as the main manipulator base coordinate system, the display screen coordinate system, etc.) in real time through the image in the display. The posture of the main manipulator relative to the reference coordinate system is the posture truly perceived by the operator. The operator controls the posture of the end of the surgical instrument by remotely operating the main manipulator. In this way, when the operator holds the handle of the main manipulator and moves it to operate the surgical instrument, based on the principle of intuitive operation, the posture change of the end of the surgical instrument felt by the operator is consistent with the posture change of the main manipulator felt by the operator, which helps to improve the operator's remote operation experience and remote operation accuracy.

[0038] In some embodiments, Figure 3 As shown, the display of the main control trolley 2 may include a stereo display 21, a main control external display 22, and a main control touch display 23. The stereo display 21 displays the image of the operation part and the system status prompt, the main control external display 22 displays the image of the operation part and the system status prompt, and the touch display 23 displays the software user interface of the main control trolley 2. In some embodiments, the image displayed by the stereo display 21 or the main control external display 22 can be determined based on the image acquired by the image acquisition device. In some embodiments, the main control trolley 2 may also include a foot pedal (e.g. Figure 4The control device 3 is connected to the main operator 20, the main control trolley display and the foot pedal respectively for signal exchange with the main operator 20, the main control trolley display and the foot pedal, and generates corresponding control instructions based on the collected control information.

[0039] Figure 4 FIG. 2 shows a schematic diagram of the structure of the main operator 20 according to some embodiments of the present disclosure. Figure 4 As shown, the main manipulator 20 includes a multi-degree-of-freedom motion arm 210 and a handle 220 arranged at the far end of the motion arm 210. The multi-degree-of-freedom motion arm 210 includes multiple joints and multiple connecting rods. The multiple joints of the motion arm 210 may include position joints and posture joints. The posture joint can be used to adjust the posture of the main manipulator, and the position joint adjusts the position of the main manipulator. The main manipulator sensor can be arranged at the posture joint and the position joint of the motion arm, and is used to obtain the joint information (position or angle) corresponding to the posture joint and the position joint. According to the acquired joint information, the position and posture of the handle of the main manipulator relative to the main manipulator base coordinate system can be determined. Those skilled in the art will understand that the same joint can be both a position joint and a posture joint.

[0040] like Figure 4 As shown, the main manipulator 20 may include 8 joints 2111-2118, wherein the first joint 2111, the second joint 2112 and the third joint 2113 are position joints, the first joint 2111, the second joint 2112, the fifth joint 2115, the sixth joint 2116 and the seventh joint 2117 are posture joints, and the eighth joint 2118 may be a clamping joint for controlling the opening and closing of the end effector of the corresponding surgical tool. The first joint 2111 and the second joint 2112 can adjust both the position and the posture of the main manipulator handle 220, but in surgical operations, they are generally used to adjust the position of the main manipulator 220. For simplicity, they can only be used as position joints in the following description. The fifth joint 2115, the sixth joint 2116 and the seventh joint 2117 can only adjust the posture of the main manipulator handle 220. The third joint 2113 can only adjust the position of the main manipulator handle 220. Based on the joint information (such as angle) obtained by the master manipulator sensor of the posture joint and the forward kinematics algorithm, the current posture of the master manipulator handle 220 is calculated. Based on the joint information (such as position) obtained by the master manipulator sensor of the position joint and the forward kinematics algorithm, the current position of the master manipulator handle 220 is calculated.

[0041] Wherein, the main manipulator 20 may include multiple connecting rods, such as multiple connecting rods 211-214 connecting position joints, and multiple connecting rods 215-216 connecting posture joints. The first joint 2111 and the second joint 2112 are connected by the first connecting rod 211, the second joint 2112 and the third joint 2113 are connected by the second connecting rod 212, and the third joint 2113 and the fourth joint 2114 are connected by the third connecting rod 213. For example, the third connecting rod 213 can be a parallelogram structure, and the third joint 2113 and the fourth joint 2114 are divided into two vertices located at the parallelogram, such as two vertices with the same side length. The third joint 2113 can be an active joint, the fourth joint 2114 is a passive joint, which can move with the third joint 2113, and the fourth connecting rod 214 can be a follower connecting rod, which moves with the third connecting rod 213. The fourth joint 2114 and the fifth joint 2115 are connected by the fourth connecting rod 214. The fifth joint 2115, the sixth joint 2116 and the seventh joint 2117 are connected by the fifth connecting rod 215 and the sixth connecting rod 216, respectively. The parallelogram structure provides 3-degree-of-freedom translational motion for the three joints at the distal end, such as the fifth joint 2115, the sixth joint 2116 and the seventh joint 2117. In addition, the design of the parallelogram structure can meet the safety, shielding and beautification requirements of the main operator. It should be understood that, for example, the first joint 2111 to the third joint 2113 can be respectively configured with a high-resolution encoder (e.g., 16 bits). In some embodiments, the first joint 2111 to the third joint 2113 may include a gravity compensation mechanism, such as a cable wheel spring mechanism, a compression spring mechanism, etc.

[0042] The above is only an example, and it should be understood that the motion arm 210 may also include fewer or more joints (for example, more redundant joints) to achieve more flexible posture adjustment through more redundant joints.

[0043] Some embodiments of the present disclosure provide a method for evaluating surgery based on artificial intelligence (e.g., Figure 5A Method 5000, Fig. 6A Method 6000, Fig. 7A Method 7000 or Fig. 8A The surgery may be performed by a surgical robot (e.g. Figure 2The surgical robot system 100 shown in the figure is performed. In the present disclosure, the surgical robot performing the surgery is described as an example, but it does not constitute a limitation on the embodiments of the present disclosure. Usually, a surgery includes multiple actual surgical stages. For example, the standard surgical process of a surgery includes a preliminary preparation stage and a surgical implementation stage. During the surgical operation performed by the surgical robot, the operator can remotely control the posture of the end of the surgical instrument 30 by remotely operating the main manipulator 20 to complete the surgical operations corresponding to multiple actual surgical stages. In some embodiments, the multiple actual surgical stages may include the stage of establishing pneumoperitoneum, the stage of surgical instrument placement, the stage of surface fat cleaning, the stage of tissue freeing, the stage of blood vessel separation, the stage of lesion removal, the stage of tissue suturing, and the like. It should be understood that the stage of establishing pneumoperitoneum and the stage of surgical instrument placement are the preliminary preparation stages of the surgery. In the stage of surgical instrument placement, the posture of the surgical instrument is adjusted so that the surgical instrument reaches the optimal operating posture and position for the surgery. The stage of surface fat cleaning, the stage of tissue freeing, the stage of blood vessel separation, the stage of lesion removal, and the stage of tissue suturing are the stages of surgical implementation.

[0044] In some embodiments, an operation may also include an additional actual operation stage that is lacking in the standard operation process. Multiple actual operation stages may also include an additional actual operation stage. In some embodiments, the additional actual operation stage may include an emergency operation stage and a remedial operation stage. The emergency operation stage may include a legitimate emergency treatment operation for unexpected situations that occur under the normal operation process, or an additional necessary operation outside the normal operation process, the purpose of which is to deal with various actual problems to ensure the quality of the operation or the safety of the operation. The emergency operation stage can reflect the difficulty of the operation and has a positive correlation with the difficulty of the operation. The remedial operation stage may include a remedial treatment operation for the consequences of the wrong operation. The remedial operation stage is an additional operation caused by the operator's error under the normal operation process or outside the normal operation process, for example, an additional suturing operation performed due to the bleeding of tissue in a normal part caused by the misoperation of the surgical instrument. The remedial operation stage can reflect the completion quality of the operation and has a negative correlation with the completion quality of the operation. For example, for conventional urological surgery, the standard operation process may include, in sequence, a pneumoperitoneum establishment stage, a surgical instrument placement stage, a surface fat cleaning stage, a tissue free stage, a vascular separation stage, a lesion resection stage, a tissue suturing stage, and the like. During the operation, due to the special location of the actual lesion or unclear boundaries with normal tissue, the lesion needs to be freed or the blood vessels separated again during the lesion removal stage. In this way, compared with the standard surgical procedure, the actual operation stage also includes an additional tissue freeing stage or an additional blood vessel separation stage between the lesion removal stage and the tissue suturing stage.

[0045] It should be understood that the above are only examples, and multiple actual surgical stages may include some of the stages or other stages, and these exemplary stages may be repeated, and the order may also be changed according to different surgeries.

[0046] Figure 5A 1 is a flowchart of a method 5000 (also referred to as "method 5000") for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure. Method 5000 may be implemented or executed by hardware, software, or firmware. In some embodiments, method 5000 may be performed by a surgical robot system (e.g., Figure 1 In some embodiments, the method 5000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 1 For example, the control device of the surgical robot system (e.g., Figure 1 The control device 3) shown may include a processor configured to execute the method 5000. In some embodiments, these instructions may be stored on a computer-readable medium.

[0047] See also Figure 5A , in step 5001, obtain a set of operation information of the surgery.

[0048] In the present disclosure, an operation information set refers to a set of operation information at multiple sampling moments during a surgical procedure. In some embodiments, the operation information at each sampling moment may include one or more of the motion trajectory data of the main manipulator handle, the motion trajectory data of the end of the surgical instrument, the type of operation action, the type of surgical instrument used, and the number of surgical instruments used, and then the operation information set may include one or more of the motion trajectory data of the main manipulator handle corresponding to multiple sampling moments, the motion trajectory data of the end of the surgical instrument, the type of operation action, the type of surgical instrument used, and the number of surgical instruments used. In some embodiments, the motion trajectory of the main manipulator handle includes a set of multiple motion trajectory data of the main manipulator handle that changes over time, and the motion trajectory data of the main manipulator handle at each sampling moment may include the current posture of the main manipulator handle at the current sampling moment, and the current posture of the main manipulator handle at multiple sampling moments during the surgical procedure can describe the motion trajectory of the main manipulator handle. In some embodiments, the motion trajectory of the end of the surgical instrument includes a set of multiple motion trajectory data of the end of the surgical instrument that changes over time, and the motion trajectory data of the end of the surgical instrument at each sampling moment may include the current posture of the end of the surgical instrument at the current sampling moment, and the current posture of the end of the surgical instrument at multiple sampling moments during the surgical procedure can describe the motion trajectory of the end of the surgical instrument. Those skilled in the art will appreciate that the above are merely examples, and the operation information set may also include other information related to the surgery.

[0049] Those skilled in the art will appreciate that the operation performed by the operator during the operation may include a single action of a single instrument, a combination of actions of a single instrument, and a combination of actions of multiple instruments. Each operation action has a corresponding action type. For example, the operation action type may include at least one of translation, rotation, clamping, and a combination of at least two actions. Among them, the combination of at least two actions may be a combination of at least two actions of a surgical instrument, such as a combination of translation and clamping actions, or may be a combination of at least two actions of multiple surgical instruments, such as clamping, translation of grasping forceps, and translation of monopolar curved scissors. In each actual surgical stage, the operator may use different types or numbers of surgical instruments to perform different surgical operation actions according to actual surgical needs and surgical progress. For example, in the tissue freeing stage, the operator may need to use three types of surgical instruments, grasping forceps, separating forceps, and / or single-stage scissors, etc., to perform surgical operations such as clamping, separation, and cutting. In the lesion resection stage, surgical instruments such as bipolar grasping forceps and / or single-stage curved scissors may be used to perform surgical operations such as electrocoagulation and electrocuting. In the tissue suturing stage, surgical instruments such as needle holders and / or grasping forceps may be used to perform surgical operations such as clamping and suturing.

[0050] In some embodiments, each operation information in the operation information set may be based on the surgical robot system (e.g., Figure 1The data displayed or generated by the surgical robot system 100 shown in the figure is obtained. For example, after the surgical instrument is successfully installed, the surgical robot system can identify the type of surgical instrument installed, and obtain the relevant surgical instrument type and number of surgical instruments. For another example, during the surgical operation, the type of surgical instrument currently used, the number of surgical instruments used, etc. can be obtained based on the control instructions such as the selection and switching of the surgical instrument. For another example, during the surgical operation, multiple joint information of multiple joints of the main manipulator of the surgical robot system can be obtained, and based on the multiple joint information, the motion trajectory data of the main manipulator handle (such as the current position of the main manipulator handle) and / or the type of operation action can be determined. For another example, during a surgical operation, the driving information of at least one surgical instrument can be obtained, and based on the driving information, the motion trajectory data of the surgical instrument end of at least one surgical instrument (for example, the current position of the surgical instrument end) can be determined, or the posture sensing data of at least one surgical instrument can be obtained through a sensor, and based on the posture sensing data of at least one surgical instrument, the motion trajectory data of the surgical instrument end of at least one surgical instrument (for example, the current position of the surgical instrument end) can be determined, or surgical video data can be obtained, and based on the surgical video data, the motion trajectory data of at least one surgical instrument (for example, the position of the surgical instrument end in different video frames) can be determined. The motion trajectory of the main manipulator handle over time can be obtained by the position of the main manipulator handle at multiple different sampling times. The motion trajectory of the surgical instrument end over time can be obtained by the position of the surgical instrument end at multiple different sampling times or in different video frames. Detailed description is given in the following embodiments.

[0051] In some embodiments, method 5000 may include obtaining multiple joint information of multiple joints of the main manipulator; and, based on the multiple joint information, determining the motion trajectory data and / or the operation action type of the main manipulator handle. In the present disclosure, the joint information (or joint value information) may include the angle of rotation of the corresponding joint relative to the corresponding joint axis or the distance moved relative to the initial position.

[0052] In some embodiments, joint information of multiple joints can be obtained based on the main manipulator sensor. The main manipulator sensor is set at multiple joint positions, and the joint information of the corresponding joints is obtained based on the main manipulator sensor to calculate the current position and posture of the main manipulator handle. For example, the current position and current posture of the main manipulator handle are calculated based on the forward kinematics algorithm. In some embodiments, the current position and posture of the main manipulator handle can be used as the motion trajectory data of the main manipulator handle at the current moment. By calculating the position and posture of the main manipulator handle at multiple different sampling moments, the motion trajectory of the main manipulator handle over time can be obtained.

[0053] In some embodiments, the change of the position and / or posture of the main manipulator handle can be calculated based on multiple joint information, and the type of operation action can be determined based on the change of position and / or posture. Alternatively, the type of operation action can be determined based on the type of joint. For example, the joints 2111-2113 of the main manipulator 20 are position joints. When the joint information of the joints 2111-2113 changes, and the joint information of the joints 2114-2117 does not change, the type of operation action can be determined to be translation. For example, joints 2115, 2116 and 2117 are posture joints. When the joint information of the joints 2115-2117 changes, and the joint information of the joints 2111-2114 does not change, the type of operation action can be determined to be rotation. For example, when the joint information of the clamping joint 2118 changes, it can be determined that the operation action is clamping. When the joint information of the joints 2115-2117 changes, the joint information of the clamping joint 2118 changes, it can be determined that the operation action is a combination of rotation and clamping. The above is only an example, and the operation action type may also be determined based on changes in joint information of joints at other different positions and their combined changes.

[0054] In some embodiments, based on multiple joint information, it can be determined which joints among multiple joints have movements exceeding a threshold value, and for joints whose movements exceed the threshold value, the action type of the operation action can be determined based on the joint type. It should be understood that the threshold value can be the range of motion amplitude that must be achieved for normal operation. When the movement of the joint is less than the threshold value, it may be considered as an error or jitter, etc., and when the movement of the joint exceeds the threshold value, it can be identified as normal operation. It should be understood that the joint type may include position joints, posture joints, and clamping joints. For example, joints 2111-2113 can be position joints, joints 2115, 2116, and 2117 can be posture joints, and joint 2118 can be a clamping joint for controlling the opening and closing of the end effector to achieve functions such as clamping and shearing. By setting a threshold value, abnormal interference factors can be eliminated, the division and identification of the operation action type can be more accurate, and the accuracy of surgical evaluation can be improved.

[0055] In some embodiments, method 5000 may include obtaining drive information of at least one surgical instrument, and determining motion trajectory data of the surgical instrument end corresponding to at least one surgical instrument based on the drive information. In some embodiments, the drive information of at least one surgical instrument (for example, the drive information of the drive motor of the flexible arm of at least one surgical instrument) can be obtained based on a drive device sensor. The drive device sensor can be set on a drive device, and the drive device is used to drive at least one surgical instrument. The current posture of at least one surgical instrument end can be determined based on the drive information, for example, the current position and current posture of the surgical instrument end are calculated based on a forward kinematics algorithm. In some embodiments, the current posture of the surgical instrument end can be used as the motion trajectory data of the surgical instrument end at the current moment. By calculating the posture of the surgical instrument end at multiple different sampling moments, the motion trajectory of the surgical instrument end over time can be obtained.

[0056] In some embodiments, method 5000 may include obtaining posture sensing data of at least one surgical instrument, and determining the motion trajectory data of the surgical instrument end corresponding to at least one surgical instrument based on the posture sensing data. In some embodiments, a posture sensor may be provided on at least one surgical instrument, and the current posture of the surgical instrument end of at least one surgical instrument may be obtained through the posture sensor data of the posture sensor. For example, the posture sensor may be an optical fiber sensor, which is arranged on the arm body (e.g., a flexible arm body) of at least one surgical instrument, and is used to sense the configuration of at least one surgical instrument, so that the posture of the surgical instrument end at the distal end of the arm body of at least one surgical instrument may be obtained. For another example, the posture sensor may be an electromagnetic sensor, and the electromagnetic sensor may include an electromagnetic induction device and an electromagnetic reflector fixed to the surgical instrument end of at least one surgical instrument, and the electromagnetic induction device relies on the mutual induction principle of the electromagnetic field to determine the posture information of the electromagnetic reflector working in its excitation magnetic field space to obtain the posture of the surgical instrument end. In some embodiments, the current posture of the surgical instrument end may be used as the motion trajectory data of the surgical instrument end at the current moment. By calculating the posture of the surgical instrument end at multiple different sampling moments, the motion trajectory of the surgical instrument end over time may be obtained.

[0057] In some embodiments, the method 5000 may further include obtaining surgical video data, and determining motion trajectory data of a surgical instrument end of at least one surgical instrument based on the surgical video data. In some embodiments, the surgical robot system (e.g., Figure 1The surgical robot system 100 shown in the figure can acquire surgical video data. For example, the surgical video data can be acquired through the endoscope in the surgical robot system. In some embodiments, the position and posture of the surgical instrument end of at least one surgical instrument in different video frames can be obtained, and the position and posture of the surgical instrument end in a video frame is used as the motion trajectory data of the corresponding surgical instrument end. In some embodiments, a visual detection algorithm can be used to obtain the position and posture of the surgical instrument end in different video frames. For example, at least one surgical instrument is provided with a positioning tag for posture detection, and the positioning tag includes multiple identifiers (for example, multiple posture identifiers and angle identifiers), which can identify multiple identifiers in different video frames, and determine the position and posture of the surgical instrument end based on the multiple identifiers. The motion trajectory of the surgical instrument end over time can be obtained through the position and posture of the surgical instrument end in different video frames.

[0058] In some embodiments, the operation information corresponding to each sampling moment can be obtained based on the sampling time interval. For example, a specified sampling time interval can be set, such as 50 milliseconds, and the corresponding operation information can be recorded every 50 milliseconds. The specific sampling time interval can be adjusted according to different surgical procedures, and can also be adjusted according to the requirements for the fineness of the division of the operation actions. In some embodiments, the operation information corresponding to each sampling moment during the operation is obtained, and the operation information corresponding to multiple sampling moments during the operation is combined into an operation information set according to preset rules. For example, the operation information corresponding to multiple sampling moments is arranged into an operation information set according to the sampling moment time series arrangement method.

[0059] Continue to view Figure 5A , in step 5003, the surgery is evaluated based on the operation information set and the pre-trained artificial intelligence evaluation model.

[0060] In some embodiments, evaluating surgery based on an operation information set and a pre-trained artificial intelligence evaluation model includes: dividing the operation information set into multiple operation information subsets corresponding to multiple actual surgical stages; and evaluating multiple actual surgical stages based on the multiple operation information subsets corresponding to the multiple actual surgical stages.

[0061] In some embodiments, the operation information set may include the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end, and the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end may be separately subjected to spatial analysis to obtain the time domain characteristics of the motion trajectory, such as trajectory shape, velocity, acceleration, jerk (third-order derivative of position), etc. In some embodiments, by analyzing the time domain characteristics of the motion trajectory (e.g., the motion trajectory of the main manipulator handle and / or the motion trajectory of the surgical instrument end), the smoothness of the motion trajectory during the operation, the accuracy of the position, the consistency of the posture changes of the main manipulator and the surgical instrument end, and the smoothness of the operation, etc., may be evaluated, and quality or difficulty evaluation information may be provided, based on which different actual surgical stages and / or the entire surgical process may be evaluated. In some embodiments, based on the time domain characteristics of the motion trajectory, a fluency time domain index for measuring the smoothness of the operation execution may be obtained, such as a velocity measurement index, an acceleration measurement index, a jitter measurement index, etc. Among them, the speed measurement index may include, for example, the time ratio MAPR (Movement Arrest Period Ratio) that the movement speed exceeds the peak speed by a given percentage, the velocity arc length VAL (Velocity Arc Length), etc. The acceleration measurement index may include, for example, the number of acceleration peaks and valleys Peaks (Peaks Metric), the inverse number of acceleration peaks and valleys IPV (Inverse Number of Peaks and Valleys), etc. Among them, the time ratio MAPR that the movement speed exceeds the peak speed by a given percentage, the inverse number of acceleration peaks and valleys IPV are positively correlated with the smoothness of the execution of the operation, and the velocity arc length VAL and the number of acceleration peaks and valleys Peaks are negatively correlated with the smoothness of the execution of the operation. The jitter measurement index is an index associated with the acceleration of the motion trajectory, and may include, for example, the dimensionless jitter index DLJ (Dimensionless Jerk), LDLJ (Log Dimensionless Jerk), etc., for example, it can be expressed by the following formula: In some embodiments, the dimensionless jitter indicators DLJ and LDLJ can be expressed by, for example, the following formula:

[0062]

[0063]

[0064] Among them, v peak is the peak speed, t1 and t2 are the start and end time of the movement, and v(t) is the speed. In some embodiments, based on the execution fluency, the surgery can be evaluated, for example, the quality of different actual surgical stages and / or the entire surgical process, surgical improvement suggestions, etc. can be evaluated.

[0065] In some embodiments, the operation information set may include the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end. The motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end may be subjected to frequency domain analysis to obtain the motion trajectory data of the surgical robot (e.g. Figure 2 The frequency domain characteristics of the motion trajectory of the surgical robot system 100 (for example, the motion trajectory of the main manipulator handle and / or the motion trajectory of the end of the surgical instrument), such as frequency spectrum characteristics and energy spectrum characteristics. In some embodiments, by analyzing the frequency domain characteristics of the motion trajectory, the execution fluency during the operation can be evaluated, and quality assessment information can be provided. Based on the execution fluency, different actual surgical stages and / or the entire surgical process can be evaluated. For example, in some embodiments, based on the frequency domain characteristics of the motion trajectory, a frequency domain index for measuring the smoothness of the execution of the surgery can be obtained, such as spectral arc length SAL (SpectralArc Length), spectral arc length SPARC (Spectral ARC Length) improved based on spectral arc length SAL, and energy spectrum index SPMR (Spectral Metric), etc., wherein spectral arc length SAL is used to calculate the arc length of the Fourier amplitude spectrum of the velocity curve corresponding to the motion trajectory within a given frequency range, spectral arc length SPARC is used to adaptively determine the frequency range and calculate the arc length of the Fourier amplitude spectrum of the velocity curve corresponding to the motion trajectory within a determined frequency range, and energy spectrum index SPMR uses the energy in a set frequency range around the main frequency of the acceleration Fourier transform to represent smoothness, and spectral arc length SAL, spectral arc length SPARC, and energy spectrum index SPMR are positively correlated with the execution smoothness of the surgery. In some embodiments, based on the execution smoothness, the surgery can be evaluated, such as evaluating the quality of different actual surgical stages and / or the entire surgical process, surgical improvement suggestions, etc.

[0066] In some embodiments, the operation information set may include the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end, and the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end may be subjected to frequency domain analysis and spatial motion analysis respectively to obtain the time domain characteristics and frequency domain characteristics of the motion trajectory (e.g., the motion trajectory of the main manipulator handle and / or the motion trajectory of the surgical instrument end). In some embodiments, the time domain characteristics and frequency domain characteristics of the motion trajectory (e.g., the motion trajectory of the main manipulator handle and / or the motion trajectory of the surgical instrument end) may be combined to identify multiple actual surgical stages, and to evaluate different actual surgical stages and / or the entire surgical process. The specific details are similar to those in some of the above embodiments and will not be expanded here.

[0067] In some embodiments, the operation information set may include motion trajectory data of the main manipulator handle and / or motion trajectory data of the surgical instrument end, and may also include other operation data related to the surgery, for example, it may include one or more of the operation action type, the type of surgical instrument used, and the number of surgical instruments used. It can combine the time domain characteristics, frequency domain characteristics of the motion trajectory (for example, the motion trajectory of the main manipulator handle and / or the motion trajectory of the surgical instrument end) and other operation information features of other operation data (for example, the timing characteristics of one or more of the operation action type, the type of surgical instrument used, and the number of surgical instruments used, etc.) to identify multiple actual surgical stages and evaluate different actual surgical stages and / or the entire surgical process.

[0068] In some embodiments, the multiple actual surgical stages include multiple actual surgical stages corresponding to multiple standard surgical stages of the standard surgical process. For example, the multiple actual surgical stages may include one or more of the pneumoperitoneum establishment stage, surgical instrument positioning stage, surface fat clearing stage, tissue freedom stage, blood vessel separation stage, lesion resection stage, tissue suturing stage, etc. in some of the above-mentioned embodiments.

[0069] In some embodiments, the plurality of actual surgical stages also include additional actual surgical stages that are lacking in the standard surgical procedure, for example, the additional actual surgical stages may include the emergency operation stage and / or the remedial operation stage in some of the above embodiments. In step 5003, the operation information set may be divided to obtain an operation information subset corresponding to the emergency operation stage and / or the remedial operation stage, and the corresponding emergency operation stage and / or the remedial operation stage may be evaluated based on the operation information subset of the emergency operation stage and / or the remedial operation stage. It will be appreciated by those skilled in the art that the emergency operation stage may reflect the overall difficulty of the operation, and has a positive correlation with the overall difficulty of the operation, and the remedial operation stage may reflect the overall quality of the operation, and has a negative correlation with the overall quality of the operation. Therefore, the overall difficulty and / or quality of the operation may be evaluated based on the evaluation of the emergency operation stage and / or the remedial operation stage.

[0070] In some embodiments, during the operation, the operation information corresponding to each sampling moment is obtained based on the sampling time interval, for example, the corresponding operation information is recorded every 50 milliseconds, and the operation information at each sampling moment may include information of m dimensions, for example, including data of 5 dimensions, namely: motion trajectory data of the main manipulator handle, motion trajectory data of the end of the surgical instrument, operation action type, type of surgical instrument used, and number of surgical instruments used. The operation information at a single sampling moment constitutes an operation information vector. The operation information vectors of multiple sampling moments during the operation are integrated into an operation information set, for example, the operation information set can be integrated into an m×n dimensional matrix, where n is the number of sampling moments.

[0071] In some embodiments, the artificial intelligence assessment model may include a division part, and the operation information set may be divided into multiple operation information subsets corresponding to multiple actual surgical stages through the division part of the artificial intelligence assessment model.

[0072] In some embodiments, the division part includes a first classification network for classifying actual surgical stages. The first classification network includes a feature extractor, a classifier, etc. The feature extractor performs feature extraction on the operation information set to obtain operation information features, such as time domain features, frequency domain features of the motion trajectory (for example, the motion trajectory of the main manipulator handle and / or the motion trajectory of the end of the surgical instrument), and other operation information features of other operation data (for example, the type of operation action, the type of surgical instrument used, the timing features of one or more of the number of surgical instruments used, etc.), etc. Based on the operation information features, a classifier is used to classify and identify the surgical stages, thereby dividing the operation information set into multiple operation information subsets corresponding to multiple actual surgical stages. For example, multiple actual surgical stages can be divided into C1 categories. For example, multiple actual surgical stages can include multiple actual surgical stages corresponding to multiple standard surgical stages of the standard surgical process and additional actual surgical stages that are lacking in the standard surgical process. For example, C1=9, which are respectively the stage of establishing pneumoperitoneum, the stage of placing surgical instruments in place, the stage of surface fat cleaning, the stage of tissue freeing, the stage of blood vessel separation, the stage of lesion removal, the stage of tissue suturing, the stage of emergency operation, and the stage of remedial operation. The operation information vectors at each sampling moment in the operation information set can be classified one by one through the first classification network, and the category of the operation information vectors at each sampling moment can be determined, so as to divide the operation into multiple surgical stages. After classification, the operation information set is divided into operation information subsets corresponding to different surgical stages. Detailed description is given in the following embodiments.

[0073] In some embodiments, the artificial intelligence evaluation model may further include an evaluation part, and the evaluation part of the artificial intelligence evaluation model may be used to evaluate a subset of operation information to obtain a surgical evaluation result.

[0074] In some embodiments, evaluating the surgery includes determining at least one surgery evaluation indicator for each of a plurality of actual surgery stages, wherein the at least one surgery evaluation indicator includes difficulty and / or quality.

[0075] In some embodiments, the operation includes multiple actual operation stages, and the difficulty assessment of each actual operation stage can be divided into multiple levels, for example, including four difficulty levels of simple, conventional, generally complex, and particularly complex, or divided into four difficulty levels of 1, 2, 3, and 4 from low to high difficulty. Different operation stages usually correspond to different difficulty levels. For example, the stage of establishing pneumoperitoneum and the stage of placing surgical instruments are the pre-operative preparation stages, and the difficulty level belongs to the simple level or the level 1 difficulty. The surface fat cleaning stage and the tissue suturing stage belong to the conventional level or the level 2 difficulty. The tissue free stage belongs to the generally complex level or the level 3 difficulty. The blood vessel separation stage and the lesion removal stage belong to the particularly complex level or the level 4 difficulty. It should be understood that the above is only an example, and the number of levels of the surgical difficulty level can be determined according to the degree of sophistication of the surgical assessment. The surgical difficulty of each actual operation stage also needs to be determined according to the specific circumstances of the operation. For example, each actual operation stage is associated with factors such as the patient's own condition, the lesion condition, and the unexpected events during the operation. The above factors need to be combined to determine the surgical difficulty of each actual operation stage. In some embodiments, the operation information of each actual surgical stage may reflect the surgical difficulty of the corresponding stage. For example, the surgical difficulty of each actual surgical stage is positively correlated with the type of operation actions involved in the actual surgical stage, the number of operation action types, the type of surgical instruments used, the number of surgical instruments used, the duration of the actual surgical stage, etc. Therefore, multiple actual surgical stages may be evaluated through multiple subsets of operation information of multiple actual surgical stages.

[0076] In one embodiment, similar to the evaluation of the difficulty of each actual surgical stage, the evaluation of the quality of each actual surgical stage can be divided into multiple levels, for example, including four quality levels of poor, general, good, and excellent, or divided into four quality levels of 1, 2, 3, and 4 from low to high quality. Based on the surgeon's personal ability, experience, surgical site status, etc., different surgical stages usually present different quality levels. In some embodiments, the operation information of each actual surgical stage can reflect the surgical quality of the corresponding stage, for example, the surgical quality of each actual surgical stage is positively correlated with the stage length of the actual surgical stage, the smoothness of the operation action, and the connection with other adjacent surgical stages. Therefore, multiple actual surgical stages can be evaluated by multiple subsets of operation information of multiple actual surgical stages.

[0077] In some embodiments, the evaluation portion of the artificial intelligence evaluation model may include a first subnetwork, the first subnetwork being used to determine at least one surgical evaluation indicator for each of a plurality of actual surgical stages, wherein the at least one surgical evaluation indicator includes difficulty and / or quality.

[0078] In some embodiments, the first subnetwork includes a second classification network for predicting at least one surgical evaluation index, and the second classification network predicts the difficulty or quality of each actual surgical stage, or the second classification network predicts the difficulty and quality of each actual surgical stage at the same time, wherein the difficulty can be divided into C2 levels, and the quality can be divided into C3 levels. For example, in some embodiments, C2=4, C3=4, the difficulty is divided into four difficulty levels of simple, conventional, generally complex, and particularly complex, and the quality is divided into four quality levels of poor, general, good, and excellent. In some embodiments, the second classification network includes a feature extractor, a classifier, etc., and the feature extractor extracts features from the operation information set to obtain operation information features. Based on the operation information features, the classifier is used to perform difficulty and / or classification identification of multiple actual surgical stages to complete the prediction of at least one surgical evaluation index of multiple actual surgical stages. Detailed descriptions are given in the following embodiments.

[0079] In some embodiments, evaluating the surgery further comprises determining a surgery improvement suggestion for at least one of the plurality of actual surgery stages.

[0080] In some embodiments, evaluating the surgery includes determining a surgical improvement suggestion for at least one actual surgical stage. For example, if the operation in the vascular separation stage is not ideal, improvement suggestions can be given for the surgical operation movements, the use of surgical instruments, etc. in the vascular separation stage to improve the quality of the surgery.

[0081] In some embodiments, the evaluation part of the artificial intelligence evaluation model may include a second sub-network, and the second sub-network is used to determine surgical improvement suggestions for at least one actual surgical stage in the multiple actual surgical stages. Detailed description is given in some of the following embodiments.

[0082] Figure 5B FIG. 5 is a schematic diagram showing the structure of an artificial intelligence evaluation model 500 according to some embodiments of the present disclosure. Figure 5B As shown, the artificial intelligence evaluation model 500 includes a division part 510 and an evaluation part 520. The division part 510 is used to divide the operation information set into multiple operation information subsets, and the evaluation part 520 is used to evaluate multiple actual surgical stages based on the multiple operation information subsets. The evaluation part 520 includes a first sub-network 521 and a second sub-network 522. The first sub-network 521 is used to determine at least one surgical evaluation index for each of the multiple actual surgical stages, and the second sub-network 522 is used to determine surgical improvement suggestions for at least one actual surgical stage in the multiple actual surgical stages.

[0083] like Figure 5BAs shown, in some embodiments, the division part 510 may include a first input layer 5101, a first feature extractor 5103, a first classifier 5105, and a first output layer 5107, which are cascaded in sequence. Among them, the first input layer 5101 is used to receive input data, for example, the input data includes an operation information set. The first feature extractor 5103 is used to extract features, for example, extract the first-level operation information features of the operation information set. The first-level operation information features can be coarse features for distinguishing surgical stages. For example, the first-level operation information features can include features associated with surgical instruments, and the surgical stages are distinguished by the types of surgical instruments; or the first-level operation information features can include features associated with surgical actions, so as to distinguish the surgical stages by features associated with surgical actions; or the first-level operation information features can also include multiple features associated with surgical instruments and surgical actions, so as to distinguish the surgical stages. The first classifier 5105 is used to classify the surgical stages based on the first-level operation information features, for example, the surgical stages can be divided into C1 categories. The first output layer 5107 is used to process and output the classification results to output multiple operation information subsets of multiple actual operation stages of the surgical process. For example, multiple operation information subsets of multiple actual operation stages can be output in sequence according to the operation time sequence.

[0084] like Figure 5BAs shown, in some embodiments, the first subnetwork 521 may include a second input layer 5211, a second feature extractor 5213, a second classifier 5215, and a second output layer 5217, which are cascaded in sequence. Among them, the second input layer 5211 is connected to the output of the first output layer 5107 to receive information output by the first output layer 5107, such as multiple operation information subsets of multiple actual surgical stages of the surgical process output by the first output layer 5107. The second feature extractor 5213 is connected to the second input layer 5211 and the output end of the first feature extractor 5103. The second feature extractor 5213 is used to obtain part or all of the first-level operation information features output by the first feature extractor 5103 as the first shared feature, and at the same time, more detailed operation information features can be extracted from the operation information subsets of multiple actual surgical stages, such as extracting second-level operation information features. For example, the second-level operation information features may include detailed features related to the flexibility and smoothness of the operation movements in each stage of the operation, such as time domain features and / or frequency domain features extracted from the motion trajectory (for example, the motion trajectory of the main manipulator handle and / or the motion trajectory of the end of the surgical instrument) that can reflect at least a part of the execution smoothness of the operation. The second classifier 5215 classifies and evaluates the surgical evaluation indicators of each actual surgical stage based on the first shared features and the second-level operation information features, wherein the surgical evaluation indicators include difficulty and / or quality, for example, difficulty can be classified into C2 category, and quality can be classified into C3 category. The second output layer 5217 is used to output the classification results of the second classifier 5215 to output the evaluation results of the surgical evaluation indicators of multiple actual surgical stages of the surgical process.

[0085] like Figure 5BAs shown, in some embodiments, the second subnetwork 522 may include a third input layer 5221, a third feature extractor 5223, a third classifier 5225, and a third output layer 5227, which are cascaded in sequence. The third input layer 5221 is connected to the output of the first output layer 5107 to receive the information output by the first output layer 5107, such as a subset of operation information of multiple actual surgical stages of the surgical process output by the first output layer 5107. The third feature extractor 5223 is connected to the third input layer 5221, and the third feature extractor 5223 can be connected to the output end of the first feature extractor 5103 and / or the second feature extractor 5213. The third feature extractor 5223 can be used to obtain all or part of the first-level operation information features output by the first feature extractor 5103 as the second shared feature. The third feature extractor 5223 can also be used to obtain all or part of the second-level operation features output by the second feature extractor 5213 as the third shared feature. At the same time, the third feature extractor 5223 can also extract other detailed operation information features from the operation information subsets of multiple actual surgical stages, such as extracting third-level operation information features. For example, the third-level operation information features can include features associated with the selection of operating instruments and operation time in each stage of the operation to focus on the detailed content of each stage of the operation. The third classifier 5225 evaluates the surgical suggestions of each actual surgical stage based on the second shared features, the third shared features, and the third-level operation information features to determine an improvement suggestion vector associated with the surgical improvement suggestion, for example, each dimension of the vector represents an improvement measure or feature, for example, it can be a surgical improvement suggestion on the selection of surgical instruments, or a surgical improvement suggestion on the control of surgical duration, or a surgical improvement suggestion on surgical operation actions, etc. The third output layer 5227 processes the output vector of the surgical suggestion to output the surgical suggestion, for example, the third output layer 5227 can process the output vector using a natural language processing network (NLP) to output text information of the surgical improvement suggestion.

[0086] In some embodiments, the partitioning part and the evaluation part (including the first sub-network and the second sub-network) may adopt a deep neural network model, such as a recurrent neural network (RNN), a convolutional neural network (CNN), etc. Those skilled in the art will appreciate that the above is only an example, and the partitioning part, the first sub-network, and the second sub-network may also adopt other network models.

[0087] Some embodiments of the present disclosure provide a method for training an artificial intelligence evaluation model, for example, Figure 5B Artificial intelligence assessment model 500 is shown.

[0088] In some embodiments, the training method of the artificial intelligence evaluation model includes the following steps:

[0089] Step 1: Obtain a large number of operation information sets. One operation information set corresponds to one operation information set, and one operation information set includes operation information vectors at multiple sampling moments during the operation.

[0090] Step 2: Annotate each operation information set, including the actual operation stage annotation corresponding to the operation information vector at each sampling moment in the operation information set, the difficulty level annotation of each actual operation stage, the quality level annotation of each actual operation stage, and the surgical improvement suggestion annotation of at least one actual operation stage. The actual operation stage can be divided into C1 category, for example, C1=9, which are respectively the stage of establishing pneumoperitoneum, the stage of surgical instrument placement, the stage of surface fat cleaning, the stage of tissue freeing, the stage of vascular separation, the stage of lesion resection, the stage of tissue suturing, the stage of emergency operation, and the stage of remedial operation. The difficulty level of each actual operation stage can be divided into C2 category, for example, C2=4, which divides the difficulty into four difficulty levels: simple, conventional, generally complex, and particularly complex. The quality level of each actual operation stage can be divided into C3 category, for example, C3=4, which divides the quality into four quality levels: poor, general, good, and excellent. The surgical improvement suggestions of at least one actual operation stage can be divided into C4 category, for example, C4=3, which includes three types of surgical improvement suggestions: surgical improvement suggestions related to surgical instrument selection, surgical improvement suggestions related to surgical time control, and surgical improvement suggestions related to surgical operation actions. In some embodiments, the operation information set can be annotated in combination with the surgical video data, preoperative medical data, and postoperative recovery data of the operation corresponding to each operation information set to obtain more objective and accurate annotations. The surgical video data can be obtained by a surgical robot system (e.g., Figure 1 The surgical information set may be annotated by an expert. In some embodiments, the annotation results of multiple experts may be obtained for mutual verification and consistency evaluation to improve the accuracy of annotation of the operation information set.

[0091] In some embodiments, the operation information set to be labeled may include motion trajectory data of the main manipulator handle and / or motion trajectory data of the surgical instrument end. The smoothness of the operation can be evaluated based on the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end. The quality level of the actual surgical stage and the surgical improvement suggestion labeling can be performed based on the smoothness of the operation, so as to train the artificial intelligence evaluation model more objectively.

[0092] In some embodiments, the execution fluency of multiple actual surgical stages can be evaluated based on the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end. The quality level labeling of each actual surgical stage and the surgical improvement suggestion labeling of at least one actual surgical stage can be determined based on the execution fluency of multiple actual surgical stages, so as to train the artificial intelligence evaluation model more objectively and meticulously.

[0093] In some embodiments, the time domain features of the motion trajectories (e.g., the motion trajectories of the main manipulator handle and / or the motion trajectories of the end of the surgical instrument) of multiple actual surgical stages can be obtained, such as trajectory shape, speed, acceleration, etc. Based on the time domain features of the motion trajectories of each actual surgical stage, at least one fluency time domain index corresponding to each actual surgical stage for measuring the execution fluency of the surgery can be calculated respectively, for example, including at least one of the speed measurement index, the acceleration measurement index, and the jitter measurement index, wherein the speed measurement index can include, for example, the time proportion MAPR, the speed arc length VAL, etc., of the moving speed exceeding a given percentage of the peak speed in some of the above embodiments. The acceleration measurement index can include, for example, the acceleration peak-to-valley number Peaks, the acceleration peak-to-valley number inverse IPV, etc. in some of the above embodiments. The jitter measurement index can include, for example, the dimensionless jitter index DLJ, LDLJ, etc. in some of the above embodiments. Based on the fluency time domain index, the quantitative value of the execution fluency of each actual surgical stage can be obtained, and function calculation can also be performed according to multiple fluency time domain indicators, for example, weighted calculation, to obtain the execution fluency of each actual surgical stage. Function calculation may also be performed based on the execution fluency of multiple actual surgical stages, for example, weighted calculation, to obtain the execution fluency of the entire surgery.

[0094] In some embodiments, frequency domain features of motion trajectories of multiple actual surgical stages (e.g., motion trajectories of the main manipulator handle and / or motion trajectories of the end of the surgical instrument) can be obtained, such as spectrum features and energy spectrum features. Based on the frequency domain features of the motion trajectories of each actual surgical stage, at least one fluency frequency domain index for measuring the fluency of the execution of the surgery corresponding to each actual surgical stage can be calculated respectively, such as the spectrum arc length SAL in some of the above embodiments, the spectrum arc length SPARC improved based on the spectrum arc length SAL, and the energy spectrum index SPMR, etc. Based on the fluency frequency domain index, the quantitative value of the execution fluency of each actual surgical stage can be obtained, and function calculation can also be performed according to multiple fluency frequency domain indicators, such as weighted calculation, to obtain the execution fluency of each actual surgical stage. It can be understood by those skilled in the art that the execution fluency of each actual surgical stage can also be comprehensively evaluated based on at least one fluency time domain index and at least one fluency frequency domain index of each actual surgical stage to obtain a more accurate fluency evaluation result. Function calculation can also be performed according to the fluency frequency domain indicators or comprehensive evaluation of multiple actual surgical stages, such as weighted calculation, to obtain the execution fluency of the entire surgery.

[0095] In some embodiments, each actual surgical stage can be divided into multiple sub-segments, and the execution fluency evaluation is performed for each sub-segment to obtain the execution fluency of each sub-segment, and the execution fluency of the corresponding actual surgical stage is obtained based on the execution fluency of each sub-segment in each actual surgical stage. In some embodiments, each actual surgical stage can be divided into multiple sub-segments based on the operation action (e.g., translation, rotation, clamping, etc.), for example, each operation action corresponds to a sub-segment, and each actual surgical stage is divided into multiple sub-segments according to the distribution of the operation action over time. In some embodiments, the motion trajectory of each sub-segment (e.g., the motion trajectory of the main manipulator handle and / or the motion trajectory of the end of the surgical instrument) is extracted, and the time domain characteristics and / or frequency domain characteristics of the motion trajectory of each sub-segment can be obtained. Based on the time domain characteristics and / or frequency domain characteristics of each sub-segment, at least one fluency time domain index and / or at least one fluency frequency domain index can be calculated. The corresponding sub-segment can be evaluated based on at least one fluency time domain index and / or at least one fluency frequency domain index to obtain the execution fluency corresponding to each sub-segment. In some embodiments, the execution fluency of the corresponding actual surgical stage can be estimated based on the execution fluency corresponding to each sub-segment, for example, it can be expressed as: i =f([λ i1 ,λ i2 ,λ i3 ,…λ iN ]), where Λ i is the execution fluency of the ith actual surgical stage, λ ijis the execution fluency of the jth sub-segment of the i-th actual surgical stage, j=1, 2, ..., N, and f is a fluency estimation function. In some embodiments, the execution fluency corresponding to each sub-segment can be obtained by weighted average estimation based on the execution fluency corresponding to each sub-segment, for example, it can be expressed as:

[0096]

[0097] Among them, w ij The weight for evaluating the fluency of the jth sub-segment of the i-th actual surgical stage.

[0098] In some embodiments, the quality levels of multiple actual surgical stages may be determined based on a mapping relationship between the execution fluency and quality levels of multiple actual surgical stages, and the quality levels of the operation information sets may be annotated based on the determined quality levels of the multiple actual surgical stages.

[0099] In some embodiments, multiple actual surgical stages can be labeled with surgical improvement suggestions based on the execution fluency of multiple actual surgical stages. For example, the execution fluency of multiple sub-segments of multiple actual surgical stages can be obtained, and surgical improvement suggestions for the corresponding actual surgical stages can be obtained based on the execution fluency of the multiple sub-segments. For example, if the execution fluency of one or several sub-segments of the actual surgical stage is lower than a set threshold, surgical improvement suggestions related to the execution fluency of the corresponding sub-segments and the operation can be given, and the operation information set is labeled with surgical improvement suggestions based on the determined surgical improvement suggestions.

[0100] Step 3: Divide the large amount of labeled operation information sets into training sets and test sets.

[0101] Step 4: Train the AI ​​evaluation model based on the labeled training set, such as training Figure 5B Artificial intelligence assessment model 500 is shown.

[0102] In some embodiments, training Figure 5BWhen the artificial intelligence evaluation model 500 is shown, a first loss function can be constructed for the partition part 510 of the artificial intelligence evaluation model 500, a second loss function can be constructed for the first sub-network 521 of the evaluation part 520 of the artificial intelligence evaluation model 500, and a third loss function can be constructed for the second sub-network 522 of the evaluation part 520 of the artificial intelligence evaluation model 500. In some embodiments, the partition part 510 in the artificial intelligence evaluation model 500 can be trained based on the first loss function first, and then the first sub-network 521 and the second sub-network 522 can be trained based on the second loss function and the third loss function respectively, so that each part of the artificial intelligence evaluation model 500 reaches the optimal state. In some embodiments, the first loss function, the second loss function and the third loss function can also be combined into a total loss function, and the partition part 510, the first sub-network 521 and the second sub-network 522 in the artificial intelligence evaluation model 500 can be jointly trained. For example, an optimization algorithm such as gradient descent can be used to minimize the total loss function of the artificial intelligence evaluation model 500 to update the model parameters in the artificial intelligence evaluation model 500.

[0103] Step 5: Test the trained AI evaluation model based on the labeled test set, for example, Figure 5B Artificial intelligence assessment model 500 is shown.

[0104] In some embodiments, the test set is used to evaluate and tune the artificial intelligence evaluation model 500. The performance of the artificial intelligence evaluation model 500 is evaluated by comparing the output of the artificial intelligence evaluation model 500 with the true label. According to the evaluation results, the hyperparameters or structure of the artificial intelligence evaluation model 500 can be adjusted to further improve the performance of the artificial intelligence evaluation model 500.

[0105] Those skilled in the art will appreciate that the training of the above artificial intelligence evaluation model is only an example, and the above training process can be adaptively adjusted or other methods can be used to train the artificial intelligence evaluation model as needed. For example, a semi-supervised method can also be used for training.

[0106] Fig. 6A 6000 (also referred to as "method 6000") for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure. Method 6000 can be implemented or executed by hardware, software, or firmware. In some embodiments, method 6000 can be performed by a surgical robot system (e.g., Figure 1 In some embodiments, the method 6000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 1For example, the control device of the surgical robot system (e.g., Figure 1 The control device 3) shown may include a processor configured to execute the method 6000. In some embodiments, these instructions may be stored on a computer-readable medium.

[0107] See also Fig. 6A In step 6001, a set of operation information of the operation is obtained. In some embodiments, step 6001 can be implemented by Figure 5A The operation information set of the surgery is obtained by a method similar to step 5001 in the above, which will not be described in detail here.

[0108] Continue to view Fig. 6A , in step 6003, surgical video data of the operation is obtained.

[0109] In some embodiments, surgical video data may be obtained from a surgical robotic system (e.g., Figure 1 The surgical robot system 100 shown in the figure is used to obtain the video data collected during the operation. For example, the video data during the operation can be collected through an endoscope to obtain the surgical video data.

[0110] Continue to view Fig. 6A , in step 6005, the surgery is evaluated based on the operation information set, the surgical video data and the artificial intelligence evaluation model.

[0111] In some embodiments, evaluating surgery based on an operation information set, surgical video data, and an artificial intelligence evaluation model includes: dividing the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages; and evaluating multiple actual surgical stages based on the multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages.

[0112] Similar to some of the above-mentioned embodiments, in some embodiments, the plurality of actual surgical stages include a plurality of actual surgical stages corresponding to a plurality of standard surgical stages of a standard surgical procedure.

[0113] Similar to some of the above embodiments, in some embodiments, the plurality of actual surgical stages also include additional actual surgical stages that are lacking in the standard surgical procedure, such as an emergency operation stage and / or a remedial operation stage. In step 6005, the operation information set and the surgical video data may be divided to obtain an operation information subset corresponding to the emergency operation stage and / or the remedial operation stage and a surgical stage video sequence, and the corresponding emergency operation stage and / or the remedial operation stage may be evaluated based on the operation information subset of the emergency operation stage and / or the remedial operation stage and the surgical stage video sequence.

[0114] Similar to some of the above embodiments, in some embodiments, during surgery, operation information corresponding to each sampling moment is obtained based on the sampling time interval t1 and integrated into an operation information set. For example, the operation information set can be integrated into an m×n dimensional matrix.

[0115] In some embodiments, the surgical video data is obtained, and the image frames of the surgical video data can be obtained based on using the time interval t2. In some embodiments, t1 and t2 can be configured to be equal or unequal.

[0116] In some embodiments, the artificial intelligence model includes a division part, and the division part of the artificial intelligence model can be used to divide the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences.

[0117] In some embodiments, the division part includes a first classification network for classifying the actual surgical stages, and the first classification network includes a feature extractor, a feature fusion device, a classifier, etc. The feature extractor extracts features from the operation information set and the surgical video data to obtain operation information features and image features, and the classifier is used to classify and identify the surgical stages based on the operation information features and image features, thereby dividing the operation information set and the surgical video data into multiple operation information subsets corresponding to multiple actual surgical stages and multiple surgical stage video sequences. The specific content is similar to that in some of the above embodiments, except that the input data of the artificial intelligence evaluation model includes surgical video data in addition to the operation information set, so that more features can be extracted to achieve accurate identification and division of the actual surgical stages.

[0118] In some embodiments, the artificial intelligence assessment model may further include an assessment part, and a surgical assessment may be performed through the assessment part of the artificial intelligence assessment model based on an operation information subset and a surgical video sequence to obtain a surgical assessment result.

[0119] In some embodiments, evaluating the surgery includes determining at least one surgery evaluation indicator for each of a plurality of actual surgery stages, wherein the at least one surgery evaluation indicator includes difficulty and / or quality. In some embodiments, evaluating the surgery also includes determining a surgery improvement suggestion for at least one of the plurality of actual surgery stages.

[0120] In some embodiments, the evaluation part of the artificial intelligence evaluation model may include a first subnetwork, which is used to determine at least one surgical evaluation indicator for each of multiple actual surgical stages. Among them, at least one surgical evaluation indicator includes difficulty and / or quality. The first subnetwork includes a second classification network that predicts at least one surgical evaluation indicator, and the second classification network predicts the difficulty or quality of each actual surgical stage. In some embodiments, the evaluation part of the artificial intelligence evaluation model may include a second subnetwork, which is used to determine surgical improvement suggestions for at least one actual surgical stage in the multiple actual surgical stages. The specific content is similar to that in some of the above embodiments, except that the input data of the artificial intelligence evaluation model includes surgical video data in addition to the operation information set, so that more features can be extracted to achieve a more accurate evaluation of the surgical evaluation indicators of each actual surgical stage, and to give more accurate surgical improvement suggestions.

[0121] Figure 6B FIG. 6 is a schematic diagram showing the structure of an artificial intelligence evaluation model 600 according to some embodiments of the present disclosure. Figure 6B As shown, the artificial intelligence evaluation model 600 includes a division part 610 and an evaluation part 620. The division part 610 is used to divide the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences. The evaluation part 620 is used to evaluate multiple actual surgical stages based on the multiple operation information subsets and the multiple surgical stage video sequences. The evaluation part 620 includes a first sub-network 621 and a second sub-network 622. The first sub-network 621 is used to determine at least one surgical evaluation index for each of the multiple actual surgical stages, and the second sub-network 622 is used to determine surgical improvement suggestions for at least one actual surgical stage in the multiple actual surgical stages.

[0122] like Figure 6B As shown, in some embodiments, the division part 610 may include a first input layer 6101, a first feature extractor 6103, a first classifier 6105, and a first output layer 6107, which are cascaded in sequence. The first input layer 6101 is used to receive input data, for example, the input data includes an operation information set and surgical video data. The first feature extractor 6103 is used to perform feature extraction and feature fusion to extract operation information features and image features and fuse them to obtain fusion features. For example, in some embodiments, Figure 6CAs shown, the first feature extractor 6103 includes a first operation information feature extractor 61031, a first image feature extractor 61032 and a first feature fusion unit 61033. The first operation information feature extractor 61031 is used to extract the first-level operation information feature, the first image feature extractor 61032 is used to extract the first-level image feature, and the first feature fusion unit 61033 is used to fuse the operation information feature and the image feature to obtain the first-level fusion feature. The first-level operation information feature and the first-level image feature can be coarse features for distinguishing surgical stages. For example, the first-level operation information feature and the first-level image feature can include features associated with surgical instruments to distinguish surgical stages by the type of surgical instruments; or the first-level operation information feature and the first-level image feature can include features associated with surgical actions to distinguish surgical stages by features associated with surgical actions; or the first-level operation information feature can also include multiple features associated with surgical instruments and surgical actions to distinguish surgical stages. The first classifier 6105 is used to classify surgical stages based on the first-level fusion feature. For example, surgical stages can be divided into C1 categories. The first output layer 6107 is used to process and output the classification results to output multiple operation information subsets and multiple operation stage video sequences of multiple actual operation stages of the surgical process. For example, multiple operation information subsets and multiple operation stage video sequences of multiple actual operation stages can be output in sequence according to the operation time sequence.

[0123] like Figure 6BAs shown, in some embodiments, the first subnetwork 621 may include a second input layer 6211, a second feature extractor 6213, a second classifier 6215, and a second output layer 6217, which are cascaded in sequence. The second input layer 6211 is connected to the output of the first output layer 6107 to receive information output by the first output layer 6107, such as multiple operation information subsets of multiple actual surgical stages of the surgical process and multiple surgical stage video sequences output by the first output layer 6107. The second feature extractor 6213 can obtain part or all of the first-level fusion features output by the first feature extractor 6103 as the first fusion shared features, and can also extract more detailed operation information features and image features from the operation information subsets of multiple actual surgical stages and the surgical stage video sequences of multiple actual surgical stages and fuse them, such as extracting second-level operation information features and second-level image features, and fusing the second-level operation information features and second-level image features into second-level fusion features. For example, the second-level operation information features may include detailed features related to the flexibility and smoothness of the operation movements in various stages of the operation, such as time domain features and / or frequency domain features extracted from the motion trajectory (for example, the motion trajectory of the main manipulator handle and / or the motion trajectory of the end of the surgical instrument) that can reflect at least a part of the smoothness of the execution of the operation; the second-level image features may include bleeding volume, bleeding time, and other features related to the difficulty and / or quality of the operation that can be identified from the picture. Figure 6C Similarly, the first feature extractor 6103 in Fig.6D As shown, the second feature extractor 6213 may include a second operation information feature extractor 62131, a second image feature extractor 62132, and a second feature fusion unit 62133, and the specific contents are not described in detail again. The second classifier 6215 classifies and evaluates the surgical evaluation indicators of each actual surgical stage based on the first shared fusion feature and the second-level fusion feature, where the surgical evaluation indicators include difficulty and / or quality. For example, difficulty can be divided into C2 category, and quality can be divided into C3 category. The second output layer 6217 is used to output the classification results of the second classifier 6215 to output the evaluation results of the surgical evaluation indicators of multiple actual surgical stages of the surgical process.

[0124] like Figure 6BAs shown, in some embodiments, the second subnetwork 622 may include a third input layer 6221, a third feature extractor 6223, a third classifier 6225, and a third output layer 6227 that are cascaded in sequence. The third input layer 6221 is connected to the output of the first output layer 6107 to receive information output by the first output layer 6107, such as a plurality of operation information subsets and a plurality of operation stage video sequences of a plurality of actual surgical stages of a surgical process output by the first output layer 6107. The third feature extractor 6223 is connected to the third input layer 6221, and the third feature extractor 6223 may be connected to the output end of the first feature extractor 6103 and / or the second feature extractor 6213. The third feature extractor 6223 may be used to obtain all or part of the first-level fusion features output by the first feature extractor 6103 as the second fusion shared features, and the third feature extractor 6223 may also be used to obtain all or part of the second-level fusion features output by the second feature extractor 6213 as the third fusion shared features. The third feature extractor 6223 can also extract other detailed operation information features from the operation information subsets of multiple actual operation stages and the video sequences of multiple operation stages and perform feature fusion to obtain third-level fusion features, such as extracting third-level operation information features and third-level image features and fusion to obtain third-level fusion features. For example, the third-level operation information features may include features associated with the selection of operation instruments and operation duration in each stage of the operation, and the third-level image features may include feature information such as the coordination of surgical instruments during the operation and the positional relationship between surgical instruments and lesions in each stage of the operation, so as to focus on the details of each stage of the operation. Figure 6C Similarly, the first feature extractor 6103 in Fig. 6E As shown, the third feature extractor 6223 may include a third operation information feature extractor 62231, a third image feature extractor 62232 and a third feature fusion unit 62233, and the specific contents are not described in detail. The third classifier 6225 evaluates the surgical suggestions of each actual surgical stage based on the second fusion shared features, the third fusion shared features, and the third-level fusion features to determine an improvement suggestion vector associated with the surgical improvement suggestion. For example, each dimension of the vector represents an improvement measure or feature, such as a surgical improvement suggestion on the selection of surgical instruments, or a surgical improvement suggestion on the control of surgical duration, or a surgical improvement suggestion on surgical operation actions, etc. The third output layer 6227 processes the output vector of the surgical suggestion to output the surgical improvement suggestion. For example, the third output layer 6227 may use a natural language processing network (NLP) to process the output vector to output the text information of the surgical improvement suggestion.

[0125] Similar to some of the above embodiments, the division part and the evaluation part (including the first sub-network and the second sub-network) may adopt a deep neural network model, such as a recurrent neural network (RNN), a convolutional neural network (CNN) and other network models. Those skilled in the art will appreciate that the above are only examples, and the division part, the first sub-network and the second sub-network may also adopt other network models.

[0126] Some embodiments of the present disclosure provide a method for training an artificial intelligence evaluation model, for example, Figure 6B Artificial intelligence assessment model 600 is shown.

[0127] In some embodiments, the training method described in some of the above embodiments may be used. Figure 5B The artificial intelligence evaluation model 500 shown is trained similarly to the method Figure 6B The artificial intelligence evaluation model 600 shown in FIG. Figure 6B When constructing an artificial model of the artificial intelligence evaluation model 600 shown in the figure, a large amount of surgical video data needs to be obtained, one surgical video data corresponds to one surgical video data, and each surgical video data is annotated. The surgical video data can be annotated in a manner similar to the annotation method of the operation information set, and the video images in the surgical video data can be annotated frame by frame, including the corresponding actual surgical stage annotation, the difficulty level annotation of each actual surgical stage, the quality level annotation of each actual surgical stage, and the surgical improvement suggestion annotation of at least one actual surgical stage. The artificial intelligence evaluation model 600 can be trained and tested using the annotated operation information set and surgical video data to obtain the trained and optimized artificial intelligence evaluation model 600. The specific content has been described in detail in some of the above embodiments, and will not be described in detail here.

[0128] FIG. 7 shows a flow chart of a method 7000 (also referred to as “method 7000”) for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure. Method 7000 may be implemented or executed by hardware, software, or firmware. In some embodiments, method 7000 may be performed by a surgical robot system (e.g., Figure 1 In some embodiments, the method 7000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 1 For example, the control device of the surgical robot system (e.g., Figure 1 The control device 3) shown may include a processor configured to execute the method 7000. In some embodiments, these instructions may be stored on a computer-readable medium.

[0129] See also Fig. 7AIn step 7001, a set of operation information of the operation is obtained. In some embodiments, step 7001 can be implemented by Figure 5A The operation information set of the surgery is obtained by a method similar to step 5001 in FIG. 6 or step 6001 in FIG. 6 , which will not be described in detail here.

[0130] Continue to view Fig. 7A In step 7003, surgical video data of the operation is obtained. In some embodiments, step 7003 can be implemented by Fig. 6A The surgical video data of the operation is obtained by a method similar to step 6003 in the above, which will not be described in detail here.

[0131] Continue to view Fig. 7A , in step 7005, a preoperative medical data set is obtained.

[0132] In some embodiments, the preoperative medical data set includes at least a portion of lesion images, pathological grades, disease history, etc. In some embodiments, preoperative medical images, such as ultrasound images, X-ray images, optical coherence tomography (OCT) scans, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, and positron emission tomography (PET) scans, may be obtained, and lesion images may be obtained through the preoperative medical images.

[0133] Continue to view Fig. 7A , in step 7007, the surgery is evaluated based on the operation information set, the surgical video data, the preoperative medical data set and the artificial intelligence evaluation model.

[0134] In some embodiments, evaluating surgery based on an operation information set, surgical video data, a preoperative medical data set, and an artificial intelligence evaluation model includes: dividing the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages; and evaluating multiple actual surgical stages based on the preoperative medical data set and the multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages.

[0135] In some embodiments, the multiple actual surgical stages include multiple actual surgical stages corresponding to multiple standard surgical stages of the standard surgical process. For example, the multiple actual surgical stages may include one or more of the pneumoperitoneum establishment stage, surgical instrument positioning stage, surface fat clearing stage, tissue freedom stage, blood vessel separation stage, lesion resection stage, tissue suturing stage, etc. in some of the above-mentioned embodiments.

[0136] In some embodiments, the plurality of actual surgical stages also include additional actual surgical stages that are lacking in the standard surgical procedure, for example, the additional actual surgical stages may include the emergency operation stage and / or the remedial operation stage in some of the above embodiments. In step 7007, the operation information set and the surgical video data may be divided to obtain an operation information subset corresponding to the emergency operation stage and / or the remedial operation stage and a surgical stage video sequence, and the corresponding emergency operation stage and / or the remedial operation stage may be evaluated based on the preoperative medical data set, the operation information subset of the emergency operation stage and / or the remedial operation stage, and the surgical stage video sequence.

[0137] Similar to some of the above embodiments, the artificial intelligence model includes a division part, and the division part of the artificial intelligence model can be used to divide the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences. The specific content has been described in detail in some of the above embodiments and will not be repeated here.

[0138] In some embodiments, the artificial intelligence evaluation model may further include an evaluation part, and the surgical evaluation may be performed through the evaluation part of the artificial intelligence evaluation model based on the operation information subset, the surgical video sequence and the preoperative medical data to obtain the surgical evaluation result. In some embodiments, evaluating the surgery includes determining at least one surgical evaluation index for each of the multiple actual surgical stages, wherein the at least one surgical evaluation index includes difficulty and / or quality. In some embodiments, evaluating the surgery also includes determining surgical improvement suggestions for at least one of the multiple actual surgical stages. Similar to some of the above embodiments, the evaluation part of the artificial intelligence evaluation model may include a first subnetwork, which is used to determine at least one surgical evaluation index for each of the multiple actual surgical stages. The evaluation part of the artificial intelligence evaluation model may include a second subnetwork, which is used to determine surgical improvement suggestions for at least one of the multiple actual surgical stages. The specific content is similar to some of the above embodiments, except that in the evaluation part, the output and input data of the artificial intelligence model include preoperative medical data in addition to the operation information and surgical video data, and the preoperative medical data can assist in determining the surgical evaluation index of each actual surgical stage, such as difficulty, or assist in determining the surgical improvement suggestions for the actual surgical stage.

[0139] Figure 7B FIG. 7 is a schematic diagram showing the structure of an artificial intelligence evaluation model 700 according to some embodiments of the present disclosure. Figure 6BSimilar to the artificial intelligence evaluation model 600 in FIG. 1 , the artificial intelligence evaluation model 700 includes a partitioning part 710 and an evaluation part 720. The evaluation part 720 includes a first sub-network 721 and a second sub-network 722. The partitioning part 710 may include a first input layer 7101, a first feature extractor 7103, a first classifier 7105, and a first output layer 7107, which are cascaded in sequence. Figure 7C As shown, the first feature extractor 7103 includes a first operation information feature extractor 71031, a first image feature extractor 71032 and a first feature fusion unit 71033. Figure 6B The first sub-network 721 may include a second input layer 7211, a second feature extractor 7213, a second classifier 7215, and a second output layer 7217, which are cascaded in sequence. The second sub-network 722 may include a third input layer 7221, a third feature extractor 7223, a third classifier 7225, and a third output layer 7227, which are cascaded in sequence. Figure 6B The difference between the AI ​​evaluation model 600 in Figure 7B The input data of the artificial intelligence evaluation model 700 in the embodiment also includes a preoperative medical data set. The preoperative medical data set is input to the second input layer 7211 and the third input layer 7221, so that the second feature extractor 7213 and the third feature extractor 7223 also need to perform feature extraction on the preoperative medical data set to obtain preoperative medical data features. In some embodiments, Fig.7D , Fig. 7EAs shown, the second feature extractor 7213 includes a second operation information feature extractor 72131, a second image feature extractor 72132, a first preoperative medical feature extractor 72133, and a second feature fusion device 72134. The third feature extractor 7223 may include a third operation information feature extractor 72231, a third image feature extractor 72232, a second preoperative medical feature extractor 72233, and a third feature fusion device 72234. The second operation information feature extractor 72131, the second image feature extractor 72132, the third operation information feature extractor 72231, and the third image feature extractor 72232 are similar to the second operation information feature extractor 62131, the second image feature extractor 62132, the third operation information feature extractor 62231, and the third image feature extractor 62232 in some of the above embodiments, and are not expanded here. The first preoperative medical feature extractor 72133 and the second preoperative medical feature extractor 72233 are respectively used to extract preoperative medical features. In some embodiments, a preoperative medical feature extractor may also be set in the network, and the preoperative medical features may be shared in the first sub-network 721 and the second sub-network 722. The second feature fusion unit 72134 and the second feature fusion unit 72134 are respectively used to fuse the operation information features, the image features and the preoperative medical features, so as to determine the surgical evaluation index and the surgical improvement suggestion based on the corresponding fusion features.

[0140] Similar to some of the above embodiments, the division part and the evaluation part (including the first sub-network and the second sub-network) may adopt a deep neural network model, such as a recurrent neural network (RNN), a convolutional neural network (CNN) and other network models. Those skilled in the art will appreciate that the above are only examples, and the division part, the first sub-network and the second sub-network may also adopt other network models.

[0141] Some embodiments of the present disclosure provide a method for training an artificial intelligence evaluation model, for example, Figure 7B Artificial intelligence assessment model 700 is shown.

[0142] In some embodiments, the training method described in some of the above embodiments may be used. Figure 6B The artificial intelligence evaluation model 600 shown is trained similarly to the method Figure 7B The artificial intelligence evaluation model 700 shown in FIG. Figure 7B When an artificial model of the artificial intelligence evaluation model 700 shown in the figure is constructed, a large number of operation information sets, surgical video data and preoperative medical data can be obtained, and they are labeled respectively to obtain training sets and test sets. The artificial intelligence evaluation model 700 is trained and tested based on the training sets and the test sets to obtain a trained and optimized artificial intelligence evaluation model 700.

[0143] FIG8 shows a flow chart of a method 8000 (also referred to as “method 8000”) for evaluating surgery based on artificial intelligence according to some embodiments of the present disclosure. Method 8000 may be implemented or executed by hardware, software, or firmware. In some embodiments, method 8000 may be performed by a surgical robot system (e.g., Figure 1 In some embodiments, the method 8000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 1 For example, the control device of the surgical robot system (e.g., Figure 1 The control device 3) shown may include a processor configured to execute the method 8000. In some embodiments, these instructions may be stored on a computer-readable medium.

[0144] See also Fig. 8A In step 8001, a set of operation information of the operation is obtained. In some embodiments, step 8001 can be implemented by Figure 5A Step 5001, Fig. 6A Step 6001 or Fig. 7A The operation information set of the surgery is obtained by a method similar to step 7001, which will not be repeated here.

[0145] Continue to view Fig. 8A In step 8003, surgical video data of the surgery is obtained. In some embodiments, step 8003 can be performed by Fig. 6A Step 6003 or Fig. 7A The surgical video data of the operation is obtained by a method similar to step 7003, which will not be repeated here.

[0146] Continue to view Fig. 8A , in step 8005, a postoperative medical data set is obtained.

[0147] In some embodiments, the postoperative medical data set includes at least a portion of incision recovery, complication, and medication. In some embodiments, incision recovery includes incision size, healing time, etc., complication includes complication type, complication duration, etc., and medication includes medication type, medication time, etc.

[0148] Continue to view Fig. 8A , in step 8007, the surgery is evaluated based on the operation information set, the surgical video data, the postoperative medical data set and the artificial intelligence evaluation model.

[0149] In some embodiments, evaluating surgery based on an operation information set, surgical video data, a postoperative medical data set, and an artificial intelligence evaluation model includes: dividing the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages; and evaluating multiple actual surgical stages based on the postoperative medical data set and the multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages.

[0150] Similar to some of the above-mentioned embodiments, in some embodiments, the plurality of actual surgical stages include a plurality of actual surgical stages corresponding to a plurality of standard surgical stages of a standard surgical procedure.

[0151] Similar to some of the above embodiments, in some embodiments, the plurality of actual surgical stages also include additional actual surgical stages that are lacking in the standard surgical procedure, such as an emergency operation stage and / or a remedial operation stage. In step 8007, the operation information set and the surgical video data may be divided to obtain an operation information subset corresponding to the emergency operation stage and / or the remedial operation stage and a surgical stage video sequence, and the corresponding emergency operation stage and / or the remedial operation stage may be evaluated based on the postoperative medical data set, the operation information subset of the emergency operation stage and / or the remedial operation stage, and the surgical stage video sequence.

[0152] Similar to some of the above embodiments, the artificial intelligence model includes a division part, and the division part of the artificial intelligence model can be used to divide the operation information set and the surgical video data into multiple operation information subsets and multiple surgical stage video sequences. The specific content has been described in detail in some of the above embodiments and will not be repeated here.

[0153] In some embodiments, the artificial intelligence evaluation model may further include an evaluation part, and the evaluation part of the artificial intelligence evaluation model may be used to perform surgical evaluation based on the operation information subset, the surgical video sequence, and the postoperative medical data to obtain the surgical evaluation result. In some embodiments, evaluating the surgery includes determining at least one surgical evaluation index for each of the multiple actual surgical stages, wherein the at least one surgical evaluation index includes difficulty and / or quality. In some embodiments, evaluating the surgery also includes determining surgical improvement suggestions for at least one of the multiple actual surgical stages. Similar to some of the above embodiments, the evaluation part of the artificial intelligence evaluation model may include a first subnetwork, the first subnetwork is used to determine at least one surgical evaluation index for each of the multiple actual surgical stages, and the evaluation part of the artificial intelligence evaluation model may include a second subnetwork, the second subnetwork is used to determine surgical improvement suggestions for at least one of the multiple actual surgical stages. The specific content is similar to some of the above embodiments, except that in the evaluation part, the input data of the artificial intelligence model includes a postoperative medical data set in addition to the operation information set and the surgical video data, and the postoperative medical data set can assist in determining the surgical evaluation indicators of each actual surgical stage, such as quality, etc., and the surgical improvement suggestions for the actual surgical stage.

[0154] Figure 8B FIG. 8 is a schematic diagram showing the structure of an artificial intelligence evaluation model 800 according to some embodiments of the present disclosure. Figure 7B Similar to the artificial intelligence evaluation model 700 in FIG. 8 , the artificial intelligence evaluation model 800 includes a partitioning part 810 and an evaluation part 820. The evaluation part 820 includes a first sub-network 821 and a second sub-network 822. The partitioning part 810 may include a first input layer 8101, a first feature extractor 8103, a first classifier 8105, and a first output layer 8107, which are cascaded in sequence. Figure 8C As shown, the first feature extractor 8103 includes a first operation information feature extractor 81031, a first image feature extractor 81032 and a first feature fusion unit 81033. Figure 7B The first sub-network 821 may include a second input layer 8211, a second feature extractor 8213, a second classifier 8215, and a second output layer 8217, which are cascaded in sequence. The second sub-network 822 may include a third input layer 8221, a third feature extractor 8223, a third classifier 8225, and a third output layer 8227, which are cascaded in sequence. Figure 8BThe input data of the artificial intelligence evaluation model 800 in the embodiment also receives the postoperative medical data set. The postoperative medical data set is input to the second input layer 8211 and the third input layer 8221, so that the second feature extractor 8213 and the third feature extractor 8223 need to perform feature extraction on the postoperative medical data set to obtain postoperative medical data features. In some embodiments, Fig.8D , Fig. 8E As shown, the second feature extractor 8213 includes a second operation information feature extractor 82131, a second image feature extractor 82132, a first postoperative medical feature extractor 82133, and a second feature fusion device 82134. The third feature extractor 8223 may include a third operation information feature extractor 82231, a third image feature extractor 82232, a second postoperative medical feature extractor 82233, and a third feature fusion device 82234. The second operation information feature extractor 82131, the second image feature extractor 82132, the third operation information feature extractor 82231, and the third image feature extractor 82232 are similar to the second operation information feature extractor 72131, the second image feature extractor 72132, the third operation information feature extractor 72231, and the third image feature extractor 72232 in some of the above embodiments, and are not expanded here. The first postoperative medical feature extractor 82133 and the second postoperative medical feature extractor 82233 are respectively used to extract postoperative medical features. In some embodiments, a postoperative medical feature extractor may also be set in the network, and the postoperative medical features may be shared in the first sub-network 821 and the second sub-network 822. The second feature fusion unit 82134 and the second feature fusion unit 82134 are respectively used to fuse the operation information features, the image features and the postoperative medical features, so as to determine the surgical evaluation index and the surgical improvement suggestion based on the corresponding fusion features.

[0155] Similar to some of the above embodiments, the division part and the evaluation part (including the first sub-network and the second sub-network) may adopt a deep neural network model, such as a recurrent neural network (RNN), a convolutional neural network (CNN) and other network models. Those skilled in the art will appreciate that the above are only examples, and the division part, the first sub-network and the second sub-network may also adopt other network models.

[0156] Some embodiments of the present disclosure provide a method for training an artificial intelligence evaluation model, for example, Figure 8B Artificial intelligence assessment model 800 is shown.

[0157] In some embodiments, the training method described in some of the above embodiments may be used. Figure 7B The artificial intelligence evaluation model 700 shown is trained similarly to the method Figure 8BThe artificial intelligence evaluation model 800 shown in FIG. Figure 8B When the artificial intelligence evaluation model 800 shown is used, a large number of operation information sets, surgical video data, and postoperative medical data sets can be obtained and annotated respectively to obtain training sets and test sets. The artificial intelligence evaluation model 800 is trained and tested based on the training sets and the test sets to obtain a trained and optimized artificial intelligence evaluation model 800.

[0158] In some embodiments, the method for evaluating surgery based on artificial intelligence may include obtaining an operation information set, surgical video data, a preoperative medical data set, and a postoperative medical data set for the surgery, and evaluating the surgery based on the operation information set, surgical video data, preoperative medical data set, postoperative medical data set, and a pre-trained artificial intelligence evaluation model. The specific method is similar to the method in some of the above embodiments (for example, Fig. 7A Method 7000 or Fig. 8A The method 8000 in FIG. Fig. 8A Taking method 8000 in as an example, in step 8005, a postoperative medical data set is obtained, and a preoperative medical data set may also be obtained, the preoperative medical data set including at least a part of lesion images, pathological grading, disease history, etc. Then, the surgery may be evaluated based on the operation information set, the surgical video data, the preoperative medical data set, the postoperative medical data set, and the pre-trained artificial intelligence evaluation model, including: dividing the operation information set and the surgical video data into multiple operation information subsets corresponding to multiple actual surgical stages and multiple surgical stage video sequences; and evaluating multiple actual surgical stages based on the preoperative medical data set, the postoperative medical data set, and multiple operation information subsets corresponding to multiple actual surgical stages and multiple surgical stage video sequences. The specific content is similar to some of the above embodiments and will not be expanded here.

[0159] In some embodiments, the artificial intelligence evaluation model provided can evaluate the surgery based on the operation information set, the surgical video data, the preoperative medical data set, and the postoperative medical data set. The structure of the artificial intelligence evaluation model is similar to the artificial intelligence models in some of the above embodiments (e.g., Figure 7B Artificial intelligence models in 700 or Figure 8B The structure of the artificial intelligence model 800 in FIG. Figure 8BTaking the artificial intelligence model 800 in the example, the input data of the artificial intelligence model includes preoperative medical data in addition to operation information, surgical video data, and postoperative medical data set. The division part 810 divides the operation information set and the surgical video data set to obtain multiple operation information subsets and multiple surgical stage video sequences corresponding to multiple actual surgical stages. The evaluation part 820 can also accept the preoperative medical data set on the basis of accepting multiple operation information subsets, multiple surgical stage video sequences, and postoperative medical data sets, so as to evaluate the surgery based on multiple operation information subsets, multiple surgical stage video sequences, postoperative medical data sets, and preoperative medical data sets. For example, the second input layer 8211 and the third input layer 8221 can also accept the preoperative medical data set respectively, and the second feature extractor 8213 and the third feature extractor 8223 can also extract features from the preoperative medical data set respectively to obtain preoperative medical data features, and the operation information features, image features, postoperative medical features, and preoperative medical features can be fused to determine the surgical evaluation index and surgical improvement suggestions based on the corresponding fusion features. The specific content is similar to some of the above embodiments and will not be expanded here. Those skilled in the art will understand that when training an artificial intelligence evaluation model, a large number of operation information sets, surgical video data, postoperative medical data sets and preoperative medical data sets can be obtained and labeled to obtain training sets and test sets. The artificial intelligence evaluation model is trained and tested based on the training set and the test set to obtain a trained and optimized artificial intelligence evaluation model.

[0160] In some embodiments, method 5000, method 6000, method 7000 or method 8000 may also include step 5003, step 6005, step 7007 or step 8007 in which evaluating surgery also includes: determining the overall difficulty and / or quality of the surgery based on at least one surgical evaluation indicator for each actual surgical stage in multiple actual surgical stages.

[0161] In some embodiments, the evaluation results of the surgical evaluation indicators of multiple actual surgical stages can be obtained, such as the difficulty level and the quality level, and the difficulty level and the quality level can be converted into 0-10 evaluation indicator quantization values, and the evaluation results of the surgical evaluation indicators of multiple actual surgical stages are assigned corresponding weights. The evaluation indicator quantization values ​​of multiple actual surgical stages can be calculated by weighted summation and other calculation methods to obtain the quantization value of the overall difficulty and / or quality of the operation. In some embodiments, the quantization value of the overall difficulty and / or quality of the operation can be further converted into the level of overall difficulty and / or quality. In some embodiments, the multiple actual surgical stages may include multiple actual surgical stages corresponding to multiple standard surgical stages of the standard surgical process, for example, one or more of the establishment of pneumoperitoneum stage, the surgical instrument placement stage, the surface fat cleaning stage, the tissue free stage, the blood vessel separation stage, the lesion removal stage, the tissue suturing stage, etc. The preliminary preparation stage, such as the establishment of pneumoperitoneum stage and the surgical instrument placement stage, is the preparation work for the formal start of the surgical operation. In the entire surgical process, the importance of the preliminary preparation stage is lower than the surgical implementation stage, such as the surface fat cleaning stage, the tissue free stage, the blood vessel separation stage, the lesion removal stage, the tissue suturing stage, etc. The operation implementation stage is the key to the operation process, which greatly affects the difficulty and quality of the operation. Therefore, a higher weight can be given to the operation implementation stage, and a lower weight can be given to the preparatory stage. Thus, the quantitative value of the overall difficulty and / or quality of the operation can be obtained by weighted summation and other calculation methods. In some embodiments, multiple actual operation stages can also include additional actual operation stages that are lacking in the standard operation process, such as emergency operation stages and / or remedial operation stages, and emergency weights and / or remedial weights are given to the additional actual operation stages. It should be understood that the emergency weight is the weight given to the additional actual operation stage (such as the emergency operation stage) corresponding to the emergency operation, which can be a higher level of weight, which can greatly reflect the overall difficulty of the operation. The remedial weight is the weight given to the additional actual operation stage (such as the remedial operation stage) corresponding to the remedial operation, which can negatively reflect the overall quality of the operation. It can be understood by those skilled in the art that the corresponding weight size can be adjusted according to the correlation between the operation evaluation index of each actual operation stage and the overall difficulty and / or quality of the operation.

[0162] In some embodiments, the overall difficulty and / or quality of the surgery can also be determined directly based on the artificial intelligence assessment model, for example, Figure 5B , Figure 6B , Figure 7B or Figure 8B The second output layer 5217, the second output layer 6217, the second output layer 7217 or the second output layer 8217 adds one-dimensional or multi-dimensional output data for outputting the overall difficulty and / or quality of the surgery. Figure 5B , Figure 6B , Figure 7B or Figure 8B When using artificial intelligence assessment model 500, artificial intelligence assessment model 600, artificial intelligence assessment model 700 or artificial intelligence assessment model 800, it is necessary to add annotations on the overall difficulty and / or quality of the surgery.

[0163] In some embodiments, method 5000, method 6000, method 7000 or method 8000 further includes determining the charging standard level and / or nursing level of the operation based on the overall difficulty and / or quality of the operation. It can be understood by those skilled in the art that the control device 3 of the surgical robot system 100 can also be configured to provide multiple surgical difficulty levels, multiple surgical quality levels, multiple surgical charging standard levels and multiple nursing levels for each surgical procedure, and multiple surgical difficulty levels, multiple surgical quality levels or their permutations and combinations can have a predetermined mapping relationship with multiple surgical charging standard levels and multiple nursing levels. For example, multiple surgical charging standard levels can include first-level, second-level, third-level, and fourth-level charging standards, and multiple nursing levels can include first-level, second-level, third-level, and fourth-level nursing standards. The permutations and combinations of multiple surgical difficulty levels and multiple surgical quality levels can form a mapping relationship with four levels of charging standards and four nursing standards, respectively. In this way, the optimal nursing standard and the most reasonable charging standard can be matched according to the actual surgical situation. The charging standard can provide a reference for charging patients and for the use fee of the surgical robot.

[0164] The method for evaluating surgery based on artificial intelligence disclosed in the present invention can objectively evaluate the difficulty and / or quality of each of the multiple actual surgical stages of the surgery, and can also give corresponding surgical improvement suggestions for at least one of the multiple actual surgical stages of the surgery, which helps surgeons to perform self-testing of skills and conduct targeted training based on the feedback results, shortening the surgeon's training cycle and improving the surgeon's surgical quality, and achieving effective surgical monitoring and management.

[0165] The method for evaluating surgery based on artificial intelligence disclosed in the present invention can achieve an objective determination of the overall difficulty and / or quality of the surgery, thereby matching charging standards and postoperative care services suitable for different levels of surgical difficulty and / or surgical quality, which is highly efficient and can form effective surgical monitoring and management. In addition, the surgical evaluation process does not require manual evaluation, thereby avoiding medical disputes caused by human factors.

[0166] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a computer to configure the computer to perform some or all steps in the method of some embodiments of the present disclosure, such as Figure 5A , Fig. 6A , Fig. 7A or Fig. 8A Some or all of the steps in method 5000, method 6000, method 7000, or method 8000 disclosed in the computer program. Examples of computer readable storage media include storage of computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0167] In some embodiments, the present disclosure provides a computer system, the computer system includes a memory and a processor. The memory can be used to store at least one instruction, and the processor is coupled to the memory to execute at least one instruction to execute some or all steps in the method of the present disclosure, such as Figure 5A , Fig. 6A , Fig. 7A or Fig. 8A Some or all of the steps in method 5000, method 6000, method 7000 or method 8000 disclosed in.

[0168] Fig. 9 900 according to some embodiments of the present disclosure. Fig. 9 The computer system 900 may include a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read-only memory (ROM) 903, and a system bus 905 connecting the various components. The computer system 900 may also include an input / output system, and a large-capacity storage device 907 for storing an operating system 913, application programs 914, and other program modules 915. The input / output system includes an input / output controller 906 mainly composed of a display 908 and an input device 909.

[0169] The mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 or computer readable medium provides non-volatile storage for the computer device. The mass storage device 907 may include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0170] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory and mass storage devices may be collectively referred to as memory.

[0171] The computer system 900 may be connected to a network 912 via a network interface unit 911 connected to the system bus 905 .

[0172] The system memory 904 or the mass storage device 907 is also used to store one or more instructions. The central processor 901 implements all or part of the steps of the method in some embodiments of the present disclosure by executing the one or more instructions, for example, Figure 5A , Fig. 6A , Fig. 7A or Fig. 8A Some or all of the steps in method 5000, method 6000, method 7000 or method 8000 disclosed in.

[0173] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor to enable a computer to execute some or all steps of the method in some embodiments of the present disclosure, for example, Figure 5A , Fig. 6A , Fig. 7A or Fig. 8ASome or all of the steps in method 5000, method 6000, method 7000 or method 8000 disclosed in the computer program. Examples of computer readable storage media include storage of computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk and optical data storage device.

[0174] Through the above description of the implementation methods, the technical personnel in the relevant field can clearly understand that the present disclosure can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present disclosure can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method in any embodiment of the present disclosure.

[0175] Note that the above are only exemplary embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A method for evaluating surgery based on artificial intelligence, characterized in that The surgery includes multiple actual surgical stages, and the method includes: Obtaining an operation information set of the surgery; and The operation is evaluated based on the operation information set and a pre-trained artificial intelligence evaluation model.

2. The method according to claim 1, characterized in that Evaluating the surgery based on the operation information set and the pre-trained artificial intelligence evaluation model includes: dividing the operation information set into a plurality of operation information subsets corresponding to the plurality of actual surgical stages; and The plurality of actual surgical stages are evaluated based on the plurality of operation information subsets corresponding to the plurality of actual surgical stages.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining surgical video data of the surgery; and The surgery is evaluated based on the operation information set, the surgical video data and the artificial intelligence evaluation model.

4. The method according to claim 3, characterized in that Evaluating the surgery based on the operation information set, the surgery video data and the artificial intelligence evaluation model includes: Dividing the operation information set and the surgical video data into a plurality of operation information subsets and a plurality of surgical stage video sequences corresponding to the plurality of actual surgical stages; and The plurality of actual surgical stages are evaluated based on the plurality of operation information subsets corresponding to the plurality of actual surgical stages and the plurality of surgical stage video sequences.

5. The method according to claim 3, characterized in that: The method further comprises: Obtaining preoperative medical data sets; The surgery is evaluated based on the operation information set, the surgical video data, the preoperative medical data set and the artificial intelligence evaluation model.

6. The method according to claim 5, characterized in that Evaluating the surgery based on the operation information set, the surgery video data, the preoperative medical data set and the artificial intelligence evaluation model includes: Dividing the operation information set and the surgical video data into a plurality of operation information subsets and a plurality of surgical stage video sequences corresponding to the plurality of actual surgical stages; and The plurality of actual surgical stages are evaluated based on the pre-operative medical dataset and the plurality of operation information subsets corresponding to the plurality of actual surgical stages and the plurality of surgical stage video sequences.

7. The method according to claim 3, characterized in that The method further comprises: Access to post-operative medical data sets; and The surgery is evaluated based on the operation information set, the surgical video data, the postoperative medical data set and the artificial intelligence evaluation model.

8. The method according to claim 7, characterized in that Evaluating the surgery based on the operation information set, the surgery video data, the postoperative medical data set and the artificial intelligence evaluation model includes: Dividing the operation information set and the surgical video data into a plurality of operation information subsets and a plurality of surgical stage video sequences corresponding to the plurality of actual surgical stages; and The plurality of actual surgical stages are evaluated based on the postoperative medical data set, the plurality of operation information subsets corresponding to the plurality of actual surgical stages, and the plurality of surgical stage video sequences.

9. The method according to any one of claims 1 to 8, characterized in that Evaluation of the surgery includes: Determining at least one surgical evaluation indicator for each of the plurality of actual surgical stages, wherein the at least one surgical evaluation indicator comprises difficulty and / or quality; and / or A surgical improvement suggestion for at least one actual surgical phase of the plurality of actual surgical phases is determined.

10. The method according to any one of claims 1 to 8, characterized in that The artificial intelligence evaluation model includes a division part and an evaluation part; The dividing part is used to divide the operation information set into a plurality of operation information subsets, or the dividing part is used to divide the operation information set and the surgical video data into a plurality of operation information subsets and a plurality of surgical stage video sequences; The evaluation part is used to evaluate the multiple actual surgical stages based on the multiple operation information subsets, or, the evaluation part is used to evaluate the multiple actual surgical stages based on the multiple operation information subsets and the multiple surgical stage video sequences, or, the evaluation part is used to evaluate the multiple actual surgical stages based on the preoperative medical dataset, the multiple operation information subsets and the multiple surgical stage video sequences, or the evaluation part is used to evaluate the multiple actual surgical stages based on the postoperative medical dataset, the multiple operation information subsets and the multiple surgical stage video sequences.

11. The method according to claim 10, characterized in that The evaluation part includes a first sub-network and a second sub-network; The first sub-network is used to determine at least one surgical evaluation indicator for each of the multiple actual surgical stages; The second sub-network is used to determine a surgery improvement suggestion for at least one actual surgery stage among the multiple actual surgery stages.

12. The method according to claim 9, characterized in that Evaluation of the surgery also includes: Based on the at least one surgical evaluation indicator of each actual surgical stage, the overall difficulty and / or quality of the surgery is determined.

13. The method according to claim 12, characterized in that The method further comprises: The charging standard level and / or care level of the surgery is determined based on the overall difficulty and / or quality of the surgery.

14. The method according to any one of claims 1 to 8, characterized in that The operation information set includes one or more of the motion trajectory data of the main manipulator handle, the motion trajectory data of the surgical instrument end, the operation action type, the type of surgical instrument used, and the number of surgical instruments used.

15. The method according to claim 14, characterized in that The operation action type includes at least one of translation, rotation, clamping and a combination of at least two actions.

16. The method according to claim 14, characterized in that The method further comprises: obtaining multiple joint information of multiple joints of the main manipulator; and Based on the multiple joint information, the motion trajectory data of the main manipulator handle and / or the operation action type are determined.

17. The method according to claim 14, characterized in that The method further comprises: Based on the driving information of at least one surgical instrument or the posture sensor data of at least one surgical instrument or the surgical video data of the operation, the motion trajectory data of the surgical instrument end corresponding to the at least one surgical instrument is determined.

18. The method according to claim 14, characterized in that The method further comprises: Based on the motion trajectory data of the main manipulator handle and / or the motion trajectory data of the surgical instrument end, obtaining time domain features and / or frequency domain features corresponding to the motion trajectory; and The surgery is evaluated based on the time domain features and / or the frequency domain features.

19. The method according to claim 18, characterized in that Evaluating the surgery based on the time domain features and / or the frequency domain features comprises: Based on the time domain features and / or the frequency domain features, evaluating the execution smoothness of the surgery; and Based on the performance fluency, the surgery is evaluated.

20. The method according to claim 5 or 6, characterized in that: The preoperative medical data set includes at least a portion of lesion images, pathological grading, and disease history.

21. The method according to claim 7 or 8, characterized in that The postoperative medical data set includes at least a portion of incision recovery conditions, complication conditions, and medication conditions.

22. The method according to any one of claims 1 to 8, characterized in that The multiple actual surgical stages include one or more of the pneumoperitoneum establishment stage, surgical instrument placement stage, surface fat clearing stage, tissue mobilization stage, blood vessel separation stage, lesion resection stage, and tissue suturing stage.

23. The method according to any one of claims 1 to 8, characterized in that The plurality of actual operation phases include additional actual operation phases that are lacking in the standard operation procedure, and the additional actual operation phases include an emergency operation phase and / or a remedial operation phase; The method comprises: Dividing the operation information set and / or surgical video data to obtain operation information subsets and / or surgical stage video sequences corresponding to the emergency operation stage and / or remedial operation stage; and The emergency operation phase and / or the remedial operation phase are evaluated based on the operation information subset corresponding to the emergency operation phase and / or the remedial operation phase and / or the operation phase video sequence.

24. A surgical robot system, characterized in that: include: At least one master console cart; At least one slave trolley, the slave trolley comprising at least one mechanical arm, the mechanical arm being used to carry surgical instruments, the master trolley being communicatively connected with the slave trolley; A control device configured to execute the method according to any one of claims 1 to 23.

25. A computer-readable storage medium, characterized in that: The storage medium is used to store at least one instruction, and the at least one instruction is executed by a computer to configure the computer to perform the method according to any one of claims 1 to 23.

26. A computer system, characterized in that: include: A memory, configured to store at least one instruction; as well as A processor, coupled to the memory and configured to execute the at least one instruction to perform the method according to any one of claims 1-23.

Images