Surgery assistance system and surgery assistance method

Abstract

The present invention provides a surgical assistance system and a surgical assistance method, wherein the surgical assistance system comprises: an image large model module, a visual large model module, a risk detection large model module and a surgeon dialogue language large model module; the image large model module segments and reconstructs the preoperative image based on the image large model, and outputs the segmentation result information of the preoperative image; the visual large model module analyzes the intraoperative endoscopic scene content based on the visual large model, and outputs the endoscopic content information; the risk detection large model module analyzes and judges based on the segmentation result information and the endoscopic content information, and outputs risk warning information; the surgeon dialogue language large model module outputs suggestion information based on the segmentation result information, the endoscopic content information and the risk warning information. With such a configuration, a comprehensive evaluation can be made and suggestions can be given to the surgeon to more intuitively assist the surgeon in reducing intraoperative risks.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a surgery assisting system and a surgery assisting method. Background Art

[0002] When using surgical robot systems for endoscopic surgery, the surgeon usually controls the robot at the doctor's console by observing the real-time image fed back by the endoscope. The surgical robot system generally only provides reminders based on process parameter limits and alarm feedback when the motor load is too large, without any other further prompts and interactive functions. This leads to the following two deficiencies:

[0003] 1. Highly dependent on the surgeon: The surgical robot system is completely dependent on the surgeon's understanding of the current scenario and information (such as the patient's medical history, etc.). The surgical robot system cannot provide reminders of abnormal situations and danger predictions during the operation. The surgical process is completely dependent on the surgeon's surgical operation ability and human eye recognition ability. Once the surgeon's hand-eye coordination is poor or fatigue occurs due to long-term operation, the surgical risk will increase.

[0004] Second, high labor costs: The surgeon needs to manually operate the whole operation at the doctor's console, and other work requires the assistance of multiple medical staff. The entire surgical operation process is completely dependent on manpower, which requires a lot of manpower for the surgical robot system, increasing the manpower burden. On the other hand, the postoperative video data is very large, and postoperative analysis also requires a lot of manpower and energy. Summary of the invention

[0005] The purpose of the present invention is to provide a surgical assistance system and a surgical assistance method to solve the problem that the existing surgical robot system relies on the operator's experience.

[0006] In order to solve the above technical problems, the present invention provides a surgical assistance system, which includes: an image large model module, a visual large model module, a risk detection large model module and a surgeon dialogue language large model module;

[0007] The image large model module segments and reconstructs the preoperative image based on the image large model, and outputs the segmentation result information of the preoperative image;

[0008] The visual big model module analyzes the endoscope scene content during the operation based on the visual big model and outputs the endoscope content information;

[0009] The risk detection large model module performs analysis and judgment based on the segmentation result information and the endoscope content information, and outputs risk warning information;

[0010] The operator dialogue language large model module outputs suggestion information based on the segmentation result information, the endoscope content information and the risk warning information.

[0011] Optionally, the operator dialogue language big model module includes an interaction unit, and the operator dialogue language big model module is also configured to obtain input information through the interaction unit; at least one of the visual big model module, the risk detection big model module and the operator dialogue language big model module corrects its own output information based on the input information.

[0012] Optionally, the operator dialogue language large model module also includes a semantic analysis unit; the semantic analysis unit is used to analyze the input information obtained by the interaction unit and correct the input information.

[0013] Optionally, the surgery assistance system further includes a data storage module; the data storage module is configured to store the segmentation result information, the recommendation information and the endoscope content information.

[0014] Optionally, the segmentation result information is image data, the recommendation information and the endoscopic content information are text data, and the data storage module is configured to combine the segmentation result information, the recommendation information and the endoscopic content information with a timestamp to form a multimodal data structure for storage.

[0015] Optionally, the surgery assistance system further includes a postoperative evaluation module; the postoperative evaluation module evaluates the surgery based on the multimodal data structure stored in the data storage module, and outputs an evaluation result.

[0016] Optionally, the postoperative evaluation module includes a splitting unit and an intraoperative task analysis unit, the splitting unit is used to split the multimodal data structure obtained from the data storage module, and send the split data to the intraoperative task analysis unit; the intraoperative task analysis unit evaluates the operation based on the data split by the splitting unit, and outputs the evaluation result.

[0017] Optionally, the image large model module is configured to segment and reconstruct the preoperative image through a first convolutional neural network; the input information of the first convolutional neural network includes the preoperative image and the medical record information processed by the word segmenter;

[0018] The visual large model module is configured to analyze the intraoperative endoscopic scene content through a second convolutional neural network; the input information of the second convolutional neural network includes the intraoperative endoscopic image and the input information processed by the word segmenter;

[0019] The risk detection large model module is configured to perform analysis and judgment through a third convolutional neural network; the input information of the third convolutional neural network includes the segmentation result information and the endoscope content information processed by the word segmenter;

[0020] The operator dialogue language large model module is configured to perform analysis and judgment through a fourth convolutional neural network; the input information of the fourth convolutional neural network includes the segmentation result information, the endoscope content information processed by the word segmenter, and the risk warning information processed by the word segmenter.

[0021] Optionally, the surgical assistance system further includes an endoscope calibration module; the endoscope calibration module is configured to perform the following steps to calibrate the endoscope before surgery:

[0022] photographing a three-dimensional calibration object to obtain a first image;

[0023] Performing rough calibration on the endoscope according to the first image and known size information of the three-dimensional calibration object;

[0024] Based on the rough calibration results, the calibration error is calculated and the position and angle of the area to be optimized are output;

[0025] Arranging a two-dimensional calibration object according to the position and angle of the area to be optimized, and photographing the two-dimensional calibration object to obtain a second image;

[0026] The endoscope is precisely calibrated according to the second image and the known size information of the two-dimensional calibration object in combination with the feature information of the three-dimensional calibration object in the rough calibration step.

[0027] Optionally, the steps of roughly calibrating the endoscope include:

[0028] Extracting first feature points in the first image, and obtaining two-dimensional image coordinate information corresponding to each of the first feature points in the first image;

[0029] Acquire world coordinate information of each of the first feature points according to the size information of the three-dimensional calibration object;

[0030] The two-dimensional image coordinate information and the world coordinate information of each of the first feature points are matched, a set of equations including the endoscope parameters to be solved is constructed, and the set of equations is solved to obtain a rough calibration result.

[0031] Optionally, the feature information of the three-dimensional calibration object in the rough calibration step includes the two-dimensional image coordinate information and world coordinate information of the first feature point; and the step of finely calibrating the endoscope includes:

[0032] Extracting second feature points in the second image, identifying a central feature point and a two-dimensional calibration object coordinate system, and obtaining world coordinate information and two-dimensional image coordinate information of each second feature point based on size information of the two-dimensional calibration object;

[0033] Combine the two-dimensional image coordinate information and world coordinate information of the first feature point, and the world coordinate information and two-dimensional image coordinate information of the second feature point, and jointly construct a group of equations containing the endoscope parameters to be solved, solve the group of equations, and obtain a precise calibration result.

[0034] Optionally, the step of calculating the calibration error based on the rough calibration result includes:

[0035] Based on the roughly calibrated endoscope parameters, mapping the world coordinate information of the first feature point calculated in the rough calibration step to a two-dimensional image coordinate system to obtain virtual two-dimensional image coordinate information of the first feature point;

[0036] The Euclidean distance between the virtual two-dimensional image coordinate information and the two-dimensional image coordinate information of the first feature point in the first image is calculated, and the Euclidean distance is determined as a calibration error.

[0037] Optionally, the step of outputting the position and angle of the area to be optimized includes:

[0038] Corresponding to the planes of multiple different tilt directions of the three-dimensional calibration object, the calibration errors of different segmented areas in the first image are calculated respectively, and according to a preset threshold, the segmented area whose calibration error exceeds the threshold is selected as the area to be optimized, and the tilt angle of the corresponding plane of the three-dimensional calibration object is determined as the angle of the area to be optimized.

[0039] Optionally, the surgical auxiliary system further includes: an endoscope fixing device, a calibration object fixing device, an object distance adjustment device, a three-dimensional calibration object, and a two-dimensional calibration object;

[0040] The endoscope fixture is used for installing the endoscope; the relative positions of the endoscope fixture and the calibration object fixture are adjustably set by the object distance adjustment device; the three-dimensional calibration object and the two-dimensional calibration object are configured to be detachably mounted on the calibration object fixture.

[0041] Optionally, the three-dimensional calibration object includes a plurality of repeated structures, each of which includes a plurality of planes with different tilt directions; and / or, the two-dimensional calibration object includes a plane, and the plane includes a plurality of first characteristic symbols and a plurality of second characteristic symbols.

[0042] Optionally, the endoscope calibration system further includes a display device; the display device is used to display:

[0043] At least one of the endoscopic image during the operation, the recommendation information, the risk warning information, the image taken by the endoscope, the segmented area of ​​the plane corresponding to different tilt directions of the three-dimensional calibration object, and the area to be optimized.

[0044] In order to solve the above technical problems, the present invention further provides a surgical assistance method, which comprises:

[0045] Based on the large image model, segment and reconstruct the preoperative image, and output the segmentation result information of the preoperative image;

[0046] Based on the visual big model, analyze the endoscope scene content during surgery and output the endoscope content information;

[0047] Based on the segmentation result information and the endoscope content information, analyze and judge, and output risk warning information;

[0048] Outputting suggestion information according to the segmentation result information, the endoscope content information and the risk warning information.

[0049] In order to solve the above technical problems, the present invention also provides a surgical robot system, which includes the surgical assistance system as described above.

[0050] To summarize, in the surgical assistance system and surgical assistance method provided by the present invention, the surgical assistance system includes: an image big model module, a visual big model module, a risk detection big model module and a surgeon dialogue language big model module; the image big model module segments and reconstructs the preoperative image based on the image big model, and outputs the segmentation result information of the preoperative image; the visual big model module analyzes the intraoperative endoscopic scene content based on the visual big model, and outputs the endoscopic content information; the risk detection big model module performs analysis and judgment based on the segmentation result information and the endoscopic content information, and outputs risk warning information; the surgeon dialogue language big model module outputs suggestion information based on the segmentation result information, the endoscopic content information and the risk warning information.

[0051] With such configuration, based on the setting of the image large model module, preoperative images can be loaded, segmented and reconstructed, and potential surgical risks can be analyzed in advance. Based on the setting of the visual large model module, real-time images of the endoscope can be analyzed. Based on the setting of the risk detection large model module, the segmentation results of the preoperative images and the real-time images of the endoscope can be integrated, analyzed and judged, and risk warning information can be output. Based on the setting of the operator dialogue language large model module, a comprehensive evaluation can be made based on the segmentation result information, endoscope content information, and risk warning information, and suggestions can be given to the operator to more intuitively assist the operator in reducing intraoperative risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Those skilled in the art will appreciate that the drawings are provided for a better understanding of the present invention, but do not constitute any limitation on the scope of the present invention.

[0053] Figure 1 Schematic diagram of functional modules of a surgical assistance system according to an embodiment of the present invention.

[0054] Figure 2 It is a functional module schematic diagram of the image large model module of an embodiment of the present invention.

[0055] Figure 3 It is a functional module schematic diagram of the visual large model module of an embodiment of the present invention.

[0056] Figure 4 It is a functional module diagram of the risk detection large model module in an embodiment of the present invention.

[0057] Figure 5 It is a functional module diagram of the operator dialogue language large model module of an embodiment of the present invention.

[0058] Figure 6 It is a functional schematic diagram of a word segmenter according to an embodiment of the present invention.

[0059] Figure 7 It is a schematic diagram of the content displayed on the display screen of the embodiment of the present invention.

[0060] Figure 8 It is a functional schematic diagram of a data storage module according to an embodiment of the present invention.

[0061] Fig. 9 Schematic diagram of two storage forms of the data storage module according to an embodiment of the present invention.

[0062] Fig.10 Schematic diagram of the functional modules of the data storage module according to an embodiment of the present invention.

[0063] Fig.11 Schematic diagram of the functions of the postoperative evaluation module according to an embodiment of the present invention.

[0064] Fig.12 Schematic diagram of the functional modules of the postoperative evaluation module according to an embodiment of the present invention.

[0065] Fig.13 Schematic diagram of two evaluation forms of the postoperative evaluation module according to an embodiment of the present invention.

[0066] Fig.14 is a schematic diagram of a display module according to an embodiment of the present invention.

[0067] Fig.15 It is a schematic diagram of the calibration process of the endoscope calibration module according to an embodiment of the present invention.

[0068] Fig.16 It is a stereogram of a three-dimensional calibration object according to an embodiment of the present invention.

[0069] Fig.17 4 is a top view of a three-dimensional calibration object according to an embodiment of the present invention.

[0070] Fig.18 4 is a side view of a three-dimensional calibration object according to an embodiment of the present invention.

[0071] Fig.19 4 is a flow chart of the rough calibration step of an embodiment of the present invention.

[0072] Fig. 20 is a schematic diagram of a two-dimensional calibration object according to an embodiment of the present invention.

[0073] Fig.21 2 is a schematic diagram of feature point extraction according to an embodiment of the present invention.

[0074] Fig. 22 It is a flowchart of the precise calibration step of an embodiment of the present invention.

[0075] Fig.23a and Figure 23b It is a schematic diagram of solving endoscope parameters according to an embodiment of the present invention.

[0076] Fig.24 It is a schematic diagram of calibration accuracy calculation according to an embodiment of the present invention.

[0077] Fig.25 Schematic diagram of a display and displayed content according to an embodiment of the present invention.

[0078] Fig.26 Schematic diagram of an endoscope calibration component according to an embodiment of the present invention.

[0079] Fig.27a and Figure 27b is a schematic diagram of an AR display device and displayed content according to an embodiment of the present invention.

[0080] In the attached figure: 10-image model module; 20-visual model module; 30-risk detection model module; 40-operator dialogue language model module; 41-speech and text recognition column; 50-data storage module; 60-postoperative evaluation module; 61-splitting unit; 62-intraoperative task analysis unit; 70-display module; 71-display screen; 72-green warning light; 73-red warning light; 74-display; 75-AR display device; 80-substrate; 81-repeating structure structure; 82-slope; 83-top surface; 84-plane; 85-first characteristic symbol; 86-second characteristic symbol; 91-endoscope fixing device; 92-calibration object fixing device; 93-object distance adjustment device; 94-three-dimensional calibration object; 95-two-dimensional calibration object; 96-calibration calculation unit; 901-endoscope; 902-image; 903-segmentation network; 904-circle; 905-triangle; 906-segmentation area; 907-area to be optimized; 908-position to be placed. DETAILED DESCRIPTION

[0081] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the drawings is often a part of the actual structure. In particular, the emphasis of each drawing is different, and sometimes different scales are used.

[0082] The purpose of the present invention is to provide a surgical assistance system and a surgical assistance method to solve the problem that the existing surgical robot system relies on the surgeon's experience.

[0083] Please refer to Figure 1 An embodiment of the present invention provides a surgical assistance system, which includes: an image big model module 10, a visual big model module 20, a risk detection big model module 30 and a surgeon dialogue language big model module 40; the image big model module 10 segments and reconstructs the preoperative image based on the image big model, and outputs the segmentation result information of the preoperative image; the visual big model module 20 analyzes the intraoperative endoscopic scene content based on the visual big model, and outputs the endoscopic content information; the risk detection big model module 30 performs analysis and judgment based on the segmentation result information and the endoscopic content information, and outputs risk warning information; the surgeon dialogue language big model module 40 outputs suggestion information based on the segmentation result information, the endoscopic content information and the risk warning information.

[0084] The imaging model module 10 (ImagingLM) can segment the acquired preoperative images (including CT images or MRI images, etc.) based on the imaging model. The segmentation result information may include image data and feature information, such as some identified risks (such as anatomical structure variation, duct variation, lymph nodes with metastatic lesions, etc.). Please refer to Figure 2 , which exemplarily shows a functional module diagram of the imaging large model module 10 (ImagingLM). Optionally, the imaging large model module 10 is configured to segment and reconstruct the preoperative image through a first convolutional neural network (CNN1); the input information of the first convolutional neural network (CNN1) includes the preoperative image and the medical record information processed by a tokenizer.

[0085] The visual model module 20 (VisualLM) can analyze the acquired intraoperative endoscopic scene content (including the real-time image of the intraoperative endoscope, etc.) based on the visual model. The output endoscopic content information may include textual description information obtained by analyzing the current endoscopic image, etc. Please refer to Figure 3 , which exemplarily shows a functional module diagram of the visual large model module 20 (VisualLM). The visual large model module 20 is configured to analyze the intraoperative endoscopic scene content through a second convolutional neural network (CNN2); the input information of the second convolutional neural network (CNN2) includes the intraoperative endoscopic image and the input information processed by a tokenizer; the input information can be obtained based on the interaction unit of the operator dialogue language large model module 40 (see the following description for details).

[0086] The risk detection model module 30 (RiskDetectorLM) can obtain the segmentation result information of the preoperative image output by the image model module 10 and the endoscopic content information analyzed by the visual model module 20, and then perform fusion analysis and judgment based on the segmentation result information and the endoscopic content information. The risk warning information output by it can be textual description information. Please refer to Figure 4 , which exemplarily shows a functional module diagram of the risk detection large model module 30 (RiskDetectorLM). The risk detection large model module 30 is configured to perform analysis and judgment through a third convolutional neural network (CNN3); the input information of the third convolutional neural network (CNN3) includes the segmentation result information and the endoscope content information processed by a tokenizer. Here, the input information of the third convolutional neural network (CNN3) includes a set of image data and a set of text data, the image data comes from the segmentation result information output by the image large model module 10; the text data comes from the endoscope content information output by the visual large model module 20.

[0087] The surgeon conversation language large model module 40 (SurgeonChatLLM) can analyze and summarize the information output by the image large model module 10, the visual large model module 20 and the risk detection large model module 30 based on the language large model, preferably analyze and comprehensively evaluate based on the prompts of the current surgical plan, and give corresponding suggestion information (such as reminders of dangerous operations, vascular variation reminders, unexpected bleeding reminders, etc.). The suggestion information it outputs may include textual descriptive information, image information or sound reminder information, etc., and may also include applying control arm resistance to the doctor's console, etc. The surgeon can proceed to the next step based on the suggestion information given by the surgeon conversation language large model module 40. It can be understood that since the suggestion information is obtained based on a comprehensive evaluation of information such as preoperative images and current endoscopic images, it can more intuitively assist the surgeon in reducing intraoperative risks. Please refer to Figure 5 , which exemplarily shows the functional module diagram of the surgeon conversation language large model module 40 (SurgeonChatLLM). The surgeon conversation language large model module 40 is configured to perform analysis and judgment through a fourth convolutional neural network (CNN4); the input information of the fourth convolutional neural network (CNN4) includes the segmentation result information, the endoscope content information processed by a tokenizer, and the risk warning information processed by a tokenizer. Here, the input information of the fourth convolutional neural network (CNN4) includes a set of image data and two sets of text data. The image data comes from the segmentation result information output by the image large model module 10; the text data comes from the endoscope content information output by the visual large model module 20 and the risk warning information output by the risk detection large model module 30.

[0088] Please refer to Figure 6 It should be noted that the role of the tokenizer is to vectorize the input text so that it can be used as a feature input to each convolutional neural network. In addition, large models such as the image large model, the visual large model, and the language large model and each convolutional neural network can be pre-trained, and the training process can refer to the existing technology, which will not be described in detail here.

[0089] The surgical assistance system provided in this embodiment uses a combination of preoperative images and intraoperative large models (including image large models, visual large models, language large models and other large models) to provide real-time operational suggestions (suggestion information) during surgery without the need to label image data, or use target detection and predefined operational instructions. The process of generating suggestion information is more autonomous, and has strong generalization capabilities, especially in non-predetermined and unknown scenarios.

[0090] Optionally, in some embodiments, the operator dialogue language big model module 40 includes an interaction unit, and the operator dialogue language big model module 40 is also configured to obtain input information through the interaction unit; at least one of the visual big model module 20, the risk detection big model module 30 and the operator dialogue language big model module 40 corrects its own output information based on the input information.

[0091] The setting of the interactive unit can be used to receive input information from the operator. It should be noted that the input information here includes but is not limited to text input information, voice input information, touch and click information and other common input information in this field. Please refer to Figure 7 , which shows the content displayed on the display screen 71 (see the following description for details), including an example of an interactive unit, such as a voice and text recognition column 41. The operator can input information in the form of text or voice.

[0092] Furthermore, the operator dialogue language large model module also includes a semantic analysis unit; the semantic analysis unit is used to analyze the input information obtained by the interaction unit and correct the input information. The semantic analysis unit can analyze the input information from the vocabulary level and the sentence level, and correct the input information in combination with the output information of the image large model module 10, the visual large model module 20, the risk detection large model module 30 and other modules, so as to further improve the accuracy and reliability of the input information.

[0093] On the one hand, if the surgeon finds that the advice information given by the surgeon dialogue language large model module 40 is wrong, the surgeon dialogue language large model module 40 can provide feedback to the surgeon dialogue language large model module 40 through the interactive unit, and at least one of the visual large model module 20, the risk detection large model module 30 and the surgeon dialogue language large model module 40 can modify their respective output information according to the surgeon's feedback to correct the error. In an alternative exemplary embodiment, the surgeon dialogue language large model module 40 can analyze specific tasks based on the input information and hand them over to the visual large model module 20 for execution (for example, requiring the visual large model module 20 to focus on analyzing the organ being cut to prepare for the next step of making suggestions). This can achieve dynamic highlighting of soft tissue separation areas, highlighted suture areas, highlighted dangerous blood vessel areas, etc. at different stages of the operation.

[0094] On the other hand, please refer to Figure 3The intraoperative endoscopic scene content acquired by the visual large model module 20 is not limited to only real-time endoscopic images, but can also include feedback and judgment information made by the operator based on the current endoscopic images. These feedback and judgment information can be fed back to the operator dialogue language large model module 40 through the interactive unit as input information, and then transmitted to the visual large model module 20 by the operator dialogue language large model module 40. Such a configuration can further improve the accuracy of the endoscopic content information analyzed by the visual large model module 20.

[0095] Please continue to refer to Figure 1 and Figure 8 Optionally, the surgical assistance system further includes a data storage module 50 (LMStorage); the data storage module 50 is configured to store the segmentation result information, the suggestion information and the endoscope content information. It can be understood that since the segmentation result information, the suggestion information and the endoscope content information contain both images and texts, the data stored in the data storage module 50 is actually a multimodal data. Please refer to Fig. 9 The data storage module 50 can store data in various forms. Fig. 9 Two storage forms of the data storage module 50 are shown, one is a local NAS server mode, and the other is a cloud server storage mode based on an encrypted channel. Of course, those skilled in the art can also configure other storage forms for the data storage module 50 according to the prior art.

[0096] For further information, please refer to Fig.10 , the segmentation result information is image data, the suggestion information and the endoscope content information are text data, and the data storage module 50 is configured to combine the segmentation result information, the suggestion information and the endoscope content information with a timestamp to form a multimodal data structure for storage. The data storage module 50 integrates data of multiple modalities into a multimodal data structure for easy storage in a storage medium.

[0097] Optional, please refer to Figure 1 The surgical assistance system further includes a postoperative evaluation module 60 (SurgeryEVA); the postoperative evaluation module 60 evaluates the surgery based on the multimodal data structure stored in the data storage module 50 and outputs the evaluation result. Based on the setting of the data storage module 50, the surgical process can be analyzed and evaluated according to the multimodal data structure stored in the data storage module 50 after the surgery, for example, the analysis result of the surgical procedure (such as the distribution of instrument movements, the time consumption of specific resection movements, etc.) is given.

[0098] Please refer to Fig.11The postoperative evaluation module 60 can perform predefined evaluations based on the multimodal data structure. The predefined evaluations include but are not limited to the duration of the operation, the proficiency of the movements, the sequence of the surgical steps, etc.

[0099] Please refer to Fig.12 The postoperative evaluation module 60 includes a splitting unit 61 and an intraoperative task analysis unit 62. The splitting unit 61 is used to split the multimodal data structure obtained from the data storage module 50, and send the split data to the intraoperative task analysis unit 62; the intraoperative task analysis unit 62 evaluates the operation based on the data split by the splitting unit 61, and outputs the evaluation result. Since the multimodal data structure is obtained from the data storage module 50, it needs to be split by the splitting unit 61 to obtain three categories of data: image data, text data, and timestamp. Then, the three categories of data obtained by splitting are input into the intraoperative task analysis unit 62 for analysis and evaluation.

[0100] The postoperative evaluation module 60 can present the evaluation results in various forms (such as comparison charts, evaluation scores, etc.). Fig.13 There are also many ways to implement the postoperative assessment module 60. Fig.13 Two evaluation forms of the postoperative evaluation module 60 are shown, one of which is manual evaluation, for example, a web page access portal can be provided, and a time-division scoring function can be provided on the page to facilitate peer evaluation. The other is automatic evaluation, for example, automatically providing comparative analysis results with sample procedures (such as expert procedures). Of course, those skilled in the art can also configure other evaluation forms for the postoperative evaluation module 60 according to the prior art.

[0101] In this embodiment, the visual large model module 20 (VisualLM) needs to analyze the content of the intraoperative endoscopic scene, and the intraoperative endoscopic scene content is based on the endoscope photography. In order to improve the accuracy, it is generally necessary to calibrate the endoscope before the operation. In the field of medical endoscopes, the most widely used endoscope internal parameter calibration method is the Zhang Zhengyou chessboard calibration method. This method takes multiple chessboard pictures at different angles at different positions in the endoscope field of view, obtains the image coordinates of the corner points of the chessboard from the pictures, and sets a corner point of the chessboard as the origin of the world coordinate system. Since the size of the chessboard is known, the world coordinate system of the corner point can be calculated, and then the equation is constructed based on the endoscope imaging model to solve the endoscope internal parameters. However, on the one hand, the calibration process of Zhang Zhengyou's chessboard calibration method is complicated and slow, and at least 15-20 images need to be taken. There are certain requirements for the endoscope shooting angle and the placement of the calibration plate. When shooting, the chessboard should cover the entire image plane as much as possible and include various angles and distances. On the other hand, Zhang Zhengyou's chessboard calibration method also has the problems of unstable calibration accuracy and strong subjectivity. Since photos need to be taken at different positions and angles, the calibration accuracy is affected by the operator's proficiency, and the calibration accuracy of different operators may vary greatly.

[0102] In order to solve the problems of complex, slow and unstable calibration process of existing endoscopes, please refer to Fig.15 In an embodiment of the present invention, the surgical assistance system further includes an endoscope calibration module; the endoscope calibration module is configured to perform the following steps to calibrate the endoscope before surgery:

[0103] Step S1: photographing a three-dimensional calibration object to obtain a first image;

[0104] Step S2: performing rough calibration on the endoscope according to the first image and known size information of the three-dimensional calibration object;

[0105] Step S3: Based on the rough calibration result, the calibration error is calculated and the position and angle of the area to be optimized are output;

[0106] Step S4: arranging a two-dimensional calibration object according to the position and angle of the area to be optimized, and photographing the two-dimensional calibration object to obtain a second image;

[0107] Step S5: Based on the second image and the known size information of the two-dimensional calibration object, combined with the feature information of the three-dimensional calibration object in the rough calibration step, the endoscope is precisely calibrated.

[0108] With this configuration, on the one hand, only a small number of images need to be taken to solve the internal parameters of the endoscope, which is a simple process and fast calibration. On the other hand, the endoscope is roughly calibrated using the first image of the three-dimensional calibration object, and then the two-dimensional calibration object is used to perform fine calibration on the area to be optimized at a specific position and angle, effectively improving the overall calibration accuracy and the robustness of the calibration results.

[0109] The three-dimensional calibration object preferably has several planes with different inclination directions, and each plane preferably has multiple easily extractable feature marks. In this way, the endoscope can perform calibration based on several planes with known inclinations on the three-dimensional calibration object in combination with the feature marks on the plane by photographing the three-dimensional calibration object.

[0110] Please refer to Figures 16 to 18 Preferably, the three-dimensional calibration object comprises a plurality of repeated structures 81, and each repeated structure 81 comprises a plurality of planes with different tilt directions. Figures 16 to 18 The three-dimensional calibration object shown includes a substrate 80 and four repeating structures 81. Each repeating structure 81 is in the shape of a quadrangular pyramid, that is, each repeating structure 81 includes four slopes 82 with different inclination directions and a top surface 83 parallel to the substrate 80. Further, each plane (including four slopes 82 and a top surface 83) has a plurality of easy-to-extract feature marks, such as graphic symbols such as dots drawn on each plane. Further, the size information of the three-dimensional calibration object is known, that is, the size of each plane of each repeating structure 81, the inclination direction and inclination angle of each slope 82, and other information are known.

[0111] In step S1, the distance between the endoscope and the three-dimensional calibration object is adjusted according to the focal length, depth of field and field of view of the endoscope to be calibrated, and then the three-dimensional calibration object can be photographed by the endoscope to obtain a first image.

[0112] In step S2, since the size information of the three-dimensional calibration object is known, the endoscope can be roughly calibrated according to the first image and the known size information of the three-dimensional calibration object.

[0113] Please refer to Fig.19 , which shows an exemplary process of roughly calibrating an endoscope. The steps of roughly calibrating an endoscope include:

[0114] Step S21: extracting first feature points in the first image, and obtaining two-dimensional image coordinate information corresponding to each first feature point in the first image;

[0115] Step S22: obtaining world coordinate information of each first feature point according to the size information of the three-dimensional calibration object;

[0116] Step S23: Match the two-dimensional image coordinate information and the world coordinate information of each first feature point, construct an equation group containing the endoscope parameters to be solved, solve the equation group, and obtain a rough calibration result.

[0117] Since each plane of the three-dimensional calibration object has multiple feature identifiers that are easy to extract, in step S21, for example, the first feature point in the first image, such as the center of a circle, can be extracted by a feature extraction algorithm, and then the two-dimensional image coordinate information corresponding to the first feature point can be obtained, that is, the pixel coordinate information of the first feature point in the first image.

[0118] In step S22, the multiple first feature points extracted in step S21 may be sorted and numbered according to their positional relationship in the first image, and the three-dimensional world coordinate information of each first feature point may be obtained based on the known size information of the three-dimensional calibration object.

[0119] After the rough calibration is completed, step S3 can calculate the calibration errors of different regions of the first image, select the regions to be optimized that need further fine calibration, and output their positions and angles, and then perform fine calibration in step S4.

[0120] Please refer to Fig. 20 Preferably, the two-dimensional calibration object includes a plane 84, and the plane 84 includes a plurality of first characteristic symbols 85 and a plurality of second characteristic symbols 86. The plurality of first characteristic symbols 85 can be arranged in an array in a certain order, and the plurality of second characteristic symbols 86 can be arranged in a specific position, and the arrangement pattern formed by the second characteristic symbols 86 and the first characteristic symbols 85 does not overlap. Fig. 20 In the illustrated example, the first characteristic symbol 85 is a circle, which is arranged roughly along a five-row and five-column array, but some of the arrangement positions are replaced by the second characteristic symbol 86. The second characteristic symbol 86 is a triangle ( Fig. 20 The triangle shown in also includes an inscribed circle to facilitate feature extraction), and the second characteristic symbols 86 are preferably arranged at special positions such as corners and centers of the five-row and five-column array. Furthermore, the size information of the two-dimensional calibration object is known, that is, the size, quantity, center position and arrangement order of each first characteristic symbol 85, and the size, quantity, center position and arrangement order of each second characteristic symbol 86 are known. It can be understood that the circle and the triangle are only exemplary and not limiting of the first characteristic symbol 85 and the second characteristic symbol 86, and other shapes can also be selected as characteristic symbols in other embodiments. The number, size, arrangement order, etc. of the characteristic symbols can also be selected and configured according to actual conditions, and are not limited to Fig. 20 shown.

[0121] Please refer to Fig.21, which exemplarily illustrates an embodiment of the feature extraction algorithm. In this embodiment, a neural network algorithm can be used to train the segmentation model. During reasoning, an image 902 taken by an endoscope is input, and the segmentation network 903 outputs a plurality of mask images of feature points, such as a circle 904 and a triangle 905. Then, the precise two-dimensional image coordinates of the feature points are obtained by calculating the center of the circle and the center of the triangle. Of course Fig.21 The illustrated example is only an example of a feature extraction algorithm and is not intended to limit the feature extraction algorithm. Those skilled in the art may adopt other feature extraction algorithms based on the prior art.

[0122] In step S4, arranging the two-dimensional calibration object according to the position and angle of the area to be optimized means arranging the plane 84 of the two-dimensional calibration object according to the position and angle of the area to be optimized output in step S3, so that the plane 84 of the two-dimensional calibration object is parallel to or coincides with the position and angle of the area to be optimized. Since the position and angle of the area to be optimized can be understood as being selected based on a certain plane of the three-dimensional calibration object, it can be understood that the two-dimensional calibration object is used to replace a certain plane of the original three-dimensional calibration object.

[0123] In step S5, since the size information of the two-dimensional calibration object is known, the endoscope can be precisely calibrated based on the second image and the known size information of the two-dimensional calibration object, combined with the feature information of the three-dimensional calibration object in the rough calibration step. Here, the feature information of the three-dimensional calibration object in the rough calibration step includes the two-dimensional image coordinate information and the world coordinate information of the first feature point.

[0124] Please refer to Fig. 22 , which shows an exemplary process of fine calibration of an endoscope. The steps of fine calibration of an endoscope include:

[0125] Step S51: extracting the second feature points in the second image, identifying the central feature point and the two-dimensional calibration object coordinate system, and obtaining the world coordinate information and the two-dimensional image coordinate information of each second feature point according to the size information of the two-dimensional calibration object;

[0126] Step S52: Combine the two-dimensional image coordinate information and world coordinate information of the first feature point, and the world coordinate information and two-dimensional image coordinate information of the second feature point, jointly construct a set of equations containing the endoscope parameters to be solved, solve the set of equations, and obtain a precise calibration result.

[0127] Since the plane 84 of the two-dimensional calibration object has a plurality of first feature symbols 85 and a plurality of second feature symbols 86 that are easy to extract, in step S51, for example, a feature extraction algorithm can be used to extract the second feature point in the second image, such as the center of the circle of the first feature symbol 85 and the center of the triangle of the second feature symbol 86, so that the center of the second image and the central feature point located at the center of the second image can be identified ( Fig. 20 The center of the triangle at the center of the image is obtained by identifying the coordinate system of the two-dimensional calibration object. Then, based on the known size information of the two-dimensional calibration object, the world coordinate information and the two-dimensional image coordinate information of each second feature point can be obtained.

[0128] In step S52, the two-dimensional image coordinate information and world coordinate information of the first feature point obtained in steps S21 and S22 are combined with the world coordinate information and two-dimensional image coordinate information of the second feature point obtained in step S51 to form a set of equations, and the precise calibration result can be obtained by solving them.

[0129] As can be understood by those skilled in the art, an endoscope is a camera that generally has external parameters and internal parameters. External parameters include rotation matrix R and translation vector T. External parameters may change at different camera positions or shooting times. Internal parameters are parameters that describe the internal properties of the camera, including internal parameter matrix M and distortion model F. disp Etc. The calibration of the endoscope can be understood as solving the internal parameters of the endoscope.

[0130] Combine the following Fig.23a and Figure 23b , the steps of implementing calibration in step S23 and step S52 by constructing and solving a set of equations are explained by way of example.

[0131] Fig.23a The figure shows a two-dimensional image coordinate system of an image captured by an endoscope, wherein o is the image origin, U and V represent two coordinate axes of the image, and the two-dimensional image coordinates of a feature point can be expressed as (ui, vi). Figure 23b The state of a calibration object in the world coordinate system is shown. In one embodiment, the central feature point of the calibration object (not limited to the central feature point, but also a certain feature point) can be set as the origin O of the world coordinate system, and the world coordinates of a certain feature point can be expressed as (Xi, Yi, Zi). By extracting multiple sets of matching feature points and obtaining the corresponding two-dimensional image coordinates and world coordinates, the following set of equations can be constructed:

[0132]

[0133] Among them, F disp is the distortion model of the endoscope, M is the internal parameter matrix of the endoscope, R is the rotation matrix, and T is the translation vector. By solving the above equations, F disp , M, R and T.

[0134] Optionally, in step S3, the step of calculating the calibration error based on the rough calibration result includes:

[0135] Step S31: based on the roughly calibrated endoscope parameters, mapping the world coordinate information of the first feature point calculated in the rough calibration step to the two-dimensional image coordinate system to obtain virtual two-dimensional image coordinate information of the first feature point;

[0136] Step S32: Calculate the Euclidean distance between the virtual two-dimensional image coordinate information and the two-dimensional image coordinate information of the first feature point in the first image, and determine the Euclidean distance as the calibration error.

[0137] Please refer to Fig.24 In an alternative example, reprojection error can be used as a quantitative method of calibration accuracy. Reprojection error refers to mapping feature points to a two-dimensional image coordinate system based on the calibrated endoscope parameters, and then calculating the distance between the virtual two-dimensional image coordinates of the mapped feature points and the real two-dimensional image coordinates. The reprojection error can be expressed by the following formula:

[0138] Err=Dist(P,T(P))

[0139] Where P represents the true two-dimensional image coordinates of the feature point, T() represents the conversion function from the world coordinate system to the two-dimensional image coordinate system, and Dist() represents the Euclidean distance. Fig.24 The arrows in the figure indicate the error direction and error magnitude of each feature point.

[0140] Furthermore, in step S3, the step of outputting the position and angle of the area to be optimized includes:

[0141] Step S33: corresponding to the planes of the three-dimensional calibration object with different tilt directions, respectively calculate the calibration errors of different segmented areas in the first image, and according to a preset threshold, select the segmented area whose calibration error exceeds the threshold as the area to be optimized, and determine the tilt angle of the corresponding plane of the three-dimensional calibration object as the angle of the area to be optimized.

[0142] Please refer to Fig.25 In an exemplary embodiment, taking a three-dimensional calibration object including four repeated structures 81, each of which includes five planes, the first image captured by the endoscope can be divided into 41 segmented regions 906. Fig.25 The segmentation is illustrated by dashed lines and solid lines. After the rough calibration, the calibration error of each segmented area can be calculated based on step S31 and step S32, wherein the segmented area 906 whose calibration error exceeds the threshold can be used as the area to be optimized 907. It can be understood that the number of areas to be optimized is not limited, and can be one, multiple or zero.

[0143] Optional, please refer to Fig.26The surgical auxiliary system further includes an endoscope calibration component, which includes: an endoscope fixing device 91, a calibration object fixing device 92, an object distance adjustment device 93, a three-dimensional calibration object 94, a two-dimensional calibration object 95 and a calibration calculation unit 96;

[0144] The relative positions of the endoscope fixture 91 and the calibration object fixture 92 are adjustably set by the object distance adjustment device 93; the three-dimensional calibration object 94 and the two-dimensional calibration object 95 are configured to be detachably installed on the calibration object fixture 92; the calibration calculation unit 96 is configured to calibrate the endoscope 901 installed on the endoscope fixture 91 according to the above endoscope calibration method.

[0145] In one embodiment, the endoscope fixture 91 is used for mounting the endoscope 901 to be calibrated. The calibration object fixture 92 is used for detachably mounting the three-dimensional calibration object 94 or the two-dimensional calibration object 95. Preferably, the calibration object fixture 92 has a movable joint with freely adjustable angle, so that the three-dimensional calibration object 94 or the two-dimensional calibration object 95 mounted thereon can freely adjust the angle relative to the endoscope 901.

[0146] The object distance adjustment device 93 includes an adjustment track and a driving motor. The endoscope fixture 91 and the calibration object fixture 92 are respectively movably mounted on the adjustment track. The relative distance between the endoscope fixture 91 and the calibration object fixture 92 can be adjusted by the driving motor.

[0147] The three-dimensional calibration object 94 and the two-dimensional calibration object 95 can refer to the previous description, which will not be repeated here.

[0148] The calibration calculation unit 96 is connected to the endoscope for communication, and can obtain images captured by the endoscope 901 , and is used to execute the above endoscope calibration method, calculate the parameters of the endoscope 901 , and thus calibrate the endoscope 901 .

[0149] Please refer to Fig.14 , Figure 25 to Figure 27b Optionally, the surgical assistance system further includes a display module 70, which is used to display the endoscopic image during the operation, the recommendation information and the risk warning information, the image captured by the endoscope, the segmented area of ​​the plane corresponding to different tilt directions of the three-dimensional calibration object, and at least one of the areas to be optimized. The display module 70 may include a display screen 71, a display 74 or an AR display device 75, and may also include a warning light, etc. The setting of the display module 70 can assist the operator more intuitively.

[0150] In an exemplary embodiment, the display module 70 may include a display screen 71, a green warning light 72, and a red warning light 73. When the operator dialogue language model module 40 outputs the suggestion information, the display screen 71 displays the suggestion information and the green warning light 72 is on. When the display screen 71 displays the risk warning information, the red warning light 73 is on. Fig.14 The display module 70 is shown as an example rather than a limitation of the display module 70. Those skilled in the art may configure the display module 70 according to actual conditions. Optionally, the display module 70 may also display the position 908 of the two-dimensional calibration object 95 to be placed, such as Fig.25 In another embodiment, the display module 70 includes a display 74, such as a liquid crystal display. The display range of the display 74 can be adapted to the complete field of view of the endoscope, and the exemplary image displayed is as follows: Fig.25 Please refer to Fig.27a In another embodiment, the display module 70 includes an AR display device 75, the content displayed by which is as follows Figure 27b As shown, it can map the position and angle of the area to be optimized to the AR display device 75 to guide the operator to place the two-dimensional calibration object. The dotted line in the figure indicates the position 908 where the two-dimensional calibration object 95 is to be placed.

[0151] In some embodiments, the endoscope fixture 91 and the calibration object fixture 92 can also be automatically moved by driving a motor according to the input position and angle, so that the placement and adjustment of the calibration object can be automated, further improving the calibration speed, which is suitable for batch automated calibration scenarios.

[0152] Based on the above-mentioned surgical assistance system, an embodiment of the present invention further provides a surgical assistance method, which includes:

[0153] Step SA: segmenting and reconstructing the preoperative image based on the large image model, and outputting the segmentation result information of the preoperative image;

[0154] Step SB: Analyze the intraoperative endoscopic scene content based on the visual big model and output endoscopic content information;

[0155] Step SC: performing analysis and judgment based on the segmentation result information and the endoscope content information, and outputting risk warning information;

[0156] Step SD: outputting suggestion information according to the segmentation result information, the endoscope content information and the risk warning information.

[0157] Preferably, the surgical assistance method provided in the embodiment of the present invention can also perform one or more combinations of the other steps of the surgical assistance system as described above. Please refer to the above text for details, which will not be repeated here. Optionally, the surgical assistance method provided in this embodiment can be compiled into a program and integrated into a readable storage medium, which can be attached to the surgical robot system, such as integrated into the control device of the surgical robot system, or can be independently set. Similarly, the surgical assistance system provided in this embodiment, its various modules can be attached to the surgical robot system, such as integrated into the control device of the surgical robot system, or can be independently set, and the present invention is not limited to this. Furthermore, the embodiment of the present invention also provides a surgical robot system, which includes the surgical assistance system as described above. The structure and principle of other components of the surgical robot system can refer to the prior art, and the present invention will not elaborate on them.

[0158] In summary, in the surgical assistance system and surgical assistance method provided by the present invention, the surgical assistance system includes: an image large model module, a visual large model module, a risk detection large model module and a surgeon dialogue language large model module; the image large model module segments and reconstructs the preoperative image based on the image large model, and outputs the segmentation result information of the preoperative image; the visual large model module analyzes the endoscope scene content during the operation based on the visual large model, and outputs the endoscope content information; the risk detection large model module analyzes and judges based on the segmentation result information and the endoscope content information, and outputs the risk warning information; the surgeon dialogue language large model module outputs the suggestion information based on the segmentation result information, the endoscope content information and the risk warning information. With such configuration, based on the setting of the image large model module, the preoperative image can be loaded, segmented and reconstructed, and the potential surgical risks can be analyzed in advance. Based on the setting of the visual large model module, the real-time image of the endoscope can be analyzed. Based on the setting of the risk detection large model module, the segmentation result of the preoperative image and the real-time image of the endoscope can be fused, and the analysis and judgment can be performed to output the risk warning information. Based on the setting of the large model module of the surgeon's dialogue language, a comprehensive assessment can be made based on the segmentation result information, endoscopic content information and risk warning information, and suggestions can be given to the surgeon to more intuitively assist the surgeon in reducing intraoperative risks.

[0159] It should be noted that the above embodiments can be combined with each other. The above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention. Any changes and modifications made by ordinary technicians in the field of the present invention based on the above disclosure are within the protection scope of the present invention.

Claims

1. A surgical assistance system, characterized in that: include: Image big model module, vision big model module, risk detection big model module and operator dialogue language big model module; The image large model module segments and reconstructs the preoperative image based on the image large model, and outputs the segmentation result information of the preoperative image; The visual big model module analyzes the endoscope scene content during the operation based on the visual big model and outputs the endoscope content information; The risk detection large model module performs analysis and judgment based on the segmentation result information and the endoscope content information, and outputs risk warning information; The operator dialogue language large model module outputs suggestion information according to the segmentation result information, the endoscope content information and the risk warning information; The image large model module is configured to segment and reconstruct the preoperative image through a first convolutional neural network; the input information of the first convolutional neural network includes the preoperative image and the medical record information processed by the word segmenter; The visual large model module is configured to analyze the intraoperative endoscopic scene content through a second convolutional neural network; the input information of the second convolutional neural network includes the intraoperative endoscopic image and the input information processed by the word segmenter; The risk detection large model module is configured to perform analysis and judgment through a third convolutional neural network; the input information of the third convolutional neural network includes the segmentation result information and the endoscope content information processed by the word segmenter; The operator dialogue language large model module is configured to perform analysis and judgment through a fourth convolutional neural network; the input information of the fourth convolutional neural network includes the segmentation result information, the endoscope content information processed by the word segmenter, and the risk warning information processed by the word segmenter.

2. The surgical assistance system according to claim 1, characterized in that: The operator dialogue language big model module includes an interaction unit, and the operator dialogue language big model module is also configured to obtain input information through the interaction unit; at least one of the visual big model module, the risk detection big model module and the operator dialogue language big model module corrects its own output information based on the input information.

3. The surgical assistance system according to claim 2, characterized in that: The operator dialogue language large model module also includes a semantic analysis unit; the semantic analysis unit is used to analyze the input information obtained by the interaction unit and correct the input information.

4. The surgical assistance system according to claim 1, characterized in that: The surgery assisting system further comprises a data storage module; the data storage module is configured to store the segmentation result information, the suggestion information and the endoscope content information.

5. The surgical assistance system according to claim 4, characterized in that: The segmentation result information is image data, the suggestion information and the endoscope content information are text data, and the data storage module is configured to combine the segmentation result information, the suggestion information and the endoscope content information with a timestamp to form a multimodal data structure for storage.

6. The surgical assisting system according to claim 5, characterized in that: The surgical assistance system further includes a postoperative evaluation module; the postoperative evaluation module evaluates the surgery based on the multimodal data structure stored in the data storage module, and outputs an evaluation result.

7. The surgical assisting system according to claim 6, characterized in that: The postoperative evaluation module includes a splitting unit and an intraoperative task analysis unit. The splitting unit is used to split the multimodal data structure obtained from the data storage module and send the split data to the intraoperative task analysis unit. The intraoperative task analysis unit evaluates the operation based on the data split by the splitting unit and outputs the evaluation result.

8. The surgical assistance system according to claim 1, characterized in that: The surgical assistance system further includes an endoscope calibration module; the endoscope calibration module is configured to perform the following steps to calibrate the endoscope before surgery: photographing a three-dimensional calibration object to obtain a first image; Performing rough calibration on the endoscope according to the first image and known size information of the three-dimensional calibration object; Based on the rough calibration results, the calibration error is calculated and the position and angle of the area to be optimized are output; Arranging a two-dimensional calibration object according to the position and angle of the area to be optimized, and photographing the two-dimensional calibration object to obtain a second image; The endoscope is precisely calibrated according to the second image and the known size information of the two-dimensional calibration object in combination with the feature information of the three-dimensional calibration object in the rough calibration step.

9. The surgical assisting system according to claim 8, characterized in that: The steps for coarse calibration of an endoscope include: Extracting first feature points in the first image, and obtaining two-dimensional image coordinate information corresponding to each of the first feature points in the first image; Acquire world coordinate information of each of the first feature points according to the size information of the three-dimensional calibration object; The two-dimensional image coordinate information and the world coordinate information of each of the first feature points are matched, a set of equations including the endoscope parameters to be solved is constructed, and the set of equations is solved to obtain a rough calibration result.

10. The surgical assisting system according to claim 9, characterized in that: The feature information of the three-dimensional calibration object in the rough calibration step includes the two-dimensional image coordinate information and world coordinate information of the first feature point; The steps for fine calibration of an endoscope include: Extracting second feature points in the second image, identifying a central feature point and a two-dimensional calibration object coordinate system, and obtaining world coordinate information and two-dimensional image coordinate information of each second feature point based on size information of the two-dimensional calibration object; Combine the two-dimensional image coordinate information and world coordinate information of the first feature point, and the world coordinate information and two-dimensional image coordinate information of the second feature point, and jointly construct a group of equations containing the endoscope parameters to be solved, solve the group of equations, and obtain a precise calibration result.

11. The surgical assisting system according to claim 9, characterized in that: Based on the rough calibration results, the steps to calculate the calibration error include: Based on the roughly calibrated endoscope parameters, mapping the world coordinate information of the first feature point calculated in the rough calibration step to a two-dimensional image coordinate system to obtain virtual two-dimensional image coordinate information of the first feature point; The Euclidean distance between the virtual two-dimensional image coordinate information and the two-dimensional image coordinate information of the first feature point in the first image is calculated, and the Euclidean distance is determined as a calibration error.

12. The surgical assisting system according to claim 11, characterized in that: The steps of outputting the position and angle of the area to be optimized include: Corresponding to the planes of multiple different tilt directions of the three-dimensional calibration object, the calibration errors of different segmented areas in the first image are calculated respectively, and according to a preset threshold, the segmented area whose calibration error exceeds the threshold is selected as the area to be optimized, and the tilt angle of the corresponding plane of the three-dimensional calibration object is determined as the angle of the area to be optimized.

13. The surgical assisting system according to claim 8, characterized in that: The surgical auxiliary system further comprises: an endoscope fixing device, a calibration object fixing device, an object distance adjustment device, a three-dimensional calibration object and a two-dimensional calibration object; The endoscope fixture is used for installing the endoscope; the relative positions of the endoscope fixture and the calibration object fixture are adjustably set by the object distance adjustment device; the three-dimensional calibration object and the two-dimensional calibration object are configured to be detachably mounted on the calibration object fixture.

14. The surgical assisting system according to claim 13, characterized in that: The three-dimensional calibration object includes a plurality of repeated structures, each of which includes a plurality of planes with different tilt directions; and / or the two-dimensional calibration object includes a plane, on which a plurality of first characteristic symbols and a plurality of second characteristic symbols are included.

15. A surgical robot system, characterized in that: include: A surgical assisting system according to any one of claims 1 to 14.

Images